09/06/2013 01:45 pm ET

McGurk Effect: When Seeing Trumps Hearing, What Is Going On In The Brain?


Listen to this.

In the brain, what we see has an effect on what we hear -- and in some cases, what we see can actually override what we hear. This concept is called the McGurk effect (named after the psychologist Harry McGurk) -- and now a new study may shed some light on why it occurs.

"We've shown neural signals in the brain that should be driven by sound are being overridden by visual cues that say, 'Hear this!'" study researcher Bradley Greger, a bioengineer at the University of Utah, said in a statement. "Your brain is essentially ignoring the physics of sound in the ear and following what's happening through your vision."

The study, published in the journal PLOS ONE, involved recording brain electrical signals from two men and two women who had epilepsy. They watched and listened to videos of a person's mouth; the person said "ba," "va," "ga" and "tha."

Three different videos were viewed. In one of them, the sound and the mouth movements matched (so if the mouth showed a person saying "ba," the sound that the study participants heard was also "ba"). In another video, the sound clearly did not correspond with the movement of the mouth (so if the mouth showed "ga," the sound the participants heard was "tha"). In the third video, the sound was only slightly off of what was demonstrated with the mouth (so if the mouth showed a person saying "ba," the sound the participants heard was "va," which might look very similar speaking-wise to "ba") -- a demonstration of the McGurk effect.

As the study participants watched these three different videos, researchers looked at brain activity in regions correlated with auditory and visual processing. They found that when the participants watched the second video (where the sound clearly didn't match with the video), brain activity linked with sound increased. But when the study participants watched the third video, where the sound and the video were slightly off, the brain activity pattern shifted to be more visual-focused.