Personal Blogs

What You See Is What You Hear

Equipment

1. Any sound recorder: phone or computer. Anything that will do the job. You need recording and playback.

2. A mirror to see your lips.

Experiment

Step 1.

Start the recorder. As distinctly as you can, and at short regular intervals, record yourself saying rhythmically, with the emphasis on the ‘B’.

‘Bat’, ‘Bat’, ‘Bat’, ‘Bat’, ‘Bat’, ‘Bat’, …

Do this for 30 seconds or so.

Step 2.

Please read through the steps below before going on.

1. Get the mirror. Look at your lips, as though you were watching the lips of a language teacher.

2. Play back the recording. If you used a smartphone switching to speaker is better than holding the phone up.

As you listen to the playback, silently mouth, rather than speak, the syllables below over and over again, lip-synching with the recording. Try to make very clear lip movements. Imagine you are speaking for an audience who are deaf and rely on lip reading to understand you.

As you mouth the syllables gaze intently at your lips in the mirror. Listen carefully to what you hear. (You’ll find you can do these three things at once quite easily.)

These are the syllables you have to mouth.

‘That Fat Bat, That Fat Bat, That Fat Bat, …’

You will almost certainly hear ‘that fat bat, that fat bat, …’, and not what you were really saying at all. When you see your lips shaping the consonants Th and F it’s impossible to hear B, whatever your ears pick up. Remember you really just said ‘bat bat bat bat bat…’ and that’s the sound reaching your ears but it’s not what you hear.

Background

This illusion is known as the McGurk effect. There is an excellent Horizon clip about it here.

Notice that it’s immaterial whether you are looking at your own lips or another person’s. Usually people watch a video like the Horizon one but as you’ve seen (or heard) seeing and hearing yourself works equally well.

Reflection

What surprises me even more is this. If I do the experiment a few times running and then only listen to the sound on its own, no lips, I still hear ‘that fat bat…’, so the effect has persistence!

The McGurk effect is all the more remarkable because we process sound quicker than vision. But what we hear – the interpretation we place on the sounds –  can be delayed and modified by related visual information.