{"id":432,"date":"2023-03-02T20:16:49","date_gmt":"2023-03-02T20:16:49","guid":{"rendered":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/chapter\/learn-it-multimodal-phenomena\/"},"modified":"2025-12-29T23:42:53","modified_gmt":"2025-12-29T23:42:53","slug":"learn-it-multimodal-phenomena","status":"publish","type":"chapter","link":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/chapter\/learn-it-multimodal-phenomena\/","title":{"raw":"Perception and Illusions: Learn It 2\u2014Multimodal Phenomena","rendered":"Perception and Illusions: Learn It 2\u2014Multimodal Phenomena"},"content":{"raw":"

Multimodal Perception: How Our Senses Work Together<\/strong><\/h2>\r\n

Although it has been traditional to study the various senses independently, most of the time, perception operates in the context of information supplied by multiple sensory modalities<\/strong> at the same time.<\/p>\r\n

\r\n

unimodal and multimodal perception<\/h3>\r\n
    \r\n\t
  • Unimodal perception<\/strong> is when we use information from just one sense, like seeing with our eyes or hearing with our ears. For example, when we look at a picture, we are using only our sense of vision.<\/li>\r\n\t
  • Multimodal perception<\/strong> is when we use information from multiple senses at the same time to understand the world around us. For example, when we hear a sound and see where the sound is coming from, we are using both our sense of hearing and vision.<\/li>\r\n<\/ul>\r\n<\/section>\r\n

    For example, imagine if you witnessed a car collision. You could describe the stimulus generated by this event by considering each of the senses independently; that is, as a set of unimodal<\/strong> stimuli. Your eyes would be stimulated with patterns of light energy bouncing off the cars involved. Your ears would be stimulated with patterns of acoustic energy emanating from the collision. Your nose might even be stimulated by the smell of burning rubber or gasoline.<\/p>\r\n

    Indeed, unless someone were to explicitly ask you to describe your perception in unimodal terms, you would most likely experience the event as a unified bundle of sensations from multiple senses. In other words, your perception would be multimodal<\/strong>. The question is whether the various sources of information involved in this multimodal stimulus are processed separately by the perceptual system or not.<\/p>\r\n

    For the last few decades, perceptual research has pointed to the importance of multimodal perception<\/strong>: the effects on the perception of events and objects in the world that are observed when there is information from more than one sensory modality. Most of this research indicates that, at some point in perceptual processing, information from the various sensory modalities is integrated<\/strong>. In other words, the information is combined and treated as a unitary representation of the world.<\/p>\r\n

    Why Multimodal Perception Matters<\/strong><\/h2>\r\n

    Recent neuroscience has identified many brain regions that respond to multiple types of sensory input<\/strong>, suggesting that humans are fundamentally multimodal perceivers<\/strong> (Spence, Senkowski, & Roder, 2009). This explains why:<\/p>\r\n

      \r\n\t
    • Movie soundtracks feel emotionally powerful<\/li>\r\n\t
    • Virtual reality seems immersive<\/li>\r\n\t
    • Food tastes bland when you have a cold<\/li>\r\n\t
    • Loud sounds appear \u201cbrighter,\u201d and bright flashes appear \u201clouder\u201d<\/li>\r\n<\/ul>\r\n

      Our brains are wired to create one coherent world<\/strong>, not separate streams of vision, sound, and touch.<\/p>\r\n

      Behavioral Effects: Multimodal vs. Crossmodal Phenomena<\/strong><\/h2>\r\n

      Psychologists study two major categories of multisensory effects:<\/p>\r\n

      \r\n

      multimodal and crossmodal phenomena<\/h3>\r\n
        \r\n\t
      • Multimodal phenomena <\/strong>concern the binding together of inputs from multiple sensory modalities and the effects of this binding on perception.<\/li>\r\n\t
      • Crossmodal phenomena <\/strong>concern the influence of one sensory modality on the perception of another (Spence, Senkowski, & Roder, 2009).<\/li>\r\n<\/ul>\r\n<\/section>\r\n

        Multimodal Phenomena<\/h2>\r\n

        Audiovisual Speech Perception<\/strong><\/h3>\r\n

        Speech is naturally multimodal: when someone talks, they produce sound waves and visual mouth movements. Watching a speaker\u2019s lips can dramatically improve comprehension, especially in noisy environments.<\/p>\r\n

        Sumby and Pollack (1954) showed that in loud background noise, seeing the speaker\u2019s mouth movements improves word recognition more than doubling the signal-to-noise ratio. In other words, watching the speaker can make speech clearer than simply turning up the volume.<\/p>\r\n

        \r\n

        One of the earliest investigations of this question examined the accuracy of recognizing spoken words presented in a noisy context, much like in the example above about talking at a crowded party. To study this phenomenon experimentally, some irrelevant noise (\u201cwhite noise\u201d\u2014which sounds like a radio tuned between stations) was presented to participants. Embedded in the white noise were spoken words, and the participants\u2019 task was to identify the words. There were two conditions: one in which only the auditory component of the words was presented (the \u201cauditory-alone\u201d condition), and one in both the auditory and visual components were presented (the \u201caudiovisual\u201d condition). The noise levels were also varied, so that on some trials, the noise was very loud relative to the loudness of the words, and on other trials, the noise was very soft relative to the words.<\/p>\r\n<\/section>\r\n

        Most people assume that deaf individuals are much better at lipreading than individuals with normal hearing. It may come as a surprise to learn, however, that some individuals with normal hearing are also remarkably good at lipreading (sometimes called \u201cspeechreading\u201d). In fact, there is a wide range of speechreading ability in both normal hearing and deaf populations (Andersson et al., 2001). However, the reasons for this wide range of performance are not well understood (Auer & Bernstein, 2007; Bernstein, 2006; Bernstein et al., 2001; Mohammed et al., 2005).<\/p>\r\n

        This improvement follows the principle of inverse effectiveness<\/strong>: the brain benefits from multisensory information most when each individual sense is degraded. You might have noticed this phenomenon when turning captions on to watch a show.<\/p>\r\n

        Another phenomenon using audiovisual speech is a very famous illusion called the \u201cMcGurk effect\u201d (named after one of its discoverers). In the classic formulation of the illusion, a movie is recorded of a speaker saying the syllables \u201cgaga.\u201d Another movie is made of the same speaker saying the syllables \u201cbaba.\u201d Then, the auditory portion of the \u201cbaba\u201d movie is dubbed onto the visual portion of the \u201cgaga\u201d movie. This combined stimulus is presented to participants, who are asked to report what the speaker in the movie said. McGurk and MacDonald (1976) reported that 98 percent of their participants reported hearing the syllable \u201cdada\u201d\u2014which was in neither the visual nor the auditory components of the stimulus. These results indicate that when visual and auditory information about speech is integrated, it can have profound effects on perception.