What Do We Know About The Senses?

In school, you learned about the five senses: sight, hearing, taste, touch, and smell. Maybe you even learned how each sense sends a message to a specific area in the brain for processing—that there is a vision center, a hearing center, and so on. But recent research suggests that this simple division is not entirely true and definitely does not paint the whole picture [1]. Studies reveal hidden connections between the senses, and these connections serve as gates that transmit information from one sense to another to allow the senses to be “blended” together. These gates allow researchers to “hack” the brain (only with the consent of the participants and for a positive purpose, of course) to improve and expand the classical senses.

We get a glimpse into these hidden sensory connections in people who have synesthesia, a conditions in which the brain mixes up the senses. Strong synesthesia, such as seeing colors when reading the letters of the alphabet or seeing shapes when listening to music, is found in a small percentage of the population. But a milder form of synesthesia is much more common, in which noises like whispering, tapping, or crinkling paper generate pleasant current or touch-like sensations. So, what can be done with these hidden connections between the senses?

Hearing Through Your Fingers

Have you ever listened to a song and felt the beat in your body? The ability to feel sounds is well-documented in nature (Figures 1A,B), and some creatures are even known to hear without having ears! This ability has its origins in the nature of sound itself. Sounds are basically waves that travel through the air and enter the ears. The waves hit the eardrums, which jiggle and send the vibrations to a structure in the middle ear, composed of three tiny bones. These bones send the vibrations to the inner ear, which is the point where the mechanical vibrations of sound become an electrical signal, which is sent to the brain through the nerves. The inner ear is filled with fluid and lined with microscopic hairs. When sound wave vibrations enter the inner ear and create waves in this fluid, the hair cells move, which generates an electrical signal. The signal is then sent to the brain’s auditory cortex via the auditory nerve.

Vibration is the key to hearing. Our lab constructed a device that can convert speech and other sounds to vibrations that are felt on the fingers (Figure 1C). We discovered that these vibrations to the fingers enhanced people’s ability to hear the speech or sound, providing an immediate increase in hearing in noisy environments! In other words, their hearing capabilities increased significantly, without any training. In addition, after an hour of training with this device, people’s abilities to hear in noisy environments doubled (Figures 1D,E) [2].

The initial results of our vibration experiments also indicate that areas in the brain that normally respond to hearing begin to respond to vibrations. We hope these findings will lead to innovations that will help the deaf and hearing-impaired to hear, in addition to helping all of us hear better in noisy environments. Could this technology be used to help people learn a second language? Could it be used in gaming or for immersion in virtual-reality environments? The implications could be unlimited!

Hearing Temperature

Have you ever sensed the heat of the shower before getting in, by only the sound of the running water? Research shows that you probably have but did not know it! When asked as part of a questionnaire, nearly all respondents stated they could not hear temperature. And yet, when we tested this ability by presenting people with sounds of water and asking them to choose whether the water they were hearing was hot or cold, they could identify hot or cold water with surprising accuracy.

Sensing temperature is another well-known phenomenon in nature. Snakes are known to hunt at night thanks to their ability to see heat radiating off their prey (Figures 2A,B). There is even a law called Dolbear’s law that formalizes the relationship between crickets chirping and the temperature in their surroundings—they chirp faster the warmer it is. Research from our lab shows that people also learn to associate the temperature of water with the sound it makes when it is poured, without ever being taught (Figure 2C).

How do we acquire this ability? Probably through our daily multisensory interactions with pouring water. Think of how many times each day you hear water while simultaneously experiencing the temperature of it. For example, when you pour yourself some ice-cold water or make some boiling hot tea, jump into a cold swimming pool or take a hot shower in the morning. These findings made us wonder whether we could actively speed up the ability to “hear” temperature with perceptual learning (Figure 2D). How good could someone become at hearing temperatures if they were trained to do it? This is a subject we are currently researching.

Creating New “Super senses”

What if your teacher really could see the pranks you play in class behind her back? Our lab has been studying whether we can expand the visual field to 360°, giving people the ability to see what is in front of them using their eyes, while simultaneously seeing what is behind them using another sense. This study showed that people acquire surprisingly high levels of accuracy “seeing” behind their backs when the visual information is conveyed as sounds using a sensory substitution device (SSD) (Figures 3A,B). This device captures a visual image using a camera and then translates the visual information into sounds, which are sent to the ears like music (Figure 3C) [3]. Our research showed that using this SSD activates the visual areas in the brain after lots of practice—showing that these areas are not used only for seeing with the eyes (Figure 3D) [4, 5]!

In 1981, David Hubel and Torsten Wiesel won the Nobel Prize for the theory of critical periods, which states that the development of the senses, for example, the development of normal vision or hearing, depends on the experience of visual or auditory stimuli in early childhood. They discovered that if animals did not experience a certain sense in the 1st weeks of life—the critical period—they could not develop that sense later in life. This idea is now being challenged by work done in several labs, including ours. For example, studies on children after cataract removal indicate that vision can be developed at a much older age than once was thought, so maybe there are multiple critical periods instead of only one! Moreover, if people can learn to use alternate senses, for example to “see” with their ears or “hear” through touch, or if people can learn to develop entirely new senses even in adulthood, this indicates that strict critical periods might not exist [6].

It seems that the more we discover about exactly how the brain is wired, the more we uncover fascinating connections between the senses that can be used to improve existing abilities and possibly to create new ones. At the beginning of the twentieth century, American psychologist William James stated that “We use only a small part of our possible mental and physical resources.” The race to discover the unused resources in the brain has already begun. Who knows what we will be able to sense in the future? With the help of technology, combined with perceptual learning, we are on the verge of becoming humans with super abilities!

Glossary

Synesthesia: ↑ A phenomenon in which stimulation of one sense leads to stimulation of another sense, resulting for example in seeing shapes or colors when hearing music.

Auditory Cortex: ↑ An area of the brain where auditory information is usually believed to be processed.

Perceptual Learning: ↑ Improvements in sensory abilities resulting from experience or training.

Sensory Substitution Device: ↑ A device that delivers information normally acquired by one sense (for example vision) through another (for example audition or touch).

Critical Period: ↑ A time period in an animal’s early life during which it must experience sensory stimuli so that its senses will develop normally.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This research was supported by an ERC Consolidator Grant (773121 NovelExperiSense) and a Horizon GuestXR (101017884) grant to AA.