Visual Search Is a Fact of Life, and Can Be a Matter of Life or Death

Many of us remember passing time on long, dreary road trips by playing “I SPY” (Figure 1A) or “Airport Scanner” on our cell phones or tablets. Older people—such those in their thirties and forties—may remember “Where’s Waldo?” a popular game found in comic books of their day. These activities are fun, because they make a game out of what we do in real life all the time—that is, look for an object of interest, or a “target,” among distracting things that we do not want, or “distractors.” These games let us experience the sweet frustration of looking for a target and not finding it, or the joy of finding it.

Sometimes, finding a target takes special training and can be a matter of life or death. Doctors train for years to become good at finding cancers in breast X-rays (Figure 1B) or chest X-rays, for example. Similarly, military specialists undergo extensive training to find enemy snipers or landmines camouflaged against the background. Disaster can strike if smart machines, such as driverless cars or medical diagnostic equipment, fail to find the targets they are designed to find. In the wild, animals must spot potential predators quickly enough to flee to safety. On the other hand, a predator may starve to death if it is not good enough at finding its prey, or if its prey finds it too easily. So, you can clearly see that visual search is a very important part of how the brain works in real life.

The Basics of Visual Search

How does the brain find what it is looking for, and how do researchers go about understanding how the brain does this? For one thing, researchers often use simplified, computer-generated visual “scenes” (Figures 1C–E). This is because it is easier to create such images in sufficiently large numbers (imagine having to find thousands of real-world images of camouflaged snipers), and because this makes visual search simpler, which in turn makes it easier to study visual search in the laboratory.

Important early studies performed by Anne Treisman (1935–2018), beginning in the 1960s, used simple images to uncover some basic things about visual search [1, 2]. For her contributions to our understanding of visual search, Dr. Treisman was awarded the National Medal of Science by U.S. President Barack Obama in 2013.

Imagine you are told to find the object that does not belong, or the “odd-man-out,” in Figure 1D. What you are looking for in this figure, the target, is the red bar. In this case, the target can be easily recognized. The red bar “pops out,” and it does so regardless of whether the distractors are blue bars or some other object, like apples. So, a target pops out if it can be distinguished from all the distractors using a basic visual characteristic, or “feature.” In the above example, this distinguishing feature happens to be color, but the distinguishing feature could also be shape, orientation, motion, depth, etc. Treisman found that the number of distractors does not matter in this case. The target can be found just as easily among four distractors as among 40 distractors. In this case, the time it takes for an observer to find the target, or “reaction time,” does not depend on the number of objects in the scene (green line in Figure 1F).

Reaction times are quite different when the target shares some, but not all, of the features of some of the distractors (Figure 1E). In this case, the target does not pop out (“non-pop-out target”), and the reaction times go up according to the number of objects in the visual scene (red line in Figure 1F).

Why does this happen? What is the simplest explanation for both types of reaction time patterns seen in Figure 1F? A critical insight came when Treisman and colleagues closely examined the errors made by the observers. When the observers misperceived what the target was (for instance, when they reported that the odd-man-out was a red vertical bar, when it actually was a red horizontal bar, Figure 1D), they were also likely to misperceive where the target was. This means that, in order to find a target correctly, one also has to perceive the target correctly. Additional experiments showed that mental ability to focus on a particular location in the visual scene—say, the bar at the top left corner of the image—is needed when the target does not pop out, and not needed when it does. This ability is called “spatial attention.” Treisman also showed that, during visual search for a non-pop-out target, we focus attention at a given location to see if that location contains the target, and if it does not, we shift the attention to the next location, and so forth until the target is found. Thus, spatial attention and visual search are closely related.

This model of visual search fully explains the reaction time patterns in Figure 1F. But, this model also explains many of the key features of how we find objects in more complex, real-world scenes [1, 2].

Different Regions in the Brain Play Different Roles in Visual Search

The findings described above tell us that visual search must involve communication between the brain regions involved in recognizing an object (“What is it?”), and the regions involved in focusing attention on a given location (“Where is it?”) and shifting attention from one location to the next (“Where to, next?”). We know something about how the brain carries out each of these pieces, although the way they work together in the process of visual search is not entirely clear yet.

In 1993, Robert Desimone and colleagues used long, ultra-thin wires inserted into the brains of macaque monkeys, to study how the brain responds while an animal performs a visual search task (Figures 2A, B). They found some brain cells that respond differently depending on which one of the two targets the animal was searching for [3]. Since monkey and human brains are so much alike, it is not surprising that very similar processes have since been found to occur in human brains during visual search, as well.

In human subjects, of course, scientists cannot introduce wires into the brain. Much of what we know about how the human brain performs visual search comes from a technique called functional magnetic resonance imaging (fMRI; Figure 2C). fMRI measures the brain activity indirectly, by looking at the changes in blood flow in the brain that happen when brain cells become active. Many fMRI studies have examined how the brain regions involved in attention respond when humans perform visual search. As expected from the earlier studies we have described different brain regions are indeed specialized for various parts of visual search [4]. For instance, some regions of the brain are preferentially active during the visual search (red arrows in Figure 2D), suggesting that they are specialized for shifting attention from one search location to the next. Some other regions of the brain are active when the subject finds the target, suggesting that these regions might help with focusing attention on the object when the subject is looking at one specific search location. How these various brain regions work together to bring about visual search and target detection remains to be understood.

Damage to Certain Brain Regions Can Make Visual Search More Difficult

If various brain regions specialized in various parts of the visual search process, as we described above, it makes sense that when one or more of these brain regions are damaged (due to stroke, trauma, tumor, etc.), visual search is also affected. This is indeed the case [5]. For instance, patients with damage to certain regions of the right half of the brain fail to find objects in the left half of the image, a phenomenon known as “hemineglect” (Figure 3F). This is because the right half of the brain handles information from the left side of the image. Interestingly, damage to the same regions in the left half of the brain does not result in any hemineglect, for reasons that remain largely unclear.

Summary

Visual search is a large part of how we see in the real world. We understand a good deal about why some objects of interest are easier to find than others. But much remains to be discovered about how the brain works during visual search. Still, it is quite clear that most parts of the brain that play a role in vision also participate in visual search, probably because visual search is such a big part of vision. Thus, damage to some parts of the brain can make visual search more difficult. In the future, computers that can perform successful visual search—such as those that can better detect things like enemies in the battlefield, cancer in X-rays, or potential hazards in airport baggage—are likely to be a bigger part of our lives.

Author Contributions

IN and AL researched the subject, obtained the requisite copyright permissions, and wrote the manuscript under the supervision of JH. All authors contributed to manuscript revision, read and approved the submitted version.

Glossary

Visual Search: ↑ Task that involves actively scanning an environment for a particular object or feature.

Odd-Man-Out: ↑ The object that does not belong.

Attention: ↑ Process of concentrating on a specific aspect of information.

fMRI (Functional Magnetic Resonance Imaging): ↑ Imaging that measures brain activity by detecting changes in blood flow.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

IN and AL were supported by the Undergraduate Research Apprenticeship Program of the U.S Army Educational Outreach Program. Research in JH’s laboratory has been supported by the U.S. Army grants W911NF-11-1-0105 and W911NF-15-1-0311, also also by grants from the National Institutes of Health and the National Science Foundation. We thank Heather Nall for her scrutiny of the manuscript.