What is Visual Search?

Have you ever lost something you needed and then had a difficult time trying to find it? Maybe you needed your backpack for school, but it was hidden somewhere in your room under a pile of clothes and toys. The act of looking for something you need (or something important) is called visual search. And some people actually search for things for a living! Not your backpack, of course. But important things like weapons or medical problems.

Many professional visual search tasks involve looking for things on computer screens or other two-dimensional (2D) displays. For instance, airport security screeners look at x-rays on computer screens to find prohibited items in travelers’ luggage, and doctors routinely scan through medical images to make sure their patients are okay. When scientists study the process of visual search, they also usually use 2D displays, and this makes sense because if they want to understand how to make people better at visual search professions, it is important to create laboratory experiments that are similar to the kinds of tasks that medical and security screeners routinely perform.

But the real world around us is three-dimensional (3D), and it can be cluttered or even downright messy! That means that the items we might be looking for may not be immediately visible—they might be partially or completely blocked by other objects. What do we do in those scenarios? We often use our hands to move things out of the way, or to rearrange a space to make things easier to find.

Imagine walking around your home and trying to find various things. Some will be easy to spot using only your eyes. In Figures 1A,B, for instance, try to find the T-rex LEGO set or a sleepy dog. If you need to find something on a shelf or figure out where your lazy pooch decided to take a nap, you probably do not need to use your hands. But imagine that you must dig a pencil out of a messy drawer or spot a particular LEGO piece hidden among many others (Figures 1C,D). In these scenarios, you might be able to search using only your eyes, but it would be much more efficient to conduct an interactive search—that is, a search using your eyes and your hands. Surely, finding your pencil is easier if you can feel for the wood of the pencil and move other items around and out of the way as you go.

Although some professional visual searchers typically look at 2D screens, many of them do conduct interactive searches. For instance, police investigators may have to uncover objects at the scene of a crime while they look for evidence; search-and-rescue responders may brush aside plant matter while looking for the footprints of a missing hiker; and a concert security guard may rifle through an audience member’s bag to make sure they are not bringing in something that they should not. These are important professions too, so if we want to understand 3D searches better and figure out ways to improve these people’s job performance, we must create laboratory tasks that mimic interactive searches.

Creating an Interactive 3D Search Task in the Laboratory

Unfortunately, scientists spend a lot of time studying 2D search, but they do not spend much time studying 3D search. So, to study interactive search, we had to design our own task for participants in our laboratory—one that was similar to 3D interactive searches in the real world. We needed to make sure that everyone did the task under the same conditions, so that we could take careful measurements of their performance. We needed a task in which there would be some targets (things that the participant was told to find) and lots and lots of distractors (other items that are not the target). For instance, if you are searching for a pencil in your desk, the pencil is the target and everything else in the drawer is a distractor. We also needed a task in which the objects could overlap, and the participants had to have the ability to move items around to get distractors out of the way when they needed to. Because we are all fans of building with LEGO bricks, we thought that having people search for LEGO targets among other distractor bricks would be a perfect way to mimic real-world interactive searches (Figure 2). We also knew that we could design a task like this very carefully and observe participants’ performance very precisely.

First, we explained the task to the participant and told them that they would be searching through trays of LEGO bricks, and that they were allowed to use their hands to help them search. We then took a set of 150 bricks and randomly scattered them on a large white tray; while we did this, there was a screen in front of the tray to make sure that the participant did not peek and get a head start! Next, the participant was handed a sample target brick (sometimes a small brick and other times a large one), and was told that their task was to find as many of those bricks as they could. When they had memorized the target and were ready to begin, we removed the screen and started a timer. Every time the participant found one of the bricks, they deposited it in a little bin (so that we could keep track) and the computer recorded how quickly the item was found. The participant searched and moved items around on the tray until they thought they had found all the targets, in which case they shouted “I am done!” and the search was over. Each participant did this several times over the course of an hour, searching for different targets on each new tray.

Active vs. Passive Search Strategies

We not only wanted to study how people conduct interactive searches, but also to explore whether we could make people better at this task. So, we asked some of our participants to try controlling their attention while they performed their searches. For one group (the uninstructed searchers), we did not give them any specific instructions, we just let them search however they wanted to. For the second group (the active searchers), we told participants to try hard to direct their attention and to actively search for the targets. You can think of this as an effortful way of searching, in which you are constantly trying to direct your eyes to where the target might be. For the third group (the passive searchers), we told them to try to let the targets “pop into mind” instead of actively seeking them out. You can think of this as a more relaxed way of searching, in which you are just casually letting your eyes move around the tray. In our previous work [1], we gave people the same sorts of instructions before they performed a search on a 2D computer screen (looking for pictures of things), and we found that a relaxed, passive approach to controlling attention actually made people better searchers. But we wondered if that strategy also worked with 3D interactive search!

What Did We Find?

In this study [2], we found exactly the opposite of what we found using computer screen searches! (Figure 3). We measured performance by recording the time it took participants to find each target. Not surprisingly, participants found the larger targets more quickly than the smaller ones. But, more importantly, we found that participants using an active attention strategy were faster than participants using a passive strategy. So, taking a relaxed approach to searching might help you search on a 2D screen, but our findings show that it is less useful when you are conducting a 3D interactive search with your hands.

It might seem to you that an active strategy would be the “default mode” of searching; that is, the way you would search even if you were not given instructions about how to do it. But our findings suggest that trying to control your attention in an active way makes you a better interactive searcher—participants in the active condition found their targets faster than the uninstructed searchers, too!

Conclusion

What can we conclude from this study? It is important to remember that this was only one experiment, so it does not mean that using an active strategy is always the best way to conduct 3D interactive search. But our work is a good start toward better understanding interactive search through careful laboratory experiments. When the search environment is cluttered and the items look a lot alike, and when the searcher can use their hands to move things around, it seems that an active attention strategy is beneficial. This might be useful to try out the next time you do a jigsaw puzzle, for example, and need to find just the right piece amongst all the other pieces!

Future work will be needed to create tasks that are even more similar to the professional scenarios that we are interested in, and to examine how controlling your attention can make you better at finding important search targets. Although our lab task was more realistic than searching for things on a computer screen, it still differs in many ways from the sorts of interactive search tasks that professionals like security guards, police officers, or forensic experts use in their daily activities. So, future experiments must be devised to more closely match the characteristics of those tasks, and to explore search performance in increasingly realistic scenarios. That will be the best way to understand whether using an active strategy or a passive one is the optimal way to find a search target. For now, just remember that an active strategy might be useful the next time you have to find that one missing LEGO brick!

Glossary

Visual Search: ↑ The act of looking for something that may be hidden or lost among other items.

Interactive Search: ↑ A type of visual search in which the observer uses their hands (or some other tool) to interact with the search environment, through moving items around or rearranging the space.

Targets: ↑ The item you are looking for in a visual search.

Distractors: ↑ The non-target items that may be present during a visual search.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.