We are interested in how cells in the visual system process information and in the functional organization of the visual system. We use complementary techniques going from psychophysics, functional MRI, to single unit recording.

We have long been interested in how tuning properties of individual neurons can be clustered at a gross level in the brain. We began by looking at the parallel processing of different kinds of visual information, going back and forth between human psychophysics, anatomical interconnectivity of modules in the primate brain, and single unit receptive-field properties. We discovered an interdigitating and highly specific connectivity between functionally distinct regions in V1 and V2 differentially concerned with processing form, color, motion and depth (Livingstone and Hubel, 1984, 1987).

Doris Tsao and Winrich Freiwald used functional MRI to localize face processing regions in the primate brain, and then used functional MRI to target single-unit recording to these functional modules. We found an astonishingly high proportion of cells in these fMRI-identified regions to highly face selective (Tsao et al, 2006).

Targeting an fMRI-identified face patch with a single-unit electrode. (top left) A semi-sagittal section through the right hemisphere of monkey M1 showing three face-selective patches along the STS. The three white arrows point to the lesion left by the recording guide tube. (bottom left) Two coronal slices showing the middle face patch in two monkeys with arrows pointing to the specific region targeted for electrophysiology. (right) Face selectivity of single units in the middle face patch 96 images of faces, bodies, fruits, gadgets, hands, and scrambled patterns (16 images/category. Each row represents one cell and each column one image. The rows have been sorted by the Face Selectivity Index and the columns by image category.

We then went on to characterize the tuning properties of this functional module, finding that face cells tend to be tuned to extremes of face parameters, such as intereye distance, eye size, etc Freiwald et al 2009). This finding suggests why caricatures are so effective in evoking identity.

Tuning of face cells to a cartoon face space. (a) Three example dimensions of a 19-dimensional cartoon space. Each row shows example values for one parameter, with all other parameters fixed at their mean. (b) Tuning curves of two example cells to each of the 19 feature dimensions. Maximal, minimal, and mean values from shift predictor are shown in gray. Stars mark significant modulation.

Currently we are again combining functional MRI and single unit recording to explore the effects of early intensive training on symbol recognition.

In humans both reading and face processing are localized to similar parts of the temporal lobe, but there is a discrepancy between the apparent innateness of face recognition and the unnaturalness of reading. However, most people do have intensive early experience with both faces and symbols. This prompted us to ask whether intensive early experience could cause monkeys to develop anatomical specializations for processing stimuli never naturally used by monkeys. We found that intensive early, but not late, experience caused the formation of category-selective modules in the macaque temporal lobe for stimuli never naturally encountered by monkeys, and behaviorally this experience produced more fluent processing of these stimuli than the same experience later in life. These results suggest a novel hypothesis, based on established work on activity-dependent plasticity in the developing brain, that causally links intensive early experience, expertise, and modular organization of the temporal lobe.

We then used this training-induced artificial domain formation to ask what governs where domains will localize. Surprisingly we found an early organization in the visual system that seems to be based on visual field maps, and provides an explanation for why various domains may end up in the same place in different people.

We are now looking at the normal development of these functional domains in infant monkeys, to find out what the earliest specialization is, what the early proto-map looks like. This is an important question: what aspects of this specialization arise innately and what from experience. Our most recent work is to raise baby monkeys who never see faces, and we will scan them when they are 8-10 months old (when normally reared monkeys develop face patches) to find out how innate the face-processing network is.

A side interest in the lab is to use what we know about vision to understand some of the discoveries artists have made about how we see. The separate processing of color and form information has a parallel in artists' idea that color and luminance play very different roles in art (Livingstone, Vision and Art, Abrams Press, 2002). The elusive quality of the Mona Lisa's smile can be explained by the fact that her smile is almost entirely in low spatial frequencies, and so is seen best by your peripheral vision (Science, 290, 1299). These three images show her face filtered to show selectively lowest (left) low (middle) and high (right) spatial frequencies.

So when you look at her eyes or the background, you see a smile like the one on the left, or in the middle, and you think she is smiling. But when you look directly at her mouth, it looks more like the panel on the right, and her smile seems to vanish. The fact that the degree of her smile varies so much with gaze angle makes her expression dynamic, and the fact that her smile vanishes when you look directly at it, makes it seem elusive.

We have been looking at depth perception in artists, because poor depth perception might be an asset in a profession where the goal is to flatten a 3-D scene onto a canvas. We found evidence that a surprisingly large number of talented artists, including Rembrandt, might be stereoblind (Livingstone and Conway, 2004). In the etching below you can see that Rembrandt portrayed himself as strabismic (with misaligned eyes). If this were the case in only one or two of his self portraits, or if he also showed other subjects with misaligned eyes, we wouldn't think anything of it, but Rembrandt most of the time portrays himself, but not other subjects, as wall-eyed, and the outward deviating eye is reversed in his paintings compared with his etchings (think about it!)