Decoding the representation of multiple simultaneous objects in human occipitotemporal cortex.
ABSTRACT: Previous work using functional magnetic resonance imaging has shown that the identities of isolated objects viewed by human subjects can be extracted from distributed patterns of brain activity. Outside the laboratory, however, objects almost never appear in isolation; thus it is important to understand how multiple simultaneously occurring objects are encoded by the visual system. We used multivoxel pattern analysis to examine this issue, testing whether activity patterns in the lateral occipital complex (LOC) evoked by object pairs showed an ordered relationship to patterns evoked by their constituent objects. Applying a searchlight analysis to identify voxels with the highest signal-to-noise ratios, we found that responses to object pairs within these informative voxels were well predicted by the averages of responses to their constituent objects. Consistent with this result, we were able to classify object pairs by using synthetic patterns created by averaging single-object patterns. These results indicate that the representation of multiple objects in LOC is governed by a response normalization mechanism similar to that reported in visual areas of several nonhuman species. They also suggest a population coding scheme that preserves information about multiple objects under conditions of distributed attention, facilitating fast object and scene recognition during natural vision.
Project description:Mammals are highly skilled in rapidly detecting objects in cluttered natural environments, a skill necessary for survival. What are the neural mechanisms mediating detection of objects in natural scenes? Here, we use human brain imaging to address the role of top-down preparatory processes in the detection of familiar object categories in real-world environments. Brain activity was measured while participants were preparing to detect highly variable depictions of people or cars in natural scenes that were new to the participants. The preparation to detect objects of the target category, in the absence of visual input, evoked activity patterns in visual cortex that resembled the response to actual exemplars of the target category. Importantly, the selectivity of multivoxel preparatory activity patterns in object-selective cortex (OSC) predicted target detection performance. By contrast, preparatory activity in early visual cortex (V1) was negatively related to search performance. Additional behavioral results suggested that the dissociation between OSC and V1 reflected the use of different search strategies, linking OSC preparatory activity to relatively abstract search preparation and V1 to more specific imagery-like preparation. Finally, whole-brain searchlight analyses revealed that, in addition to OSC, response patterns in medial prefrontal cortex distinguished the target categories based on the search cues alone, suggesting that this region may constitute a top-down source of preparatory activity observed in visual cortex. These results indicate that in naturalistic situations, when the precise visual characteristics of target objects are not known in advance, preparatory activity at higher levels of the visual hierarchy selectively mediates visual search.
Project description:We used functional magnetic resonance imaging (fMRI) to demonstrate the existence of a mechanism in the human lateral occipital (LO) cortex that supports recognition of real-world visual scenes through parallel analysis of within-scene objects. Neural activity was recorded while subjects viewed four categories of scenes and eight categories of 'signature' objects strongly associated with the scenes in three experiments. Multivoxel patterns evoked by scenes in the LO cortex were well predicted by the average of the patterns elicited by their signature objects. By contrast, there was no relationship between scene and object patterns in the parahippocampal place area (PPA), even though this region responds strongly to scenes and is believed to be crucial for scene identification. By combining information about multiple objects within a scene, the LO cortex may support an object-based channel for scene recognition that complements the processing of global scene properties in the PPA.
Project description:Abstract The lateral occipital complex (LOC) is a set of areas in the human occipito-temporal cortex responding to objects as opposed to low-level control stimuli. Conventional functional magnetic resonance imaging (fMRI) analysis methods based on regional averages could not detect signals discriminative of different types of objects in this region. Here, we examined fMRI signals using multivariate pattern recognition (support vector classification) to systematically explore the nature of object-related information available in fine-grained activity patterns in the LOC. Distributed fMRI signals from the LOC allowed for above-chance discrimination not only of the category but also of within-category exemplars of everyday man-made objects, and such exemplar-specific information generalized across changes in stimulus size and viewpoint, particularly in posterior subregions. Object identity could also be predicted from responses of the early visual cortex, even significantly across the changes in size and viewpoint used here. However, a dissociation was observed between these two regions of interest in the degree of discrimination for objects relative to size: In the early visual cortex, two different sizes of the same object were even better discriminated than two different objects (in accordance with measures of pixelwise stimulus similarity), whereas the opposite was true in the LOC. These findings provide the first evidence that direct evoked fMRI activity patterns in the LOC can be different for individual object exemplars (within a single category). We propose that pattern recognition methods as used here may provide an alternative approach to study mechanisms of neuronal representation based on aspects of the fMRI response independent of those assessed in adaptation paradigms.
Project description:To optimize processing, the human visual system utilizes regularities present in naturalistic visual input. One of these regularities is the relative position of objects in a scene (e.g., a sofa in front of a television), with behavioral research showing that regularly positioned objects are easier to perceive and to remember. Here we use fMRI to test how positional regularities are encoded in the visual system. Participants viewed pairs of objects that formed minimalistic two-object scenes (e.g., a "living room" consisting of a sofa and television) presented in their regularly experienced spatial arrangement or in an irregular arrangement (with interchanged positions). Additionally, single objects were presented centrally and in isolation. Multi-voxel activity patterns evoked by the object pairs were modeled as the average of the response patterns evoked by the two single objects forming the pair. In two experiments, this approximation in object-selective cortex was significantly less accurate for the regularly than the irregularly positioned pairs, indicating integration of individual object representations. More detailed analysis revealed a transition from independent to integrative coding along the posterior-anterior axis of the visual cortex, with the independent component (but not the integrative component) being almost perfectly predicted by object selectivity across the visual hierarchy. These results reveal a transitional stage between individual object and multi-object coding in visual cortex, providing a possible neural correlate of efficient processing of regularly positioned objects in natural scenes.
Project description:Repeated exposure to the same stimulus results in an attenuated brain response in cortical regions that are activated during the processing of that stimulus. This phenomenon, called repetition suppression (RS), has been shown to be modulated by expectation. Typically, this is achieved by varying the probability of stimulus repetitions (Prep) between blocks of an experiment, generating an abstract expectation that 'things will repeat'. Here, we examined whether stimulus-specific expectations also modulate RS. We designed a task where expectation and repetition are manipulated independently, using stimulus-specific expectations. We investigated to which extent such stimulus-specific expectations modulated the visual evoked response to objects in lateral occipital cortex (LOC) and primary visual cortex (V1), using functional magnetic resonance imaging (fMRI). In LOC, we found that RS interacted with expectation, such that repetition suppression was more pronounced for unexpected relative to expected stimuli. Additionally, we found that the response of stimulus-preferring voxels in V1 was generally decreased when stimuli were expected. These results suggest that stimulus-specific expectations about objects modulate LOC and propagate back to the earliest cortical station processing visual input.
Project description:Mental imagery is a critical cognitive function, clinically important, but poorly understood. When visual objects are perceived, many of their sensory, semantic and emotional properties are represented in occipitotemporal cortex. Visual imagery has been found to activate some of the same brain regions, but it was not known what properties are re-created in these regions during imagery. We therefore examined the representation during imagery for two stimuli in depth, by comparing the pattern of fMRI response to the patterns evoked by the perception of 200 diverse objects chosen to de-correlate their properties. Real-time, adaptive stimulus selection allowed efficient sampling of this broad stimulus space. Our experiments show that occipitotemporal cortex, which encoded sensory, semantic and emotional properties during perception, can robustly represent semantic and emotional properties during imagery, but that these representations depend on the object being imagined and on individual differences in style and reported vividness of imagery.
Project description:Object interaction requires knowledge of the weight of an object, as well as its shape. The lateral occipital complex (LOC), an area within the ventral visual pathway, is well known to be critically involved in processing visual shape information. Recently, however, LOC has also been implicated in coding object weight before grasping-a result that is surprising because weight is a nonvisual object property that is more relevant for motor interaction than visual perception. Here, we examined the causal role of LOC in perceiving heaviness and in determining appropriate fingertip forces during object lifting. We studied perceptions of heaviness and lifting behavior in a neuropsychological patient (M.C.) who has large bilateral occipitotemporal lesions that include LOC. We compared the patient's performance to a group of 18 neurologically healthy age-matched controls. Participants were asked to lift and report the perceived heaviness of a set of equally weighted spherical objects of various sizes-stimuli which typically induce the size-weight illusion, in which the smaller objects feel heavier than the larger objects despite having identical mass. Despite her ventral stream lesions, M.C. experienced a robust size-weight illusion induced by visual cues to object volume, and the magnitude of the illusion in M.C. was comparable to age-matched controls. Similarly, M.C. evinced predictive fingertip force scaling to visual size cues during her initial lifts of the objects that were well within the normal range. These single-case neuropsychological findings suggest that LOC is unlikely to play a causal role in computing object weight.
Project description:To determine how responses evoked by natural odorant mixtures compare to responses evoked by individual odorant chemicals, we mapped 2-deoxyglucose uptake during exposures to vapors arising from a variety of odor objects that may be important to rodents in the wild. We studied 21 distinct natural odor stimuli ranging from possible food sources such as fruits, vegetables, and meats to environmental odor objects such as grass, herbs, and tree leaves. The natural odor objects evoked robust and surprisingly focal patterns of 2-deoxyglucose uptake involving clusters of neighboring glomeruli, thereby resembling patterns evoked by pure chemicals. Overall, the patterns were significantly related to patterns evoked by monomolecular odorant components that had been studied previously. Object patterns also were significantly related to the molecular features present in the mixture components. Despite these overall relationships, there were individual examples of object patterns that were simpler than might have been predicted given the multiplicity of components present in the vapors. In these cases, the object patterns lacked certain responses evoked by their major odorant mixture components. These data suggest the possibility of mixture response interactions and provide a foundation for understanding the neural coding of natural odor stimuli.
Project description:Regularities are gradually represented in cortex after extensive experience, and yet they can influence behavior after minimal exposure. What kind of representations support such rapid statistical learning? The medial temporal lobe (MTL) can represent information from even a single experience, making it a good candidate system for assisting in initial learning about regularities. We combined anatomical segmentation of the MTL, high-resolution fMRI, and multivariate pattern analysis to identify representations of objects in cortical and hippocampal areas of human MTL, assessing how these representations were shaped by exposure to regularities. Subjects viewed a continuous visual stream containing hidden temporal relationships--pairs of objects that reliably appeared nearby in time. We compared the pattern of blood oxygen level-dependent activity evoked by each object before and after this exposure, and found that perirhinal cortex, parahippocampal cortex, subiculum, CA1, and CA2/CA3/dentate gyrus (CA2/3/DG) encoded regularities by increasing the representational similarity of their constituent objects. Most regions exhibited bidirectional associative shaping, whereas CA2/3/DG represented regularities in a forward-looking predictive manner. These findings suggest that object representations in MTL come to mirror the temporal structure of the environment, supporting rapid and incidental statistical learning.
Project description:The extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model, as well as object representations, are more widely distributed across the brain than previously acknowledged and that functional searchlight can improve model-based similarity and decoding accuracy. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.