Brain uses universal shortcuts to blend senses, study reveals
Researchers have identified how the brain creates flexible representations that seamlessly merge information from touch, sight, and sound — even when signals arrive at different times. The discovery could transform how companies design VR systems, prosthetics, and brain-computer interfaces that feel more natural and intuitive to users.
Originaltitel: Cross-modal invariants as the foundational basis of emerging brain circuitry representations.
Hjärnan integrerar sinnesintryck från olika modaliteter genom att bygga upp så kallade cross-modal invarianter — representationer som fungerar oberoende av sammanhang. Denna mekanisk förmåga förklarar hur vi kan uppfatta ljud och känsel tillsammans även när de inte inträffar samtidigt, vilket varit kritiskt för överlevnad. Forskare vid École Normale Supérieure i Paris och Lunds universitet visar att unimodala invarianter — baserade på kinetiska strukturer — uppstår i nervsystemet före kortex och har förankring i fysik. Cross-modala invarianter däremot lärs individuellt och domineras av kortexen, vilket gör dem mycket variabler mellan personer och utbredda i kortikala representationer. Detta förklarar varför hjärnans flerkanalintegration varit svår att kartlägga. Insikterna får tillämpningar inom gränssnittskonstruktion och virtual reality, där förståelse av dessa mekanismer möjliggör mer intuitiva människa-maskin-interaktioner.
Our perception of input from one sensory modality is malleable by inputs from other sensory modalities, also when they are non-coincident in time. This function has been critical for survival throughout evolution. How it arises may reveal fundamental principles of brain operation and inform the design of more intuitive human-machine interfaces and Virtual Reality (VR). Here we outline how the brain can acquire what we refer to as cross-modal invariants, brain representations that are applicable across any context to integrate information from multiple modalities. We begin by reviewing evidence that early unimodal brain circuitry processing of tactile inputs in the CNS, before the cortex, is organized around kinematic invariants. Published data is compatible with kinematic invariants in retinal processing, too. We show how such invariants can form natural components of biologically adapted versions of the previously introduced plenhaptic and plenoptic functions, plus a proposed matching plenauditory function. Behavioral observations, we argue, indicate that cross-modal invariants in addition rely on the capability of recording forward information in working memory and therefore this function should be dominated by the cortex. Whereas the unimodal invariants appear to reside in pre-cortical structures and have a ground truth in physics, we explain why cross-modal invariants have not. Therefore cross-modal invariants will need to be individually learned. This likely renders them intelligible only from the point of view of the intrinsic functional organization of the circuitry operations that emerge from that learning. Hence, cross-modal invariants can be expected to be more individually variable and furthermore ubiquitous in the representation space of the cortical circuitry, which can explain why cross-modal integration in the brain has proven so elusive to map out and understand.