Seeing Beyond Sight: Multisensory Inference under Degraded Visual Input

Author: Chan, Yeuk Chin Ailene

Year: 2026

Degree: Dissertation (Ph.D.)

Advisor: Shimojo, Shinsuke

Committee Members: O'Doherty, John P.; Rutishauser, Ueli; Shams, Ladan; Shimojo, Shinsuke

Option: Computation and Neural Systems

Abstract

Navigating a complex environment requires the human nervous system to continuously integrate noisy and often conflicting sensory signals. To do so, the brain must solve the causal inference problem: determining whether signals arise from a common source and should be integrated, or from independent sources and should remain segregated. While Bayesian Causal Inference (BCI) provides a principled framework for this computation, it remains unclear how these mechanisms operate under conditions of severely degraded or absent visual input.

This dissertation investigates multisensory inference across a spectrum of visual uncertainty using complementary behavioral and computational approaches in three regimes: clinical low vision, physiological absence at the blind spot, and controlled degradation in immersive virtual reality with visuotactile interactions. In the low vision study, we find that multisensory illusions are not enhanced despite chronic impairment. Instead, BCI accurately captures behavior at high-visibility locations, where inference parameters remain comparable to sighted controls. At low-visibility locations, model fits become substantially weaker and behavioral responses more variable, limiting reliable interpretation of fitted parameters. Importantly, this likely reflects increased behavioral noise and the exploratory nature of the experimental design, which was optimized for broad spatial sampling rather than exhaustive inclusion of conditions required for robust model fitting at each location.

In the blind spot, where visual input is structurally absent, illusory percepts persist and remain highly structured. BCI provides an excellent account of behavior, revealing stable binding tendencies alongside increased visual uncertainty and broader priors. This demonstrates that multisensory filling-in arises from probabilistic inference under uncertainty rather than strengthened multisensory binding.

In a virtual reality visuotactile task, visual reliability is systematically manipulated in a naturalistic 3D environment. Here, BCI continues to explain behavior best under high reliability, but its predictive power decreases as visual uncertainty increases. We see a similar pattern of increased visual uncertainty and broader priors in response to experimentally reduced visual reliability. Notably, BCI model fit scales with task performance, with higher accuracy observed in individuals whose behavior more closely approximates a Bayes-optimal observer, indicating more effective multisensory integration.

Across all three studies, the results reveal a consistent pattern: the brain’s causal inference architecture remains stable across vastly different visual conditions. Rather than increasing binding tendency, the system adapts by recalibrating sensory uncertainty and prior expectations. However, as uncertainty increases, behavior deviates from Bayes-optimal predictions, reflecting both computational and representational limitations.

Together, this work refines our understanding of sensory compensation by showing that multisensory interactions are shaped primarily by local sensory reliability rather than broad changes in binding tendency. By identifying when and why BCI succeeds or falls short, this dissertation provides a more nuanced account of perception under degraded conditions and offers useful considerations for the design of sensory substitution systems.