Centre for the Study of Perceptual Experience

Maurice Ptito

“There is more to vision than meets the eye: The tongue as tactile gateway to the occipital cortex”

Abstract: The tongue display unit (TDU) is a sensory substitution device that translates visual information into electrotactile stimulation that is applied to the tongue. Blind subjects can learn to use the TDU in various visual tasks, including orientation, motion and shape identification and spatial navigation. We used the TDU in conjunction with brain imaging techniques in order to study the cerebral correlates of cross-modal brain plasticity. The results show that when blind subjects use the TDU to perform visual tasks that are known to activate the dorsal and ventral visual streams in the sighted, they activate the same brain areas. This suggests that motion and shape processing are organized in a supramodal manner in the human brain and that vision is not necessary for the development of the functional architecture in motion and shape processing areas. We also used the TDU in spatial navigation tasks and we showed that blind but not blindfolded sighted subjects activated their visual cortex and right parahippocampus during navigation, suggesting that in the absence of vision, cross-modal plasticity permits the recruitment of the same cortical network used for spatial navigation tasks in sighted subjects. We also looked at the anatomical reorganization of the blind’s brain through various techniques including VBM, FDG/PET, cortical thickness, TMS and SEPs.

Michael J Proulx

“Sensory Subsitution: Learning to See with Sound”

(co-authors, David Brown and Achille Pasqualotto, Queen Mary University of London)

Abstract: Vision is the default sensory modality for normal spatial navigation in humans. Touch is restricted to providing information about peripersonal space, whereas detecting and avoiding obstacles in extrapersonal space is key for efficient navigation. Hearing is restricted to the detection of objects that emit noise, yet many obstacles such as walls are silent. Sensory substitution devices provide a means of translating distal visual information into a form that visually-impaired individuals can process through either tactile or auditory perception. Here we will review findings from electrotactile, vibrotactile, and auditory sensory substitution systems for the processing of visual information that can be classified as what (object recognition), where (localization), and how (perception for action) forms of processing. The current state of the art suggests that the different forms of sensory substitution are better at some tasks than others. Spatial navigation in particular provides a useful model for comparing sensory substitution systems, with important implications for rehabilitation, neuroanatomy, and theories of cognition.

Amir Amedi

“Seeing colored images with music using the EYEMUSIC: from perception to visual rehabilitation”

Abstract: The exciting view of our brain as a highly flexible task-based and not sensory-based raises the chances for visual rehabilitation, long considered unachievable, given adequate training in teaching the brain how to see. Recent advances in rehabilitation approaches, both noninvasive, like sensory substitution devices (SSDs) which present visual information using sound or touch, and invasive, like visual prosthesis, may potentially be used to achieve this goal, each alone, and most preferably together. Visual impairments and said solutions are being used as a model for answering fundamental questions ranging from basic cognitive neuroscience, showing that several key visual brain areas are actually highly flexible, modality-independent and, as was recently shown, even visual experience-independent task machines, to technological and behavioral developments, allowing blind persons to 'see' using SSDs and other approaches. I will present evidence that SSDs can be potentially used as a research tool for assessing the brain's functional organization; as an aid for the blind in daily visual tasks; to visually train the brain prior to invasive procedures, by taking advantage of the 'visual' cortex's flexibility and task specialization even in the absence of vision; and to augment post-surgery functional vision using a unique SSD-prostheses hybrid, including color information using a new tool we developed in the lab called the EyeMusic. This SSD is aimed at converting colored images with music. Taken together the reviewed results suggest a brighter future for visual neuro-rehabilitation.

Laurent Renier

“Sensory substitution devices: Seeing with sound, hearing with touch”

Abstract : Sensory substitution refers to the use of one sensory modality (e.g., hearing) to supply environmental information normally gathered by another sense (e.g., vision) while still preserving some of the key functions of the original sense. For example, the use of auditory signals might give information about visual scenes. The development of sensory substitution devices has profoundly changed the classical definition of sensory modalities and contributed to the emergence of a form of “artificial synaesthesia”. In the last decade, our knowledge about cognitive and brain mechanisms involved in sensory substitution has grown considerably bringing new insights into human perception and neural plasticity. Thanks to technological advances and scientific achievements, sensory substitution has become a real alternative for restoring some functions of a defective sensory organ (e.g., sight in case of blindness or hearing in the case of deafness). We will review here some of the major questions raised by sensory substitution (e.g. the exact nature of perception), demonstrates how the study of sensory substitution enhances our understanding of human perception and brain plasticity and provides an overview of rehabilitation potentialities.

Peter Konig

“Properties and mechanisms of sensory augmentation”

Abstract: Enacted theories of consciousness conjecture that perception and cognition arise from an active experience of the regular relations that tie together sensory stimulation and associated motor actions. By employing the technique of sensory substitution and sensory augmentation previous experiments explored this assumption. In the latter study the sensory augmentation device delivered global orientation information by mapping directional information of a compass to a set of vibrators, activating the element pointing north. Here we use it to investigate the impact of newly supplied directional signals on cortical plasticity, sensory processing and spatial cognition.

The training belts consisted of 30 piezoelectric, vibrotactile ac- tuators, a 3DM GX3 compass, a control-box and battery packs. They were to be worn by the subjects during all waking time. A dedicated MRI-compatible belt was based on identical piezoelectric vibrators. Ten subjects (age 19-32y, four female, one control) were wearing the belt during all waking hours over a period of six weeks. We compared belt-on and belt-off conditions in a series of measurements including homing, multimodal integration, nystagmography, sleep-EEG, fMRI, and subjective methods before, during and after training.

(1) In the homing task using on polygons of varying complexity we observe a slight reduction of the systematic error and a larger reduction of the stochastic error in belt-on condition after the training period. (2) Integration of the newly supplied signals with visual information in a psychophysical task was rather limited and just noticeable differences of rotation (yaw) were surprisingly high in belt-off as well as in belt-on condition. (3) Nystagmography demonstrated an increase of the time constant of per-rotatory nystagmus (slow phase) after training in the belt-on condition as compared to the belt-off condition indicating a firm integration in sensory processing. (4) Sleep-EEG uncovered an increase of REM-sleep during the early training phase. In contrast, no such change is observed in stage 3 sleep. (5) Most areas that were reported in a previous fMRI study on navigation could be replicated in all our subjects. Furthermore, we observe widespread cortical activation in belt-on condition after training as compared to the pre-training baseline. (6) Subjective reports indicate that by training with the feel- space belt the scope of perceived space grows wider and includes areas that are not within reach or directly visible; subjects feel more secure in known as well as previously unknown environments; and navigational abilities improve and emphasize an egocentric reference frame.

The data provide evidence for an integration of the newly sup- plied signals in sensory integration (homing, nystagmography), cortical processing (sleep-EEG, fMRI) and spatial cognition (subjective methods).

Jonathan Cohen

"On Some Limitations of Sensory Substitution"

Abstract: Sensory substitution devices can be thought of as
alternative channels for carrying the information normally delivered
by an impaired modality. Unfortunately, because sensory modalities
carry information about a very wide range of properties/events;
consequently, the design challenge of building informational
alternatives for them is significant.

Here is an attractively simple reaction to this concern. We might hope
to ensure that the substituting channel captures everything wanted
by just building SSDs that carry information about the form(s)
of basic energy normally carried by the impaired modality they are
intended to replace (and adding lots of computational power to
code up whatever can be derived from the basic energy). Thus, for
example, if we could build a device that carries in some alternative
way information about the distal distribution of light intensity --
the basic form of energy to which visual receptors are normally
responsive --  our device could, in principle, represent everything
vision can represent: color, shape, form, motion, and so on.

Unfortunately, I will argue, this attractively simple idea fails. For
there appaer to be properties represented by our sensory modalities
whose exemplification is not fixed by the representation of the
distribution of (the relevant forms of) basic energy. Hence, a SSD
whose basic representational vocabulary is limited to the distribution
of such basic energy will inevitably leave things out.

None of this shows that SSDs cannot be useful technologies that
improves lives. It does, however, suggest that if we want SSDs to
represent what sensory modalities normally represent, we'll have to do
more than preserve the representation of basic energy to which the
substituted modalities are sensitive.

Jerome Dokic

"The role of noetic feelings in sensory substitution"

Abstract: I shall first argue that the phenomenology of ordinary perception is dual: it involves both a sensory aspect, which determines the representational contents of the experience, and an affective aspect, which is constituted by various noetic feelings, such as the feelings of presence, familiarity, and certainty. These feelings are best understood as conscious emanations of implicit metacognitive monitoring and control processes. The duality of perceptual phenomenology is relevant to an assessment of the analogy between sensory substitution and ordinary perception. Beyond some anecdotal evidence to the effect that sensory substitution does not involve the “feel” or “qualia” of ordinary perception (for instance with respect to familiarity), two questions arise. First, what are the analogues, if any, of metacognitive monitoring and control processes in the case of sensory substitution? Second, to what extent does the existence of such processes in sensory substitution depend on the perceptual interpretation of sensory substitution? In other words, could the subjects have feelings of presence or familiarity in the absence of genuine sensory contents in the putative substituted modality?

Malika Auvray and Ophelia Deroy

"Audio-Visual Substitution"

Abstract: Despite the numerous studies and research programs devoted to their development, sensory substitution devices (SSDs) have failed to live up to their goal of offering something close to the recovery of a lost perceptual skill. As we will stress, the most timely issue is therefore to understand where this limitation comes from, and subsequently to redefine more realistic promises for SSDs. Here we will focus on one of the main obstacles presented by current SSDs, and one of the most relevant for their users:  the amount of learning they require before they can be properly used, together with the high cognitive and attentional load they continue to demand afterwards. We will highlight that addressing this problem requires theoretical and technical adjustments: First, a more systematic investigation of the various processes underlying the integration of these devices; second, a better model of what can be achieved through learning the most optimal couplings between the various dimensions of the signals coming from the intact sensory modality and the targeted modality. In this way, we will bring together the analogy between SSD-use and reading skills, and the promises of using intuitive crossmodal correspondences to ease learning and use.

Creole

Colour provides a lot of information about the world, allowing us to separate and identify objects, recognise scenes and facilitate communication between the blind and the sighted. While most sensory substitution devices translate luminance into sound or touch, in complex natural environments full of shadows and complex lighting, luminance is no longer a reliable cue to extract objects from their background, or to navigate through the real world. While previous attempts have been made to create colour sensory substitution devices, there was no easy way of comparing their approaches to colour representation.

Our device, the Creole, is a tablet device that allows the user to listen and feel the colours on the screen by pressing down on them with their finger. This approach allows the user to use their finger movements like eye movements to scan over the image and understand the distribution of colours. By leaving the spatial element to the user, the majority of sound and touch stimulation is left to representing colour itself. This allows us to compare the advantages of using different colourspaces and their
encodings in a wide range of tasks.

If you would like to find out more on this colour sensory substitution device, please e-mail Giles Hamilton-Fletcher directly or visit the lab website.