A Speculative Research on the use of particles system and point clouds combined with echolocation and synesthesia to help blind and deaf individuals fully perceive the environment
Imagine experiencing a classical music concert as a colourful visual experience in which each sound evoke a vibrant orb of colour. In the classic animated movie, Fantasia (Walt Disney, 1940), during one of the interludes, the Maestro introduces a captivating personification of the “Soundtrack” itself; this entity reacts to the music, creating beautiful waves, bright colors, and captivating images, bringing to life the harmony and rhythm in a stunning show of colors and patterns.
Synesthesia's Condition
Synesthesia is a neurological condition where the stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. For example, a person with synesthesia may perceive letters or numbers as inherently coloured or associate certain sounds with specific colours. Now imagine the condition of synesthesia turned upside down and applied as a means to make up for disabilities; synesthesia could potentially be used to enhance accessibility for individuals with blindness and deafness. But how?
Visual-to-Auditory Synesthesia for the Blind: For individuals who are blind, visual-to-auditory synesthesia could be harnessed to convert visual information into auditory signals. Through the use of specialized devices or software, visual inputs such as text, images, or spatial information could be translated into soundscapes or auditory cues, allowing blind individuals to perceive and interact with their surroundings more effectively. For example, objects in the environment could be represented by distinct pitches, tones, or rhythms, providing spatial awareness and facilitating navigation.
Auditory-to-Visual Synesthesia for the Deaf: Conversely, auditory-to-visual synesthesia could be leveraged to translate auditory information into visual stimuli for individuals who are deaf or hard of hearing. Sound frequencies, patterns, or rhythms could be mapped to visual representations such as colors, shapes, or movements, enabling deaf individuals to perceive and interpret auditory signals through visual means. This approach could enhance communication, facilitate music appreciation, and improve access to auditory information in various contexts.
In the famous rain scene from Daredevil(2003, Mark Steven Johnson) we can see the tormented blind hero played by Ben Affleck standing on a rooftop with Elektra. He uses rain as a way to visualize the environment; thanks to its supernatural hearing he’s able to locate every raindrop, each of one it’s producing a distinct sound. But that’s not just fiction, it is based on a true “technique” called echolocation.
Echolocation it’s often associated with bats and dolphins, whom generates sounds to retrieve echoes from the surrounding to perceive the distance from the environment. But human can learn it too; blind individuals can make clicking noises with their mouths, tapping a cane on the ground, or using specialized devices that emit sound pulses. When these sounds hit objects in the environment, they bounce back as echoes. By listening to the timing, intensity, and direction of these echoes, blind individuals can form mental images of their surroundings. With practice, individuals can discern the size, shape, texture, and distance of objects based on the characteristics of the echoes they receive.
The same Lidar systems has been invented from the technique of echolocation, using pulses of light and measuring how long takes each light to come back. This process generates incredibly detailed 3D scans of the environment, often represented as a Pointcloud — a collection of discrete data points in space that define the objects and terrain within.
Particles’ Echolocation
Now, envision each of these points within the Pointcloud emitting a distinct sound upon their return. The visualization of the environment transforms into a symphony of echoes, similar to the sensation experienced by Daredevil as raindrops reverberate off his surroundings. This constant pulsating echo provides a dynamic and vivid representation of the environment, offering a real-time auditory map of the surrounding space.
Andrew Thaler, used all this techniques and his knowledge of ocean science to apply the echolocation from dolphins to normal sight. In 2018 he built a head-mounted LiDAR array that lets you see the world like a dolphin via vibrations sent through your jaw. The headset wasn’t used for blindness because it doesn’t help you visualise the surrounding but it helps you extend it through vibrations and sound. But that’s the perfect case of Serendipity, where a discovery without a real objective become a base to solve a real problem.
.Lumen is a research startup founded in 2020 and was headquartered in Cluj, Romania. .Lumen mission is that of giving assistance to blind individuals, enabling them to see the environment and to replace the guide dogs because of their limited numbers compared with the number of blind people. Their headset The . lumen glasses understand the environment, objects, position, and movement, and where it is safe for a blind person to move, and compute interaction paths to wanted objects. But even in that case the .lumen glasses doesn’t help to visualise the environment but give just direction to avoid obstacles.
But is it actually possible to convert echo into “visual maps” of an environment? A study (Retinotopic-like maps of spatial sound in primary ‘visual’ cortex of blind human echolocators by Liam J. Norman and Lore Thaler) confirms that while blind individuals may not “see” in the traditional sense, their brains can still create a representation of the environment similar to what sighted individuals experience visually. So what would it be like when in the future the echolocalization technology become much more effective?
With tounchdesigner I realized possible hypothesis of particle echolocation; in the prototype, through sound hitting a point cloud the space can be visualized and reconstructed. Chaos and oblivion becomes order through the sound pulse and returns the image as a set of particles(sound waves). This space is my bedroom, captured through the LiDar system and then converted to a PointCloud file.
Imagine if it were possible through this system to “visualize” through sounds a person we know but cannot see. Well I have imagined it for you, inspired by the cult Daredevil rain scene mentioned above.
Also, colours can be seen through sound, because each one of them has its frequency; Neil Harbisson (color cyborg) is an artist who was born completely color blind and because of that he implanted himself an antenna which convert colours directly into sounds. A chip hidden under the scalp converts the signal into vibrations, which the implant wearer can hear as if they were sound, through bone conduction.
Particles’ Chromesthesia
The same concept can be reversed for individuals who are deaf. Chromesthesia, or sound-to-color synesthesia, is a condition where sounds trigger experiences of color, shape, and movement involuntarily. Those with this condition consciously perceive color associations with sounds in their daily lives. We also know that colours and sounds have both a wavelike character, but still they are very different.
“They are vastly different in both size and speed. Both color and sound cover a range of wavelengths, but color is not restricted to a single wavelength. The pitch of the note A has a frequency of 440 Hz. If notes were sounded around the A but of slightly smaller or greater frequencies, the result would produce pulses in the sound and an unclear tone. As the size of difference in frequency is increased, the sound would be dissonant and very unpleasant to listen to for an extended period of time. Sound is not continuous in this sense. Color on the other hand is continuous.”
(An Investigation Into the Relationship Between Sound and Color — Joe Goldsmith, http://www.people.vcu.edu/~djbromle/color-theory/color01/Relationship-color-sound-joe_goldsmith.html#:~:text=Sound and color (light) both,restricted to a single wavelength.).
That means that converting light to sound is possible but would be limitative; on the other hand converting sound to colours it’s much more effective because the audio spectrum is much more limited than the light spectrum. That’s something we are already experiencing in everyday life; specific music is associated with specific colours or is applied in colour-grading in cinematography. But how can this be used to make up for deafness?
Augmented Reality is the answer. Cochl.Sense with Nvidia Riva and its ASR(automatic speech recognition which it’s able to understand a wide range of sound other than speech) is designing for the Microsoft Holo Lenses 2 an interface integrated with the glasses that will tell which sound is it and where does it come from. But for now it’s just straight audio recognition and conversion to text.
What if in the future, you could visualise the sound as if it was physical (chromoesthesia) as colours and vibration of the surroundings? The AR headset transform audio signals into dynamic visual representations. As sound waves propagate through the environment, they trigger the emission of colorful particles that dance and swirl in response to the intensity, frequency, and timbre of the sound. The device incorporates a sophisticated color-mapping algorithm that assigns specific colors to different sound frequencies, creating a visually intuitive representation of the auditory landscape.
Through a workflow between blender and touchdesigner, I applied a particle system that reacts to audio and changes color based on frequency. Then the result was “transformed” into touchdesigner to map the sound and detect its direction.
That’s my corridor, but now it can be heard
Future will be certainly more accessible for everyone, the convergence of technology and human experience opens doors to a more inclusive world for individuals with diverse sensory abilities. From echolocation-inspired LiDAR systems to synesthetic AR headsets, innovative solutions are reshaping how we perceive and interact with our environment.
Check also the Behance project:
https://www.behance.net/gallery/194733919/Particles-Synesthesia
