DSDN112 - Project 1 Synthetic Synesthesia (Touching Sound) - Finals

Synthetic Synesthesia 

(Touching Sound)

This project began with an investigation into sound. The phenomenon of sound interested me because as of yet there is no solid science to explain the experience, the essence of sound. Sound waves are made up of particles that have shifted and vibrated as a bi-product of an action, they hit our eardrums, are transformed by the inner ear into electrical impulses to the brain. There is no physical way to explain how these electrical impulses create the conscious experience that is sound. This is reasoning behind sound illusions such as the “Mcgurk effect” and suggests that combining the right physical factors can create illusions of experiences. I then began developing an idea that would allow me to create an illusion of experience by fusion of the two senses, sound and touch. My aim was to create a form of synthetic Synesthesia, where the user feels like they are touching sound. By creating five textures that when triggered play a sound, which I have designed to match the corresponding texture, the object produces a multi sensory experience for the user.

DSDN112 - Project 1 - Final Aesthetic

Final look of my interative object

For the final presentation of the object i wanted to make sure the experience was entirely dedicated to the feeling rather than people getting overwhelmed with all the knobs, faders and dials on the midi controller. I wrapped the controller in canvas as a response to this, covering any sign of "technology". This means the user will gain a completly organic experience that is prodeced by what they are feeling and hearing, not what they are looking at. 

DSDN112 - Project 1 - Demonstrative Video

View video here on Vimeo

Seeing as i cant physically give you my object after presentation here is a video of it in action.

 - Note: The aesthetics of my final object are different as the one shown here is just a concept.

DSDN112 - Project 1 - Further Texture Development

 Sandpaper texture with improved gradient 

from smooth to rough

 Pipe-cleaners with larger surface area and more 

colors to enhance surprise effect

Sanded balsa with staircase texture to match sound 

Sponge with increased surface area 

Soft synthetic wool with increased surface area

After talking with Angela about my design it came to my attention that there were a few areas that needed developing. The way the user interacts with each surface is crucial in the designs success. In order for the sound to be triggered the user must apply enough pressure to each surface. I solved this by extruding each surface away from the keys of the controller. This created more torque so less pressure needed to be applied to the surface to push down each key. Secondly the surfaces required the user to run their hands along the surface rather than just touch to achieve the full experience. We talked about making the surfaces sloped, the the user would be forced to run there hands along the surface. However in testing the user had to use an uncomfortable wrist/hand position to achieve this. I decided to instead make the surfaces longer. This had a positive effect as most users instinctively wanted to run their hands along the long surface rather than just touch.  

DSDN112 - Project 1 - Texture Development

I have refined my textures to make them more interesting to the touch as well as making them more tactile. The balsa i have carved inlays to the user will feel a repetition in texture, as there is a repetition in the sound. The sandpaper has had a smooth to rough texture added to it as the sound builds up. 

DSDN112 - Project 1 - Texture Testing

Trail 1 of 6 testing how people match textures to sound

I have been testing how accurate each sound is to it texture on 18 different people. I have displayed the 6 most common combinations users gave. It was interesting to find that there was a consistant pattern of users matching the wool, sponge and pipe cleaners to their appropriate sounds however sandpaper and balsa were inconsistent. This has lead me to the conclusion that I need to make these textures more defined and retest until i get consistent results. 

DSDN112 - Project 1 - Sound And Texture Development

Textures I am using and below are the samples that convey these textures

1 - Wool

2 -  Sponge

3 - Pipe Cleaners

4 - Balsa

5 - Sandpaper

DSDN112 - Project 1 - Sound Creation

Creating Sound To Match The Textures  

Ableton Live is the program i am working in to create the sounds as well as trigger them in he experiment 

 The sandpaper sample that is triggered when the sandpaper is touched 

"Massive" is the virtual instrument I have used to create the sounds

The sounds I have created to match my textures have been created entirely from scratch as sine and saw waves using the virtual instrument "Massive". When the textures are touched it presses down a key on the midi controller sending a "MIDI" note to Ableton on the laptop which I have mapped to trigger the sample which the user then hears through the headphones creating a simultaneous touch/hear experience. I must articulate that this is NOT a keyboard, it is a midi controller meaning it makes no sound on its own marly sends signals to the computer. All actions and sounds expressed by the controller have been created and programed by myself. I am the owner and creator of all experiences created by this experiment. Below is a tutorial on how I made one of the sounds in "Massive". It should  demonstrate how the instrument works for those unfamiliar. 

How I created the "Sand Paper" sound on "Massive"

DSDN112 - Project 1 - Concept Trial

Initial Concept

First concept Experiment displaying the materials I will use 

Initial Testing

Initial testing of "Touching Sound"

After I had finished a concept of my experiment i immediately put it to the test. I began with trailing 5 people and asked them about their experiences. The overall comments where good with the users gaining a muti sensory experience as the touched the textures. They understood the idea and had fun engaging in the experiment. The criticism i got where that a few textures didn't quite match the sound. These where the balsa and the sand paper. I will develop this further but i was happy to learn that the other three materials reflected their sounds accurately    

DSDN112 - Project 1 - Development - The Human Mind and Its Interaction With Sound

The Human Mind and Its Interaction With Sound. 

Sound is essentially codes in the form of sound waves. Therefore there is no scientific explanation for how the brain turns these "codes" into sound experiences. This means the mind can be easily tricked into creating false experiences from certain sounds. There are a few phenomenons that easily demonstrate this. 

"Some pieces of music consist of high-speed arpeggios or other repeating patterns, which change only subtly. If they’re played fast enough, the brain picks up on the occasional notes that change, and links them together to form a melody. The melody disappears if the piece is played slowly.

Compare these recordings of Christian Sinding’s Frühlingsrauschen (“Rustle of Spring”). At the higher speed, the changing notes linger in your perception long enough to be linked into a melody, but at the lower speeds they’re too widely separated."

The shepherd's paradox is another sound illusion in which pairs of chords sound as if they are advancing up the scale, but in fact the starting pair of chords is the same as the finishing pair. If you loop this sample seamlessly then it should be impossible to tell where the sample begins and ends.

The Shepard's Paradox

These interactions suggest to me that I can manipulate the way the brain experiences sound by including textures and removing sight. Thus creating a isolated sound and touch sensory experience. 

DSDN112 - Project 1 - Storyboard

DSDN112 - Project 1 - Proposal

112 – Proposal – Seeing and Feeling Sound

For my interactive object I want to create an experiment that allows the mind to visualise and feel sound that you are hearing. I want to create sounds that the brain directly associates the way certain material look and feel like. This will in a sense be and illusion of the mind as the objects appearance and texture have no visual association but I want to create an object that suggests the human brain has the ability to visualise and feel what it is feeling, creating a multisensory experience. I intend on creating this experience buy attaching contrasting textures to different notes on a keyboard. I will use the software “Ableton Live” to make each key trigger a sound that I feel is reflective of the materials texture. As people will have a varied expectance of what each sound should look or feel like I will have to survey a certain amount of people. I will create different sounds and different materials and ask a selection of people to match certain textures with different sounds. I will repeat and refine this selection to the point where I am getting consistent results with pairing of materials to sounds. As we have no way to explain how our brain turns soundwaves, light waves and electrical impulses through our nervous system into an experience I hope to give the user a new way of considering this phenomenon. I feel that combining sensors is a good way of investigating this, and I also hope through my findings this idea can be replicated in a more practical direction, weather it is used to enhance the way musicians play instruments, or to help someone blind experience sight without necessarily seeing.     

            

DSDN12 - Project 1 - Development - Sound Experiments

I have begun my investigation into how our brain deals with sound in ways that dictate the information on what we need (What we hear) and what we dont need. The cocktail effect is a particular phenomenon that manifested my interest in this concept. The cocktail effect is the phenomenon that occurs when we are listening and absorbing information of one conversation in a room full of separate conversations at the same volume. Although, physically all the sound waves of these conversations are reaching our eardrums, or brain filters them out, so we here them as background noise, almost subconsciously quieter than the conversation that has our attention. 

Another phenomenon that has begun to interest me is binaural beats. These are sound artifacts that induce a physical response and can improve brain function. Also when played in constant repetition they have the ability to subconsciously remove themselves from your auditory senses enabling you to concentrate on listening to something else. To experiment with this phenomenon i wrote the first half of this blog post with this binaural beat playing. I was able to compose and type the first paragraph in under 2 minutes. For this current paragraph i have been listing to this, a synthesizer producing randomly pitched notes. I have watched the five minute video twice and only got this far through the paragraph. What you can begin to conclude is that the ability to concentrate can be changed by what the auditory sense is engaging in.

DSDN112 - Project 1 - Studio 2 - Fun Vs Function

Guitar Tuner and Bubble Gum

I have chosen a juxtaposition of objects that stimulate various senses for a purely functional or aesthetic purposes. The Guitar tuner is deigned to stimulate your visual senses in a practical way. Its black on white text and responsive LEDs mean it is easy for the user to visually find the pitch they are looking for. The guitar tuner is also very tactile due to its rubber buttons. These are nicely spaced as to hit the button the user intends to hit and give a quick response when pushed. The guitar tuner also involves the auditory sense with a clear and loud tone that lets the user reference what note they need to find. The guitar tuner is a hard object to have fun with, however it is very engaging to three out of the five sensors making it extremely functional.

In contrast it would be hard to find a practical use for bubblegum short of a substitute for Blue-tak, though it is a far from useless item. The bubblegum heavily engages all five sensors, and is highly consumed product. 

DSDN112 - Project One - Studio 1 - Five Sensory Items

Finding five items that engage more than just our sight started out as a challenge to me, but the more I thought about how my 5 senors interact with everyday objects, the faster i found myself saturated with applicable designs. For me it was in looking past pure aesthetics of an item that i found items with well considered ergonomics and ease of function engaged several sensors.   

 Bubblegum

Surprisingly, I found this simple item stimulated the most sensors when interacted with. Initially, its looks with the bright colors entrances the eyes. The semiotics of its shape and color instantly associate the products with thoughts of candy, then as the gum is unwrapped the scent gratifies those initial feelings. The texture and flavor serve its function as a food product however the secondary function of the consumer blowing bubbles also engages the ears. I prime example of a product that engages all five sensors. 

 Rubber Coffee Mug

The way the ergonomics of the mouth piece mould to the lips simulate the touch sensors. Its look and feel are integrated seamlessly to create an experience of drinking the coffee, not just keeping it warm.

 Authentic Iphone Earplugs

Without wanting to seem cliche, there is more to this item then i just being something i have i my pocket. I believe apple have created a fashion and design icon through the way these earplugs interact with your sensors. Firstly the modernist design and marble white simplicity is reminiscent of the Parthenon. Easily recognizable when worn, it would be hard to find a person in the western would that doesn't subconsciously think "Apple" when they see these. Obviously these serve the primary function of engaging the ears, but the way they have been designed to feel against your ears has been well considered.

Midi Keyboard

The black on white keys are an instant visual aid to to associating this with its function of being and instrument. The way it creates sound obviously engages the ears and the years of its existence has crafted the generic design of the keys to become ergonomic and efficient.  

Mandarin

Nature has subversively crafted a fruit that engages our sight, touch, smell and taste.