View the original post
Designing an interactive, audiovisual exhibition — a Case Study:
How I designed and build RE:SONANCE — a digital exhibition that takes viewers on an interactive, audiovisual 3D journey with sounds based on naturalistic soundscape and harmonising, captivating visuals.
When the second wave of the pandemic hit Sweden I thought to myself stress levels are at an all time high with financial insecurity, uncertainty about the future and social isolation. Spending all our time indoors has led to more people wanting to explore the outdoors and nature, allowing people to meet others at a safe distance and breaking out of our own four walls.
How could we take a part of nature and combine it with a feeling of connectedness? What if we could make it easily accessible to take a break, explore and relax?
Check it out yourself before reading further — Use desktop and Chrome for best experience: Visit the exhibition
It all started with me attending a digital exhibition and it made me think if I could create one myself. A few days earlier I stumbled across Cables a visual development tool for interactive WebGL, which opened a whole new world of possibilities to me. Wanting to learn this new tool, I set up to brainstorm ideas on what would be of value and also would give me the space to create something a bit more artsy — within constraints. One day being outside with my dog listening to the soft crushing of ocean waves on rocks it hit me — a multi-sensory experience with visuals and sounds. I wanted it to be interactive, with beautiful visuals and relaxing sounds. For the sound part I definitely needed some help, so I reached out to my good friend Emily Kornaat and explained her my idea with a sketch (the one you see on the top with just dots). My idea was to create a universe where organic shapes exist and you can wander around, choosing which one you want to listen to, and that the visuals would reflect the sounds. Luckily she was as excited as me!
Kick-starting the design process through desk research
Down the rabbit hole of researching nature sounds and the effects it has on our brains! After reading lots of articles about our mind, auditive stimulation, relaxation & stressor sounds and psycho-evolutionary theory I nailed down our concept and vision:
“An immersive, interactive audiovisual 3D exhibition. Each interactive shape has a unique base layer of a naturalistic soundscape, recorded by family and friends all across the globe. Added onto the base layer is a unique interpretation of the feeling the sound evokes. When tapped or clicked selected bass sounds ring out. The shapes look and animations are a visual interpretation of the naturalistic base and also remind the viewer of it.”
Pinpointing value for viewers and ourselves
- Experiencing the natural world can provide relief from the stress and anxiety of the COVID-19 outbreak
- Facilitating Mood Recovery through natural Soundscapes
- Creating a feeling of connectedness in a time where physical closeness is not possible with features naturalistic sound patterns from all around the globe
- Reducing stress through constant noises of a pleasant pitch
- Designing and producing a visually interesting exhibition, that makes people curious
- Capturing the feeling of naturalistic sounds and adding on to them
Visually coding and interpreting sounds as WebGLs
Oh, the highs and lows of learning a new tool. It was quite difficult at first to get a hang of Cables. Cables is visual coding, you have OPs that you place and then connect to each other — similar to nodes. After lots of tutorials, creating ugly things and looking at other peoples patches I was finally able to set up a shape base. I could reuse this same base but tweak or exchange effects somewhat easily. Every shape is visually unique and my own artistic interpretation of the naturalistic base sound or finished audio from Emily. I sometimes created the shape first and it inspired Emily, and sometimes her audio inspired my visuals.
After finishing the shape, I added the code for the interaction, which is visually a bit different depending on the shape, but it is always a deform effect of some kind — and it plays the bass of the track. I wanted first that the viewer is able to drag, and click and interact in many different ways, but that turned out to be very difficult and time consuming, and too much for this scope.
Evoking feelings through uniquely crafted sound
All tracks have naturalistic soundscapes that were kindly provided by friends and family all over the world — or we recorded them ourselves. The sounds are Emilys personal interpretations of what feelings these sounds provoke. She produced the finished tracks using different methods like Musical Typing in GarageBand or even singing to achieve different ranges of sounds.
Giving special meaning through a name and logo
We named this exhibition RE:SONANCE combining the words resonate (“evoke images, memories, and emotions”) and sonance (“Sound”). I designed the logo for it, keeping it very clean, in black & white to let the colorful shapes stand out. I first played around with different fonts, then decided on the strong sans-serif Bebas Neue. I really wanted to give this logo a 3D space, and move away from flat design. I achieved that with the blend effect and playing with opacities and the 3D effect in Illustrator. Also applying a bit of monochromatic noise in photoshop, gave this logo more character, instead of a perfect finish.
Embedding a digital exhibition and hoping for the best
Surprise, I also coded the website! Simply by using HTML and CSS to create a very simple base to be able to click through the different shapes. Then came the realisation that browsers have specific rules about playing audio. I first tried to somehow work around it, which was very difficult because by no means am I a developer — I found a script code snippet online that would play the audio after the DOM Content has loaded. It works, not perfectly — but it works.
For implementing cables on HTML, it was quite easy — you can download a .zip with all your files for your patch. After finishing all the pages, embedding everything I created a subdomain and uploaded all through a FTP. Well, as you can imagine, a 3D interactive patch with sound is quite heavy on a server, so it was so slow and not smooth at all. Change of plans: I found a way to host it on Cables server itself and embed just an iframe into my HTML. The result now being a smooth running digital exhibition!
Unfortunately there are major problems on mobile (from Cables side). Textures do not get loaded, so the shapes are just dark — I added a little note for users when on mobile that this experience is not designed for mobile and to please switch to a desktop. I wanted to minimise frustrations here and give the user insights of why they would encounter problems, they can still look at it on mobile, but understand it is not intended this way.
Check it out yourself — Use desktop and Chrome for best experience: Visit the exhibition
“This made me realise how much I miss going on art exhibitions! Really liked the look and feel of it and together with the sounds it was a very nice experience.”
“Wow!!! Truly amazing!”
“I found the results of this so inspiring that I had to take a look at the case study for this: and it’s such a well documented and interesting story of exploration. For anyone interested its a really worthwhile read!! And great example of a first testable product.”
“How beautifully done! You guys have so much talent.”
– Quotes from visitors
Honestly, at some point in this project I thought to myself that I bit off more than I can chew. And I am so happy I pulled through. I learned so much through this incredible project and was able to really let my creativity go wild — visually and through with problem solving. Having to code visually and the website was a nice reminder of how parts work together, and how often it can be difficult to navigate through technical constraints that you did not consider before. I’m really happy and got an amazing positive response from so many lovely people too. Big thanks to my friend Emily too, it was so much fun to work on this!
Dream big start small
It’s easy to get overwhelmed, when you have big expectations and then you don’t reach them. Through this project I realised that I love being creative and dreaming, but equally important it is to set yourself up for success by starting with a smaller scope of a project.
Finding creativity in a new medium
I really needed some fresh, creative wind in my sails. I noticed how easily we get into a routine and how as designers we need to try out new tools and have projects where we just can be creative. I really enjoyed working with a 3D tool and also interactivity! More of that!