The world of today is a salad of sounds, pictures, odors, action and feeling. None of it can be separated from each other and perceived separately, they come as a kit. You see a lightning and then you hear a bang. You hear a cat's "meow" and you don't imagine a dog. You smell a flower and most probably - you think of a flower, visually.
Music is strongly extended by visual information as well - music videos, visualizations in night clubs and backgrounds in live concerts. It helps to construct the full-flavoured experience out of the peace. I was interested into real-time connections between the sound and the picture and the interaction of the artist between those two.
The project was developed at the University of the Arts Bremen, Germany in the summer 2006 as the final bachelor project in digital media. It covered theoretical thesis as well as development of the software and hardware setup. The theoretical part is downloadable from here in PDF format.
The performance delivers different visuals each time it's being run, here are some videos captured while experimenting with the software. Quicktime required.
After the final presentation at the university the project has been paused for an unknown period of time because there is no access to the needed hardware, that earlier was provided by the school, due to lack of financial support.
For the time of the development a blog was created to document the process. The website will expire soon, nevertheless can be checked here.
The performance consists of two DJ turntables that spin special time-coded vinyl records from Ms.Pinky, one simple Wacom Graphire A5 size graphic tablet and software programmed in Max/MSP/Jitter which analyses live music, interprets inputs from vinyl and tablet and outputs real-time motion-graphics.
For technical reasons at the time of the making the software was split up in two parts, first - running on a MacBook Pro - analyzes the incoming audio signal and sends data over local network to the second part - running on a PC - which picks up the data and generates visuals.
Audio analysis is fairly simple - audio signal is being split up into 25 bands where each band represents one frequency and it goes from 0Hz to 22kHz or in other words from low sound (bass) to high sound (treble). When we hear a loud bass kick, the values on the left ar higher, when we hear nice detailed high tones, the values on the right respond. Later on I can map these low frequency values to some "rough" visual effects to reflect the drum- and bass line of the sound and the high frequencies to fine details that make up the scene.
But not only music controls what we see on the screen; there are custom ways of interacting with the visuals. First of all there's the interface of the application where the artist can tune different properties to achieve different visual data.
Then there is the time-coded vinyl which delivers two values each - an absolute position (of the needle) value and a velocity value which is as big as fast the vinyl turns and negative or positive when vinyl turns backward or forward respectively. By touching the vinyl records, scratching them or anyhow modifying the position of the needle different properties of a visual object can be influenced in real-time. It's an exclusive experience when you can turn an old-school vinyl instead of pushing buttons on an anonymous computing device and get electrified visual effects in return.
The visual scene consists of 4 different objects with numerous properties which float in a 3-Dimensional space. There had to be invented a way to mix between these objects live on the stage. Conventional video mixing techniques (transition from one picture to another known as cross-disolve) wouldn't work because the rendered action is true OpenGL 3D and we have four action figures. Instead a graphic tablet was used as a mixing device - it has 4 corners and you can freely move from one to another in any direction. The mixing happens while one object morphs into another. The result between objects is an object as well. As there are four objects, there is an undefined number of possible morphs.