Matt Donald - assistant principal

I have always been fascinated by interdisciplinary art. Along with my passion for music creation and performance, I am also interested in colour, technology, traditional and contemporary art forms, theatre, design, and more. But what I find particularly intriguing is the intersection of those art forms. I believe that great art has always aimed to make connections between the lines of classification. Perhaps true art is interdisciplinary, but that’s a topic for another time.

In February, I decided to delve deeper into the intersection of art and technology. I aimed to create two pieces of interactive new media art, combining visual and musical elements responding to the movement of the viewer.

 I thought I would document the process, from conception to creation, along the way.

The process

I admit that accomplishing a new artistic practice in four days would be challenging! So, to prepare myself, I researched and learned some of the technology beforehand, knowing that I would only be able to scratch the surface. I relied heavily on a self-curated curriculum that taught me the fundamentals and how to apply them. Nevertheless, my goal was to learn something entirely new in a very short amount of time.

I swiftly landed on a piece of software called TouchDesigner. However, my first investigations suggested that I might need a Kinect camera to sense movement in the viewer & a Windows machine; I have neither. But then I discovered a script that incorporates Google’s MediaPipe into TouchDesigner. MediaPipe only needs an image or a camera feed to classify what it sees (amazing!). The script I found allowed me to use my computer webcam (or any camera that the computer thinks is a webcam) to analyse my posture, track my hand movements, and even recognise my facial expressions.

 

My earliest test was simple – a proof of concept ahead of time.

Can I use MediaPipe to make an orange circle track the tip of my index finger?

Success.

Day One

A little delayed due to a sick daughter, I started working on a fluid system, my first piece. The idea was to create a simulation in which the viewer’s hand movement would generate swirling shapes of fluid on the screen and produce musical notes at the same time. But before I could get to the fun interactive part (or even the music bit, which is much more in my wheelhouse), I had to build and code the simulation. There were many mistakes, ‘nodes’, and Python scripts (I had never coded in Python before). However, I managed to make some progress towards the first step.

I appreciate the above image shows little of the day’s success, but you’ll have to trust me; this took some time! What you are observing is known as a ‘network’ in TouchDesigner. It serves as a central hub for all the connections between various components.

Tomorrow’s challenge would be to complete the fluid simulation, add interactivity, and possibly incorporate the music.

Day Two

Another less-than-full day due to an invitation to join the New Wine tour that night (that’s the balance of a creative lifestyle). I had just a few minutes to continue before needing to be in for rehearsal. Unfortunately, I came to the end of my instructions, and the simulation didn’t work. It was almost there, but not quite. I was disappointed and frustrated because I didn’t understand enough to fix it. Yet. I had to leave for my rehearsal with an incomplete project. I would have to start over again tomorrow.

Day Three

Undeterred by yesterday’s unsuccessful conclusion, I decided to retrace my steps meticulously, one by one. I identified the issue; a single digit “1” in one of my scripts should have been a “0”. It was a satisfying moment seeing the swirling red fluid following my mouse pointer around. Sometimes, you need to double-check everything, you know?

Some colour changes later & I had the experience beginning to take shape.

Now following the mouse was one thing, but I needed to replace that interactivity with moving my hand in front of the camera. I also needed to get to grips with how this was to generate music.

I began by working on the musical aspect, using a package that is native to TouchDesigner and allows for direct control of Ableton Live (which is my go-to choice of music production software). I used a bit of Python code to generate a single note and send it to Ableton each time the mouse pointer changed position. Once Ableton received the note, it was processed and sent to a synth. Now, both notes and motion responded each time I moved the mouse.

I used MediaPipe to track the position of the tip of my right hand’s index finger, similar to my initial test. After making a few adjustments, I replaced the mouse with my hand and successfully triggered notes by moving! However, the fluid simulation wasn’t as dynamic as it had been with the mouse.

Do you ever wake up with random thoughts about the things you’re working on? A single midnight thought gave me the solution to bringing those dynamics back!

It worked. For those interested, here’s what the patch looks like. A lot of things connect to a lot of other things(!), but it worked.

Now, it needed some tweaking for sure. Perhaps the colours could modulate differently; maybe the sounds could respond differently; I wonder if the interaction could behave differently? But version one was complete. My eldest daughter tried it and said, “It’s like a galaxy!”.

I had one day left to explore. With hopes to tidy this up and create a second piece, I still had some work to do, but momentum was building.

Day Four


Day four opened with a strong sense of optimism and determination to complete a second piece. I reviewed my instructions and learned new techniques. But thanks to the knowledge I gained from the previous piece, this time was much smoother, and I completed the instructions in (comparatively) record time.

In my first piece, I explored how movement can generate visuals and music. For my second piece, I took a different approach. The animation of this visual included several concurrent movements, which gave it a flowy, wavy, and gradient look. I decided to apply that motion to the music by animating the volume of each note in a chord using the same parameters that animated the colour or shape. I settled on using an M9#11 chord for those interested. The result was instantly satisfying; the movement of the visual and the harmonic movement of the music worked together in unity, even though I didn’t know the direction!

But it needed interaction.

I used the same MediaPipe script to bring my hands into play again. This time, I used both my hands and a larger motion. When I brought my hands together, the circle in the animation would shrink, and when I moved them further apart, the circle would grow.

This worked well but it needed to adjust what was being heard too. I added an arpeggiator to the chord playing (it plays each note of the chord one after another). I set it to change the tone of those moving notes in partnership with the circle’s size. After a little play-time (it’s important to make time to enjoy the creation process) I wanted to make the arpeggiator’s speed adjustable based on my hands’ proximity to the screen. I intended to make it so that the closer my hands were to the screen, the slower the arpeggiator would play, and the further away they were, the faster it would play. My goal was to create a sense of focus that intensified as I moved my hands closer to the screen.

EDIT: In fact, it landed on a higher & lower hand position controlling the speed but that wasn’t intentional – I need to review it again(!).

A few tweaks later and I had a second piece.

I returned to my first to discover the success of my second leaving me wanting the interaction to be tighter. I explored how to process the data generated from my finger better and adjusted it accordingly.

Capturing Success

I completed my two art pieces and even decided on the names for them; a (short) series called InMotion.

However, it was a difficult challenge to capture the essence of the art in a video. Unfortunately, I wasn’t able to create the tidy videos that I wanted in four days (but I will create them and share them soon, they’re likely below by the time this is up online).

 

As a creative person, I believe it’s important to explore new things and challenge myself with new mediums. I learned a lot and as I knew would be the case, have only begun to discover this world. But it means my imagination is now full once more. You should try it.

Although I can’t take full credit for these pieces, I took a lot from the others who taught me how to do this (and I’m sure I still don’t quite understand everything). Receiving the inheritance of artists’ current and past is part of every creative practice. Not so we can merely copy but so we can learn from the others who have taken ground before us. So although I acknowledge I stand on their experience and ability to teach me, these pieces are still my take on it.

So, what will you create next? What new medium are you going to try?