“A Fun Game For Me”
Collaboration with Scott Leinweber
Disclaimer: Due to run-time issues between my programs and Unity, I’m currently not able to get an in-time screen recording of the live interaction.
Our voice is one of our first actuating and collaborative tools.
I pursued a concept where I poise my voice as the authority to manipulate an object based on my satisfaction or dissatisfaction with it. Stemming from my interest in artifacts of utterance and how verbal-visual systems facilitate cross-contextual translation, I’ve begun to explore how voice interactions construct intimate power structures between machines and people.
This is a game that is fun for me and is only for me.
To understand how to build a speech detection and analysis system for this context, I read up on modern speech processing techniques and phrase to phoneme transcription systems while also exploring different models of audiovisual meaning-making.
This project consists of applications: Max/MSP, OSC, Wekinator and Unity. I’m also using a MOTU to interface a mic with the Max patch, doing the pre-processing for incoming audio.
Max Patch Functionality: Extracts 25 Bark Coefficients, the fundamental pitch and peak amplitude of an incoming audio signal, sends 27 output features using port 6448 over Open Sound Control.
Wekinator Neural Net Model: Maps 27 features from Max to 1 output.
Unity Scripts (UnityOSC/C#): Listen to output value from Wekinator to control the Y scaling of the cylinder and brightness of the point lights.
Rebecca Fiebrink’s Machine Learning for Artists and Musicians course at Goldsmith’s University, https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v
Cognitive Architectures: Models of Perceiving and Interpreting the World https://medium.com/r/?url=https%3A%2F%2Fwww.wikipendium.no%2FTDT4137_Cognitive_Architectures
Unity OSC Receiver by heaversm: https://medium.com/r/?url=https%3A%2F%2Fgithub.com%2Fheaversm%2Funity-osc-receiver
Special thanks to
Thank you to Scott Leinweber, a collaborator on this project. Scott’s major contribution for this instance of the project was reworking C# scripts to get Unity OSC up and running
Thank you to Luke Dubois, composer, artist and co-creator of Jitter for Max/MSP who generously sat down, schooled me on voice and noise processing in Max and helped me rework my patch
Thank you to Aman Tiwari for helping me debug in dire times of need
Thank you to Claire Hentschker for project consulting, conceptual free-styling and moral support
Thank you to Nitesh Sridhar for misc.!
Thank you to Golan and the Frank-Ratchye Studio for Creative Inquiry for giving me a space and community to work in