Ok, so the project has expanded to try to answer an additional questions
- “can 3d gesture interfaces provide an *actual* alternative to current sound editing techniques?”
- “what gestures can be used to improve sound editing?”
- “which gestures are appropriate and intuitive?”
- “what are intuitive/appropriate gestures for the execution/performance of sound?”
This is essentially a gesture recognition and mapping problem. First, how do I accurately register that a specific gesture is being executed? Then, how that gesture does that gesture relate to the sound process, both conceptually and technically?
While researching gestures and sound I found Jules Francoise’s work on Machine Learning systems (Gaussian Mixture Regression) for continuous motion sonification.
Thanks to Shawn Xu from CMU’s HCI program for pointing me towards the Gesture Recognition Toolkit (GRT) for C++, which I’ll be testing the Hidden Markov Model (HMM).
Here’s what I’ve found on machine learning for openframeworks:
- ofxLearn
- kinect + GRT + openFrameworks (tutorial from GRT)
- ofxPattern (older)
Visual
On the visual side of things, I found Kyle McDonald’s ofxAudioDecoder addon, which I’ve managed to blend into the leap motion world.
In the image above, we can see a black line across the waveform. This line is related to the selection gesture as previously illustrated. In this gesture, the space between the index finger and thumb determine size of the selection window. The position of the wrist relates to the starting position of the window.
Sound
On the sound side of things, I’ve had to switch to Max/MSP because **Pure Data was unable to process audio and OSC messages and the same time**, WTF?
This changes things up quite a bit since I was planning on integrating pd entirely into the app with ofxPd. Since Max can’t do this (to my knowledge), I think I’ll search for other alternatives a bit later on, something that’ll allow me to release a single stand alone application that will process the leap motion controller data directly and perform the sound manipulation/synthesis, possibly a C++ library like Tonic or Maximilian. We’ll see what happens.
