Project 3 – Interact: BOO, by John Horstman & Nisha Kurani

by jhorstma @ 9:14 am 21 February 2011

Mario Boo drawn in real space with Kinect, by Horstman & Kurani

The idea behind this project was to use the Kinect to create an illusion for the viewer that items displayed on the computer screen existed in the real world. This is accomplished by drawing a virtual character on-screen in a location that the viewer can’t see in the real world – i.e., the viewer’s “blind spot.” The virtual character moves to stay in the viewer’s blind spot as the user moves around, so that the viewer can never see the real-world location of where the virtual character would be.

For the virtual character we selected the “Boo” ghost from the Super Mario Brothers video game series.  In the games, the Boo sneaks up on Mario when Mario is facing away from Boo.  When Mario faces Boo, Boo stops advancing on Mario and hides his face.  Boo’s behavior is similar enough to our goal behavior to make him a great candidate to star in our project.


There are two features we considered critical to the success of the project:

  • Positioning Boo correctly in relation to the user
  • Drawing Boo in front of or behind objects depending on his location

The Boo location is calculated from the cross product of two OpenNI skeleton limb vectors: the neck and the right shoulder. This cross product gives us the 3D position of the Boo in relation to the base of the viewer’s neck. From this, we can calculate the position of the Boo in the frame simply by adding the Boo’s relative position to the position of the viewer’s neck.

The z-value of the vector is used for two critical calculations. First, because the z-value is the depth of the Boo in the scene, it is used to determine whether the Boo is drawn in front of or behind the objects in the scene. The z-position of the Boo is compared to the z-value for each pixel in the scene’s depth pixel array. If the Boo z-position is less than the depth value of the scene, then the appropriate Boo pixel is drawn; otherwise, the scene pixel is drawn.

Second, our code uses the z-value of the vector to determine which Boo image is drawn. If the z-value is positive, then the Boo is positioned behind the viewer looking at the viewer’s back, and we should be able to see Boo’s face on the computer screen. If the z-value is negative, the Boo is closer to the Kinect than the viewer is, so the Kinect is looking at the back of the Boo. Our code loads a different Boo image for the Boo’s back than his front, and the Boo is drawn accordingly.

Once those critical features were implemented, we added the following details:

  • Scaling Boo based on his proximity to the camera – This was accomplished by plugging the aforementioned z-value of the Boo location vector into an openFrameworks resize() function.
  • Gentle floating motion – Everyone knows that ghosts hover, and we made Boo slowly waver up & down by plugging an incrementing variable into a cosine function and using the result as a y-offset for Boo’s position.
  • Evil laugh & background music – Boo’s menacing laugh is triggered by his position as he rotates around you, specifically when the z-value and x-value of his position vector change to positive. The background music is taken from the ghost house levels of Super Mario World, setting the tone for a spooky encounter.


All programming for this project was performed in openFrameworks. The following addons made our project possible:

  • ofxKinect – Processing Kinect input
  • ofxOpenNI – Skeleton recognition
  • ofxVectorMath – Vector calculations

We worked toward our vision in the following sequence:

Running the OpenNI skeleton recognition successfully in openFrameworks

Placing Boo in relation to the tracked skeletons


Vector calculation for Boo location


Depth calculations for Boo/object occlusions


Swapping Boo images to face the user


Scaling Boo based on distance from the Kinect camera


Gentle floating motion for Boo


Sound effects & background music


We initially struggled to find a robust method for calculating the Boo position, with unsuccessful attempts at using trigonometry & the Pythagorean theorem. After learning that openFrameworks can treat points as vectors, using the cross product was the simplest & strongest solution.

The Boo/object occlusion was another serious hurdle, largely because of our difficulty understanding the data format of the depth pixels. Many thanks to Ben Gotow for helping us to figure this out.

This project was the first time that either one of us had programmed with openFrameworks, and we’re both novices at C++. This proved to be quite a challenge unto itself, given the time constraints of the assignment. However, a few late night coding sessions and some very helpful classmates got us where we needed to be.


We owe the success of this project to the following people:

Golan Levin

Dan Wilcox

Ben Gotow

Contributors to the ofxKinect and ofxOpenNI libraries

The presentation

Final Presentation Slides (PDF)

The source code


This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2019 Interactive Art & Computational Design / Spring 2011 | powered by WordPress with Barecity