P3 final blog post

by honray @ 2:16 am 10 May 2011

So, Alex & I decided to create a flocking effect with the Kinect, inspired by Robert Hodgin’s previous work in this area.

For this project, I was involved in the more low-level technical aspects, such as getting the Kinect up and running and finding out how to get the user data from the api. We spent a fair amount of time figuring out how to get the Kinect to work on the PC with a library that would give us both depth and rgb data on the detected user. After trying to use a processing library which eventually did not work, we opted for using OpenNI & nite. Since we both had PC laptops, this was arguably our best bet. Getting the project to work under Visual Basic was a hurdle, since there was poor documentation for the OpenNI library and how to get it up and running on VB.
Once we did get it running, I wrote a wrapper class to detect users, calibrate, and then parse the user and depth pixels so that Alex could easily access them using her particle simulator. We iterated on the code, and looked for ways to improve the flocking algorithm so that it would look more realistic and work better. We also spent a lot of time tweaking parameters and finding out how to represent the depth of the user as seen from the camera. The flocking behavior didn’t look quite as nice when we first implemented it in 3D. As such, we decided to use color to help represent the depth data.

And this is what we ended up with:

 

Ben Gotow-Generative Art

by Ben Gotow @ 8:54 am 23 March 2011

I threw around a lot of ideas for this assignment. I wanted to create a generative art piece that was static and large–something that could be printed on canvas and placed on a wall. I also wanted to revisit the SMS dataset I used in my first assignment, because I felt I hadn’t sufficiently explored it. I eventually settled on modeling something after this “Triangles” piece on OpenProcessing. It seemed relatively simple and it was very abstract.

I combined the concept from the Triangles piece with code that scored characters in a conversation based on the likelihood that they would follow the previous characters. This was accomplished by generating a Markov chain and a character frequency table using combinations of two characters pulled from the full text of 2,500 text messages. The triangles generated to represent the conversation were colorized so that more likely characters were shown inside brighter triangles.

Process:

I started by printing out part of an SMS conversation, with each character drawn within a triangle. The triangles were colorize based on whether the message was sent or received, and the individual letter brightnesses were modulated based on the likelihood that the characters would be adjacent to each other in a typical text message.

In the next few revisions, I decided to move away from simple triangles and make each word in the conversation a single unit. I also added some code that seeds the colors used in the visualization based on the properties of the conversation such as it’s length.

Final output – click to enlarge!

Looking Outwards – Simulations/Forms

by Chong Han Chua @ 5:23 pm 27 February 2011

I am interested in various growth simulations forming into objects that we can use in our daily lives. Something similar to nervous attack but used in a way to aesthetically shape furniture.

A good example would be http://i.materialise.com/blog/entry/5-amazing-full-sized-furniture-pieces-made-with-3d-printing The entry where the stool takes the form of a flock is extremely interesting. I’m wondering if any work can be done in particle systems to simulate this.

In addition, while researching on fluid simulations, I came across this, http://memo.tv/ofxmsafluid marked here for reference just in case the fluid idea in my head connects

I’ve seen a few wonderful processing 3d printing simulation of organic forms. http://www.michael-hansmeyer.com/projects/project4.html aswell as this awe generating piece http://www.sabin-jones.com/arselectronica.html

Lastly, my buddy’s work at www.supabold.com

Huaishu Peng + Charles Doomany: Project 3- Neurospasta

by cdoomany @ 3:22 am 25 February 2011

Neurospasta (Greek word usually translated as “puppets”, which means “string-pulling”, from nervus, meaning either sinew, tendon, muscle, string, or wire, and span, to pull), is a game without any defined objective or goal, but rather a platform for experimentation and play.

Neurospasta is am interactive two-player game in which the player’s physical movements are mapped to a 2D avatar. The game’s graphic interface consists of a set of manipulation functions which enable players to interact with their avatar as well as the other player’s.

In terms of software, we used Open NI for the skeletal tracking and OpenFrameworks for the UI and texture mapping.

*For future consideration, Neurospasta would include a larger selection of manipulation functions and in an ideal environment would be able to capture and generate avatar textures for accommodating new player identities.

Susan Lin + Chong Han Chua — Balloon Grab, Project 3 Final

by susanlin @ 12:42 pm 23 February 2011

The Technical Magic

Technically, this project was lots of fun and frustration, all in the same sentence. The kinect depth map is not a very stable in general. Blob detection tends to be unstable across frames as a result.

The hand detection algorithm is overtly complex but it works like this:

  1. For every depth of 15 values or so, it thresholds front and back.
  2. For the thresholded image, it uses the openCV contour finder to look for blobs within a certain range.
  3. Cull if the blobs are too high or low in the frame.
  4. Calculate the ratio between blob and boundingRect.
  5. Calculate the distances between the centroid and the perimeter. Cull if there are no more than 4 peaks in the difference in distances. The idea is to start the recognition by looking at an open hand.
  6. Iterate through current row of hands. If there is already an existing hand in the same location, assume they are the same and merge them together.
  7. If not, add a new hand.
  8. There is also a lifetime function to calculate basically a confidence level for the blob. Because the kinect drop frames, it is essential to actually “remember” that there was a hand in the same location.

The closed hand detection uses the above, but looks if the smaller blob overlaps the existing larger area. As we can observe in the video, it is fairly good but not entirely robust. There are a few techniques to improve detection going forward.

  1. Use real dimensions to calculate length. We can estimate hand size by using real measurements.
  2. Use bounding polygon versus blob instead of bounding rect. This way we can account for the “holes” in the hand and get a more accurate result.
  3. Remember the size, depth and location and guess intelligently.

The rest of the sketch uses box2d and various other enhancements such as a gradient drawer that takes in any arbitrary RGB color on each end. Another challenge that we had in the project was to deal with C++ code. C++ is en entirely too verbose language compared even to Java, which says alot. Another bug that I encountered was that the vector class actually uses the copy constructor on the push_back function. This actually led to 2 different copies of the same object at one point in the program. In short, pointers is such a big hairy mess. However, this was solved in the version demoed, hence the animations.

All in all, pretty fun! I’ll continue in working on the robustness of the hand detection and hopefully able to get some hand gestures going on.

Process + Back-story

At first, we were planning to generate the art using code. The first two prototypes were done in Flash and Processing, to get the feel for the physics of the floating and collisions. However, when porting to C++ it became evident that our project’s scope was somewhat large. We cut trying to generate images and having implementing a system that animated the pngs.

As for the story, we wanted to play with the idea of the “Red String of Fate.” Ran out of the time, but ideally, the hands would be tied together with red string and the players would not be able to float unless all participants were clenching their hands (grabbing balloons). There was a lot of thought for all the cases. Here’s a snippet:

2 Hands:
Co-op
Rise: can only rise when both hands are closed
Fall: when EITHER hand is open

1 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op

3 Hands:
Co-op with “weighted” areas on string
Rise: all 3
Fall: any 1 hand
1 open, 2 closed: can’t even float up
2 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op

shawn sims-roboScan-Project3

by Shawn Sims @ 12:36 pm

roboScan is a 3D modeler + scanner that uses a Kinect mounted on a ABB4400 robot arm. Motion planning and RAPID code are produced in Robot Studio and Robot Master. This code sends movement commands and positions to the robot as well as the 3D position of the camera. C++ and openFrameworks are used to plot the depth data of Kinect in digital 3D space to produce an accurate model of the environment.

This work was done by Shawn Sims and Karl Willis as the first project of the research group Interactive Robotic Fabrication. The facilities of the Digital Fabrication Lab in the School of Architecture at Carnegie Mellon were used in the making of this project.

There were many technical hurdles along the way. We needed to “teach” the robot the coordinates of the Kinect IR camera in order to accurately move the perspective through 3D space. We also attempted to calibrate the Kinect. We successful did this but it is in fact hard to tell if it made a difference. All-in-all we learned a lot about the robot and about the difficulties of moving a camera around and still trying to extract continuous and accurate data.

Many thanks to dFab, Zach Ali, and Jeremy Ficca

Project 3 | Kinect Flock | Alex Wolfe and Ray Lin

by Alex Wolfe @ 12:36 pm

Developed using Cinder + OpenNI + NITE +Xbox Kinect

After seeing what the Kinect was capable of, we were interested in using it to generate a particle system that would respond to user movement and depth placement. By pulling out the depth map, we were able to isolate the user to an extremely high level of accuracy.

The particles flock to the silhouette of the user when he or she is still and exhibits flocking/swarming behavior otherwise, creating mesmerizing shapes that ebb and flow between the recognizable and unknown.

Creation Process

Each point  that comprises my body pulled from the kinect depth map has a gravitational pull on one particle in the simulation. The strength of this force is inversely related to how fast I’m moving, so when I stand perfectly still the particle zooms to the point it corresponds to, and when I move its free to wander and behave on its natural flocking tendency. Thus you get these really compelling visuals as my silhouette breaks and reforms depending on how fast I’m moving.

Ray set out getting the kinect to actually work with Windows and pull the depth map/user data out from it using the OpenNI library and NITE drivers. Almost all of the current kinect support is for Macs, so this was no small feat! We attempted our project in Processing and Openframeworks, but finally settled on Cinder for our developing environment.

early concept sketch

 

 

Here’s a short clip of some process sketches I did in Processing in order to explore the flocking behavior before porting it over with some more hard hitting graphics to Cinder. And countless thanks to Robert Hodgin for his fantastic “Hello Cinder” tutorial

 

more from Alex: alexwolfe.blogspot.com

more from Ray:

Kinect VJ’ing

by chaotic*neutral @ 10:30 am

playing around testing the kinect @ brillobox pittsburgh with Keeb$, Cutups, and Freddy Todd (detroit)
stackinpaper.com/​

Openframeworks.cc
tuio.org/​
osculator.net/​
OF, FFT OSC, MSA 3D Shape, Kinect, TUIO

SamiaAhmed-Final-Interactttt!

by Samia @ 9:48 am

Process:

I wanted to create an interaction that was  physically reactive — given that the kinect requires you to engage physically, it made sense to me to bring the interaction off of the screen and into a real space. Golan pointed me towards Marie Sester’s Access project, which does exactly that, engage an inbetween space with a spotlight that automatically follows people.

 

There were lots of techincal challenge in this project for me — getting the right drivers, learning about working with hardware and sending information over serial, dmx protocol, as well as working with the kinect, (very, very) basic image processing, blob tracking.

I approached the project from two ends, first, working with a strobe light (which has only 4 channels) and blob tracking separately, and then finally meeting in the middle so that the blob position controlled the light position of a moving light.

In the end math got in the way a little bit, instead of calculating the position of where the lights should pan and tilt to using trig, I created a grid over the kinect data, and plotted the light movement to points, which was conceptually easier for me to understand. It works, but there’s some choppy-ness.

Project 3: 2D ♥ 3D – Mauricio Giraldo

by Mauricio @ 9:36 am

2D ♥ 3D is a Kinect-based online multiuser interactive environment. The project is still under development and currently allows multiple web-based users to interact with and augment the physical space of the Kinect participants with virtual objects.

Background

3D is the talk of the hour. From cinema to the living room, 3D is becoming simultaneously a selling and a divisive point. The point of this project is to mesh both points of view and make a commentary about the banality of this “issue”. The project is inspired by many previous works including:

The Bauhaus school of art, architecture and design also explored theatre. I was mostly interested in its highly geometric and formal qualities.

Peter Gabriel’s Sledgehammer video demonstrated an interesting optical phenomenon: human shapes may be inferred by the sole movement of points.

UYUYUY is a short film by Santiago Caicedo which, while designed to be watched with 3D glasses, is mostly composed of 2D elements.

The web has always been a medium of my interest. I try to include web-related functionality in my projects. The Red Agency website has an interesting chat functionality mixed with navigation and 2D parallax scrolling interface.

Technical exploration

I wanted to have a multiuser conversation between the physical and the virtual; 3D and 2D. This was my first C++ project so the learning curve was quite steep and full of obstacles. I started exploring the networking possibilities in openFrameworks including ofxOsc and ofxNetwork deciding upon the latter communicating with the web via a Java-based XMLSocket server (I know “basic” web sockets perform better but not enough time to explore that).

The project also uses ofxOpenNI for skeleton detection and ofxBox2d for physics/collision detection. An initial attempt was made to use ofxMSAPhysics but for some reason I could not make it work well with ofxOpenNI. The web interface uses Flash.

One challenge was how to visualize and represent the activity in either end (Flash being the 2D environment and openFrameworks the 3D). Further exploration must be made in order to have a more interesting visual representation in both ends.

Interdependence

The two main components of this project are mutually dependent: the 3D-interacting users require that web users are present and creating objects they can interact with. 2D web users require the “presence” of 3D skeletons for them to observe and affect. This “conversation” between 2D and 3D is the main element of the project.

Download the code (ZIP 20 MB) which is completely unsupported. Contains:

  • Flash CS5 .FLA file (for the web-based interface)
  • Eclipse Java project folder (for the socket server)
  • openFrameworks project folder

Presentation (PDF 471 KB)

OF 3d

by chaotic*neutral @ 9:27 am

Initial graphics for integrating with networked kinect skeletons.

Dane + Max !Kinect Project – You Mirror Me

by dpieri @ 5:00 am

You Mirror Me is a site for creating collaborative video animations. Animations start with one base sequence that is a five second video recorded by one person. This video is split up into 50 frames which are then served to users. When contributing individual final frames you are served a random frame from a base video and have five seconds to re-create it. A photo is taken by your webcam and sent back to the server.

Once each frame of a base video has been recreated a new video is stitched together and shown on the home page.

Here is a link to our presentation:

http://danepieri.com/prez.pdf

Project 3 – Huaishu Peng & Charles Doomany

by huaishup @ 1:06 pm 21 February 2011

Neurospasta

a Kinect based 2-player interaction game

 

CONCEPT

For our main concept we will be developing an experimental game in which two players have the ability to manipulate their own avatar as well as the other player’s avatar. The players will have the choice to interact, fight with their avatar against one another, or just simply explore different ways of manipulating the appearance and/or ability of their avatar.
The player’s avatar will consist of a rag-doll/skeletal model which will mimic the actions of the player via kinect video/depth capture.

The graphic interface will provide a menu for selecting a specific body part and a manipulation function menu for changing certain characteristics or abilities of the selected part.

 

THE GAME

After trying really hard for a week, we implemented Neurospasta (Greek word usually translated as “puppets”, which means “string-pulling”, from nervus, meaning either sinew, tendon, muscle, string, or wire, and span, to pull), a game without any defined objective or goal, but rather a platform for experimentation and play.

Neurospasta is am interactive two-player game in which the player’s physical movements are mapped to a 2D avatar. The game’s graphic interface consists of a set of manipulation functions which enable players to interact with their avatar as well as the other player’s.
In terms of software, we used Open NI for the skeletal tracking and OpenFrameworks for the UI and texture mapping.

 

FUTURE PLAN

*For future consideration, Neurospasta would include a larger selection of manipulation functions and in an ideal environment would be able to capture and generate avatar textures for accommodating new player identities.

Meg Richards – Project 3

by Meg Richards @ 1:01 pm

Mix & Match Interactive Toy


Mix and match toys and games are a popular activity for children. It typically appears as an upright cylinder or rectangular prism with multiple standing human or animal figures independently painted or carved into the surface. The shape is divided into thirds that rotate independently, allowing the different heads, torsos, and legs of the painted or carved figures to align. They have existed in many forms of media and targeted different age groups over their two-hundred year history. Changeable Gentlemen, a card game published around 1810 targeting adults, was one of the earliest manufactured mix and match games. Adaptations in base material and figure subjects made it more suitable for a younger audience, and modern incarnations are generally as a classic toy for children.

I chose a mix and match toy as the interaction subject because it’s a ubiquitous toy and gives almost all the players a baseline familiarity. Also, while the media and figures are numerous over the last two hundred years, the method of interaction has always been hand manipulation. The Kinect enables the player to use their entire body to control the figure. The figures are from a card game “Mixies” published in 1956 by Ed-U-Cards. The player can rotate through different heads, torsos, and feet by swiping above, at, or below the player’s midsection, respectively. Swiping to the left will rotate to the next card to appear, and swiping to the right will rotate to the previous card. The figure will follow a player’s horizontal and vertical movement, bending, and if they turn around.


I used OpenFrameworks, ofxKinect, and OpenNI. Once the skeleton is detected, the images are overlayed at the position and angle of the corresponding section. Hands and arms do not directly affect card placement, so they are free to control the card rotation. Sweep/wave detection is simply a matter of the end of the left or right arm moving over a certain horizontal distance within a time threshold. If the user’s back is turned, the back of the card is displayed instead of the obverse.

Project 3 – Paul Miller and Timothy Sherman

by ppm @ 12:53 pm

Magrathea uses the kinect camera to dynamically generate a landscape out of any structure or object. The kinect takes an depth reading of what’s built on the table in front of it, which is then rendered live onscreen as terrain using openFrameworks and OpenGL.

The depth image is used as a heightmap for the terrain. A polygon mesh gradually morphs to match the heightmap, creating a nice rise and fall behavior. Textures are dynamically applied based on the height and slope of the mseh. For example, steep slopes are given a rocky texture, and flatter areas a grassy one. As the user builds and removes, the landscape correspondingly grows and sinks out of the ocean, shifting into a new configuration.

Landscapes can be made from anything, such as blocks, boxes, the human body, and even a giant mound of dough.

We both learned OpenGL and openFrameworks for this project.

If we were to continue this project further, we’d do so by adding more textures with more complex conditions, learning shaders and improving the graphics, populating the world with flora and fauna based on certain conditions, and possibly allowing for color-coded objects that could be recognized and rendered as specific features, say, a statue, or giant Yggdrasil like tree.

Project 3–Interaction–Kinect Hand Waving

by Ben Gotow @ 10:16 am

What if you could use hand gestures to control an audio visualization? Instead of relying on audio metrics like frequency and volume, you could base the visualization on the user’s interpretation of perceivable audio qualities. The end result would be a better reflection of the way that people feel about music.

To investigate this, I wrote an OpenFrameworks application that uses depth data from the Kinect to identify hands in a scene. The information about the users’ hands – position, velocity, heading, and size – is used to create an interactive visualization with long-exposure motion trails and particle effects.

There were a number of challenges in this project. I started with Processing, but it was too slow to extract hands and render the point sprite effects I wanted. I switched to OpenFrameworks and started using OpenNI to extract a skeleton from the Kinect depth image. OpenNI worked well and extracted a full skeleton with wrists that could be tracked, but it was difficult to test because the skeletal detection took nearly a minute every time the visualization was tested. It got frustrating pretty quickly, and I decided to do hand detection manually.

Detecting Hands in the Depth Image
I chose a relatively straightforward approach to finding hands in the depth image. I made three significant assumptions that made realtime detection possible:

  1. The users body intersects the bottom of the frame
  2. The user is the closest thing in the scene.
  3. The users hands are extended (at least slightly) in front of their body

Assumption 1 is important because it allows for automatic depth thresholding. By assuming that the user intersects the bottom of the frame, we can scan the bottom row of depth pixels to determine the depth of the users body. The hand detection ignores anything further away than the user.

Assumptions 2 and 3 are important for the next step in the process. The application looks for local minima in the depth image and identifies the points nearest the camera. It then uses a breadth-first search algorithm to repeatedly expand the blob to neighboring points and find the boundaries of hands. Each pixel is scored based on it’s depth and distance from the source. Pixels that are scored as part of one hand cannot be scored as part of another hand and this prevents near points in the same hand from generating multiple resulting blobs.

Interpreting Hands
Once pixels in the depth image have been identified as hands, a bounding box is created around each one. The bounding boxes are compared to those found in the previous frame and matched together, so that the user’s two hands are tracked separately.

Once each blob has been associated with the left or right hand, the algorithm determines the heading, velocity and acceleration of the hand. This information is averaged over multiple frames to eliminate noise.

Long-Exposure Motion Trails
The size and location of each hand are used to extend a motion trail from the user’s hand. The motion trail is stored in an array. Each point in the trail has an X and Y position, and a size. To render the motion trail, overlapping, alpha-blended point sprites are drawn along the entire length of the trail. A catmul-rom spline algorithm is used to interpolate between the points in the trail and create a smooth path. Though it might seem best to append a point to the motion trail every frame, this tends to cause noise. In the version below, a point is added to the trail every three frames. This increases the distance between the points in the trail and allows for more smoothing using catmul-rom interpolation.

Hand Centers
One of the early problems with the hand tracking code was the center of the blob bounding boxes were used as the input to the motion trails. When the user held up their forearm perpendicular to the camera, the entire length of their arm was recognized as a hand. To better determine where the center of the hand was, I wrote a midpoint finder based on iterative erosion of the blobs. This provided much more accurate hand centers for the motion trails.

Particle Effects
After the long-exposure motion trails were working properly, I decided that more engaging visuals were needed to create a compelling visualization. It seemed like particles would be a good solution because they could augment the feeling of motion created by the user’s gestures. Particles are created when the hand blobs are in motion, and more particles are created based on the hand velocity. The particles stream off the motion trail in the direction of motion, and curve slightly as they move away from the hand. They fade and disappear after a set number of frames.

Challenges and Obstacles
This is my first use of the open-source ofxKinect framework and OpenFrameworks. It was also my first attempt to do blob detection and blob midpoint finding, so I’m happy those worked out nicely. I investigated Processing and OpenNI but chose not to use them because of performance and debug time implications, respectively.

Live Demo
The video below shows the final visualization. It was generated in real-time from improv hand gestures I performed while listening to “Dare you to Move” by the Vitamin String Quartet.

JamesMulholland-Project3-Interaction-WillOTheWisp

by James Mulholland @ 9:38 am

Will-O-The-Wisp

Scary Tale or Fairy Tale?

Background:
A Will-o-the-Wisp in western folklore is a small glowing creature that appears at night to travelers, often near swamps or marshes. In these stories, this sprite lures these unknowing wanderers into the deep marsh to be lost. Once the individual is far enough away from safety, the glow disappears and the person is never heard from again.


However… this will-o-the-wisp is benevolent. And fun to play with! If you’re lucky, you may just find your way out of the swamp.

Project Inspiration:
Contact Juggling – (e.g. crystal balls from Labyrinth)

Tools:
openFrameworks! [code]
FireWorks and Photoshop [graphics]

Successes:

  • Ball moving along skeleton
  • Flows across hands
  • Floats nicely above appendages
  • 2-player!

Further improvement:

  • position averaging (still jumpy)
  • environmental graphics (incl. background subtraction)

Process:

We used the openFrameworks with OpenNI add-on to provide the skeleton data. The ball follows the skeleton to a point lower to the ground (like gravity). Size of ball image is mapped to the depth captured by the kinect to give the illusion of depth.

Fog effect – created using difference clouds in Photoshop and displayed with alpha adjusted in openFrameworks.
Glowing Ball and Glow Effect – drawn in FireWorks with semi-transparent pixels and placed into video.

Ball image placement is offset perpendicular to the skeleton being tracked and placed relative to the outermost edge of the person (achieved through depth threshold).

Credits:

Behavior tracking + movement
by Le Wei

Depth thresholding + graphics
by James Mulholland

Project 3 – Interact: BOO, by John Horstman & Nisha Kurani

by jhorstma @ 9:14 am

Mario Boo drawn in real space with Kinect, by Horstman & Kurani

The idea behind this project was to use the Kinect to create an illusion for the viewer that items displayed on the computer screen existed in the real world. This is accomplished by drawing a virtual character on-screen in a location that the viewer can’t see in the real world – i.e., the viewer’s “blind spot.” The virtual character moves to stay in the viewer’s blind spot as the user moves around, so that the viewer can never see the real-world location of where the virtual character would be.

For the virtual character we selected the “Boo” ghost from the Super Mario Brothers video game series.  In the games, the Boo sneaks up on Mario when Mario is facing away from Boo.  When Mario faces Boo, Boo stops advancing on Mario and hides his face.  Boo’s behavior is similar enough to our goal behavior to make him a great candidate to star in our project.

Features

There are two features we considered critical to the success of the project:

  • Positioning Boo correctly in relation to the user
  • Drawing Boo in front of or behind objects depending on his location

The Boo location is calculated from the cross product of two OpenNI skeleton limb vectors: the neck and the right shoulder. This cross product gives us the 3D position of the Boo in relation to the base of the viewer’s neck. From this, we can calculate the position of the Boo in the frame simply by adding the Boo’s relative position to the position of the viewer’s neck.

The z-value of the vector is used for two critical calculations. First, because the z-value is the depth of the Boo in the scene, it is used to determine whether the Boo is drawn in front of or behind the objects in the scene. The z-position of the Boo is compared to the z-value for each pixel in the scene’s depth pixel array. If the Boo z-position is less than the depth value of the scene, then the appropriate Boo pixel is drawn; otherwise, the scene pixel is drawn.

Second, our code uses the z-value of the vector to determine which Boo image is drawn. If the z-value is positive, then the Boo is positioned behind the viewer looking at the viewer’s back, and we should be able to see Boo’s face on the computer screen. If the z-value is negative, the Boo is closer to the Kinect than the viewer is, so the Kinect is looking at the back of the Boo. Our code loads a different Boo image for the Boo’s back than his front, and the Boo is drawn accordingly.

Once those critical features were implemented, we added the following details:

  • Scaling Boo based on his proximity to the camera – This was accomplished by plugging the aforementioned z-value of the Boo location vector into an openFrameworks resize() function.
  • Gentle floating motion – Everyone knows that ghosts hover, and we made Boo slowly waver up & down by plugging an incrementing variable into a cosine function and using the result as a y-offset for Boo’s position.
  • Evil laugh & background music – Boo’s menacing laugh is triggered by his position as he rotates around you, specifically when the z-value and x-value of his position vector change to positive. The background music is taken from the ghost house levels of Super Mario World, setting the tone for a spooky encounter.

Process

All programming for this project was performed in openFrameworks. The following addons made our project possible:

  • ofxKinect – Processing Kinect input
  • ofxOpenNI – Skeleton recognition
  • ofxVectorMath – Vector calculations

We worked toward our vision in the following sequence:

Running the OpenNI skeleton recognition successfully in openFrameworks

Placing Boo in relation to the tracked skeletons

 

Vector calculation for Boo location

 

Depth calculations for Boo/object occlusions

 

Swapping Boo images to face the user

 

Scaling Boo based on distance from the Kinect camera

 

Gentle floating motion for Boo

 

Sound effects & background music

Challenges

We initially struggled to find a robust method for calculating the Boo position, with unsuccessful attempts at using trigonometry & the Pythagorean theorem. After learning that openFrameworks can treat points as vectors, using the cross product was the simplest & strongest solution.

The Boo/object occlusion was another serious hurdle, largely because of our difficulty understanding the data format of the depth pixels. Many thanks to Ben Gotow for helping us to figure this out.

This project was the first time that either one of us had programmed with openFrameworks, and we’re both novices at C++. This proved to be quite a challenge unto itself, given the time constraints of the assignment. However, a few late night coding sessions and some very helpful classmates got us where we needed to be.

Props

We owe the success of this project to the following people:

Golan Levin

Dan Wilcox

Ben Gotow

Contributors to the ofxKinect and ofxOpenNI libraries

The presentation

Final Presentation Slides (PDF)

The source code

BOO


Project 3 – BOO

by nkurani @ 9:11 am

Mario Boo drawn in real space with Kinect, by Horstman & Kurani

The idea behind this project was to use the Kinect to create an illusion for the viewer that items displayed on the computer screen existed in the real world. This is accomplished by drawing a virtual character on-screen in a location that the viewer can’t see in the real world – i.e., the viewer’s “blind spot.” The virtual character moves to stay in the viewer’s blind spot as the user moves around, so that the viewer can never see the real-world location of where the virtual character would be.

For the virtual character we selected the “Boo” ghost from the Super Mario Brothers video game series.  In the games, the Boo sneaks up on Mario when Mario is facing away from Boo.  When Mario faces Boo, Boo stops advancing on Mario and hides his face.  Boo’s behavior is similar enough to our goal behavior to make him a great candidate to star in our project.

Features

There are two features we considered critical to the success of the project:

  • Positioning Boo correctly in relation to the user
  • Drawing Boo in front of or behind objects depending on his location

The Boo location is calculated from the cross product of two OpenNI skeleton limb vectors: the neck and the right shoulder. This cross product gives us the 3D position of the Boo in relation to the base of the viewer’s neck. From this, we can calculate the position of the Boo in the frame simply by adding the Boo’s relative position to the position of the viewer’s neck.

The z-value of the vector is used for two critical calculations. First, because the z-value is the depth of the Boo in the scene, it is used to determine whether the Boo is drawn in front of or behind the objects in the scene. The z-position of the Boo is compared to the z-value for each pixel in the scene’s depth pixel array. If the Boo z-position is less than the depth value of the scene, then the appropriate Boo pixel is drawn; otherwise, the scene pixel is drawn.

Second, our code uses the z-value of the vector to determine which Boo image is drawn. If the z-value is positive, then the Boo is positioned behind the viewer looking at the viewer’s back, and we should be able to see Boo’s face on the computer screen. If the z-value is negative, the Boo is closer to the Kinect than the viewer is, so the Kinect is looking at the back of the Boo. Our code loads a different Boo image for the Boo’s back than his front, and the Boo is drawn accordingly.

Once those critical features were implemented, we added the following details:

  • Scaling Boo based on his proximity to the camera – This was accomplished by plugging the aforementioned z-value of the Boo location vector into an openFrameworks resize() function.
  • Gentle floating motion – Everyone knows that ghosts hover, and we made Boo slowly waver up & down by plugging an incrementing variable into a cosine function and using the result as a y-offset for Boo’s position.
  • Evil laugh & background music – Boo’s menacing laugh is triggered by his position as he rotates around you, specifically when the z-value and x-value of his position vector change to positive. The background music is taken from the ghost house levels of Super Mario World, setting the tone for a spooky encounter.

Process

All programming for this project was performed in openFrameworks. The following addons made our project possible:

  • ofxKinect – Processing Kinect input
  • ofxOpenNI – Skeleton recognition
  • ofxVectorMath – Vector calculations

We worked toward our vision in the following sequence:

Running the OpenNI skeleton recognition successfully in openFrameworks

Placing Boo in relation to the tracked skeletons

 

Vector calculation for Boo location

 

Depth calculations for Boo/object occlusions

 

Swapping Boo images to face the user

 

Scaling Boo based on distance from the Kinect camera

 

Gentle floating motion for Boo

 

Sound effects & background music

Challenges

We initially struggled to find a robust method for calculating the Boo position, with unsuccessful attempts at using trigonometry & the Pythagorean theorem. After learning that openFrameworks can treat points as vectors, using the cross product was the simplest & strongest solution.

The Boo/object occlusion was another serious hurdle, largely because of our difficulty understanding the data format of the depth pixels. Many thanks to Ben Gotow for helping us to figure this out.

This project was the first time that either one of us had programmed with openFrameworks, and we’re both novices at C++. This proved to be quite a challenge unto itself, given the time constraints of the assignment. However, a few late night coding sessions and some very helpful classmates got us where we needed to be.

Props

We owe the success of this project to the following people:

Golan Levin

Dan Wilcox

Ben Gotow

Contributors to the ofxKinect and ofxOpenNI libraries

The presentation

Final Presentation Slides (PDF)

The source code

BOO

Le Wei and James Mulholland – Project 3 Final Will o’ the wisp

by Le Wei @ 8:47 am

temporary!

James will be posting the video at his post here.

Next Page »
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2017 Interactive Art & Computational Design / Spring 2011 | powered by WordPress with Barecity