The Comic Kinect: Maya Irvine, Ward Penney, Emily Schwartzman, Mark Shuster

by mirvine @ 3:47 am 21 February 2011

The Comic Kinect
Our group came together over the realization that everyone wishes they were in a comic book. We were interested in working with multiple users and encouraging them to interact in new ways.

With this in mind:
Our project aims to create an experience that detects, visualizes, and encourages two users’ physical contact with one another.

Identifying the Challenge
After looking at a lot of comics, tv shows, and movies, we came to the conclusion that one of the most significant elements of comic book fighting is the visualization of impact energy or the impending impact of energy. This effect is achieved through the combined use of several techniques such as:


Dynamic Typography

Exploding Elements

and Symbols

These characteristics make up a recognized language of comic book fighting that can be seen across styles and mediums. Here we see similar methods being used in a Manga illustration, a contemporary X-Men panel, and a scene from the 1960’s Batman tv show.

Scoping: Making It Manageable

There were a lot of ideas we wanted to try with this project including tracking velocity, giving depth to graphics, and integrating sound, but after experimenting with speech bubbles and glow we decided to focus our energy on typography and exploding symbols as a way to visualize users’ interactions with one another, and use filters to place them in the comic book world.

Task Breakdown
Given the multifaceted nature of this project, we decided to have a clear separation of tasks amongst our team..

Filters: Mark
The movie A Scanner Darkly was a big inspiration. It’s application of filters borrows heavily from the comic book hand-drawn aesthetic, despite it’s use of code.

Mark started by mastering mean-shifting in openCV, followed by edge detection to create a black outline. Mad props for the valiant attempt at Shaders, and for fixing the insane memory leaks.

Motion Graphics: Emily and Maya
Emily and Maya focused on creating the dynamic visualizations that occur on impact.

We started by deciding by experimenting with several words and symbols, both hand-drawn and generated.

Our effects were created with a number of openFrameworks addons.

Word motion with Tween:
Fading with ofxFadable:
Lightning explosion with ofxBox2d:

Our biggest challenge was making these programs integrate with the skeleton interaction detection. There was much discussion about variables and what information we were going to have access to for each limb. Emily was dubbed the “Addon Ninja.”

Skeleton Interaction Detection: Ward
Ward really took on openNI and did a lot of the heavy lifting as far as detecting limb interaction. We had a lot of entertaining user tests.

Some of our challenges:
Leaking 30mb of memory every second.
Addon and pointer madness!
A very slow debug cycle.
Integrating many different parts.
Animating over time.
Continually shifting scope.

Our Video After Integration:
There are still a lot of bugs to work out in order to achieve the interaction we wanted, but hopefully those will be resolved in the future.

New Demo Video!
After a bit more tinkering with the code, we we able to get a much clearer result. :D


  1. The Fantastic 4! Great work. Below are the group’s Etherpad comments from today’s crit.

    Dealing with 4 people merging code is where something like Git is very helpful. Continuous integration would have also helped.

    I think the filter could be better with a halftone pattern and a somewhat shifted color hue. The only things that should be black are contours of the depth image.

    Also, you have the depth image, so you should really be able to make the skelletons pop. Put a high saturation colored background where people aren’t standing and put a big fat border around the person. Yeah, Golan said it.
    Some sort of histogram tansform might work well.

    I wish there was more glow. It would be -awesome- if the background flashed a color with every hit.

    Ambitious. I don’t envy you having to merge code from 4 contributors. It’s a good proof of concept that you were able to get it to this point.

    Has an 80s music video feel…

    This is fun. Over the top acting should sell it in your video.

    Great job splitting up the workload! Love the filter and effects. Good job :)

    Appreciate the breakdown of how the action effects are achieved.

    I like the breakdown of workload; a lot went into this program, it’s neat to know who did which bit.

    Great presentation. Really cool to see the separate parts and then integration. Seems like there are still some integration issues though.

    Great use of Scott Pilgrim as your example. Sound effects would be really cool too.

    Nice presentation/explanation. Love the parts of the project and how you guys split the work up.

    Cool concept. Way fun!

    Well designed & articulated presentation — exemplary.
    “Everybody wishes they were in a comic book”. Nicely stated motivation.
    Nice identification: “visualizations of impact energy, or impending impact energy”.
    Nice statement that “the bar is high because people have seen something like this before.”
    Good visual research, too.

    I love the divisions of labor, love the fact that you all wrote code. Very important.
    — Mark: 2D surface filters
    — Ward: skeleton processing
    — Maya & Emily: dynamic typography & energy graphics

    good, succinct way to sum up your concept in 2 sentances, good example/background images

    good way to break down the work among the team members

    I like the breakdown of work between team members – it looks like everyone contributed and that’s great.

    I really like the skeleton interaction bit – really great approach, and the visualization of it looks great. I feel like you guys are REALLY close with the final visualization. Just need to get the timing of the graphics right.

    your final looks like a SUPER INTENSE fight

    Comment by Golan Levin — 21 February 2011 @ 1:14 pm
  2. Great video!

    Comment by Dan Wilcox — 26 February 2011 @ 3:41 pm
  3. Above the fold on for all of Friday! (

    Comment by Ward Penney — 27 February 2011 @ 2:58 am
  4. Front page Saturday on! (

    Comment by Ward Penney — 27 February 2011 @ 2:59 am
  5. Kotaku!!5772854/bam-pow-kinect-can-turn-your-life-into-the-batman-tv-show

    Comment by Ward Penney — 1 March 2011 @ 3:02 pm
  6. Funny comments on this one:

    Comment by Ward Penney — 1 March 2011 @ 7:58 pm

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2020 Interactive Art & Computational Design / Spring 2011 | powered by WordPress with Barecity