Project 3: 2D ♥ 3D – Mauricio Giraldo

by Mauricio @ 9:36 am 23 February 2011

2D ♥ 3D is a Kinect-based online multiuser interactive environment. The project is still under development and currently allows multiple web-based users to interact with and augment the physical space of the Kinect participants with virtual objects.


3D is the talk of the hour. From cinema to the living room, 3D is becoming simultaneously a selling and a divisive point. The point of this project is to mesh both points of view and make a commentary about the banality of this “issue”. The project is inspired by many previous works including:

The Bauhaus school of art, architecture and design also explored theatre. I was mostly interested in its highly geometric and formal qualities.

Peter Gabriel’s Sledgehammer video demonstrated an interesting optical phenomenon: human shapes may be inferred by the sole movement of points.

UYUYUY is a short film by Santiago Caicedo which, while designed to be watched with 3D glasses, is mostly composed of 2D elements.

The web has always been a medium of my interest. I try to include web-related functionality in my projects. The Red Agency website has an interesting chat functionality mixed with navigation and 2D parallax scrolling interface.

Technical exploration

I wanted to have a multiuser conversation between the physical and the virtual; 3D and 2D. This was my first C++ project so the learning curve was quite steep and full of obstacles. I started exploring the networking possibilities in openFrameworks including ofxOsc and ofxNetwork deciding upon the latter communicating with the web via a Java-based XMLSocket server (I know “basic” web sockets perform better but not enough time to explore that).

The project also uses ofxOpenNI for skeleton detection and ofxBox2d for physics/collision detection. An initial attempt was made to use ofxMSAPhysics but for some reason I could not make it work well with ofxOpenNI. The web interface uses Flash.

One challenge was how to visualize and represent the activity in either end (Flash being the 2D environment and openFrameworks the 3D). Further exploration must be made in order to have a more interesting visual representation in both ends.


The two main components of this project are mutually dependent: the 3D-interacting users require that web users are present and creating objects they can interact with. 2D web users require the “presence” of 3D skeletons for them to observe and affect. This “conversation” between 2D and 3D is the main element of the project.

Download the code (ZIP 20 MB) which is completely unsupported. Contains:

  • Flash CS5 .FLA file (for the web-based interface)
  • Eclipse Java project folder (for the socket server)
  • openFrameworks project folder

Presentation (PDF 471 KB)

1 Comment

  1. Hi Mauricio, here are comments from the crit.

    You’re doing an excellent job, for the most part, at explaining some pretty complicated concepts in concise ways. Also, excellent job chillin’, Riley.

    From Team America: “You have balls. I like balls.”

    The goal of getting Skeletons into Flash seems key. That the flash users cannot see the skeletons seems a pretty big deal for me. The presentation explained technical issues well, but need a summary of the project before anything else. Riley’s work just not being shown at all was strange – could we at least see it and plans for integration? It just seems totally dropped.

    I think there could be more variation in terms of interaction. I don’t really get the feedback from the web users i.e. I don’t understand how the web interaction affects the balls.

    I really like the interaction and the demo was great./

    The presentation felt a little slow at first, I didn’t know how to apply the inspirational slides ot what you created at the end.

    I think the project needs work, but I can see how miscommunication played a large part into the process in general… Good luck for the future, I think if you continue to develop it, the kinect user vs. internet users dynamic could be really interesting.

    [Golan: So, after 10 minutes, i FINALLY understand WHAT your project is. The project itself is great but I have some issues with the presentation. While I get the idea that you’re interested in the tension between 2D & 3D, we’re 8 minutes into the presentation and I’m still not sure WHAT you were working towards, WHAT you made. Several minutes spent showing other people’s work… I know we’ve asked for context, but there are two kinds of context we need; (1) historic/cultural context, but also (2) a telescopic overview of WHAT you’re working on. Also, Mauricio, since this is a joint presentation, please try to be more inclusive in your discussion of the project and how it came together from your collaboration with your partner. So far, you’ve only said “I” (“my interests”, “I developed”, “I had to do a custom thing”). Why not..let Riley speak too? What were his contributions?

    Your presentation needed a simple overview statement: “an interface by which web users can interact with live people, such that web users can place virtual objects into a simulation around the skeletons of the live users”.

    It looks like you guys weren’t able to agree on a concept. If a collaboration isn’t meant to be, try to realize this as early as possible …]

    Unfortunately, I don’t really buy that it’s not possible to send the skeleton data. Even with 10800 floats per second (and you don’t need floats — you could use low-res single bytes), that’s only 4 bytes * 10,800 = 43200 bytes = 43K/sec, or less. I don’t know much about flash sockets, but that should be totally doable. I mean hell – you should be able to push almost 4.3_MB_ per second.

    I like the inspiration examples. After the inspiration, the technical part got a little slow. Put that after the demo. Playing the game is kinda fun. A little more bounce off my body would be good. I was heading the balls like a soccer ball, and juggling them, but they were too ‘floaty’. Maybe something more like gravity for the fall rate would be good. Also, the body points as balls could have been more fun if they were something more flat to hit the balls easier.

    The inspiration pieces were interesting, but it would have been cool to see how they actually influenced your work – it didn’t seem like they guided your work past the very simple “forced 2D from 3D” concept

    ^ Agree with the above, but I do like the historical perspective. Would be better mixed in throughout.

    Particles look a bit slow. It’s just circles, right?

    Lots of background. Overwhelming. Not sure what the purpose of the online part is? It’s not clear how people on either end are interacting with each other. Is this supposed to be something where anybody anywhere can use it or do a group of people in the same room have to arange to use it together

    I wonder if you could have used some compression to get the skeletons to work.

    ==GEEK ALTERCATION== ? what is this? IT’S A SOCKET yeah, you have to learn how these things work in order to use them effectively … is that being a “geek”? Not an insult.

    While it’s definitely impressive on the technical side, it would really improve the project to have actual images (characters, background, shiney effects) to really establish a connection with your audience rather than just code generated ellipses. Too bad Riley’s stuff didn’t get integrated.

    It was hard to connect what was going on on your screen with what I was doing in the browser. Not totally convinced that I was actually creating any effect.

    Agree that this is very technically impressive. From a visual standpoint I wish there was some way of differentiating the users, perhaps with a subtle color shift or something. There were circular shapes in the users and circles they were interacting with so maybe some variation in shape? How did you collaborate on this project? This was not clear from the presentation. It was nice to see where the inspiration came from but it was a bit long-winded.

    The motion is smooth, and I’m really impressed with the collaborative element (multiple users connecting remotely). I think adding a few more visual flourishes – something with the red balls, perhaps – would have made things a bit more impressive.

    er, what collaborative effort? Riley hasn’t said anything so I can’t gauge his involvement. Would have been nice to see what he did do, even if it coudln’t be applied to the final outcome

    Comment by Golan Levin — 23 February 2011 @ 12:45 pm

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2020 Interactive Art & Computational Design / Spring 2011 | powered by WordPress with Barecity