#06 (Due 3/2)

Deliverables #06 has two components, due at the beginning of class on Friday March 2:

Looking Outwards #06

For this Looking Outwards assignment, write a brief report on a new media project which has something to do with the body or gesture. Follow the (usual) instructions for a Looking Outwards report here. Please title your blog post nickname-LookingOutwards06, and give it the WordPress Category, LookingOutwards06. For staring points, you are welcome to refer to our most recent lecture notes, CreativeApplications (e.g. their gesture tag), MediaArtTube, or beyond.

SOFTWARE FOR A SKELETON

For this project, you are asked to write software which
creatively interprets, or responds to, the actions of the body.

You will develop a computational treatment for motion capture data (broadly understood). Ideally, both your treatment, and your motion capture data, will be ‘tightly coupled’ to each other: The treatment will be designed for specific motion-capture data, and the motion-capture data will be intentionally selected (or performed) for your specific treatment.

Code templates for Processing, three.js and openFrameworks are here.

Ten Creative Opportunities

It’s important to emphasize that you have a multitude of creative options — well beyond, or alternative to, the simple concept of a “decorated skeleton”. The following ten suggestions, which are by no means comprehensive, are intended to prompt you to appreciate the breadth of the conceptual space you may explore. In all cases, be prepared to justify your decisions.

  1. You may work in real-time (interactive), or off-line (animation). You may choose to develop a piece of interactive real-time software, which treats the mocap file as a proxy for data from a live user (as in Setsuyakurotaki, by Zach Lieberman + Rhizomatiks, shown above in use by live DJs). Or you may choose to develop a piece of custom animation software, which interprets the mocap file as an input to a lengthy rendering process process (as in Universal Everything’s Walking City, or Method Studios’ AICP Sponsor Reel).
  2. You may use more than one body. Your software doesn’t have to be limited to just one body. Instead, it could visualize the relationship (or create a relationship) between two or more bodies (as in Scott Snibbe’s Boundary Functions or this sketch by Zach Lieberman). It could visualize or respond to a duet, trio or even a crowd of people.
  3. You may focus on just part of the body. Your software doesn’t need to respond to the entire body; it could focus on interpreting just a single part of the body (as in Theo Watson & Emily Gobeille’s prototype for Puppet Parade, which responds to a single arm).
  4. You may focus on how an environment is affected by the body. Your software doesn’t have to re-skin or visualize the body. Instead, you can develop an environment that is affected by the movements of the body (as in Theo & Emily’s Weather Worlds).
  5. You may position your ‘camera’ anywhere — including a first-person POV, or with a (user-driven) VR POV. Just because your performance was recorded from a sensor “in front” of you, this does not mean your mocap data must be viewed from the same point of view. Consider displaying your figure in the round, from above, below, or even from the POV of the body itself. (Check out the camera() function in Processing, or the PerspectiveCamera object in three.js, for more ideas. If you’re using three.js, you could also try a WebVR build for Google cardboard.)
  6. You may work in 3D or 2D. Although your mocap data represents three-dimensional coordinates, you don’t have to make a 3D scene; for example, you could use your mocap to control an assemblage of 2D shapes. You could even use your body to control two-dimensional typography. (Helpful Processing commands like screenX() and screenY() , or unprojectVector() in three.js, allow you to easily compute the 2D coordinates of a perspectivally-projected 3D point.)
  7. You may control the behavior of something non-human. Just because your data was captured from a human, doesn’t mean you must control a human. Consider using your mocap data to puppeteer an animal, monster, plant, or even a non-living object (as in this research on “animating non-humanoid characters with human motion data” from Disney Research).
  8. You may record mocap data yourself, or you can use data from an online source. If you’re recording the data yourself, feel free to record a friend who is a performer — perhaps a musician, actor, or athlete. Alternatively, feel free to use data from an online archive or commercial vendor. You may also combine data from different sources; for example, you could combine your own awkward performance, with a group of professional backup dancers.
  9. You can make software which is analytic or expressive. You are asked to make a piece of software which interprets the actions of the human body. While some of your peers may choose to develop a character animation or interactive software mirror, you might instead elect to create “information visualization” software that presents an ergonometric analysis of the body’s joints over time. Your software could present comparisons different people making similar movements, or could track the accelerations of movements by a violinist. You could create quantitative comparisons of people’s different gaits.
  10. You may use sound. Feel free to play back sound which is synchronized with your motion capture files. This might be the performer’s speech, or music to which they are dancing, etc. (Check out the Processing Sound Library to play simple sounds.)

Technical Options & Resources

You are provided with code templates in three different programming environments, loading three different kinds of motion capture data. (Not all possible combinations are supported.):

Code templates for loading and displaying motion capture files in the BVH format have been provided for you in Processing (Java), openFrameworks (C++), and three.js (a JavaScript library for high-quality OpenGL 3D graphics in the browser). You can find these templates collected together here: https://github.com/CreativeInquiry/BVH-Examples. It is probably worthwhile browsing what three.js can do.

We have purchased a copy of Brekel Pro Body v2 for you to use to record motion capture files, and we have installed it on a PC in the STUDIO; it can record Kinectv2 data into BVH, FBX, and other various mocap formats.

Our Three.js demo (included in BVH example code):

skeleton

Our Processing demo (included in BVH example code):

processing-skeleton

As an alternative to the above, you are permitted to use Blender, Maya or Unity3D for this project— if and only as long as you write code in the internal scripting environment. Kindly note, however, that the professor and TA may not be able to support projects developed in these alternative environments. If you use them, you should be prepared to work independently. Note that if you are working in Blender, Maya or Unity, you may prefer to use the FBX format.

Summary of Deliverables

Here’s what’s expected for this assignment.

  • Sketch first! Draw some ideas. Study people moving, perhaps on YouTube.
  • Make or find a motion capture recording. Be sure to record a couple takes. Keep in mind that you may wish to re-record your performance later, once your software is finished.
  • Develop a program that creatively interprets, or responds to, the changing performance of a body as recorded in your motion-capture data. (If you feel like trying three.js, check out their demos and examples.)
  • Create a blog post on this site to hold the media below.
  • Title your blog post, nickname-Mocap, and give your blog post the WordPress Category, 06-Mocap.
  • Write a narrative of 150-200 words describing your development process, and evaluating your results. Discuss the relationship between your specific motion file, and your treatment. Which came first?
  • Embed a screengrabbed video of your software running (if it is designed to run in realtime). If your software runs “offline” (non-realtime), as in an animation, render out a video and embed that.
  • Embed an animated GIF of your software.
  • Embed a still image of your software.
  • Embed some photos or scans of your notebook sketches.
  • Embed your code (using the WP-Syntax WordPress plugin to format your JavaScript and/or other code).
  • Test your blog post to make sure that all of the above embedded media appear correctly. If you’re having a problem, ask for help.

Good luck!

Posted in