Quick Links to our Spring 2011 Kinect Projects

by Golan Levin @ 1:56 pm 26 May 2011

This page presents a quick link index to all of the interactive projects created in our semester course which made use of the new Microsoft Kinect depth-camera. These included a number of 7-week final projects, as well as sixteen small projects from a 2-week Kinect-specific unit.

Kinect-based final projects (May 2011):

Kinect-based small projects (February 2011):

 

Alex Wolfe | Final Project | Reaction Diffusion Textiles

by Alex Wolfe @ 4:17 pm 12 May 2011


Abstract

Knitting is often dismissed as a way to pass time for the excessively hip, or the excessively old, and it’s potential for digital fabrication ignored. But really, at its most basic level, knitting is an extremely robust method of transforming one continuous line into a three dimensional flexible form, incorporating color pattern, and also texture and volume seamlessly.

For this project I wanted to explore knitting as a way to fabricate textiles based off of generative systems, choosing to focus on Reaction Diffusion.

 

Inspiration

Although digitally controlled generative patterns aren’t really in existence, hand knitters for ages have been applying mathematical principles in order to not only produce interesting color patterns, but also form and texture. 

(from left to right, machine knit generative fair-isle by Becky Stern, hand knit from Sandra Backlund's pool collection, and Cables with Whisky sweater by Lucy Neatbody)

My inspiration for this project largely came from Sandra Backlund, one of my absolute favorite knitwear designers. By hand, and without any particular pattern she creates beautiful volumes and textures. However, due to the tedious nature of the work, she is only able to produce a handful of pieces each season, each entirely unique and unreplicable. I started wondering if this process of creating knitwear could be digitalized and algorithmically generated, like the work I’d done before with flocking. Becky stern also has been doing some amazing work with hacking knitting machines to print out patterns in fair-isle, and I was delighted to find out that similar texture producing stitches could also be created in the same methodology. However, as anyone who has knitted before knows, just randomly generating stitches, crossing your fingers, and hoping for the best often produces horrible and ill fitting results. Fortunately, there is a large community of knitters who have been applying mathematics, such as fibonacci sequences, cellular automata, and mobious strips, to the craft for ages. Lucy Neatbody was actually one of the first to apply such methodology to texture, using probability to pseduo randomly cable a sweater.

Process

Since even the most intricate patterns can be reduced to a very simple set of rules, knitting lends itself well to combining with more traditional generative system algorithms. After experimenting with several, including Voronoi diagrams and Diffusion-limited aggragation, I decided to focus on Reaction Diffusion, using Grey Scott’s equations.

Reaction Diffusion is a chemical reaction that produces a large range of emergent patterns in nature, from fish scales, to leopard spots and zebra stripes, whose hosts are often killed so their pelts can be used for textiles. By using it as a basis for a pattern making application, I liked the potential for creating and then fabrication patterns of our own.

I began experimenting in processing, using Toxiclib’s excellent simUtils library, “growing” reaction diffusion to fit various patterns and forms, and then focusing on creating multiple layers of it in order to produce variety in pattern


However, Processing became frustratingly slow and unwieldy, especially when it came to seeding the diffusion off an underlying layer of the reaction. So following a tutorial by Robert Hodgin and rdex-fluxus on how to recreate Reaction Diffusion with shaders in Cinder I developed my final application.

Producing “swatches” of pattern by adjusting the parameters of the equation and user interaction, I then translated them into patterns for knitting by analyzing the colors of pixels in the corresponding row, and translating them into a series of cables and eyelets.

Since plans for using a computer controlled knitting machine fell through (which ended up being a much larger part of the project than I had ever intended), I then hand produced the above pattern on my own machine.

Doing it by hand instead of on a computer controlled machine took a significantly longer time (for something that was rather small), but it allowed me to incorporate certain textural stitches such as cables that would have been impossible otherwise. I really like the final product, and actually wherever there were 3×3 cables in the pattern produced the best results, but those are really difficult to produce on the machine (you have to knit those rows manually since the stitches are too tightly pulled). The 1×1 cables of which there was an abundance of in that middle section are much more subtle.

Conclusion

Although I am extremely pleased with the final swatch, ideally I wanted this project to operate on a much larger scale. Hopefully, one could have the application and machine set up so that once the user finishes creating their own swatch and entering their measurements, garments could be printed out immediately incorporating their unique texture in a few hours rather than the 28-30 it took to produce the swatch. Working with more advanced shaders in Cinder, and learning more than I ever thought possible about knitting machines, how they operate, (and also how to fix terrible broken ones purchased from craigslist!) were also excellent things to know I learned from this project. I’m very excited to pursue this project further, and am working on making a prototype dress using the algorithm, and modding a machine to just print this during the summer

 



Final Project: Transit Visualization & The Trouble With Large Datasets

by Max Hawkins @ 12:02 am

81B bus in Pittsburgh

If you’ve ever come into contact with the Pittsburgh Port Authority bus system you’re likely familiar with the following situation: You come to the bus stop just before the time when it’s scheduled to arrive. Minutes, often tens of minutes later the bus arrives—full—and is forced to pass you by. Frustrated and disappointed you give up on the bus and walk or drive to your destination.

In my two years in Pittsburgh I’ve become quite familiar with this situation, and this semester I was lucky to come into come across an interesting dataset that I hoped would help me understand why buses so frequently let us down.

The Data

It turns out that every bus in Pittsburgh (and in most other large transit systems) is equipped with some fairly sophisticated data collection equipment. A GPS-based system called AVL or Automatic Vehicle Location records the location of every bus in the fleet once every 500 feet. Another device called an APC or Automatic Passenger Counter is installed on the doors of each bus to track the numer of people boarding and departing the buses. The data from these devices and others is recorded in the bus’s computer and downloaded to a central server at the Port Authority in the depot each night.

This data is then periodically transferred to Professor Eddy in the statistics department at Carnegie Mellon to support an ongoing research project studying the Pittsburgh bus system. I discussed my interest in the bus system with Professor Eddy and he graciously granted me access to the data—all 40 million records—to play around with.

The Work

I was surprised, humbled, and frankly a bit disappointed with how difficult it was to deal with 40 million rows of data. Since I’m a data mining newbie, I unfortunately ended up spending the majority of my time wrangling the data between formats and databases.

However, I did achieve some interesting (if modest) results with the data I was able to process:

First, I created a “hello world” of sorts by using R to plot one day’s worth of AVL data in a map

Though graphically this visualization is quite rough, it gives you an idea of the type of data contained in the bus dataset. Simply plotting the data gives a rough outline of the map of Pittsburgh in a way that echoes Ben Fry’s All Streets. The data alone is all that’s required for the map to emerge.

To better understand the data, I built a console that allowed me to interactively query the data and make quick visualizations. The system, built in Protovis and backed by MongoDB, allowed me to make queries based on the type, location, speed, and passenger count of the bus.

I created the above visualization using the console I created. It shows the path of every bus operating at 6:00 AM. The color of the line describes the route that the bus is servicing and the width of the line describes the passenger load. The map is zoomable and live-configurable using javascript.

The most interesting insight that I was able to gain from this console is that single buses often service multiple routes in a given day. The 71D you ride might service two or three other routes in a given day.

The other (troubling) thing that the console alerted me to was the amount of noise present in the data. Like most real-world sensor data, the location data collected by buses is quite fuzzy. One of the things I struggled with the most was cleaning this data to understand how the records were related and how far along the route each bus was.

At this point I realized that the modest computing power available on my laptop would not be enough to process all of the data efficiently and I decided to bring out the big guns. In retrospect it may have been overkill, but I decided to throw the data into the School of Computer Science’s 80-computer Hadoop cluster

What’s still to be done

Why It’s Important

== Transpo Camp & Context ==

==

== The Trouble With Large Datasets ==

SamiaAhmed-Final-Final

by Samia @ 8:29 pm 11 May 2011

Daily Life is a generative book.

Daily Life is a generative book. With it I strove to codify and programmatically structure the rules I use to design so as to make a book that designs itself. Daily Life is a program that reads in data (schedules I kept sophomore year of everything I did every day), allows me to choose a color palette and grid structure, and then generates a book of a desired length.

Inspiration
After wrestling with the previous assignment (generative), I found that I had a hard time grasping what it means to make generative art or design, and the process of going about making generative work. In that previous project, I dove in head first into the code, and ultimately that hindered me, because I did not have a sense of what my vision was that I was implementing in code. I realized in doing that project, that I needed to find a better approach to generative work, rooted in my understandings of the process-driven design approach. In this final project, I tried to encorporate more of that. I spent time thinking and sketching about how I wanted to create parts of my book, and that when I finally made it to coding, I was making fewer important design decisions, and instead figuring out the best ways to implement and create the vision that I had already defined.

References
The Written Images book.

PostSpectacular genre book covers

Amazing But True Stories about Cats

Process
My process for this project began with sketching.

As I began working, one of my hardest challenges was working with two halves of the program — creating a system for the rules and output, as well as creating the visualizations themselves. My first checkpoint was making a program that allowed me to create a number of pdfs that I determined (documented in my checkpoint blog post). After that, it became a game of implementation, firstly, building the grids and color palettes (the color palettes took a lot of time wading through toxilibs documentation, which is rather robust, but has fewer examples than I would have liked), and then creating a series of different visualization-methods called by the program.

Reflection

Overall, I feel like I had a solid learning experience with this project, scoping it out in a ways that gave me concrete objectives to reach. I found that when I got in to the details of it, that generative work is incredibly nuanced and difficult to manage. Though I had a fairly successful output, there were so many details of codifying the rules of design that I wanted to see implemented, that even just dealing with the smallest visual paradigm resulted in many many many edge cases that needed resolution. As a result, the final product I have (though I think I did make a lot of strides forward) is no where as near to sophisticated as I would have liked (especially in terms of typography). I was amazed at how much time and detail it took to realize and resolve very small aspects of the design of the pages. Additionally, I found that by spending so much time dealing with the “data-ness” of my data (past schedules), that I felt somewhat limited in how to represent it, as opposed to simply creating a narrative of visualizations that I specifically curated for the book. However, without the guiding constraints of working with a set of data, I’m not sure that I would have had as successful of an output, because I may have just spent my time worrying about what I was making, as opposed to spending time making that thing.

Timothy Sherman – Final Project – Never Turn Your Back On Mother Earth

by Timothy Sherman @ 4:22 pm

Whatever is built on the table is rendered onscreen as a landscape. Landscapes can be built out of anything. Four heroes explore the world you build. There are many wonderful things in it. But many dangerous things as well. Who knows what will befall them?

Made using openFrameworks with ofxKinect and ofxSpriteSheetLoader. The sprites were loosely based off of examples included with ofxSpriteSheetLoader.

Inspiration
This project was developed from an earlier project by Paul Miller and me, Magrathea. More information on the inspiration and process for that project can be found through that link, I won’t reiterate it here. Instead, I’ll talk about my own inspiration for the particular direction I took this older work in.

I felt Paul and I had been successful with Magrathea, and a lot of the positive feedback we received had to do with how much potential the work seemed to have. A lot of this feedback had to do with the potential of using the engine for a game, and this was something that particularly piqued my interest. I have a strong interest in game design, and the ability to use such an accessible and non-traditional mechanic/control scheme for a game was very exciting. This is what made me decide to work from Magrathea for my final project.

I took inspiration from a number of other sources:

I’ve always been enchanted by the Pikmin games, made by Shigeru Miyamoto at Nintendo. The feeling of controlling a mass of little characters across this giant world that dwarves them and you really appealed to me, and the bond the player forms with the little creatures was something I wanted to evoke.

I also took inspiration from the Lemmings games, in which you have to give tools to a group of steadily marching creatures in order to help them survive through the danger-filled, booby-trapped landscape that they aren’t intelligent enough to navigate on their own.

I wasn’t sure for a lot of this process whether to go down the road of a game, or make a more sandbox-styled project, but I knew I wanted interaction with many characters like in these titles.

Process

I’ll talk mostly about design issues here, as technical issues are covered in my other blog posts: Hard Part Done and Final Project Update.

The process for finding a game using the Magrathea system wasn’t an easy one for me. I went through a number of ideas, and talked to Paolo Pedercini for advice in developing a mechanical system that worked. The design didn’t really come together until I had to make my “Late Phase” presentation in front of the class, recieve their feedback, see what was working, what wasn’t, etc. It seemed from the feedback I recieved at that session, that people were connecting to the character enough when he was incredibly simple, stupid, and unchanging, so I decided that expanding that, by adding more characters with more moods, deeper behaviors, etc. would result in an engaging and fun end result.

I added the abilities for characters to wander, instead of just seek the current flower, which enabled me to have more flowers, more characters, a troll that can wander as well, etc. The characters were also given new animations for running from the troll in fear, drowning, and a ghost that rises from the point where they died. This helped to flesh them out and make them feel more alive, which led to users enjoying a stronger feeling of connection and interaction with the characters.

Conclusion

After the exhibition, I gained a lot more perspective on this project. If anything, my belief that the core mechanic of building terrain is intensely interesting has been strengthened – a lot of people were drawn in by this. The characters though, seemed to be the hook that stuck a lot of people in, and I definitely think that people developed connections with them. There was a range of approaches – some ignored them, some obsessed over specific ones, some tried to just get rid of the troll whenever he appeared, some tried to help the troll eat everyone, or drown everyone.

I noticed that for small children, the Play-Doh was the biggest appeal at first – the actual landscape generation was totally ignored. After a few minutes though, they began to pay attention and care about the landscape – the characteres, I’m not sure ever made much of an impact, though they certainly noticed them. More feedback might be necessary in order to really hammer the point home.

I’m happy with how this project turned out overall – a lot of people seemed to really enjoy it during the show. I think theres a lot more room to explore using the terrain generation system, as well as systems using the kinect above a table. The most requested / asked about feature was color recognition – This is something I know is capable, but I didn’t have time to implement, or design a reason to implement it. Using multiple colors of play-doh perhaps begs this question, however.

Icon and Summary

Whatever is built on the table is rendered onscreen as a landscape for characters to explore.

Maya Irvine – Final Project – Blogpost

by mirvine @ 3:43 pm

Generative personal icons that use data from Last.fm to describe an individuals music taste. These icons are meant to provide a quick method for determining musical compatibility, enhancing the social aspect of last.fm.

Inspiration
This project was largely inspired by developments in the music industry, including Arcadefire’s interactive video for “The Wilderness Downtown”.
http://www.thewildernessdowntown.com/

I am really interested in the idea that the days on top-down marketing techniques and design are over. People want their visual experiences to be every bit as custom tailored as their music is. It is from this observation that I set about creating a project that would use music to influence design. I had many lofty ambitions, including ideas for custom cd packaging, web app/interface, and logos, but in the end I decided to focus my attention on a single data source, last.fm, and single output, icons that can be used to identify music taste.

References

I was really inspired by the work that is being done in the field of generative print design. There seems to be a very severe split in the design world between those that do digital/interactive work and those that do print, and the two could do a lot more to inform and improve each other. The generative logos for Nordkyn and the MIT Media Lab achieve this particularly well. Nordkyn especially, for tying a graphic so seamlessly to a changing dataset.


http://www.underconsideration.com/brandnew/archives/where_the_cold_wind_blows.php


http://www.underconsideration.com/brandnew/archives/mit_media_lab_full_of_squares.php

I also really liked this project I found on creative applications that uses processing to create patters to be printed on a newspaper. I appreciate the ability to create beauty out of such simple forms and the re-inventing of an old medium.


http://www.creativeapplications.net/processing/these-are-patterns-processing/

Process

After our discussions in class I decided that in order to make the generative work I was envisioning, I really needed to work with a data set. I also liked everyone’s input that it would be interesting to work with the transactional aspect of this project, while keeping in mind the idea of “customizable album art for mass production.”

After searching through song databases and looking at alot of web-apps, I decided to use Last.FM.

Last.FM is a site that allows people track what music they are listening too. It also works as an “online personal radio” suggesting songs based on the users past listening. After checking out the API, it seemed like exactly what I needed, a dataset relating to a specific users musical taste.

After looking into using xml and getting a bit discouraged, I found a java binder made by janni kovacs, allowing you to use the Last.FM API with java/processing.
http://www.u-mass.de/lastfm

From here I developed a plan for generating simple designs based on user input and retrieval of the genera tags of their most listened to artists, plotted in a circle.

My first task was to tackle the problem of communicating with last.fm, and plotting the results.

I was very excited about making this an application on the web. That would allow it to be used by anyone, and solve the problem of gaining access to someone’s Last.FM account. I asked Max about this and he helped me mock up the login link below.

My first iteration of this project tried to manipulate an algorithm to act differently as it drew around the circle, “responding” to the present tags. I wanted to achieve the visual complexity that this so easy to generate when using code and so difficult when designing by hand. I couldn’t figure out a good system for this however, and decided to abandon that idea for one that was much clearer in it’s intent and read.

This was the iteration I had for the final show. It is a lot clearer in it’s intention and people seemed to respond well to inputing their information and returning back a graphic. However, a diagram of the general layout of the tags was needed for it to meaningful. The aesthetic was clearly not right as well. The colors were distracting and the shape was not appealing enough.

For my final iteration I first went back and did some serious re-ordering of the tags list. Getting those into a better, continuous order made thing instantly easier to work with. Next I changed my method of drawing to bezierVertex so I could make a continuous, filled shape. There was a lot of geometry involved with figuring out the curves of the shape in relation to the radius of the circle without overlaps, or loops that quite stumped me, but thankfully spencer came to my rescue on that. I also decided to get rid of all color and just focus on making interesting and meaningful shapes.

The Closer to the edge of the black circle, the more prevalent the genera, and the more frequent the white spikes, the more tags in that “area” of music were found.

Image Documentation

Video

Introspective

I ended up in a very different place with this project than I originally intended. I think that this was a product of overstepping my abilities as a computery person, but I learned a ridiculous amount and I’m happy with my final product. I’m really interested in the way that traditional design applications can be informed and expanded by computation, often making design more individualized and dynamic, and I got to touch on that process here, wich was a great experience.

I have improved the aesthetic of this project since the final show. I regret that I wasn’t more prepared for presentation on that day as I think my project would have been much better received. I am glad I had time to refine the form of the icon. That being said, there are still several improvements I would like to make. This project is still lacking an online interface which would be the first step to making it an actually usable. I would also like to connected it to a physical form or practical application, otherwise it remains in the world of data visualization, wich was not its initial intent.

These applications don’t have to be complex. The world of music has so many objects attached to it that you can see how a simple shape could be used across a range of items to create a language.

I owe a big thank you to Ward Penney, Paul Richardson, Max Hawkins, Golan Levin, and Spencer Williams for showing me things like iterators, hashmaps, content management systems, ruby, jar file unzipping, collections, proper indenting, and basic geometry.

Thanks for the fab semester!

Icon for your project

Decription
Generative personal icons that use data from Last.fm to describe an individuals music taste.

//Maya Irvine
//Carnegie Mellon University
//Interactive Art and Computational Design Spring 2011
//Thanks to Spencer, Max, Paul, Ward, and Golan for the helps

String myApiKey = “6f74e5f9e8ef323f8d108c71eba6c762”;
String user = “sunnydope”;

HashMap types = new HashMap();
String mostPopularTags[];
Float[] angles;
Float angle;

PFont font;
PFont font2;

void setup(){
size(800, 800);
smooth();
background(255);
mostPopularTags = loadStrings(“tags.txt”);
noLoop();

//set up arrays
angles = new Float[mostPopularTags.length];

// load most pop tags
float angle;
for (int i=0; i < mostPopularTags.length; i++) { // angle = (i*0.0174532925)*(2.92682927); angle = 2*PI/69 * i; mostPopularTags[i] = mostPopularTags[i].trim(); angles[i] = angle; types.put(mostPopularTags[i], angle); } println("mostPopularTags: "+mostPopularTags.length); } void draw() { Collection chart = User.getTopArtists(user, myApiKey);
for (Artist artist : chart) {

// for each tag in the artist tags
Collection artistTags = Artist.getTopTags(artist.getName(), myApiKey);

Iterator tagIterator = artistTags.iterator();
for (int i = 0; i< min(3, artistTags.size()); i++) { Tag artistTag = tagIterator.next(); float angle = getTagAngle(artistTag.getName()); if (angle != 0){ fill(0); font = loadFont("DIN-Medium-12.vlw"); textFont(font, 9); text(artistTag.getName(), width/2 + 300*cos(angle) - 15, height/2 + 300*sin(angle)); // text(artistTag.getName(), width/2, height/2+ angle*20); } } } font2 = loadFont("DIN-Regular-30.vlw"); textFont(font2, 30); textAlign(CENTER); text(user, width/2, 750); // map HashMap tags = new HashMap();
for (Artist artist : chart) {
Collection artistTags = Artist.getTopTags(artist.getName(), myApiKey);
Iterator tagIterator = artistTags.iterator();
for (int i=0; i=lastIndex) {
i-= lastIndex;
repeat = true;
}
String firstTag = sortedTags[i];

// IMPORTANT NUMBERS
int upperAnchorLimit = 250;
int lowerAnchorLimit = 200;
int controlAngleDifference = 10; // degrees
int controlRadius = 25;

// do a curve bw firstTag and secondTag
float r1 = (upperAnchorLimit-lowerAnchorLimit)*tags.get(firstTag)/maxval+lowerAnchorLimit;
float a1 = getTagAngle(firstTag);
float x1 = width/2 + r1 * cos(a1);
float y1 = height/2 + r1 * sin(a1);

float aC1 = a1+controlAngleDifference*2*PI/360;
float aC2 = a1-controlAngleDifference*2*PI/360;

float cx1 = width/2 + controlRadius * cos(aC1);
float cx2 = width/2 + controlRadius * cos(aC2);
float cy1 = height/2 + controlRadius * sin(aC1);
float cy2 = height/2 + controlRadius * sin(aC2);

if (i==0 && !repeat)
vertex(x1,y1);
else
bezierVertex(cx1,cy1,cx2,cy2,x1,y1);

if (repeat)
i+=lastIndex;
}

endShape();

}

Float getTagAngle(String tag)
{
for (int i = 0; i < mostPopularTags.length; i++) { if (mostPopularTags[i].equals(tag)) { return angles[i]; } } return 0.0f; }

Algo.Rhythm – Final Report

by huaishup @ 3:26 pm

Algo.Rhythm is a physical computing music machine with social behaviors. It consists of a set of drumboxes with can record outside input and replay it in different ways like LOOP, x2SPEED, /2SPEED and so on. By arranging different functional drumboxes in 3D space,users will experience the beauty ofcreating music with algorithm.

INITIAL Inspired by the stunning Japanese drum show KODO in Pittsburgh a couple of weeks ago, Cheng Xu & I had the idea of re-perform those variational and euphoric drum pattern by using a set of “drum robots”. In the show, there were less then 10 people and each handled one or two drums; everyone’s drum patterns are not very complicate but as a whole, these simple drum patterns mixed together and sounds subtle and amazing.


INSPIRATION
Taptap

Taptap is a fantastic project done by Andy Huntington
a couple of years ago. It demonstrate the concept of using modular knock boxes to create drum rhythms. Each knock boxes has a input sensor on top and a stick as an actuator. It will remember every input and repeat the beat once. By physically connect two knock box, it start create complex drum rhythms.   Solenoid Concert

This smart project using MAX/MSP to control different solenoids. The hardware is not complicate but the result is really impressive. CONCEPT We found Taptap is a good start. We share a similar idea to create drum pattern using modular robots. But the different thing is that we are more interested in using different “algorithm” to do this. Our initial plan is to let each drumbox can get drum pattern input from all sides, and has different drum output like LOOP function, x2SPEED function and so on. We also proposed that some of the drumbox should have 2 sticks so that it can pass the drum beat to 2 children simultaneously, like a decision tree. PROCESS To test the idea, I made a processing sketch to do the simulation. The result is pretty good. With only a loop function drum and several x2SPEED drum, we can create a relatively beautiful music rhythm.

For the hardware, we put a lot of thoughts in it. We thought about different housing but finally end up as a cube shaped drum for the reason that a cube is easy to pile up and the shape is organic.

We also designed and made our own arduino compatible board. However, even if we designed and ordered the board 3 weeks before the deadline, we could not get the stuff in time. So in the current prototype, we still using the really big arduino board.

 

 

 

We made 5 different prototypes to test the concept and make it work. Here are some working process photoes:

 

FINAL

 

After experiencing several failures, the final prototype looked like the above image. It has 5 sides that can accept tap input and one or two drum sticks as the actuator. The two boxes with one stick are different functioned. One has the LOOP function, the other has the x2SPEED function.

Though the current version is not very actuate, it can demo the concept of passing drum pattern between different drumboxes. I am pleased with the result after experiencing a really hard and frustrated time.

The next step is to revise the current design to make the drumbox robust and sensitive. And I will surly keep working on it.

Final Project Documentation

by Chong Han Chua @ 11:46 am

The idea for TwingRing originated from the idea of generating sound through Twitter streams. There might be some abstract interest in generating music from Twitter stream, but it is probably difficult to translate musical sense from sentential sense. It then occurred to me that text-to-speech is yet another form of computer mediated sound generation. It follows that having the computer to speak something is funny, having the computer speak tweets would be much funnier. In other words, it’s serious LOLz.

And thus the idea was born. The idea is then fleshed out; rather then simply converting text-to-speech, it would be much more interesting if it simulates a conversation between two parties. A duet of computer generated voices would create a “context” in which the conversation is happening. And the “context” of a conversation, to rewind through the progress of communication tools, would be something like a phone call. To add to the concept” of the project, the phone call would be decidedly traditional, sporting the familiar shrill Ring Ring to it, and the entire idea of not taking itself too seriously and hence kitschy was born.

On a serious note, TwingRing examines the notion of conversation between people, especially as communication mediums changed. The telephone was the first device that enabled remote communication between two people. Then the next game changer was the mobile or cell phone, which basically enables the same communication channel between two people. Then there was text, and more recently, the rise of social media tools, or social communication tools such as facebook and twitter. However, the notion of conversations do not change, but merely shifted as these tools enabled people to communicate differently. This project aims to examine how the idea of twitter conversations, when put into the context of a phone call, would sound like.

Actually, it’s really just for LULz.

There are a few technical pieces required for this to happen. Let’s look at the larger architecture.

There are a few technical pieces required for this to happen. Let’s look at the larger architecture.

The two main pieces that enables this piece is the Twitter API and the Google Translate API. There are a few parts to the twitter API. The current implementation uses the search API. The search API retrieves tweets based on a search term up to five days historically, which is a current big limitation of the system. The Twitter API is used to look for conversations from a particular user that mentions another user. This is done for both sides of the conversation. The retrieved tweets are then sorted chronologically so there is a guise of some sort of conversation going on. However, the way that tweets are used currently, @mentions are much less about conversations rather than simply “mentioning” another user.

When that is done, the message is then examined, common acronyms replaced with long form words, and then fed into the Google Translate API. The API has a “secret” API that returns a mp3 when fed with text. However, there is a 100 character limitation currently. The JavaScript examines the tweet and breaks them into 2 chunks if the tweet is longer than 140 characters. The name of the user is then used to check against a small list of database to determine the “gender” of the user. The mp3 is then fed to a flash based sound player, and pitch bent based on the “gender” of the user.

User interface is built in HTML/CSS with a heavy dose of JavaScript.

In retrospect, the project would have benefited from a stronger dose of humor. As the project iterated through from the rough design to the clean design, I personally feel that some of the kitschy elements are dropped in favor for a cleaner identity. However, there is definitely this whole notion of being able to play up the kitsch-ness of the site. There is still a need to rework some of the API elements to make it easier to work with. There is perhaps a problem with discovery of tweets by users, both because of the limitation in the search API as well as @mentions not being sufficiently dominant as a method of conversing on Twitter. There can be some more work done on that front, but what I have for now, is really, just kinda LOLz.

Meg Richards – Final Project

by Meg Richards @ 11:17 am

Catching Flies

Using a Trampoline and Microsoft Kinect

This project is a game that incorporates a trampoline into the player’s interaction with a Kinect. The player controls a frog by jumping and using his or her arm to extend the frog’s tongue and catch flies. The goal is to catch as many flies as possible before reaching the water.

Trampolines offer a new degree of freedom while interacting with the Kinect, and I wanted to create a game that demonstrated how one could be used to enhance the Kinect experience. Unlike regular jumping, a trampoline creates a regular and more predictable rhythm of vertical motion and can be sustained for a longer period of time.

Starting with that concept I originally began to create a side-scrolling game similar to Super Mario Brothers that would have the user jump to hit boxes and jump over pipes and enemies. I abandoned this attempt when it was clear that
it was uncomfortable for the player to adjust their actions to one that affected an orthogonal vertex. Additionally, complex object intersection became arbitrary complexity for a project attempting to achieve a proof of concept of rhythm
detection, skeleton tracking while bouncing, and hop detection.

Frog Arms

In the next iteration, the player was a frog whose arms would reflect their relative vertical position. A spring force at the end of each hands directs upward motion slightly outward and creates the illusion that the player is a frog jumping forward at each hop.



Hop Detection

The most interesting phase of the project involved player hop detection. One full cycle of ascending and descending could be broken down into logical phases of a bounce. Using the middle of the shoulders as the most stable part of the body from which to determine the acts of ascending and descending, every update cycle would pull that vertical position and place it into a height cache. The height cache was an array of height values treated as a sliding window: Instead of comparing the current y position to the most recent y position it was compared to some defined number of former y positions. As the current y position would have to be larger than all the former readings within the window, the sliding window reduced the likelihood of a bad reading or jitter causing a false determination that the player has started a new phase of ascending or descending. After a period of ascension, the player is considered in stasis when a future y position fails to be larger than all members of the height cache.

This is reflected in the following logic:

bool asc = true;
bool desc = true;
for( int i=0; i < HEIGHT_CACHE_SIZE; i++) {
    float old = properties.heightCache[i];
    //height pulled from tracked user is relatively inversed
    if(old > y) {
        desc = false;
    }
    else if(old < y) {
        asc = false;
    }
}
 
// if we are either ascending or descending (not in stasis)
// update the previous state
if( asc || desc) {
    // on first time leaving stasis; preserve former asc/desc states
    if(!properties.inStasis) {
        properties.wasAscending = properties.isAscending;
        properties.wasDescending = properties.isDescending;
    }
    properties.isAscending = asc;
    properties.isDescending = desc;
    properties.inStasis = false;
}
else {
    if (!properties.inStasis){
        properties.wasAscending = properties.isAscending;
        properties.wasDescending = properties.isDescending;
        properties.isAscending = false;
        properties.isDescending = false;
        properties.inStasis = true;
    }
}

Finally, with this logic in place, a hop is accounted for by the following:

if( wasAscending() and isDescending() and 
        (abs(y - properties.average_weighted_y) > REST_Y_DELTA)) {
    properties.num_hops++;
}

NB: The REST_Y_DELTA is used to account for the trampoline’s affect on a player’s rest state.

Gameplay – Flies

With the foundation in place, gameplay became the next phase of development. Continuing with the frog theme, I introduced flies as a desired token within the game. As the arms were presently unused, they became the tool used for fly interaction. Once the fly came close enough (they would gradually become larger as they approached the player), the player could wave their hand above shoulder level to swipe the frog’s tongue across the screen and gather any flies that hadn’t escaped off screen.



Gameplay – Stats

In order to show a player’s progress and bring encouragement, their number of hops and the number of flies their frog had collected were displayed on clouds at the top of the screen.

Gameplay – Endgame

Finally, the end game became the grass descending into the sea. Each hop would keep the grass from descending some marginal amount increasing the length of the game and thus the time available to catch flies.



Conclusion

While gameplay left a lot to be desired, I believe playing to my strengths and focusing on logical determinations of the jumper’s state and the resulting generic ‘jumper’ class and header files, which can be dropped in to future games involving rhythmic vertical motion, will ultimately prove worthwhile. After the public exhibition, it became clear that smaller children were having trouble getting proper hop detection and corresponding calculations. Once I fix some of the code that relied on incorrect constant values, it should be ready for general release.

Icon & One Sentence Description

Fly Catching is a game using a Kinect and trampoline where the player controls a frog that can hop and catch flies.

Passengers: Compositing Assistant + Generation

by chaotic*neutral @ 11:12 am

This project conceptually started before this class. I was working on compositing myself into Hollywood car scenes. I was using a crappy piece of software to live preview and align the actor within the background plate they would be composited into. I realized I could write a bunch better application to do the job. It allows the live feed + image to be paused together to look closer/discuss with camera assistant, toggle between images, crossfade between images, load directory of images. It is a very simple application, but a tool for something much greater. That was the first step of the process.

The compositing assistant allows me to take a live HD image over firewire from the film cam, and cross fade between that video feed and the background plate. Think onion-skinning. A by product of this process are interesting screenshots where self and original actor are morphed.

The second process was to make these films generative, so they could theoretically be infinitely looping. Currently these films only reside on youtube and dvd’s – the generative versions are created to an hour long dvd – so perceptually a viewer seems them as infinite. It could also run on a computer.

The Markov Chain code is implemented and works, but I have yet to add a GUI for tweaking of values.

INSPIRATIONS

Inspiration comes from a lot of experiences in cars. People moving in and out of vehicles. Instead of action based, or narrative driven, I thought I would rather have a scene of superficial nothingness, but when you start examining thin-slices (http://en.wikipedia.org/wiki/Thin-slicing), the viewer projects something there. (Kushelov effect)

(a facebook status update)
I park on the street, start rummaging through my bag. Girl idk is confidently walking towards my car on the phone. She opens my passenger side door and gets in. I say, “oh hello.” She looks freaaaaked out and says, “oh jesus, you’re not my dad.” I respond, “No Im not, but I guess I could pretend to be if you want.” She apologizes and states how confused she is, gets out and continues walking down the street.

Literature:
Remainder, Tom McCarthy
Platform, Michel Houellebecq

THE FUTURE

I still have yet to see these projected large format and in context with each other. I need to play with presentation and how they function together as a whole. I need to continue pushing the scenes and play with genre. Right now the films come from a very specific time range and very specific film stock, thus they look similar. What if it was a horror film car scene? Or a sci-fi car scene, etc? I need to create more. My goal is 10. I currently have 5, but one of which I wont ever show.

JamesMulholland-FinalProject-geneHackman

by James Mulholland @ 10:44 am

geneHackman (no relation)

Created using Processing, geneHackman is a radial visualization for genealogical data. Individuals, represented as nodes, are placed a distance from the center relative to the year they were born, with 2000 at the center moving backward in time as they move outward. The paternal lines are traced in darker green; the maternal lines are traced in lighter green. The software reads common GEDCOM files to generate the relationships.

About the Project

I previously developed a version of the family tree program (found here) but the interaction was incomplete and I didn’t make the information particularly clear. After diving back into it I found few errors in the parsing code that prevented many key people from being drawn. In the new version I accounted for longer data entries so that a full person is created.

Conceptually, my original had a problem as well. the up/down tree structure just couldn’t work at the level I wanted it to, even after I drew only the main lines, skipping the siblings (at least for now). To be able to understand the family network a little better, I developed the circle rendering to view the entire lineage cleanly and all at once.

However, since my rendering accounts for specific years born the information shown is more dense than the example. Once parsing the data, the code takes another pass at each person and calculates age from birth- and death-dates or estimates the age if one is not already calculated.

Video Demo

Interaction

Currently the controls are a little crude, but they are mostly to explore the diagram as a designer, and not as a user.

Controls:
z – zoom out
x – zoom in
0 – re-center on middle
left/right/up/down arrows – pan around

Inspiration

There are few resources to render family heritage in traditional tree structures. GEDCOM is a common program to manage genealogy data. There are a few others that offer to render the family tree (even in 3D), but no software currently shows the entire tree in a way that allows a full view of the family over the entire 1000 year span.

The divided circle angles was inspired by a handful of family tree images from poster companies.

Artistically, I found inspiration in star-finder maps that help to identify constellations.

Regarding color choice, there is little meaning in the color itself other than identifying gender: male (dark green) and female (lighter green).

Process

The primary challenge of drawing the relationships is to show each connection, without letting the connections overlap each other (at least, without some form of extra ). Overlapping paths cause confusion and prevent the user from seeing true relationships. To make sure there were no overlaps, I calculated the angle of spread for each person by dividing the angle of the child by two. The calculation relies on knowing the number of generations from the person in the center. The angle calculation looks like this:

Angle = 180 / ( 2 ^ (GEN-1) )

So the strength of this is that it is mathematically impossible for one child-parent relationship line to cross with another (but does not account for incest which would provide a crossover. The drawback is obvious: by the time you get more than 7 or 8 generations out, the density on screen becomes too intense to make sense.

Conclusions

Like many coding projects I spent hours going down rabbit holes that didn’t get me anywhere. I feel the efforts paid off though and the key features to navigate and understand the diagram are in place. A handful of bugs regarding color rendering are still in place and hope to fix them soon. The code does work on other GEDCOM files, but takes some tweaking. The code is customized for my own dataset, but down the line I hope to have plug-in-play capability with other data files.

Unfortunately, there are still problems in the data itself (I believe there are multiples of one person linked as a father to himself!) but the nice thing is that my navigation makes these errors apparent.

I would like to add more functionality soon including extra controls via controlP5 library to adjust parameters more easily. A filter-like search feature would be interesting but requires broader reflection on the overall functionality and interaction.

Future versions could allow more accurate zooming and more sophisticated line drawing using bezier curves (my first attempts were way off!). Since there are a couple historic figures in the tree, it might be nice to pull from wikipedia if data is available.

Marynel Vázquez – Future Robots

by Marynel Vázquez @ 5:08 am

Future Robots

  627 crowdsourced ideas about robots in the future.




INSPIRATION


As the field of robotics advances, more intelligent machines become part of our world. What will machines do in the future? What should they be doing? Even if we know what we would like machines to do, would other people agree?

To address these questions, I made a (crowd-sourced) book that compiles predictions and emotional responses about the role of robots in the future. The ideas collected about the use of robotics technology are mainly a product of popular culture, personal desires, beliefs and expectations.

We may not know what will happen in 10, 20 or 30 years from now, but we know what we want to happen, what we fear, what excites us. Is the robotics field expected to impact our future for better? Hopefully, the book product of this work will be a valuable sample of opinions.



BACKGROUND


Crowdsourcing has been used in the past for collecting creative human manifestations. One of the most famous projects that used Amazon’s Mechanical Turk for collecting drawings is Sheep Market by Aaron Koblin. The author of the project paid 2 cents to 10,000 Turkers to draw a sheep.

More recently, Björn Hartmann made Amazing but True Cat Stories. For this project, cat stories and drawings from Turkers were compiled in the form of a physical book. More details about this project can be found in Instructables.com.

In term of data collection, another important reference is the work of Paul Ekman on emotion classification. Ekman’s list of basic human emotions includes sadness, disgust, anger, surprise, fear, and happiness.

When collecting information from Turkers, a traditional scale for ordinal data is the Likert Scale. Different Likert-type scale response anchors have been studied by Wade Vagias. In particular, the traditional “level of agreement” scale turned out to be very useful for the Future Robots book.



THE PROCESS


CROWDSOURCING DATA

The ideas, opinions and drawings presented in this book were collected through Mechanical Turk. The data collection process went as follows:

  1. Workers were asked to complete two sentences: “In X years, some robots will…”; and “In X years, robots should…”. The possible values for X were 10, 20 and 30.
  2. For a given sentence from step 1, five workers had to agree or disagree with the statement that it was good for our future. Each of them had to select a feeling that would represent their major emotion if this sentence became true. The possible feelings they could choose from were Ekman’s basic emotions.

    Workers also had to agree or disagree with the statement that they foreseed a bad future if the sentence became true.
  3. A drawing of a robot was collected for each of the sentences from step 1.

The first step of the data collection process involved 627 sentences proposed by the workers (refer to this post for more information). Five different people had to give their opinions about each of these sentences in step 2, while only one person had to provide a drawing (see this other post for more details on steps 2 and 3 – there you can find the Processing.js code used to collect drawings -).

Note that different workers responded throughout the data collection process, though some completed many of the steps previously described.

MAKING THE BOOK

Mechanical Turk outputs comma-separated-value (.csv) files with the workers’ responses. Python was used to process all this data, and generate summarized files with results and simple statistics. Some of the data was later used by Processing to generate emotion charts such as the following:


In 10 years, robots may make us feel...

A LaTeX class was created for the book after organizing all the images and graphics that were generated. This class was based on the classic book template, though it alters noticeably the title page, the headers, and the beginning of each section. Once the layout was set, a Python script was run to automatically create all the sections of the book as LaTeX input files.

The pdf version of the book can be downloaded from here. A zip file containing all LaTeX source can be downloaded from here.



VIDEO






CONCLUSIONS


My first impression was: the future is going to be bad! When I started reading the completed sentences proposed by the Turkers, I encountered many ideas that resembled famous Hollywood movies were the Earth and our race is in danger. Certainly, popular culture had an impact in the Future Robots book. Nonetheless, after generating the graphics for the book, my impressions drastically changed. A lot of happiness can be seen throughout the pages. Even though the data shows that there is concern about a bad future because of robots, the expectations of a good future are significant. From the ideas collected, for example, the expectation of a good future in 10 years has the following distribution:


Goodness

If I had had more time for the project, I would have computed more statistics. For example, having a chart that compares the future in 10, 20 and 30 years from now would be a nice addition to the book. Also, an index of terms at the end of the book would be an easy and interesting way of exploring the data.

Another format for the project could have been a website. This option is very promising because, in general, it is easier to access for people than a physical book. A Processing visualization could be made to incorporate additional ways to analyze the data.

Body Type – Mauricio Giraldo

by Mauricio @ 4:07 am

A typeface is an environment for someone’s expression: using typefaces we write documents, create posters and subtitle movies. We follow the type creator’s restrictions and design decisions (kerning, spacing, proportion) when using her fonts. Good typefaces are created by highly-skilled people and can take several years to create. Body Type aims to allow anyone to create a highly expressive and personal typeface using only their body and hand gestures.

Body Type is an excercise on freestyle computing and was built using OpenFrameworks, the Microsoft Kinect sensor, OpenNI, FontForge, Potrace, ImageMagick, PHP, TouchOSC and the Apple iPhone.

Background

This project builds on a previous exercise on generative art that made use of the Kinect OpenNI skeleton system to create letterforms. I decided to take the idea further and create a self-contained application that would allow anyone to create a TrueType font using their body and gestures. The software should allow people to create type using their body in a way evocative of light painting photography:

love more.

Technical information

Body Type was created using the Microsoft Kinect sensor which was controlled by a C++ application created using openFrameworks and its OpenNI extension. This allowed for the 3D input received from the sensor to be converted into variable-width outlines and skeletons. The next issue was creating a TrueType font (.TTF file) from these silhouettes. The font-generation process goes through several stages in order to achieve this:

  1. Using openFrameworks to generate a black and white bitmap representation of each letter.
  2. ImageMagick is used to convert the image to a format compatible with FontForge.
  3. FontForge makes use of Potrace to vectorize the bitmap and then generates the final letterform and font.
  4. Since Body Type was displayed in the STUDIO for Creative Inquiry as part of an exhibition, web functionality was added so that font-creators could send themselves the resulting files. PHP was used to create a compressed ZIP file of all image and TTF files and send it to the specified email address.

The created font has six different possible parameters to determine a letterform’s visual attributes: skeleton lines, skeleton dots, line width, upper body, lower body and hand trails. Since controlling these parameters would be quite complicated to do by using gestures or on-screen menus, a TouchOSC iOS overlay was created to allow remote control via Open Sound Control:

Body Type iPhone console

Results

During the STUDIO exhibition dozens of fonts were created. Below are samples of some of them.

shawn sims outline
By Shawn Sims

heather knight flower alphabet
By Heather Knight (dingbats)

chinese whispers
By Cong Ma (chinese characters)

faste bold
By Haakon Faste

meta 2
By Terry Irwin

Creating the font is just part of the process. A font is made to be used. This project acknowledges the limitations of the fonts created both technically (since they lack proper letter spacing) and alphabetically (since they contain only representations for letters A to Z and numbers 0 to 9). However, these fonts allow for “freestyle” graphic experimentation:


By Paulo Pinto


By Juan Carlos Cammaert

Body Type
By Mauricio Giraldo Arteaga

Further work can be explored creating complete characters sets.

Icon:

Short description:
Body Type aims to allow anyone to create a highly expressive and personal typeface using only their body and hand gestures.

Le Wei – süss

by Le Wei @ 1:19 am

süss

süss is a sound generator where the user creates lines that play sounds when strummed. The user controls the length and color of each line, which in turn affects the pitch and voice of the sound it makes. By putting lines near each other, chords and simple melodies can be created. Currently, the project is controlled using a trackpad and supports a few different multi-touch interactions, so it can easily be adapted to touchscreen devices such as the iPad or iPhone in the future.

INSPIRATION
Since I have never worked with audio in any projects before, I decided this would be a good time to learn. I was inspired by a few projects which incorporated music and sound with a visual representation of what was being heard. In some projects, the sound was support for the visuals (or vice versa), but in other projects the two aspects were combined quite seamlessly. I was especially interested in applications that allowed users to create their own music by making something graphical that then gets translated into audio in a way that looks good, sounds good, and makes sense.

mta.me is actually very similar to what I ended up doing for this project, although I only realized this just now when looking back on my research. This is a visualization of the NYC subway schedule, and every time a line is crossed by another it gets plucked and makes a sound.

A couple of other, more painting based projects that I looked at:

PROCESS

I knew from the beginning that I wanted to make a visual tool for creating music. However, I didn’t really know what a good way to represent the sounds would be, and I didn’t know how to handle playback of the audio. So I did some sketching to get some initial ideas flowing.

I also set up some code using TrackPadTUIO [http://www.010175.net/?browse=Monolithe], which gave me a way to access what the trackpad was recording. This got me started on thinking about ways to finger paint with sound.

I also chose to use maximilian [http://www.maximilian.strangeloop.co.uk/] to synthesize sounds. I had a lot to learn in this area, so in the beginning stages I only had really simple sine waves or really ugly computery sounds. Nevertheless, I gave the user the option of four sounds represented by paint buckets, that they could dip their fingers into and paint with around the trackpad while the sounds played. I decided to have the y-position of the fingers control the pitch of the sounds, but I wasn’t really sure what to do with the x-position, so I tried out tying it to some filter variables and some other random stuff. Eventually, I got tired of playing around with such ugly noises and decided to see if I could make real notes, so I implemented a little touchpad keyboard. It sounded okay, but it really wasn’t what I was going for in my project, so I eventually got rid of it. However, the keyboard was a good exercise in working with real notes, which made its way into the final iteration of my project. Even so, I was pretty unhappy with my progress up until a couple of days before the final show.

Two nights before the final show, I decided to scrap most of what I had and switch to a new concept, which ended up being süss.

My main motivation for switching concepts was to simplify the project. Before this, I had too many sounds and visuals going on with all the paint trails and long, sustained notes. By having the user create areas that get activated through their own actions, sounds are only played every once in a while, and we get closer to features such as rhythms and melodies. Another success was when I figured out how to use envelopes with the maximilian library, which led to slightly more pleasant sounds, although they are still not great. With help from my peers, I was also able to give the project a facelift so that it looks much better than what I had in my intermediate stages.


REFLECTION
From the beginning of my project up until the very last days, I only had a general idea of what I wanted to do. A big problem throughout the process was my indecision and lack of inspiration in figuring out exactly what would be a good visualization and interaction for my project. At my in class presentation, I was still not happy with what I had, and I had to do a lot of work to turn it around in time for the final.

I’m pretty satisfied with what I have now, but there are definitely areas that could be refined. I think the project could be really helped by adding animation to the strings when they get plucked. As always, the different voices could be improved to sound more beautiful, which would contribute to making nicer sounding compositions. And the way I implemented it, it would really be more appropriate for something with a touchscreen, so that your fingers are actually on top of the lines you are creating.

I learned a lot from this experience, especially in the realm of audio synthesis. Before this project, I had no idea what waveforms, envelopes, filters, and oscillators were, and now I at least vaguely understand the terms (it’s a start!). I also know how hard it is to make sounds that are decent, and how easy it is to write a bug that almost deafens me when I try to run my code.

100×100 and quick summary

süss is an interactive sound generator using touch interactions to create strings that can be “strummed”.

shawn sims-Robotic Mimicry-Final Project

by Shawn Sims @ 9:21 pm 10 May 2011

The future of fabrication lies in the ability to fuse precision machines with an intuitive, creative process. Our built environment will become radically different when standardization is replaced with robotic customization. This project is designed to generate a collaborative space for humans and robots by offering a unique design process. ABB4400 will mimic. ABB4400 will interpret + augment. You + ABB4400 will share and create together.

flickr set

Process
We set out to produce live RAPID code in order to control a fabrication robot, typically programmed for repeatability, in real-time. This meant we needed to interface first Robot Studio ABB simulation software. This insured that we would sort out bugs before using the real, potentially dangerous robot. We created a local network where the robot controller listened only to the IP address of the computer running the openFrameworks application.

Once live control was established we then limited the work area in order to avoid collision with the workspace. This meant that every iteration of RAPID code being sent we were checking to make sure that the quaternion point was within the 3d workspace. This was once of the more challenging parts of the project. We needed to coordinate the axis’ of our camera sensor, ofx quaternion library, and the robot’s computer.

The main focus of this research was to really invest in the interaction. In order to demonstrate this we wanted the robot to mimic you 2d drawing live and then remember that drawing and repeat and scale it in 3d. We have come to call this the “augmentation” during our collaborative design process with the robot. In order to reveal in video and photographs the movement we placed a low wattage LED on the end of the robot and began to light write.

Below is a video demoing the interaction and the process of light writing. here is also a link to the flickr set of some of the drawings produced.

Susan — SketchCam

by susanlin @ 7:09 pm

Top
SketchCam is exactly what it sounds like: a webcam mod which enables viewers to see themselves as a moving sketch through image augmentation. Viewers are able to see themselves as a black&white or sepia sketch.

Inspiration
I was drawn to 2d animation which had intentionally left process work, or sketchy, qualities behind in the final production. The beauty of the rough edges, grain, and tangential lines added to the final piece unlike any polish piece would be able to accomplish. In terms of an exhibition, I wanted to see if I could make something that was interactive, since the final pieces were meant to be showcased publicly.

Here are a few pieces which piqued these feelings:

Background Reference

Originally over-thinking the colorization issue http://dippix.tp.chiba-u.jp/papers/CIC13_Horiuchi.pdf

Sifting through a ton of Learning Processing before stumbling upon Brightness Thresholding  http://www.learningprocessing.com/examples/chapter-15/example-15-10/

And realized I could simply colorize as 2-toned sepias.

Edge Detection http://www.pages.drexel.edu/~weg22/edge.html Sobel/Laplace algorithms and background studies before reproducing Sobel successfully.

Reading about Optical Flow http://people.csail.mit.edu/bkph/papers/Optical_Flow_OPT.pdfhttp://robots.stanford.edu/cs223b05/notes/CS%20223-B%20T1%20stavens_opencv_optical_flow.pdf (openCV before switching to Processing)

and trying to find simpler examples to learn quickly from http://andybest.net/2008/07/particle-engine-optical-flow/http://processing.org/discourse/yabb2/YaBB.pl?num=1222478633

Finally, using physics via Box2D http://www.shiffman.net/teaching/nature/box2d-processing/ particularly, http://www.shiffman.net/itp/classes/nature/box2d_2010/blob/

Process

I half-heartedly committed to the idea of analyzing cute characters, perhaps via Mechanical Turk before being swept away by great 2d animations. The entire thought can be found in this previous post. Brainstorming time and precious time before crunch-time was lost changing my mind… Unfortunate, this previous idea might have scaled a lot better and leverages some existing strengths of mine (producing cute like mad).

Before this madness: I tried coding in C++ using OpenCV via Code::Blocks, only to struggle just getting things compiling. In the interest of time/feasibility, I switched to Processing.

Pieces which ended up working

Brightness Thresholding

and

Sobel Edge Detection

After some sighing and frustration of trying to get things to play along with Optical Flow, I took another detour: instead of producing one robust augmenter, thought having 2 or 3 smaller pieces of a bigger project would help showcase the amount of work that went into this.

And a detour on a detour:

Pretty reminiscent of Warhol actually…

Of course, that’s been done… (Macbook Webacam filter)

What’s been done

Introspective

It is evident that this pursuit was a very erratic one which could have benefited from a bit of focus or external constraints. Overall, it has been a great learning experience. If anything, the whole experience has told me that I am a slow coder, and benefit from creating things visually, instead of algorithmically. I prefer to slowly absorb new material and really dissect it for myself, unfortunately, that ends up being a timely process that requires more attention than if I were scrambling a visual together. Help is nice, but there is a larger of issue of the balance between not getting enough help and getting too much help. If anything, my preference is to get less help, at the cost of being slower, so long as I learn. The best thing academia has given me is the vaster timeline and flexibility to learn anything I would like to, largely at my own pace.

The biggest downfall for this final project was simply the lack of time between job onsite interviews and capstone project report. Sticking with one key “magic” would have been better instead of splitting the already limited attention across colorization, filters, camera processing, optical flow, and edge detection. Committing to an idea would have helped too. Not feeling the pressure of having to feel like I needed to create something not already done could also be a part of this struggle. Also, there was a ton of researching, dead-ends, and things which did not inhabit the final form (ironically, considering the ideal project outcome).

All-in-all, if I were to ever uptake another coding pursuit, it’s evident that good timing will be key to success. What I got out of IACD was a bunch of initial exposure to both my pursuits and my peers’ pursuits. If anything, I hope it refined my process for the better to produce code-based projects and show the amount of heart put in each step of the way… I was successful in the sense I had the opportunity which forced me to look at a ton of different new code-related things.

100×100 and one-liner

SketchCam is exactly what it sounds like: a webcam mod which enables viewers to see themselves as a moving sketch through image augmentation.

Asa & Caitlin :: We Be Monsters :: The Final Iteration

by Caitlin Boyle @ 6:05 pm

The joint project of Caitlin Boyle and Asa Foster, We Be Monsters is an interactive collaborative kinect puppet built in Processing with the help of OSCeleton and openNI. It allows a pair of users to control the BEHEMOTH, a non-humanoid quadruped puppet that forces users to think twice about the way they use their anatomy. With one user in front and another in the rear, participants must work together to Be Monsters! The puppet comes to life further with sound activation, with user’s sounds past a certain threshold causing the BEHEMOTH to let loose a stream of star vomit.

( also available on vimeo)

The two users must come to terms with the fact that they represent a singular entity; in order to get the BEHEMOTH moving in a convincing manner,  movements have to be choreographed on the fly- users must synchronize their steps, and keep their laughter to a minimum if they don’t want the puppet projectile vomiting stars all over everything.

WHY a 2 PERSON PUPPET?

vs.

Why not? Inspired by the muppets and chinese lion costumes, We Be Monsters was initially developed for Interactive Art/Computational Design’s Project #2, a partner-based kinect hack. One person puppets had been made for the kinect before, one of the most famous being Theo Watson & Emily Gobeille’s, but as far as we knew, a 2 person puppet had yet to be executed. The first version of We Be Monsters was completed and unleashed upon the internet back in March.

(also available on vimeo)

 

Version 1.0 solidified our derp-monster aesthetic and acted as a stepping stone for further iterations, but we ultimately decided to go back to the drawing board for the majority of the code. Ver. 1 was only taking x and y points, and thus was essentially rendering the kinect into a glorified webcam. The program took in our joint coordinates from the skeleton tracking and drew directly on top of them, forcing puppeteers to stand in a very specific fashion; as you can see in the video, we spend the majority of time on tip-toe.

see that? we’re about to fall over.

Using the puppet in this fashion was tiring and awkward- it was clear that the way we dealt with the puppet pieces needed to be revamped.

In order to make the user more comfortable, and to take proper advantage of the kinect, we had to change the way the program dealt with mapping the movement of the puppet to our respective joints and angles.

MAKING THE BEHEMOTH USER-FRIENDLY

Asa dove into figuring out the dot product of each joint family “an algebraic operation that takes two equal-length sequences of numbers and returns a single number obtained by multiplying corresponding entries and then summing those products.”

a QUICK BREAKDOWN of DOT PRODUCT and JOINT FAMILIES

Joint family : 9 values that make up the x, y, a nd z point of three relative joints; in this example, the shoulder, elbow, and wrist.

Dot product : the combined angle of these three points/9 values.

Thanks to this math-magic, a person can be much more relaxed when operating the puppet, and the results will be much more fluid.

In layman’s terms, our program now reads the angles between sets of skeleton joints in 3 dimensions, instead of 2. We finally started utilizing the z point, so the program now takes depth into consideration. This is preferable for a multitude of reasons; mostly for the sheer intuitive nature of the project, which was practically nonexistent in the first iteration: Mr. BEHEMOTH moved as nicely as he did because Asa & I understood the precise way to stand in order to make it not look like our puppet was having a seizure.

Now, users can stand in any direction to puppet the BEHEMOTH; it’s much more enjoyable to use, as you can plainly see in the video for We Be Monsters 2.0.

MUCH better. We can face the camera now; the new joint tracking encourages users to be as wacky as possible, without sacrificing the puppet’s movements- and you can see by my flying hair that we’re jumping all over the place now, rather than the repetitive, restricted movements of our first attempt.

Puppeteers can now jump around, flail, or face in any direction and the puppet will still respond; users are more likely to experiment with what they can get the BEHEMOTH to do when they aren’t restricted in where they have to stand.

Making the BEHEMOTH scratch his head; something that would have been impossible in the first iteration of the project. DO NOT ATTEMPT without a wall to lean on.

SOUND ACTIVATION and STARVOMIT

Caitlin created a sound trigger for the star-vomit we dreamed up for IACD’s Project 3. Now, users can ROAR (or sing, or talk loudly, or laugh), said sound will be picked up by the microphone, and the stream of stars will either be tiny or RIDICULOUS depending on the volume and length of the noise.

+ =

The ability to produce stars was added as a way to further activate the puppet; we were hoping to make the project more dynamic by increasing the ways players can interact with and manipulate the BEHEMOTH. The result was marginally successful, giving the users the option to dive headfirst into MONSTER mode; the stars force the puppeteers to make loud and elongated noises. This, in turn, pushes the users to embrace the ‘monster’ inside and step away from humanity, at least for a short burst of time.

Asa roars, and the BEHEMOTH gets sick all over the place. Note his raised arm, which controls the top part of the puppet’s head, allowing for the stars to flow in a larger stream.

The goal of the project was to push people into putting on a (friendly) monstrous persona, and, cooperating with a friend, learn how to pilot a being more massive than themselves; in this case, the BEHEMOTH.

IN THE FUTURE, the BEHEMOTH will hopefully be joined by 3 and 4 person puppets; the more people you add into the mix, the more challenging (and fun!) it is to make the puppet a coherent beast.

We are also hoping to further activate the puppet with Box2D and physics (floppy ears, bouncy spines, blinking eyes, swaying wings); this was a step that was planned for this iteration of the project, but was never reached thanks to our novice programmer status; we’re learning as we go, so our progress is always a little slower than we’d hope. We did not get enough of an understanding of Box2D in time to incorporate physics other than the stars, but will both be in Pittsburgh over the summer; it’s our hope that we can continue tinkering away on We Be Monsters, until it transforms into the behemoth of a project it deserves to be.

– Catilin Rose Boyle & Asa Foster III

 

 



 


Paul Miller – Final Project

by ppm @ 5:10 pm


Blurb: “The Babblefish Aquarium” is a virtual aquarium where users can call in and leave a message, and a fish is generated from the pitches of their voice. Try it at babble.zapto.org.

As I explain in the video:
“Please Leave a Fish at the Beep” or “The Babblefish Aquarium” is a website at babble.zapto.org where users can call a phone number and record a message and a virtual fish is created from the pitches of their voice. The fish appear in the aquarium along with the first few digits of the caller’s phone number.

Pitch detection on the recordings is done in Pure Data, with higher pitches making fatter sections of the fish’s body, and lower pitches making narrower sections. Learned: Pure Data’s “fiddle” pitch detection object works great at 44.1 khz with a nice mic, but not so well with an 8 khz cell phone recording. In my patch, I’m doing a band-pass to filter out high and low noise, and then further rejecting detected pitches which fall outside an expected human vocal range.

The backend is Twilio communicating with my web server written in Java (using the ridiculously convenient HttpServer object) which communicates with Pure Data via OSC. My server keeps a list a phone numbers and detected pitches, which the graphical web client (JavaScript & HTML5) periodically requests to generate and animate the fish.

Also learned: the strokeText JavaScript function is hella slow in Safari (not other browsers though). Use fillText for decent FPS.

The reception at the final show was encouraging. Here’s a map of some of the area codes I got:

It would have been sweet if I could automatically take the area codes and produce a .kml for viewing on Google Earth, but I didn’t find an efficient area-code-to-latitude-and-longitude solution, so the above map was annoyingly made by hand.

The recordings I got were very entertaining. Most were either funny vocal noises or speech regarding the fish or the project like “I’m making a fish!” or “Make me a BIG, FAT, FISH.” I got a few songs, some whistles, a number of confused callers (“Hello? Is anybody there?”) and one wrong number who apparently thought they had reached an actual answering machine and left a message regarding someone’s thesis project.

Things to have done, given more time: The graphics are crap. It could do with a pretty aquarium background image, some seaweed or something for the fish to swim between, some animated bubbles… The fish also could have more interesting behavior than swimming in straight lines and staring awkwardly at each other. Flocking would help.

The pitch detection is somewhat crap, though it’s hard to fix that given my reliance on Pure Data and the audio quality of the cell phones. This caused the correspondence between sound and fish shape to be rather weak. Combined with the delay between leaving the message and the fish appearing on the screen resulted in poor user feedback. Participants didn’t feel control over the shapes and therefore (generally) didn’t try to express themselves by making certain kinds of fish. However, the delay did add an element of suspense, which participants seemed to find very exciting.

A solution would be to ditch the cell phones (as I initially intended to do, before my classmates urged me not to) and just have a single reasonable-quality microphone mounted in front of the computer at the show. This would facilitate better pitch detection and remove the time delay, but also remove the mystery and technological prestidigitation of the phone-based interaction.

In all, I’m most excited about the recordings I got. Some of them are very funny, and I hope to find use for them (anonymously, of course) in future projects. Hopefully, the Internet discovers my project and leaves me more interesting sounds.

Emily Schwartzman – Lyrical Similarity

by ecschwar @ 4:30 am

Lyrical Similarity 

What can you tell about an artist through their song lyrics? Typically songs and musical artists are analyzed on metrics based on the music, not on the song lyrics. This project explores patterns and similarities of musical artists based on song lyrics. Numerous studies and visualizations were created with various text analysis tools and techniques to look at a dataset of 100 song lyrics for 95 artists, selected from Rolling Stone‘s “Top 100 Artists of All Time.”

Inspiration

For this project I wanted to explore text-based visualizations. I decided to focus on song lyrics as my data source. Typically music is analyzed on the quality of the sound or the music itself, not from the perspective of the lyrics. I thought this would be an interesting opportunity to look at what the song lyrics might communicate or say about songs.

My initial inspiration came from other work done in the text visualization space. A couple of examples, shown below, include Stefanie Posavec’s Literary Organism project, and Ben Fry’s Origins of Species project.

Another interesting project is Stephan Thiel’s Understanding Shakespeare. What I appreciate about these projects is the breadth of views that they looked at to examine and understand the data.

Fernanda Viegas and Martin Wattenberg created a visualization of song lyrics looks at an interesting angle, parts of the body. This piece was part of their Fleshmap project and visualizes genres of songs based on how frequently different body parts are mentioned in the lyrics.

Data

I downloaded song lyrics using the musixmatch API. Initially I intended to look at song lyrics over time, analyzing collections of individual songs from different artists by decade. However, I ran into an issue when I was acquiring the data from musixmatch. I was only able to get 30% of song lyrics, and this did not feel like a good representation of individual songs. I put the lyrics that I had for each decade into one txt file, and ran them through an online tag cloud generator, just to get an initial impression of the data. There were no surprises here, but the bigger issue was that I did not feel good about the data. (Shown below, tag cloud for Billboard top 100 songs from 1980 and 2010)

Instead, I decided to look at collections of songs by artists. Using Rolling Stone’s “Top 100 Artists of All Time” as a starting point, I downloaded up to 100 songs for each artist on the list, which returned 30% of 100 songs. These song lyrics were stored in one .txt file for each artist. All of the lyrics were acquired using the musixmatch API via Processing.

 

Analysis

Once I had all of the song lyrics, I had to figure out how to analyze the lyrics to prepare the data for a visualization. This process involved exploring many different avenues, looking at different text-analysis libraries and software as a starting point, and then deciding what angle to take on the data. This process was a back and forth between trying to figure out what angle to take on the data and how to best visualize that. Below are some sketches from throughout the process that look at possible ways to visualize the data.

AntConc, one of the tools that I installed and used to look at the data, provided word frequencies, word pairing frequencies, and concordance plots of where words appear in each txt file. Below is a concordance plot created with AntConc and a view of the interface showing frequent word clusters.

LIWC (Linguistic Inquiry and Word Count) was another tool that I came across, which “…can determine the degree any text uses positive or negative emotions, self-references, causal words, and 70 other language dimensions.” Unfortunately I did not have access to the full program, but on the LIWC website a sample set of data could be returned for an individual text file. The metrics included in this set were self references, social words, positive emotions, negative emotions, cognitive words and big words (words greater than 6 letters long). With help from Mauricio, who kindly gave me a PHP script to scrape this data from the LIWC site, I was able to upload each of the song lyric files and store the LIWC metrics for each artist in one txt file.

 

Visualization Studies

I began with a simple plot of the LIWC data to get a sense of what it looked like. To see where each artist ranked, I created a series of vertical plots, one for each metric, to map out the name of each artist. Some interesting groupings were visible, in particular the outliers, but the rest of the artists were not legible. For example, one immediate surprise was discovering that Simon and Garfunkel ranked the highest for negative emotions, which I was not expecting.

I took this visualization a step further by making it interactive, so that only the selected artist would be white and the other artist names would fall back at a lighter opacity. If you rollover one artist, you can see that artist highlighted in each plot to see where they fall for each of the metrics. As an attempt to add another layer of information, I accessed the last.fm API to get genre information for each artist. I add a list of the genres to the left of the visualization, so that when you rollover each genre, all artists that fall under that category will be highlighted. Although this allows for an interesting exploration, the data still feels disjointed in this visualization.

In another study, I created 2-dimensional plots for each of the metrics, to look for where the most significant relationships might be. For example, one plot looked at positive emotions vs negative emotions, or self references vs social words. I printed the artist names for this study instead of plotting points, which actually detracted from the legibility of the charts. However, some relationships were still visible. Shown below are sketches from this part of the visualization process, and a view of the 2-dimensional charts. Download PDF of charts

To better consolidate the data into a more meaningful visualization, Golan recommended looking into SVD (Singular Value Decomposition), which is a way of reducing multidimensional data into 2 dimensions. Looking at a 2-dimensional plot would map out similarity of artists based on the LIWC analysis of the song lyrics, so relationships between artists would be more visible. I did some research to look for existing tools to do this and came across an SVD library in LingPipe, which is a “…tool kit for processing text using computational linguistics.” I was able to get this running on my computer with Eclipse, but the data it returned did not quite translate into data that could be plotted in a 2-dimensional chart. I attempted to understand the math behind it, and looked into other libraries as well, but unfortunately was unable to figure this out on my own. Golan kindly shared code that would take my data and do SVD, returning 2 values that could be plotted in Processing. The initial results were difficult to read, since many of the artists fell in the same region.

To help with this, I used Golan’s Polynomial Shaping Functions to scale the data to optimize legibility. Result after applying shaping functions shown below.

 

Final Visualization

For the final visualization I ended up with a stylized version of the SVD results, which was formatted at poster size. Ideally I wanted to take this a step further by synthesizing the data more and adding a more meaningful layer of interpretation. I discussed the results with several people with a range of perspectives and opinions on music, and got different interpretations of the significance of the artist groupings. However, this in combination with my own interpretation did not leave me feeling confident enough about adding this to the final piece. Despite not adding this information, the piece still invites discussion and reflection on what the viewer thinks the groupings mean. Download PDF of Final Poster

 

Reflection

I was somewhat disappointed with where I ended up for the final poster. From a design standpoint, the legibility was unfortunately compromised in my initial attempt to stylize the results. However, this could easily be resolved with another iteration on this particular design. However, I would ideally like to take this visualization a step further and add in an extra layer of information that would provide some insightful interpretation of why certain artists have similar lyrics. I think that part of my struggle to do this was in my selection of artists to analyze. I chose the Rolling Stone’s “Top 100 Artists of All Time” as a starting point because I thought it was a credible source that would provide a good range of artists from different decades and genres. However, some of these artists fell outside of my own knowledge of music, which made it difficult to fully interpret the results on my own. I would be interested to create a similar visualization with the 100 most frequently listened to artists from my own musical library, to see if I can draw some more meaningful conclusions from the findings.

Despite these frustrations, I learned a lot of new processes and techniques for scraping, analyzing, and visualizing data, which I think will be valuable to know for future information visualization projects. I realize a significant part of the process of creating an information visualization involves finding, analyzing and prepping the data. From a technical standpoint, that was my biggest challenge to overcome on this project. However, another significant part of creating a successful visualization is having an interesting question or angle to look at the data from, which I also struggled to settle focus on early on in the process. After working through this project, I can appreciate how challenging it is to create a meaningful and interesting text visualization, given the complexity of language and finding patterns and meaning within a body or bodies of text.

 

John Horstman – Final Project

by jhorstma @ 4:09 am

My final project is an interactive video installation named Twilight Conductor. By using arm gestures in front of a webcam, the audience can “conduct” a sunset, changing the color of the atmosphere and flinging stars up into the evening sky.

 

Ideation

We had freedom to set our own course for the final project, so I chose to create a webcam video installation. Many of the interactive video installations we had seen in class really impressed me, such as Utterback’s Text Rain, Liquid Time and Untitled 5, and Levin’s Interstitial Fragment Processor. I liked their ability to draw people in with their immediate responsiveness and sense of playfulness. Though the Kinect’s depth cam offers potentially deeper interactive experience, I wanted to create a piece that anyone could download and run on their local machine; many people have access to a webcam, but very few folks have a Kinect available.

The idea for conducting a sunset came from the book/movie “The Phantom Tollbooth.” The character Chroma the Great is in charge of conducting the world’s sunset each day, controlling the unique sounds and colors and placing all of the stars in the sky.

 

Implementation

I began by coding the color changes of the sky based on motion in front of the camera, because that was the most complex part. I chose to capture the motion with optical flow for several reasons. First, because it was capable of capturing both the amplitude and direction of the motion in the camera feed; this was important for both changing the sky color in the appropriate places according to the audience participant’s motion and for throwing the stars in the corresponding direction and velocity. Second, there is already an optical flow function included in the OpenNI openFrameworks add-on. The add-on did not include a wrapper for this function, and since I’m a pretty inexperienced openFrameworks programmer Prof. Levin had to give me some guidance there before I could use the function in my code.

My code identifies the motion in the camera feed, divides the screen into 20 bins from left to right, and totals the motion in each bin. The sky is drawn as a triangle fan, with 20 triangles drawn radially from an origin at the bottom center of the screen. The total motion in each horizontal bin is used to calculate the color change at the triangle fan vertices: greater total motion creates a darker blue, eventually darkening all the way to black.

The stars in the sky are Box2D objects, created using the Box2d openFrameworks add-on. If the amplitude of the motion in the scene totals above a certain threshold, then the code fires stars into the sky in the direction of the motion. Each star has a mass and friction, so that they fire from the origin with a certain velocity, then slow down and stop so that they hang in the sky. The direction of the stars is determined by summing all of the optical flow vectors into a total motion vector. Each star radius is calculated from a randomization function. The stars twinkle by randomizing the value of the alpha channel of each star. The stars slowly move across by the sky by applying a rotation matrix to the star positions and incrementing the angle of rotation.

The ambient sounds in the scene are two WAV files created by the user offthesky at The Free Sound Project. One sound is attached to each audio channel, and the volume of each sound file is set in proportion to the amount of motion in each half of the screen (left and right). A WAV file of a chime noise is played upon the creation of each new star.

 

Challenges

As mentioned earlier, Prof. Levin helped me to get the optical flow function working, which was critical to the success of the project.

I would have liked to change the color of the sky in a manner more similar to a real sunset, fading from light blue to red to dark blue, but I had difficulty controlling the RGB channels in a way that mimicked this behavior.

One of the big hurdles that surfaced at the time of installation was tuning the code to behave well in the installation environment. There were several coefficients in the equations that controlled the the motion of the stars and the changing the color of the sky that had to be adjusted in order to create a pleasant, predictable interaction.

In its current state, the code crashes after a certain number of stars are fired into the sky, which was happening about every 3–5 minutes at the final project exhibition. I think this could be resolved by adding some logic to call the destructor on stars that fall out of bounds of the screen – for which there is currently a function in the Box2D add-on, though that function is not working at the time of this writing.

 

Future Revisions

To improve upon the project, I’d like to revisit the color calculation algorithm to make the sunset behave in a more realistic way.

There are a few features that I had wanted to add earlier, but ran out of time before the exhibition. For example, I was thinking about adding generative sound synthesis to the piece rather than just playing back WAV files; adding a more responsive audio dimension could create a richer interactive experience. I’d also like to add some physics to make the stars generate constellations; if I could get my hands on a data file with coordinates for the stars that appear in major constellations, I don’t think this would be too difficult.

 

Source code

Download the package

Next Page »
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
(c) 2017 Interactive Art & Computational Design / Spring 2011 | powered by WordPress with Barecity