Robots and the future?

I would like to collect opinions about robots in the future (what they will be doing and what they should do). I decided to try Amazon Mechanical Turk, and started paying little cents for people to give ideas about robots.

I had to use different strategies to get responses from people. I started with a very descriptive task with more than 20 lines of descriptions and examples about how to complete it. The task consisted in completing 6 sentences: “In X years, some robots will …”; and “In X years, robots should …” with X = {10,20,30}. Not many people filled the survey, and so I had the impression that too much description at the beginning of a task might make people think it’s too hard and not worth the effort. I simplified my description over time, and changed how much I was paying many times. At some point seemed like paying too much was also counterproductive, as people started thinking that they had to do more than simply filling the short sentences. After several trials, I concluded that paying $0.03 for the 6 sentences was the more effective strategy.

There is always the possibility that people will cheat in MTurk. To try to reduce the amount of rejected tasks I was getting with my 3 cents, I decided to change the task to rather write two 30-words paragraphs about about what robots will and should do. I started paying 6 cents for the paragraphs without much luck; not many people was completing the task. I raised the incentive to $0.11, but just a few people reacted to the new price.

My last test consisted in asking about “robotics” instead of “robots”, as suggested by a few people in the class. During the past data collection, I realized that too many people was thinking about robots killing us, controlling us, etc.. and wanted to see if I could get some more diverse set of answers. I got just a few responses this time as well.

The unprocessed list of valid ideas I got about robots is presented at the end of this post.  Probably I ended up getting ~600 different ideas, though I still need to count the repetitions. I would like to be able to visualize the data in different formats (year, will/should, topic, agreement), so I started clustering the ideas by topic or by a single word description. Also, as soon as I organize a little better the data, I want to ask people to judge the ideas. Meaning, say how much you agree that robots will be doing this or that robots should be doing this other thing.

I would like to collect drawings about robots capable of doing what the people described. For this task I plan to ask people to draw by hand, take a picture, and upload it to MTurk. I expect price calibration to be an issue again :S

Finally, here’s the list of ideas (each sentence either starts with “In X years, some robots will…” or “In X years, robots should…”, but this is omitted to simplify the reading of the post),

Currently the RAPID output is working smoothly. The only issues currently are some coordinate system misalignment. The x,y,z orientations of the robot (RAPID), kinect (ofx), and the TCP (tool center point) are all different. We have solved most of these issues but just need to scale the input. Currently the interaction method is you move your hand and a vector is created from your chest to hand. That vector determines where the robot looks or where it operates.

Some hurdles we have covered….
+ Making openNI actually 3d instead of a flat image
+ RAPID output is clean with no errors (except joint limits when it moves to a position out of range)
+ Simulation is running live in Robot Studio (virtual controller) via OF

Things to do…
+ Map/Scale Quaternion inputs from openNI to limit input. This will minimize joint limit errors
+ Create a tool to operate on a soft material
+ Create a safe interaction

I’m revisiting the Kinect for my final project. I’m separating the background of an image from the foreground and using OpenGL GLSL Multitexturing Shaders to apply effects to the foreground.

GLSL Shaders work in OpenFrameworks, which is cool. However, there’s a trick that took me about three days to find. By default, ofTextures use an OpenGL plugin that allows for non-power of two textures. Even if you use a power of two texture the plugin is enabled and allocates textures that can’t be referenced from GLSL. FML.

The first GLSL shader I wrote distorted the foreground texture layer on the Y axis using a sine wave to adjust the image fragments that are mapped onto a textured quad.

I wrote another shader that blurs the foreground texture using textel averaging. You can see that the background is unaffected by the filter!

So there actually has been some great strides. I got in touch with Andrew Salomone through Becky Stern and he agreed to help me create a prototype on his computerized knitting machine sometime next week! Which definitely constitutes conquering probably the hardest “hard part”. He also sent me the manual to his machine so I can convert my patterns over into something recognizable.

Until then, I received this wonderfully bulky old LK100 Singer and maybe a pound of terrible acrylic off of craigslist with which to play around with. (Since I probably won’t have an excessive amount of time for prototyping on Andrew’s machine, might as well make sure the pattern looks right).  The lady who sold it to me showed me how to make a simple gauge swatch like so…..and also how to cable. Which is time consuming but braindead simple.

After looking at knitwear designers like Sandra Backlund, and the Cables after Whiskey sweater pattern  (that selects where cables will be placed with a probability function that produces pretty interesting results) both shown below, I want to create patterns with pseudo random cables/pleats in order to generate interesting volumes (ideally still very simple tube-based dresses).

As far as a “meaningful function”, perlin nosie/ flocking would  be sort of perfect, its so graceful, but I feel like that horse is quite dead by now. So I was looking at Reaction Diffusion algorithms, Grey Scott in particular. Given a standard boring ugly sweater pattern like this one, you can control the rate of diffusion to literally “grow” the diffusion to fit the pieces you are creating.

This creates very interesting results, especially if you knit so that the cables are nested within each other or maybe mapping each isolated circle to an eyelit. Since this pattern is actually impossible to translate into knitted cables due to their orientation and proximity, it could warp the fabric in good or bad ways. I’m also debating the merits of using an actual generative system, or just having at with simple probability.

This sketch begins with parallel lines, and then decides if a given line will intersect with any other with a random probability. I also realized this sort of technique could easily be applied to a sensor and collect some kind of data and visualize it that way. So I guess solving my previous “hard part” really has just given me alot more room to brood over this one. I think its important that its not just a textured knit produced electronically, but that the driving pattern underneath has some sort of relevance/significance other than pure chaos.

I’m working on reproducing the Blob at http://hakim.se/experiments/html5/blob/03/. So far, I’ve gotten box2d on javascript working, and am working on trying to implement a blob with the same kind of fluidity and dynamic.
I’ve thought about implementing a bunch of particles attached by springs, where each particle is connected to by an even number of springs, half on the left and half on the right.

Using MongoDB and MapReduce, analysis of my dataset has finally become tractable. It’s still early, but I feel like I have the analysis framework mostly taken care of which gives me time to focus on the visualization.

This simple (and ugly) visualization of downtown Pittsburgh and the surrounding area maps the average speed of buses to colors. Each pixel represents a roughly 500 foot square region. The color interpolates from red for the regions where the buses travel on average the slowest to green for areas where buses are moving quickly. For time’s sake, I only sampled around two percent of the data when creating this image. An image sampling the entire dataset would have fewer holes and less noise.

Next I’m going to work on creating more effective and aesthetic visualizations of this data and extracting new data through analysis. There are some low-hanging fruit here (bicubic interpolation of color values, map overlay) and also some more challenging directions I can take things like map distortion for isochronic maps.

Hello!
Thus far, I’ve been building the pieces of my project, and have a somewhat working, page-generator
Roadmap sketching:

Half one: generating pdfs! It’s happening! Currently generating single pages, this may be ideal, however, because I’m going to have to make an action in photoshop to automatically cut up spreads to be printed (hooray doublesided printing and perfect binding!).

Half two: building components of “visualizations” using my personal-schedule-data as a jumping off point to generate small viz that will be recombined on spreads to create the book. Right now, I only have two, and the mostly suck.

So right now, I can generate a rather simple, currently boring book, with a user specified page count

Now that the frame work functions, I need to really build all of the different visualization pieces, as well as rules for combining them, as well as start to deal with things such as variable pages size.

Kinect Portal

KinectPortal Sketch

Tasks

1. Develop acrylic panels with handles and Vellum on one side for users
2. Threshold capture the blob contours with openCV in depth grayscale
3. Discover corners with Dot Product testing
4. Inverse Perspective Transform to get real “square” and discount user tilt
5. Draw 3D image
6. Projection mapping back onto square in 3D space

Currently working on Thresholding capture (step 1). It’s just gonna take all weekend.

I have a good grasp of how the Kinect works, so I don’t think there’s going to be much technical challenge there. The other thing that I’m kinda toying with is integrating an iPad into the loop. I’m starting to look at programming for the iPad and evaluating if this can be done within the time frame. I’ll be a happy camper if I can integrate that and build a fun self discovery app.

Many of the hard parts of my project have been conquered. I’ve successfully implemented cross-domain YouTube video queries that will load search results and allow the user to select videos to load into the application. I can then have multiple HTML5 videos be displayed at a time. Finally, I can programatically play clips of the video and make them loop.

The next difficult part will be creating a global clock, implementing a GUI for clip selection, and also making an interactive timeline. Now that I know the video player will work, the rest will be a matter of testing and implementation.

For my hard part done, I’ve solved the problem of how to insert characters into the landscapes. I’ve gotten 2D sprites that can move around the landscape, getting drawn at the correct level despite however the landscape is moving under them, and even scaling to different heights when they’re further away or closer. In this project, the x/y plane is rotated, so the grid the landscape is drawn onto is the x/y plane, and the rising landscape rises on the z axis. I had to first map sprite coordinates onto this coordinate system, then figure out how to draw them on the x/z plane, perpendicular to the landscape. As I’m not used to OpenGL, this ended up being a lot harder than it sounds at first.

My next steps are related to how my project has evolved. At this point, I’m feeling fairly certain that I want to turn this project into a 2-player, cooperative game. One player will control an onscreen character with a controller, while another will build the terrain this character can walk on. The character-player will have to avoid falling into the water (lava?), roaming flying enemies (birds? jellyfish? clouds of death?), and other hostile monsters, and visit four different islands before returning to the starting location. The builder-player will build the paths the character-player can walk on, but will have a limited amount of resources to work with. By collecting falling objects, the character-player can unlock the ability for the builder-player to use more colors of clay to build with, making things easier for themselves as well.

More interaction will be available, for example, the builder-player can protect the character-player by building a mountain behind him which pursuing monsters can’t get past (but blocking the characters own backward progression as well). Designing the enemies and interactions of this is probably my next task, and coding a few proof-of-concepts to test various gameplay mechanics. Also, I’ll probably be coding a basic color recognition/tracking system to allow me to turn off/on various colors of clay or colored blocks (or give them different effects).

I’m VERY MUCH looking for any and all feedback on this, or just people to playtest stuff once I’ve got a build, so talk to me or comment on this or email me. (timothy at cmu dot edu).

Thanks!

I have user’s button presses on their cell phones being delivered via Twilio (thank you to the pirate pader who recommended it) to a Java application on my computer, using the HttpServer class in Java 6. You call my number, and then press buttons, and the numbers you press show up on my screen after about a half-second delay. I can also get your phone number (useful to differentiate users) and the home city/state/zip of your phone. And I can record your voice and play audio files to you, as well. So the basic technical challenge is solved. I still am not decided on my project, though. I’ve got several ideas for games that could be played on a large screen with many players. From the big-screen-lots-of-users interactions I’ve seen in the past, the challenge is to give each user enough control over the system to become invested in it. One should not feel one is fighting the other users in order to experience the game. I’m also thinking about what sort of things it would be cool to control anonymously with your phone. Like maybe a giant projection where each person gets control of a few pixels, and together people could build images. Or a radio station. Or a robot. The challenging with button-based interaction is that there’s no fine control there are only 12 binary switches, and you can only push one at a time, and there’s a half second delay before the system reacts to your choice. Perhaps a voice-based interaction would be better, although I don’t have real-time voice with Twilio. I can only record you and then download the recording asynchronously.

So there were speedbumps of many shapes and sizes. Large, small, technical and personal; production just kind of ground to a halt. This prompted us to basically bunker down and focus on our two largest challenges: a.) recording a file consisting of Kinect data to use during debugging, and b.) nailing down the angle tracking algorithm for our baseline skeleton-tracking-stick-figure-program. Caitlin took the reins on the data recorder, I worked on the maff.

The Kinect data recording bit is pretty straightforward and rather uninteresting, so I’ll just summarize: instead of having to get off our ass every time we press compile, we want it to just play a dummy so we can code a LOT more efficiently. We currently have a small snippet of data to access any time we need it.

The math bit is a hair more complicated, but can be summarized by saying that there are quite a few sets of quite a few points, which all need to run through an algorithm quite a few times. The goal is to create a baseline stick figure program, onto which we can build our puppets (or any other future programs for that matter), and we needed angle data for each of the joints on the skeleton, in three-dimensional space. The initial equation worked to calculate one of the angles at a time, but calculating all eight simultaneously became a data structure puzzle. My first instinct was to make arrays to hold these points, and multi-dimensional arrays to hold the sets of points, and then doing math with waaaay too many of [these] suckers. With Golan’s help explaining an object-oriented approach vs. the convoluted arrays that I had been using, we are now well on our way to finishing up the integration into the stick figure program.

In other news, this whole thing is somewhat changed by the fact that we did not get the Tough Art residency at the Pittsburgh Children’s Museum, meaning that we now do not have some floating due date in the future to worry about and thus just want to make this thing WORK. And well. More updates to follow.

Hard Part Solved update

What’s happening

I started this project without a clear idea as to how I wanted it to look or what sort of interaction I wanted to use. But so far, I’ve been working with the trackpad on my computer to create a little fingerpainting program, so I am going to narrow my focus to see how I can use this in a way that suits my project. I’ve also been able to integrate some sound output into the fingerpainting program, so that the way you move your fingers around the trackpad controls the sound coming out of the computer. I think this is a good foundation for implementing the rest of my project, I just need to make some decisions on how I want it to work in the end.

What’s solved

• Trackpad input, using TrackpadTUIO [http://www.010175.net/?browse=Monolithe]
• Sound output, using maximilian [http://www.maximilian.strangeloop.co.uk/]
• Combining them

What now

• Thinking about what I actually want (sketching, brainstorming)
• Learning how to make sounds that aren’t ugly (using math and stuff I guess)

Inspiration

Goal Recreate gestural animation effects using code.

Hurdle Final form: drawing tool? layer on top of video? images?

Napkin Sketch I call it “LoFi”

Question Recommended short-films to watch or things to read about animation/rotoscoping techniques? :)

Inspiration:

“Musical instruments come in a wide variety of shapes, sizes, and forms. Most have a long history, sometimes thousands of years, and their basic structure derives in part from the accidental discoveries of early musicians, in part from the properties of the physics of vibrating strings, columns of air, membranes, and reeds. Very little attention has been paid to the ergonomics of the instruments. As a result, they often require awkward body positions, such as the contortion of the left hand required to play the violin and related stringed instruments, and sometimes exert great strain: look at the bulging cheeks of brass players, or the calluses on fingers tips of string players.

I am convinced that if the instruments were introduced today and forced to undergo ergonomic review for health and safety, they would fail.

The piano, for example, is relatively straightforward to understand, but incredibly difficult to master. The learning time is measured in years. Note that there are two parts to learning an instrument. One is the physical mastering of the mechanics itself: how to hold the hands, posture, and breathing. Many instruments require demanding physical exertion or special blowing techniques. Some require different rhythms in each hand simultaneously, and some require use of both hands and feet simultaneously (harp, piano, organ, percussion).”

– Don Norman, Living With Complexity

Design Goal:

Create an experimental digital instrument – the design of the instrument will not be constrained by the acoustic properties of its physical form or by traditional design paradigms, but will be centered on providing an ergonomic and intuitive interface for musical expression.

Hardware/Software:

• Arduino + Processing + MIDI

Interaction:

• Possibly incorporate gesture recognition/ haptic feedback/ parallel screen-based feedback with physical interaction

• Synesthetic associations for notation and aiding performance; utilize multiple modalities

[?]

Are there any projects that you have seen that address a similar goal?