Project 3: faceframe

Author: Dev Summary: A video puzzle that is responsive to the location of the user's face Abstract: Traditional puzzles are static, and this project began as a means to make them more dynamic/interactive. I ended up creating a video puzzle, which users by default would solve by identifying factial features. I decided to emphasize this by using a head tracking library to only display puzzle pieces that contained the face. Repository:


You can play around with this project here:

Note that this project has been tested with the most recent install of Chrome. It is likely that other browsers will not support some of the features that I am using.


This idea was a stepchild of another which involved using video with a Chrome Extension. Unfortunately, at this time Chrome Extensions don’t support video input. I found this out a bit late, but decided to use what I learned with HTML5 video processing to make something else.

I was inspired by an N64 video game from my childhood, Banjo-Kazooie which had a video puzzle as a mini-game.

I started off by trying to split a video feed into a set of images.

video to  images

I then controlled the placement of these pieces by setting their initial state randomly. And I also added mouse hover interactions.

hover to select piece

Hovering was followed by click and drag, which (then temporarily) would allow me to interact with my puzzle.

working click and drag

I then looked into hand gesturing in javascript, something I wish I had done earlier. I wanted the user to be able to interact with the puzzle by using their hands to grab tiles. I came across this library: which seemed extremely promising. However, when I got around to using it, it simply didn’t work as advertised – the detection of my hand was too variable for my purposes.

I decided to ditch the idea, and try going for brightness based motion. I wrote code for it that worked, but after testing it on some friends, found that it was somewhat unnatural.

selecting a piece based on brightness

The tests told me one thing – people playing with tiles would consistently move their head to refresh images on the tiles. Completing a face is easier than completing some random background image. I decided to hook this concept with the fact that the face tracking javascript libraries worked reasonably well. The one I had tried headtrackr.js was very well done.

I started by using headtrackr to select pieces, and found this had an interesting effect. I evolved this into a puzzle which would darken tiles which didn’t have a face, and properly display those that did. I wanted to force users to really use their faces.

headtracking to select pieces

Playing with opacity:

opacity filter + facedetetion

Pairing opacity and face detection:

facedetection + opacity pairing

Current State and Conclusions


At the moment this project lets users interact rectangular portions of their webcam’s video feed. These tiles can be rearranged however the user chooses. Tiles that don’t contain the user’s face are purposely obscured to encourage users to use face location as an interaction.

I think there is some fun appeal in a project like this. And am happy that I have further exercised my web development skills in this course. I really regret two things about my implementation:

  1. Interaction technique was not thought out early enough, especially given that it was meant to be the crux of this project.
  2. Experimentation and reimplementation left less time for polish.

Hopefully I can learn from this in my future works.

Future Work / Expansion

  • I really think video puzzles could be fun as a multiplayer game. Maybe with the task of completing someone else’s field of view would be interesting.
  • I also looked into things like microphone input. Maybe the visible squares around the users face could increase based on how loud the user was?
  • I can buffer the video images from the past and cause some tiles to display video slowly, or video from the past. Not entirely sure how to make this interaction sensible though.
This entry was posted in project-3 on by .

About Dev

I am a senior ECE major who enjoys the outdoors and gaming. I spent my first half at CMU taking low-level technical courses and am now completing my second half taking some creative user-interface/interaction courses.

Leave a Reply