rolerman-Mocap

Preamble:

I know the assignment was to work with the body, and what I made focuses only on the face. I apologize for that and tried to avoid it – I think for this project I wound up sliding into things I felt comfortable with rather than giving myself the space to try something new. I tried for a while to get real-time body information with just the webcam on my laptop, but hit a bunch of roadblocks with that. Facial recognition on the laptop, though, is much more common and I was able to figure it out more quickly.

That said, this week I made:
 

the HOT mirror

I was really interested in working with real-time for this project, so I created the HOT mirror. The HOT mirror makes use of Men’s Fitness’s list of “20 hottest female celebrities in the business”. It takes your UN-HOT face, and chooses new beautiful eyes, nose, and a mouth for you from the sexiest eyes, noses, and mouths out there. Looking good!

I found a nice, front-facing image for each celebrity on the list, and chopped up the image into left eye, right eye, nose, and mouth. Every time the space bar is hit, I choose a new random selection of eyes, nose, and mouth, from the 20 celebrities.

Thank you to the following HOT ladies

  • Scarlett Johansson
  • Jessica Biel
  • Jennifer Lawrence
  • Megan Fox
  • Mila Kunis
  • Melanie Iglesias
  • Kate Upton
  • Michelle Rodriguez
  • Amber Heard
  • Julianne Hough
  • Vanessa Hudgens
  • Lauren Cohan
  • Minka Kelly
  • Jamie Chung
  • Rihanna
  • Elsa Pataky
  • Jessica Alba
  • Maria Menounos
  • Sophia Bush
  • Beyonce

Code

I used the library tracking.js, because I wanted to host this project online after I was done. However, I didn’t anticipate the limitations of doing this in the browser! For one, once the video feed got to be too large, the library wasn’t able to keep up with the larger amount of pixels, and would stop working. Also, the tracking is pretty jumpy and discontinuous. If I were to do this again, I would just use OpenFrameworks.

Honestly, the real challenge with this project was experimenting with and installing tons of software, Python OpenCV, OpenFrameworks, and several JavaScript libraries, and trying to figure out which ones I could get working and which ones had capabilities that I needed. Once I committed to tracking.js, things were much smoother.

<!doctype html>
<html>
<head>
  <meta charset="utf-8">
  <link rel="stylesheet" href="assets/demo.css">
 
  <script src="../build/tracking.js"></script>
  <script src="../build/data/face-min.js"></script>
  <script src="../src/alignment/training/Landmarks.js"></script>
  <script src="../src/alignment/training/Regressor.js"></script>
 
  <script src="../node_modules/dat.gui/build/dat.gui.min.js"></script>
  <script src="assets/stats.min.js"></script>
 
  <style>
  video, canvas {
    position: absolute;
    margin-left: 100px;
    margin-top: 100px;
  }
  </style>
</head>
<body>
      <video id="video" width="300px" preload autoplay loop muted></video>
      <video id="video2" width="300px" preload autoplay loop muted></video>
      <canvas id="canvas" width="300px" height="400px"></canvas>
 
 
  <script>
    window.onload = function() {
      var video = document.getElementById('video');
      var canvas = document.getElementById('canvas');
      var context = canvas.getContext('2d');
 
      leftEye = Math.round(Math.random() * 12) + "-left.jpg";
      rightEye = Math.round(Math.random() * 12) + "-right.jpg";
      lips = Math.round(Math.random() * 12) + "-lips.jpg";
      nose = Math.round(Math.random() * 12) + "-nose.jpg";
 
      var leftEyeImg = new Image();
      leftEyeImg.src = "./img/" + leftEye
      var rightEyeImg = new Image();
      rightEyeImg.src = "./img/" + rightEye
      var lipsImg = new Image();
      lipsImg.src = "./img/" + lips
      var noseImg = new Image();
      noseImg.src = "./img/" + nose
 
      var tracker = new tracking.LandmarksTracker();
      tracker.setInitialScale(4);
      tracker.setStepSize(2);
      tracker.setEdgesDensity(0.1);
 
      tracking.track('#video', tracker, { camera: true });
 
      tracker.on('track', function(event) {
 
        context.clearRect(0,0,canvas.width, canvas.height);
 
        if(!event.data) return;
        var lex0, lex1, ley0, ley1;
        var rex0, rex1, rey0, rey1;
        var lx0, lx1, ly0, ly1;
        var nx0, nx1, ny0, ny1;
 
 
 
        event.data.landmarks.forEach(function(landmarks) {
          for(var l in landmarks){
            // left eye: 0 to 11 (x) and 10 to 22 + 3 (y)
            if (l == 0) lex0 = landmarks[l][0]
            if (l == 10) ley0 = landmarks[l][1]
            if (l == 11) lex1 = landmarks[l][0]
            if (l == 22) ley1 = landmarks[l][1] + 3
 
            // right eye: 15 to 14 (x) and 13 to 23 + 3 (y)
            if (l == 15) rex0 = landmarks[l][0]
            if (l == 14) rex1 = landmarks[l][0]
            if (l == 13) rey0 = landmarks[l][1]
            if (l == 23) rey1 = landmarks[l][1] + 3
 
            // lips: 27 - 1 to 29 + 1 (x) and 28 - 1 to 30 + 1 (y)
            if (l == 27) lx0 = landmarks[l][0] - 1
            if (l == 29) lx1 = landmarks[l][0] + 1
            if (l == 28) ly0 = landmarks[l][1] - 1
            if (l == 29) ly1 = landmarks[l][1] + 1
 
            // nose: 16 - 2 to 18 + 2 (x) and 15 to 17 + 3 (y) 
            if (l == 15) ny0 = landmarks[l][1] 
            if (l == 17) ny1 = landmarks[l][1] + 3
            if (l == 16) nx0 = landmarks[l][0] - 2
            if (l == 18) nx1 = landmarks[l][0] + 2
 
 
          }
          context.drawImage(noseImg, nx0, ny0, nx1 - nx0, ny1 - ny0)
 
          context.drawImage(leftEyeImg, lex0 + 3, ley0, lex1 - lex0 + 1, ley1 - ley0 + 2)
          context.drawImage(rightEyeImg, rex0 + 2, rey0, rex1 - rex0, rey1 - rey0 + 2)
          context.drawImage(lipsImg, lx0, ly0, lx1 - lx0, ly1 - ly0 + 7)
 
        });
 
      });
 
      var gui = new dat.GUI();
      gui.add(tracker, 'edgesDensity', 0.1, 0.5).step(0.01).listen();
      gui.add(tracker, 'initialScale', 1.0, 10.0).step(0.1).listen();
      gui.add(tracker, 'stepSize', 1, 5).step(0.1).listen();
 
    };
  </script>
 
</body>
</html>