I am looking into implementing a novel approach for 2-D camera rectification as an OFx plugin. This method of rectification requires no input from the user (provided your image/camera has EXIF data which is almost always the case) – forget about making and using a checkerboard to rectify your image!

I learned about this technique when I took the Computer Vision course (taught by Dr. Sohaib Khan) in Fall 2012 at LUMS. We covered one of the recent publications by the CV Lab at LUMS: “Shape from Angle Regularity” by Zaheer et al. in Proceedings of the 12th European Conference on Computer Vision, ECCV, October 2012. This paper uses ‘angle regularity’ to automatically reconstruct structures form a single view. As part of their algorithm, Zaheer et al. first identify the planes in the image and then automatically 2D rectify this image solely relying on ‘angle regularity’. That’s the part I’m interested in.

Angle regularity is a geometric constraint that relies on the fact that in structures around us (buildings, floors, furniture etc.), straight lines in 3-D meet at a particular angle most commonly 90 degrees. Look around your room, start counting the number of 90 degree angles you can find and you’ll see what I mean. Zaheer et al. use the ‘distortion of this angle under projection’ as a constraint for 3D reconstruction. Quite simply, if you look at plane from a fronto-parallel view you shall see the maximum number of 90 degree angles possible. That’s what we’ll search for: we look for the “… homography that maximizes the number of orthogonal angles between projected line-pairs” (Zaheer et al.).

Following the algorithm used in the paper (and MATLAB code available at http://cvlab.lums.edu.pk/zaheer2012shape/) I plan to generate a few more results to see how practical it is- from the results given in the paper, it seems very promising. The algorithm for 2D rectification relies on searching the lines in the image, assuming line-pairs to be perpendicular, and then using RANSAC to separate inliers and outliers line-pairs in order to optimize 2 variables (camera pan and tilt, Note: focal length is known from EXIF data). The MATLAB code relies on toolboxes provided within MATLAB (for example RANSAC) which I should be able to find open-source C++ implementations of- The algorithm, although conceptually straight-forward, might not be as easy to implement and optimize when working with C++. I would work towards it and judge the time commitment it requires.

Once I’m done coding the plugin, I would want to make some cool examples that demonstrate the power of this algorithm. If the frame-to-frame optimization is fast (i.e. the last frame’s homography seeds the initial value for the next one), I could try to make this real-time.

I have not yet come across a C++ implementation of this technique, and the only OFx plugin for camera rectification that exists right now (ofxMSAStereoSolver) depends on the checkerboard approach.

Paper: http://cvlab.lums.edu.pk/sfar/
Aamer Zaheer, Maheen Rashid, Sohaib Khan, “Shape from Angle Regularity”, in Proceedings of the 12th European Conference on Computer Vision, ECCV, October 2012

Related OFx Plugin: ofxMSAStereoSolver by memo

Ideas for final project

For my final project I want to do something fun with projectors. For several reasons:1. I have a projector at home and I immensely enjoy it. 2. Among all forms of displays, projectors probably have least defined shape and scale. It can magnify something that only occupies a few pixels on screen to a wall-sized (even building-sized) image. I think it is really magical. 3. I found some really good inspirations in the projects Golan showed in class. One of them is Light Leaks by Kyle McDonald and Jonas Jongejan:

This project reminds me of a thing that almost everyone used to do: using a mirror to reflect light spots onto something / someone. I still do that with the reflective apple sign on the back of my cellphone. I think the charm of this project comes from the exact calculation they did that resulted in the beautiful light patterns. Though reflecting light spots with disco balls alone is a cool enough idea, the project would not be as stunning as it is if the light leaks had been randomly reflected onto the walls in random directions.

A little experiment

I was really amazed by the concept that ‘projection can be reflected’, so much that I carried out this little experimentation at home:

This is the projector I have at home. We normally just put it on a shelf and point it at a blank wall.

 I

The other night I put a mirror in front of the light beam it projected. The mirror reflected a partial view of the projected image onto a completely different wall (the one that’s facing my bedroom, perpendicular to the blank wall). It’s hard to tell from the photo but the reflected image was actually quite clear. As I rotate the mirror around its holder the partial image started to move around the room in a predictable elliptical trajectory. That meant I could sort of control the location of the image in 3D space by setting the orientation of the mirror. If I could mount a mirror on a servo and control the orientation of the servo, with some careful computation (that I don’t know how to do yet), I’d be able to cast the projection to anywhere in a room. Furthermore, the projected image doesn’t have to be a still image – it could be a video, or be interactively determined by the orientation of the mirror.

This opens an exciting possibility of using the reflected light spot as a lens into some sort of “hidden image”. For example, the partial image could show the scene behind the wall it’s projected onto. The light spot in a sense becomes a movable window into the scene behind – and it’s interactive, inviting people to move it around to explore more of the total view. Or, the projector + mirror could become a game where the game world is mapped to the full room. Players can only see part of the world at once through the projected light spot, and they move their characters by rotating the mirror / interacting with other handles that manipulate the mirror.

If all I want is to project an image to an arbitrary surface, why not just move the projector?

Well, the foremost reason is that moving a mirror is way easier than moving a projector. The shape of the projection can also be based on the mirror shape so that we don’t always get an ugly distorted rectangle. The idea can be relatively easily set up with the tools we have in hand. Another motivation is that by having multiple mirrors, each reflecting certain region of the raw projection, the original image can be broken into parts and reflected in diverging directions. They can all move independently, but ultimately using light from the same projector. Light Leaks uses this advantage very well to emit light spots in numerous directions.

 

So that’s what I’ve thought of so far. There are still many undetermined parts, and I’m not sure how challenging it would be implementation wise. I have played with Arduino only in middle school and it was such a disaster that I completely stayed away from hardware in college. But I’m willing to learn whatever is needed to make the idea come true. I’m currently still researching other works for inspirations, also trying to make sure that my idea isn’t already done by someone else.

Other inspirations:

Chase No Face – face projection. Also discussed in class.

More to come…

Update:

Taeyoon and Kyle pointed me to this project.