A few weeks ago I met a young designer interested in digitally augmented mirrors: in particular he was interested in messing with people faces. Since this is the kind of stuff I have some experience with, we ordered a couple drinks and brainstormed about how he could do this and that.
I ended up writing a little demo showing how to easily change parts of people’s face in realtime and, since I think it could be helpful for other people too, I wanted to share it and quickly explain how it works.
Basically I track the user’s face with Jason Saragih’s library and create a mesh that can be overlaid on the lower part of the tracker’s face mesh; then I can use this “partial mesh” to create a UV map from the user’s mouth expression, or to blend a saved mouth expression into the live feed.
You can find the source code on my github and here’s a video showing how it works: