A paper presented this summer on Eulerian Video Magnification showed some incredible techniques for extracting information invisible to the naked eye, from video. I’m looking at it this week to see how much I can understand what the approach is.
Here’s how it works.
Using the precedent used by some heart rate sensors, the Eulerian video magnification proposes to be able to amplify the changes in color that happen as blood flows through the body. In finger heart rate sensors, an infrared light emitter shines through the finger to an infrared light receiver, and as blood is pumped through the finger, the light received is obstructed at the same intervals creating a reading. In a similar fashion, the color of our skin changes slightly as blood is pumped, it is this change that Eulerian video magnification attempts to detect. Variation in movement is also amplified to detect things like breathing.
• “consider time series of color values at any spatial location then amplify variation in given temporal frequency band of interest”
ie. changes in redness in a face, may be assumed to indicate change in heart rate
• “study and amplify the variation of pixel values over time in a spatially multi-scale manner” then amplify the color changes over time at certain positions.
The process begins by deconstructing the initial video frames into different spatial frequency bands then filtered with temporal bandpass – determining the filter depends on what one chooses to observe.
ie., for motion – use broad passband, to observe color, narrow passband
Then each spatial frequency band receives “temporal processing” which basically compares particular pixel locations over time. The frequency layer that is determined to show information of interest is magnified and all layers are put back together. This final output is like the original video clip, but with the exaggerated motion or color.
The degree of magnification required is determined by (very complex) equations that balance between enough magnification to become significantly observed, and not so much magnification that noise is let in. The Eulerian method also allows the potential to amplify certain areas of video, in order to isolate the areas of interest and not anything beyond.