Prior Art: Imaging Systems as Biosensors

My primary interests in the Quantified Self concern two major topics:

  • Imaging Systems as Biosensors. A great example of this is Eulerian Video Magnificationwhich can detect a subject’s pulse using imaging alone.
  • Population-based Quantification. This is the general concept of gathering data across multiple people rather than just an individual, giving the opportunity to make broader observations that can apply to individuals as well as the group itself.

The first place I saw these two things combining was in simple crowd analysis systems, such as counting the number of people passing by a camera:

Subburaman, Venkatesh Bala, Adrien Descamps, and Cyril Carincotte. “Counting people in the crowd using a generic head detector.“ Advanced Video and Signal-Based Surveillance (AVSS), 2012 IEEE Ninth International Conference on. IEEE, 2012.

I feel that imaging systems have tremendous potential to get much more accurate mood and disposition information that other sensor systems. For example, in their “Sense of Space” project, Al-Husain et. al. link a physiological response to location:

Al-Husain, Luluah, Eiman Kanjo, and Alan Chamberlain. “Sense of space: mapping physiological emotion response in urban space.“ Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 2013.

For another example, in the Mappiness app people can manually log their emotional states to generate a map of the general feeling of places.

I posit that if you could look at the face of each person tracked in these systems, you could measure an extremely accurate mood for each person, passively.

Perhaps my research could be as “simple” as having a series of cameras across a number of locations, panning across populations and recording the emotions captured on people’s faces. This data could be similarly mapped as in “Sense of Space” or “Mappiness” but be much more accurate about the recorded emotional states.

However I would like to also generate conclusions beyond a simple mapping of place to emotion. The potential for applying a system of this to affective computing is huge, however I could not find any significant prior art that applied gathered emotional states to improve user experiences. I was at least expecting a paper showing how simplifying a user experience when the user is stressed improved effectiveness or productivity. The only thing I could find was this article:

Nasoz, Fatma, Christine L. Lisetti, and Athanasios V. Vasilakos. “Affectively intelligent and adaptive car interfaces.” Information Sciences 180.20 (2010): 3817-3836.

In this study, Nasoz et. al. reflected detected emotional states back onto a driver of a virtual car to encourage better behavior. However, the paper focused mostly on the detection of emotional states (no easy task given the set of inputs they used) rather that how the possible actions the car could take affected the user’s mood or task effectiveness.

I would like to bridge this gap, and find a set of ways to reflect the data gathered by my proposed system onto a crowd to improve a situation. Some ideas:

  1. Simply providing an indication of the emotional state of a crowd to the crowd by asking the obvious question: “Why is everyone sad/happy/angry?”. Done right, this could empower the crowd to fix the problem.
    1. Variation: Allow the crowd to vote on the problem/cause, hopefully illuminate and empower the crowd collectively.
  2. Same as above, but single out a person in the crowd who happens to be an emotional outlier, e.g. “Why are you sad? Everyone around you seems to be happy.”
  3. Show the emotional gradient based on location, either in map form or just on direction, to help people physically move to happier locations. This could be a fun interface for helping people find a place that has a more positive vibe. Over the span of a city, this could be a more “intuitive” metric for addressing the Fear of Missing Out (FOMO) – giving you a direction of where you’re more likely to have a good time, you can move to maximize your own happiness.
  4. Same as above, but as a more “intuitive” version of Foursquare. Rather than deciding whether a place is interesting based on your friends, one could make a decision based on whether or not the general mood of people at a place is positive.
    1. An interesting side effect of this could be a negative-bias to places that are frequented by people with generally “cool” dispositions, e.g. places frequented by hipsters would be graded negatively.

1 thought on “Prior Art: Imaging Systems as Biosensors”

  1. Great post. On a side note, the Eulerian Video project is fascinating, it inspired my own thinking in terms of the possible use of other kinds of imagery (i.e. satellite and remote sensing).

    The “intuitive” version of Foursquare could be a very relevant project for style-conscious cities like New York; I’m curious about what may result from quantifying emotional qualities of trendy spots.

    Since you are interested in emotion, and environment, I think your concept has potential overlap with:
    Woonyung Choi bio and lit review

Leave a Reply