Craig Protzel

PROJECT DEMO

We presented our final project last Tuesday and the demo at that time was unfortunately not very successful.

This past week we focused much of our time on creating a functional “mobile system” that could actually be used in a real world hospital environment.

To accomplish this we decided to scale back and eliminate the second projector component of the project.  In terms of materials, we designed our own “creative” AR markers out of both acrylic and wood.  We wanted to create pieces that would appear playful and fun. The initial iterations and final markers are pictured here.

Regarding hardware, we had planned to use a small Pico projector, but the throw was not wide enough to accommodate a mobile unit with a goose neck mount only a few feet above the projected surface.  We decided to use a more standard short throw projector that would give us a wider projection.

In terms of the camera, we initially planned to use the same Logitech Webcam we had been testing with during the tech demos.  However, this camera it proved difficult to accurately mount, so we opted for a mini USB cam.  We attached this camera close to the projector’s line of sight.  Here are some images of the projector/camera mount.

We then created an additional Processing sketch to properly calibrate the camera and projectors field of view.  We used a digital zoom within processing as well as multiple cross-hairs to accurately scale and align the camera and projector.  With those two pieces of hardware in sync, we could ensure accurate projection mapping.

On the software side, we created a system where the computer could interpret which pieces were located at which corner of the board and, as a result, project different imagery based on the varying order of the pieces.  Once the camera found the board, i.e. all four corner-markers, a specific pre-loaded image would be passed into a keystone mesh and projection-mapped accurately on the board.

If an image is being projected and the camera loses track of any of the corner pieces, it will keep projecting the current image.  We thought this was important because if a user were to actually play on the board, they would constantly obscure the camera’s sight of the markers.  So, we wanted to ensure the experience would not be interrupted if the markers were blocked from the camera’s view. We did run into some difficulties when certain pieces were crossed and the mesh would get “tangled,” causing a crisscrossed inverted projection.

Prior to class, the entire system seemed to be working fairly well.  We decided to fully demonstrate the goal of the project, it made sense to move the camera-projector stand into the classroom for the actual classroom presentation.  The system failed miserably to track any of the markers within the classroom.  I think the change in environment, which included new lighting conditions, a different colored table, a new distance and angle of the camera/projector from the table must have contributed to a poor demonstration.  When we checked the quakity of the camera’s view, it was pretty low, but that was the same quality of view we had been testing with.  In retrospect, I would have left the system where it was and had the class come to it instead.

We continued working on both the system and the code over the next few days to improve the reliability.  We tried better cameras, which helped somewhat, but we weren’t able to get the calibration as accurate as we had with the mini-USB cam.  Also, the digital zoom required to accurately calibrate the cameras to the projector caused the image quality of the new cameras to suffer as well.

We also tried shining IR light on the pieces to help the camera better interpret them.  We only saw some improvement when we shined a huge IR light, which would not be feasible as a mobile hospital unit. We also realized that our original markers may not have been as robust as the original tech demo markers provided to us by Craig Kapp.  All that being said, we were able to get the project into a much better working state using all the original materials.

In terms of what is next, I have ordered some retro-reflective tape and plan to keep working on this project using an IR camera.  I’d like to create IR AR markers that can be tracked by an IR camera.  I’m hoping this will be the most robust and functional solution.

I’d also like to spend more time working on the actual application.  We got so caught up in trying to design a practical system that we didn’t spend as much time as we would have liked incorporating interactive content into the application.

Hopefully, over the summer I will improve the project.  What excites me about this its potential use in a hospital environment is that if we can get it working well with IR markers, it opens up the possibility to place the markers in any space, point a camera-projector unit at those markers and have an immediate mobile projection mapping unit up running very quickly.

AR TECH DEMO

After doing some additional research, we’ve realized a number of practical constraints.  First, nothing can be permanently affixed within a hospital room, specifically a camera or a projector.  Second, we’ve been told we have approximately :30 to get something up and running before its dismissed and discarded.

So, we have decided to create a mobile system where a camera and projector are attached to a gooseneck arm, which would be affixed to a mobile platform.

Second, we are abandoning the variability of lego pieces that can be assembled in any fashion and choosing to use AR Markers to accomplish the computer vision.  AR markers are easily recognizable by a camera, provide specific location, rotation, and identity of objects, and can function in a variety of environments.  By limiting the variability of what the computer is looking for, we \think we can have a much more reliable and accurate response.  We’ve been working with some paper markers and a standard web cam to get the computer side functionality in working order.

Currently, we have four markers reading in as the corners of a board and an additional marker registering as a “game piece.”  We also have incorporated a keystoning library to allow us to properly insert an image onto the board.  The image will conform to the position/orientation of the board, which will ultimately allow for an accurate projection.  The vid above demonstrates the current tech functionality.

For the final project in Computational Cameras, I am working with Andrew Lazarow.  We are both interested in the field of “Medical Media” and believe the hospital is a ripe environment for interactive content.

We started the project by examining hospital spaces, particular the hospital room, to explore possible project ideas.

In our research, we found a number of practical consistencies.  Hopsital rooms are designed with optimum functionality in mind.  Health standards require them to be clean and sterile; the need to easily move equipment and personnel in and out of the space requires openness; the diversity of patients in terms of illnesses and backgrounds requires a space applicable for anyone; and the transient nature of a patient’s visit requires the room to be easily reset for the next patient.

As a result, hospital rooms tend to be plain, barren, lightly colored, empty, non-descript, and impersonal.  Patients are thus forced to recover in a space that is devoid of personality.

In terms of the actual materials within a hospital room, there are some consistent features as well.  Almost always, there is a bed, a bedside tray that extends over the bed, a tv, and at least one barren wall or curtain.  The ceiling above the bed is usually flat with fluorescent lights.  There is often some furniture such as a small table and chair to accommodate visitors.  If a patient is lucky enough to have their own room, there is a possibility to add some form of limited personalization on a bedside table or counter such as flowers, pictures, or other belongings.  But, if a patient is in a shared room, the space for this might be even more limited.

With these characteristics in mind, we continued exploring possible projects.  We then came across this video for the Intel Oasis project:

We were inspired to create something along these lines but adapted for a Children’s Hospital.  Our idea was to create an in-room playspace for a child’s hospital room that would offer them a more humanized hospital experience and a more personalized room environment.

How are we going to do this?

Using the bedside tray as an interactive platform, a patient can create objects, through blocks or legos, which they can play with on the tray.  A camera mounted above the bed on the ceiling would detect these objects in terms of color/shape/orientation/quantity.  A projector, also mounted above, would projection map visual/audio responses onto the tray.  A second projector would be oriented towards a wall in the room and would project imagery corresponding to the actions/objects occurring on the tray.

The ultimate goal is to build on the existing hospital room infrastructure to create an immersive play space that will allow a patient to take control and ownership of their space.  As a result, they will feel more comfortable in their “own rooms” and, ideally, experience the benefit of improved health recovery.

For this week’s assignment to play with cameras in interesting locations, I experimented with the Junaio app on the iPhone as well as a GoPro HERO Cam as a “where-able camera.”

For the Junaio app, I was able to load this black and white image of a heart to act as an AR code.

I’ve been thinking about the possibility of having a first-aid app that can offer basic AR instruction for actions such as CPR.  So, when the heart image is recognized, ideally placed over a person’s actual heart, a pair of hands would appear and instruct a person on how they should push on a person’s chest to conduct CPR.

I was able to load the following image as an animated file and get it to play when the app sees the “heart” sketch.

However, because it’s a video file, it didn’t play on top of the camera feed, but rather took over the screen and played on its own.  To get video to play over the camera’s live view, Junaio requires additional encryption of the movie file.  If I continue down this path, I will definitely explore this process.

I also realized I had an SD HERO Cam that’s been packed away in a box I brought from Los Angeles.  I totally forgot about this little camera, which, with its protective housing and harness, is perfect for using anywhere and everywhere.  I decided it would be fun to take the camera out for a spin and see what kind of footage I could get by placing it in different locations.  Here’s a little video of what I was able to capture.

For this blog posting, we were asked to explore the questions/ideas/thoughts surrounding an environment where cameras are everywhere.  What does it mean if everything we do and everything going on around us is being recorded, whether by picture or video.  Is this a good thing, a bad thing, a scary thing, an amazing thing?  I guess it’s probably a bit of all of that.

My gut reaction is twofold.  First, if everything is being recorded all the time, I would imagine a system where there is so much data that it would be impossible, if it isn’t already, for humans to watch everything.  We will selectively seek out certain pieces of recorded information for a variety of purposes, but I would guess a majority will go untouched and unwatched, as is probably the case with most surveillance footage existing now.  But, as a result of the exorbitant amount of information, we will become increasingly dependent upon computers to process/sort/evaluate/filter all of this data.  This part seems a bit scarier to me given that our reliance on computer intelligence will only increase as the volumes of data increase.

I don’t fear that computers will be doing the bulk of the work.  What I somewhat fear is a system where we become so reliant on these computer systems and the systems themselves become so intertwined and convoluted that we could potentially lose our ability to control over how they operate.  As a result, we steadily approach a world in line with E.M. Forrester’s “The Machine Stops” where we are at the mercy of a computer system that no one really understands.  This future is probably a very far off scenario, but it’s worth considering as our lives become increasingly dependent on technology very few of us actually understand.

Looking at this “cameras everywhere” environment from another standpoint, one of benefit, I stumbled upon Microsoft’s SenseCam.  The SenseCam is “a wearable digital camera that is designed to take photographs passively, without user intervention, while it is being worn.”  It’s also built with a variety of electronic sensors that monitor the surrounding environment, including temperature, light, and motion.  The camera can be set to take a photo based on any of these sensory inputs.  The motivation for the SenseCam was a Microsoft Medical Media Lab study focusing on helping doctors manage their daily responsibilities in a hospital.

According to Microsoft, “capturing audio or video data using technologies like SenseCam throughout a clinician’s day could replace the onerous documentation requirements driving up costs and draining efficiencies in healthcare. Design the systems to ubiquitously and automatically capture the audio and video encounters of patients in the hospital.”

In this scenario, there’s definitely a trade-off for a doctor in terms of personal privacy versus public documentation, but there seems to be potential for making a doctor’s life considerably easier as they try to navigate a hectic fast-paced day and, at the end of that day, try to review and document everything that occurred.  The choice of taking snapshot photos as opposed to video allegedly provides an even easier means to scan through and review the day’s activities.

Now, I don’t think this camera solution is appropriate for all professions or environments, but for one in which there are extreme levels of liability, responsibility, multi-tasking, and pressure to keep track of anything and everything, I think this is a viable solution.  In terms of the actual environment, I would consider a hospital a “semi-public” space where surveillance and monitoring would be expected, and even wanted, from most parties involved.  Given this acceptance of recording in the given space, the SenseCam would not dramatically change the landscape of what already occurs within a hospital.

However, I don’t think forcing this type of accessory onto a doctor would be appropriate either.  I would like to see a doctor have the option to use or not use the SenseCam.  I also don’t think audio recordings of confidential health information or private conversations with other doctors, patients, and patients families would be appropriate.  Microsoft claims they have experimented with audio recording, but that it’s “not featured in the current hardware.”

All that being said, having a stop-motion like string of pictures to remind a doctor of everything they did that day could relieve a great deal of pressure put on the doctors to constantly document everything they do.  Also, hopefully, it could help them provide a more accurate description of what occurred and when it happened.

So, on one hand we run the risk of turning our lives over to the mercy of cameras and computers everywhere for the sake of ability over function, while on the other hand we can realize actual benefits from utilizing this technology in meaningful and appropriate fashions.  I hope we can skew towards the latter in terms of results and direction.  However, as is usually the case, all things will find a balancing point.  My prediction is that we’ll find ourselves somewhere in the middle… and by middle I mean skewed to the former.

For the Comp Cameras midterm, Thiti and I ran into some difficulties with the Kinect camera, projector, and OF.  So, per DanO’s advice to go with the flow, we deviated from our intial aquarium idea (which we decided we’d tackle as our Spatial Media midterm project), and came up with “Where’s DanO?”  “Where’s DanO?” involved using a Kinect camera and a projector, both mounted above a table to track the movement of the tops of Pringles cans.  As the Pringles cans were slid around the table, a portion of a “Where’s Waldo?” image would be revealed on the top of the can.  Except, in this “Where’s Waldo?” photo, we inserted an image of DanO , hence ‘Where’s DanO?”

We used Pringles cans because the design of the cans allowed us to set a unique threshold for the height of the top of the can.  The Kinect camera was set to only look for blobs within a certain Z threshold, which was the exact location of the top of a can if it were placed directly on top of the table.  Also, because the cans are so tall, to slide them around the table, the user would naturally hold the can at its based and move it around.  Because of this, there was no hand/arm interference in the area the Kinect was looking, which would allow for accurate blob tracking.  Third, because the cans we cylindrical, the circular tops were ideal for projecting because they did not create a need to rotate an image based on the orientation of the can.

In terms of code, we ended up working in Java in Eclipse.  We set the sketch to determine the center point of a certain type of blob.  Once it had found the center point of an appropriate blob, we had it unmask a circular portion of the “Where’s DanO?” image at that location.  The projector was set up to accurately projection map the image onto the top of the can.  Here are some photos to get a better sense of what we did.

The most exciting part of the presentation for us and the class proved to be the end, when after finding all the hidden DanOs, (we doctored 3 Waldo images) we enjoyed the fruits of our labor by eating the Pringles.