Group Members & Roles:
Claire Bradley – Steinhardt, Digital Media Design for Learning; Project Management, Communication with Strivright
Ayanna Seals – Poly, Integrated Digital Media; Researching Existing Digital Platforms and Technology, Lead on Coding Prototype
Ting-yun (Angela) Yeh – Steinhardt, Occupational Therapy; Researching Therapy, Lead on Organizing Literature Review
Name of Device: We chose the name “Sensory Steps” for our game. This incorporates the original conception of our group project, the sensory room. We also chose to develop a Kinect game which uses a sensor to read physical movements. The second part of the name refers to the fact that there is a progression of these therapy activities that the children must progress through.
Primary Population the Device is Being Developed for:
Our device is being developed for Strivright, the Auditory Oral School of New York. Strivright is a preschool for hearing impaired students that aims to prepare their students for mainstream schools by the time they are in 1st grade.
There are two main user groups that we are designing for within Strivright; the students and their therapists. The students will be playing the game, while the therapists will be setting up the game using the navigational components of the interface.
The Students – The students are 0-5 year olds with hearing impairments. Most of the students at Strivright have some type of device to help them hear. Many students have cochlear implants while others use hearing aids or Baha which use the bone structure in order to amplify sound.
Some students also experience motor delays and may vary in their cognition and level of speech.
Given this range of diagnoses and abilities, it is important that we work closely with the therapists to ensure that we include activities in our design that will be applicable to all of the children at Strivright.
The Therapists – There are many therapists who work in the school and would use our game in their therapy. Therapists are working with many children and using different types of equipment and activities on a regular basis.
It is important that we keep the needs of the therapists in mind in order to ensure that our game is used regularly. The game should be easy to set up, incorporate activities that are currently used by the therapists, and allow for flexibility so that the therapist can adapt them to their needs and the needs of any child they might be working with. We would also like to make the job of the therapist easier. It will hopefully be possible to do some of the work for them by incorporating features to save the progress of the students or allow therapists to take and save notes on a given session or child.
Our device is being developed in order to assist with the physical and occupational therapy sessions at Strivright. Specifically our device will help with therapy that focuses on the students’ vestibular system.
Description/Materials: Sensory Steps is a Kinect game that features a series of short activities that provide exercises for students who have trouble maintaining balance in various positions. The Kinect mirrors the physical movements of the student and allows them to interact with digital objects that are shown on a screen.
- Investigation of Population
Nowadays, interventions for children who have hearing impairments are often focused on their communication and verbal skills. However, recent research shows that these children might encounter vestibular dysfunctions, which influence children’s ability to balance and their sense of spatial orientation. According to Maes et al. (2014), 20% to 85% of children with sensorineural hearing loss demonstrate some type of vestibular dysfunction and tend to display higher percentages of motor deficits, specifically balance deficits.
The cause of motor difficulties among children with hearing impairment is not yet understood. According to Livingstone and Mcphillips (2011), there are two possible explanations for this problem. First, vestibular deficit theory suggests that because the cochlear and vestibular systems are closely connected, hearing loss caused by damage to the inner ear is likely to result in damage to the vestibular system as well (Livingstone & Mcphillips, 2011). In other words, according to this view, the motor difficulties of children with partial hearing stem from physical structures within the child. Another explanation is that children with partial hearing do not experience as much physical play as typically developing children because of delayed language and communication skills (Livingstone & Mcphillips, 2011). This may result in a delay of their motor development.
Many children with hearing impairments, including those at Strivright, receive cochlear implants in order to correct or lessen their hearing impairment. Research has shown, however, that these implants have little impact on the vestibular or motor deficits that these children experience (Rajendran & Jeevanantham, 2012).
According to Gronski (2013), in order to maintain balance, children need to receive appropriate input from the sensory system. If a child has difficulty with vestibular function, that child will benefit from therapy that focuses on focuses on both the underlying motor difficulties as well as strategies to compensate for these difficulties on occupational tasks (Gronski, 2013). Exercise programs that provide visual input to patients and incorporate muscle movements specific to the vestibular system have been shown to be effective in increasing postural control and motor performance of children with hearing impairment (Rajendran & Jeevanantham, 2012). As the research shows, children’s motor impairments are more pronounced in tasks when their eyes are closed, suggesting that children with vestibular and/or auditory difficulties rely on their visual and other sensory input to maintain their balance (Maes, L. et.al, 2014). Thus, not providing them visual information will result in decreased performance during therapy tasks (Maes, L. et.al, 2014). Thus, it is important that we provide visual feedback in our game.
- Therapeutic Use of Kinect
Research of therapeutic uses of the Kinect with children with motor deficits is limited because the technology is still in its infancy. However, according to Chang et al. (2011) and Luna-Olivia et al. (2013), Kinect system games provide a more natural user interface, allows users access to complex environments, fresh content to make repetitive tasks more enjoyable, and allows therapists to store data. Therapy done using a Kinect has also been shown to produce more correct movements than therapy done in a traditional manner (Chang et al., 2011).
Research has also implied that there may be some drawbacks or obstacles to using Kinect games for physical and occupational therapy. For example, Kinect games may exceed the cognitive and physical abilities of the user or may provide too much stimulation (Cheung, et al., 2013). Furthermore, because the Kinect technology is so new it has been shown to have trouble tracking users’ bodies and is designed to have one user, a problem when therapists need to be in close physical proximity to their patients (Cheung, et al., 2013). These study provide some useful perspectives on the problems that we will need to meet and solve in our project.
Chang, Y. J., Chen, S. F., & Huang, J. D. (2011). A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Research in developmental disabilities, 32(6), 2566-2570.
Cheung, J., Maron, M., Tatla, S., & Jarus, T. (2013). Virtual reality as balance rehabilitation for children with brain injury: A case study. Technology and Disability, 25, 207-219.
Gronski, M. (2013). Balance and Motor Deficits and the Role of Occupational Therapy in Children Who Are Deaf and Hard of Hearing: A Critical Appraisal of the Topic. Journal of Occupational Therapy, Schools, & Early Intervention, 356-371.
Livingstone, N., & Mcphillips, M. (2011). Motor skill deficits in children with partial hearing. Developmental Medicine & Child Neurology, 836-842.
Luna-Olivia, L., Maria Ortiz-Gutierrez, R., Cano-de la Cuerda, R., Martinez Piedrola, R., M. Alguacil-Diego, I., Sanchez-Camarero, C., & Del Carmen Martinez Culebras, M. (2013). Kinect Xbox 360 as a therapeutic modality for children with cerebral palsy in a school environment: A preliminary study. NeuroRehabilitation, 33, 513-521.
Mellon, N., Ouellette, M., Greer, T., & Gates-Ulanet, P. (2009). Achieving Developmental Synchrony In Young Children With Hearing Loss. Trends in Amplification, 223-240.
Rajendran, V., Roy, F., & Jeevanantham, D. (2012). A preliminary randomized controlled study on the effectiveness of vestibular-specific neuromuscular training in children with hearing impairment. Clinical Rehabilitation, 459-467.
Rine, R. (2009). Growing evidence for balance and vestibular problems in children. Audiological Medicine, 138-142.
Narrative of our Process
First Frankenstein Model
Our original idea was to build an interactive mat or blanket that would respond to the users touch by providing feedback in the form of light, vibration, sound, etc.
Visit to Strivright
We made our first trip to Strivright in early October. During this visit we met with the OTs and PTs at the school and toured the building. We were able to see some of the classrooms and the activities that are done in therapy. We were able to get a feel for the atmosphere of the school and observe their mission in action.
The school building is new and a OT/PT gym is still under construction. Instead of wanting a piece of equipment for the sensory room that they plan to develop, the therapists said that what they would really like is something that would enhance their current therapy methods. Specifically they want something that will help to develop the students’ vestibular or balance skills. They would also like to increase the reward kids get for success during therapy in order to make it more engaging. They also emphasized the importance of providing biofeedback to the kids during therapy in order to help them perform correct behaviors.
After visiting Strivright we reformulated our idea in order to fit the needs of the school. We thought that our original idea of an interactive blanket or mat could be turned into a mat placed on the floor that would sense whether children were equally distributing their weight while participating in therapy. We thought that this map could communicate with a digital screen that would provide an animated reward when the child’s weight was evenly distributed.
We did some initial research into existing technologies that are able to sense physical input and translate it into digital output. We came across the SESNSEable Shoes project from Canegie Mellon.
and thought that we could develop something similar using an Arduino. When a child performed a task correctly in the physical world, it would be picked up by the sensors. The sensors would then communicate with the digital platform and provide a reward to the child in the form of an animation on the digital screen.
We also developed navigational wireframes to depict the interface that the therapists would interact with to operate the digital challenges. We thought it would be helpful to therapists to be able to save and analyze student progress and be able to add notes about a specific task.
After developing our initial wireframes we sent them to Strivright in order to get their feedback on both the concept in general and the sketches. Their feedback was generally positive.
We applied for and received an award from the prototyping fund from the Entrepreneurial Institute and Greenhouse. We also attended a workshop for the fund that featured a speaker from IDEO who discussed the importance of prototyping and the different forms it can take. He presented many case studies of prototypes that they have developed for their work. Receiving this fund will allow us to put more money into our design and show our work to a larger audience at the showcase at the end of the semester.
After reflecting on our idea and receiving feedback from our midterm presentation to the class, we realized that our idea of programming sensors to communicate with a digital platform might be overly complicated. Unless the therapists were interested in receiving the kind of specific data that the sensors would be measuring, we might get the same results from using a Kinect game which would be much easier to develop. In order to get a better idea of what the therapists were hoping to gain from incorporating new technology into their therapy routine, we sent the following questions to the therapists at Strivright:
- We would like to have a list of the skills you tend to focus on in therapy. Since we have a limited amount of time to develop our prototype, we are hoping to focus on one or two of these skills while still developing a system that would allow for all of the skills to be incorporated in the future. So, if you could select a few skills that are most essential for us to target that would be great.
2. For those one or two skills, what is the range of abilities that we would be designing for? In other words, what are the differences between what the most advanced students and the ones who need more help are able to do.
Also, for these skills, what are the types of behaviors that you look for or encourage during therapy?
3. What would be the most helpful things for you to receive from the technology we are building? We know that you were interested in providing biofeedback for the kids, but what else would you like? Would you be interested in having the technology provide data in order to save and track the performance of students? For example, would you like to be able save information such as the date students completed tasks? Or would more specific data such as measurements of weight distribution be helpful?
4. What kind of technology are you and the other therapists accustomed to or feel comfortable using? Do you have experience with tablets, Kinect or other game consoles, projectors, or computers? Do you have experience with Apple products or Android and PC?
5. How often do students have therapy sessions to work on the skills we would target? How often would you be using what we develop to assist with this therapy? How long would a typical session last?
6. Are there any tasks, colors, or visual rewards that the students would enjoy for the digital display? Is there anything that they would not like? Would physical feedback (like gentle vibration) be helpful during therapy to encourage the behaviors you’re looking for during therapy?
7. How much experience do the students have with technology? Are they comfortable looking at screens? Do they have experience using tablets, computers, game consoles, etc.?
- Do you have an estimated range for the height of the students?
9. How many kids have cochlear implants vs. other types of hearing aids, and are there differences in the therapies or behaviors of these two groups?
10. Do you have any suggestions of resources that we might look into to better understand the connection between the vestibular system and the formation of the cochlea?
After receiving their feedback we realized that the data was less important than providing biofeedback and exciting rewards. So we decided to switch gears again and move forward with developing a game for the Kinect.
We decided to narrow our focus to two of the skills that were mentioned in the response to the questions; standing still while maintaining balance, and standing still while reaching.
We did some brainstorming for what the digital tasks could include for these skills:
Standing for Period of Time
- Creating a segment (3 seconds) and if they stand for that segment of time something comes up on the screen, the longer they hold their position the more happens on the screen
- Multiple of the same object appearing (stars, bubbles, etc.)
- Object growing (tree, flowers, balloon, bubble, etc.)
- Animation that only runs if they are doing something correctly (if they remain standing it will run, if they move it will pause, if they stand again it will resume)
- The longer they stand something on the screen changes
Standing and Reaching
- Controlling movement of character
- Feeding or giving to character
- Grab something
- Collecting things (food, apples, etc.)
- Puzzle game (collecting pieces and putting them into puzzle)
We also received the following input from Livia, our guest speaker:
- Less overwhelming, more abstract, given a goal, track progress in an entertaining but not distracting
- Flower, Flow, Journey
- Dealing with frustration → rewarding resilience and having personal bests
- Survey to parents about media exposure (things they like and are exposed to)
- check out indie games
After this session we decided that for our standing still task, either bubbles of varying size and colors should appear on the screen as the child remains standing or as a child remains standing a single bubble gradually grows and then pops. We selected these two variations because we felt that they were simple enough that they would not distract the child from the difficult physical task that they were engaging in at the same time.
For our reaching task we decided that a single balloon should appear on one side of the screen and the child would need to pull it over to a bunch of balloons that would appear on the opposite side of the screen. Again, we felt that this simple animation wouldn’t interfere with the child’s concentration on the physical task at hand.
We set up a time to visit the school again to do some user testing. We were anxious to get feedback from both the instructors and the students on our wireframes so far. In order to get more informative feedback from the children, we developed some simple animations in Keynote that mimicked what the game would look like. We also wanted to observe some therapy sessions with several children at a range of ability levels in order to see what tasks we might be able to mirror in our game.
However, the major reason we wanted to visit the school again was to determine whether the Kinect would be able to track the movements of the children, given their small size. Therapy sessions also often involve the children sitting or kneeling, making them even smaller.
Results of User Testing
- The Kinect was able to read the movements of children at a range of heights. It has a little trouble detecting feet. We should be able to code around that.
- The regular therapy activities that we observed encompasses activities that we could mimic in the Kinect activities
- The wireframes we’re too simple and children were losing interest. The final iterations need more bells and whistles. The therapists recommended psychedelic colors, random sounds, screen filling graphics and animations.
- The therapists recommended including different themes so the children don’t become bored of the chosen animations. Some suggestions were animals and vehicles.
- The therapists also recommended that instead of providing the children with a target body shape on the screen, we should use an animation on the screen to show the children what they should do and then the children will imitate the action done by the animation
We asked the therapists to tell us two activities they would like us to move forward in developing. They chose standing while reaching and standing and crouching because they use them often in their regular therapy sessions.
Developing the Game
After the user testing we set to work coding the game. Ayanna worked in Unity 3D and C# to create the game for Kinect v2. We chose to move forward with the standing while reaching activity rather than the standing and crouching because we thought it would me simpler to develop, a welcome quality when you are attempting something new. Ayanna was able to create objects on the screen that the player can interact with my using their arm movements. These objects can be picked up and moved around the screen.
Prototyping Fund Showcase
We were able to present our prototype at the prototyping fund showcase on December 4th at the Poly Greenhouse. We had a table set up that showed some of the wireframes that we have developed, a poster that summarized our work so far, and a short demo using the Kinect. The showcase was well attended and we were able to share our work with many people who stopped by our table. We even connected with one Poly professor who has used a Kinect game in physical therapy with adults.
Final User Testing that Never Was
We had hoped to fit in another user testing before the end of the semester. We wanted to test what we had so far of the actual Kinect game with the children and go over the final wireframes with the therapists to receive feedback. However, we were unable to make the game easy enough for the children to operate it. In its current state, in order to interact with an object on screen, a player needs to position a hand-shaped cursor over a given object by moving their arm and then close their hand making a grabbing gesture. This can get rather difficult because the cursor location is a little off from the location of the player’s actual arm. This is why we didn’t feel the game was ready to be tested with the kids. We decided to just send a demo video to Strivright along with our final wireframes and the list of features we had discussed in order to get their feedback. It was disappointing that we weren’t able to visit the school one last time, but we didn’t want to have the children become frustrated with the game.
(Prototype project file can’t be uploaded here)