Talk Birdy Group

Project Title: TalkBirdy Web Application

Group Members’ and Primary Roles:

Charmaine Sum – UX Design

Lisa Song – Visual Design

Anne Borromeo  – Research


Primary Population For Whom the Device is Being Developed:

The primary population includes the speech pathology researchers and the speech pathology students whose ages range from 20-60 years old.  They specialize in academic research for disorders that affect an individual’s ability to speak intelligibly. The TalkBirdy app is used for academic research, crowdsourcing, and as a supplemental learning tool for disordered speech production. 

What is TalkBirdy?

TalkBirdy is a crowdsourced platform for clinical research and training in speech production particularly the ‘r’ sound. This project is headed by Tara McAllister Byun and Daphna Harel. The resulting ratings will be used in research on speech therapy. This will be a replacement for the current method of gathering ratings, which is through a mechanical turk task.

Why is crowdsourcing important?

Online crowdsourcing represents a promising source of valid ratings of speech, but high cost and the need for technical expertise currently stand as barriers to large-scale uptake of this method. This online platform can allow research to obtain crowdsourced speech ratings through an interactive digital training tool targeting students and clinicians in communication disorder. It will allow researchers to obtain valid measure of speech at a fraction of the time and cost that is required by traditional methods.

Purpose of the Device:

The TalkBirdy web application that will be used to rate the quality of speech in short speaking samples. 

TalkBirdy Fall 2016 Project

Our goal is to develop a front end web application  that will be used to rate the quality of speech in short speaking samples, with research headed by Tara Byun. The resulting ratings will be used in research on speech therapy. This will be a replacement for the current method of gathering ratings, which is by a mechanical turk task.


Research

Literature Review 1

Children with phonological delays or disorders are known to give speech patterns that are far different from patterns of normal adults. McAllister Byun & Harel (2016) investigated 40 speech tokens through Visual Analog Scale and binary ratings scales with samples gathered from 29 naive listeners via Amazon Mechanical Turk. The study noted the advantages of using binary versus continuous rating scales when aggregating responses over large numbers of naive listeners recruited through online crowdsourcing. Stimuli used in this study was the mispronunciation of the North American English/r/. The researchers concluded that both binary and VAS crowdsourced listener ratings produced high levels of validity. Both tools also had high correlations with an acoustic gold standard.  

Byun, T. M., Harel, D., Halpin, P. F., & Szeredi, D. (2016). Deriving gradient measures of child speech from crowdsourced ratings. Journal of Communication Disorders, 64, 91-102. http://dx.doi:10.1016/j.jcomdis.2016.07.001

Literature Review 2

In a study by McAllister Byun & Hitchcock (2012), 11 children were observed in a period of 4 to 6 weeks given traditional treatment and another 4 to 6 weeks biofeedback treatment for the misarticulation of /r/. Based on the results of the investigation, there was no significant difference in the scores of children before and after traditional treatment. However, the /r/ sounds that were produced after the biofeedback treatment were recognized as perceptually and acoustically correct in children having difficulty with the /r/ misarticulation.

Byun, T. M., & Hitchcock, E. R. (2012). Investigating the Use of Traditional and Spectral Biofeedback Approaches to Intervention for /r/ Misarticulation. American Journal of Speech-Language Pathology, 21(3), 207. http://dx.doi:10.1044/1058-0360(2012/11-0083)

Crowd Sourcing Importance

  • This interactive tool targets students and clinicians in communication disorder
  • Can allow researcher to obtain valid measure of speech at a fraction of the time and cost
  • Lower barrier of technical expertise
  • Can help with research process

Similar research projects for speech therapy with Front-End Application – NiNiSpeech

http://cs.ninispeech.com

    • ninispeechNiNiSpeech enables people who stutter to take control over their speech, using unique mobile technologies.
    • NiNiSpeech offers people who stutter a safe platform to engage in speech activities, build confidence and experience spontaneous interaction with speech partners who share ambition, sensitivity and mutual commitment to make progress.

Project Development Schedule

talkbirdy schedule

schedule-1schedule-2


User personas:

  1- Principal Investigator

  • Age: 32 / Gender: Female /Highest Education Level: PhD in Linguistics

Background: The principal investigator works in the speech research project at NYU. She has a good background with technology.

How does the principle investigator use the TalkBirdy app?

  • Evaluates student researcher’s ratings
  • Groups clips together
  • Uses the ratings & sound clips for further research
  • Supervises the student researcher
  • Aims to improve quality of communication for users of TalkBirdy

  2- Student Researcher (Speech Language Pathology Student)

  • Age: 24 / Gender: Male / Highest Educational Background: Undergraduate

Background: The student researcher is currently studying Speech Language Pathology at NYU. He is really immersed in technology where he is up to date with the latest softwares. He is also very active on social media.

How does the student researcher use the TalkBirdy app?

  • Listens to speech samples
  • Rates sound clips in a spectrum 15-20 minutes each session

Uses TalkBirdy as an extra learning tool

3- Administrator

  • Age:  40 / Gender: Male /Highest Education Level: PhD

Background: Web Development, Linguistics, Statistics. Good knowledge in Technology and Web Development

How does the principle investigator use the TalkBirdy app?

  • Should be able to do everything a researcher can do, as well as push experiments from tokens anyone has uploaded
  • Once a token has been rated enough times, so that we no longer have uncertainty about its true rating then the administrator should be able to mark this token as “completed” or “gold” and set its “true value”
  • Group different categories together and make recommendations to student researchers
  • Connect the front end and the back end of TalkBirdy

User Testing was done by the the Interactive Prototype. We tested with the administrative researchers,  Speech Language Pathologists, and student researchers.

User Testing Questions:

  1. How do you select a categories to begin a testing session?
  2. Do you know what question you are currently on?
  3. How do you go back to the main page to select another session?
  4. How do you view or edit your personal information?
  5. Are the instructions for TalkBirdy clear?
  6. What do you think of the visuals of TalkBirdy?
  7. What do you think of the overall flow of TalkBirdy?

User Testing Feedback:

user-testing2

  • add feedback note here – lisa

After the User Testing, we made changes to the design:

  • Added a list of questionnaire for sign up
  • Added a Themes Page for different categories of test
  • Changed the rating accuracy measurement from comparing the users with the average to the amount of gold tokens per test
  • Added an instructions overlay screen for the ratings page to guide users how to rate the clips
  • Added personal profile information and allow it to be edited

Final Design:

The final design for the webapp of TalkBirdy aims to be functional and engaging.

The Final Interactive Prototype for TalkBirdy:
https://invis.io/PM9LB1XXA

sign_in

onboarding_questions

themes

sessions

instructions

rating-overlay

rating-2

rating-last

finished

completed_session

rating_accuracy

account_info


Challenges:

The main challenge for TalkBirdy is to create a really clear flow for the testing process.

The main goal of the project is to create a web application that will be used to rate the quality of speech in short speaking samples for researcher. The Resulting ratings will be used in research on speech therapy. The web application will act as a replacement for the current method of gathering ratings, which is by a mechanical turk task. Crowdsourcing is a big part of this project and we want to get as many users to use this so that the ratings would be accurate.

To achieve this, we aimed to develop an engaging and user friendly website that is not overly “gamified” as this is a research tool. We also needed to match the existing requirement and standards that the TalkBirdy researchers have been working on previously.  The most challenging for us to come up with a clean and user friendly flow that is really easy for a user to carry out a long testing session. We also wanted to create fun visuals that are not too distracting from the content.


Costs for Developing TalkBirdy: 

The cost of developing the web application of TalkBirdy is quite minimal. Most of the project was done by the students in the current DAT team as well as a backend development team from last year’s DAT class. The leading researchers also received grants for this project.


Future Plans: 

The next steps for TalkBirdy will be for the backend dev team to finish building out the front end web application that is designed by the team. We have worked on specs, style guides as well as exported all the assets for them in order to assist the back end team to develop the project smoothly. The backend development team should finish the front end development of TalkBirdy by Dec 12, 2016. The front end application will also be linked with the back end data section by the back end development team.

Specs for the Design

wireframe-spec-instructionswireframe-spec-instructions-copy

 

wireframe-spec-themeswireframe-spec-sessions

 

wireframe-spec-instructions-2wireframe-spec-overlay

 

wireframe-spec-ratingswireframe-spec-done

 

wireframe-spec-completedsessionswireframe-spec-ratingaccuracy

wireframe-spec-edit

Style Guide 

color-spec fonts

Asset Folder 

asset-folder


Journal:  Documentation of Weekly Progress

October 4, 2016 (Week 1) – Meeting with Tara and Daphna (principal investigators of TalkBirdy), write out project plans and goals

This PDF is our plan to kick off the project TalkBirdy Fall 2016: talkbirdyfall2016planplan-1

plan-2

plan-3

October 11, 2016 (Week 2) – Locked down schedule, sketched our initial visuals (look and feel explorations), card sorting, sketching, brainstorming of possible visuals and plans for TalkBirdy, wireframes (V1), user flow (V1)

fullsizerender-1fullsizerender-4

fullsizerender-3

fullsizerender

talk-birdy-card-sorting user-flow

October 18, 2016 (Week 3) -Sketching, created user profiles, wireframing (V2), visual exploration selection and refinement, prepared for presentation.

intro

introduction

rating  completed history

accuracy

October 25, 2016 (Week 4) – Midterm presentation, paper prototype, visual sketches, prepared questions to ask the researchers for our meeting, set up a meeting with the researchers and previous TalkBirdy backend group, further literary research

Midterm Presentation: 

https://docs.google.com/a/nyu.edu/presentation/d/1v6osBdfpN27XfwbinT8jNJ53JSZp0rDxUnnTgV0LhAQ/edit?usp=sharing

midterm-birdy

November 1, 2016 (Week 5) – Second meeting with researchers and previous TalkBirdy backend group, locked down the visuals, digital wireframing, interactive prototype (V1), discussed the questions for revision.

sign_in

home

instructions

rating

finished

finished_copy_2

November 8, 2016 (Week 6) – Visual design revisions (gold token revision, font, progress bar,  buttons), short feedback presentation in class

visual3

visual2

visual1

November 15, 2016 (Week 7) – Visual design revisions (bird icons, gold tokens, font, volume control, clouds), sign-up information page, instructions were revised, interactive prototype V2, prepared questions for client meeting 3

List of Questions:

  • Do we create a signup section? What information are we asking our user for the sign up section?
  • Do we want to have feedback for users when they have put the rating on the wire? Should there be a sound or like a visual cue?
  • What is on the instruction screen? Do we want to have a tutorial?
  • Do we want to let the users go back to the previous clip?
  • Do we want our users to have a replay function for each sound clip?
  • Do we have any numerical markings on this scale? Some indicator to how good a clip is?
  • What are the acceptable labels on the ratings page? Good or bad?

sign_in

sign_in_question

session

sessions

instructions

rating

finished

completed_sessions

rating_accuracy-_history

edit_personal_information

November 22, 2016 (Week 8) – Front-end website updates, added overlay screen for instructions, locked down the account information page, added sign in and sign up session pages, visuals – changed the colors, font, and size to make it more accessible, sent everything to backend team

November 29, 2016 (Week 9) – Specs for TalkBirdy, exporting and packaging the assets for the backend group, finalizing all the documentation for the website, preparing for our presentation


References:

Byun, T. M., Halpin, P. F., & Szeredi, D. (2015). Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders, 53, 70-83. http://dx.doi:10.1016/j.jcomdis.2014.11.003

Byun, T. M., Harel, D., Halpin, P. F., & Szeredi, D. (2016). Deriving gradient measures of child speech from crowdsourced ratings. Journal of Communication Disorders, 64, 91-102. http://dx.doi:10.1016/j.jcomdis.2016.07.001

Byun, T. M., & Hitchcock, E. R. (2012). Investigating the Use of Traditional and Spectral Biofeedback Approaches to Intervention for /r/ Misarticulation. American Journal of Speech-Language Pathology, 21(3), 207. http://dx.doi:10.1044/1058-0360(2012/11-0083)

Ruben, R. J. (2000). Redefining the Survival of the Fittest: Communication Disorders in the 21st Century. The Laryngoscope, 110(2), 241-241. http://dx.doi:10.1097/00005537-200002010-00010

NiNiSpeech. (n.d.). Retrieved December 06, 2016, from http://www.ninispeech.com/