\n This project explores the emotional dialogue between human movement and generative sound. By training a custom machine learning model on dancers\u2019 movements, the system detects three core emotional expressions\u2014Sadness & Inner Struggle, Conflict & Tension, and Freedom & Liberation\u2014and responds with generative sound textures mapped to each emotional layer. Inspired by contemporary dance practice and therapeutic movement exploration, this project empowers performers to \u201cspeak\u201d through their bodies and let their emotions shape the sonic space. It challenges the traditional performer\u2013music hierarchy by making sound reactive to the dancer\u2019s internal state, not just their external actions. The result is a dynamic, live emotional instrument that blurs the line between choreography and composition. <\/p>\n
\n The system uses ml5.js\u2019s neuralNetwork with time-series input (position, velocity, acceleration) from body tracking data to classify movement into three emotional categories. Real-time audio feedback is generated using Tone.js, including layered ambient textures, string glissandos, and percussion that dynamically respond to dancer input. The model was trained on annotated motion clips and deployed in a browser-based interface using p5.js and webcam capture. <\/p>\n
\n This work draws from dance therapy, the Laban movement system, and emotion recognition in computer vision. Inspired by personal experiences of reconnecting with the body through dance, the project aims to create an expressive platform where bodily emotion is not only seen\u2014but heard.\n <\/p>\n","protected":false},"excerpt":{"rendered":"
An AI-powered system that detects emotional expression in dance and generates responsive sound in real-time, transforming movement into music.<\/p>\n","protected":false},"author":0,"featured_media":2548,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[13,25,23],"class_list":["post-1116","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-30","tag-machine-learning","tag-music","tag-performance"],"_links":{"self":[{"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/posts\/1116","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/comments?post=1116"}],"version-history":[{"count":1,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/posts\/1116\/revisions"}],"predecessor-version":[{"id":1604,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/posts\/1116\/revisions\/1604"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/media\/2548"}],"wp:attachment":[{"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/media?parent=1116"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/categories?post=1116"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itp.nyu.edu\/thesis\/archive\/2025\/wp-json\/wp\/v2\/tags?post=1116"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}