Sign Lines
Lucia Gomez
Advisor: Tiri Kananuruk
Sign Lines is a paint plotter that will represent American Sign Language in an abstract way, allowing a participant who knows ASL to transform any sentence they sign into a colorful painting.
Project Description
Sign Lines is a paint plotter that will represent American Sign Language in an abstract way, allowing a participant who knows ASL to transform any sentence they sign into a colorful painting.
The motion paths of a user's hands through space are mirrored by strokes of a paintbrush on a canvas. At the same time, the detailed handshapes that a user's fingers form while signing determine the paint colors of the brush strokes. In this way, the plotted painting is a small takeaway that captures what was signed in a particular moment in time, condensing a personal 3D message into a 2D space.
As with most technological advances relating to language (ex: LLMs, machine translation, language learning apps) sign languages are often excluded. Although it’s estimated that there are over 300 distinct sign languages in the world, and over 70 million deaf and hard-of-hearing people who communicate via sign language, the technology doesn't yet exist for accurate sign language machine translation.
Sign Lines aims to combine ASL, art, and technology without relying on machine translation and to include the Deaf community in creative experiences made possible by emerging technology. At the same time, Sign Lines hopes to educate the hearing community about the linguistic depth and beauty of sign language.
Portfolio: www.lucia-gomez.dev
Instagram: @lugoo.dev
Technical Details
Python
TouchDesigner
MediaPipe
Raspberry Pi
Flask
Arduino
Custom PCB
Axidraw
Watercolor
Research/Context
Christine Sun Kim
Christine Sun Kim is a famous Deaf artist who often works with sound and music. She creates art that represents how she as a deaf person visualizes sound. In 2020 she was invited to perform the national anthem in ASL at the 2020 Super Bowl. The image below shows her interpretation of the song in ASL gloss, taking full advantage of the spatiality of ASL. To me, transcribing the interpretation in this manner that shows the signs’ physical position in space relative to her body is a unique representation of the song’s meaning.
soundwave.love
This project, made by an ITP alum, is a physical representation of the audio waveform trend I described above. Customers can submit an audio recording and receive a custom piece of jewelry that represents the waveform. This inspired me to create some kind of takeaway or gift for my project’s output.
si5s
… and other sign language writing systems. These frameworks tried to create orthographies for sign languages that are not based on existing alphabets. They aren’t very widely used outside of academia. But they did create a system capable of representing the semantic meaning attached to a wide variety of gestures. While my project will drop the semantic meaning, I’ll need some kind of generative system.
Licia He’s paint plotters
Licia He is an artist who has created an open-source framework for working with paint plotters. She has experimented with lots of different techniques and materials to provide a comprehensive guide for how she produces her beautiful plotter paintings. I’m particularly inspired by her watercolor paintings made with an Axidraw plotter. I want a similar hand-drawn and non-precise look and feel for my plotter output.