Posted on

Extend Steaming

Yet another dimension of interaction you may enjoy while streaming.

Wenhe Li

https://youtu.be/PR42Sxhw9O4

Description

Nowadays, the interaction in live streaming is limited to verbal comments either on top of the content or at the side. However, streaming as entertainment, its content should be a joint effort of both the streamer and the audience. This project aims to find a new medium that could primarily extend the interaction dimension while streaming and watching streaming content. Therefore, Augmented Reality (AR) will be the ideal medium for streamers and audience in this project.

In detail, this project will add three interactions:

1) Tap & Text. It is like traditional verbal comments, but users(audience & streamers) can leave comments on any physical location. In this way, the comments should have a stronger connection to the surroundings/context.

2) Tap & Place. It is like the Tap & Text, but it places 3D models. Later, I want to make those models interactable to users by just getting the user's figure gestures from the camera.

3) Tap & Play. Instead of placing models and texts, this interaction will place multimedia(Video, Gif, and even other Streaming Content) to the physical location. The above two forms of interaction aim to connect the streamer and audience by offering a mutual interactable item ob both sides.

The whole idea of this project is to bring more dimensions to the live streaming interaction system and allow more audience to engage in the streaming content. As T.L Taylor comments on live-streaming by saying, “It is a form that plays with the boundary lines between audience and producer.” I hope this project could show the futuristic view of what audience-oriented live streaming could be.

IMA/IMB Shanghai
INTM-SHU.401.3
Capstone Studio (Shanghai)
Mobile,VR/AR
Posted on

家: An Art Book on Intergenerational Trauma

“家” is a multimedia art book that chronicles my experience with intergenerational trauma within my Chinese American family.

Maya Yanjie Wang

https://youtu.be/hejROfwPtnA

Description

My book exposes and chronicles intergenerational trauma within my Chinese American family. Three generations of my family’s history are recounted through interviews, poetry, photos, and paintings. Topics such as parent/child relationships, domestic abuse, mourning, and diaspora are addressed. I created this book not only to document and spark discussion about difficult and painful familial relationships, but also to reconcile my own experiences in an act of catharsis. The narrative progresses and regresses in a nonlinear way to reflect the true process of healing.

I invite viewers to interact with my work and think about their own experiences in tandem. Discussing family trauma is stigmatized, and when left unresolved, trauma manifests in dysfunction that is inherited by the next generation. The conception of this project stemmed from a desire to reconcile my own experiences, but it has developed into the creation of a metaphorical space where people can discuss and reflect on the shared issues and experiences of a minority identity. Art that comes from an honest and personal place has the ability to resonate with people and be a medium for self-reflection – despite the deeply varied human experience – because of universal themes.

IMA/IMB Shanghai
INTM-SHU.401.1
Capstone Studio (Shanghai)
Art,Narrative/Storytelling
Posted on

Surrounds

An app or a solution that uses AR technology to manage user conversations while doing live streaming.

Jinzhong Yu

https://youtu.be/K1oP1jwbKFQ

Description

Surrounds is an app or a solution that manages chats and conversations while doing live streaming. It adopts AR technology to categorize user comments and binds them to specific items in the streaming to make conversations organized.

Currently, there exist several ways of interaction during live streaming — Virtual Gifts, Comments Systems, or Danmaku Systems. However, all information or chats are totally wrapped up and mixed together. They are vaguely bound only to the streaming itself, rather than the specific topics inside. Survey shows that most users find it annoying when over 25% of the screen is covered by danmaku (a set of scrolling texts floating on the screen), especially when the texts are little related to the topic. So, when the streaming is covering large scale of topics or presenting complicated ideas or scenes, chats are completely garbled and hard to locate.

So, I designed Surrounds, which uses Google ARCore to detect and track objects in the streaming in order to categorize the ungrouped messages into various chatting groups. By using Firebase Realtime Database, the messages can be synchronized along with the streaming data and kept permanent. When the user opens a live streaming session, the application will automatically find objects on the screen and attach a clickable indicator on them. By clicking the indicator, the user will join a public group where all the messages here are specifically related to the object. This design allows users to receive only the intended information they are interested in, and hence reduce the spam of information.

Although the system is not perfectly designed and lots of other issues need to be concerned, it sophistically provides users a structured and clearer view of information during live streaming. It makes the users target the topic they are favor of and hence helps the users refine their inspiration on streaming.

IMA/IMB Shanghai
INTM-SHU.401.3
Capstone Studio (Shanghai)
VRAR,Mobile
Posted on

From Border to Border

An Immersive Short Story in Virtual Reality

Sarah Tahir

https://youtu.be/ZFpWjliTV-w

Description

From Border to Border is an immersive Virtual Reality experience that explores nostalgia, growing up, and being ensnared by the past. The experience takes the form of a journey in three parts: The Sandbox, The City, and The Sea. Throughout this journey, the visitor is both propelled forward by a vision of the sea and called back by an idealistic picture of the sand. In the end, the visitor finds the sea and realizes it was not what they envisioned. They are adrift and helpless, with only one way out.

Please visit the project website to watch the film and learn more. This experience is best viewed through a VR headset on YouTube VR, but may also be viewed as a 360 video in browser.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
VR\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\AR,Narrative/Storytelling
Posted on

Haptic Breathing Guidance Device for Flute

A haptic interface for breathing guidance in flute learning.

Yinmiao Li

https://youtu.be/SX_Ju6qKnJ8

Description

Haptic devices have made improvements in music learning. They can resolve the problems of space and time constraints in traditional learning methods and reduce the difficulties of learning music notations and terms. Previous haptic systems for flute fingering position learning proved that the learning rate is faster, and forgetting chance shrinks compared with learning from videos. However, flute performance consists of fingering and breathing. And existing haptic interfaces deal with fingering positions only. Research on technological-aided breathing guidance is mostly visual guidance or audio guidance. Haptic interfaces related to breathing were not used as guiding methods but mainly used to measure the people’s body position while breathing and provide feedback accordingly. Therefore, I intend to expand the flute learning from partial haptic guidance to full haptic guidance. This research will design a haptic interface for breathing. And this breathing guidance interface is a wearable device that is used to haptically instruct flute learners only on breathing control for playing the flute. The device leads the flute beginners breathing at the precise point during the song. The research intends to examine the users’ acceptance of using haptic guidance methods for breathing. This study not only provides a more convenient and accessible learning method for people with interests in flute or wind instruments but also has the potential to be applied to breathing guidance in the medical field.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
Education,Wearables
Posted on

Music Harry

A casual game to explore more about music

Yudi Jia

https://youtu.be/_PxYbcnqwKI

Description

This project is an interactive game to help amateurs learn more about music in an interactive and interesting way. Many music lovers want to further explore the world of music, such as composing their own songs and making musical accompaniments and arrangements but are restricted by their limited knowledge of basic music theory. However, existing traditional music theory classes are not that playful and interactive, and also need a long time to learn. Other musical apps imitating the piano/guitar interfaces only allow users to get a sense of one instrument. Thus, to help users learn and explore music in an interesting way, this project will illustrate music elements vividly using cartoon-like interface and characters. It is not gamification learning, it is a game which hopefully can let players know more about musical elements during their leisure time. While gamification learning focuses more on how to design the learning methods and material by imitating game mechanics, my project is purely a casual game which provides a bit of learning value.

This game is developed using Processing and p5.js, and imitates the interface of Super Mario so that users can quickly understand how to play the game. The notes and chords will be visualized as cartoon symbols or figures, floating in the air or moving on the ground. When the user captures the cartoon figures, musical knowledge will be displayed on the screen, and total scores will be updated each time the user interacts with the game characters. After playing this game, users will get a sense of musical notes (C,D,E,F,G,A,B), possible/popular chords that can be used in composing a music piece, and might also be able to design a chord progression for a set of melodies.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
Play/Games
Posted on

Magic Wish

A magic space for children to create & cast their own spells, with the hope that they can learn from Positive Psychology embedded.

Quinny He

https://youtu.be/ITuAWD6XaIw

Description

We have many tools that help kids to learn: Scratch for kids to access programming, Variation Playground for kids to learn music, Cognimates for kids to understand AI concepts, and we even have a whole museum exploratorium for kids to explore scientific phenomena at their own pace. However, all these learning experiences focus on talents, which are non-moral (such as understanding physics). Seldom do people consider helping children develop strengths, which are moral and contribute to virtues (such as the ability to love and be loved). In the field of Positive Psychology, strengths development contributes to personal happiness in both childhood and adulthood. Can we design a learning tool specifically for kids to learn happiness? Located in a dark room, the project creates a magical space for children to create and cast their own magic spells. With the guidance of a magic book teaching children how to use our magic wand, children can use different gestures to control the floating spells projected on a giant transparent canvas and design a new spell for their own wish. For instance, a “heart” shape spell for the wish that “I want my parents to get back together”. This project draws inspiration from Positive Art Therapy, the art therapy for positive psychology interventions, provides our young audience with a magical space to learn optimism, overcome grief, and regain hope.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
Education
Posted on

MediaMoney

MediaMoney provides simple analytics for any public Twitter account. With this handy tool, you can even estimate how much advertising money your favorite Twitter accounts can make with each tweet.

Hank D Wu

https://youtu.be/qo2QN_LnkZU

Description

The chief purpose of MediaMoney ties in directly with a relatively new phenomenon: the monetization of social media. Despite rapid changes in the social media landscape, such monetization has remained rather opaque to content consumers. While many consumers now understand that social media can be a real business that drives very real profits, there remains quite a bit of misinformation regarding the quantity of such payments. Some consumers tend to underestimate the revenue generated by social media influencers (likely because few formal transactions ever take place directly between the consumer and the influencer). However, others also validly overestimate revenue, especially after viewing content depicting the lavish lifestyles of such influencers. The reality is much more nuanced. Consuming “free” media on platforms such as YouTube, Instagram, and Twitter is a large part of an important value creation process. Thus, the primary objective of this project is to cut through the opaqueness and shed light on what truly happens.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
Education
Posted on

Subjective Mapping: (Re)visit NYUSH and its memories

A digital subjective map of the 8th floor IMA space at the NYUSH Pudong campus, where users leave and share their memories attached to any location.

Zheng Zhang

https://youtu.be/plVQvrSiIhQ

Description

Distinct from traditional mapping/cartography which intends to represent a territory as close to reality as possible, subjective mapping aims at producing a map using one’s own physical sense, emotion and memories that are not necessarily objective. Subjective mapping can be understood as the cognitive process that one reproduces his/her “story of experiences” at a specific location through making a map of the place (Nomadways). In creating such a map, one “stores and expresses his/her emotions into minimalistic shapes”.

The physical territory of the IMA spaces, added with personal annotations, is a community-created subjective map. The idea of this project is born from the context that our school is moving to a new campus in the Shanghai Qiantan area, and the current building will no longer be accessible by the NYU Shanghai community.

This project is aimed at creating a digital heritage of the current campus with students’ memory attachments, so that we can go back and revisit our alma mater when we are not able to do so physically in the future. Given the COVID-19 situation, the goal of this project can be extended to provide a community space for students, fellows and faculties to bond with each other while the campus is inaccessible. The project utilizes 1) Mozilla Hubs as the frontend and backend to host both the virtual space of the Pudong campus as well as personal memory pins in various forms, and 2) Spoke by Mozilla to create the 3D model of the campus itself. Given time constraints, this project focuses on objectively mapping the 8th floor of the Pudong campus and intends to create a subjective map of spaces related to IMA. The project will allow users to immerse themselves in the building, interact with other online users, and to drop down “pins of memory” at any location.

In a broader sense, this project will enable the current and future NYU Shanghai community to have an accessible space to document their individual attachments about the Pudong campus. Furthermore, it can also be a place for people to share their attachments, and thus enhance connections within the community.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
VR\\AR,Tool\\Service
Posted on

FAN-SEE: A collaborative website for fans

Fan-see is a crowdsourced video project which recreates a fans' version of the Fancy music video to show the creativity and engagement of K-pop fans.

Yiqi Chang

https://youtu.be/lcLzSzxvZCQ

Description

Fans can select any frame from the Fancy music video and redraw it in their ways. Then, fans are asked to upload their drawings to the website. All the frames will be pieced together to recreate a fans’ version of the music video. Fan-see’s goal is to present the K-pop fandom as a group of active audiences. K-pop fans engage with the K-pop industry and act as participatory consumers. They not only consume music but also create their artworks ranging from drawings to videos. Borrowing the frame-by-frame music video style from the Johnny Cash project that was created in memory of the American musician Johnny Cash, Fan-see tries to showcase transnational K-pop fans’ involvement in community and creativity.

IMA/IMB Shanghai
INTM-SHU.401.2
Capstone Studio (Shanghai)
Culture