We have all played and enjoyed games, but how do people actually design and develop them? How to describe a game from a professional standpoint? What are the basic elements and structure in video game development? How do game designers create an interactive experience for the player? What about prototyping and iterating in development? This course explores these questions and others through playing, analyzing and making games over 14 weeks. Students will understand game not only as an entertaining production and business model but a form of interactive media impacting current life and future. Students will be introduced to game design concepts, emphasizing the development: paper and digital prototyping, develop iteration, interactive narratives design and embedment, object-oriented programming, 2D/3D game art design, sound effects composition and user testing. For the course project, students will work in teams and create games in multiple projects, from board game focusing on gameplay prototype to digital playable experience with creative game art designs. This course leverages Unity, a game engine that uses C# based programming language. Basic knowledge of any programming language will come in handy. Prerequisite: Creative Coding Lab, Interaction Lab or equivalent programming experience. Fulfillment: IMA Major Electives; IMB Major Interactive Media Elective.
From the history of visual music and abstract film to the contemporary notion of live cinema, this course will be an exploration of the synesthetic relationship between sound and visuals in a real-time performance setting. Dating back as far as the 18th century, systems have been invented to produce images alongside music linking the two through formalized arrangements. Current media technologies make developing such systems both more approachable and more expansive in their scope. Through readings, viewings, and case studies students will gain an understanding of the history and theory of live audiovisuals. During the course students will team up to develop and master a real-time audiovisual system of their own invention. The class will culminate in a show in which they will present their work through a live performance.
With the rise of mobile computing platforms such as smart phones and tablets, location has suddenly become a key element in the production and consumption of media. In this online course, designed for NYU Shanghai Interactive Media Arts majors studying abroad, students will be encouraged to simultaneously explore their unique study away site, as well as to consume, research, critique, and create location-based media for mobile devices. Students will be introduced to GPS (Global Positioning System) technologies through activities such as geocaching and GPS drawing. We will next investigate geocoding, geotagging, and geofencing through the application of JavaScript mapping platforms CartoDB and Google Maps. Students will then explore an emerging technology known as Bluetooth Beacons, which can be used to create custom positioning systems and to facilitate location awareness in mobile devices. Students will be asked to then produce, as a final project, a game that engages participants in a location or locations, as well as in locative media in any number of forms. Note: This is an online course featuring both synchronous and asynchronous learning opportunities. Registration is limited to IMA Majors studying at NYU’s global sites other than New York or Abu Dhabi.
The class will teach how to architect and lead a virtual production by creating a dialogue between the Producer, Director, and Cinematographer in filmmaking with the Technical Producer and Director in creative technology. The class will cover an overview of all of the technical skills required to produce a remote virtual production through the lens of a project manager making administrative and creative decisions. This class will culminate in a real-time 3D project exploring motion capture and virtual production that will adapt a pre-existing cinematic work with the class themes in mind.
Today’s internet, made up of mostly text documents and two-dimensional images and videos, is the result of historical limitations in bandwidth, graphics processing and input devices. These limitations have made the internet a place where the mind goes, but the body cannot follow. Recent advances in motion capture devices, graphics processing, machine learning, bandwidth and browsers, however, are paving the way for the body to find its place online. This course will explore embodied interactions in the browser and across networks. Specifically, we’ll explore TensorFlow.js models like PoseNet and BodyPix, and Microsoft Kinect in p5.js and Three.js. Assignments will consider designing engaging embodied experiences for individual and social interactions online. Experience with Node, HTML and JavaScript is helpful but not required. ICM level programming experience is required. The course will have weekly assignments that explore embodied interaction online. Assignments will begin with exploring single points of interaction (i.e. one mouse or one joint), and progress to considering full bodies and multiple bodies in one browser. Students will have a 2-3-week final project with which they will delve more deeply into the subject matter in one piece of work. Students will have readings/watchings focused on embodied and networked user experience. Some influential works that will likely be assigned/discussed are Laurie Anderson’s “Habeas Corpus,” Todd Rose’s “The End of Average,” and Myron Krueger’s “Artificial Reality.” The course examples will be taught in Javascript using web technologies/frameworks. However, students are welcome to work in their preferred medium.
Juxtaposed to traditional comics, Experiential Comics combines emergent tech, unconventional comic book art/structure, and game engines to offer users a more immersive, continuous storyworld experience. Challenging the status quo of classic and contemporary digital comics, students will explore new technologies/world-building techniques better suited to craft innovative comic book narratives and formats –worthy of the Fourth Industrial Revolution. Students will ingest a brief history of classic and digital comics formats, collaborate with comic book artists to design engrossing characters, engage in world-building sessions, play with Unity/Unreal engines to generate avatars/ virtual environments, work with actors in motion capture/volumetric capture studios, learn the latest iteration of the Experiential Comics format, and share their unique expressions of Experiential Comics in a final presentation. Throughout a 7-week period, the course will be divided into 7 themes 1) The Disconnection of Digital Comics 2) Classic and Unconventional Comics Continuity 3) Marvel vs DC vs Insert Your Universe Here 4) Fourth Industrial Revolution Technologies 5) Capture & Creation 6) Infinite Engagement and Unlocking Immersive Format 7) Experiential Comics Presentations. Each weekly class will be divided into two halves 1) Exploration of Theme/Discussion 2) Process, Practices, & Play. This course requires CL: Hypercinema or equivalent experience.
This course is designed to provide an overview of visual storytelling in the newsroom. We will explore a variety of narrative formats and design principles, learn about reporting techniques for visual stories, touch on the best practices and ethics of journalism and work on collaborative exercises and assignments.
How can light, sound and design transform the human experience within a given space? How can psycho-geography be manipulated through audio-visual techniques? In what ways have and will technology allow spaces for sonic entertainment to be more immersive and experimental? Through an exploration of audio-visual techniques (i.e. VJing, MIDI-ing devices, sound synthesis, projection mapping, experiments with spatial sonic composition) along with discussions on how counterculture movements have used music and design as a vehicle for political dissent and community building, students will be invited to imagine new club spaces for social contexts beyond pure aesthetics. Assignments will include the development of different forms of interactive spaces for expression. Ableton (and free DAWs), MaxMSP, Isadora, and Unity will be used within this course.
Masks have been used around the world since antiquity for ceremonial and practical purposes, as devices for protection, disguise, entertainment and bodily transformation, made to be worn or displayed. Sociologist Erving Goffman wrote about the everyday life as a masked theatrical performance. The performative aspect of our lives today is ever so present in our use of social media, where we present a curated version ourselves for the immediate visual consumption of others. In our `Selfies`, we can assume a multitude of identities and characters. Recent tools and platforms have evolved social media portraiture to an art form and have created new opportunities for artists to create and distribute interactive augmentations, forming new relationships between artists and viewers. This class explores the developing language of social media portraiture enhanced by Augmented Reality. Students will: – review masks in art history, leading up to today – ideate, design and develop an interactive mask (AKA effects/lenses/filters) – learn to use the Meta Spark software to create AR effects. This course requires CL: Hypercinema or equivalent experience.
Time-based art, performance – and theater most specifically – should be perfect manipulators of experience. Many creators of time-based art look for the “universal” in content and overlook what we all have in common in form: brains. What happens in the minds of all truly happens (what happens in the lobby also truly happens). How can we use art to make our brains experience the same things? What behind-the-scenes work can we employ to manipulate experience. Film scoring works on us in ways we don’t perceive in the moment. Can we pay closer attention to this when making work? Sometimes what we have for lunch effects our experience of art more than the hours of labor that went into the most minuscule of decisions when making that art. Is empathy always ethical? Show don’t tell. How does the art take place in the room in front of us? How can public performance not be awkward? How do we experience a performance rather than watch it.
he virtual expansion of screens began during the 1960’s with the exploration of head-mounted displays. Since the 60’s, virtual reality has been explored in a multi-disciplinary context including philosophy, design, arts, behavioral therapy. Baudrillard, with his publication of Simulacra and Simulation (1981), declared that human experience is being replaced by a simulation of reality (HyperReality). His theories brought the dystopian narrative of the virtual to mainstream pop-culture, as seen in films such as The Lawnmower Man and The Matrix . Contrary to Baudrillard, Canadian VR Pioneer Char Davies brings a more positive perspective to Virtual Reality, “facilitating a temporary release from our haitial perceptions and culturally biased assumptions about being in the world, to enable us, however momentarily, to perceive ourselves and the world us freshly.” Throughout the class, the friction between Baudrillard and Davies will create the foundation of our exploration of Virtual Reality, where we will use room scale headsets and game engines to create meaningful “temporal experiences” exploring themes from behavioral sciences to narrative storytelling. We will be exploring ● existing VR projects, popular culture references and theory. ● concepts such as sense of embodiment (SoE), social VR design, and interactive storytelling techniques. ● methods for designing, modeling and rigging avatars for VR. ● live and pre-recorded animation. ● spatial audio techniques such as ambisonic sounds engines. ● packaging and distributing applications for social VR. This is a production class, along with a theoretical foundation, in which we will prototype projects with networking, inverse kinematics, raycasting and face tracking technologies to explore questions such as “how does the viewer become part of the experience?” and “how does the real space relate to the virtual worlds we design?” In the second half of the class, students will work in groups to build a final social VR project based on their exploration of the above framework.
As game and interaction designers we create systems and choices that can either prey upon our psychological foibles or help us avoid decision pitfalls. It is our responsibility to understand how we decide, to consider the ethics of the systems we create and to practice designing systems in a purposeful manner. Game Design & The Psychology of Choice will provide interaction and game designers with an understanding of the factors that influence behavior and decision-making by looking at the intertwining of cognitive psychology and economics through the development of behavioral economics. These disciplines study behavior on the individual and group level, often revealing some of the why behind the rules of thumb and folk wisdom that game designers come to intuitively. But understanding the why—why we fall into decision traps; why certain tradeoffs tax our brain more than others; why we are overconfident about our abilities; why certain decisions make us uncomfortable—allows us to more purposefully apply our design craft, both in and out of games. Finally, as a class, we will take what we learn about how we think and create series of game experiences based around key cognitive science concepts. Assignments may include: •Mod a cognitive science experiment into a game or experience •Analyze and present a game through the lens of cognitive science and behavioral economics •Create game or experience based around a particular insight from cognitive science or behavioral economics
Whether it’s through photo realistic scans found in current-gen video games or the cartoonish and low-fi aesthetic of Bitmoji there is no limit to ways in which the body and the self are represented in digital spaces. This 2 credit class will look at how avatars have been historically used in the realm of art, commerce, and entertainment and utilize existing avatar creation tools to develop projects that examine identity, body politics, and contemporary performance. In class we will cover the basics of Unreal Engine, photogrammetry, 3D scanning, and model rigging although students will be encouraged to use existing skill sets and creative thinking to complete some of the smaller week-by-week assignments. The class will culminate with a short performance, small installation or single/multi-channel video piece using one or more of the techniques covered in class. This can be a solo project or a group project. In this class students will: – Explore how avatars can be utilized in your creative practice – Gain an introductory understanding of Unreal Engine, photogrammetry, model rigging, and 3D scanning. – Learn how to recontextualize digital spaces for the purposes of art, installation, and performance. – Broaden your thinking of what performance can be, both in a physical setting and digital setting. – Think critically about how physical bodies inhabit digital spaces and how the hardware and software we use reinforces the acceptance and value of certain kinds of bodies.
“Lies. Hoaxes. Conspiracies. Rumors. Propaganda. Fake news is an age-old phenomenon—but the internet is making targeted misinformation cheap and scalable. That is affecting politics, public opinion, and the everyday experience of the internet. In this 6-week class, we will explore the cutting edge of “fake news” by engaging in ethical research and fabrication. Participants will manufacture and observe a controlled “fake news” event. We will experiment with command-line tools for doctoring video, neural nets and deepfakes to fabricate reality, Twitter bots, behavioral psychology, and the dark underbelly of the ad economy.”
Until recently 3D sound was a novelty reserved for special uses and reaching a limited audience, no medium in popular culture has been as inherently dependent upon spatial audio as virtual reality. The widespread and standardized implementation of surround sound in film brought cinema to a new level of immersion, but is limited to theatrical exhibition and home theater systems. Today a considerable amount of content is consumed on mobile devices and laptops which excludes the cinematic experience of spatial sound. With the current rise of cinematic VR and the blurring line between gaming and experiential VR, spatial audio is no longer just an added bonus, but rather a necessity in designing immersive VR experiences. In this course we will explore the emerging field of 3D sound design and for both 360 video and game engine-built VR using a digital audio workstation, Unity, and 3D audio plugins.
This course investigates aesthetic and technical aspects of sound for video games and interactive 3-D environments. Artistic implications of the technology are also explored from the perspective of the electronic composer and performer. Students will work with a game engine to create an immersive interactive environment. Additional topics include: Csound, Java and other relevant technologies. Completion of a final project, class presentation, as well as several weekly assignments is required.
Many of us have played and enjoyed games. Have you ever wondered how people actually design and develop them? Can a games as a profession and practice be described? What are the basic theories, structures and skills involved in game development? How do game designers create compelling interactive experiences for the player? How do they respond to feedback, prototype and improve these experiences? This course explores these questions and others through playing, analyzing and making games over 14 weeks. Students will understand games not only as entertainment, production and business models, but as a form of mass interactive media and culture. Students will be introduced to game design concepts, emphasizing all stages of game development: paper and digital prototyping, iteration, interactive narrative design, object-oriented game programming, 3D/2D game art creation, sound effects composition and user testing. For the course project, students will work in teams and create games in multiple projects, from simple board games to a digital game with original game art, mechanics and other design elements. This course leverages multiple tools for game prototyping and design, but will focus on Unity, a game engine that uses C# based programming language, for game build-up.
We will explore a range of programming languages, systems, and activities designed to help learners acquire computational skills and become creative problem solvers and project designers, including arts and interactive projects. We will create projects in turtle geometry, animation, and programmable embroidery (Snap!, Turtle Geometry, TurtleArt, and TurtleStitch), and in simulation systems which model complex systems in the life and social sciences in order to acquire a deeper understanding of their underlying phenomena (NetLogo). The course is fundamentally about ideas, and how some powerful ideas from computation can empower a learner to be a better creator and problem solver, acquire a deeper understanding of social and scientific phenomena, and become a self-directed learner. We will identify these ideas and actively engage with the pedagogical theories that underlie embodying them by creating with systems designed for children, beginners, or people coming from disciplines which traditionally had less emphasis on computing-based tools. We will emphasize reflection on our own learning within the course.”
In this course, students will work in teams to design digital learning resources and experience designs at the intersection of music, coding, arts, and technology. The course will begin with an introduction to emerging trends in learner engagement and design-based research, especially related to web- and mobile-based musical experiences and principles of making music with new media. Innovations in and applications of musical creativity, interactive technologies, user-centered design & engagement, scaffolded learning, creative learning, pedagogies of play and making, and educational entrepreneurship in Chinese contexts will also be explored. The market for creative educational experiences in afterschool settings for youth in China is exploding. For-profit educational service companies are competing and searching for differentiated, learning experiences in music, coding, and creative project based learning that will attract high-paying parents looking for the best supplemental education for their children. This practical, hands-on course will explore questions such as: How can we design engaging, creative learning experiences that are relevant to the cultural goals and needs of today’s youth in China, while laying the foundation for creative learning for the workforce of tomorrow? What are engaging, effective creative learning resources, and how are they best implemented in Chinese learning settings? How can we take advantage of young people’s near ubiquitous love of music and technology to facilitate creative learning? Students will work together in teams and paired with a partner audience of learners and teachers in Shanghai drawn from local and regional international schools (e.g. Alibaba’s Cloud Valley), local afterschool programs (e.g., Music Lab), and cultural partners (e.g., Shanghai Symphony). Together they will assess the needs and opportunities of partner students and teachers, and engage in a two-stage iterative and reflective co-design process prototyping custom learning resources and experience designs with their partner end users. At the end of the course, students will present and demo their learning resources as part of a public showcase to an external audience of partners, educators, technologists, musicians, entrepreneurs, and experience designers in Shanghai. Prerequisites: None.