Computation and Data (Old Structure)

Topics in Computation and Data: Mobile Application Development +

John Henry ThompsonSyllabus | IMNY-UT.220 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024

One of the most transformative consumer products in history, the iPhone remains the standard bearer for great design and user experience. With the latest versions of iOS and iPhone, Apple puts depth sensing and augmented reality in our pockets. How do we take advantage of this incredible platform to produce our own compelling experiences?

This course will be a hands-on workshop where we explore the world beyond generic apps and push the boundaries of what’s possible on iOS hardware. Each week, you’ll be asked to complete a programming exercise meant to foster your understanding of iOS application development. We’ll leverage existing open source libraries to quickly build out your app with features such as real time communication and cloud storage.

We aim to create distributed instruments for computed expression.

Full-time access to an iOS device and a Mac laptop computer running the latest operating system and development tools are required.

Prereq: Some programming experience (such as ICM) and willingness to learn Apple’s Swift programming language.

Prerequisite: Creative Computing (IMNY-UT 101)

Real-Time Media +

Carrie Sijia WangSyllabus | IMNY-UT.285 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Art and Design (Old Structure), Computation and Data (Old Structure), Elective, Fall 2024, Media and Entertainment (Old Structure)

This course focuses on designing, developing and delivering real-time, performative work using audio and video elements. The class will have an emphasis on using MaxMSPJitter and other tools to create performative experiences that dynamically combine interactive elements such as video, sound, and code, allow for the unfolding of engaging narratives, and generate compelling visuals in real time.

We will look at various examples of both multimedia performances and installations, explore how we can apply the technologies we have learned to design real-time systems, and discuss methods we can use to make our work more engaging.

The class is three-fold and divided into tech tutorials, discussions of existing examples, and in-class performances.

Introduction to Machine Learning for the Arts +

Daniel ShiffmanIMNY-UT.224 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024, Programming and Data (New Structure)

Prerequisite: Creative Computing (IMNY-UT 101) OR equivalent coursework.

An introductory course designed to provide students with hands-on experience developing creative coding projects with machine learning. The history, theory, and application of machine learning algorithms and related datasets are explored in a laboratory context of experimentation and discussion. Examples and exercises will be demonstrated in JavaScript using the p5.js, ml5.js, and TensorFlow.js libraries. In addition, students will learn to work with open source pre-trained models in the cloud using Runway. Principles of data collection and ethics are introduced. Weekly assignments, team and independent projects, and project reports are required.

Front-End Web +

Lenin CompresIMNY-UT.228 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024

This course will provide a foundation for understanding modern web development with a focus on front end technologies and accessing public data. The forms and uses of these technologies are explored in a laboratory context of experimentation and discussion. This studio stresses interactivity, usability, and the quality and appropriateness of look and feel.
Students will create two web applications, including one that leverages public APIs and Javascript libraries. The goal of the course is for students to learn how to think holistically about an application, both by designing a clear user experience and understanding the algorithmic steps required to build it. Assignments are arranged in sequence to enable the production of a website of high quality in design and engineering.

Topics in Media Arts: AI for Creatives +

Tyler PeppelSyllabus | IMNY-UT.281 | Last updated: March 12, 2024 2 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024

A survey and hands-on workshop on AI augmentation of creative communication.

We will get hands-on with AI tools for photography, design, art, writing, UX, illustration, and video.
We will produce, critique, and exhibit creative work that tests the creative capabilities of these new AI tools.

Questions we will consider:
How does AI influence the creative process?
How does this new human/machine collaboration challenge my role as a creator?
Who is creating these new tools and how do they work?
What is the longer-term impact of AI technologies on the creative professions?
Our goals are to understand AI technology, use it to produce creative work, and use what we learn to influence the larger conversation about creativity and AI.

Will AI diminish or enhance human creativity? It’s up to us.

The Code of Music +

Luisa PereiraSyllabus | IMNY-UT.222 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024, In-Person, Programming and Data (New Structure)

In this course, students learn how to create musical systems –pieces that incorporate randomness, interact with their listeners, or evolve over time, in the browser.

We will start by creating audiovisual instruments and sample-based interactive songs, as students review their p5.js skills and are introduced to the Tone.js music library. Then, we will turn to a structured exploration of the elements of music, focusing on rhythm, melody, timbre, and harmony. For each, we will hold listening sessions, represent and manipulate the element in code, and interact with it via a range of existing interfaces. Students will explore the possibilities that computation and interactivity open up by designing and implementing a series of interactive studies.

The last few weeks of the semester will be dedicated to introducing algorithmic composition techniques such as Markov Chains and Neural Networks. During this time, students will also develop their final project: an interactive/generative musical piece that builds on their previous classwork.

Throughout the course, students are encouraged to bring in their musical tastes and interests into the classroom. This class is a good fit for students who are interested in:
– Creating interactive music pieces and digital instruments.
– Deepening their understanding of how music works. All musically-curious students are welcome: previous experience with music and audio will be useful, but is not required.
– Continuing to develop coding skills. Creative Coding or equivalent programming experience is required.

About Luisa Hors: https://www.luisapereira.net/

Prerequisite: Creative Computing (IMNY-UT 101)

Networked Media +

Sam HeckleIMNY-UT.223 | Last updated: March 12, 2024 4 Point, 9.75 Ready to Schedule (Do not edit) YES, All, Computation and Data (Old Structure), Elective, Fall 2024, In-Person, Programming and Data (New Structure)

The network is a fundamental medium for interactivity. It makes possible our interaction with machines, data, and, most importantly, other people. Though the base interaction it supports is simple, a client sends a request to a server, which replies; an incredible variety of systems can be and have been built on top of it. An equally impressive body of media theory has also arisen around its use.

This hybrid theory and technology course will be 50% project driven technical work and 50% theory and discussion. The technical work will utilize JavaScript as both a client and server side programming language to build creative systems on the web. Technical topics will include server and client web frameworks, such as Express, HTML, CSS, templating, and databases. The theory portion of the course will include reading and discussion of past and current media theory texts that relate to the networks of today.

**** it is HIGHLY recommended you take Front End Web Development (or have equivalent front end web development experience) to get the most out of this course. We will be going over fundamentals of HTML/CSS but it would be useful to have prior knowledge ***

Topics in Media Art: Generative Art with the Unity Game Engine +

Dave SteinSyllabus | IMNY-UT.281 | Last updated: October 20, 2023 2 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, Media and Entertainment (Old Structure), New, Spring 2024

This course will provide an overview of important topics of generative art. On a weekly basis we will cover a new topic, review examples of work within this topic and discuss their influence in generative artworks as well as in a broader art context.

In addition, we cover the fundamental concepts of the C# programming language and its application within the Unity game engine. C# is a widely used, very fast and efficient programming language and can perform significantly faster than P5 and Processing. As such, creating generative art projects using Unity and C# will make our projects faster with higher definition and larger detail than a typical Javascript sketch.

This course is designed for students who want to continue their creative coding practice and are interested in more advanced coding techniques while building their knowledge of C# and Unity. Students should have a solid understanding of programming concepts such as arrays, classes and objects and be comfortable with creative coding (such as with P5).

Topics in Media Art: Shared Minds +

Dan O'SullivanSyllabus | IMNY-UT.260 | Last updated: October 20, 2023 4 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, New, Spring 2024

What capabilities does computational media have for depicting and conveying the experience of our minds? In using the new possibilities of machine learning networks to create media, what should we take or leave from cinema, social media and virtual reality?

In this course we will start out by turning inward to reflect on how our mind transcends time and space and how artificial neural networks might better capture the multidimensional space of our thought. We then turn to using cloud networking and databases to share our thinking with other people across time and space. Finally we need to flatten everything back into 4D interfaces that, while being stuck in time and space, can reach our embodied, emotional and experiential ways of understanding of the world.

The class will operate at a conceptual level, inviting students’ empirical, psychological and philosophical investigations of the nature of their experience and how to convey it with art and story. It will ask students to look critically at existing computational media’s tendencies to bore, misinform, divide or inflame its users.

But this is also very much a coding class where students will prototype their own ideas for new forms of media first with machine learning models like Stable Diffusion using Huggingface APIs or Colab notebooks, and then with networking and databases using Firebase or P5 Live Media, and finally with 3D graphics using the threejs library. Students can substitute other coding tools but game engines will not work for this class. The coding is in javascript, with touches of python, and is a natural sequel to Creative Computing.

Topics in Media Art: Computational Image Deconstruction +

Alan WinslowIMNY-UT.260 | Last updated: October 20, 2023 4 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, Spring 2024

This class explores how as creatives, we can take the wealth of data that each still image contains and re-purpose it. In the first few weeks of the course, students will develop an understanding of technical and creative photographic techniques through lectures, hands-on assignments, and critiques. As the class progresses, students will develop a series on a particular topic of interest (portraits, architecture, street photography). Using p5js we will explore simple scripts to extract information or manipulate the images (what are the most represented colors in the photos? What are the RGB values that make up the image? Can we add movement to the picture?). At the end of the course, students will present their series.

Prerequisite: Creative Computing
About Alan Winslow: www.alanwinslow.com

Topics in Computation and Data: Nature of Code +

Lenin CompresSyllabus | IMNY-UT.220 | Last updated: October 20, 2023 4 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, New, Programming and Data (New Structure), Spring 2024

Can we capture the unpredictable evolutionary and emergent properties of nature in software? Can understanding the mathematical principles behind our physical world help us to create digital worlds? This class focuses on the programming strategies and techniques behind computer simulations of natural systems. We explore topics ranging from basic mathematics and physics concepts to more advanced simulations of complex systems. Subjects covered include physics simulation, trigonometry, self-organization, genetic algorithms, and neural networks. Examples are demonstrated in JavaScript using p5.js.

Prerequisites: Creative Computing

Instructor Daniel Shiffman Website: https://natureofcode.com/

Experimental Photography +

Ellen NicklesIMNY-UT.232 | Last updated: October 20, 2023 4 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, In-Person, Spring 2024

What are all the ways that you saw or made a photograph this week? How are those ways similar and different? How do those pictures function in your life and in society? What is a photograph? This course repeatedly asks these questions by using emerging computational tools to design alternative forms of making and interacting with photographs. The forms and applications of these tools, such as those for creative coding, physical computing, and machine learning, are explored weekly in technical tutorials and hands-on workshops. These are informed by discussions of critical debates in photography and various practitioners working with photographs, past and present. The homework includes readings, short writing responses, and photography assignments. Prerequisites: Comm Lab: HyperCinema (or similar coursework exploring communication and storytelling with digital tools) and New York’s IMA Creative Computing (or similar coursework with creative coding using the p5.js JavaScript library and programming for physical computing using Arduino microcontrollers). Note that prior experience with physical computing using the Arduino platform is required for this course. Please feel free to contact the instructor if you have any questions about the course.

Chatbots for Art’s Sake +

Carrie Sijia WangIMNY-UT.233 | Last updated: October 20, 2023 4 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, In-Person, Spring 2024

This class aims to repurpose existing chatbot technologies and use them for the sake of art. The class is twofold, students will engage in labs and workshops to learn and practice different techniques— such as p5.js, p5.speech, RiveScript, RiTa, and Alexa Skills—to create functional chatbots. They will also participate in lectures and discussions that look at the different roles Artificial Intelligence plays in human society, including but not limited to authority, companions, or simply reflections of the humans it interacts with.

Topics in Media Art: Three.js for Makers +

Brian HoSyllabus | IMNY-UT.281 | Last updated: October 20, 2023 2 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, New, Spring 2024

“In this increasingly online world, the internet has proven to be a powerful tool that can connect us with one another, host meaningful experiences, and provoke critical thinking. In this class, students will have an opportunity to learn about breaking out of the 2D web page and the fundamentals of working with 3D on the web.

This course hopes to introduce new avenues for creative expression and experimentation via the web and promote learning practical web development skills through experiential learning. Students will use Three.js to create dynamic and immersive web-based experiences that push the boundaries of what is possible online.

The course is intended for technologists who have no programming or computer science background but are interested in 3D exploration on the web. Nothing more than a basic understanding and familiarity with CSS, HTML, and Javascript is required.”

Topics in Media Art: Interactive Multi-Screen Experiences +

John Henry ThompsonIMNY-UT.281 | Last updated: October 20, 2023 2 Point, 9.9 YES, Computation and Data (Old Structure), Elective, IMA Major Elective, New, Spring 2024

We experience screens daily in many forms: in our hands, on our desktops, on walls and public installations as we travel. This course will explore the creative possibilities of real-time interactive and reactive art on screens in various forms. Using the recently developed p5VideoKit we will create standalone installations. p5VideoKit is a new library of live video effects – building on p5js – presented as a dashboard for mixing video in the browser. This library allows the user to apply visual effects to live video from connected cameras and sensors or streaming from devices on the internet. p5VideoKit is open source and can be extended with the user’s p5js code for a plethora of visual effects and interactivity. One possible application of p5Videokit would be a public facing installation allowing anonymous people on the street to use their hand held devices to interact with large street facing screens, thereby collaborating on real time creation of “digital graffiti”.

Building on ICM, students will learn how to adapt simple sketches into components of p5VideoKit so that algorithms can be quickly composited and orchestrated into more complex works. Students will also learn how to edit and share code beyond the p5js editor, use nodejs/javascript to automate deployment of installations, and remotely configure dedicated computers with long running installations. Several dedicated computers and screens will be available to preview installations on the floor and street facing areas of the 370 Jay Street campus.

Prerequisites: ICM or equivalent coding experience.

About John Henry Thompson: http://johnhenrythompson.com