Libraries like TensorFlow.js and ml5.js unlocked new opportunities for interactive machine learning projects in the browser. The goal of this class is to learn and understand common machine learning techniques and apply them to generate creative outputs in the browser. This class will start with running models in the browser using high-level APIs from ml5.js, as well as explore the Layer APIs from TensorFlow.js to train models using custom data. This class will also cover preparing the dataset for training models. At the completion of this course, students will have a better understanding of a few machine learning models, how do they work, how to train these models, and their use case to creative projects. Students will also be able to create interactive ML web applications with pre-trained models or their own models. Prospective students are expected to have taken an ICM (Introduction to Computational Media) course, or have an equivalent programming experience with JavaScript, HTML, CSS.
Generative machine learning models open new possibilities for creating images, videos, and text. This class explores the idea of how artists, designers and creators can use machine learning in their own design process. The goal of this class is to learn and understand some common machine learning techniques and use them to generate creative outputs. Students will learn to use pre-trained models, and train their own models in the cloud using Runway. For each week, we will discuss the history, theory, datasets, application of the machine learning models, and build experiments based on the model. In addition to Runway, we will be using JavaScript libraries like the p5.js, ml5.js, and TensorFlow.js, and software like Photoshop, Unity and Figma. Students are expected to have taken ICM (Introduction to Computational Media), or have equivalent programming experience with Python or JavaScript. A list of ML models we will be covering: Image generation: StylanGAN: https://github.com/NVlabs/stylegan BigGAN: https://github.com/ajbrock/BigGAN-PyTorch Style Transfer Fast-style-transfer: https://github.com/lengstrom/fast-style-transfer Arbitrary-Image-Stylization: https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization Semantic Image Segmentation/Synthesis Deeplab: https://github.com/tensorflow/models/tree/master/research/deeplab Sapde-coco: https://github.com/NVlabs/SPADE Image-to-Image Translation: pix2pix: https://phillipi.github.io/pix2pix/ pix2pixHD: https://github.com/NVIDIA/pix2pixHD Text Generation LSTM gpt-2: https://github.com/openai/gpt-2
With Machine Learning models are getting smaller, and microcontrollers are getting more computing power, Machine Learning is moving towards edge devices. This class explores the idea of how machine learning algorithms can be used on microcontrollers along with sensor data to build Physical Computing projects. In this class, we will learn about TensorFlow Lite, a library that allows you to run machine learning algorithms on microcontrollers. We will talk about common machine learning algorithms and techniques and apply them to build hands-on interactive projects that enrich our daily lives. Students will learn to use pre-trained models, and re-train the models with sensor data. We are going to talk about Image Classification, Transfer Learning, Gesture and Speech Detection. For each topic, we will first discuss its history, theory, datasets, and applications, and then build simple experiments based on the topic. Prospective students are expected to have taken Introduction to Physical Computing and Introduction to Computational Media course, or have equivalent programming experience with Arduino and JavaScript.
A site for IMA NY Students to find equivalent courses outside of IMA NY
For most students joining IMA in Fall 2022 and beyond, there is a new program structure that affects the categorization of courses on this site:
Any class in any IMA major elective category (ie "Art & Design") refers to the IMA program structure previous to those entering in Fall 2022. If you are in the class of 2026 (most entering Fall 2022 or later), any course in an IMA elective category are generic IMA electives in the new structure.