NIME 2022

Syllabus Fall 2022

In this course students design and build digital musical instruments and ultimately create a live performance using them.

Over the semester, we look at examples of current work by creators of musical interfaces, and discuss a wide range of issues facing technology-enabled sound performance, such as affordances, mapping, links to acoustic instruments, site specificity, gestuality. Readings and case studies provide background for class discussions on the theory and practice of designing musical controllers and the process of integrating them in a musical performance, with attention to the idiosyncrasies that emerge from the figure of the inventor / composer / performer.

Students design and prototype a musical instrument – a complete system encompassing musical controller, algorithm for mapping input to sound, and the sound output itself. A technical framework for prototyping performance controllers is made available. As they prepare their prototypes for a final live concert where they (or guest musicians) will perform with their instruments,  students focus on musical composition and improvisation techniques.

Prerequisites: ITPG-GT.2233 (Introduction to Computational Media) and ITPG-GT.2301 (Physical Computing).

09/12. Session 01. Introductions. A brief history of musical expression. NIMEs.
09/19. Session 02. Instrument 1 performance. Inspiration NIMEs. Conceptual + Technical Framework.   
09/26. Session 03. Instrument 2 demo + technical discussion. MIDI vs OSC.
10/03. Session 04. Instrument 3 demo. Elements of sound. Percussive and pitched sounds. Sampling and synthesis.
10/11*. Session 05. First Garage performances, Group A. *note: this is a Tuesday
10/17. Session 06. First Garage performances, Group B.
10/24. Session 07. Artist talk: Johann Diedrick . Guidelines for concept presentations.
10/31. Session 08. Concept presentations with guest critics: Rena Anakwe, Sarah Ibrahim, Helen Leigh.
11/07. Session 09. NIME principles, process, and evaluation criteria.
11/14. Session 10. Updates, Sound/music space resources, Artist talk: Andy Cavatorta
11/21. Session 11. Updates, part 2. Methods for performing. Performance logistics.
11/28. Session 12. Rehearsal (Garage).
12/05. Session 13.  Dress Rehearsal (Garage).
12/10 and 12/11. NIME SHOW at The Garage at 370 Jay St Garage, 6pm.
* Please keep Saturday and Sunday afternoons open for sound+light check, set up and tear down.
12/12. Session 14. Show review.

Sessions

This section will be updated weekly with class notes, assignment details and resources.

09/13. Session 1.

  • Introductions
  • Syllabus overview
  • A very brief history of musical expression
  • What is NIME. Acoustic vs Digital Instruments
  • Activity: one gesture => one sound

Assignments

  • Create a musical instrument and come ready to perform it for 30s – 1min.
  • Read:
  • If you are not familiar with Ableton Live, borrow a production laptop from the ER and do these tutorials under Help/ Help View:
    • A Tour of Live
    • Recording Audio
    • Creating Beats
    • Playing Software Instruments

09/13. Session 2.

  • First Instrument Performances
  • Discussion: what is an instrument?
  • Inspiration NIMEs
  • Conceptual framework (see image under Resources)
  • Technical framework
    • Analysis of student instruments
    • Demo: Sensors => Arduino (controller) ==MIDI==> musical parameters in Ableton

Assignment. Instrument 2: sensors > MIDI > Ableton

Create an instrument with the following components: 

1. A gestural interface with at least two interface controls: one with on/off states (like a push button), one with a range of values (like a potentiometer). Your interface can either be physical (sensors connected to Arduino), or digital (for example, skeleton tracking connected to p5).

2. A controller that reads the control values, maps / thresholds / smooths them if needed, and sends out MIDI messages (note on, note off, CC). The controller can be an Arduino sketch or a p5.js sketch.

3. A sound generator (Ableton Live) that receives the MIDI messages, and plays back, synthesizes, or modulates sounds in response.

Be intentional in picking your controls: consider making your own switches out of any conductive material (rather than a standard push button); consider using sensors like flex, stretch, vibration, accelerometer, gyroscope (rather than a standard potentiometer). What gestures and form factors might each control afford?

Resources

Source: Wessel & Wright, NIME – 01 (highlighting is my own)

09/26. Session 3.

  • Instrument 2 demos and discussion: troubleshooting, sensors, mapping to instrument parameters, alternative connection methods like Firmata + the Arduino device in the Live Connection Kit.
  • Communication protocols: limitations of MIDI; OSC as an alternative.

Assignment: OSC hello world + Instrument 3

  1. Implement an OSC ‘hello world’ using one of the methods under Resources below. You can do this at the Wednesday workshop, or on your own.
  2. Choose one of the following prompts:
    • Iterate on Instrument 2, refining the components you paid less attention to this week. For example:
      • Focus on the controller: Do you need to smooth your sensor’s input? Average it? If you are using thresholds, what are they? Do they need calibrating?
      • Focus on the sound generation:
        • What ranges do the parameters you are controlling currently vary within? Try different ranges. Try inverting them. Try controlling different parameters.
        • Try different instrument presets, find two presets that contrast with each other, interpolate between them.
        • Using the same interface, try controlling wildly different instruments.
    • Create a new instrument, with a different interface.
    • Reimplement your instrument: this time, use OSC instead of MIDI.

Resources

  •  Introduction to OSC
  • Most creative coding frameworks have OSC libraries:
  • To control Ableton Live via OSC, you will need an intermediary between the system sending OSC messages and Live.
    • If the sender is your own sketch/patch/app, you can:
      1. Make your own Connection device using Max for Live. In this case, the OSC Monitor included in the Live Connection Kit can be a useful troubleshooting tool.
      2. Make an intermediary app using any of the libraries above. The app receives the OSC messages from the controller, and converts them to MIDI messages that are sent to Live and mapped using MIDI Learn.
    • If the sender is a standard hardware/software controller like LeapMotion or TouchOSC, you can use the equivalent Live Connection Kit devices to receive their messages, and the process will be very similar to using the MIDI Learn feature you used last week.
    • A standard hardware/software controller like TouchOSC or LeapMotion, which the Live Connection Kit has devices for. You could also follow their protocol specs as you construct the messages in your own device.
    • Note: the Connection Kit also has an OSC Send device which receives MIDI notes and sends OSC messages. This means you can send OSC messages OUT of Live and into other systems (for example, your own sketch/patch/app).

09/26. Session 4.

  • Instrument 3 demos
  • Percussive and pitched instruments
  • Elements of Sound: pitch, volume, timbre
  • Timbre: Sampling and Synthesis. Technique: Additive Synthesis.

Resources

  • Sampling: free sample sources
    • Radio Aporee – a corpus of high quality sounds from around the world, navigable in a number of ways, including a map
    • Freesound – a collaborative database of sounds
    • Macaulay Library – a wildlife media archive
    • Philharmonia – a collection of sound samples recorded by the London Philharmonic Orchestra
    • Ableton Sample packs – a variety of free sample packs from Ableton
    • Spitfire Audio Labs – a series of free software instruments to use with any DAW
  • Intro to Synthesis Max/MSP patches from class

Assignment: First Garage Performance
Pick one of the instrument experiments you’ve done so far. Either next class or the week after, you will do a mock performance with it on the Performance Garage stage, for the rest of the class. Time is tight, so please arrive 10 minutes early if you can. To prepare:

  • Do a couple of improvisation sessions to explore the expressive possibilities of your instrument. What kinds of gestures does it afford? What different sound spaces can you get into?
  • Plan for a 3 – 5 minute performance and rehearse it. Where do you start? Where does that lead? How do you end? Is there enough to explore in the instrument to sustain your interest for that long? Keep in mind: if you are engaged in playing the instrument, chances are the audience will be, too.
  • Consider the audience’s point of view. Will they be able to understand what you are doing, and how it affects the music? Do you need a detail camera? Anything else?

The purpose of this early performance is for everybody to get a chance to:

  • Break the ice and perform on stage.
  • Get acquainted with the Garage space and its equipment
  • Get a sense of what the flow of the final performance will be
  • To do the above ahead of conceptualizing the instrument+performance you will work on during the second half of the semester.

We will have a setup ready for you to connect to the following, from the stage:

  • Stereo audio. Connect your audio output via an 1/8” audio cable (we’ll have an aux cable ready, like in class), or via XLR / 1/4” cables, which you’ll bring. These will go to a Scarlett audio interface on stage.
  • Projector and large screen (behind the stage). Connect your USB-C or HDMI video output (we’ll have an HDMI cable ready, with an USB-C dongle)
  • Detail camera. If it makes sense for your project, you can project a closer view of your instrument using a detail camera mounted on the stage table. You can connect the camera:
    • directly to the projector, or
    • get the camera stream into your computer via USB-C or HDMI ports, and choose how to project it (side-by-side with your p5 sketch? your Max patch? your own video content? or, processed in some way?

While you are not performing, you will be writing notes to your peers, with this feedback sheet as a guide.

Session 5. First Garage Performances, Group A

Session 6. First Garage Performances, Group B

  • More NIME inspiration:
    • augmented instruments
    • participatory instruments
    • tangible scores
    • analog electronics

Assignment: Start defining your NIME project

  1. Review the work we have done so far: your instrument explorations, your demos and performances (including the last one, in the Garage), the feedback from the class; the NIME conceptual framework, the inspiration projects, our discussions on the elements of sound and technical implementation. Reflect:
  • What has caught your interest and imagination? This may be an instrument’s sound, its form, material or the gestures involved, the affect or mood of a performance, a theoretical concept, a sound generating or manipulating technique, a possibility that is opened up by technology, anything else.
  • What was engaging, effective, intriguing in your experiments?
  • What would you like to develop or explore further?

2. Start defining the concept for your final – the instrument you will develop through the rest of the semester, and perform with at the end. Remember: collaborations are encouraged.

In Session 8 (in two weeks), you will present your concept to guests, who will offer guidance and feedback. You can find guidelines on what to include in your presentation in the Session 7 assignment section below -these might help clarify what exactly I mean by concept, too.

3. Visit the websites of our guest speaker for next week, pick a couple of pieces that seem interesting, look at them more closely. Bring questions to ask them during the Q&A.

Session 7. Artist talks

  • 12:10. Johann Diedrick – Johann is an artist, engineer and musician that makes installations, performances and sculptures for encountering the world through our ears.

Assignment: Prepare your concept presentation / project proposal

Format

You will each have 4 minutes to present, followed by 3 minutes of feedback from our guests (if there are collaborations, we may have a bit more time). Time will be tight; to make sure everyone gets their allotted time to present:
– Practice your presentation, timing yourself.
– Before class, add your slides to this folder, so that everybody presents from the same computer and we save some swapping time.

Guidelines

Here are some template slides with guidelines on what to include in your presentation – note that there are comments in the presenter view.

You can think of your presentation as a project proposal: guests will be aware that you are close to the beginning of the creative process. The goal is to communicate your ideas in a clear and concise way –if our guests understand what you are imagining, they will be able to offer richer, more specific feedback. If they are confused, they might spend their time asking more general questions.

Session 8. Concept presentations

Guests: Rena Anakwe, Sarah Ibrahim, Helen Leigh

Session 9. NIME Principles, Process, Evaluation Criteria.

  • Aesthetic and Design considerations:
    • NIME performance challenges.
    • Approaches for making mapping transparent: visual feedback, interaction metaphor, simple and direct interfaces.
  • NIME principles and process
  • Activity: NIME evaluation

Resources

Assignments

Session 10. Updates, Sound/music space resources, Artist talk

  • Collective success criteria wrap-up – a compilation of the group’s collective criteria is here.
  • Show production updates + poll.
  • Check-ins + feedback, part I
  • Resources for refining sound/music spaces – note that there are additional comments in the presenter view.
  • 13:30: Artist talk by Andy Cavatorta – Andy is sculptor working with sound and robotics. While his work is powered by software and robotic technologies, its focus is on experience and emotional narrative.

Assignments

  • Follow your own plan, and advance your project.
  • Recommended: as you refine your mapping choices, read the paper under Resources. It discusses mapping beyond one-to-one (one-to many, layers), and an interesting experiment where it was found that a challenging mapping choice kept performers engaged for longer than the easier-to-learn alternative.

Resources

  • “The Importance of Parameter Mapping in Electronic Instrument Design”, M. Wanderley, NIME-02

Session 11. Updates. Methods for performing. Performance logistics

  • Check-ins + feedback, part II
  • Show production updates: functions, posters, rehearsal logistics, technical riders
  • Performances: collective analysis

Assignments

  1. Get your instrument ready for rehearsal.
  2. Rehearse, and time yourself. You have 10 minutes in total, including:
    • setup (i.e. 3 mins)
    • performance (i.e. 5 mins)
    • teardown (i.e. 2 mins)
  3. Submit a first version of your Technical Rider (you can update this later)
  4. Review/submit your success criteria/feedback questions. After you perform, we will read the top three questions under “audience” to the group, and they write their responses on printed feedback sheets (make sure to bring a pen).
  5. Recommended: sign up for extra rehearsal time in the Garage the week before the performances (Dec 6 and Dec 7). Please do not sign up for more than one rehearsal slot this week. We will do another round of sign-ups after class next week. An audio engineer will at the space during these times, so you can connect to the audio system.
  • Note: You can also sign up for time in the Garage directly: find an open spot on the calendar and make a request using this form. This is managed by the 370 Jay St staff –approval is not guaranteed, and may take a few days. You will likely need to bring your own speakers – you can request audio and lighting staff to run the house systems, but they may not be available.

Session 12. Rehearsal

Please come 10 minutes early, and get your instrument and any equipment ready to plug in before 12:10. This will ensure everyone gets a chance to rehearse – time will be tight!

Here is the plan:

  • We will meet at the Garage at noon, and start the rehearsal at 12:10 sharp.
  • The rehearsal order is here.
  • We will keep track of the duration of your setup, performance, and teardown – if you ran over, we will let you know so you can adjust for the dress rehearsal.
  • Keep track of who is performing:
    • When you are the second in line to perform, go backstage and get your instrument ready. For example, if you’re performer #5, you should go backstage when #3 starts moving towards the stage.
  • After each performance, the performer’s feedback questions will be read to the group. While you are not performing, you will be writing notes for your peers on a feedback sheet like this – make sure to bring a pen!

The setup will be similar to what we had for your first performances in October:

  • Audio
    • Stereo (most of you). Connect your audio output via an 1/8” audio cable (we’ll have an aux cable ready, like in class), or via XLR / 1/4” cables, which you’ll bring. These will go to a Scarlett audio interface on stage.
    • 4 channel (for Dave!). We will connect the 4 audio outputs on your interface to the wall panel via XLR cables. The distance from the edge of the stage (downstage) to the wall plate is approx. 11 feet.
  • Projector and large screen (behind the stage). Connect your USB-C or HDMI video output (we’ll have an HDMI cable ready, with an USB-C dongle)
  • Detail camera. If it makes sense for your project, you can project a closer view of your instrument using a detail camera mounted on the stage table. You can connect the camera:
    • directly to the projector, or
    • get the camera stream into your computer via USB-C or HDMI ports, and choose how to project it (side-by-side with your p5 sketch? your Max patch? your own video content? or, processed in some way?)
  • Note: as we discussed in class, we are considering having a secondary screen on-stage during the show, for streaming the detail documentation camera. This will depend on the space available on-stage, given the Egress area.

Assignments

  1. Finish your instrument. After dress rehearsal, you should only do minor tweaks, not add new functions or make significant changes.
  2. Rehearse your performance as many times as you can. Make sure it is under 5 minutes.
  3. Practice setting up and tearing down your project, too, with your assigned stage helper. Ideally, do this in a space separate from where you’re working, and keep in mind the distances from your project to the stage mixer, and so on. Make sure setup + teardown don’t take over 5 minutes, combined.
  4. Submit:
    1. Your performance/instrument title and description (see Promotion section here). Monday, 12/05 at midnight is a hard deadline: these need to be proof read, put into the program design, and sent to print in time for the show.
    2. Your technical rider.
    3. The names and emails of your two guests for Saturday, here.

Session 13. Dress Rehearsal

Please have your instrument to our Green Room, ready to plug in (room 233 on the 2nd floor), before 12:10. This will ensure everyone gets a chance to rehearse – time will be tight!

The schedule for the Dress Rehearsal is here. Please keep in mind: if you are not ready by your slot time, you might not be able to rehearse today.

The assignments are the same as last week – please re-read them to make sure you are not missing anything!

NIME SHOW

  • Please review our Production sheets which include the schedule for each day, production team roles and contacts, and more.

Session 14. Show Review

Policies

Evaluation

Grades will be determined according to the following breakdown:

  • Regular Assignments 40%
  • Participation and Attendance 30%
  • Final Project 30%

Please see ITP’s statement on Pass/Fail which states that a “Pass” is equivalent to an “A” or a “B” while anything less would be considered a “Fail”.

We will have weekly assignments that are relevant to material from the previous class. These assignments are required and you should be prepared to show/talk about them in class. It is expected that everyone in the class will create and maintain a blog for their assignments.

Attendance is mandatory. Please inform your teacher via email if you are going to miss a class. Two unexcused absences is cause for failing the class. (An unexcused lateness of 10 minutes or more is equivalent to 1/2 an absence.)

This class will be participatory, you are expected to participate in discussions and give feedback to other students both in class and participate with their projects. This (along with attendance) is 30% of your grade.

Class will culminate with a final performance of student projects. You are expected to participate in the final performance of your NIME, as well as previous rehearsals throughout the semester. This will comprise 30% of your grade.

Statement of Academic Integrity

Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else.

Statement of Principle

The core of the educational experience at the Tisch School of the Arts is the creation of original academic and artistic work by students for the critical review of faculty members. It is therefore of the utmost importance that students at all times provide their instructors with an accurate sense of their current abilities and knowledge in order to receive appropriate constructive criticism and advice. Any attempt to evade that essential, transparent transaction between instructor and student through plagiarism or cheating is educationally self-defeating and a grave violation of Tisch School of the Arts community standards. For all the details on plagiarism, please refer to the Tisch School of the Arts, Policies and Procedures Handbook, which can be found online at: https://tisch.nyu.edu/student-affairs/important-resources/academic-integrity-policy

Statement on Accessibility

Please feel free to make suggestions to your instructor about ways in which this class could become more accessible to you. Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.

Statement on Counseling and Wellness

Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYU Wellness Exchange 212-443-9999. Also, all students who may require an academic accommodation due to a qualified disability, physical or mental, please register with the Moses Center 212-998-4980. Please let your instructor know if you need help connecting to these resources.

Statement on use of Electronic Devices

Laptops will be an essential part of the course and may be used in class during workshops and for taking notes in lecture. Laptops must be closed during class discussions and student presentations. Phone use in class is strictly prohibited unless directly related to a presentation of your own work or if you are asked to do so as part of the curriculum.