Digital Audio Synthesis Techniques/MIDI

In the Digital Sound Lab course we focused on tools for creating basic audio material, and generally treated it as if it were to be used in a linear narrative context. The purpose of this class is to expose you to the other side of the coin, namely the ways in which you can create audio material in a non linear, more interactive or spontaneous context. The primary tool we'll be using for this purpose is the Max programming environment from Cycling74.  With the recent addition of jitter to Max's already formidable arsenal of tools, the course will fall into roughly three categories: MIDI (Musical Instrument Digital Interface), DSP (Digital Signal Processing via Max's MSP component), and Jitter (realtime video processing via Max). Other topics to be explored in this context will be Phil Burk's JSYN, Disk Didkovsky's JMSL, and possibly CSound, for a little background.

It's a pretty simple thing to generate streams of musical notes via the input of your choice (midi device, hair dryer, pulse rate), but there are many types of control available through MIDI and I'll try to keep you focused on ways to go beyond the simple creation of ‘keystrokes'. MIDI is a useful low bandwidth tool for supplying input to and output from a system, but as CPUs have increased in capacity and speed, the need for MIDI as communication for hardware synthesizers is shifting over to one of several choices for control of ‘softsynths', synthesizers that exist in software alone.  Max is primarily a midi tool, which was transformed a few years ago into a complete digital audio creation suite. Now we also have Jitter, which adds live video, graphic and 3d tools to the suite.  We will start with an orientation/overview to put things into perspective, along with an introduction to the sound labs. Then we will focus on the basic elements of midi and the midi studio, which will quickly get us into Max, the mastery of which will be our main goal. People come into the class with varying degrees of experience, and I'll endeavor to calibrate the learning curve accordingly.

Your participation and suggestions will play a major role in shaping the direction of the class. If you have other topics you wish to have attention given to, don't hesitate to suggest them. I am recommending three books for you to buy, which will be as close to a "text" as we'll get: Composing Interactive Music, by Todd Winkler ( MIT Press, ISBN 0-262-23193-x). It is a Max 'cookbook', with hundreds of examples and ideas (and included software examples) which can help you create. For an overview of MIDI and some good technical charts, get MIDI for the Professional, by Lehrman & Tully. Gordy has ordered these items, which are available at the book store, or, of course, amazon.  Usually we'll be creating something new in class. It is crucial that you bring in the work you've done during the week, to provide raw material for discussion, and also as a way to get feedback or help. Keep your memory stick handy for easy transfer of materials into the classroom computer, or in the case of midi you can easily place them in your fargo directory and ftp them right onto the desktop. It's also easy now to transfer files to the classroom computer from whatever workstation you're at on the floor via ethernet. You can also retrieve class examples and missing max objects, patchers from the classroom machine to the workstation you're at. All the floor machines use the standard itp login/pwd.  You are also encouraged to sign out cd burners to back up your larger work, if you've been generating large video or audio files, for instance. The floor routinely wipes the drives, and you don't want to lose all that work.  IMPORTANT: Any files you pass back and forth across ftp need to be properly named, to avoid corruption. For Max patchers, you can simply save as a 'max text' file from within the program (be sure to use .txt as the file suffix). The other option is to archive the file into a .zip or .sit. The Macintosh does not attach a suffix by default, so if you try to simply email me with your file attached, if it doesn't have the proper suffix, my pc will misinterpret it, as will most any unix server you put it on. You can save a Max file as .mxb, which is the Max identifier, please don't use .pat (also in use) because Photoshop also uses that suffix for their PATterns.

Be sure to check my web page periodically for updates, syllabus changes, and most importantly, for the Soundlab Help Files. My email is: daniel.palkowski@nyu.edu. In class I'll give you a better email address, though. I'll be around on Mon. afternoons before class, and will be available after class as well. You can also call if you need assistance in the labs (can't get sound, midi, etc.)  I'll give the numbers in class. There is also a list where you can post questions,comments, news, etc., it is itp-sound. You get to it via http://home.nyu.edu, and subscribe with your net id and a password which I'll give in class. I'll also give you an IM address so you can send me an aol IM  from any pc if you have trouble or a question.

Here is a general plan for material to be covered. The structure and content is subject to change, depending on your interests, new floor acquisitions, and the general state of a particular piece of hardware/software in the classroom:

A Brief History of Electronic Music Technology, MIDI in Transition
Discussion of the major trends in electronic music, playback of samples from different eras, overview of technologies

Practical: INTRODUCTION TO THE MIDILABS
Turning on the system, getting sound, review of the mixing board, troubleshooting. Manual control of the synths via their front panel display, using the MIDI Express to rout control. Finding and using the Max tutorials.

Assignment: Get yourself a minidisk (or cassette) to bring into one of the audio stations. Sign out some time on one and work on your mastery of the audio routing in the system by:
1. Figuring out how to get sound from the synth into the mixer.
2. Figuring out how to get the sound from the mixer to the input of the mini disk recorder (or cassette).
3. Figuring out how to access different patches on the synth, and record them onto disk or cassette (if you aren't a keyboardist or musician, no worry: just play a single key with the sound you've selected).The sounds you pick will be used in a later exercise.
4. Go over the terms below, make sure you have a basic understanding of the parts of a synthesizer. Here are some test questions: what does it mean to oscillate? when you filter sound, what part of the sound is being filtered? A diagram of an envelope often resembles a cycle of a sound wave. How are the two different?
5. You can start looking at the Max tutorials at any time. Here are crucial tips:
copy the various Tutorial folders to your own media..there are max tutorials, msp tutorials and jitter tutorials, each being a Max patcher in its own right. Some of the tuts require other supporting files, such as graphics, video, etc. They should all be found in a folder called 'Media' in the Max 'Patchers' folder. The directory structure is a little different between the Windows and Mac versions. If you have trouble finding things, just do a search for the names above.
The help files are just as useful as the tutorials themselves. Copy the help folder (max-help) to your media as well.
When you open and play with a help file or tutorial, you should save it under a different name, (e.g. Katy-MaxTut1.pat), so you don't inadvertently save over the original help file or tut. Also, alt- or option-clicking on any Max,msp or jitter object will open its help file, which is also a working patcher.

Useful Bibliography: MIDI bibliography list
Check out The Electronic Music Foundation for their history of electronic music section.

The MIDI Interface

This session will be a high level discussion about MIDI hardware: how things are connected, and how MIDI works in the practical sense. In the process, we'll also deal with any associated audio considerations, patching, wiring, etc. We'll then explore the built-in MIDI UI in MacOSX and WindowsXP and discuss practial ways to connect systems together. Finally, we'll compare a typical hardware synthesizer (Korg M-1 or Wavestation) with similar software devices (Reason modules) and discuss the computer integration of each into a studio.
Main topics:Modes, Channels, Ports, Hardware/cables, How MIDI flows, discussion of serial communication, wiring, the parts of a MIDI interface.

Assignment:
Launch Reason. Create your own new bank, setting up the audio and midi settings to allow live control from a USB keyboard controller. You may need to configure AudioMidi setup on a mac, though usually simply plugging in the keyboard after launching will cause it to work.
Once you have the software listening to the keyboard, create a 14 channel mixer, a Subtractor analog synth, and an NN-XT sampler into your bank. Browse through patches until you find something you like, then spend some time studying the synth interfaces, notice which controls are common to both and which are unique. Use the attached midi keyboard to trigger sounds from the modules, be sure to experiment with the keyboard controls (all of them) and observe how each is mapped into the user interface of the selected device. Listen closely to the changes you can produce by use of the controls. Save your bank frequently.

Terminology

A MIDI Timeline

The MIDI specification

In this class we’ll delve deeply into the structure of MIDI, including a review of the binary and hexadecimal numbering systems and their usefulness in coding MIDI. We will also use Max to view incoming and outgoing MIDI data..

Assignment:
1. Study the MIDI implementation chart. Review the structure of status bytes and data bytes. Commit to memory the hexadecimal and decimal codes for the following status bytes: note-on (90-9F), patch change(C0-Cf), control message(B0-Bf), and the control message data bytes for volume (07), sustain (40), pan (0A), and modulation (01) controllers.
2. On the Wavestation, go into the midi editing mode and figure out how to turn local control on/off. Figure out how to switch from poly to multi mode, how to change the transmit channel. When you're done, the settings should be: local control: disabled, mode: multi, transmit channel: 1. Also, figure out how to map midi control 1 (mod wheel) to some parameter of the particular program you're editing.
3. Using the data chart, construct the melody Yankee Doodle (in C) as a set of 1-line lists in hexadecimal, using properly formatted midi commands (hint: here's the first note, assuming we're starting on middle c, on midi channel 1: 90 3C 40 90 3C 00 I suggest that you make three columns & label them STATUS, DATA1 and DATA2)
After you've figured out the commands and notes, think about the melody. What's the simplest way you could describe the rhythm, or duration, of each note in the tune? After each line, put down a number that you think expresses the relative rhythm of the note with each of the others.
4. Practice converting numbers between 0-255 into hex, decimal and binary.

Max and MIDI:Live Input Processing

In this session, we'll use Max for one of it's original purposes: as a tool for enhancing a live performance using MIDI. We will take the output from a midi controller (or two) and create a patcher that will deal with the incoming data in a number of interesting ways.
Topics will include parsing and reformatting MIDI data, recasting and interpreting, MIDI delay effects, enhancing the patcher UI with the ascii keyboard.

Assignment: If you haven’t been doing the MAX tutorials, NOW is the time to start. You should be able to get through the first twenty in a couple of afternoons. Try doing them this way: copy the tutorial folder on to your own zip disk. Run the tutorials from there. That way you can make any kinds of alterations to them you want with out the fear of writing over the originals. Remember that each midi station has a unique configuration, so it will probably be necessary for you to follow the note below.
Always explore the patcher(s) we design in class, since they'll mean a lot more to you than something out of a folder. I post the patches to the blackboard 'course documents' area soon after each class. But don't just look at them..download them, then butcher them to your own tastes. Don't worry if you end up with a mess, just try different things. That's the best way to get warmed up. Eventually you'll probably approach the program from a more 'problem solving' approach, b ut for now just play. If you're modification ends up doing something you like, I always encourage you to have a mini disk loaded up & ready to capture some of the audio output of your concoction, then you'll have something to play in the next class.
Worth repeating: Max patchers exist in two forms - either a Max Binary file, or a Max Text file. A Max Text file is just that: text. When you save to that format from the max program, you can send your work to anyone via email. It is far less reliable to send a Max Binary file, and if fact is discouraged. There is absolutely no difference in what gets downloaded at the other end. Be sure to append .txt to the file name. For your max binary files, I also urge you to use either .mxb (max binary) or .pat (patcher). The latter is sometimes confused in some machines with Adobe Photoshop's 'pattern' suffix, though, which also uses .pat, so .mxb is best.
NOTE RE TUTORIALS: The tutorials are intended to be generic and may need to be modified for the particular workstation you’re studying on. Make sure that the letter indicated in any ‘note-out’, ‘ctlout’, or other object that sends midi data out of the computer is represented by the synthesizer you want to use. You check by going to FILE | MIDI SETUP, then look on the right side of the dialog box at the output devices. Just change whatever letter represents the synth (by dragging on the letter) till it matches the object’s letter. There are many software synthesizers out there, which lets you sidestep the awkwardness of getting output to an external device. One is Quicktime (you have to use qt6 for osX), via a max object called 'qtmusic' (see the blackboard site), the other is SimpleSynth, which is a free download, and which should be on all the floor machines , The sound will come out of the mac itself. An alternative to the file|dialog approach is to doubleclick the object and select the midi destination from the popup list that appears. I have found that that is sometimes necessary even when you set the letters in midi setup to match.

Terminology

Max and MIDI: Simple Algorithmic Techniques

Having explored live techniques, here we'll use Max as an algorithmic composition/performance tool, and consider ways in which you can make the computer do some of the 'thinking' for you via the use of constrained random processes. We'll also start prettying up our workspace by using sub-patchers, bpatchers, send/receive msgs, etc.. The goal is to start small and build a complex mechanical process through Max's 'object-oriented- nature.
In the process, we'll discuss methods currently in use to shape musical structure, including numerical systems, chaos/fractals, string replacement (L-systems), etc.

Assignment: Using one or more of the techniques explored in class, create your own algorithmic sound machine. Of course, you can also simply hack into the class demo.
Do a Google search for the equation "3n+1" and see what you find. See if you can incorporate it (or one of its type) to generate some sort of midi data in Max.

MIDI Sequencing

This class will explore the concept of MIDI sequencing, that is, the laying down of time-stamped data into a file, which may be stored, played back, of further processed. We will combine ideas from the first two classes to create an interactive midi recording system. Can't say much more, since the result is always a surprise! We’ll also delve into integration of simultaneous apps (e.g., Max and Pro Tools, Finale, etc.) via Mac OSX Core MIDI.

Assignments: Create for yourself a sequencer that will allow you to:
1. input from a keyboard or controller, any type of midi, and show you what type of data is coming in.
2. easily select different midi channels to record new material without erasing the old.
3. create a 'click'track', to keep yourself in tempo (hint, use the tempo object, pound out a single note (use channel 10, note# 42 on the Apple DLS Synth) ).
4. have features for setting the program (patch). Once you have something that works to your liking, set the output device to 'Max Out 1' (easiest way is to doubleclick on your final [midiout] object).
While Max is running, launch Pro Tools and make a new project, then choose File|New Track|Midi. In the mix window (Window|Show Mix), set the input to Max Out 1 (should show up in the popup window). Then create an audio track and use Reason as its insert (multichannel plugin|Instrument|Reason). Switch back to Max and record some keyboard (or you can open any type0 format midi file into the [seq] object). Set the midi playback of your patcher to loop (have it bang back into the seq object from the right outlet of itself), switcdh back to Pro Tools, record enable your midi track and start recording...you should be able to capture the midi output from Max..

MSP: Max Signal Processing

We'll start by reviewing the parts/capabilities of an analog 'synthesizer' via Reason’s Subtractor. We'll then explore the ways in which these sounds are achieved in the digital world.
Precursors (fortran, csound) and recent alternatives (JSYN) to MSP will be discussed, as well as the pros and cons of doing cpu-based synthesis as opposed to relying on midi controlled synths.
We'll listen to samples of electronic music through the years that use various technologies and discuss some of the aesthetic changes (and challenges) that electronic techniques have brought to sonic art.

Main topics: synthesis from scratch - creating waves using cycle~, phasor~ and lookup tables; Controlling signal flow and amplitudes; creating envelopes; ways to complexify timbres: FM, Additive synthesis and wavetabling; basic integration with MIDI.

Assignment:
1. Browse through various MSP tutorials (Applications | Opcode | Max | MSP Stuff | MSP Tutorials). Be sure to do the first five, which are introductory. There is also an excellent folder of MSP examples in the same directory.
2. Experiment with the basic objects cycle~ and phasor~, observe the effects of plugging one cycle~ or phasor~ into the inputs of another; practice using gate~ objects to control the flow of signal. Be sure to route your objects through a set of *~ objects to control the amplitudes. Be sure to check out my David Zicarelli feedback variation patcher on Blackboard. As always, feel free to hack it to pieces.
3. Integrate an MSP element into the sequencer we created in the previous class, either as an audio voice or as a control element. Save under a different name. Consider the issues involved in saving a live or studio performance using this tool that would contain both midi and audio data: how would you get the two completely different technologies to save and load properly?

MIDTERM PRESENTATIONS. Present a Max patcher that you’ve made, and demonstrate what it does. You may base the project on the Music Box assignment, your own research, or it may be a prototype of or prelude to your final project.

2. Remaining class time will be a recap of the midi specification, including status/data bytes, modes, channels, controllers, chief data representation methods, etc. plus Q&A on the previous week's topic of elementary synthesis.

At this point you should let me know if you have a solo or group piece you'd like to present in the May Show, so I can start the works in motion towards getting you a space, equipment, etc.

Sampling

MSP has many tools for navigating and manipulating digitized audio, aka samples. Now we'll explore the use of buffer~ as a tool for holding temporary audio information, and the ways it can be referenced by a wide assortment of tools, such as groove~.

Assignment:
Create a patcher that records input and automatically writes the recorded buffer~ to an auto-named file in a designated folder. Writing can either occur as soon as recording is done, or via a button. Once you've confirmed that you know how to do the trickier auto-writing method, feel free to modify the write command so that you can name the file whatever you like. Hint: use the sprintf object to set up the autonaming. A fun variation to the above, in lieu of auto-writing files, is to make a 'ping-pong' recorder, which alternately records into one of two buffer~s, switching after a specified time. You can create an 'echoplex' effect this way, for instance, by feeding back some of the output of one buffer into the other and blending with the live input. Watch your levels, though!

MSP: Advanced Concepts

We will review concepts from the previous class, and introduce more advanced methods of controlling and analyzing audio, including pitch detection, FFT functions, and a host of sample based applications. We'll play with some of the most useful msp objects and patchers which have been created by third parties, including granular synthesis, FFT related objects, such as fiddle~, and the powerful physical modeling suite of objects, percolate~.
We will also cover the poly~ object, how it can be used to multiply your instrument efficiently, and how to set it up with the pattr objects

Assignment:
Customize one of the Percolate instrument 0bjects by adding a user interface to it. Encapsulate your instrument in a bpatcher, so that the percolate object and wiring is hidden, and only the user interface portion is visible. Use plenty of the Max UI tools (panels, color, info boxes) to spice up your interface.
By now you should also have a good handle on what you'd like to present for a final project. Max/MSP is very open-ended, so the idea can take pretty much any form you can dream up.

Jitter: The Basics

Jitter is an integrated suite of objects for Max which allow you to control data as superimposed matrices. This makes it perfectly suited for video work, since a video screen is basically one big matrix of pixels. In this class we'll learn what is necessary to set up basic functionality of jitter and explore some of the most useful manipulation tools, such as jit.op, jit.qt.effect, jit.rota, etc., on still images and imported video files.

Assignment:
From here on out, most of your work sould be put toward your final project. but here is an exercise to get you started with jitter:
Assignment: Load a still image into [jit.qt.movie] ( [...] denotes a Max object). Use [jit.scissors] to split up the image into four quadrants, then use a different jitter object on each of the four quadrants. Finally, use [jit.glue] to reassemble the four pieces into a single matrix again.
Variation: instead of regluing the four, use a few [jit.xfade] objects to crossfade between the quadrants, and send the result to a new [jit.window].

Jitter: Live Input Processing, Maximizing Quality, Manipulation of Sound.

Jitter is a huge topic by itself, and these three sessions will barely scratch the proverbial surface. Version 1.5 has many enhancements, and the holy grail (hi resolution, fluid, realtime video no matter how much processing you throw at it) inches ever closer. We'll explore ways to get the best looking output from your jitter patcher, and how to use DV I/O via firewire. Other tools particularly relevant to this class will involve ways to visualize audio input, and translate video into some sort of audio control.

Assignment: Experiment with the following patchers: tutorials 19,20,21

Jitter: Working with 3d

One of the nice features of Jitter is it's ability to leverage other technologies seamlessly into the Max environment, such as open gl. Here we'll look at how Jitter imports, arranges and modifies objects created in programs such as Maya, Lightwave, etc. This class will also serve as a polishing class for your work in progress.

Assignment: Finish Projects!

Final Presentations, Party, etc.