All posts by Max Horwich

Interaction (no real progress but lots of ideas)

I haven’t made much concrete progress since last week. I spent the early part of this week getting a camera feed into Three.js, which I succeeded in doing. I started experimenting with a processing sketch for tracking the center of change , but I couldn’t figure out how to convert the whole thing into p5 before the rest of the week caught up with me and I had to put it down for the time being.

However, I am confident and excited moving forward. I’ve started pulling together the pieces for my final project, which is also the final project for every other class I’m in, which means after this weekend, it will be the only thing I have to work on for the rest of the semester.

From this gem of a web 1.0 site, I’ve scraped .mus files — an esoteric musical file format similar to MIDI — for over 100 songs from the Sacred Harp Songbook. I will be using these files to generate a Markov Chain to algorithmically generate this style of music in real time. Because the music is inherently “spatialized,” with sound coming from all corners of the room, it is perfectly suited to a 3d environment like Three.js.

I have more specific ideas regarding execution that I’ll talk about in class — I don’t think I have time to get them all out in the next 25 minutes — but I’d like to put some time into explaining why I want to do this. Beyond a desire to create something transcendently beautiful (which I believe this will be), it serves as a useful proof of concept in developing a more universal system.

The field of Cantometrics seeks to provide a qualitative analysis of all the musics of the world, studying what makes the music of each culture unique, and by extension, what unites them all. It’s a mid-twentieth century concept that has blossomed recently with the aid of computerized analysis, but to my knowledge has not yet sought to actually generate anything with all those data points. This could be a first step driving this field of study toward creation. If such a system could be developed for music, it stands to reason that it could be developed for any form of expression.

In pursuit of this goal, I’ve been diving into several Javascript libraries that deal with generative text and speech synthesis. I also used Python for the first time to pull the .mus files (I didn’t even really know what Python did at the beginning of this week but wow Python is amazing). I have a lot to learn, and this is going to be a lot of hard work. I can’t wait.

Elements

I really felt the split between the conceptual and technical parts of this class this week. I walked away from Remix feeling inspired to go reinvent the way people communicate, only to spend an entire week trying and failing to upload a dancing alien to a website.

Having spent most of the week trying and failing to make one change after another to my previous code, I realized the problem was there was too much going on in the code that I didn’t understand. So I started a new script from scratch to figure out what each individual part of the code was doing. I’m most of the way there:

osc element

It doesn’t do much yet, but it sure looks cool. Can’t wait to see what it looks like when I know what I’m doing a little better.

THREE.JS setting

My first experiment with a 3D environment in Javascript. I took a panorama of a path through the woods from Flickr, flipped the image upside down, and then crossfaded back and forth between the original and the inverted image.

Having done a bit of work in VR last semester, I’ve been interested for a while in how to cut between scenes in a completely immersive environment. I’m attracted to the hyper-reality created by the gradual fade between a recognizable environment (a path in the woods), a semi-recognizable environment (that same image inverted) and something completely abstract (the fractal patterns created by the interplay of trees, ground and sky as the two images cross-fade). I look forward to continuing to explore and develop this environment as I gain a better understanding of how to navigate Three.js.

An Alphabet of Meaning

A college friend of mine once told me something that has stuck with me for nearly a decade and a half. She was considering a film major because, she told me, “film is a better form of communication than language.” She ended up studying neuroscience and computers instead, but that statement changed the way I think about art and media forever.

I never would have thought to put it in those terms, but of course film is a better form of communication than language. Film can contain language, of course, but it also wraps that language in music, lighting, scenery, characters, a story, camera angles and special effects, all edited to the microsecond with surgical precision, often by dozens or hundreds of people. It’s hard to imagine a better tool to convey meaning, precisely because it encompasses and integrates so many other things that convey meaning.
Meaning is a tricky thing. No matter how carefully one tries to express it in any form, it’s always liable to be missed or misconstrued, sometimes for better or worse. Any form of communication is inherently a two-way street, whether it’s really interactive or not (books, for example, can be misread or expounded upon, depending on who reads and interprets them). But it’s in that dependence on an audience that meaning comes to life. I find it useful to turn to musicologist Christopher Small and his concept of musicking, (that is, music as a process (verb), rather than an object (noun)):
“The act of musicking establishes in the place where it is happening a set of relationships, and it is in those relationships that the meaning of the act lies. They are to be found not only between those organized sounds which are conventionally thought of as being the stuff of musical meaning but also between the people who are taking part, in whatever capacity, in the performance; and they model, or stand as metaphor for, ideal relationships as the participants in the performance imagine them to be: relationships between person and person, between individual and society, between humanity and the natural world and even perhaps the supernatural world.”
This incomprehensibly vast network of relationships is where meaning is created. And as we become an increasingly global society, this network of relationships grows evermore vast and incomprehensible. One strange side-effect has been the proliferation of CGI-heavy summer blockbusters (jokes don’t always translate into other languages, but explosions do, so these do better than comedies in overseas markets). However, I refuse to believe that exploding robots are the only message worth sending to a global audience. But more messages to send depends on more messengers to send them.
I’m not suggesting that the next step is to democratize filmmaking — that’s already happened. And moreover, film (broadly defined) is having a moment now because every person on earth is staring at a screen for most of their waking hours; but as we’re frogmarched away from our screens to mixed reality contact lenses or neural lacing or whatever terrifying sci-fi bullshit comes next, film as we currently understand it will go the way of the zoetrope.
To truly universalize communication in its most effective state, we need an alphabet of meaning, a catalog of things that cannot otherwise be expressed, and might mean different things to different people. For instance, one symbol might refer to a day at the beach for one person, a hug from their grandmother for another, or the smell of fresh cut grass to someone else. Alternately, another symbol might mean the feeling of just missing the train when you’re already late for work for one person, or that time in second grade when you called the teacher “mom” to another person.
It’s hard to imagine the form such symbols would take, and at this point I don’t really have a clear concept. Some combination of art, linguistics, neuroscience, and deep learning may approximate it, but in all likelihood our current technologies can’t quite get us there. However, I believe that it is essential to work towards in whatever way we can. The challenges that we will face over the rest of the 21st century (and hopefully beyond) will depend on our ability to communicate increasingly deep and complex ideas, to look past differences of language and culture and to focus on what we all share as humans.