Embodied Cinema and the Body Language of Trees

MOVIES AND PERCEPTION

Cinema communicates with human hearing and seeing as if direct sensory evidence. Even before motion-picture-sound media first played to an audience and prior to the movies’ confluence with the cultural mainstream, this scarcely scrutinized paradox had already asserted itself in the preconceptions and strivings of the technology’s inventors. André Bazin in The Myth of Total Cinema observed how film’s development, long before it acquired a language, proceeded from a driving ambition on the part of these 19th century industrialists to achieve “an integral realism, a recreation of the world in its own image, an image unburdened by the freedom of interpretation of the artist or the irreversibility of time” [Bazin 1946, 21]. 

And while we watch films with our eyes and ears, we experience them with our minds and our bodies [Bordwell 2012]. The world of a film exhibits a spatiotemporal and causal structure that resembles our sensible world. It may be true that our species did not evolve to watch movies, but movies have nevertheless evolved to match the way our cognitive and perceptual faculties operate [Cutting and Candan 2013]. The art of narrative filmmaking has endeavored throughout its progress to reproduce cues that people use to identify objects and to grasp spatial relationships in ordinary situations. A moving camera, not just the movement of things in the picture, transports our sensing bodies into the scene [Gibson 1979].

Cinematic devices like flashbacks, closeups (both visual and aural), sonic perspective shifts, and constructivist editing have been treated as analogues for mental processes such as memory, attention, inference, and projection. Rules of camera movement and the logic of cutting are optimized to register dynamics and multiplicity of a human point of view. Referring to shared knowledge of the world and an acquired grasp of cinematic editing conventions, the audience might be said to collude with the filmmaker in the construction of a coherent story. 

The 19th century philosopher-psychologist William James likened human attention to “an alternation of flights and perchings.” Yet despite its selectivity and perpetual fluctuations, we exercise the impression that our consciousness is continuous. Likewise, we readily ignore the perceptual discontinuity of cuts in a film so long as the sequence reveals a connected space or develops a continuous line of action. Our brains proceed to fill in the gaps and to link consecutive clips according to an associative logic that we practice in everyday life.

Unlike any previous system of human communication, cinema has assimilated operations of being alive and sensually embodied in the world (looking, listening, moving, etc.) as the very substance of its language. Film phenomenologist Vivian Sobchack describes the experience of the motion picture as a reciprocal interchange of perception with expression. “A film is an act of seeing that makes itself seen, an act of hearing that makes itself heard, an act of physical and reflective movement that makes itself reflexively felt and understood.” It is an experience that is mutually possessed by filmmaker, film, and spectator, through common structures of embodied existence. Simultaneously having sense and making sense for us, the cinema literally “returns us to our senses” [Sobchack 1992].

Audio-Vision and Synesthesia

‘Audio-vision’ is the name given by composer-filmmaker Michel Chion to the privileged relationship between image and sound that is bestowed upon them by film, wherein their co-presence is at once bound together and peculiarly prioritized by the framing of the visual. This audiovisual coupling is experienced by spectators not as a simple addition of two independent sensory channels. Rather image and sound are mutually inflected and transformed by one another. “The outcome of the combination does not consist in perceptions of images and sounds as such, but in perceptions of space, matter, volume, meaning, expression, and spatial and temporal organization” [Chion 2010].

In Chion’s lexicon ‘synchresis’ describes a psychological response to audiovisual simultaneity that can occur even when sound and image originate from different sources. “The forging of an immediate and necessary relationship between something one sees and something one hears at the same time (from synchronism and synthesis) . . . is what makes dubbing and much other post-production sound mixing possible” [Chion 1994, 224]. “Without synchresis, sound would be obliged to resemble reality and would have much reduced expressive possibilities (because, it must be said, sound in films only roughly resembles sound in real situations)” [Chion, 2010].

‘Spatial magnetization’ of sound by the image is correlated with humans’ predisposition to hear a sound emanating from the location where we see its source, even if the sound actually reaches our ears by way of acoustic reflection or is played on a loudspeaker that has been installed at the back of the theater. Furthermore, because film sound is bounded by no auditory container, we tend to interpret its location with respect to the edges of the visual frame [Chion, 2010].

‘Added value’ accrues to the image in consequence of a spectator’s unconscious response to the audible component of the audiovisual couple.

“By added value I mean the expressive and informative value with which a sound enriches a given image so as to create the definite impression, in the immediate or remembered experience one has of it, that this information or expression ‘naturally’ comes from what is seen, and is already contained in the image itself.”

Chion 1994, 5

Once upon a time Aristotle proposed that humans are endowed with five senses, each enabled by its respective sensory organ. Twenty-three centuries later, as neuroscience extends our knowledge about cross-modal plasticity of the brain, we have departed from construing the senses as a collection of discrete faculties as we begin to grasp their fundamental interconnection. Once regarded as a neurophysiological quirk analogous to crossed electric wires, synesthesia is reconceived in contemporary aesthetic theory to refer to cooperation among sense receptors and to acknowledge the multisensory integration that underlies ordinary consciousness [Merleau-Ponty1945, Marks 2011].

Influenced by Chion’s insights while reflecting on his own years of experience observing the world through a camera, ethnographer-filmmaker David MacDougall contemplates the capacity of cinema to affect spectators not only through vision and hearing, but to simultaneously engage our other senses as well. What sensations a film does not evoke directly, we are quick to summon and project back upon the film from personal recollections of place, situation, touch, taste, and smell.

“The cinema makes possible an almost endless number of combinations of sight and sound to evoke an equally endless number of sensory impressions. To the five senses can thus be added such further perceptions as those of temperature, space and distance, weight, pressure, and texture. These should perhaps properly be considered derivative senses, from touch and vision, but they are no less distinct or real for the person experiencing them. For example, where should we place the perception of wetness or dryness, the perception of viscosity? Each surely involves something more than touch.”

MacDougall, 2014

Visual and acoustic closeups in film, by calling attention to surface topographies and textures, produce a synesthetic effect that refers vision and audition to the sensation of touch. The sound of a worker’s shovel scraping concrete pavement in order to clear a path through snow conveys to our bodies everything about the temperature and saturation of the air, as well as the texture and obduracy of the slosh.

Umwelt and Atmosphere

The sense of place, or experiential space, is a consensual relation that arises in the interaction of an embodied subject and the perceived world. Not only perception but also expression—the ability to give meaning to the world—are inextricably linked to embodiment.

On the subject of atmosphere, philosopher Gernot Böhme has written, “To sense oneself bodily is to sense concurrently one’s being in an environment, one’s feelings in this place” [Böhme 1995, 21]. No surprise that weather and feelings are closely conjoined. A dip in barometric pressure, likely to portend an approaching storm, is also notorious for aggravating joint pain and triggering headaches. Whereas a color meter can detect that the sky is overcast this morning, you and I are furthermore aware that the mood in the air is gloomy or oppressive or listless or threatening.

Another sense of the sense of place that readily conspires with cinematic construction is suggested by the German word ‘Umwelt.’ The naturalist Jakob von Uexküll tried to imagine the physical world as lived and perceived by different animals [Uexküll 1934]. He used the word Umwelt to denote the organized experience—or point of view—unique to any creature, which depends on that particular creature’s sensory and cognitive apparatus. Different organisms have discrete Umwelten, even though they share the same environment.

In cinematic space, corporeal knowledge takes precedence over symbolic communication. Our bodies feel their way around a location in screen space, engaging with the body of the camera operator as well as with the bodies of the subjects. Rules of interpersonal space and social distancing to which we routinely adhere are frequently broken by the camera and the microphone, placing the spectator in a situation of intimate proximity to people and things. Projected on the screen, objects and bodies can transcend their boundaries and exert a presence that is sensual and emotional as well as material.

We apprehend the physical world not from any fixed point, but in the process of walking and pausing and turning and looking around. Perception of the environment and the acquisition of spatial memory naturally coincide with the proprioception of our mobile bodies. MacDougall reminds us that: 

“The cinema operates in yet another way to affect the spectator corporeally through its construction of imaginary spaces and its evocation of real ones… Our sense of space in the cinema relies upon recognition, but also upon piecing together the shots into a larger imaginative structure. In participating in this construction, we are drawn further into the film in mind and body.”

MacDougall 2006, 24-25

THE SOCIAL LIVES OF URBAN TREES

A tree growing in the sidewalk is an architectural organism. It organizes its urban milieu and, through its body language and habits, gives definition to public space. Continuously adapting—to geophysical stresses, vehicular surges and emissions, sudden cloud bursts, and the unpredictable behaviors of other resident species—its form is never completed. Intimate negotiations between tree and built environment ensue as the growing plant accommodates itself to prevailing forces over a timeframe that is exceedingly slow compared with ours.

Initiated by one of us, Rachel Strickland, The Social Lives of Urban Trees is an experiment that merges environmental sensing with observational cinema techniques—inventing a cinéma vérité approach that takes cues from revelations which emerge from ongoing data capture. Video auto-portraits of urban street trees chronicle architectural, environmental, and sociological transactions of these colossal characters with their everyday surroundings and urban cohabitants. The project employs solar-powered cameras, microphones, and sensory instrumentation installed in tree canopies to reveal circumstances surrounding the morphological intentions, adaptive expressions, subterranean reversals, and ordinary routines of the tree. Pursuing a more-than-human approach to multispecies encounters in a densely populated city, this new genus of observational cinema mediates arboreal and human life—registering their disparate perspectives regarding shared territory, their respective architectures, socialities, mobilities, and temporalities.

Rendering the tree’s experience in counterpoint with perspectives of other dwellers and passersby, the result is a polylinear and polytemporal video/sound work which reveals territorial use and motion patterns occurring in timescales that otherwise elude human perception. While examining how multispecies interactions impact the well-being of the tree itself, we are also exploring ways to record and represent what the trees “listen to,” and what they may have to say. 

This work develops technologies for media capture, distillation, and presentation that extend the absolute limits of current possibility for long-duration data sensibility, environmental journalism, and video reportage. Our project title pays tribute to journalist William Whyte’s ingenious 1970’s film study of Manhattan sidewalks, plazas, and pedestrians, The Social Life of Small Urban Spaces—a work of enduring revelation to urban designers.

Treecorder is the name for the audio-video-sensor recording instrument designed for mounting in the canopy of an urban street tree for extended durations of a year or more. A modular system, the treecorder interfaces multiple camera-microphone units and electronic sensors with a single-board computer that controls recording and transmits data via 4G. An accelerometer tracks the movements of a windblown branch; a precision dendrometer measures hourly fluctuations in stem circumference; additional sensors record ambient conditions such as temperature, humidity, sky color and luminance. The computer can activate video and sound recording based on sensor data, or at specified times of day such as sunrise and sunset. In October 2019 a weatherproofed, electrically shielded, solar powered, and networked prototype of treecorder was installed and tested for two months in a large linden (Tilia americana) that grows in a busy courtyard on Columbia University’s campus.

Sensual Data: Extending the Senses of Cinema

A tree is marvelously engineered for withstanding environmental stresses, as well as for solving the problems of being large, fixed-in-place, and long-lived. One version of the tree’s story investigates its interactions with the environment and cohabitants, while lending expression to its character and coping mechanisms. The tree in its urban milieu has stories that not only escape the timeframes of ordinary human experience, but also disregard conventions of cinematic pacing. The next phase of this project will explore a process of merging environmental sensor data with video and audio signals to produce a new cinematic synthesis that reorganizes sense and sensing.

It is true that video and audio technology can record data in their own right. As media of cinema and storytelling, they furthermore have the power to articulate sensor data as sensual data. With The Social Lives of Urban Trees we are interested in extending the senses of cinema by incorporating environmental recording techniques that venture beyond the traditional apparatus of cameras and microphones. 

Deviating from data visualization and sonification techniques that represent mathematical information with abstract graphs or sonic maps and musical transformations, our experiment aims to directly integrate data from multiple environmental sensors with the perceptual immediacy of cinematic image+sound long familiar to moviegoers, rendering sensor data intuitively accessible through common filmmaking maneuvers. The fluctuating value output by each environmental sensor will be assigned to unique video and audio processing parameters, such that it continuously inflects the video or audio signal. What results is a movie stream that shimmers gently or undergoes subtle contrast and color tonal shifts without interrupting the legibility of the underlying scene [Figures 1-3].

Audio filtering is used in filmmaking to develop the sense of a physical space among sound elements. By adjusting knobs on a mixing console, the sound mixer can impart an illusion of changes occurring in the listener’s environment, and modify the perceived position, distance, orientation and acoustic properties of sound sources within it. A sound can be made to feel closer or farther, to turn toward or away, or convey the feeling that a large object has come between the sound source and the audience. By mapping the tree’s environmental variables onto audio filter parameters, we intend to translate and render these variables as a sensory experience that can be “felt,” and experienced in synchretic context with visuals and other display components.

Principles of montage composition, polylinear and polytemporal construction, along with an interest in mapping senses and biorhythms of the tree to those of a human body, all play into our approach for rendering data cinematically, while exploring a sensor-augmented approach to cinema production.

Tree and Soundscape

Sound makes the distance of the image possible, surrounding audients in the space of a world whose visible aspect is collapsed to a procession of discrete clips projected on a flat screen. Aligned with this image procession, the soundtrack propagates through real air to establish the dimensions of the narrative world and to assert continuity across scene divisions. 

The simplest relationship between image and sound is synchronicity—when a visible movement or change is accompanied by an audible effect, its impression becomes more vivid. Sound can also constitute the subject, source, or motive of visible action—it may not merely accompany the spectacle of a flock of birds taking off (commotion of flapping wings), but contain their cue (the backfire of a motorcycle) as well. Sounds are not confined within the boundaries of the screen—a sound emanating from outside the field of view might alert us to activity occurring elsewhere, or lead our gaze to a new vantage point. Sounds associated with one scene can overflow pictures of another to induce anticipation or to bind disconnected views in a common acoustic space. Because sounds assume the coloring of the particular location where they are produced, variations in timbre supply information about features of the landscape. The same fly buzzing around acquires a different aura when we transfer it from a cramped sleeping tent to a cavernous empty hall, or shoo it out the window. Acoustic closeups, like directional microphones, allow us to tune in individual sounds that would otherwise drown in the din [Balázs, 1948].

Sound recording with treecorder will support several strategies of development in post-production. First, as acoustic phenomena in real-time sync with visuals. Second, as a sonic substrate for projecting the tree’s environment into the audience space through a layer of signal processing driven by sensor data, detailed below. Third, as continuous, structured capture of long-term-average spectral sound, which is expected to reveal cyclical characteristics of the tree’s surrounding location. 

Ear-Based Situational Sense Mapping

Situational awareness is acquired by our bodies through the sense of hearing. Reverberance, time-delay, Doppler, filtration, and intensity effects tell us a story about the size, shape, arrangement, physics, and materiality of other bodies in the environment where we find ourselves; what is near and what is far; what is in motion and what is still. An illusion of this multi-dimensional experience can be achieved through the use of digital signal processing. By mapping our protagonist tree’s environmental variables onto such parameters, and applying them to site-recorded audio—essentially placing the tree’s environment at the knobs of the mixing console—we intend to translate and render what the tree experiences bodily into a form that humans can receive sensually and understand spontaneously.

Example 1:  humidity sensor triggers (video) image blur.
Example 2:  temperature sensor triggers (audio) high pass filter.

Extrinsic and Intrinsic Sensing

Sensors that measure temperature, humidity, luminance, and soil moisture are monitoring aspects of the tree’s environment. An accelerometer that measures branch vibration and a dendrometer that records hourly fluctuations in the tree’s girth track processes that are intrinsic to the body of the tree. Other examples of intrinsic sensing apparatus include sap flow meters, thermal imaging cameras, and ground-penetrating radar.

Considering soundtrack as the more visceral component of cinema compared with image, we speculate that it may prove effective to use audio filtering parameters for data that is more intrinsic to the tree’s experience and video processing parameters for data that registers ambient conditions.

Example 3:  temperature sensor triggers (video) contrast.
Example 4:  accelerometer (vibration sensor) triggers (audio) pitch shift.

Polylinear Construction

Departing from one-way linear cinema played on a single rectangular screen, this project pursues a cinematic paradigm that undoes habitual ways of framing things, employing ambulatory, multiple, and simultaneous viewpoints—such as humans exercise when orienting ourselves in physical space. Polylinear video affords a language that is more akin to architecture than conventional narrative formulas, for capturing and articulating a kind of experience that belongs to many different dimensions. While split screen composition is commonly used by filmmakers for displaying simultaneous events that are spatially remote, we shall specifically cultivate this multi-channel technique for articulating polytemporal timescales that intersect or overlap. 

Sound Spectra Sampling

Sonically, New York City is dominated by traffic noise. Patterns in noise density emerge on daily, weekly, and seasonal cycles as vehicular activity ebbs and flows. Employing an integrating-average sound pressure level (SPL) meter, we shall capture a continuous spectral data log which can be used to understand, graph, and ultimately resynthesize the traffic spectrum as audio, time-scaled to synchronize with polytemporal media. Proposed parameters: third-octave bands from 10hz to 20khz, 5-minute integrated averages. Predicted useful ranges: third-octave or one-octave bands, averaged over the span of 2, 5, or 10 minutes.

Polytemporal Timescaling 

Trees do not escape their predators by running away. Their movement is essentially growth, which occurs at a rate that is invisible in the scale of human time. Timelapse film has revealed the graceful choreography of plant growth which would otherwise elude our perception. But in the process of imaging this expanded timeframe that belongs to the tree, we and the other animals literally disappear, erased by our own mobility.

In the interest of transposing tree-time to human-time, we imagine a series of scalar transforms whose output values are significant for humans while their input values remain consistent with timeframes of consequence to trees. By condensing 24 hours duration to the interval of a musical beat, for example, the environmental noise profile of a day at the location can be perceived as a rhythmic pattern. 

FLOWERS, FRUITS, AND SEEDS

Some 40 years since William Whyte’s Street Life Project disrupted old axioms of urban design, The Social Lives of Urban Trees updates Whyte’s evidentiary methods—integrating image/sound capture with new sensory technologies. Different from the present profusion of environmental monitoring and datafication experiments, this exploration pursues a coalescence of digital sensing with classic observational cinema. The work’s unusual conviction is that the essence of cinematic structure is not narrative, but more fundamentally inheres in dynamics of framing and the translocations of an embodied gaze. Here cinema’s potential for articulating multiple points of view gives expression to the manifold perspectives of multispecies urban life.

During a pandemic quarantine when our presence in the world is either confined inside, or else exposed so as to be simultaneously dangerous and vulnerable, the yoga mat becomes for several minutes every day an unbounded place where like a tree I am both sessile and free to move. As an animal I enact my intentions by activating my muscles. The tree articulates its intentions through deliberate placement of new leaves and shoots. “Plants grow not so much in opposition as in contiguity with the ecological niche they inhabit, as evidenced by the maximization of their surface exposure,” writes phyto-philosopher Michael Marder. “A rooted mode of being and thinking is, then, characterized by extreme attention to the place and context of growth and, hence, by a sensitivity that at times exceeds that of animals” [Marder 2012].

Set forth as a prospectus for the next phase of an experimental project, this writing is itself a work-in-progress, a cloud of pollen released to the breeze, risking obliteration for the scarce chance of alighting on a compatible flower. Forest ecologist Peter Thomas has observed that pollination by wind “is like trying to get a letter to a friend at the other end of the village by climbing onto the roof and throwing an armful of letters into the air and hoping that one will end up in their garden” [Thomas 2000, 129].

Our experimental documentary production intends to culminate in a multi-stream video/soundscape for gallery or public space installation, along with open source specifications for an off-the-shelf treecorder unit that can support urban arborists and environmental journalists in new strategies for research and reportage. Long term goals include expanding the cast of tree characters, extending the timespan of observation from months to years, creating a website that lets people interact with polylinear video, sound, and sensor data from the treecorders. The project aims to influence scholars, engineers, planners, and designers interested in new ways of understanding and documenting the city, and appeal to an upcoming generation of urban foresters who acknowledge trees’ contributions to environmental and human health, yet may not fully grasp complex interactions that influence the health of the trees themselves. Children, too, who have special powers for entertaining other species’ viewpoints, are an audience we wish to reach.

APPENDIX

PROJECT TEAM AND ADVISORS

Jim Campbell, Jonathan Cohen, Seth Cluett, Patricia Culligan, Jordan Geiger, Kevin Griffin, Jeremy Hise, Jon Kuzmich, Golan Levin, Jeremiah Moore, Jane Nisselson, Nida Rehman, Richard Sheryll, Rachel Strickland, Brian Wilson

ACKNOWLEDGMENTS

Brown Institute for Media Innovation at Columbia University

References