We have learned the conventions of reading images since pictograms were carved in caves. As humans pictorially represented their understanding of the world, they learned to read metaphorical visual representations of history and the environment that consolidated their experiences and identity. The rules for understanding visuals are, however, more open to interpretation than those of language. Spoken and written language appeared as universal conventions. The universal rules of each language manifest cultural differences that reflect particularities between each group’s interpretations of their cosmology. For example, in Mandarin there are no specific tenses. This is linked to Confucianism which views time as circular and on going, as opposed to linear and limited. Poetry is the closest use of language to a pictorial representation of the world. Poetry plays along or defies the rules of grammar to convey emotions. Conventions for interpreting audiovisuals also change depending on time and location. German expressionism used theatrical mise-en-scéne to convey an existential anguish and Freudian revelations of the plot and characters. While, the auteurs of the French new wave adopted a more ‘realistic,’ fast-paced editing style to convey reality. When we analyze these films, we use these conventions for contextualization. However, when having an image and a word side to side without a context, we still draw more intricate interpretations from the image than from the word. The image, unless it is a symbol, does not have a universal meaning attached to it. Images leave space for interpretation that is affected by personal experience.
So far television and cinema have educated us with audiovisual narrative conventions. We understand that montages convey parallel space and time. A modern cinema viewer does not faint at the sight of a train coming towards her/or him on the screen, like it happened in the time of The Cinema of Attractions. Video games have trained a generation to understand that a non-anthropomorphic being on the screen is ‘you.’ In fact a whole new set of social interactions has sprung from online multiplayer games. It is easy for them to meet each other in the middle ages, fly and hang out with dragons and elves and in less than a second meet in the ‘real world’ and get married. The influx of audiovisual media in our daily lives makes it seamless for us to jump from a reality behind a screen to a tactile reality that we call our world. People are able to understand the spatial difference, yet we don’t necessarily see ourselves as different personas when we take part in each world. The way people interact in a ‘virtual’ and a ‘real’ spaces reveals pieces of their personality. Some people feel more comfortable expressing themselves in a videogame space than in their office. Even though, we are taught to differentiate reality from audiovisual media our self identification while participating in these realms is not necessarily disregarded. The I is still present while participating in audiovisual media, whether is watching or interacting with it.
We are in a time where the digital imagery in audiovisual format is not totally divided from the environment we experience. We are capable of embedding information into images. We are capable of attaching a location to a picture, rasterize a photograph, convert it into vectors and/or pixels and find patterns amongst shapes and colors that allow for us to draw conclusions. This makes facial recognition possible. Now Facebook suggests names for faces and google earth shows you pictures of your street. The tangible difference between the fantasy world behind the screen and the real world in front of it, is not that clear anymore.
Encoding and decoding written text changes our interpretation of it. We can access it easier, we can draw patterns and communicate faster. But language is ultimately a human construct. Images are the way we experience the world. Of course, there is sound, touch and smell. But if we are able to decode images, we are decoding part of our phenomenological experience of our environment.
Translating images into binary code goes beyond comprehension and enters the realm of interpretation and evaluation. Computers make no judgment, they interpret based on codes and deliver. We, humans, are the creators of the codes used to interpret these images. The deciphering of the visuals we experience is the ultimate bridge between linguistic semiotics and phenomenology. Perception, thought, memory, imagination, emotion, desire, volition to bodily awareness, embodied action, and social activity, are manifested through tactile and audiovisual perception. Decoding these experiences into a universal code will change our relationship to our environment and ourselves.
We can encode and decode images without attaching universal meanings to them. In order to allow for an individual to have unique audiovisual experiences, the audiovisual content that we transmit via binary code should be presented similar to how we experience it in the ‘real world’. This is what’s so exciting about developing audiovisual experiences using the kinect. One could either use it’s technology to tell us whether we are moving right or wrong, whether we are dancing properly or punching the right object on the screen or we could challenge our kinesthetic awareness beyond our rational inclination to contextualize and give meaning. I have been writing real world in parenthesis, because there is no absolute definition for the way all humans experience space and time. There is no universal definition for emotions, dreams and the physical relationship between humans and their environment.
We can use this technology to challenge the reality we experience, by being aware of our movement, observing people and places we usually don’t stop to look at, distorting our spatial awareness, playing with our focal and marginal visual awareness, distorting our self consciousness, deviating the purpose or intention of our actions, challenging our social interactions and everyday activities regarding our surrounding world and culture. This type of interaction with digital representations of time and space can change the constraining way we perceive our body in relationship to the environment and others. We could use digital images to change the way our faces look and potentially wear virtual masks. How would people treat us if we look like we belong to a different race or gender? How would we perceive others? What analysis can be read from facial expressions while we look different? We are already experiencing these types of challenges in virtual realities. The exciting time is coming when worlds blur and meet.