Firstly we will consider strategies and methodologies for the design and composition of sonic narratives (soundtexts) in non-linear environments; for example the design of immersive soundscapes within installations that employ multiple parallel soundtexts; and in particular new compositional methods for building terrain based mobile location sensitive audio experiences.
But let us begin with a simple question of our location; how might our auditor position themselves in the vast and complex web of vibrations that form the sonic environment? The answer is both simple and fundamental ~ our sensorium demands that each and every one of us inhabits the epicenter of the sonic world; we permanently occupy a mobile sweetspot (to employ the parlance of the audiophile). There is of course no better place to be ~ whilst we share equally in the sonic commons we are simultaneously privileged as the absolute ruler of a unique sonic realm.
In effect, our senses form a ‘Procrustes Bed’ (1) upon which the palpable world is forced to comply. Therefore, that which we naturally assume to be comprehensive and exhaustive is simply a small portion of a vast spectrum that extends well beyond our perceptual hearth and home.
The second alarmingly simple issue is that of Immersion. Whilst achieving a convincing sense of immersion in the form of an artifact demands considerable skill (and generally piles of expensive equipment) it is ironically an inescapable condition of our quotidian experience. We are immersed in the womb, bathed by pulsing body fluids and maternal speech alike; upon issue into the world we are henceforth saturated in subtle vibrations and alarming noises whether awake and asleep, like it or not!
Be no afeared; this isle is full of noises,
Sounds and sweet airs that give delight and hurt not.
Sometimes a thousand twangling instruments
Will hum about mine ears; and sometimes voices
Tah, if I then had waked after long sleep,
Will make me sleep again; and then, in dreaming,
The clouds methought would open and show riches
Ready to drop upon me, that, when I waked,
I cried to dream again.(2)
To conclude this preamble; for all, who like myself are attracted to complex forms of symmetry, consider this; the soundscapes that we perceive as intricate and seamless natural compositions (a forest at dawn or the metropolis at rush hour) are in reality conglomerations of independent and unrelated sounds. What appears to the auditor as a structured and syncopated whole, arises from a vast array of largely unrelated sonic events, some intentional, many accidental ~ all are spatially displaced but all converge upon our ears which form the centre of the soniferous universe in which we are immersed.
In a complimentary fashion, the auditors themselves form another set of spatially displaced and by and large, independent sensory portals ~ these two spatial matrices are overlaid with the result that each auditor experiences a subtly different version of sonic reality!
It would be useful to recap and get the lay of the land with respect to what we understand (or at least assume) immersive audio to be! I began by deflating the issue somewhat, claiming our day to day experiences to be, by definition immersive, however whilst this is indeed the case we may bring a closer scrutiny to constructed (or cultural) forms of immersive audio which may have a variety of specific manifestations and meanings that extend what we normally experience.
For example, we could consider the special acoustic environment of a concert hall, taken in combination with the very specific listening behaviour of the (western) audience, as an attempt to differentiate the subjective experience of the auditor from the quotidian, amplifying a sense of immersion and presence.
Alternatively, we might consider the concert hall experience as relatively passive when compared to an auditor who performs, enacts or summons a spatial audio work into being via various forms of physical interaction; and through the investment of physical energy and intellectual curiosity develops a subjective experience that goes beyond the close attention and active listening of the concert audience.
Deep immersion.
As a way of getting our feet wet this section examines the construction of sound narratives together with the mechanisms for engaging with and experiencing, spatially distributed audioworks. I shall use my own creative and research projects that exhibit various forms of Immersion to tease out these issues, using four basic approaches that are categorised as follows: –
1. Three dimensional speaker arrays with dynamic spatial audio.
2. Environmental and public soundart projects.
3. Interactive multi channel projects.
4. Location sensitive terrain based spatial audio research.
The four categories of soundart projects described as examples each deal with issues of immersion, with the construction of narrative and modes of interaction in a variety of ways. Each work category describes different strategies for composition and content development for immersive environments and identifies varying auditor experiences highlighting concepts of Linearity and Non-linearity.
The works also allude to the changes in concepts of sonic immersion (particularly in reference to Public Space) by indicating how the technologies of audio transmission and reproduction have increasingly enabled and encouraged forms of selective hearing (this is further illustrated in Sonic Commons below).
Sonic narratives; linear and non-linear compositional strategies.
Before we examine the projects in detail, it is useful to pose a few general questions about the nature of sound design and narrative construction. We typically experience cultural sounds in traditional linear forms, obvious examples being music, story telling, radio and cinema presentation. All of these have many features in common with the history of Theatre and employ standard dramatic devices that by-and-large operate with a strict temporal metre. As such, these devices generally cohere with our experience and understanding of the passage of time in daily life, arranged as sequential events and causal chains, operating in a linear fashion.
Both the Novel and Cinema have adopted poly-vocal perspectives that narrate reality from plural perspectives and temporal frames. The Ciné-Roman writing of the French Nouvelle Vague authors, such as Alain Robbe-Grillet is saturated in cinematic montage techniques that have gradually evolved a public familiarity with temporal distortions and reversals, the Flashback being the most iconic technique. Music too has seen many attempts to counter conventional linear compositional strategies. The stochastic and dynamic music of the Fluxus movement (John Cage et al) can be characterised as an attempt to make strange, following the concept of Ostranenie, a technique proposed by the Russian formalist critic Viktor Shklovsky (3) that problematised and prolonged the process of perception, temporarily estranging familiar objects and events in order to render them freshly visible. However it may well be that whilst de-railing conventional narrative or sequential structures operates strongly within the process of composition and performance, it may function only weakly in the process of reception and engagement on the part of the audience, therefore creating an asymmetry between the genesis and reception of the work. If we then consider the wide range of contemporary audio-visual forms that span the gamut from pro-saic audio tours to computer interactives and sophisticated sound art projects, what compositional and structural differences are apparent when compared to traditional forms?
The four categories of work examples that follow will attempt to identify and illustrate developments in contemporary sound art practice that contribute towards a sense of immersion, unpacking key issues such as; the transition from passive to active listening frequently coupled with performative listening ~ physically active participation in works; the development of spatially and environmentally distributed soundworks and the development of non-linear forms of composition and deployment.
3D Dynamic Speaker Matrix.
The Projects; “Transit of Venus” and “Drift”.
Full fathom five thy father lies,
Of his bones are Coral made;
Those are Pearls that were his eyes;
Nothing of him that doth fade,
But doth suffers a sea-change
Into something rich and strange.
Sea-nymphs hourly ring his knell.
Ding dong,
Hark, now I hear them, ding dong bell. (4)
“The Transit of Venus” (5) a collaborative project with Finnish artist Simo Alitalo, was commissioned by the Nykytaiteen Museo, Helsinki (KIASMA) and YLE (Finnish National Radio) in 1999; the “Drift” project was commissioned by Hull Time Based Arts, UK also in 1999. Both projects employ spatially dynamic 3D speaker arrays with audio trajectories mapped to X,Y and Z axes. Both projects also enfold private or covert sound narratives. Those in the Transit of Venus are encountered by using headphone equipped electromagnetic listening devices in the form of paddles which are used to probe a series of sculptural objects; whilst those in Drift emanate from a sculptural object within the work.
These works were generated on multitrack audio recorders with the 3D trajectory information en-coded via a Lake Technology “Huron” DSP engine. Each channel of audio content being painted into 3D space in a realtime, gestural manner by using a hand-held Polhemus 3D tracker. The resulting X,Y and Z audio trajectories generate a fully dynamic, multi-channel soundscape deliv-ered at a very high amplitude ensuring a powerful physical presence, spatially encapsulating both the auditor and a secondary layer of covert audio narratives. In the case of The Transit of Venus the visitor encounters a series of Ship’s life rafts, repurposed as pools containing several large submerged Pacific marine shells.
The act of literally dipping a headphone-equipped paddle into the water simultaneously reveals pools of liquid audio, which can be mixed by moving the paddle blade between the mollusk shells. This element of the project functions by using the simple principals of induction coupling. An induction coil housed within each shell is fed an audio signal from an amplifier (i.e. voltage). Matching coils are built into the paddle blades, which when placed in proximity to the active coil in a shell, is energized and sends a corresponding voltage to a small audio amplifier housed in the shaft of the paddle and thence onto headphones as audio ~ voila!
“Drift” is a dynamic three dimensionally spatialised soundwork installed in the fish hold of the “Artic Corsair” the last remaining Humberside sidewinder trawler, now moored as a museum ship, in Hull (Yorkshire, UK). The soundscape works directly with both the physical environment and the historical context and is structured to avoid a strict linear composition – so that visitors may choose to pass through the soundspace at their own pace, remaining to experience a full programme cycle – or simply sampling sections of the work.
The audio is projected by three types of speaker arrays; the principal matrix of x8 speakers delivers the dynamic spatialised audio, immersing the visitor with sounds derived from the Ocean, from Fish, the Fishing and Fish processing Industry.
Four sub-woofer units mounted in pairs at either end of the longitudinal axis of the hold project a panning low frequency wave-surge, literally capable of shaking the vessel!
Finally two small voice coils, housed in a central sculptural vessel in the form of an Ark, recount the English vernacular names, with a counterpoint of Latin Classifications, of fish from Australian waters – i.e. all fish which the “Artic Corsair” would never have netted. (6)
These two projects play with concepts of Immersion in literal, conceptual and metaphorical ways. The encounter is extremely physical, subjecting the auditor to a massive crash and surge of swirling audio, projected at visceral levels but beneath the surface, quieter water and quieter tales lie. Both works are multi-layered aqueous metaphors, the one being composed of life rafts, shells, paddles and water; the other drawing people down into the fish hold of a Sidewinder trawler, itself floating on the River Humber.
Environmental and interactive systems.
The Projects; “Meta-Diva”, “Theorem” and “Lotus”.
Why is Soundart like a CockRoach!
My interest has always been to design and build arrays of relatively simple devices, with robust and minimal technological components that when taken in aggregate develop apparently complex and almost unrepeatable sonic behaviours. Bearing this model of simplicity in mind, I am drawn towards the metaphor of the Cockroach, a species that has evolved a system of distributed nerve ganglia and low central processing ‘overheads’ rendering the creature simple, robust, and thus ecologically successful. So to an extent my solar powered environmental audio works emulate the salient features of multiplicity, simplicity, autonomy and frugality but even so are able to deliver an extremely rich, complex and immersive soundscape, which naturally is unscripted and to a large extent unpredictable.
“Meta-Diva” is a solar powered environmental Sculpture that generates a digital audio soundscape. The structure of the sculpture employs the metaphor of plant biology and consists of a grouping of thirty units designed for installation in a Lake setting (as in a bed of reeds). Each unit comprises a 3.5 metre tall ‘stem’ that supports a small solar panel and terminates in a flo-rette of eight small exponential speaker horns. The audio in each unit is provided by miniature digital audio chip, coupled to a digital timer, set individually so that each of the thirty units have a unique time signature. The audio chips contain short samples of natural history sounds, bird and insect song and frog voices.
This combination of multiple sound sources, individual time signatures, file lenghts and the fluctuations of the solar power supply give the soundscape an un-cannily natural presence, that gives rise to an almost emergent behaviour in which, although we might hear individual elements repeat. The overall soundscape forms an infinite mix, somewhat like the always familiar, but never repetitive sounds of a creek. In operation, the soundscape blends so seamlessly with the natural acoustic ecology it is extremely difficult to distinguish the artificial from the natural background.
“Theorem” is a broad-brush audio-portrait of the Paul Sherrer research laboratories and its scientific community. This prototype work, developed during a seven-month residency, was integrated into the research and development resources of the PSI Laboratories (Villigen Switzerland) with the audio content being drawn from a variety of sources that reflect the intellectual life of the Scientists, the physical lab environments and the broader environmental ambience that together form a large sound library.
Collection techniques include conventional microphony; electro-magnetic recording and vibration recording; brief interviews recorded with the Scientific Staff of PSI who outline their concepts of ele-gance and beauty within the Scientific paradigm and offer examples of elegant equations; recordings of the wider rural environment in which the PSI Laboratory is located, including a tonal palette of Swiss cowbells!
Structurally “Theorem” consists of an ensemble of individual audio and sensor units linked by a computer network.
The acoustic behaviour of the ensemble is modeled upon the subtle balance of order and chaos found in the natural environment. The work emulates the situation where we are surrounded by multiple individual sound sources, belonging to a variety of acoustic types (for example bird song and insect noises) most of which share some type of collective or community behaviour, for example call and response in bird song and excitation in insect communities.
Theorem is designed to operate as an ensemble of some forty modular sound sculpture units spatially installed in an outdoor setting at the lab. Each unit carries an identical cache of audio files structured as a hierarchy of playlists ~ in effect a family of sounds. The individual files that make up a playlist are designed to have different durations, which naturally develops complexity in the soundscape.
Sonic variation in each unit and change in the overall audio behaviour of the ensemble’s composition is driven by two principal criteria, human proximity and Cosmic Muon (7) detection rates.
Human proximity detected as sound and movement, excites the ensemble, waking the array from sleep, or interrupting and re-setting the units during play. Once an individual unit receives an excita-tion-command it signals the ensemble to individually select an appropriate play list and subsequently select an individual song file at random. Thus, the ensemble flocks to a common playlist with all units playing songs from the same family in chorus.
Thereafter sensors in the work detect variations in the rate of Cosmic Muon radiation, which steer the audio content selection process away from uniformity, gradually diversifying the material played across the array, generating an increasingly heterogeneous and unpredictable composition
“Lotus” is a large public environmental sound-sculpture, that floats in a lake. The soundscape is delivered by a multi-part, solar-powered system that distributes a series of traditional ten Japanese haiku poems (each comprised of three lines) via miniature digital audio caches.
A solar accumulator and timer circuit was developed to permit operation without recourse to power storage. By employing the slight fluctuations in solar power availability, the work generates variations in the timing of audio chip activation with the result that the ensemble produces an infinite-mix of thirty voice phrases, never repeating exact patterns. In this manner the ten original Haiku are fused into one über Haiku. (8)
All three projects all emulate the structure of natural acoustic ecologies, being composed of multiple voices, spatially dispersed throughout a physical terrain. Furthermore, the audio content is syncopated primarily through naturally occurring stochastic processes, for example, the variability of photo-voltaic power from sunlight or the detection of Cosmic particles giving the works a quasi-organic behaviour. Parallel to the visual experience of Sculpture, about which one may move, viewing the work from all angles, an encounter with these soundscapes is also contingent upon the position and movements of the auditor generating a sense of surroundedness, of depth and of ‘parallax’. So strong is the perception of audio naturalism in, for example “Meta-Diva” that on more than one occasion visitors to the work have point blank refused to accept that the soundscape issued from the installation, steadfastly claiming it to be real.
Spatially arrayed, multi-source, multi point, multi-channel.
The Projects; “Seed”, “2 or 3 Things…” and “The Naughty Apartment”.
“Seed” made its debut in 2002 at “SoundCulture” in Pheonix Arizona; “2 or 3 things…” was developed for the “School of Sound” festival at the Southbank complex, London in 2003 and the “Naughty Apartment” was commissioned by “Experimenta” for its “House of Tomorrow” exhibition, Melbourne 2003.
Silence is Golden….
One could be forgiven for imagining these works were developed as ideal soundworks for a Museum context, as at first encounter they appear to be totally silent! Only when equipped with a special reader can the viewer/auditor reveal elements of the soundscape.
All of these examples incorporate highly developed narrative content presented in multiple-parallel, simultaneous audio-streams. They are characterised not only by the performative role the audience must adopt to encounter the audio content but how they choose to negotiate a complex but spatially dispersed, composition equipped with a device that only offers a single point of audition.
“Seed” is a sonic installation that metaphorically collides our agricultural lexicon of the minefield with the narratives of the Old Testament and the contemporary disasters of military and ideological conflict.
It does so by inviting the viewer/auditor to literally enter a sonic-minefield where visitors equip them-selves with a simple mine detector that allows them to search for and listen to the sonic terrain emit-ted by the mines.
Perhaps to their surprise, the small facsimile landmines, each resting at the centre of an Islamic prayer mat, do not voice strident political commentary, the sounds of battle or doctored media grabs! Instead the encounter is with a sonic world of Arabic music, some ancient and some contemporary, overlaid with voices, in Arabic and English, which enunciate the ninety nine attributes of Allah, each attribute supported by a brief extract from the Koran.
In English we speak of mines, sown in fields or laid somewhat akin to an egg, or perhaps a cunningly laid trap. This is a domestic and agricultural lexicon whose familiar words belie the barbarous intent of these small kernels of violence. Mines are ontological devices; they lie in wait for the future! Such a concept is resonant with the Old Testament parable of the sowing of seed, in which the germs of the future are broadcast, as if by chance, across a varied range of terrain, some fertile and fruitful, and some stony and barren ~ an ecology of destiny.
In a more obvious combination of the fruitful and the fatal, we might recall the recent aerial seeding of Afghanistan with small yellow packages, some round and some square, some containing food but others deadly ordinance, actions in which cynicism and incompetence are indistinguishable.
Whilst the physical geography of Islam acts as the historical context for the mytho-poetic spaces and narratives of the Old Testament so to it acts as a repository for hundreds of thousands of landmines – a testament of the failure of military solutions to generations of economic and political instability. Should we heed the adage “As you sow, so shall you reap” then an optimistic future in the region is less than assured.
“Seed” therefore proposes a place of complexity and ambiguity within which to contemplate the sim-plistic and unilateral position of current military and political events. It is after all sobering to consider that the death-toll inflicted by landmines (principally in the developing world) is equivalent to the appalling destruction of the World Trade Centre – repeated five times each year.
Fictionalising Fiction!
In the beginning; as a thirteen-year old lad I was once left to kick my heels for the day in central London, along with a couple of friends. Being both wary of getting lost in the capital and having little or no money we decided to visit the cartoon cinema, housed in those days, on the main concourse of Victoria station. Somehow, our encounter was not, as we had imagined, with Bugs Bunny or the Invisible Man, as it was each Saturday morning but with something indescribable and utterly alien. I left the cinema with mixed emotions no longer an innocent for I had seen Godard and my experience of cinema had changed irrevocably!
Many years later (in 2003) whilst a Visiting Fellow at Stanford University I constructed a miniature audio project entitled ‘One or Two Things…’ (9) that was subsequently exhibited at London’s South Bank Centre as a feature of the ‘School of Sound’ (10) festival (April 2003).
This work, the conceptual forerunner of “The Naughty Apartment”, consists of a series of seven miniature reconstructions of my favourite scenes from Godard films, each encased in a small glass dome mounted atop a camera tripod and equipped with a hidden audio transducer. Visitors equipped themselves with a hand-held device sporting a magnifying glass (to better view the minia-ture scenes) and a loop-antenna designed to pick-up the audio track accompanying each scene.
The conceptual issue at stake relates to how we deal and engage with the fictional spaces and narratives of cinema – in particular how we incorporate and ‘carry’ these fictions within us. During the construction of ‘One or Two Things’ I realised that I held two wildly inaccurate assumptions.
Surprise number one came as I attempted to locate the scenes and sequences that I recalled so well. I was to discover that they, for the most part did not actually exist in Godard’s works per se and the images that I had internalised and held so dearly, were in effect hybrid forms, embroidered, imploded and superimposed; this was somewhat sobering!
Wildly inaccurate assumption, number two; as I attempted to transpose these filmic narratives into miniature sculptural dioramas a second form of slippage occurred as I realised that the discontinuous narratives that constitute film proved difficult to congeal into a sculptural tableau. A filmic event constructed via montage appears strangely like a Medieval continuous-narrative frieze when it is recruited as a three dimensional form; the temporal flows of montage frozen into a static form. Even more complex is the transformation of (the memory of) filmic audio. The soundtracks were forced to comply to the Procrustes bed of my memory, re-constituted and neatly sutured into place with the precision of digital surgery, creating a Chimera, part the work of Godard, part that of my none too precise recall.
https://www.sonicobjects.com/index.php/sonicobjects/more/quint_de_loup_ii/
“The Naughty Apartment” (11) attempts a similar approach to fiction – but here the original being a text, is even more open to the re-constructive imagination – the principal irony being the central venue of the novel was and remains a physical architectural reality!
I am one of those readers who become obsessed with certain authors and who rapidly exhaust their oeuvre, naturally becoming despondent when the supply of novels dries up!. The Russian writer Mikhail Bulgakov is one of these passions and in particular, the magic realist novel “The Master and Margarita” (12) which was completed in 1940 but only published in 1966 and which caused a sensation in Soviet culture.
To cut a long and complicated story short “The Master and Margarita” is an allegorical and darkly sa-tirical tale of a visit to Moscow by Satan and his entourage to conduct Satan’s Ball. Much of the ac-tion unfolds within the writer’s own apartment (the infamous apartment No.50 of 302-bis Sadovaya Street). As fate would have it the Bulgakov apartment still exists and currently enjoys a shrine like status – a site of pilgrimage for Bulgakov buffs.
“The Naughty Apartment” is for me an exploration of architectural space in the realm of fiction and fiction within architectural space! I have never visited No.50 of 302-bis Sadovaya Street in person but my good friend Andrei was able to draw a rough plan of it on the back of an envelope between drinks. This was close enough for me as I had for many years in my imagination, inhabited the rooms of that apartment, I knew the creaks in its floors and kitchen odours!
Each of the eight miniature rooms, spread before the visitor in the sculptural installation, contains a vital scene from the novel, the dramatic events elaborated by an audio narrative, written in the very same apartment building ~ perhaps as Bulgakov would have it, in the 5th Dimension!
For the technically curious, the transmission of these soundscapes is simple operating with the same principals of induction as employed in the “Transit of Venus” and “Seed”. Each of the laser-cut acrylic model apartments, miniature film sets and facsimile landmines conceal a simple wire loop antenna, which although ‘powered’ by an audio signal in fact emits only a weak, local electromagnetic field (i.e. the models do not make any sound per se). Each hand-held unit contains another coil-antennae that when placed in close proximity to the model’s antennae is energised by the electromagnetic field. The resulting electrical signal processed by a small audio amplifier being sent to headphones as audio. Whilst technologically simple this method of audio transfer via electromagnetic induction is beguiling in operation, a re-enchantment that resonates with the visual fascination for the miniature dioramas.
Seeking Immersion and performative listening.
The principal challenge in all of these works is how to construct an engaging sonic microcosm that will focus and sustain the attention of the audience without the prop of a traditional linear narrative sequence. In each of these works there is no overall composition, rather thematically grouped audio flows simultaneously from a physical array of multiple sources. The work may only be negotiated, source by source in a type of performative listening, structured at the whim of the visitor.
How are audio narratives designed for such situations and how do Auditors choose to navigate such works? Imagine being in a profoundly dark room with only a flashlight capable of a narrow focus. The room is full of detailed objects and texts, all of which have strong associations and meanings in common. A visitor will survey the surroundings, either randomly or with some structure, encountering fragments here and there but all the while trying to draw the visual cues together into some form of spatial and narrative map.
This sleuth-like encounter is even more complicated when experienced in the sonic domain. Whereas in the visual world, objects are frequently, perceived as a Gestalt, sounds (and soundtracks) are both dynamic and transient. In an environment composed of an array of multiple and spatially dispersed sound sources an Auditor must perform extremely complex mapping tasks that demand the memory to resolve content in both spatial and temporal domains in order to apprehend the overall fabric of the work (see The Art of Memory 13 ).
From the composers perspective the audio content although linear and narrative requires a treatment that acknowledges an essentially non-linear (or more accurately non-sequential) audience behaviour and experience. Therefore, the content needs to be robust in that it must allow for random points of entry and exit whilst remaining intelligible and compelling as a narrative compositional structure.
To achieve this the work must be thematically coherent but achieve this without imposing a hierarchy ~ thus facilitating a situation in which narrative fragments begin to coalesce into relation-ships.
In reality, the engagement with the sonic content is to a large degree facilitated and directed by the sculptural elements. Each of these works exhibit a strong relationship between the aural and visual content and employ visual foci as strange attractors to initially attract, then maintain attention. Physiologically and psychologically, the linkage between auditory and visual cues is particularly strong, psychoacoustic effects frequently compensating for or correcting the sonic image to corre-spond with environmental expectations or visual references. We only need to consider the voice of the ventriloquist that appears to emanate from the dummy (or the sound of a jet aircraft when ren-dered in the horizontal format of a 5.1 system i.e. no vertical axis but which still appears to have verti-cal elevation) to understand the synthetic and corrective relationship between vision and audition.
The Nomadic Ear ~ journeys in location aware spatial audio.
The AudioNomad (and prototype Sonic Landscapes) project may be simply defined as virtual and/or augmented audio reality systems (14) that adopt a naturalistic or landscape metaphor of our sonic ex-periences, operating via a metaphor of sonic-cartography and able to co-locate virtual audio with physical features of the environment. The following will highlight some of the methods and chal-lenges encountered during the development of the AudioNomad system, illustrated by two classes of project; location-aware, surround-sound installations mounted on ships and mobile handheld devices for individual pedestrian use.
AudioNomad for beginners….
In AudioNomad sound functions conceptually to reveal information or create an aesthetic effect, si-multaneously immersing and situating the auditor. AudioNomad comprises a series of cross-disciplinary art and science projects working with the concept of GPS-driven, location-based audio applications. The project outcomes generally take the form of artworks (interactive installations and exhibits) that enable a user or audience to experience a virtual audio world that is intended to be per-ceived as though it was situated within the real world, in such a manner that the user experiences the synthetic sounds as apparently originating from real objects. The audio content often entails a combination of oral histories, archival audio, local historical information, field recordings and music related to the general cultural context and specific site locations. The outcome is a culturally significant, temporary artifact, generated by a creative synthesis of novel location-based audio composition tools, global positioning systems, audio technologies, and software engineering.
Conceptual Origins ~ Tristram Shandy.
There is a marvelous passage in “The Life and Opinions of Tristram Shandy” (15) that describes a unique map, made at one to one scale; that is a map made to fit exactly over the physical features it represents! The AudioNomad research programme operates a sonic cartography with very similar characteristics, due to the potentially vast scale of the geographic area available to the GPS enabled system and amplified by the fact that the sound composition is performed by the mobile presence of the user traversing real geography.
Conceptual Origins ~ Ars Memoria.
Yet, another literary source provided the conceptual impetus for the development of a sonic cartogra-phy able to seed a physical environment with virtual audio memories.
The storage and retrieval of audio content within a complex soundscape, virtually associated with real landscape objects, has its precedence in the classical mnemonic system for storing rhetoric. In “The Art of Memory” (pub. 1964) Frances Yates (16) paints a vivid picture of the antique technique that enabled Orators to place memory objects (such as lengthy quotations) within the labyrinthine spaces of classical architecture. By visualizing an architectural interior, real or imaginary, the speaker might place here a red cloak over a sculpture (as a mnemonic trigger) and there, a sword on a table to locate yet another passage. By memorizing a stroll through this virtual architecture an Orator could retrieve a vast amount of correctly sequenced rhetoric.
The AudioNomad project transmutes such imaginary architectural space into the cartographic space of a digital map (itself a representation of the physical site of the project) and develops a complex sonic landscape by and assigning soundfiles, trajectories and other acoustic and navigational properties, at multiple locations within this virtual domain.
Whereas the classical rhetorician would re-play a walk through an imaginary architecture, to sequen-tially retrieve the elements of a speech, the participant in an AudioNomad project literally walks in a real environment, their position and orientation driving the software to render an immersive and dy-namic soundscape via surround enabled headphones. The user perceives individual audio events to be ‘located’ at specific points in physical space and as these share similar acoustic properties with the surrounding ambient sound a seamless nexus is formed between the real and the virtual suggesting a type of parallel audio world, in which memories of particular sites are invoked alongside contemporary reality.
“Syren”, a shipboard open speaker augmented audio environment.
Alternatively the AudioNomad system can be successfully deployed at a much larger scale in the case of massive surround speaker arrays mounted on large mobile platforms, such as the ship-board works “Syren” and “Syren for Port Jackson”. These shipboard augmented audio reality projects allow passengers to link the multichannel surround-sound experience with the visual surroundings. Geo-spatial information, automatically accessed as the ship navigates the electronic charts, is associated with the ship’s track, via a high-resolution GPS system coupled to a digital compass. This positional information is in turn be used to render a surround-sound, 3D soundscape corresponding to proximate physical features.
Syren was first deployed on the passenger cruise ship Opera on the Baltic Sea as part of the International Symposium on Electronic Arts (ISEA) in August 2004. A subsequent development entitled Syren for Port Jackson was run over three days in March 2006 in conjunction with the conference New Constellations: Art, Science and Society at the Museum of Contemporary Art in Sydney. This project modeled dense layers of aural content on and around the city waterfront, juxtaposing archival material with oral histories and wild fictions with political critiques.
These maritime projects test non-linear compositional methods and the strategies of virtual sound-scape design at a much larger scale than the intimate grain of handheld mobile, location sensitive systems, delivering a contemplative synthesis between soundscape and landscape.
How do the compositional scenarios and creative objectives of AudioNomad differ from the sound sculptures and installations previously described, which in various forms seek to immerse, surround and spatially direct our listening attention? Further, how may we differentiate them from our experience of mobile audio in the public realm, which is increasingly tailored to individual needs and preferences?
Traditional audio composition, manifest as music, radio and cinema for example, operate in a linear manner and are constrained in the temporal realm and generally employ standardized dramatic tropes to convey meaning. Moreover, these media forms, developed in and for a non-place, for example the utopia of the concert hall, or the everywhere and nowhere world of radio. Even when we begin to consider the conjunction of audio content and site, in the guise of the Protozoan form of mobile audio narratives ~ it is evident that the numerically activated Museum audio tour, is essentially a talking catalogue that draws visitor along a pre-selected pathway to each P.O.I. (Point of Interest). A linear, non-branching route, fixed in time and space, with a sequential narrative proposing audio content as a didactic experience and a user interface that is at best awkward.
In our quotidian reality non-linear experiences are hard to define and therefore difficult to come by! Possibly the most prosaic example would be Hypertext with its associative branching networks, now ubiquitous in the Internet. Another familiar form is the Labyrinth style computer game that demands ‘Lab-rat in a maze’ type navigational skills! Could it be however that we are selling ourselves short by defining these forms as non-linear, whereas they are in fact rather complex, multi-choice structures composed of a myriad of sequentially linear experiences ~ heavily branched ‘trees’ composed of linear twigs?
Where the AudioNomad project proposes a departure is firstly in the embrace of non-linear (or semi-non-linear) compositional structures, secondly in the convincing perceptual fusion of virtual sounds and real physical locations and finally with a user experience that is seamless an unobtrusive.
The term semi-non-linear is an acknowledgement of the pragmatic constraints imposed by the implicit durational nature of sound itself but more properly, the physical limitations of pedestrian navigation. There is an obvious and real difference in the capacity to freely move within the virtual space of a digital cartography (i.e. the AudioNomad editor) and the navigation of obstacles in the real world, ruled by the laws of Physics and hopefully by common sense!
It is possible to design a sonic cartography for a green-field site (such as a featureless expanse of desert) that in theory provides almost infinite permutations in navigation and therefore equally infinite performances of the spatial composition. However, the concomitant lack of visual features and as-sociated cultural references of such a tabula rasa radically diminishes the motivation for particular navigational choice. Therefore, most real-world compositions can proceed with a series of practical assumptions, informed by cartographic data, that generate a set of design constraints concerning the users behaviour and potential movements ~ the hunch being that the average user will not attempt to walk through walls, walk on water or leapfrog from one end of the compositional space to the other!
The design of AudioNomad soundscapes on one hand hovers between the uncertainty of knowing the precise route that will generate the real-time spatial composition and on the other, the purposeful creation of thematic audio zones associated with physical sites as well as the related web of potential navigational options. Each manifestation of the composition will naturally be different but in the manner of strange attractors most will contain significant proportions of shared content.
The Sonic Commons; an embrace or retreat?
Privatisation.
Contemporary western culture takes such notions as the private and the intimate very seriously, re-garding them as both fundamental and natural rights. So closely linked are they to the basis of in-dustrial capital that it is easy to overlook the historical reality, where private space, as opposed to the public vis-à-vis is a relatively recent luxury commodity!
In the audio realm the communication technologies of the telephone and wireless broadcast have created and proliferated the possibility of intimate listening spaces within the public domain. Recent developments in mobile audio devices such as the cell phone and personal listening systems have amplified the transformation of the sonic commons, punctuating it with myriad imploded private soundscapes.
Such immersion in the self, such selective listening, is easily construed as a turn away from public and shared aural forms towards an individualised and commodified aural experience. This movement has strong parallels with the recent embrace of political and economic tendencies away from the collective and communal and towards a valorisation of the individual and the privatised.
The concept of aural privacy, once inextricably linked with either spatial isolation (a conversation in camera) or with furtive behaviour (whispering) now strikes us as remarkable. The internalisation of sonic narratives has an interesting precedent in the discovery of silent reading; for we forget that be-fore the 5th century the literate were also performers of written texts. The first known citation of silent reading was recorded by St Augustine in reference to a 5th century monk Ambrose.
“When he read his eyes scanned the page and his heart sought out the meaning, but his voice was silent and his tongue was still. Anyone could approach him freely and guests were not commonly an-nounced, so that often, when we came to visit him, we found him reading like this in silence, for he never read aloud.”(17)
To gauge the significance of this shift in behaviour imagine a London Tube at peak hour with the en-tire carriage intoning articles from the Times and the Telegraph!
Telephony; locatedness and public speech.
Whilst it is common knowledge that technological forms of sound reproduction have had a dramatic effect on the manner in which we experience sound in the public realm, we are less aware of the un-derlying transformations in relation to the spatial location, temporal displacement and the virtual elimi-nation of provenance that mark recorded and transmitted audio.
Murray Schaefers (18) Schitzophonic splitting of a sound from its original source en route to being em-balmed in a recorded or transmitted medium is at the very heart of both the temporal and Spatial dis-locations with which we are now so familiar. Schitzophonic audio therefore runs counter to the pow-erful and fundamental psychoacoustic linkages between the eye and the ear, forming the perceptual glue that instantly identifies a sound with its source and location. This disassociation of sound and source is enshrined in the history of Electroacoustic music as Acousmatiques. (19)
The original fixed landline (point to point) telephone represents our one of the earliest experiences of schitzophonic audio. Even so, the early telephone system marked the geo-spatial location of those in dialogue to the point that each correspondent associated the signal with both a personality and a physical surrounding and therefore to some extent, the telephonic act became a sonic bridge be-tween familiar sites. At each end of the line, an imagination of the distant site, a parlour with over-stuffed chairs and a mothers dress, a formal wood paneled office and the smell of pipe tobacco and so on.
Thus the landline partially diminished the spatial otherness implied by communication at a distance by frequently reinstating a supplementary knowledge of the distant location. Contemporary telephonic communication has however become increasingly de-territorialised and de-racinated. The mobile phone essentially promotes a dialogue between nomads, obliterating the concept of fa-miliar location or environs. It is not without irony that the first question posed during a mobile phone conversation is not ‘How are you?’ but ‘Where are you?’ with the inevitable response ‘I’m on a bus!’
Along with mobility the cell phone has initiated forms of social evolution (or devolution). Originally phones were mounted on walls their earpieces at head height ~ it was, of course, impolite to talk to a stranger in a sitting position, it was also considered improper to ‘chat’ on a telephone (something apparently Women were inclined to do). Early telephone companies went to considerable lengths to reserve the device as a business machine and in some cases strove to keep them out of private homes! Nineteenth Century telephone aficionados would be alarmed at the prosthetic-like applica-tion of Bluetooth headsets and the spectre of the glossaliaic pedestrian (merrily talking to invisible correspondents and gesticulating wildly). Such people, once shunned as mad, are now tolerated in some countries, although such behaviour is still considered socially unacceptable in many public spaces in Europe (the train system for example frequently demarcates areas where they may and may not be used).
Wirelessness, smallness and mobility.
With the tranny and the boombox the development of transistors delivered miniaturisation and ipso facto true portability, the consequence being that radio and phonographic replay now could leave the home (and the power outlet) and head for the streets, the beach and the ghetto. This Sonic Assault has two phases; Intrusive and Implosive audio. The first of these audio modalities might be considered invasive or at least expressive and is exemplified by the Ghetto blaster and its more recent incarnation, mega-bass low-rider vehicular sound systems. Whilst the old boy with the transistor glued to one ear, listening to the cricket (or the ball-game) is not considered as noise pollution ~ the dude with the Boom-Box is trying really hard! The Ghetto Blaster in effect re-ritualises sound in public space and makes an unequivocal claim on cultural space.
In marked contrast to the expressive nature of the Ghetto-Blaster we are currently witnessing an im-plosion of Audio-Worlds (as if in recoil from an overload of Urban stress) into the micro-acoustic-ecologies of the Walkman, the cell phone and the iPod.
This tendency initiated by the Walkman and now conferred upon the iPod nullifies the vis-à-vis of Public Space transforming collective experience into serial withdrawal ~ A retreat, perhaps a respite from the press of bodies in the commuter train, an escape from the pressure of being a (social-being) within the anonymous Crowd. The general and desired use of mobile entertainment audio is to isolate the user from anonymous public situations (Crowd) and transitory geographical/spatial situations (Transit) Public Transport being the ideal nexus. The audio-bubble effect also extends to the monotony of the gym treadmill, the boredom of air travel and ironically to the delights of jogging.
It is perhaps therefore not surprising that the popularity of mobile personal audio systems (and the concomitant desire to be sonically isolated) is less prevalent in non-urban contexts; the social fabric of smaller and rural towns generally lack the principal drivers, anonymity, crowding and their nexus, mass transit. The pressure to conform to recent audio fashion is probably also diminished by a re-duced exposure to advertising hype.
To be optimistic we might embrace the greater community of consumers and indulge in the simplistic embrace of the notions of the freedom of choice within the free-market economy of music(s), especially if we adopt the view that now all music is world music, a commodity form set free from ethnic and cultural boundaries by the corporate sector. We may now assert and affirm our individuality by the esoteric nature of our playlists, sharing them even, in generous acts that freely gives that which is not legally ours (sorry Sony records but thank-you Limewire).
That which remains…
The counterpoint to an audio world composed of myriad private mobile soundscapes is found in it’s negative envelope, that which remains as public aural space inhabited by those weak and fractured signals that escape from earbuds and headphones. Unlike the hauntingly somatic riffs of a street saxophonist, playing to no-one in particular, late at night these are transient B.P.M. signals just audible enough to attract the attention, but instantly discarded as irrelevant and redundant. The ear constantly and redundantly hunting for meaningful patterns, a mechanism reminiscent of British Army Intelligence audio torture, once practiced against IRA political prisoners!
Re-Situation and Re-Immersion.
Other recent technologies are however starting to reverse these paradigms of isolation and with-drawal from social and geo-spatial situations. Locative forms of media are beginning to ‘situate’ the participant in a geographic and cultural context at both the theoretical and experiential level that po-tentially might reinstate an electronically mediated vis-à-vis.
Audiotours and audio informatics in public space.
Conventional audio-tours typically associate specific information to an object or site via a simple num-ber tag, delivering a didactic narrative in mono (sometimes stereo). In general tours are structured as a linear programme both in terms of narrative and physical movement. More sophisticated systems employ infra-red (or similar) to automatically deliver the appropriate audio within a limited range of the object or environment to visitor equipped with Infra-red (I.R.) equipped headsets ~ As I.R. operates only in line of sight conditions this system can work well in architectural contexts where room spaces are delineated into thematic audio-spaces. Likewise Radio Frequency Identification (R.F.I.D.) tags or induction loop devices can be used to trigger files and can work well in Museum contexts that demand no more that didactic information at the correct point. This has the advantage of eliminating the user interface as well as reducing the necessity for strict adherence to the tour route.
More evolved ‘audio tours’, such as those available on Alcatraz or on the old Liberty Ship at Fort Mason in San Francisco Bay USA, begin to incorporate an awareness of the overall spatial context. In-formation mixed in stereo (and occasionally in binaural) situate the participant in the architectural context, often using descriptions of key features to synchronise the progress of the journey. Narrator voices both provide didactic commentary and navigational prompts with the result that participants begin to focus upon their physical position and synchronisation within the architectural context.
An engaging linear tour of this kind has the capacity to focus the participant to the degree that they become immersed, not solely in the audio as content but in a synergetic combination of audio as narrative and audio as didactic or navigational instruction. By responding to the navigation cues auditors are obliged to visually identify architectural and environmental features, meshing them with the narrative content and navigational information ~ if not immersed, then they are at least fully occupied!
Navigational Voices.
Creative projects develop this positional awareness by the introduction of fictive narratives and/or the amplification and distortion of spatiotemporal frames, as with Janet Cardiff, who employs the technically simple means of binaural recording (i.e. environmental ambiance recorded by in-ear mikes, in-situ) to recreate a hi-fidelity spatial rendering of space. These however are subsequently cut-loose in terms of position and orientation as the re-play devices are not location sensitive. This naturalism is overlaid with other fictions that rapidly develop a rich layering of physical ambiance and psychological space.
Other artists have pursued a more technically defined sense of location by employing GPS devices to tie audio events (compositions even) to geo-spatial position. Teri Reub and Ian Mott have both evolved systems that rely upon the coupling of laptops with hand-held GPS units to give an approxi-mate spatial triggering for stereo files but which do not have an immersive capacity or a physical ori-entation. Reub’s earlier work, built around more or less, linear walking track geographies presented the participant with relatively few spatial options to deviate from a prescribed path. Stories were col-lected and situated along the trail, able to be activated by large GPS trigger zones; a good solution where the accuracy of the technology is limited and when a precise correspondence between the audio content and the environmental context is not essential. Ian Mott’s early GPS works operate along similar principles. Audio and tracking technology, packed into wheeled shopping bags pro-vided a soundscape of mono audio, related to position but without orientation, distance or spatialisa-tion properties. (20)
Futures and Conclusion.
Edison’s Ars Memoria concept for the phonograph.
“Your words are preserved in the tin foil and will come back upon the application of the instrument years after you are dead in exactly the same tone of voice you spoke in then…..This tongueless, toothless instrument, without larynx or pharynx, dumb, voiceless matter, nevertheless mimics your tones, speaks with your voice, speaks with your words, and centuries after you have crumbled into dust will repeat again and again, to a generation that could never know you, every idle thought, every fond fancy, every vain word that you chose to whisper against this thin iron diaphram”.(21)
Edison conceived the phonograph plain and simple as a memorial device, a means to archive the transient voices of relatives as a sonic counterpoint to the family photo album. That the future of the phonograph was to rapidly evolve into a commercial device driven by musical entertainment is with hindsight an obvious irony, but one that Edison both missed and was resistant to. Naturally we should not overlook the fact that Edison was partially deaf! (22)
GIS worlds ~ the environment as a polyglot audio archive.
Notwithstanding the overwhelming use of audio recording technology harnessed to the commercial mill of the music industry, Edison’s presentiment concerning the mnemonic use of audio has a ring of truth. The potential to develop intelligent, interactive audio-cartographies, as outlined in the Audi-oNomad project, in which powerful GIS technologies (23) serve ubiquitous mobile devices may well see a world in which audio memories reside in every nook and cranny, attached to URL’s domiciled at the nodes of a global 10cm grid.
In the vein of Pygmalion, the Edison Company turned its hand to manufacturing talking dolls, produc-ing several thousand in the 1890’s. This uncanny embodiment of the voice in the mechanical flesh of a puppet is today transformed into a range of (not so smart) mobile devices; but devices that will within a short period of time, become intelligent companions, potentially far more sensitive to physical location and the invisible flows of data than ourselves. By way of a conclusion, please take a walk in NokiaTown with an AudioNomad scenario!
Read the Night out in NokiaTown treatment.
Footnotes: –
1. Procrustes, a figure from Greek Mythology possessed an Iron bed upon which he invited his visitors to lie. The taller guests who proved too long for the bed were subject to amputation, whilst the shorted visitors were stretched until they fit nicely!
2. Caliban in The Tempest by William Shakespeare.
3. Victor Shklovsky a Russian Formalist developed a concept of “Ostranenie” (or defamiliarisation) in literature which he explained thus: –
The purpose of art is to impart the sensation of things as they are perceived and not as they are known. The technique of art is to make objects ‘unfamiliar’, to make forms difficult, to increase the difficulty and length of perception because the process of perception is an aesthetic end in itself and must be prolonged. Art is a way of experiencing the artfulness of an object; the object is not important.”
4. Ariel’s song in the Tempest by William Shakespeare.
5. The Transit of Venus takes as its starting point the voyages of colonial expansion across the Pacific tracing a European sounding-out of a terra incognita. This is a meeting of different aural sensibilities, a contest of spatial and environmental perceptions and a collision of geographies; the Maori considered James Cook’s ship, the “Endeavour” to be a floating island, naturally they were correct in that it transported not only men and equipment but another cultural system!
The principal aim of the project is to construct an ‘Impossible Object’ in the form of a Sonic Cartography of Pacific encounters that critically examine the period of colonial expansion from historical and ideological perspective’s. The project is designed to operate across a wide range of modalities; sculptural installation, soundscape, radio works (commissioned for Australian and Finnish national broadcast corporations) and interactive web projects.
6. Drift is what the Ark did – aimlessly and noiselessly (as long as you disregard the menagerie!). Drift takes on another layer of significance when translated to French as derive. This is what members of the “Situationist Internationale” liked to do as a form of higher-order pastime (requiring about four days and fuelled by large quantities of high-octane alcohol). In both cases one is eventually washed up somewhere, tired, damp but still alive!
What Noah forgot I guess was Fish – I assume there were no fish on the Ark; maybe this means fish are free of guilt (which is perhaps why they are so good to eat!). So in a Jonah(esque) trope the Arctic Corsair has netted an Ark (a fish free cosmos) but one which constantly intones all those species of fish that never swam under the keel of the Arctic Corsair – all the warm vernacular fish names from my new Antipodean homeland.
7. Muon particles are formed by Cosmic Rays, which emanate from deep space and have high energy. When cosmic rays impact the nuclei of air atoms in the upper atmosphere they first produce short lived Pions, which subsequently decay into Muons that generally continue towards the Earths surface. I worked with the Muon Detection scientists at the Paul Scherrer Lab to develop a system that would control the distribution of audio content, according to Muon sensor counts in the “Theorem” project.
9. Hommage to “Deux ou trois choses que je sais d’elle (Two or Three Things I Know About Her) Jean Luc Godard (1966)
10. School of Sound < http://www.schoolofsound.co.uk>
11. “The Naughty Apartment” is the title of chapter 7 of Bulgakov’s novel “The Master and Margarita”.
12. “The Master and Margarita” by Mikhail Bulgakov, commenced writing in 1928, but only published in a complete final Russian version in 1973, Khudozhestvennaya Literatura.
13. “The Art of Memory”, Francis Yates, University of Chicago Press,1966.
14. Augmented Audio Reality refers to a system in which allows an auditor to experience ambient/local sounds whilst simultaneously overlaying these with additional audio information.
Virtual Audio Reality refers to a system that immerses an auditor in a dynamic and spatially active audio environment, which may or may not be linked to a corresponding visual domain (real or virtual). The audio supplied is intended as a total environment and supplants any local or ambient sound.
VAR is not essentially concerned with a functional relationship to events and objects in physical reality, it is best employed in totally VR environments or where there is a desire to diminish or suppress the links between the visual and the aural in the quotidian world (as in the iPod). AAR on the other hand has a vital concern to link synthetic audio events and compositional; strategies with aspects of the physical environment through which the ‘AudioNomad’ is navigating (whilst simultaneously navigating the parallel cartographic/sonographic software).
15. “The Life and Opinions of Tristram Shandy, Gentleman” (published between 1759 and 1767) by Laurence Stern.
16. Details of Francis Yates (ibid 8).
17. Confessions a series of thirteen autobiographical texts by St. Augustine of Hippo written between AD 397 and AD 398.
18. Schafer, Raymond Murray (1977). The Tuning of the World. Random House Inc.. ISBN 0394409663.
19. Acousmatics (from the Greek Akousma, what is heard) has its origins wi