SoundSpace – Lit Review/Proposal

“. . .the art work . . . is no longer a static object or a pre-defined multiple choice interaction but has become a process-like living system.” Christa Sommerer and Laurent Mignonneau, Art as a Living System 1998

Advances in communication technology have caused fundamental changes in the way people interact with one another. This interaction often occurs with people in isolated environments, and what some may say impersonal settings.
On the other side of the coin to this mediation is the use of technology to enhance communication between people within physical proximity. “Systems enabling large audiences to interact offer numerous possibilities for entertainment, education, and team building”2.
Woolsey and Semper in ‘Multimedia in Public Space’, define public space as being, “Public; that is, they are social, people come to them with other people, to interact or to watch other people doing things. They are spatial; the nature of space and how people use it is key to their design. And finally they are timeless, or to put it another way, the timing of interaction is exceedingly variable and idiosyncratic with the participant.1”
In the American Psychologist, a report suggests that greater use of the Internet leads to shrinking social support and happiness, ”They found a direct correlation between participants level of Internet use and their reports of social activity and happiness. As their use of the Internet increased, the participants reported a decrease in the amount of social support they felt and in the number of social activities they were involved in. They also reported being more depressed and lonely.” (Isolation increases with Internet use, Scott Sleek, 1998).
”Teenagers spend increasing amounts of time immersed in television, video games, and music from their iPods-activities where they listen rather than speak. As a result, they don’t get much practice at communicating clearly with others, and they aren’t exposed to a wide vocabulary” – (Linguistic Anthropology, 2006)
This explains why further research has to be done into the way we use technology. Where and how it is used, and whether more interactive, better design can change these social trends in technology use to create more sociable environments, making use of communal space and for social interaction.’

NATURAL ACTION
A range of products and services are beginning to emerge that have more sensibility to the actions we as humans natural make. Rather than define the action we should make to get a response these technologies are allowing the actions we are already making to have stronger or more extended reactions.
Mega Phone a commercial service for engaging the public in interactive advertising, takes away a lot of the effort associated with other forms of mobile phone controls, no slow SMS typing just call and away you go. By using the natural action of vocals as well as the keypad people can control the visuals on their large-scale screen displays.
Another such example of user input via their vocals is Primal Source, an installation using audio detection devices to determine the volume levels of the audience. A light show in the sky was directly affected by the amount of noise people (estimated at approx. 200,000 over the course of the night) where making. The added benefit here is how the audience encourages the display and the display encourages the audience. The height of the show is reached through a collective excitement. This working together to get the most out of the installations is a nice example of public sharing enhancing the experience for all.
An interface that gained huge public interest this year due to it being featured in the musician; Bjork’s live performances is the Reactable, a round illuminated table interface based on a modular synthesizer.
The table is a multi-touch tangible interface, which works when you place special objects onto the table. The table will recognise it and play a sound or add or modify a sound already playing depending on the object placed onto the Reactable. Also moving these objects around will have certain effects on the audio. The table encourages collaboration to create electronic music.

Garth Paine is an artist who has done a lot of interactive installations that use interaction and audio. One of these installations was called Map1. This installation was held at Span Galleries in Melbourne, Australia. Map 1 explores ways in which humans develop and re-evaluate cognitive mappings of personal relationships with their environment. Human expectations, frustrations, desires and experiences are usually expressed to the outside world as a physical or aural response (Garth Paine, 1998). Map1 uses an installation that uses sensors that gather information on people’s location and also their physical movements within the installation space. This information is then fed to a computer controlled music synthesizer, which plays audio and alters sounds based on a person’s movement with in the space.

The Air Piano is a prototype of a musical interface. This interface is a long thin device that detects hand movements in the air above the Air Piano to play and control software instruments. The Air Piano use LEDs to provide feedback about the users hand movements. The hand movements can control the virtual keys and faders above the air piano, and example of this is if you placed you hand on a virtual key the length of the note is determined by how long you hold your hand on that virtual key. The distance and hand speed also control different aspects of the interface. The Air Piano virtual keys use MIDI messages so that the software know what sound to play, this also allows the Air Piano to be loaded with lots of different MIDI instruments depending on what the outcome required is. Even though it is a prototype of a music interface, this type of interface could be adapted to many different types of applications.

Mobile gesture interfaces a concept by Kitchen Budapest is using as the name suggests gestures much in the same way the previous examples used vocals, “by connecting our characteristic gestures with the inclinometer attached to the cell phone, we don’t need to send a text message anymore, but are able to reach our friends, acquaintances, and colleagues through a simple gesture.3” This application for a technology called accelerometers which are increasingly being built into mobile phones, is another great example of looking towards action that we make naturally and connecting them with natural responses.
Johnny Chung Lee has made a series of applications exploring the potential of the Nintendo Wii remote. The expense of setting up these systems is minimal when considering the possibilities of this technology. Johnny has done 3 examples. Firstly, multi-point interactive white boards where people can write on a wall similar to the use of a Wacom Tablet. This digital information can then be stored/translated to other displays. The other 2 systems Lee has explored so far are fingertip tracking, and desktop virtual reality displays. Each of these examples uses the abilities of the Wii remote in different ways, creating a very comprehensive toolbox for developers of interactive systems. The technology it uses is an infrared camera that will detect infrared light. While the camera and software available have limitation there are a lot of further application of this accessible technology.

PHYSICAL LOCATION/
ENVIRONMENT DETECTION.
Sonar isn’t exactly a new technology, yet a team at New York University’s Tisch School of the Arts in their project Urban Sonar have combined it with some other common technologies; Bluetooth, Arduino and a heart rate monitor to create an interesting device used for measuring the individuals proximity to other physical objects. This data is recorded and then visualized for later reference.
Radio-frequency identification or RFID has been used predominantly as a new kind of barcode system. One of its most successful applications is the octopus payment system used in Hong Kong for public transport. In addition RFID has great potential for creating interactive systems where identification of objects and/or people is a core theme. RFID has the ability to detect though other physical objects. There have been a number of interactive displays using this technology but in a very experimental manner.

ENVIRONMENTS
Renowned musician David Bryne explored the idea of bring an environment to life with music in his piece Playing the Building. In this work David has turned a building, The Battery Maritime Building, New York, into a giant musical instrument control via a piano. People are invited to stroke the keys on the piano each keystroke resulting in a disproportioned response in another room. We had a meeting with Melbourne based Eness to learn more about interactive environment design. They are definitely all about pushing what they do and developing new and interesting ways of creating engaging space. “We could continue doing what we have done already, there is a market for it. But that wouldn’t be much fun”. Eness develop a lot of their own software and hardware, like Pixile a 3D engine for wrapping projections around objects. The Pixile software also houses a sophisticated particle system that can take on any number of forms, they have variables like colour, speed, quantity, and direction. Pair these with a bunch of input controls like motion tracking, colour detection, audio levels and you have one powerful interactive environment.

THE SIGNIFICANCE OF MUSIC
Music has had it’s place in society for thousands of years, being found within every human culture known across the globe, today popular musicians are worshipped for their skill and talent. Therefore it’s no surprise that learning the mechanics of music during childhood can heavily work toward benefiting a child’s development. Understanding rhythm and melody can enhance a child’s co-ordination, motor skills and challenge their creative intellect. Whilst the slow process of learning to become proficient at playing an instrument can teach patience, responsibility and increase their self esteem when they accomplish their goal of (for example) playing a piece of music. Studies have also revealed that learning music can enhance spatial awareness, reading ability, verbal memory and mathematic achievement. ”This combination of experiences could have a positive impact on cognition, particularly during the childhood years, when brain development is highly plastic and sensitive to environmental influence”(Huttenlocher, 2002)

Creativity is a talent that is being sought more and more in every aspect of business today, the world is discarding the boring corporate blanket that has controlled the economy for so long and seeking alternative ways to go about business. Technology has enabled endless possibilities whilst also having created a generation unhappy with the rigid confines of a soulless working environment. Technology has also broadened the possibilities of being creative with music, making it far more accessible to be creative in sound with financially accessible software and hardware. The skill is enhanced further when other people/instruments are introduced and focus on teamwork becomes essential. Teamwork within a band creating/writing music involves understanding others wishes and forte’s and compromising to reach goals. Sport has long been the sole provider of teaching teamwork to children, but that need not be true. In any case working together to defeat another team could be seen to encourage competitiveness, where as working together to create is more aimed at self satisfaction, which we believe is more important than “proving yourself” via defeating someone else. One could easily argue that if more people played music together the world would be a happier place!

THE MUSICAL SEQUENCER
The idea of mechanically or electronically playing a musical sequence emerged late in the 19th century with mass-produced Pianola piano rolls becoming a reality in1896. Yet the mechanics of how those early piano rolls operated is still comparable to modern digital sequencers used in contemporary music. A piano roll is a roll of paper with holes punched in it. The position and length of the perforation determines the note played on the piano. The piano roll moves over the ‘tracker bar’, which generally has one hole for each piano key. When a perforation passes over the hole, the note sounds.

Today in the age of creating your own musical sequences on the home computer, the method of arrangement has taken on the form of a virtual modifiable grid, which tells the machine how long for and when to play a specific sound, this is known as step sequencing. The first mass produced and affordable sequencers or drum machines based on the step sequencing grid concept became available in the late 1970s and boomed in popularity in the mid 1980s with the emergence and interest in synthesizers and other digital advancements in music.

We have been researching ways in which the virtual grid of step sequencing can become a more tangible interface, in which the mechanics of step sequencing are immediately clear. The most obvious way would be to construct an actual physical grid so the user could actually see how the sounds are arranged in the arrangement’s entirety at all times. Further research into experimental musical interfaces has encouraged us to expand on the idea of a grid made of squares to possible work with other shapes that can fit together i.e. Hexagons.

Also inspired by research and our initial interest in creating multi user/public interfaces is our eagerness to explore the idea of creating a form of sequencer on a room sized scale. The idea of a group of people (a school group for example) working together to create a sequence of sounds depending how they interact with a space is something we feel could benefit a public exhibition space.

ACMI OBSERVATIONS:

Location
ACMI (Australian Centre for the Moving Image)

Description
Walk through of ACMI’s interactive gallery space, taking field notes along the way.
Aim: To note the demographics represented within the space; how people were acting in the space; and the physical space itself. In particular we were looking at how people were moving through and around the venue, and physical traits common to this kind of location that may be useful when considering the design of interaction.

Conclusion
The foyer is a well light clean space. The open space is inviting yet not overly exciting. There are some brochure stands, light box style displays, plenty of wall and floor space, escalators and an information desk. The floors and walls (which are typical of other venues of similar purpose around the world) rarely meet square the floor not a constant level. ACMI’s interactive gallery space is basically a long thin room with dividing walls placed through out depending on what the current exhibition requires. The space is dark with low lighting, this helps with all the projection works that a regularly held at ACMI.
The only lighting is on works or information signs, although the space does have lights to light the whole room if needed. All the walls and roof are painted black to aid with this, and the carpet on the floor is also very dark.

People that stand around in the foyer are in general waiting for friends or waiting to speak to one of the staff at the information desk. Some flip thought the available brochures, but nothing engaging occurs in this space. (A space with opportunity to be activated) The more formal downstairs space of ACMI is explored in general silence; very little chatter or socialising in general was observed. The “Correspondence” exhibition that was on show was a more serious show, it’s audience left to observe the pieces than interact. “Correspondence” was, for the most part, a film exhibition. The exhibition used partitioned walls, this stopped the audio from each film mixing and becoming a blurred mess of sound. Overall the space is a very adaptable space that could cater just about any type of interactive exhibition or activity.
People’s actions ranged from walking around by themselves, reading blurbs and skimming through film exhibits, to sitting and embracing as much of the short films as possible, to simply looking at the most visually enticing pieces. Families with children especially tended to skim the exhibition, stopping primarily for the more visually enticing and bizarre pieces.
When designing our interface this will need to be taken into consideration. Allowing for varying levels of involvement and time commitment to getting something out of the interface. As one would expect of such a venue the people within represent a broad array of demographics. There were older adults, younger adults, and children of both sexes. A few of trends emerged however. There was predominance in males as well as young parents with they children. Excluding one baby in a pram, the children range in age from around 6 – 12. When comparing a weekday to a weekend we see that the common factor is children. The parents get replaced with teachers or other carers however the 2 common factors are children (primary school age) and young adults aged 18 – 30 (predominance male).

Upstairs in the “Memory Grid” the general ambience is a lot more fun and tended to have much more children. Families sat in the small cubicles watching short films, roamed around, sat on the computers and played games, generally for no more than 5 minutes. There is no particular sequence to the exhibition so people just wandered to whatever stimulated them (or their children) first. The Fed Square styled architecture flows through into this room and has many angled walls and doorways. This is purely to keep it consistent with the rest of the building and is not a requirement for an interactive space.

ARCADE OBSERVATIONS:
Location
Galactic Circus
Description
Galactic Circus is an indoor interactive theme park mainly consisting of video games and other interactive entertainment. This environment is primarily aimed at children but does also attract an older crowd. In order to reach the best interface for our environment, we have done some observation of time based interface systems.

Aim
The aim of the observation at Galactic Circus was to observe which interactive device was the most attractive to people and how they were interacting with the device. This includes environment, visual aspects, interactivity and sound.

Conclusion
The environment was very colourful and bright with lots of flashing lights, with each device competing to grab people’s attention. There was lots of music playing in this environment; again this music was competing with music and sound effects from other devices. In particular we focused on a game at Galactic City, “Mega Mega Dance Dance Extreme”. Watching how people interacted with this dance machine reinforced the idea of our interface needing to be for varying levels of proficiency. While some were extremely skillful at responding to the on screen prompts others were a lot slower. When compared with what we are proposing we see that an interface that responds to peoples involvement rather that compares them with predetermined sequences will give the user far greater control and empower them to be creative and improvise.
Also watching people use this device we noticed that they didn’t watch what they were doing with the floor control, but they’re on screen instructions and responses. By using a floor controlled system we enhance the opportunity for people to look at and therefore interact with their fellow players, rather than faced by a simple screen display. From this we gained the knowledge that the most popular are the games that gave users the sense that they were really involved in the game through body moments and unique interaction controls.

KORG ELECTRIBE SEQUENCER
OBSERVATIONS
Description: Observing and filming people attempting to use a commercially available Korg musical sequencer with which they have no prior knowledge of how to use. To simplify matters and make it actually possible we informed each of the three users that they should concentrate on the bottom half of the unit (the dials and buttons on the top half are all concerned with sound modulation effects, delays, reverb etc), reduced the measure of the sequence to one bar – removing the need to figure out how to change bars and we also informed the users where the 8 sounds available to make a rhythm from were located on the machine.

Aim:
To identify how well people can identify and interpret the concept of a virtual gridded step sequencer from the Korg module’s interface which, it was expected, would prove to be difficult to quickly make sense of, even though it is known in the marketplace as one of the easiest to use.

Results:
User 1 – Steven, 18yo male
Steven quickly identified the play button and within 2 minutes through pressing of random buttons figured out how to assign the sounds to a place on the grid. However we distinctly got the impression that Steven didn’t really understand quite what he was doing, very unlikely that he had the virtual image of a grid in his head.

User 2 – Sarah 20yo female
Although initially intimidated by the amount of buttons and knobs, Sarah also noticed the play button fairly quickly, however it took a while longer to figure out how to assign the sounds to a place on the grid as it cycled through. Her final rhythm was very simple with only 3 of the sounds being assigned a place on the grid.

User 3 – Karen 48yo female
Karen was taken aback by the amount of options displayed on the Korg module and took an amount of time to simply start experimenting. Several times she indicated she was opposed to continuing as she felt she couldn’t accomplish anything. Eventually Karen accomplished a simple beat but really had no idea how she accomplished it.

Conclusion
We were actually surprised at the relative successes of the users in creating a musical sequence; its possible that the information provided to them before the users began was too specific for an accurate test of the interface. However upon questioning after the testing none of the users successfully explained how they were actually creating the sequence and since we are interested in making the grid of sequencing more tangible and an easily identifiable concept, the experiment was successful in proving that the interface of such a device could be improved. Steven seemed to quite enjoy making rhythms and we got the impression he would have been happy to sit and play with the module for some time.

PAPER PROTOTYPING
Description
To take our proposed interface solutions further we have engaged in series of paper prototyping exercises.
Testing the accessibility of floor based interfaces, we have used a com¬bination of black and various coloured paper arranged on the floor. The initial state of the display is black tessellating shapes filling the floor. On the reverse side of each black shape is a coloured alternative. Asking a range of people to enter into the space with no instructions on how to act. Wall graphics will provide all other information about how to use the interface as well as display reaction. As participants move through the space we will flip the black shapes they stand on to simulate an initial response to their actions. In addition to this visual response, we will manually trigger a corresponding sound on a Korg music sequencer. This sound will hold its position in a musical loop while that shape remains stood on. Certain colours will correspond with particular sounds.

Aim
We focused on the level of engagement of people within the space; how accessible the floor interface is as well as how intuitive peoples interaction with the system was. The intention of the testing was on gauging the level of complexity in musical sequences people are able to create with such a simple construct, as well as the ease of doing so. How people worked together in order to collectively achieve these responses and their interaction with one another was observed to gain better understanding of how to tailor our solutions to enhance these relationships. }

Conclusion
The paper prototyping showed us that our concept for our interface was very easy for the participants to understand, making it very easy for the participants to use and fun. We also found out that participants found the interface not to be intimidating and that they wanted to interact with it. Paper prototyping also gave us an insight into the way people would move with in the space and how people would react to the sounds they have activated. From the paper prototyping we didn’t come out with any major problems or need to make alterations to the concept interface.
PROPOSAL
Our proposal is to design a multi-user musical interface. The interface is to encourage people into physical interaction and social interaction with the intention to get people to creatively collaborate to create a music loop. The interface will also teach the fundamentals of rhythm and timing in music.

For this exercise we are proposing the interface be installed into a gallery space such as at the Australian Centre for Moving Image (ACMI) at Federation Square in Melbourne, although the interface can be installed into and adjusted to fit almost any sized space.

How the interface will work is when the space is empty there will be only illuminated dots of different colours on the ground. Each dot will represent a different instrument or sound. The amount of dots available will depend on the size of the space the interface will occupy. When a users walks over to a dot and users press their foot onto the dot a ring interface will circle around the user on the floor. This ring interface will be broken up into 8 segments, with each segment representing a beat in the music loop. Each segment will light up one at a time in a clockwise direction, this will indicate the timing of the loop.
To use the interface all the user has to do is press their foot onto any of the segments, this will light up the segment and turn it on. When the timer gets to a segment that it is turned on it will play a sound relating to the instrument dot the person initiated. This allows people to collaborate and work together to make music as each person can be controlling their own ring interface attached to a instrument. When a user walks out of their interface the interface will fade back to the original dot on the ground and the sounds initiated via the their interface will have faded out.

If a person walking into another ring that is already occupied, the ring will grow from eight segments into sixteen. Depending on the size of the space, each ring can occupy up to four people at anyone time, allowing four people to control thirty two segments for that instrument. This element of the interface further builds on the teamwork aspect.