Interacting with Media Architecture
1Introduction
Over the last years, urban environments and public places emerged as prime locations for deploying digital technologies. Besides large scale digital displays, an increasing number of media facades (see Figure 1) are embedded into the urban landscape (Bouchard 2007; Seitinger 2008; Winogard 2001), thus becoming more and more ubiquitous (Weiser 1991). In a common sense, the term media facade describes the idea of turning the facade of a building into a huge public screen. This is achieved by either equipping its outer shell with interactive, light-emitting elements or by projecting digital content onto the building (Bullivant 2006; Haeusler 2009; Schoch 2006). The display might appear as a secondary skin of the building. Media facades can be classified based on different characteristics and properties. Among others, these might include their technical composition, as well as the main principles of how media content can be displayed. With the increasing number of media facades, urban environments are on their way to being turned into dynamic and programmable surfaces (Greenfield/Shepard 2007). Media facades provide a new way for accessing information at a large scale in urban spaces. Due to their technical requirements, their size and resolution, they can make information of different granularity accessible for a large audience. Passersby can not visually access information but often also interact with it with the help of various interaction techniques, that will be described in this chapter. The visualized information can range from low resolution ambient visualization of data such as energy consumption or traffic to high resolution content such as dynamic videos and text. In the remaining part of this chapter, we will analyze interactive media facade settings with respect to their technical requirements and their potential for providing access to information of different granularity. Besides media facades, themanifold use of light and light-emitting elements in urban environments plays an important role in the architecture of such spaces.
Haeusler (Haeusler 2009) identified three different uses of light in urban spaces: light architecture, media architecture, and media facades. He distinguishes these terms as follows: Light architecture subsumes the illumination of a building using both daylight and artificial light to underline parts of the building and to create a certain atmosphere. Media architecture uses a similar approach, but also includes all aspects of dynamically displaying media, such as dynamic graphics, dynamic text and spatial movement, but with a strong focus on dynamic content. Media facades build on this by including media to transform the building facade into a communicative element, that is embedded into the architecture of the building. Note that there is a fluent transition between these three terms.
The definition of architecture as a whole includes both the design of the entire environment around media facades and its effect on a given urban space (Haeusler 2009). Thus, whereas media facades embed communication into a facade in the form of digital media, media architecture describes the cultural, social and economic implications of these facades for the immediate environment (Haeusler 2009). It is noteworthy that Haeusler’s notions of media facades are created from a architectural perspective. Since we are dealing with media facades as large-scale digital screens from a human-computer interaction perspective, we do not explicitly address the architectural implications of media facades and we therefore do not distinguish between light architecture and media architecture. Thus, we define the term media facade as follows:
Media facades are digital public screens with arbitrary form factors and of varying resolution, which are created by either equipping the outer surface of an architectural building with controllable, uniformly shaped, light emitting elements or by projecting digital content onto it. Media facades are embedded into the architectural structure of the underlying building and transform the building into a communicative element.
Unlike designing the actual media facade (i.e., the underlying hardware) for a particular building, designing interactive media facade installations today is far from being standardized. There is neither a common ground for which technologies and which interaction techniques work in a particular situation, nor is there any categorization that puts the components of media facade installations into context. Existing installations usually are one-of-a-kind deployments that (1) are specifically designed from scratch for the particular building, and (2) only to a limited extent exploit the spatial context of the setting as well as the building’s surrounding. The design is usually driven by trying to make an idea somehow fit into a given location, rather than analyzing the location and context of the media facade in question to design an installation that specifically addresses the properties of the facade’s environment.
In this chapter we analyze the technical opportunities of interactive media facade installations regarding the properties that frame the installation, its spatial context as well as the applied interaction techniques and utilized input devices. Based on our analysis, we derive a taxonomy for describing interactive media facade installations, which provides insights about the appropriateness of different technologies in particular situations. The taxonomy considers (1) the technical and spatial properties of a media facade and its surroundings, as well as (2) different input modalities. The goal is to inform the design of interactive installations for media facades in such a way that designers can choose a suitable interaction technique based on the facade’s particular setting addressing technical opportunities, as well as limitations. Within this scope, we investigate different ways of participating in a media facade installation, and we analyze properties of media facades and their deployment environments in terms of their impact on interaction.
2Related Work
While many installations are one-of-a-kind deployments, researchers have begun to analyze systematically interactive media facade installations. It is important to consider the unique technical properties and requirements of this whole domain in order to create successful and enjoyable installations.
Dalsgaard et al. presented one of the first systematic analyses of the design of interactivity for media facades (Dalsgaard et al. 2008). Using their Design Space Explorer, they present a framework for managing multiple sources of information and domain concerns for the design of collaborative design process, which combines these aspects in scenarios for design concepts (Dalsgaard et al. 2008). The Design Space Explorer captures and gives an overview of design materials and forms, domain locations and situations, interaction styles, and content types. It is a tool to gain an overview of the design space, to conceptualize key aspects of interaction design, as well as to support communication and discussions among designers, clients and partners. Furthermore, it provides a platform for designers to combine these aspects into scenarios for design concepts (Dalsgaard et al. 2008). Dalsgaard and Halskov further expanded this work (Dalsgaard/Halskov 2010), identifying eight key challenges that need to be addressed when designing for media facades in an urban context. In this chapter we are mainly addressing the challenges (1) new interface, (2) integration into physical structures and surroundings and (4) developing content to suit the medium. Halskov and Ebsen provide a conceptual framework for addressing how content for a media facade may be designed taking into account the specific qualities of the display of media facade interfaces: scale, shape, pixel configuration, pixel shape, and light quality (Halskov/Ebsen 2013)). Van de Moere and Hill analyze the concept of research through design in the context of teaching urban computing (Moere/Hill 2009). They further point out the importance of the context of an urban installation for its design process (Moere/Wouters 2012). Reeves et al. provided a taxonomy to classify public interfaces according to the extent to which the interactions of a user and the resulting effects are hidden, revealed or amplified for spectators (Reeves et al. 2005). Korsgaard et al. approached the question of how to design for media facades by discussing how they structure their design processes to address specific sets of challenges raised in the literature (Korsgaard et al. 2012).
Fischer and Hornecker analyzed the spatial aspects in the design of shared encounters for interactive media facades (Fischer/Hornecker 2012). They reflected on various urban technology interventions by analyzing their spatial configuration in relation to the structuring of interaction. They introduce the term Urban HCI, which focuses on urban settings where context is not only a location point but also an activity carried out – it emphasizes situations composed of the built environment, the interfaces and the social context. Understanding the spatial configuration of the spaces around a media facade is an important factor for the design and development of successful interactive media facade installations. When creating such systems, the spatial configuration of the media facade’s environment further influences the appropriateness of certain content and it further restricts the applicability of different interaction techniques.
The aforementioned works aim at formalizing specific aspects of interactive media facade installations. Yet they focus on specific aspects and are thus not suitable for holistically describing the overall installation – including its technical specification – in order to categorize and compare it. It is noteworthy that there is work on multi-person-display ecosystems (Terrenghi et al. 2009). However, their taxonomy is not applicable to interactive media facade installations. Due to their physical characteristics, the outdoor urban environment, and the various available input modalities and interaction styles, a taxonomy characterizing interactive media facade installations needs to address those particular characteristics as well as the technical parameters of the setting. In the following, we therefore review existing interactive media facade installations and identify characteristic (and sometimes unique) properties and technical capabilities to ultimately create a taxonomy.
3Interactive Media Facade Installations
Researchers, artists and designers have been exploring media facade installations in a variety of ways: (1) they experimented with different input modalities and interaction techniques in a playful and artistic manner; and (2) they shared their valuable experiences and lessons learned as a solid ground for developing novel, universal interaction approaches. The following review of existing installations will serve as the basis to extract characteristic properties that frame and shape interactive installations.
3.1Whole Body Interaction
With the design intervention Aarhus by Light, Brynskov et al. created an interactive installation for a concert hall (Brynskov et al. 2009). It was installed on the glass facade of the building, which was fitted with 180m2 of media facade consisting of semi-transparent LED screens (see Figure 2). The goal of this installation was to engage local citizens into new kinds of public behavior to explore the potential of digital media in urban life. Within three interactivity areas in front of the facade (marked with colored carpets), the silhouettes of people standing on the carpet were tracked. Their movements were mapped to playful creatures on the media facade to encourage a curious and playful investigation of the expression among the users. The movement of the virtual characters further served as visual feedback on the users’ movements. Brynskov et al. (Brynskov et al. 2009) further revealed valuable insights around the themes of people’s interaction patterns, initiation, as well as their engagement with the installation and their interaction style. This led to an overview of social interaction patterns. Dalsgaard et al. further investigated the ability of this design intervention to support engagement (Dalsgaard et al. 2011). They describe how the presence of territorial issues (people’s social encounters) lead to social activities around the installation.
Fritsch and Dalsgaard utilized the whole body interaction approach from the Aarhus by Light installation in an interactive design intervention (Fritsch/Dalsgaard 2008). Their Climate Wall was an installation at an historical building in Aarhus, where it was running during the climate conference Beyond Kyoto21. It displayed generated, fragmented climate statements, giving passersby the opportunity to take part in the ongoing climate debate. People could grab and move around words by body movement to form new climate statements. During interaction, users received visual feedback on their input through the motion of controlled word. As key aspects from the Climate Wall and Aarhus by Light installations, the authors revealed their insights into how people interact with such novel urban interfaces and how they can be engaged to interact. They utilized media facades to create engagement, where the particular interaction technique supported the playful character of the installation.
Gutiérez et al. showed how whole body interaction can enhance the experience of a user during interaction by taking the interaction to a spatial level. With Lummoblocks (Gutiérrez et al. 2009), they created an interactive Tetris game, running on the media facade of the Medialab Prado22 in Madrid (see Figure 2). The aim of the installation is to provide a playful, interactive, space-located experience and engaging social interaction between users and spectators. The facade showed the game itself, combined with a live video feed displaying a bird’s eye view of the users in front of the facade. The installation mapped rotating and moving actions (the two dominant interactions in Tetris) to two separate interaction spaces, thus forcing the two players to collaborate: one had to rotate a Tetris block, while the second player moved it to the right spot. Players had to do so by running around within the boundaries of their particular interaction space.
With the installation 12m4s, the LAb[au] group created an interactive installation that generated a real-time visualization of passersby based on their average walking speed (Klanten et al. 2011). The movements are tracked in real-time with cameras and ultra-sound sensors to generate a visual (3D particles) and auditory (granular synthesis) scape on the facade. The visualization is based on the position, orientation and speed of passersby. It is projected on a 12m long Mylar23 screen (an electro insulating film made of PETP), combining projection and reflection of the passerby’s body,while creating an interactive space in between the digital and the body space. The characteristics of the Mylar screen amplify the visual effects. The screen remains a mirror on non-enlightened zones, but turns transparent on enlightened zones.
The Night Lights installation by YesYesNo combines three different interaction techniques (Klanten et al. 2011) to display interactive silhouettes and animations on a projected media facade with 3D projection mapping: tracking movements of people standing on a platform, hand gestures above a light-table and waving of mobile phones. The installation turned a building into an interactive playground. The goal of the installation was to adapt the concept of shadow puppets to allow any passersby to become a performer. There are dedicated interaction zones for the particular interaction techniques, which are designed as small stages, where people can walk up and interact with the projection. The movements of people within these interaction zones are tracked and amplified by an approximately 15m tall projection on the building.
The building of La Vitrine Culturelle in Montreal, Canada, is equipped with a small, low-resolution media facade of approximately 23m2 consisting of 35000 RGB LEDs that change their color as a reaction to the movements of passersby (Haeusler et al. 2013), thus providing various animations and media content. When walking past the facade, the walking direction is mapped to animated arrows. When people stop and stand in front of the facade, they can create further animations through body gestures. These animations range from snowflakes around the user’s silhouette to movable light spots. Although designed as temporary installation for Montreal’s city of lights festival in 2009, it was the first interactive media facade in North America to be turned into a permanent installation due to its huge popularity.
The aforementioned installations illustrate the importance and suitability of body-movement as input modality as it usually has a playful character combined with performance elements and enables people to interact without a dedicated interaction device. However, its applicability as general interaction technique to interact with complex content on distant screens is limited as it requires extensive instrumentation of the space.
3.2Pointing
A common interaction technique utilized in various interactive installations is pointing. With Spread.gun, Fischer et al. present an interactive shared encounter for media facades, inspired by established forms of graffiti culture (Fischer et al. 2010). The installation took place within the scope of the 2008 Media Facade Festival in Berlin, Germany with the aim of creating a digitally augmented forum in public space. The tangibility of the interaction is mentioned as the most important part of the design work. The stationary interaction device – called Spread:gun – is a model of an ancient cannon that a user rotates to aim at a particular point on the facade. A user can enter a text message through a digital touch screen. While aiming at the facade, a virtual crosshair is displayed as visual feedback. The position of the crosshair is calculated with data from two potentiometers that are integrated in the cannon-like interface for the horizontal and vertical axis. By shooting the cannon, a color bag is virtually shot onto the projected facade. The color spots on the facade are displayed enclosing the text messages of the particular user. One disadvantage of this installation is that the interaction device is highly tailored to the purpose of the installation and environment around the particular facade. They point out the effect of the social and spatial setting and describe how the location and the surroundings may drastically alter the context of the installation.
As a conceptual enhancement of Spread:gun, Fischer et al. presented the SM-Slingshot (Fischer et al. 2010). Instead of a stationary input device, they provided a mobile, custom built input device, based on the metaphor of a wooden slingshot. The aim of the installation was allowing people to shoot information onto public screens. The slingshot device is equipped with a small LCD display and a mobile phone keyboard, allowing a user to enter text messages. By aiming and shooting at a particular point on the media facade with the slingshot, a user can shoot his entered message on the facade together with a virtual color bag, analog to the Spread.gun approach. Additionally, the entered text messages are twittered in real-time. In comparison to Spread:gun, SMSlingshot has the advantage of providing a portable and flexible interaction technique. With TXTual Healing, Notzold followed an approach very similar to SMSlingshot, applying the same interaction metaphor and feedback mechanisms utilizing the model of an assault rifle to fire text-based messages onto a projected media facade (Klanten et al. 2011).
For the described approaches, a user only points at the media facade to transfer locally created content onto the facade. In contrast, the installation Laser Tag by the Graffiti Research Lab allows for continuous pointing interaction to create content directly on the media facade (Klanten et al. 2011). As depicted in Figure 2, users can paint and write onto the media facade by pointing at it with a laser pointer. The scope of this installation was to create a playful approach for allowing people to digitally write on buildings in public spaces. In an attempt to provide a universal interaction technique for media facades, Gehring and Lander introduced the GPS Lens (Gehring/Lander 2013), a technique to control a pointer on a large-scale distant screen by pointing at the screen with the mobile device. The position of the pointer is calculated based on GPS, accelerometer and compass data from the mobile device’s built-in sensors.
Most of these installations have in common that they utilize a pointing approach where a laser is tracked by cameras. Although this is a trivial, from a technical perspective easy to realize approach providing immediate visual feedback on the pointing, it’s applicability as a general interaction technique for permanent media facades that are not permanently monitored is rather limited. From the technical side, the tracking does not work with standard, free to use laser pointers. The utilized lasers require special permissions to be used in public. From an interaction perspective, it is hardly possible to distinguish between users, or limit the interaction to particular users, as there is no association between user and laser pointer. Such a system only detects a certain number of laser dots, yet it is not aware of the absolute number of lasers used.
3.3Touch & Gestures
Both direct touch and gestures with mobile devices have been explored with different media facade installations. Within the scope of the Illuminating York Festival24,in conjunction with the GaiaNova Productions Ltd., the OMA International group presented the Wall of Light installation (Klanten et al. 2011), an artistic installation to paint in real-time on a projected media facade. Using the Tag Tool – a digital graphics tablet for painting – as the input device, people were able to paint onto the facade by drawing on the Tag Tool with a digital pen. The installation was intended as a canvas for daily performances of artists, but they also allowed visitors to use the installation to express themselves. According to the creators, the performance of the inexperienced visitors shows that the installation is intuitive and easy to use.
The urban media installation Touch by the LAb[au] group allows passersby to interact with the Dexia Tower in Brussels, Belgium (Klanten et al. 2011). They installed a stationary multi-touch screen at the base of the building to allow people to individually or collectively manipulate the color of the approximately 4200 windows of the 145 m high tower in real time. This window animation facade allows for both dynamic and static inputs. People can create basic geometric forms on the tower as well as change its color. The concept of this installation tries not to treat the facade as a plain 2D surface. The goal is to integrate characteristics of the building, such as orientation, scale or volume, to create an interactive experience. The stationary screen displays a virtual representation of the buildings surface, on which the users get immediate visual feedback on their interactions.
With MobiSpray (Scheible/Ojala 2009), Scheible and Ojala presented a digital graffiti tool utilizing a mobile phone as a virtual spray can. Users can spray digital dabs onto a projected media facade. The movement of the spraying nozzle is based on the orientation of the mobile phone, determined by continuous data from the built in accelerometer. Adedicated drawing client application on the mobile phone provides controls to adjust the properties of the virtual spray can such as blob size, colors or intensity. A user can draw onto a facade by performing spray gestures with the mobile phone, similar to a real spray can. Since there is no absolute mapping between the spray canvas and the facade, the advantage of this approach is that it does not require any calibration when used in a different location for a different building. Shamma et al. utilized and adapted MobiSpray to create a collaborative space for the voice of a local community to make statements about the world around them (Shamma et al. 2009). Participants and spectators can observe how digital graffiti is created while they receive feedback provided by dancers interacting with the virtual projected content and physical space. To create graffiti, people can use MobiSpray and they can additionally upload text messages as well as images from the mobile phone to the media facade.
With the installation iRiS, Boring et al. presented an approach utilizing the interaction through live video pattern (Boring et al. 2011). Users aim their device at the facade and observe it in live video, while can point through the display at the facade. Touch input occurring on the mobile device is projected onto the facade, giving the impression that users directly touch the surface of the building (Boring et al. 2011). This approach requires a direct line of sight and is therefore limited to the visible part of the facade. With facade maps, Gehring and Krüger introduced an approach to interact with media facades having 3D form factors (Gehring/Krüger 2012). They apply cartographic map projections to make the whole facade visible at once on a map of its surface. This map is displayed on a mobile device, where users can interact with the facade with touch input on the mobile device’s screen.
A common problem of such installations is they are often both stationary and highly tailored to the one specific setting and usage scenario. This limits their applicability as general input techniques to interact with media facades.
3.4Remote
We are investigating direct interaction with media facades in a sense that to be able to interact with a facade, the user has to be within the potential interaction space in front of the facade. Especially for high resolution facades displaying advertisements, operators can remotely interact with the facade over web interfaces or dedicated software in order to provide the displayed content. Although they are interacting with the facade, we do not address this form of interaction since it can be instead considered as pure provisioning of content and it does provide passersby with the opportunity to interact with the facade. However, there exist different approaches providing people with a remote interface to interact with a media facade without the requirement of being within the potential interaction space or even the display space. While the aforementioned installations focused on allowing the people within the display space to interact with the facade, providing remote interfaces is usually driven by different design intentions. First, providing remote interfaces does not particularly focus on the people within the display space. It allows a wider range of people to interact with a facade while the interaction could be either limited to a particular group of people like for example all inhabitants of the city, or it could be allowed to all people that can access the remote interface (e.g., over the internet). Second, media facade installations providing remote interfaces often do not focus on continuous interaction. They rather provide means for sharing content with a media facade (e.g., by uploading images, videos or animations) or to participate by expressing yourself and visually branding or customizing a building.
One of the first installations is the Blinkenlights project (Haeusler 2009). The upper eight floors of an office building were turned into the at that time world’s biggest interactive computer screen. They created a window raster animation facade by equipping the windows of the building with 144 individually controllable lamps in total, which resulted in display with 8x18 pixels. To control the content, people had to call a dedicated phone number with their mobile phones. Once connected, they could either control a virtual cursor on the facade or activate previously uploaded animation with a code also entered with the key pad of the mobile phone. The world’s first interactive RGB media facade, the Marnix facade in Brussels, Belgium, allows people to participate by taking control of the media facade’s content (Haeusler et al. 2013). The operators of the facade provide a free-to-download animation interface, with which people can create and upload their own animations from anywhere in the world. This installation is very popular and frequently used from people around the world to share content from wedding proposals, to political statements, to artful animations. The media facade of the Rundle Lantern building in Adelaide, Australia was built with “the vision of creating an experience that would capture the imagination of the city and add beauty to peoples’ lives” (Haeusler et al. 2013). This low-resolution media facade consists of more than 700 individually controllable RGB panels that are wrapped around the building. To control the content of the facade, people can use a web interface to create and upload their own animations. With its frequent use, this installation has developed into a tool for continuous community engagement.
The aforementioned installations mostly have a playful, performative or artistic orientation. The interaction as well as the applied input techniques and devices were particularly designed for the specific installation. Besides the playful and artistic character of the installation, an important aspect when interacting with a media facade is its resolution and form factor. As we have seen, the resolution can easily vary from screens with 144 pixels to high-resolution screens capable of displaying sophisticated content. Furthermore, the majority of the presented installations require a direct line of sight between user and facade to interact. This would leave a user with only a part of the actual facade as an interactive screen. When developing novel interaction techniques, we need to address these issues in order to exploit the media facades’ full potential as urban screens.
4Characteristics of Interactive Media Facades
Media facades are commonly built to fulfill a particular purpose. They are based on different technologies and are situated in unique, dynamic settings. This makes a comparison of different media facade installations a challenging task. Interactive media facade installations usually apply different input modalities utilizing a wide range of input devices with various technical requirements. Furthermore, the technical properties and spatial arrangement of the interactive space itself influence the applicability of technologies to provide user input.
4.1Context of the Setting
Media facades are usually situated in a dynamic setting with unique characteristics making it difficult to compare installations and situations that arise from them. Nevertheless, there are characteristic properties such as instrumentation of the space or spatial aspects in the context of a media facade’s deployment space which have a strong influence on the applicability of different technologies. Furthermore, the intended interaction style (i.e., performative interaction) also introduces constraints. The properties can be brought in to categorize and compare different media facade installations in different contexts.
4.1.1Spatial Aspects
Fischer et al. analyzed spatial aspects – the spatial configuration of the space around the installation – in the design of shared encounters for interactive media facades with respect to the spatial configuration in relation to the structuring of interaction (Fischer/Hornecker 2012). In combination with the resolution of the media facade itself, as well as its underlying technology (e.g., pixel-based, projected, etc.), the configuration of the space around a media facade has more impact on the appropriateness of content than on the general applicability of particular interaction techniques. Size and configuration of the space determine distances and angles from which users can see the facade and its content. To allow the users to perceive the displayed content as a whole from within this space, the content must be designed with respect to the spatial configuration of the setting. When dealing with multiple users interacting simultaneously with the facade, the spatial configuration of the potentially interactive space can further determine if the particular users can see each other directly or if they are at least aware of each other. The spatial aspects also influence the communication between the users and hence, the social configuration of the setting.
Concerning different interaction techniques, the spatial configuration of the setting represents a scaling factor rather than a criterion for the general applicability of a particular technique. Regarding the physical space that is required to perform an interaction, interaction techniques in the context of media facades in general only require minimal space – the immediate space around the user – to be applicable at all. For interaction that does not involve whole-body movement of the user, the minimal space is usually represented by the space required to operate the input device. When involving interaction techniques such as direct pointing, it further requires a direct line of sight between the user and the particular target area on the facade. Concerning whole body interaction, size and spatial arrangement of the involved spaces determines the possible scale of the mapping between body movement and reaction on the facade. Furthermore, it influences the potential number of maximum users that can interact simultaneously, since each user requires a certain amount of the available space. In summary, we can say that the spatial configuration of the setting represents a scaling factor rather than a criterion for the general applicability of a particular technique and therefore has no direct influence on the general applicability of a particular interaction technique. Hence, we only assign it a minor role in our taxonomy.
4.1.2Instrumentation
The technical instrumentation of a media facade’s environment has a direct influence on the applicability of different interaction techniques. Different interaction devices and interaction styles require the instrumentation of the potential interaction space to different extents. If users can interact with the media facade from a single place, usually only a small space needs to be instrumented, e.g., by providing wifi, installing a camera or providing an input device. If users can interact from anywhere within a given space, usually the whole space needs to be instrumented with the necessary technologies. While this is not a big deal when using wifi and the users’ mobile phones as input devices, this can result in a very complex setting when for example the users’ body movements need to be tracked within large spaces. In the following, we point out the influence of different characteristic properties of interactive media facade installations on the technical instrumentation of the potential interaction space.
4.1.3Interaction Style
Interaction with media facades can be designed in various ways, resulting in interaction with different characteristics. This requires the potentially interactive environment to be instrumented to a different extent:
1. Performative interaction is usually realized by tracking the body posture or movement of a user. This is usually achieved by instrumenting the potentially interactive space with additional hardware such as cameras or sensors.
2. Direct interaction – such as direct pointing – requires either tracking the current position of the user, as well as the direction he is pointing at, or tracking the location of a cursor on the facade, initiated by a pointing device (e.g., a laser pointer). The latter is usually realized with cameras pointed at the facade and hence requires less instrumentation than tracking a user’s body movements within a large open space.
3. Indirect or relative interaction techniques (e.g., using a mobile device’s built-in inertial sensor to relatively control a pointer on the facade) usually do not require an extensive instrumentation of the potentially interactive space. The technical capabilities required to determine the user’s input are usually provided by the input device itself (i.e., a mobile device or an interactive terminal placed in the space in front of the facade).
4.1.4Connection
When using dedicated input devices (e.g., custom-built hardware or smart phones), there is the requirement for a permanent data connection between input device and media facade. With the increasing availability of public wifi networks and data plans for mobile devices, the demand for connectivity can be considered as a minor concern, and not as a dedicated instrumentation of the space around the media facade.
4.1.5Number of Users
Aside from the interaction technique itself, the number of supported users is influenced by the technical instrumentation of the potentially interactive space. The instrumentation can be either a permanent installation of stationary hardware, the distribution of mobile input devices, or both. For example, if we consider whole body interaction where the user’s movements are tracked with a camera, the number of users is limited to the number of users trackable by the available cameras. To allow for more users, the number of cameras has to be increased. For the sake of simplicity, we omitted the required computing power to process the actual camera data in this example, since this again depends on the particular algorithms, update rates and the detailed realization. Similarly, when using dedicated input devices, the number of users is limited by the number of provided devices and can be increased by distributing more devices up to a certain limit given by the application itself. In general, we can say that issues related to number of users can be handled either by hardware, software or a combination of the two.
4.1.6Interaction Distance
As mentioned before, interacting with a media facade usually implies interacting at a distance. As can be seen when comparing techniques for interacting at a distance, such as SMSLingshot (Fischer et al. 2010) or LaserTag (Klanten et al. 2011), they require different minimal viewing distances and they can potentially work from great distances. Depending on the desired interaction distance, this can require the instrumentation of large spaces. On the flip side, if a designer aims for interaction from very short distances within the direct proximity of the facade, some interaction techniques are not applicable if the intended interaction distance is below the required minimal distance.
4.1.7Location Awareness
The degree to which an installation can be aware of a user’s position within the potential interactive space is also determined by the instrumentation of the environment. In the least constrained case, the application only needs to be aware of whether a user is within the potential interaction space or not. Since this is covered anyway by the chosen interaction technique, as a user needs to be within the potential interaction space for the technique to work, we can treat this as though the installation not being aware of the user’s location and this case therefore does not need further instrumentation. If the facade needs to be aware of whether a user is within its direct proximity – which is often the case for interactive media facades installed at ground level – only the immediate space around the facade needs to be instrumented. If the exact location of a user within the potential interaction space needs to be known, the amount of required instrumentation increases with the size of the space and of course the accuracy with which the location needs to be provided.
The spatial configuration of the potentially interactive space again only plays a minor role for the general possibility of instrumenting it with different technologies. The size of the space simply indicates the amount of resources required. Or, conversely, the available amount of resources determines the size of the space that can be instrumented.
4.2Input Modalities
When designing interactive installations for media facades, the applied interaction techniques in combination with the utilized input devices shape the interaction and have a huge impact on the usability and the user experience. Furthermore, they determine the technical requirements for the instrumentation of the potential interaction space in front of the facade. When choosing a particular input modality (e.g., touch, pointing, etc.) and a particular interaction technique (e.g., camera tracking, interaction through live video, etc.) there is always a trade-off between the technical requirements for utilizing the particular input modality and the granularity of the input and the triggered experiences. Hence, the utilized input modalities are an important factor for a systematic approach to categorize interaction with media facades. While we reviewed interactive media facade installations grouped by the input modality they use, in the following, we will investigate the characteristics of the particular input modalities as well as their influence on the installation.
4.2.1Whole Body Interaction
With whole body interaction approaches, the human body itself is used as an input device, such an approach requires tracking body movement within the potential interaction spaces. Hence, those spaces have to be instrumented with different technologies, depending on the accuracy and the extent to which movements are supposed to be tracked. With the instrumentation of the space, and turning every body entering the potential interaction space into an input device, this input modality is especially suited for reactive media facade installations where the communication with the facade happens implicitly. Whole body interaction also turned out to be well suited for performative interaction within an artistic or playful context (Brynskov et al. 2009; Fritsch/Dalsgaard 2008; Haeusler 2009; Haeusler et al. 2013). In this context, mapping body movement within a particular space – physically running around within the space – to motion on the media facade is a common pattern. This allows for a continuous interaction through the whole potential interaction space. Furthermore, to passersby, users seem to perform in front of an audience, which usually draws attention and encourages passersby to engage in the social and physical setting (Brynskov et al. 2009; Fischer/Hornecker 2012).
For whole body interaction, the accuracy of the tracking and hence of the interaction itself heavily depends on the utilized tracking technologies, the speed of the movement, and environmental conditions such as lighting conditions. From this, we argue that whole body interaction is more suited for coarsely grained interaction where the accuracy of input is not of great importance. This is especially the case for low-resolution media facades that are not capable of displaying complex content. For fine-grained, precise interaction with complex content, we need more accurate interaction modalities. Concerning the number of users, whole body interaction is suitable for multiple users interacting simultaneously. It is only limited by the technical capabilities of the utilized tracking technologies and the size of the instrumented space.
4.2.2Pointing
Direct and indirect pointing are further common techniques for interacting with a media facade (Klanten et al. 2011). Both usually require a dedicated input device with which the user points at the facade in order to interact (e.g., to control a pointer). While for direct pointing the movement of a visual cue that is initiated by the pointing device – usually a laser pointer – is tracked on the facade with the help of cameras, in the indirect approach, the input device itself calculates the relative movement or in some cases the position of the pointer and communicates it to the facade. Hence, the amount of required instrumentation of the potential interaction space is rather low, reducing the effort needed to set up the installation, while allowing for fine-grained interaction with complex content at the same time. The low instrumentation makes such an approach highly suitable for portable, non-permanent media facade settings.
It is common to use direct or indirect pointing in combination with non-permanent, projected media facades (Klanten et al. 2011; Scheible/Ojala 2009). In the case of direct pointing, a camera can be mounted onto the projector for observing the facade and tracking pointers. In the case of indirect pointing, the pointing can be realized with a portable mobile input device while no further hardware – the projection system aside – is required. It is common to use a projector which is connected to a mobile computer. For direct pointing, the most common pointing device is a laser pointer which also might be mounted onto or integrated into custom input devices. The position of the pointer on the projected facade is then usually determined with the help of a camera mounted onto the projector. For indirect pointing, a common input device is a standard mobile phone, which allows for calculating the relative pointer movement based on the movement of the device itself or based on touch input on the mobile device’s screen. Since in both cases dedicated input devices – usually portable – are needed, such an approach usually needs an ongoing monitoring of the setting.
Although multiple users could interact with the media facade in parallel, this is hampered by two factors: (1) Every user needs an input device. Hence, the number of potential users is limited by the number of input devices. However, if the user’s own mobile phone is used as the input device, then this is less of an issue. And (2), when using direct pointing with a laser pointer-like device, distinguishing between respective users is challenging. Such a system usually only detects a certain number of laser pointers on the screen, but it is not aware of how many lasers they are coming from and which pointer belongs to which user. To match a pointer to a particular user, further instrumentation of the input device is needed.
4.2.3Touch and Gestures
Gestural user input by direct touch is a widespread approach often used with permanent installations. When applying touch as the input modality of choice, dedicated input devices are needed. This is a common approach utilized in various permanent media facade installations of various resolutions (Haeusler 2009; Haeusler et al. 2013). For such installations, a stationary input device (e.g., a input terminal or an interaction booth) is usually provided within viewing distance of the facade. While this comes with the benefit of keeping the required instrumentation at a minimum (e.g., only the stationary input device needs to be installed and usually no instrumentation of the further environment is required), this comes with the disadvantage of usually limiting the number of users to one or two. Furthermore, this leads to stationary settings, not exploiting the spatial aspects of interaction with media facades. While interacting, users are required to be within the direct proximity of the stationary input device. This can also turn into an advantage when realizing complex interactions requiring specialized or dedicated input hardware to address the peculiarities of a particular setting. Concerning the resolution of the facade, this approach is in general highly suitable for both low-and high-resolution facades.
4.2.4Web and Remote Interface
A further approach to allow users to interact with the content of media facade is providing external – usually web-based – interfaces to upload and modify the displayed content (Klanten et al. 2011). While such an approach does not require instrumenting the environment in front of the media facade, it also does not particularly address the passersby in front of the facade within the display space. Furthermore, users do not get immediate visual feedback on their interactions. The main focus of such approaches lies in supporting a one-directional interaction, allowing a wide range of users to provide content for the media facade pushing this approach more in the direction of being reactive rather than being interactive. By providing remote interfaces, the target audience for interacting with the facade facade can be maximized since basically every person around the globe with access to the internet could potentially interact. However, we believe that this does not go along with the main idea of interacting with a media facade. It rather constitutes pushing content onto a media facade and we consider this as the minimal amount of participation that can be allowed.
5Taxonomy
The goal of this chapter is to derive a taxonomy of interaction with media facades using different technologies. The aforementioned properties and characteristics of interactive media facade installations determine the applicability of different approaches for allowing interactivity with different characteristic traits. In addition to determine suitable approaches for particular settings, we can consult the taxonomy to compare interactive media facade installations in different contexts. In general, we address the following properties in the taxonomy, since – as it occurred when reviewing existing installations – they have proven to influence the characteristics of an installation as well as the applicability of different interaction approaches:
Instrumentation versus no instrumentation of the potential interaction space: The technical realization of interaction can require instrumentation of the potential interaction space in front of the facade to different extents. This can vary from providing custom input devices to equipping the space with complex infrastructure.
Utilized input modalities: Different input modalities usually come with different technical requirements. They vary in the level of user engagement they require. Furthermore, depending on their characteristics, there are input techniques more suitable for high-resolution media facades and some that work well with low-resolution facades.
Stationary versus mobile interaction: Interaction can be carried out in either a stationary or a mobile way. For stationary interaction, users are either provided with a stationary input device, or the interaction requires the user to remain in a particular place. Whenmobile interaction is possible, users usually can move around within the potential interaction space and interact from arbitrary places. Furthermore, they often can continuously interact while moving around the potential interaction space. The particular movement of the user can also be a part of the interaction itself.
Resolution of the media facade: Media facades can have different resolutions, ranging from very low resolutions of less than 200 pixels to high-resolution screens with several million pixels. This for example influences the options for displaying visual feedback such as pointers or controls necessary for particular input techniques, as well as the complexity of the displayable content. Furthermore, when the interaction is carried out while moving around within the potential interaction space, perceived resolutions might change dynamically.
The media facade’s form factor: Different forms of interactivity can involve the presence of a user in various ways. Users can be required to be within a certain distance from the media facade in order to interact with its content in general. A user could also explicitly interact with a media facade by changing his distance from the facade. Furthermore, when providing means for remote interaction, a media facade might not require a user to be present at all. When considering the media facade as a whole and not only the particular part visible from a user’s current point of view, the form factor of a media facade can introduce additional complexity. With interaction techniques requiring a direct line of sight between the user and the target area on the facade, interacting with all parts of the facade is not possible without relocating and moving around the facade. Furthermore, the media facade’s content might never be completely visible as a whole to a user, which also can constrain the interaction.
The aforementioned properties are dependent on each other in different ways. We combine them into a taxonomy of interaction with media facades as follows: We initially classify the interactive installation according to the utilized input modality and whether it requires the instrumentation of the potential interaction space or not. Regarding input modalities, we consider whole body interaction, pointing, touch and gestures and remote interaction. We subdivide the installations into stationary ones, where the position from which a user interacts is fixed, and mobile ones, where a user can interact from different locations and also especially when moving between location. For the input modalities, we subdivide the applicability of the particular input modality for low- and high-resolution media facades. For each installation, we point out the number of supported users. This results in the spatial layout of the taxonomy as depicted in Figure 3. The layout was inspired by the taxonomy for mobile device input on external displays, presented in (Boring 2010).
Based on the introduced taxonomy, we revisited existing interactive media facade installations and categorized them accordingly as depicted in Figure 3. It turned out that whole body interaction usually demands static instrumentation of the potential interaction space, usually resulting in interaction from a static position with a limited number of users. Furthermore, the classification revealed that whole body interaction is usually applied in combination with low-resolution media facades. Pointing as well touch input also require the instrumentation of the potential interaction space. Even in the case of using a laser pointer as the pointing device, a camera is still required to track the pointer movement.
While pointing installations usually address high-resolution media facades, touch and gestural input are regularly applied for both low- and high-resolution media facades, which we believe involves using additional input devices, handling the mapping between input and facade resolution. Manipulating a media facade’s content through remote interfaces (e.g., web interfaces) does not require the instrumentation of the space in front of the media facade, but it usually also does not require a user to be present within the display space near the facade.
6Discussion
To our knowledge, this is the first taxonomy of interaction with media facades that comprises the unique properties and characteristics presented by media facades in general. It aims at the interactivity of media facades to go beyond considering them purely as information broadcasters. Upon review of existing media facade installations, it becomes apparent that such an installation does not require any instrumentation, allows for the users’ mobility (and potentially mobile input) and is independent of the facade’s form factors. Generally, fewer installations can be found that do not require instrumentation of the space the facade is located in. This limits the deployment of such facades as the instrumentation is often cumbersome and cost-intense. The taxonomy further reveals that – so far – installations that follow these characteristics only allow for controlling pointers. While this is an acceptable implementation of an interactive display, media facades are a separate device class (i.e., they are enormous in size and allow for a large number of users) and it is thus questionable whether pointing (as known from personal computers) is the appropriate and meaningful way for interacting with such displays. Virtual Projection presents an exception to interacting with pointers (Boring/Baur 2013). Although it has not been implemented on media facades, their implementation allows for a more direct coupling between personal mobile devices (e.g., mobile phones) and a large display, which may well be an entire media facade as in the case of (Boring et al. 2011).
6.1Using the Taxonomy
Our taxonomy is targeted towards practitioners and designers who aim at deploying an interactive media facade installation. We consider two approaches: (1) defining the level of instrumentation based on desired input modalities; and (2) basing the input modality on the level of instrumentation present and more importantly possible in the space. In the first approach, for example, a designer may choose to have whole body interaction as the dominant input modality. This limits the space for deployment in question to one that can be instrumented (with the exception of Night Lights (Klanten et al. 2011)). In the case where the level of instrumentation is fixed, practitioners and designers can thus select from a range of techniques that have been deployed before. For example, if instrumentation is not possible and users are mobile, there are almost no limits with respect to the input modality. Yet, when an unlimited number of users is required, for example, some techniques appear more suitable than others.
Overall, we believe that our taxonomy, filled with a broad variety of existing installations, allows designers and practitioners to choose the best possible candidate for a new installation. In addition, the taxonomy enables researchers to fill in the gaps in case new combinations within the taxonomy are required (e.g., a stationary, mobile, whole body interaction with high tracking resolution).
6.2Limitations
Our taxonomy was based on an extensive review of both literature as well as existing media facade installations, yet with the intent of being generally applicable to future installations. However, we are aware of limitations that arise from their public and outdoor nature. First and foremost, the taxonomy does not capture a facade’s visibility. For example, a facade may only be partially visible due to other passersby or other buildings obstructing (parts of) the facade from a given viewing location. Additionally, the technical implementations may be affected by changing light conditions (e.g., daylight versus night). Finally, our taxonomy does not capture other external influences such as traffic in a given space that may interrupt a given interaction temporarily.
While these factors naturally affect interaction with media facades, they are highly unique to a given environment. We consider them part of the architectural planning. Our taxonomy helps in that certain aspects can be excluded before designing an interactive experience based on the layout of the space the media facade is located in.
Bibliography
Boring, S. (2010): “Interacting Through the Display: A New Model for Interacting on and Across External Displays”. Dissertation, LMU München. Fakultät für Mathematik, Informatik und Statistik.
Boring, S.; Baur, D. (2013): “Making public displays interactive everywhere”. Computer Graphics and Applications, IEEE 33:2, 28–36.
Boring, S.; Gehring, S.; Wiethoff, A.; Blöckner, A.M.; Schöning, J.; Butz, A. (2011): “Multi-user interaction on media facades through live video on mobile devices”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11. New York, NY: ACM, 2721–2724.
Bouchard, D. (2007): “Embodied Emergence: Distributed Computing Manipulatives”. PhD thesis, Massachusetts Institute of Technology. Dept. of Architecture. Program in Media Arts and Sciences.
Brynskov, M.; Dalsgaard, P.; Ebsen, T.; Fritsch, J.; Halskov, K.; Nielsen, R. (2009): “Staging urban interactions with media façades”. In: T. Gross; J. Gulliksen; P. Kotz; L. Oestreicher; P. Palanque; R. Prates; M. Winckler (eds.): Human-Computer Interaction INTERACT 2009 (Lecture Notes in Computer Science 5726). Berlin u.a.: Springer, 154–167.
Bullivant, L. (2006): Responsive Environments: Architecture, Art and Design. London: V&A.
Dalsgaard, P.; Dindler, C.; Halskov, K. (2011): “Understanding the dynamics of engaging interaction in public spaces”. In: P. Campos; N. Graham; J. Jorge; N. Nunes; P. Palanque; M. Winckler (eds.): Human-Computer Interaction INTERACT 2011 (Lecture Notes in Computer Science 6947). Berlin u.a.: Springer, 212–229.
Dalsgaard, P.; Halskov, K. (2010): “Designing urban media façades: cases and challenges”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’10. New York, NY: ACM, 2277–2286.
Dalsgaard, P.; Halskov, K.; Nielsen, R. (2008): “Towards a design space explorer for media facades”. In: Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat, OZCHI ’08. New York, NY: ACM, 219–226.
Fischer, P.T.; Hornecker, E. (2012): “Urban HCI: spatial aspects in the design of shared encounters for media facades”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12. New York, NY: ACM, 307–316.
Fischer, P.T.; Zöllner, C.; Hoffmann, T.; Piatza, S. (2010): “VR/Urban: SMSlingshot”. In: Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied Interaction, TEI ’10. New York, NY: ACM, 381–382.
Fischer, P.T.; Zöllner, C.; Hornecker, E. (2010): “VR/Urban: Spread.gun – design process and challenges in developing a shared encounter for media façades”. In: Proceedings of the 24th BCS Interaction Specialist Group Conference, BCS ’10. Swinton, UK: British Computer Society, 289–298.
Fritsch, J.; Dalsgaard, P. (2008): “Media Façades beyond interaction”. In: Proceedings of the 20th Australasian Computer-Human Interaction Conference, OZCHI ’08. New York, NY: ACM, 8–12.
Gehring, S.; Krüger, A. (2012): “Façade map: continuous interaction with media façades using cartographic map projections”. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, UbiComp ’12. New York, NY: ACM, 471–480.
Gehring, S.; Lander, C. (2013): GPS Lens: “GPS based controlling of pointers on large-scale urban displays using mobile devices”. In: Proceedings of the 2nd ACM International Symposium on Pervasive Displays, PerDis ’13. New York, NY: ACM, 115–120.
Greenfield, A.; Shepard, M. (2007): Urban Computing and its Discontents. New York, NY: The Architectural League of New York.
Guitiérrez, C.; Canet, M.; Puig, J.; Lloret, J. (2015): “Lummoblocks.” http://www.lummo.eu/lummotetris.html. Accessed: 2015-10-19.
Haeusler, H.M. (2009): Media Facades – History, Technology, Content. Stuttgart: avedition.
Haeusler, H.M.; Tomitsch, M.; Tscherteu, G. (2013): New Media Facades – A Global Survey. Stuttgart: avedition.
Halskov, K.; Ebsen, T. (2013): “A framework for designing complex media facades”. Design Studies 34:5, 663–679.
Klanten, R.; Ehmann, S.; Hanschke, V. (2011): A Touch of Code – Interactive Installations and Experiences. Berlin: gestalten.
Korsgaard, H.; Hansen, N.B.; Basballe, D.; Dalsgaard, P.; Halskov, K. (2012): “Odenplan: a media façade design process”. In: Proceedings of the 4th Media Architecture Biennale Conference: Participation, MAB ’12. New York, NY: ACM, 23–32.
Moere, A.V.; Hill, D. (2009): Research through design in the context of teaching urban computing. Street Computing Workshop (co-located with OZCHI ’09), Melbourne.
Moere, A.V.; Wouters, N. (2012): “The role of context in media architecture”. In: Proceedings of the 2012 International Symposium on Pervasive Displays, PerDis ’12. New York, NY: ACM, 12:1–12:6.
Reeves, S.; Benford, S.; O’Malley, C.; Fraser, M. (2005): “Designing the spectator experience”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’05. New York, NY: ACM, 741–750.
Scheible, J.; Ojala, T. (2009): “MobiSpray: mobile phone as virtual spray can for painting big anytime anywhere on anything”. Leonardo 42:4, 332–341.
Schoch, O. (2006): “My building is my display”. In: Proceedings of the Conference on Education and Research in Computer Aided Architectural Design in Europe: eCAADe’06, eCAADe, Wien.
Seitinger, S.; Perry, D.S.; Mitchell, W.J. (2009): “Urban pixels: painting the city with light”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’09. New York, NY: ACM, 839–848.
Shamma, D.A.; Scheible, J.; Sheppard, R.M. (2009): “Graffiti dance: interaction of light, information, and environment”. In: Proceedings of the 7th ACM Conference on Creativity and Cognition, C&C ’09. New York, NY: ACM, 479–480.
Terrenghi, L.; Quigley, A.; Dix, A. (2009): “A taxonomy for and analysis of multi-person-display ecosystems”. Personal and Ubiquitous Computing 13:8, 583–598.
Weiser, M. (1992): “The computer for the 21st century”. Scientific American 265:3, 94–104.
Winograd, T. (2001): “Interaction Spaces for Twenty-First-Century Computing”. In: HCI in the New Millenium. New York: Addison-Wesley, 259–276.