2.1Ambient digital displays

Concepts of ubiquitous or calm computing inspired decades of research determined to implement Weiser’s vision of calm technology. One of the notions that emerged from these efforts is the ambient display of real-time information (Mankoff et al., 2003). At the core of this concept is the idea of communicating information at the periphery of the user’s vision, using a range of modalities, such as visuals or sound (Tomitsch et al., 2007). The premise is that ambient information displays allow people to access information while being engaged in a primary activity.

For example, the Ambient Orb14was a consumer product that implemented this idea by translating real-time information into shades of color. Userswere able to choose from a range of data sources, such as weather data, surf conditions, or stock markets, which would then be mapped to a color scale. Apart from allowing people to receive information without having to turn on one of their personal computing devices, the Ambient Orb was also designed to match the character of a home environment. Similarly, other ambient information displays were proposed, such as the informative art series (Holmquist and Skog, 2003), which communicated information, such as weather data and the arrival of new email messages, through visuals implementing the style of well-known artists. Using this approach, information could be consumed through artwork hung on the wall, or embedded into wallpaper (Skog, 2004), again matching the character of home environments, rather than forcing humans to enter the computing environment (Weiser, 1991).

2.2Pervasive urban displays

Although computers today are ubiquitous, this is far from Weiser’s original vision of computers receding in the background, being there to support people in their day-to-day experiences without being in their face (Aylett and Quigly, 2015). How this vision can achieved is therefore still a matter of discussion (Rogers, 2006; Bell and Dourish, 2007), however the enabling technologies continue to evolve, leading to new forms of digital interfaces and paradigms. Of particular relevance in the context of this chapter is the advancement of digital display technologies as an interface between people and the digital world. Public displays in the form of LED or plasma screens are now pervasive in the built environment. However, their primary focus in many cases is to compete for people’s attention – which follows the same single-attention approach implemented by personal computing devices. The advancement of display technologies in the built environment, it seems, has led to a ‘signage on steroids’ approach. Displays are flashing at passers-by to announce early bird parking rates (Figure 1), and advertising content is given significantly more attention and display space than the information that actually matters to people, demonstrated for example in airport environments (Aylett and Quigly, 2015).

This trend is contrasted by the rollout of media facades as building-scale low-resolution displays, which use LEDs to complement or support the physical topology of buildings. In many ways, media facades were enabled by similar technological advancements that led to the vision and realization of ubiquitous computing. However, it was mostly architects and artists who drove their implementation rather than computer scientists and they have seen little to no attention in the human-computer interaction (HCI) community until Dalsgaard and Halskov’s (2010) paper on challenges for designing urban media facades. On the one hand, architects argue that media should be treated as building material with the aim to support the character and appearance of the architecture (Haeusler et al., 2012). On the other hand, the HCI community has mostly focused on strategies for enabling passers-by to engage and interact with large displays (Hespanhol et al., 2012; Memarovic et al., 2012; Ackad et al., 2013) and media facades (Wiethoff and Gehring, 2012) and studying people’s behavior around them (Fischer and Hornecker, 2012; Tomitsch et al., 2014). We can currently see a convergence of media facades and public display research, manifested through shared discourse venues, such as the International Symposium on Pervasive Displays and the Media Architecture Biennale, which bring together academics and practitioners working across the disciplines of HCI, architecture, design and visual art.

This chapter therefore takes an inclusive approach and proposes the use of media architecture as an umbrella term for a holistic design approach to multi-scale urban displays, from small embedded interfaces to large media facades. To that end, the chapter also attempts to integrate concepts from HCI and architecture with the aim to advance approaches for real-time data access in the city.

2.3Ambient architectural displays

Ultimately, media architecture installations can offer similar characteristics to ambient information displays (Mankoff et al., 2003; Tomitsch et al., 2007). A key element of this approach is that the media layer is appropriately integrated into the built environment. Media facades achieve this integration by using media as architectural building component (Haeusler, 2009). However, in most cases to date the media takes the form of ambient patterns rather than communicating information (Haeusler et al., 2012; Tomitsch et al., 2015).

More recently, advancements in sensing technologies and networked devices, have led to the conception of media façade installations, that communicate information. For example, Colangelo (2014) described an application for the existing media façade on the Ryerson Image Centre in Toronto, Canada, which displays two types of real-time information: first, the wind speed and direction is visualized as a blue wave on the building, and second, occurrences of the hash-tag #homelessness on Twitter trigger a red pulse (Figure 2). Another example of a building-scale media façade that communicates information is the National Football Stadium in Lima, Peru. The media façade installation uses distributed microphones to capture the noise levels within the stadium during games to visualize the mood of the spectator audience through light patterns on the building façade (Haeusler et al., 2012; page 134).

Similarly, media architecture installations of smaller scale integrate real-time data as media input and physical form to convey information in an ambient way. For example, the Hybrid Media Display project15, described in more detail by Alexander Wiethoff and Marius Hoggenmüller in their chapter, visualizes the arrival time of underground services using a combination of text-based information and low-resolution ambient visualization drawing on the metaphor of an hourglass.

3.1The vision of the real-time city

Urban digital media are transforming the city landscape not only on its surface, evidenced through large public screens and media facades, but also through an invisible network of digital service infrastructures, which are continuously collecting and generating data throughout the city. For example, sensors collect information about vehicle traffic, electricity usage, the city’s public transport network, its water network for leakage detection, etc.

In the real-time city, this data is transmitted to some form of control interface and visualized in real-time. There is a trend towards establishing centralized control rooms, in which these data streams are brought together and displayed on large arrays of digital screens for analysis and to inform decision-making. For example, the ‘Centro De Operacoes Prefeitura Do Rio’ in Rio de Janeiro, Brazil, collects data from 30 different government agencies, to monitor the life in the city in real-time 24 hours a day (Ferguson, 2013). The aggregated data, which ranges from traffic to the social mood in the city to weather conditions, is visualized on a giant control interface. The aim of this centralized real-time control and data analytics center is to enable sustainable development and economic growth.

While these data centers offer promising opportunities for making cities more livable, they have seen some criticism, mostly associated with the top-down approach in which they are implemented (e.g. Townsend, 2013; Hill, 2013). In line with this criticism, Kitchin (2014) describes three concerns about the real-time city, being technocratic governance, “corporatization” of city government and panoptic cities. Kitchin links these concerns to the observation that smart city proposals are based on an approach that is narrow in scope, the fact that smart city agendas are heavily promoted by large ICT and service companies, and the risks associated with recording data about all aspects of city life in a central database that is processed through corporate systems.

Following this criticism, there have been arguments that data generated by citizens should remain freely and openly accessible to all citizens. How this data can best be used to the advantage of citizens is still unclear (Townsend et al., 2010). An approach, implemented by an increasing number of cities around the world is to provide open application programming interfaces (APIs) that make it possible for people to implement their own applications using civic data. To date, most of the applications that have emerged through these initiatives are in the form of websites or mobile apps. However, with data about public transport, traffic conditions and other civic services becoming openly available, there are opportunities to connect this movement with the principles of media architecture in order to create new situated urban interfaces that connect citizens with the digital real-time layer of the city.

3.2Accessing the digital layer of cities

The rise of the internet has led to opportunities for innovating the way cities are being managed and run. The term ‘digital city’ is generally used to describe any aspects of city making and governance that involve digital ingredients. For example, one of Sydney’s suburbs was said to be the first digital city in Australia in a media announcement in 201016. The statement listed digital strategies, such as providing free public wifi and mobile apps for parking and other municipal services. Although free public wifi is still heralded as innovation in smart city master plans, the more interesting aspect in the context of this chapter, are the apps made possible through the provision of data. For cities, this is an important shift from not only collecting the data but also making it available to developers and designers in a standardized format. Governments around the world are joining the open data movement, releasing access to civic data, ranging from real-time transport data to garbage collection schedules. This movement is an important first step to release data from the smart city control centers and making it available to citizens.

An increasing number of cities around the world, such as Chicago, further decided not to attempt creating apps themselves but to rather encourage the community to decide how to use this data and to develop their own applications. The city of Chicago even promotes these emerging “civic data apps” through the Chicago Digital initiative and platform17.This move may indicate an acknowledgment of the creativity of citizens, who use civic data to solve problems they observe within their own communities. At the same time, it might be seen as indication that cities lack the technical competence and resources to conceive and deploy civic data apps themselves. An approach that has been adopted by cities and organizations around the world to spark civic data app development are hackathons, in which designers and developers come together to compete for prizes awarded for innovative ideas. Despite their popularity, this raises the question whether the people, who invent such apps are indeed the ones experiencing the issues that the apps aim to address. Nonetheless, these trends depict an important shift in how governmental institutions understand their responsibilities. Rather than safeguarding the data generated by the city and its citizens, they are seeing their role as an interface between the city and the citizens.

A variation of hackathons are so-called “hothouse” events, which involve the mentoring and support of winning participants beyond the event. For example, Transport for NSW, the transport authority for the state of New South Wales, Australia, ran a hothouse in Sydney in 2013, which successfully pushed the development of real-time transport apps (Tompson, 2015).

On a more conceptual level, such civic data apps provide portals into the digital layer of the city. They grant access to data that was previously hidden and not available. Through advanced sensing and mobile technologies, it is now possible to know at what time the next train arrives, to see around the corner for finding parking spots, or to take a virtual glimpse into buildings to find the nearest bathroom facilities with access for people with disabilities. Civic data apps can be seen as protagonists of true bottom-up solutions that enable smarter living in today’s cities. However, to date they are mostly confined to websites and mobile devices, requiring people to a) possess a personal computer or mobile device, b) understand how to use these devices, and c) know about the apps and how to access them.

Media architecture, as described by Brynskov et al. (2013) and in other chapters of this book (e.g. M. Hank Haeusler’s chapter on designing content) provides a framework to transgress the boundaries of personal computers and mobile devices, weaving civic data into the built environment and providing in-situ access when and where it matters. To achieve this, the next section proposes the concept of the city as operating system.

3.3The city as operating system

Smart city solutions offered by large ICT providers are typically sold much like a proprietary operating system that consists of sensors and IT infrastructures, essentially locking governments into that particular provider (Townsend, 2013). Where this approach fails is that these top-down systems consist of standardized modules and restrict any adaptions to be carried out by the provider. In many ways, this is similar to the outdated model used for operating systems on desktop computers in the 1970s and early operating systems for smartphones. Even the first version of Apple’s operating system developed for the original iPhone was originally a closed system that confined the development of new apps to Apple. In 2008 by opening up their mobile operating system, Apple made it possible for anyone with the appropriate tools and skills to develop an app thatwould be available to every iPhone (and later iPad) user.

As described earlier, many cities are already moving towards offering and managing access to civic data through standardized APIs. As such, the city offers similarities to personal computing devices in that it consists of a series of input and output channels. The term ‘operating system’ here is used in a much more conceptual sense compared to how it is used in the context of desktop computers and smartphones. But essentially the city is providing the infrastructure on top of which ‘user interfaces’ are built along with pre-existing input and output mechanisms. Input channels include urban activities, such as traffic or pedestrian flow, and environmental conditions, such as air quality, temperature, light, etc. Output channels include surfaces, such as the street or building façades, and urban furniture, such benches, street lamps, etc.

Of course, the city is a messy operating system and it is difficult to impossible to develop a clearly defined, universal API. But conceptually this metaphor adds to the emphasis that the citizen is given not only as user but also by enabling bottom-up solutions to emerge, opening up new opportunities for DIY media architecture as described by Glenda Amayo Caldwell and Marcus Foth in their chapter. Similar to desktop and smartphone operating systems, providing access to the city’s input channels, allows anyone who has the tools and skills to build user interfaces. Using the concept of the city as operating system moves possibilities beyond the development of mobile apps. Much more, this concept through the lens of media architecture makes it possible to think about city apps as access points sitting on top of the city’s infrastructure.

The next section will take a closer look at these forms of ‘city apps’ and how they can contribute to making cities more livable.

4.1Real-time data access through city apps

The presence of digital technology in the definition of city apps plays an important role to distinguish the design of city apps from traditional urban design. In that sense a park bench is not a city app, while a traffic light push button for pedestrians is a city app. This is essential, as digital technology a) enables people to interact with the city’s infrastructure through the city app or for city apps to respond reactively to the behavior of people, and b) provides the means for mapping digital information into a form that can be accessed by passers-by.

From the perspective of the real-time city, public screens represent a promising platform to reveal digital information in a physical context, with the capability to update their content in real-time. An example for this type of use of public screens that has already found its way into many cities is the use of digital screens at transport hubs to display upcoming services (Figure 3). One might argue that it is much more economic to rely on smartphones to disseminate this information, however such an approach would exclude people who do not own or know how to operate a smartphone. Having access to information in-situ has also been reported to be the preferred method compared to smartphone applications (Rahman, 2012).

Traffic lights can maybe considered to be the first examples of city apps. The first electric traffic light was installed in Cleveland, Ohio, and featured a green and red light to control the flow of traffic at a busy intersection. Traffic lights are an interface into the city’s infrastructure. They use simple lights as output channel to dynamically visualize road conditions to drivers and pedestrians. Traffic lights may even be described as early forms of media architecture, although they only offer a two or three pixel resolution. More advanced implementations of traffic lights now feature numerical displays that show a countdown of the red or green phases (Figure 4). An example for a traffic light that uses digital technologies to add even further value to pedestrians is StreetPong18 (known as ‘ActiWait’ in its next iteration), which turns the traffic light’s push button device into a high-resolution interface. The interface allows people to play pong while waiting for the light to turn green. To encourage social encounters, the game is played with another person waiting on the other side of the road. As outlined by Patrick Tobias Fischer and Eva Hornecker in their chapter, designing shared encounters through media architecture can enrich everyday life in cities. The StreetPong interface also provides a visualization of the remaining waiting time through an animated, declining bar in the background of the game (Figure 5).

City apps can also take the form of situated data visualizations (Vande Moere and Hill, 2012) or street info graphics (Claes and Vande Moere, 2013). Such interventions typically use more complex and larger datasets than the simple information associated with traffic lights or the arrival time of public transport services. For example, in a project investigating the concept of urban neighborhood displays, we used mechanical low-resolution displays to visualize the electricity consumption of households (Figure 6). The displays also offered passers-by a way of interacting with the display through a simple push button, which triggered an animation based on the electricity consumption patterns of all participating households. In another study we used large analogue displays to visualize patterns of domestic electricity consumption along with a neighborhood ranking (Vande Moere et al., 2011). We found that real-time data was perceived to be useful even when it was updated on a very low cycle (once a day in the case of our study). Indeed, the low update rate added a concrete value to how study participants experienced the intervention, as the daily updates became an important recurring part of their lives during the 7-week period of the study. Urban interventions that display complex datasets can increase social awareness and discourse, however their design needs to be carefully informed to ensure people can easily understand the information provided (Valkanova et al., 2015).

4.2The human scale of city apps

Using the notion of city apps places an emphasis on citizens as users of the intervention. This approach complements the focus of media architecture and smart city solutions, which is typically on the architecture, the cities and their governance. Similar to how some scholars in urban planning and architecture have put forward arguments for considering the human scale of cities (Gehl, 2013; Burke, 2016), city apps introduce a human scale to media architecture. Just as the success of mobile apps is heavily dependent on the employment of a human-centered design approach, city apps rely on the citizen being at the core of the design process for their successful deployment.

The experience of cities is even more complex compared to how people experience products, since people compose their own city. As de Waal (2014) describes, “they use a network of cities that is spread over the whole city”, meaning that people live in one place, work in a different suburb across town, shop in their favorite shopping center or sometimes in the city center, and spend their spare time in recreational places, such as an indoor pool or a nearby park. People compose the city as a network of places that meet their particular requirements.

The way people experience the urban environment is therefore not so much a result of strategic urban planning, but rather “driven by the individual planning of the inhabitants of the city” (de Waal, 2014). It is difficult to design for a specific urban experience, similarly to how it is impossible to design the user experience of a digital product. As designers we can only design the conditions for a great experience to unfold (Zmijewski, 2008). On a higher level the design of city apps should therefore employ design frameworks established from other domains, such as participatory design (Floyd et al., 1989) and action research (Lewin, 1946).

On a more concrete application level, it is important to understand the citizens’ needs in a particular environment or situation, and to identify opportunities for city apps to empower citizens. A useful approach is to think about city apps as tools that provide citizens with superpowers. For example, real-time data displays at bus stops allow people to “see” the whereabouts of their service before it arrives at the stop; and digital countdown displays at traffic lights enable drivers and pedestrians to exactly know how much longer they have to wait for the light to turn green.

Ultimately, city apps that are designed around citizens and their needs can help to improve the livability in cities, for example by improving the experience of public transport services, but also lead to more sustainable use of resources, by making the actions of people visible.

4.3The physical context of city apps

City apps are always used in a particular physical context. Many city apps further consist of a physical as well as a digital user interface. A traffic light push button with feedback light, for example, has a physical input component (the push button) and a digital output (the light signal). The TetraBIN project (Bai and Tomisch, 2015) consists of a higher resolution output channel compared to a traffic light push button. It uses an LED screen wrapped around a city bin to turn the act of disposing rubbish into a playful experience. In this example, physical input is in the form of trash being disposed into the bin, which is translated into digital output mapped onto the LED screen (Figure 7).

Media architecture provides a framework for the physical design of city apps, as it specifically described the integration of media and the built environment. Seen through the lens of city apps, media façades exemplify a perfect integration of the digital and the physical user interface. The screen element of a media façade is closely integrated with the building layers, if not a complete integration into a new hybrid structure (Haeusler, 2009). Although from a technical perspective the image production still takes place on certain parts of the façade, the visual imagery is designed to be perceived as part of the built architecture (Tscherteu and Tomitsch, 2011). This is fundamentally different to media displays that are installed without close connection to the underlying building layers, described earlier as urban screens. If an urban screen is attached to a building, the screen and the building façade remain two separated layers both in terms of the technical installation as well as in the way the two elements communicate their function. If urban screens appear as freestanding, independent architectonic elements, they tend to take on the single purpose to communicate media content (McQuire et al., 2009). The same principle can be translated to the integration of digital displays of any scale, from 30-inch plasma screens to small, embedded displays.

In traditional screen-based interaction design, the software usually outlives the device platform. For example, mobile apps are continuously developed, and remain available and functional for new smartphone models. It’s relatively easy to update software and push out a new version if any adaptions are necessary to ensure it runs on the new model. The physical user interface of city apps is much more complex, or in some cases impossible, to replace. Replacing the physical user interface is associated with high costs and time-consuming. Unlike with software updates, updates to the physical user interface cannot be rolled out at the push of a button.

It is therefore crucial to consider the life span of city apps in their design. How long a city app will be deployed for determines not only the implementation but also the design of the city app. If it is meant to be deployed for several years or even decades, it may be necessary to allow any necessary updates or replacements through the design of the physical user interface, similar to Brand’s (1994) notion of ‘shearing layers’ in architecture. A potential solution is to separate the digital user interface from the underlying physical structure, or designing them in a way so that they can be easily replaced. Technology is evolving at such a rapid pace, that it is necessary to allow for specific components to be upgraded or replaced, without having to completely redeploy the display, including fixtures, cabling, etc.