CHAPTER2
Fundamentals
This first section of concepts, Fundamentals, represents a set of perspectives that are useful when considered as frameworks for thinking about how to investigate and plan a design for the problem you are considering. The big difference between this and the following sections is that, in my experience, each of these fundamentals are necessary to be considered in any AT/IA project, whereas the concepts in the sections on Models and Technique may be only applicable to certain kinds of projects. In this fundamental category are the following topics:
•artificial intelligence (AI)/intelligence augmentation (IA);
•design for failure;
•distributed cognition;
•scaffolding;
•situated action/cognition;
•socio-technical environments;
•universe of one; and
•wicked problems.
2.1ARTIFICIAL INTELLIGENCE (AI) → INTELLIGENCE AUGMENTATION (IA)
Short Definition: Instead of using technology and machine learning to replace cognitive behavior (artificial intelligence), use these tools to leverage existing cognitive abilities to achieve the same ends. It is arguable that the history and key developments of HCI are based on this approach. Mice, desktops, cut, and paste are all based on a vision of IA (see the first chapters of Markoff’s remarkable book on the early development of personal computers (Markoff, 2005))
Longer Description: Traditionally, supporting end-users with AI meant that the problem that they were facing, but could not accomplish (e.g., deciding what choice to take, what to do next, how to get a desired result), would be solved for them and the solution presented to the user. The IA approach instead provides tools and information to leverage existing ability of the user to solve the problem. Here is an example. In the near future, smart kitchens will be able to know the contents of the family’s pantry and refrigerator. From this, the AI approach might be to generate a shopping list (or better still to order for delivery) based on past stock levels; the IA approach could instead suggest what items to purchase based on possible meals that could be made with the current food in the home, and also make suggestions about replenishing the commonly used stock in the kitchen, and based on the responses to the suggestions it alters the expectations about what the family needs to buy on an on-going basis. A less futuristic example is spelling correctors: the AI model simply replaces the “wrong” text with what it assumes the writer meant, and the IA version presents a choice of words, including adding the “wrong” spelling to the list of correctly spelled words.
There are several benefits to splitting the act of cognition rather than making the decision for the end-user.
•Let each part of the cognitive process do what it does best. Computational machines count, enumerate, estimate probabilities, and control motors better than humans; humans know their desires, visually identify things (in general and for now), understand the context of the task in a nuanced way, and feel most comfortable with a sense of control. By splitting the task and fitting the appropriate parts into the best place, the task becomes more tractable and the user is more satisfied with the results.
•One of the results of supporting people by leveraging the end-user’s abilities to produce the cognitive act (e.g., making a decision), rather than replacing cognitive acts of the end-user with a pre-calculated answer or decision, is that you give the human the feeling of being active in the process. This supports a sense of personal dignity and autonomy. By putting them in a active relationship with the cognitive process, they are committed and motivated to adopt the system better than just being a passive recipient of instructions. Moreover, it facilitates context-sensitive cognition.
There are two interesting themes in changing the design focus from using AI to using IA to support tasks or decision activities. First is the Replacement → Empowerment axis. When computational support consists of replacement of the cognitive act with pushing a complete solution to the user several things can happen: the answer is not MY answer, the answer is wrong, and the answer may return the desired goal but the intermediate steps are not workable or desirable. The difference between replacement and empowerment can be the difference between adoption and abandonment. When the user feels ownership and participation in the cognitive process they are more motivated to use the result. Next is the Emulate → Complement axis. What is being pointed out there is that in the case of AI the machine/system is not just doing what it is best at but it is also being forced to emulate the human cognitive process (if only at the “goodness of solution” utility function level). However, if the solution process involves both the end-user and the AI system the parts complement each other and thus are more effective. Of course this assumes, like most economic theories, that users are rational (i.e., homo economicus) and choose optimally, which sometimes is not the case at all, something the designer needs to keep in mind.
An excellent example of the efficiency of the IA approach (but not necessarily the empowerment aspects of IA) is the system for self-pricing produce provided in some Carrefour supermarkets in France (Figure 2.1). Typically, in European supermarkets the shopper is expected to self-price their selected produce. This usually involves putting the produce in a plastic bag and looking at a number on the description/price label next to the produce bin, then taking the bag to a pricing scale and pushing the number associated with the type of produce on a screen. The scale then emits an adhesive sticker with a barcode, description, and price of the bag of produce. The way that these stores implement self-pricing is that you bag your purchase, and place it on the scale, repeating this for each kind of produce they want. Based on information gathered from a camera and the scale itself (Bronnec, 2015) (e.g., the characteristics of the produce), the screen produces four possible identifications of the produce. If it is in the list you touch that picture (see Figure 2.1), if not, the scale allows you to input the name of the item on a touch keyboard on the screen. For me it is right about 90% of the time.5 This eliminates the cognitive load of keeping the produce ID number in your head while you are walking to the scale (which can be quite a problem with those of us with short-term memory issues), as well as not forcing you to go to the scale after each selection, you could choose all your produce then do all the pricing.
One of the marks of IA design is that it leverages existing knowledge and metaphors commonly known by end-users. This requires designers to be familiar with the target end-users, an existing requirement for good design, but in this case not understanding the end user may lead to failure in use. This may be particularly important for new adopters or those on the wrong side of the digital divide; what are assumed to be helpful shortcuts become daunting fences.
Figure 2.1: Carrefour vegetable pricing system. Author’s picture, 2015, Carrefour Saint Jean De Luz.
2.1.1CANONICAL PAPER
There are many streams that lead into this topic (see the extended references in Part 3). However, this one source stands out for both an early vision of the direction, and for its wide citation:
Engelbart, D. Augmenting Human Intellect: A Conceptual Framework (Engelbart, 1962).
2.1.2AT EXAMPLES
The first AT example is the MAPS (Memory Aiding Prompting System) (Carmien, 2006b), a research system to support adults with cognitive disabilities performing activities or tasks that are too difficult for them to do unaided. MAPS ran on a PDA (in this case a HP ipaq) platform to present verbal and pictorial prompts in sequence. Furthermore, this set of prompts comprised a script that guided the user through completing a task. The PDA provided error correction functionality via dynamic, situated scripting and “panic button” functionality (using wireless connectivity). A PC-based application provided tools for script creation, modification, and sharing with other users via a web-based repository of scripts. A web script repository has numerous tested script templates assisting the design of new personalized successful scripts.
MAPS research has explored independence specifically in the following contexts: (1) to extend the ability to choose and do as many activities of daily living as possible; (2) to be employed, but without the constant or frequent support and supervision of a professional job coach; and (3) to prepare meals and shop for weekly groceries. Independence is not at odds with socialization; it is the foundation of inclusion and engagement in society.
The MAPS Environment
MAPS (Carmien, 2007) consists of two major subsystems that share the same fundamental structure but present different affordances for the two sets of users: (1) MAPS-DE, for caregivers, employs web-based script and template repositories that allow content to be created and shared by caregivers of different abilities and experiences; and (2) MAPS-PR, for clients, provides external scripts that reduce the cognitive demands for the clients by changing the task.
The MAPS-Design-Environment (MAPS-DE)
The scripts needed to effectively support users are specific for particular tasks, creating the requirement that the people who know about the clients and the tasks (i.e., the local caregivers rather than a technologist far removed from the action) must be able to develop scripts. Caregivers generally have no specific professional technology training nor are they interested in becoming computer programmers. This creates the need for design environments with extensive end-user support to allow caregivers to create, store, and share scripts (Fischer and Giaccardi, 2006). Figure 2.2 shows MAPS-DE for creating complex multimodal prompting sequences. The prototype allows sound, pictures, and video to be assembled by using a filmstrip-based scripting metaphor.
Figure 2.2: The MAPS-design-dnvironment for creating scripts. From Carmien and Fischer (2008).
MAPS-DE supports a multi-script version that allows caregivers to present the looping and forking behavior that is critical for numerous task-support situations. MAPS-DE was implemented on a Microsoft OS (Windows 2000 or XP) platform connecting to and supporting PDAs that run the WIN-Compact Edition (WIN-CE) operating system.
The MAPS-Prompter (MAPS-PR)
MAPS-PR presents to clients the multimedia scripts that support the task to be accomplished. Its function is to display the prompt and its accompanying verbal instruction. MAPS-PR has a few simple controls (see Figure 2.3): (1) the touch screen advances the script forward one prompt and (2) the four hardware buttons on the bottom, which are mapped to: (i) back up one prompt, (ii) replay the verbal prompt, (iii) advance one prompt, and (iv) activate panic/help status. The mapping of the buttons to functions is configurable to account for the needs of individual users and tasks.
The current platform for the MAPS-PR is an IPAQ 3850. The system was implemented for any machine that runs the WIN-CE operating system. MAPS-PR has been installed on units that have cell phone and GPRS functionality. The prompter software was originally written in embedded Visual Basic, and then ported to the faster and more flexible C# .net environment. The prompter software supports single-task or multi-task support.
Figure 2.3: The MAPS Prompter (MAPS-PR). From Carmien and Fischer (2008).
More to the point, the MAPS system was tested in training a 19-yr old with developmental disabilities to sort clothes in a very specific way (i.e., by type, then by size, then by color (in a very definite order)). She was working with a job coach in the traditional way and switched to using MAPS to support doing the task. In this case the intent was not particularly to train her to do the job but that using the MAPS prompter would be a cognitive orthotic and this task configuration would allow her to easily switch to new tasks with the only cost being the creation of a new script. This worked very well for her; the job coach said that the “training” time was cut in third. However, there were two problems the coach wanted to solve for the job trainee: (1) occasionally she would run out of work and it would be good for her to have another task that she could always do; and (2) when she decided that there weren’t any more tasks to do the prompter offered a task that encouraged her “soft” skills (e.g., going to the supervisor and telling them you have run out of sorting to do). In the job coach’s experience “Lacking this skill will “deep six our people” if they just stand there after completing a task. They need to be able to find something to do until they see their supervisor” and “90% of this is a soft-skill.” In looking at what MAPS could provide to the situation, she felt that it would be difficult to put soft skills into a script. As a result, these last sub-scripts were quite critical; a frequent cause of people with cognitive disabilities loosing semi-sheltered jobs that are out in the public is a deficiency of these “soft skills.” An example of this is the situation where the worker comes to the end of the task and just waits there, or arriving at work and not greeting the supervisor or other employees (Carmien, 2006b).
In this case the job trainee had enough cognitive ability to recognize the end of the task but not enough to initiate the soft skill activities. MAPS was modified (Figure 2.4) so that small initial prompts of subscripts were provided from which to choose. The ability to choose from a pictorial menu empowered the person with cognitive disabilities and motivated her to “own” the sub-task thus increasing the probability of interacting smoothly with co-workers and supervisors.
Figure 2.4: Selecting sub-scripts. In MAPS Prompter. From Carmien (2011).
2.1.3CONCLUSION
This concept is key to designing AT for people with cognitive disabilities, because these tools can support the cognitive deficiencies the user brings in a way that exactly compensates just enough to get the task done. The act of making IA systems is more-or-less figuring out how to flexibly implement “just enough.”
To a large extent, moving from an AI approach to an IA system is deciding where to place the divider between what the systems brings to the task and what the user brings to the task. By doing this optimally the person using the system can be engaged and feel genuinely autonomous. Properly done, this increases the generalizability of the system in terms of different tasks, and leverages the user’s abilities to recover from potential errors. The tricky part is to properly access what the user can offer, what must be supplied by the system at minimum to succeed and the fit between the two. These concepts will be discussed in Sections 4.1, 2.2, and, most fundamentally, 2.3.
2.2DESIGN FOR FAILURE
Short Definition: Complex AT is by its very nature prone to functional failure or not achieving the intended goal. As a result it is necessary to design these systems with the ability to robustly accomplish their intended purposes by having the ability to dynamically mitigate such errors.
Longer Description: Design for failure (DfF) is a response to the inherent, but not obvious, flaws of complex AT that is used as cognitive support. The basic process is to detect an error, classify it using a “two-basket” approach for making decisions about whether to provide human intervention or to automatically mitigate the error condition, and finally intervene in the error with the appropriate mitigation strategy. This section describes a framework for classifying, capturing, and designing possible mitigation strategies. This framework is comprised of four types of failure events. Additionally, it provides a multi-step program for finding vulnerable parts of a system. This is a particularly long section, as it is not only intended to discuss the concept but also to provide an “algorithm” for creating a design for failure system.
2.2.1INTRODUCTION
AT designed to augment cognitive ability has a history of fragility, difficulty of generalization (Carmien, 2011), and complexity (LoPresti et al., 2004) leading to a lack of wide adoption. Critical parts of the problem are the fragility of modern IT technology and the unbounded range of the application. Smartphones run out of batteries, network connectivity comes and goes, small devices can be easily lost or stolen, all leading to a complete halt of the assistive or cognitive orthotic support (Cole, 1997). Additionally, these systems are typically used in uncontrolled environments. For instance, there are traffic jams (Carmien et al., 2005) and unexpected complications of simple tasks. These situations, when looked at analytically, are usually solved with situated cognition (Suchman, 1987; Lave and Wenger, 1991), an ability that may be missing from the end-users of these systems.
In the CLever project (CLever, 2004), a series of interviews with parents of developmentally disabled young adults explored the use of hand-held portable devices to guide their children in using public transportation. Many caregivers expressed concerns about technology leveraging their children into situations that, if the support failed, would have catastrophic consequences (Carmien, 2007). While further exploring this topic (Sullivan, 2004), the CLever group collected many examples of people with limited cognitive ability “disappearing” in the public bus system, only to turn up many hours later due to becoming lost (Sullivan, 2003). Similar interviews with parents about multimedia script support of jobs for their cognitively disabled children led to their concerns about the system not being able to respond to unexpected situations in performing their jobs. In their experience, lack of immediate support often triggered a panicked response leading to employment problems (Carmien, 2007).
Why and how do these technologically complex and unbounded applications invite failure? The first and most common concern is that the technology used is not robust; it works in most cases but not reliably enough for the target population. Often, typically skilled end users are proficient enough to work around these problems or are able to triage error messages/failures, but frequently those users with cognitive deficiencies cannot overcome them. The systems themselves are built on complicated and layered fragile operating systems, services, and devices. It is common to experience computer operating systems halting, particularly in relation to networking. Further, the user population of the application is too small to guarantee thorough testing. Contributing to failure in mobile systems are connectivity issues that are context dependent. Examples of these are interactions between radio and buildings/topology leading to intermittent service. More frequent and with larger consequence are power issues when away from a charger, and running out of power.
The complex nature of these systems, having system server, mobile device, wireless connectivity, and unexpected environmental concerns (e.g., traffic jams, misidentified signs/tags, etc.) leads to having many failure modes. The support for the user and the user’s interactions often need multiple sequences of correctly happening events/actions to succeed. An excellent example of this is the COACH (Cognitive Orthosis for Assisting with aCtivites in the Home) project (Mihailidis, 2007) which had to adapt to guiding people with dementia through many possible paths to correctly washing their hands. Furthermore, younger people often design interfaces for these systems. Without careful planning, designs are produced containing metaphors and affordances not obvious or right-sized/text/color contrast, etc.
Complex systems designed for the “real” world (in contrast to medical or sheltered workshop environments) are, by nature, unbounded. Real-world events are not nailed down, controllable, or predictable (with some statistical exceptions). The intersection of user, context, and device present a type of combinatorial explosion problem space of possible failure/error states.
Responding to this, I have developed an approach called design for failure (DfF). This is not a particularly new way of designing complex technology. The ubiquitous RAID systems (Patterson et al., 1988) that use a number of hard drives to provide reliable data storage is a design based on the assumption that hard drives fail. DfF is also an approach in software engineering in general (Sommerville, 2001). In this case, the approach means to spend a significant amount of design time on design focusing on the inevitable failure of the system. To obtain this, a fair amount of time must be spent on developing failure scenarios. This DfF approach can also ensure good design for the system when it does not go into a failure mode, not unlike the way well done accessible design can lead to better design in general.
This section will focus on some general principles:
•device failure;
•environmentally caused failure;
•failure due to user action; and
•failure due to supporting technology in special contexts.
Further examples from two projects (ASSISTANT (2012a) and MAPS (Carmien, 2004b)) illustrate the DfF approach.
2.2.2DOMAIN BACKGROUND
The sorts of systems described here typically extend ability or attempt to aid in compensating for lost ability of people with cognitive disabilities. Therefore, this is a discussion of orthotics (Pollack et al., 2003) rather than AT per se. An illustration of the difference is a navigation system that supports your use of transportation and deciding which way to walk, but does not actually transport you there. Examples of these types can include navigation systems, task support systems (for work or recreation), communications systems, email and browser systems, and applications on PCs and tablets specialized for people with cognitive disabilities.
System failure means that an error has occurred in the expected behavior of the system, in this case an error in a socio-technical system. Socio-technical systems are composed of a user, a technical system, work practices, and the use context. Classical human error research (Reason, 1990) can give some insight into the problems with designing complex AT. However, the incidence of comorbidity (multiple disabilities) results in an effective “universe of one” as the design target (Erikson, 1958). The complexity of user baseline ability and daily and, in general, temporal variability of these abilities (Cole, 2013) make the assumptions that research based on typical or even highly skilled people difficult to extrapolate from.
Often the big difference between these technologies and more mainline AT is that obtaining the desired result only happens after a long string of connected events, and the failure at any one point (missing the bus stop to exit on) can cause the entire task to fail. The number of things that can go wrong (e.g., the grocery store has moved the contents of one isle to another side of the store) can make the error space very big.
Finally, the cognitive and experiential resources of the targeted user can be quite limited, making the system’s use much more “brittle” than when used by a more typical population. An example of this is the use of a smartphone by an elder who may find accessing the on-screen keyboard tricky (too small keys on screen, popping back and forth between entry modes i.e., keyboard disappears without warning). This can become upsetting enough to trigger abandoning use of the application and perhaps any smartphone application entirely. This is magnified by the resultant AT abandonment (Phillips and Zhao, 1993) in exactly the population whose use of it may be life changing, not just convenient.
2.2.3BACKGROUND
This study of DfF is illuminated from two perspectives: (1) human error studies and (2) the HCI version of distributed cognition (Hollan et al., 2001) and situated cognition/action (Suchman, 1987). Between them, these complex systems can be made more robust and adoptable.
2.2.4ERROR
Modern, cognitive science-based studies of human errors start with Norman’s (1983), Lewis and Norman’s (1986), and Reason’s (1990) work in the 1980s and 1990s. Their work on error classification (slips and mistakes) and turning an analytical eye to attention and memory as critical components supported design guidelines such as “forcing functions” and appropriate system responses to internally recognized error states. Dekker (2006) added to this the deep examination of error, describing it as systemic behavior. By seeing error as being comprised of the user, the system and the environment, designers can focus more clearly on capturing and mitigation of the error.
One of the more useful discoveries by these researchers is the actual distribution of errors in the world. The possible enumeration of incorrect actions and errors in any given task is a very big number, approaching the size of combinatorial explosion if you count all the elements and links that have to be present and happen correctly to perform even a simple task.
Fortunately, the reality is different. Human error is neither as abundant nor as varied as its vast potential might suggest. Not only are errors much rarer than correct actions, they also tend to take a surprisingly limited number of forms, surprising, that is, when set against their possible variety. Moreover, errors also appear in very similar guises across a wide range of mental activities. Thus it is possible to identify comparable error forms in action, speech, perception, recall, recognition, judgment, problem solving, decision-making, concept formation, interpretation and the like (Reason, 1990).
Thus, it is more like a Pareto distribution, which frees the designer to work on the 20% of error types that constitute 80% of the actual errors.
2.2.5INTELLIGENCE AUGMENTATION
The second important part of this problem is that these complex systems are designed to provide a cognitive orthotic for elders, children, and people with cognitive disabilities. These systems are designed not to replace intelligence or cognition but to leverage existing abilities to compensate for missing ones (CLever, 2004; Fischer et al., 2004). This is an AT variant of the approach started with Engelhard’s description of intelligence augmentation (Engelbart, 1962). Intelligence augmentation (IA) in this domain depends on the correct application of distributed cognition. The capturing of the error state may or may not depend on the process of distributed cognition (i.e., the cognitive artifacts). However, mitigating the error within the system (in contrast to bringing in a helpful human) will involve generating and delivering such artifacts.
Again, the most appropriate (e.g., humane, require the smallest effort, and best fitted to the user) response is to support the user in their desires (Fischer, 2001b). This may be continuing to the end of the task or it may be safely and cleanly abandoning the task or returning home.
Strategy and Examples
The Two-Basket Approach
The discovery that the majority of the cases of error or failure are clustered in only several types of errors makes the problem tractable. However, there are still these events that can be identified as possible errors but cannot either be classified with confidence, may not be amenable to solutions, or may only be amenable to solutions that themselves are so complex that make them (the solutions) highly vulnerable to failure themselves. Those errors that can be classified and easily mitigated are put in one category and the (less probable) errors that cannot either be classified or, if correctly classified are not easily mitigated, into another. Fortunately, the second basket is statistically much smaller than the first basket (e.g., sleeping past the bus stop vs. a bus driver just leaving the bus). Therefore, for the second basket, the strategy is to bring in a human, ranging from a friend/caregiver to possibly emergency personnel.
Another simplifying strategy is not to focus on the cause but work with the error in a “stateless” fashion. Of course, if possible the end user and/or a caregiver should be informed that the phone battery is not holding a charge, but the important issue is the best way to help a user with a dead phone.
MAPS and ASSISTANT Projects
The two systems used as examples in this section are MAPS and ASSISTANT. MAPS, the author’s dissertation project (Carmien, 2004a), is a system to support performing tasks by the user which cannot be done unaided. MAPS used a PDA-based prompter, guiding the user by displaying task segments that the user can perform. A set of these effectively leads the user through the accomplishment of the desired action. MAPS consisted of software to display scripts of prompts and collect performance data about use and a PC-based script editor that allows the creation/editing of tasks scripts using user supplied photos and recorded verbal prompt instructions (Figure 2.2).
The editor also provided utilities for loading the scripts onto the PDA-based prompter and collected performance logs. Supplementary functions of the MAPS system include a set of template scripts (>300) on a MAPS server on the internet and a prototype of an error trapping/help summoning sub-system based on queries embedded in each script prompt which, when triggered, sent SMS messages to a designated caregiver’s cell phone. The system also includes undo functions, video-based help features, and an error-trapping visual programming environment.
ASSISTANT’s (2012a) objective was to provide accessible support for the use of public transportation by older adults. ASSISTANT did this through an online application for trip planning and the provision of guides for multi-step journeys (Figure 2.5), with information provided to help the end-user to get from the last transit stop to the final destination. The main target group of the ASSISTANT project is mobile older people, aiding them in using public transportation by providing only relevant information, at the right time and in the appropriate format, to the user via audio, visual and haptic cues. ASSISTANT provided safety and security by providing error-trapping and remediation functionality. The user gets waypoint directions via prompts on a smartphone, both in written and audio formats, as well as with a map.
The ASSISTANT system identified the vehicle to board and signals to the user its arrival. The system also informs the user of when to exit the bus/metro using real-time location data. ASSISTANT provides a web browser-based route editor and personal preferences editor (Figure 2.1), a server that coordinates routes and checks for errors, and an application—a Personal Navigation Device (PND) (Figure 2.2)—running on a smartphone (Android or iPhone).
Figure 2.5: ASSISTANT route planner. From Assistant Project (2015).
For each topic there is a section describing the error’s cause, how the error is recognized, and the mediation the system applies. Note that the cause section does not describe the chain of causality that produces errors, but describes the proximate cause of the problem.6
Device Failure
Cause: It is increasingly common to find support systems of this type being composed of a server of some sort and a mobile device. Server failure is a fairly uncommon (shy of original software bugs) occurrence and can be reduced further by hardening the device (i.e., uninterruptable power supplies, RAID redundancy, etc.) or moving the service into the cloud. However, both the mobile device and the connections to the mobile device can be quite fragile. Accordingly, there could be a dead battery, a lost phone, a stolen phone, and any number of things that might make the phone inoperable by the end user. The connection between the phones may fail too, there may be no phone service, there may be no GPS service, or worse yet there may be no data connectivity between the server and the phone. Finally, the phone may be partially broken—the screen may break, the Bluetooth earpiece may fail, etc.—hardware problems of all kinds that allow some phone functionality (typically as just a phone) but not other critical abilities.
Capture: The capturing of error states in the device failure category typically requires repeated polling of the possible failure points. In the ASSISTANT project, this polling happens on both sides of the vital mobile server connection. Therefore, at the smartphone side the ASSISTANT application monitors the status of the GPS service, the phone, and data connections and the battery level (Figure 2.6). On the server side, there is a periodic polling (ping) of the smartphone’s connected state and part of every data exchange is the status of the battery charge.
Figure 2.6: PND notification screen. From Assistant Project (2015).
By collecting this data the system can flag connectivity and hardware problems in real time (Figure 2.7). However, it is important to discriminate between intermittent problems and real problems. This is particularly important to prevent type 1 errors, i.e., detecting and responding to an error when there was not that problem. False positives can contribute to AT abandonment (Kintsch and dePaula, 2002) by frustrating the user with false alarms.
Mediation: In the case of ASSISTANT device/connection failure there are, like the sensing, two sides to the mitigation activities.
Figure 2.7: Navigation guidance (left: with GPS, right: with no GPS). From Assistant Project (2015).
On the server side, when the pinging of the smartphone fails over a predetermined period of time (to accommodate temporary intermittent disconnections), the system sends a SMS message to a contact the end user has specified. This is also called a “dead man switch” in other engineering contexts. The un-implemented part of this design in ASSISTANT specifies that the message be sent to the first person in an escalating list, if that first person, say a cousin, does not send a SMS reply to the sender in a specified time the system then sends the same SMS to the next contact, and so on until a contact replies with a SMS that indicates that the message has been received and that they will be responsible for the user with the disconnected phone. The last entry in the contact list is usually an emergency service such as the police of a private contractor. In this way there is a safety net below the user that will always provide support should things go terribly wrong. Note that the end user can choose not to have this function at all or not to call in the police or an ambulance service; this can be specified in the preferences pane of the route editor. The content of the message has in it as many specifics as the system knew at the time of disconnect so that the called person can then provide appropriate help. An example of the SMS is: “Hello, this is <my name>, it is <time> on <date>. I am sending this to you at <time> on <date> From my phone <phnumber>. My location right now is <currloc> and I am going to <dest> on <last bus taken> and I think I am in trouble. I need some help.” If the server determines that this is a case of the battery running out, this is included in the message. This single contact SMS notification is currently implemented and tested in the ASSISTANT system.
The PND informs the user of low battery conditions with a dialogue box, and every screen in the navigation set can display the loss of data, GPS, or phone service to the user. When the real-time vehicle location data provided by the public transportation administration is not available, the navigation screen informs the user that it is now giving directions based on the route schedule and not real-time data (see Figure 2.7). By giving these updates of potential problems the user can figure out a workaround to the loss. Alternatively, by pressing a button, the user can have a SMS message sent to contacts from the phone with appropriate status information. The end-user also has the option of calling the contacts via the phone directly with a push button on the help screen.
Environmentally Caused Failure
Cause: Another mode of failure, outside of the user’s actions is when the environment changes unpredictably, in a way not expected either by the user or tacitly by the task execution sequence.
An example in the ASSISTANT system may be that the route has been changed at the last minute by the transportation agency. This may not yet be reflected in the transportation systems database. Perhaps there is an accident or some unexpected construction or a very popular event has caused a traffic jam so large that the driver has decided to drive around the entrance to the stadium rather than in front of it.
For a user of the MAPS system the problem may occur when the arrangement of the cleaning supplies for the job that the MAPS prompter guides the person with cognitive disabilities to do (which was too complicated to do without) has been moved.
Capture: In ASSISTANT the PND and its server monitor the position of the user’s smartphone and the location of the vehicle. It continuously compares that with the expected location and time according to the route that was loaded into the PND from the server at route creation time. Additional deductions from this can be made from this data. For instance, if the location information indicates that the PND is moving very slowly (0–3 MPH) when the user is expected to be on a tram (10–25 MPH) the system can deduce (with due concern for the problems type 1 errors can cause) that the user is now walking or standing still.
In MAPS, the log of the user stepping through the series of multimedia prompts that guide her to do a task may also have time stamps. MAPS can conclude that there is a problem if the user stays on one prompt for a much longer time than would be broadly normal for this person to do for this segment of the task, or (more interestingly) the user is “rocking” back and forth over two or more prompts, centered on the step that is to be accomplished at this stage of the task. The experimenters also observed the user going through the task steps very rapidly, much more rapidly than could be accounted for by actually doing them. It turned out that she had memorized these steps (from previously cooking exactly these cookies in her special education classes many times before) and one of the cooking instructions triggered her performing the next five steps. As a result, she just smoothly did them all at once. While not an error, in that the guided task was accomplished, it did signal that this particular 36-step cookie cooking script could be, for this particular user, condensed to incorporate the “skipped” steps into the one cumulative step.
Mediation: ASSISTANT, when it determines, by real-time location and velocity tracking, that an error state has occurred, it starts the error recovery process. This may mean (1) instructing the user to exit at the next stop and recalculating the new best route to the goal, (2) informing the user of what is going on, (3) pushing the new route to the PND, and (4)starting the route guidance. In designing this mitigation, the design team first struggled with elaborate plans for both guiding the user back to the place where their route differed from the planned route and then getting them back on the original track. This was both pragmatically difficult and hard to represent to the user. However, after several attempts it was discovered that it was much easier (and able to reuse code and GUI to do so) to simply generate a new plan. The new route was calculated starting from where they were now, and was aimed at the original target destination. This was implemented and tested in the field prototype with positive results. If the user set in the preferences panel of the route planner that they are uncomfortable with this happening or with any new route that has more than n steps (i.e., takes three different buses and a metro ride to get to the goal from where they are now) (Figure 2.8), the server initiates a SMS message to a trusted contact to resolve the problem. This resolution may involve “rescuing” them or just getting in touch with a contact by phone for some reassurance.
Figure 2.8: Setting the users preference of the maximum number of steps in a route mitigation. From Assistant Project (2015).
Failure Due to User Action
Cause: This category contains a larger percentage of difficult to capture and categorize errors than the others. Users can be very clever when trying to accomplish tasks. These tasks may be a job or navigating routes. In doing so, the mismatch between metal models and environment/device may generate actions that are legitimate attempts to move forward but errors nonetheless. While these errors, when captured, may frequently fall into the second categoty (those whose remediation requires human intervention) the research-based statistics of clustered mistakes still holds, and the system can still be quite effective. Sometimes the user is unable to make a decision about which way to proceed in following the route instructions. In interviews with Colorado Front Range travel trainers for young adults with developmental disabilities (Sullivan, 2003), the most frequent scenario brought up was sleeping through their bus stop. Another common concern is that the environment would be so noisy that the passenger might not hear their stop being called out. In the MAPS project a user could take too long in doing a task (Carmien and Gorman, 2003) or in a job requiring moving through a building (e.g., delivering corporate mail) the MAPS prompter may not be triggered by proximity to a Bluetooth beacon at one of the mail delivery stations (Gorman, 2005).
Capture: The ASSISTANT server and the PND both monitor and compare location and velocity information with what should be happening if the correct route was being followed, especially when correlated with vehicle real-time location information. Capturing the error state this way gives not only the existence and type of the error but also the current location and time of the user.
In the MAPS system, a prototype had a time that each task in teh script was expected to take. When the advancement to the next step prompt did not happen for a significant about of time over the expected task duration an error was flagged. A prototype of the caregiver editor provided a set of screens to build up an error trap and corresponding mediation plan, as shown in Figure 2.9.
Figure 2.9: Error trapping and mediation insertion into prompt of a script. From Carmien (2005).
Mediation: For some user errors a good way of mediation is for the system to provide better ways to support the user in making decisions. So ASSISTANT, which is a waypoint-to-waypoint navigation guidance system, also provides a map to both inform the user of their current location and also reassure them about their progress on the route. Often the problem is not with representation of the route to the user, but a result of a lack of attention to the PNDs instructions. To mitigate the lack of attention the PND can be programmed to vibrate when the next step on a task is about to be presented. The use can also choose to have the screen blink in a similar situation. The default mode of prompting for the route instructions is through a Bluetooth earpiece, which does not require continuous attention that instructions on a smartphone screen do—a push of information rather than requiring the end user polling the status of the screen.
When the ASSISTANT server does determine that the user is, for whatever reason, not continuing on the correct route at the proper velocity, it flags an error state, recalculates the route, and, if appropriate, instructs the user to exit the vehicle. The user can also initiate this process via the help screens (Figure 2.10) of the PND application, and on the same screen, the user can summon a contact to help them.
Figure 2.10: PND help screen. From Assistant Project (2015).
MAPS sent SMS messages with the location of a step in the task that had been not reached within a caregiver/end user specified amount of time. Similarly, skipping steps or taking too long to accomplish a task can (depending on the user’s preferences) trigger redisplay of the missed task step.
Failure Due to Supporting Technology in Special Contexts
Cause: High complexity navigation or task-guiding applications are embedded in a matrix of supporting and helping applications and systems. For instance, ASSISTANT relies on WebServices for messaging and remote procedural calls between its modules, on GPS for location and velocity information, on SMS for contacting help 4G mobile data service for connecting to the ASSISTANT server, and on Bluetooth to connect with an earphone. All of these have different levels of reliability, range, and sensitivity to environmental conditions.
One of the “killer applications” in the 21st century is the various GPS systems offered for automotive and other vehicles. These, combined with digital maps and simple AI-based optimal route generators, have changed the way that we drive and travel.
They are reliable (depending on the correctness of the map used) and widely used from family vacations to taxis to police vehicles (and of, course by the original users, the armed forces). However, commercial units are not reliable for pedestrian urban navigation (Weimann et al., 2007; Andersson, 2012). This is mostly due to the well-known canyon effect. In the process of designing the support system for the “last KM” (the trip from the last transportation stop to the door of the route’s goal), the design team in Helsinki ran tests (one example is Figure 2.11) that led them to design around the use of GPS in the standard way.
Additionally, in ASSISTANT, while the transit systems did have reliable GPS systems that were coordinating with their administration centers, and the transit systems provided ways for us to utilize that information, sometimes the real-time location of the transportation systems vehicles are not available.
Figure 2.11: Actual urban route taken vs. smartphone GPS. From VTT (2013).
Capture: This is a good example of the two typical approaches to working with this class. In some cases the best approach is to work around the problem. That is to say, by avoiding the standard use of the untrustable technology. In other cases, capturing and providing mitigation is more appropriate. An example of the first approach was avoiding altogether the typical urban use of GPS and an example of the second approach was having the design sense the lack of real-time data when the calls to the real-time data server returned an error message, and then providing a workaround.
Mediation: ASSISTANT designers acknowledged the unreliability of existing GPS-based navigational aids for pedestrian navigation guidance, but knew that there was useful knowledge that the GPS gave and were really motivated to produce something for the elders that was reliable and would avoid giving any guidance that might frustrate the user and trigger abandonment (Kintsch and dePaula, 2002). The solution was three-fold.
1.Provide an appropriately sized map for the user from the last stop to the goal. This was possible because the system knows where the user is when the bus or metro has reached their last stop and they have acknowledged debarking (Figure 2.7, left).
2.Provide a compass pointing to the last goal during the walk from the stop to the door of the route’s end. How this was done was averaging the GPS signal over time and knowing absolutely where the end goal was and having a reliable compass in the smartphone. The drawback is that the compass pointed not at the next waypoint but directly at the front door of the goal, but this was carefully explained to the end-user in the accompanied short user’s manual (Figure 2.12, right).
3.Use this same averaging over time technique with the GPS position to determine if the user is walking the wrong way and alert them of this with the smartphone. Again, it was important to gather average locations and choose a span of time (typically one to several minutes) that they were going the wrong way so that small movements were not misinterpreted as being confused about where the end goal was. This was particularly important as working with the compass may require backtracking to walk around buildings or other obstacles, again driven by the line between providing useful timely guidance and raising false alarms that contribute to system abandonment.
Figure 2.12: Last KM guidance (left: map view, right: compass view). From Assistant Project (2015).
The other problem in ASSISTANT was addressed by changing the guidance screen with adding text on the screen alerting the user that the schedule is being used, and the user manual advises the user that when that happens they need to be a bit more flexible in terms of planning the next steps (i.e., the bus may arrive in something like 3 min). Field tests so far show this to be small enough not to compromise the reliability of the system.
Developing a Culture of Design for Failure
A culture that supports DfF sees failure as a whole system attribute not only a human problem. Dekker points out that we need to move from seeing error as the cause of a mishap to seeing that mishaps and failures are symptoms of deeper, systemic, trouble. He calls this the New View of human error (Dekker, 2006) and one of the marks of this approach is seeing complex systems as not intrinsically safe, in contrast to the more traditional view that complex systems are basically safe with human error undermining that basic safety. With this figure ground reversal it becomes much more natural to find the points of potential failure.
Design for Failure with Scenarios
DfF works well with a scenario approach (Rosson and Carroll, 2001), but the traditional scenario design approach needs to be extended to include branches in action. It needs to anticipate, at each critical point, where things often go wrong, how to capture, and how to mitigate. Here, domain experts become invaluable, both in identifying critical points and estimating the relative probability and consequences of failure.
DfF error scenarios can be generated by enumerating failure points from anecdotal narratives, from end-users and proxies, and from literature/best practices collections (among other sources). The basic outline of our approach has been:
1.prioritize failure importance from probable frequency, consequence (i.e., danger) probabilities, context (i.e., time and place), and results (i.e., confusion matrix and type 1 errors lead to abandonment, type 2 errors lead to danger);
2.determine ease of detection for each error scenario; and
3.based on 1 and 2 sort into the two baskets. In general, high consequence, low frequency and/or hard to detect events are sent to human support and high frequency, easy to detect, easy to mitigate, and lower consequence are good candidates for automatic error correction.
A DfF approach adds a requirement for a module to monitor and continuously evaluate the progress of a system-supported activity. Additionally, the system should be easy for the end-user to autonomously generate an error flag, as well as provide any summoned help context information (and possibly a log of the user action, depending on privacy issues).
Category |
Mitigation Approach |
Device failure |
Dead man switches—constantly monitor the critical device (e.g., ping every 20 s) and if no response in reasonable time automatically summon help |
Environmentally caused failure |
Notify user when captured, replan guidance if possible, or else notify contact person by phone or SMS |
Failure due to user action |
Offer to rework guidance (e.g., “re-calculating route” on automobile GPS) |
Failure due to supporting technology in special contexts |
Use what parts of the technology that are functioning and provide as much decision support as possible (i.e., “there is a printer within 3 m of you, it may be inside the room you are outside of”) |
One nice property of incorporating DfF into the system from the beginning it that it lends itself very nicely to iterative improvement. Once status monitoring, error trapping and mitigation modules are implemented it is easy to include increasingly sophisticated error capture and mitigation technology, starting with being rule based and progressing to incorporating real-time machine learning approaches.
DfF adds a level of a robustness to complex cognitive AT as well as removes a common trigger for abandonment (which is the fate of up to 60% of high functioning AT (Reimer-Reiss, 2000)). But the few examples here show just how difficult implementing this approach is, not just in design but also in system architecture. Moving from well-bounded environments (interacting with just the screen and the network, without relying on external context) to dynamic AT in the wild will require both careful examination of what can be done to support existing skills of the user, what can be done by the system, and what will require external human assistance (Table 2.1). Further research will increase the percentage of the first two, but there will always be a need for the last.
2.2.6CANONICAL PAPER
As I have pointed out, the idea of DfF is not new to engineering in general, however as a bounded HCI problem, there are few publications that focus on this principal. A good introduction to this is in classical engineering terms is Patterson et al. (1988).
2.2.7AT EXAMPLES
The discussion of the DfF approach in the ASSISTANT and MAPS system above are two examples of how design for failure works and how to design with DfF.
The COACH Automated Prompting System
The Assisting with aCtivites in the Home (COACH) system (Mihailidis et al., 2008; Hoey et al., 2010) (Figure 2.13) is a Cognitive Orthotic system, e.g., a system that leverages existing skills to preform tasks that the user cannot do unaided or unguided. Within the spirit of IA most of the systems described in this book are orthotic (aiding the user to use what they have) rather than prosthetic (replacing what the user does not have). For example, contrast back or forearm-wrist braces with artificial legs or hands. This prototype is a result of a multi-discipline team and a reasonably successful third generation that took them from 2002 to 2010 to develop. The COACH system is an excellent example of how many different domains and person hours it takes to actually make an IA cognitive orthotic that works. The most effective (and needed) supports for independent aging and efficient use of caregivers are those categorized as activities of daily living (ADL) (and also Instrumental activities of daily living which support ADLs). COACH is designed to support proper handwashing by people with dementia in a home environment.
Figure 2.13: COACH system handwashing. From Mihailidis (2007).
The COACH system is composed of three modules: hand tracking, planning, and prompting (Figure 2.14). The hand tracking module uses video images (e.g., an overhead camera mounted above the sink in a washroom) to identify the position and actions of the user’s hands and objects. Planning module translates the image information into washing progress information using a partially observable Markov decision process (Hoey et al., 2010). Finally, the prompting module, if directed by the planning module, provides guidance, in appropriate visual and audio formats, to the hand washer. The selected level of assistance is determined by an estimate of current user’s lucidity and responsiveness provided by the planning module. COACH in use is continuously guiding the end-user, and the planning module can accept several paths or sets of sub paths for correct washing, there are multiple correct paths in COACH washing (Mihailidis, 2007; Czarnuch and Mihailidis, 2012; Czarnuch et al., 2013) (see Figure 2.15)
Figure 2.14: COACH modules and functions. From Mihailidis (2007).
This ability to detect the path and respond to anyone of several correct actions, missing one part that can be done in a different order, makes COACH an excellent example of designing for failure from the beginning of the project. The actual intention was to have people that really needed it and were not particularly using this to train in proper handwashing but to support this IADL behavior while the dementia progressed, up to a not reliable point.
Figure 2.15: COACH plan and steps (left) and COACH modules (right). From Mihailidis (2007).
2.2.8CONCLUSION
Design for failure is not really much more than an extension of requirements gathering combined with appropriate system architecture modularity. The main message is that systems, when used by people without a strong ability to compensate for troublesome situations, should anticipate common problems and be ready with mitigating actions. These can range from automatically reconfiguring interfaces, responses, and plans for the user to bringing in human assistance of appropriate urgency. While the need is most obvious in the examples described above, some awareness of this approach should be brought to any computationally supported system.
2.3DISTRIBUTED COGNITION
Short Definition: Distributed cognition (DC) is the view that cognitive acts are (typically) based on both the internal assets of the person and the cultural structures and artifacts that support intelligence or cognition in a given human action.
Longer Description: Gregory Bateson remarked that memory is half in the head and half in the world (Bateson, 1972; Pea, 1993). We exist in a world full of examples of this distributed cognition: the shopping list that “remembers” for us, the speedometer on our car, the position of the toggle on our light switch (up for on), the very words that we are reading right now. Acts and knowledge are not constructed unilaterally. An interesting question is “Where is the boundary between my knowledge and the context that supports my knowledge?” (Salomon, 1993). Distributed cognition is an approach that views the cognitive act as a result of a system comprising an actor (memory, attention, executive function), the artifacts in the actor’s environment, the context, and possibly other people. These artifacts can be as concrete as a notebook and as ethereal as language. Viewing cognition in this fashion can enable analysis and prediction of cognitive behavior that has a basis beyond the solitary human mind.
Distributed cognition attempts to analyze problem-solving behavior with a unit of analysis that spans individuals, artifacts, and others (Hollan et al., 2001). The artifact can provide external support (i.e., amplification and transformation) for cognitive acts that may be beyond the ability of the unaided mind (e.g., cube roots). The artifact may be providing a true cognitive orthotic role, as in the MAPS prompting system, or may just extend sensory abilities as in the classic blind man’s stick in Gregory Bateson’s example:
But what about “me”? Suppose I am a blind man, and I use a stick. I go tap, tap, tap. Where do I start? Is my mental system bounded at the handle of the stick? Is it bounded by my skin? Does it start halfway up the stick? Does it start at the tip of the stick? (Bateson, 1972).
Distributed cognition is a cognitive science model, in the sense that it basically is concerned with the individual’s internal cognitive processes and the support/extension that artifacts can provide, in contrast with the sociological/ethnologist view that sees the user and artifact as part of a system of relationships (Suchman, 1987). One view of distributed cognition is that it is attempting to describe how distributed units are coordinated, how information is represented, stored, and transformed, and in turn how the representation of information transforms the task at hand (Pea, 1993). In this sense, the representation and computational mechanism that manipulates the representation become part of the cognitive process. But this transformation is not a static event.
In distributed cognition, one expects to find a system that can dynamically configure itself to bring subsystems into coordination to accomplish various functions (Hollan et al., 2001).
Therefore, the system is often dynamically interacting between the agents and objects, each modifying and mutually supporting the effort toward the system goal.
From the distributed cognition perspective, this process of “external cognition” (Carroll, 2003) consists of agents creating and using information in the world, rather than simply within their heads, to do three key things: (1) externalize information to reduce memory load (such as reminders); (2) simplify cognitive effort by “computational offloading” onto an external media; and (3) allow us to trace changes, for example over time and space, through annotation (Perry, 2003). The external cognitive artifacts or mediating artifacts that support this offloading increase memory capacity; in addition, the representation held in the artifact may “not simply augment, or amplify existing human capabilities. Rather, they transform the task into a different one” (Norman, 1993).
To analyze a task or environment from a distributed cognition perspective one needs to answer three questions (from Pea, 1993):
1.What is distributed (i.e., different components of the problem-solving process as well as the product)?
2.What constraints govern the dynamics of such distributions in different time scales (e.g., microgenesis, ontogenesis, cultural history, and phylogenesis)?
3.Through what reconfigurations of distributed cognition might the performance of an activity system improve over time?
The process of deconstructing the problem with this framework can be useful in creating a system that distributes knowledge in the world (Norman, 1990) by redistributing expert skills into a system. In this case, what is distributed are mnemonic and executive triggers and content, the constraints on the system are the timeliness and fit of the prompts to the current context, and the improvement of the performance over time maps to both error correction and scaffolding concerns.
By viewing the cognitive system as a system comprising an actor and mediating artifacts with the perspective of distributed cognition, one can look at goals and plans to attain these goals as being effected by a system comprising actors, singly or in groups (e.g., classes of actors), mediating artifacts, and their interactions. There is no particular bias in this perspective toward human actors; all elements are evaluated on the same plane (Nardi, 1996b). Distributed cognition looks for cognitive processes wherever they may occur, on the basis of the functional relationships of elements that participate together in the process. In distributed cognition, one expects to find a system that can dynamically configure itself to bring subsystems into coordination to accomplish various functions (Hollan et al., 2001). Distributed cognitions perspective is biased toward a goal, the cognition; in contrast to focusing on the process, as in activity theory.
2.3.1EXAMPLES OF DC IN OUR DAILY LIFE AND MARKS OF DC
The gaining of the view of distributed cognition (DC) is like owning a Saab—once you get one you see examples everywhere. For some this is a brilliant insight, for others it seems like just common sense. In this book I try and explicate these ideas as not only explanatory or theoretical principles, but to show that using them gives genuine guidance in the design process. In this case, by understanding and examining distributed cognition you will be led to look closer at the artifacts and the division of effort that DC implies. So as I was exposed to these ideas in L3D I would ask myself, “what difference will this make in my work?”, not “Wow, another fascinating insight” or ‘This is useful for classifying these things (systems, events, actions, etc.)”. If it does not make a difference in what you are doing, it probably does not belong here.
The classic example of DC is the transition from memorization of epic poems (i.e., The Iliad) and religious texts (i.e., the Quran, the Vedas) to writing and reading; here the this side of the DC is my ability to read and the that side are books. Maps are the same; before you had memorized trails you did not need to know how to read a map, however, now knowing how to read a map you can use that map to “memorize” any trail. Similarly street signs allow us to locate ourselves in new places without memorizing the organization of a city or village. Of course the premier example of DC is the computer.
There are several important attributes of DC that the designer needs to know (Table 2.2). The first is that distributed cognition changes the task. This is obvious from our book example, that before reading the task was using all sort of mnemonic tricks to memorize a few large texts, but with writing the task becomes how to decode the letters to make words (in my head7). Armed with the ability to read, a person can now walk into a library and effectively have memorized (because of her ability to read and the collection of books) thousands of texts. So, the gaining of the knowledge to do this sort of cognition also changes, to teaching reading. Another example of DC is the transition of grocery clerks from reading the prices on the grocery items to scanning them with barcodes. This changed the task from picking up the can (for instance) and turning it to see the price sticker and then pushing the keys on the register in that amount to passing the can over the scanner and listening for the feedback beep of success. As with many transitions to DC there are also unexpected side effects that can be quite beneficial—now the consumer (and the store) have a record of what was purchased because the database entry corresponding to the bar code also has descriptive data that can be used.
Along with changing the task, one side effect is deskilling the user of the previous ability that is being replaced. A personal example is our family’s use of GPS on family vacations. Previously we poured over maps to plan the route to be driven, and because of this had a pretty good map of the route, and often a rudimentary layout of the cities on the route, before we left. Now with accurate and reliable automobile-mounted GPS systems, it is not unusual to input the goal, have the system calculate the route, and leave only knowing the distance and time of arrival. On one trip from Bonn to a hotel outside of Paris the GPS indicated that we should take the next off-ramp and then we would be a half .5 km from the hotel. Unfortunately, the exit was under re-construction and as we passed it the GPS just recalculated the route and we found ourselves 20 min later at the same exit. We did not have a map of the area so, being the computer scientist that I am, I drove 30 min at right angles to the way we were going and then followed the directions; we arrived at the hotel from the other direction. Had I had a map I would have seen that the following exit would have allowed us to come back on side roads.
Another example of deskilling is the controversy about the use of calculators in primary schools. There was much concern that allowing them would deskill the children’s already gained arithmetical ability. In 2011, The National Council of Teachers of Mathematics decided to recommend allowing their use in the classroom,8 however (as with the slide rule), it was now very important for students to learn estimation skills to ensure that the calculator’s seeming accuracy was at least in the right “ballpark” of the right answer. This may be an example of an auxiliary skill that could be required to ensure that the DC system does not make things worse.
For many implementations, particularly in the domain of supports for people with cognitive disabilities, it is important to fit the external support to the particular, perhaps unique, existing abilities of the end-user. And those abilities must be sufficient to effectively utilize the support system. Figure 2.16 shows how the fit can be critical for adoption and use. In the upper region are those people who could use the system but their cognitive skills are high enough so that using the system would be more work than not and produce the same result—they don’t need it; in the lower region are those people that really need the system but don’t have the resources sufficient to use the DC artifact, so they need it but can’t use it. Those in the middle section can use it, and do need it. The tricky part is to make sure that the middle section is not too narrow so that the designer ends up basically developing bespoke software. The workaround here is personalization. Note also that the population in the middle may change over time, for instance people suffering from Alzheimer’s may decline in ability over time so rapidly that by the time they have mastered the DC artifact they can no longer benefit from the system.9
Figure 2.16: Bandwidth of need and ability.
2.3.2CANONICAL PAPER
Distributed cognition explicates that cognitive phenomena generally are best seen as distributed processes. The theory does not do away with the notion of individual cognition but emphasizes studying instances of socioculturally distributed cognition: how cognition is distributed across people and artifacts, and on how it depends on both internal and external representations. Hollan et al.’s paper “Distributed cognition: Toward a new foundation for human-computer interaction research,” while not the first discussion of this approach, is the one most pointed to (Hollan et al., 2001).
2.3.3AT EXAMPLES
Memorize task steps vs. the MAPS prompting system.
2.3.4EXTERNAL AND INTERNAL SCRIPTS
Scripting can be seen as an instance of distributed cognition. Cognitive scientists look at knowledge representation, particularly operational knowledge, in terms of scripts and frames (Schank and Abelson, 1977). In the MAPS system’s view of scripts, however, they are regarded as exterior and supportive rather than as internal structures. Traditional rehabilitative use of scripts is intended to lead to the memorization of the script steps, thus tying together these two perspectives. In the MAPS system, scripts are designed to be external supports when the internalization of the sequence of instructions is not possible. From this, one can discuss internal scripts as sequences of behavior that have been memorized and can be appropriately evoked to accomplish a desired task, and external scripts as the distributed cognition artifacts to simulate an internal script (Carmien et al., 2007). Figure 2.17 illustrates the external cueing of extant “atomic” behaviors by an artifact or human support. The top portion refers to a person with sufficient internal scripts to accomplish the whole task, the bottom two sections of the illustration refer to a person with all the “atomic” behaviors to accomplish the task but not having the internal scripts to tie them together in sequence and detail. The middle section demonstrates a case in which the external support is more than the person needs to accomplish the task, therefore possibly creating confusion or boredom (see Section 2.4). The bottom section shows the right level and fit of external support.
Figure 2.17: Internal and external scripts. From Carmien (2006b).
As an example of the process of internal scripts, consider when children become old enough to dress themselves that the various executive and mnemonic tasks involved with selecting, donning, and fastening clothing become part of an internal script that can be appropriately “run” when required. For some people with cognitive disabilities, the various internal scripts involved in the task of going to the store to buy milk may not be available; perhaps all the components but the travel component exist and are appropriately accessible. MAPS may provide an external script, in the form of prompts, to use the bus to the store, to accomplish this whole task.
Figure 2.17 demonstrates the relation of these internal and external scripts. Even people suffering from severe cognitive disability have functioning internal scripts for simple functions such as eating or walking. The MAPS system envisions its external scripts as bridging the gaps where the internal scripts do not support the complete task behavior.
2.3.5CONCLUSION
DC is the understructure of all AT/IA systems. Shifting the line that divides what is brought to a task by the end-user, who may not have the resources required, and providing computational support for the missing abilities, can level the playing field. Even if the system does not explicitly look like an example of DC, it may provide very useful insights into the problem of spending a bit of time identifying what parts of the problem are based on this concept.
2.4SCAFFOLDING
Short Definition: Scaffolding is the support given during the learning process which is tailored to the needs of the student with the intention of helping the student achieve his/her learning goals (Sawyer, 2006). Scaffolding can also be part of task support and dynamic scaffolding can be a personalization fit for AT systems.
Longer Description: From educational domain, notion is that the scaffolding provided support for the difficult parts, allowing the existing skills of the student to achieve success early in the process, and as the learning process continues the scaffolding can be slowly eliminated until the student has mastered the skill being studied and can accomplish them without help. An elaboration on this concept is extending or dynamic scaffolding, a recent example of which is smart cars automatically pulsing brakes when skidding.
Supporting human goals with computational artifacts must take into account the variability of human need and have a goal of fitting them, one measure of which is the concept of flow (Csikszentmihalyi, 1990) (Figure 2.18). The notion of optimal flow is that humans perform most comfortably in the zone where the task is not too difficult or too easy. When a task it beyond current ability there is a danger of abandonment; when it’s too easy then it may become boring (and also abandoned). The original work by Mihaly Csikszentmihalyi was with typically abled adults, but it can be extended to our population, and problems ameliorated by providing scaffolding stuctures to compensate for either condition. Dynamic on-the-fly or static history-based scaffolding can reflects flow optimization. As skills change then challenges and tasks change.
Figure 2.18: Flow. Based on Csikszentmihalyi (1990).
Therefore, one way of providing an optimized set of supporting tools is to provide scaffolding that fits the user’s model. Doing this employs changing the interface and supporting functionalities as the user model changes. This model reflects the user’s current skill set and needs, examples of which are the work done teaching programming, learning the scientific method, and guiding collaborative learning. A somewhat failed example of scaffolding was the actions of menu items in some versions of Microsoft Office where the items displayed in a given menu reflect the most recent set of items selected. This example is a particularly apt one of scaffolding changing inappropriately, as for many users it is disorienting and caused more problems than a non-changing menu set did.
In dynamic systems for daily life support the scaffolding can manifest as changing levels of detail in task support, based on use history (which is part of the user model) and on sensor data. For instance, if a set of prompts for a task was rapidly triggered through, faster than actually doing the steps, the system could contract those steps into a single step. Just as internalization of a subtask can trigger scaffold retraction, so too, for an aging population with progressively decreasing cognitive abilities, must scaffolding automatically extend to provide well-fitted support.
2.4.1CANONICAL PAPERS
The research on the subject of scaffolding comes primarily from educational domains, but significant work in other domains is presented in Part 3.
Wood, D., J.S. Bruner, and G. Ross (1976). “The role of tutoring in problem-solving.” (Wood et al., 1976).
Wood, D. and D. Middleton (1975). “A study of assisted problem-solving.” (Wood and Middleton, 1975).
2.4.2AT DESIGN EXAMPLES
In MAPS trials, in some cases, as parts of a script become memorized and become part of an internal repertory of sub-scripts (see Figure 2.17), the longer set was replaced in a subsequent script with a single cue for that internal memorized set of instructions. For instance, a script for making cookies had 40 steps and one trial participant initially started the script by looking at the prompt, sometimes repeating the verbal instructions, and then following the instructions (i.e., remove this ingredient from the refrigerator, take the baking pan from the cabinet, turn on the oven . . . ), when she came to the 10-step instruction sequence for actually mixing and placing the cookie dough she skipped through the steps with pauses between each prompting screen, and, at the end of the set, stopped and prepared the cookies on the baking pan. When she finished that set of steps she went back to the look, act, look loop until the end. The trial observer saw this and confirmed the observation by reviewing the log and saw that the set of steps were displayed with only seconds between them until the log came to the script step that went back to the 10 or so second span for looking and acting. In this case, the participant had made these cookies (the script had been made by a special education professional from a standard set of cooking scripts for people with cognitive disabilities) previously and had internalized the steps. In this case, the next time the script was given to the student the set of prompts for that sequence was replaced with two steps and she had no problem accomplishing the task.
In another task, an adult with intellectual disabilities was being guided in folding clothes from the dryer. Interestingly, this person held down a job in a local gym, was able to cook and take a bus to work independently, and had a personal bank account, but seemed to never be able to succeed in folding and storing his clothes from the dryer properly. The segmented steps of the task were sufficient for the talks with one exception—he was not able to master the folding and hanging of his pants properly, so his parents rewrote the script to expand the pants folding prompt into many steps and he was successful. As his use continued he became better at pants folding and the steps were contracted as in the example above. In MAPS, the expansion and contraction of the scripts scaffolding were done manually based on logs and observations, but it is not difficult to envision an automatic expansion and contraction based on logs when new scripts were loaded, and further to have the dynamic prompting happen in the process triggered by task performance in real time. One way to capture this might be to monitor the logs and react to cases where the end-user rocked back and forth over steps in the script and have the prompting system dynamically expand the sub-set of prompts to give more detailed guidance.
Figure 2.19: Components of Mohamed’s scaffolded game system. From Mohamad (2005)
Yehya Mohamed’s work on dynamically expanding and contracting a task level of difficulty based on biometric information is an example of dynamic scaffolding (Tebarth et al., 2000; Velasco et al., 2004). His adaptive interface is an excellent example of dynamically extending and contracting scaffolding (Mohamad, 2005) (see Figure 2.19). He evaluated existing adaptive systems and developed his by analyzing the arousal states of the user with non-invasive biofeedback sensors. The dissertation project provided a reliable means to recognize the arousal states of the user that complements research attempts in this area that interpreted arousal states by other external signals of the human body, like gestures, speech intonation, blood pressure, skin conductivity, etc. His system used a combination of sensors and a set of effective algorithms to infer the arousal state of the user (Figure 2.20). This information was used to modify the behavior and the interfaces of the application (Figure 2.21). In particular, a therapeutic system was developed where Interface Agents—which themselves can communicate through mimic, speech, and affective expressions with the user can modify their behavior according to the arousal status of the user. His system uses the skin conductivity sensor as the input device to measure the user’s arousal states. By monitoring this biometric information and estimating emotional levels he is able to make the elements in the pedagogical interface more or less difficult, thus by keeping the interface of system adapted to the optimum flow of the user he maximized the pedagogical efficacy of the system. He does this by having the system adapt the task difficulty to the maximum limit of the child’s abilities by analyzing continuously the child’s performance. When the system determines that the child’s perception of the difficulty of the task becomes too low, it adjusts the task to be more challenging; and when it is too hard, it makes the task harder. In this way it keeps the level of challenge optimal for the learning process.
Figure 2.20: Raw sensor data and classifier. From Mohamad (2005).
Figure 2.21: TAPA game interface. Fom Mohamad (2005).
In conclusion, dynamic expansion and contraction of guidance and increasing the difficulty and ease of effort presented to the user (isomorphs of the same problem), can be a good way of fitting the tool to the model of the end-user as internalized scripts and skills change.
2.4.3CONCLUSION
Scaffolding has tremendous promise for AT/IA, but with few exceptions (Mohamad, 2005) existing implementations require manual intervention. Truly dynamic scaffolding will require advancement in contextual sensoring and inference, which is only available non-trivially in the laboratory or in research projects. Dynamic versions will be based on estar user modeling (a user model that reflects, and keeps this as history, the current actions and context of the end-user) (see Section 4.4) and inference machine learning systems. Perhaps the best approach is to start with the low-hanging fruit (see Section 4.2) that computationally emulates manual scaffolding extension and retraction. An exciting time to be a researcher!
2.5SITUATED ACTION/COGNITION
Short Definition: Situated action “stresses the knowledge ability of actors and how they use common-sense practices/procedures to produce, analyze and make sense of one another’s actions and their local or situated circumstances” (Doerry, 1995). Simply, situated action is how people act in a situation (Suchman, 1987). Sometimes referred to as situated cognition, the main point here is that actions in performing a task are often based on the actual situation at hand as well as any supporting guidance that is previously prepared.
Longer Description: Lucy Suchman, in her canonical exposition of this concept, said the following (Suchman, 1987).
Rather than attempting to abstract action away from its circumstances and represent it as a rational plan, the approach is to study how people use their circumstances to achieve intelligent action. Rather than build a theory of action out of a theory of plans, the aim is to investigate how people produce and find evidence for plans in the course of situated action. More generally, rather than subsume the details of action under the study of plans, plans are subsumed by the larger problem of situated action.
The critical point here is that, in the real world, plans and guidance are almost always not literally sufficient to accomplish the task at hand. Closely examining any sufficiently complex set of actions, it becomes clear that the tacit knowledge of the world and domain needs to be added to the set of instructions to be successful. More importantly, mis-estimating the background knowledge held by the performer can sabotage the activity. We have all seen this in an attempt to guide elderly relatives to fix a computer problem that to a technologist is trivial, with a common result that the elder feels that there is “something wrong with them” in trying to act on the instructions. This particular experience can lead to abandonment of the use of the technology entirely.
Therefore, the first important point is to correctly identify the level of tacit and implicit background knowledge the targeted end-user may bring to the use of the system. Furthermore, what seems obvious to the designer may be mysterious to the end user not only because of a murky interface but because of a lack of a useful model of what is going on in the background of the task Note that I used the term “useful” rather than “accurate” in describing the model that the user projects on the technological system; it is not necessary that a user understand the actual details of the system, only that their understanding maps enough to the system for their purposes. An example is that of modern file systems in personal computers, in order to copy a file from one folder directly to another there is no need to understand the representation that a windowing OS provides nor FAT tables or Inodes.
Second, what looking at a task through the situated action lens leads to is thinking about error trapping and mitigation (see Section 2.2), and enumeration of possible points in the act that may need provided help in the right way at the right time. Tool tips, while not widely used in AT, are a good example of providing contextually appropriate help at the right time.
Finally the insights of situated action provide a theoretical basis for having an expectation of the inevitable failure of any set of instructions or prompts due to the intersection of environment, infrastructure, user, and artifacts. Armed with this insight, a designer of AT tools for intelligence augmentation is, to some extent, protected from making naive assumptions as the basis of their design process. More about this is discussed in Section 2.2.
The above discussion takes a specialized perspective on the much more broad insights that Suchman and other researchers present. However, the core insight of the inevitable inability of any set of guidance tools in solely supporting task completion has been a constant guide in whatever success I may have had in projects and systems.
2.5.1CANONICAL PAPER
Suchman, L.A. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. (Suchman, 1987)
2.5.2AT EXAMPLES
Situated action/cognition is not really a basis for design of AT/IA, as discussed above. However, the design process of the last (and first) kilometre guidance in ASSISTANT (see Section 4.2) was a result of putting together situated action with the process of design for failure (see Section 2.2). The designers knew that, for urban pedestrian navigation, GPS was not a reliable guide. Yet, the project was committed to provide support for this critical part of the elder’s journey. So taking those parts of urban GPS that were reliable (see Section 2.2 for details), the system recognized when there were breakdowns in the process (being careful to avoid type one errors) and provided support for end-users to (1) know that something was wrong and (2) give them the most reliable information available to remedy the error.
2.5.3CONCLUSION
The importance of the perspective of situated action is to make the designer humble. Suchman’s gift to AT/IA is that you will never be able to simply support task completion in a bubble no matter how sophisticated you get. And it works both ways—determining what the user is trying to do on the basis of observed behavior (basically sensor inputs) is never trivial. For me, reading her work was, in my humble opinion, something like Gödel’s theorem. There is not a closed system solution for many of our problems, at least those in the natural world (i.e., with the exception of systems that are self-contained, like Coach).
2.6SOCIO-TECHNICAL ENVIRONMENTS
Short Definition: Socio-technical environments (STE) or sociotechnical systems (STS) are a way of looking at technology, workplace practices, and end-users (as well as all stakeholders) as an integrated whole. By using the whole system or environment as a unit for analysis, the fitness and efficacy of the system is improved and the dignity and intelligence of the human elements can be supported.
Longer Description: The development of the socio-technical approach to technology system design is well documented (Coakes et al., 1999; Mumford, 2009). Looking at technology and its use in STEs came out of the Tavsitock group in post-war England. STEs are composed of two basic elements: the technical part and the social part. The technical subsystem comprises the devices, tools, and techniques needed to transform inputs into outputs in a way that enhances the economic performance of the organization. The social system comprises the employees (at all levels) and the knowledge, skills, attitudes, values and needs they bring to the work environment; additionally, the socio element encompasses the relations between the social roles, including authority and legal dimensions. By looking at the world of work as a whole system and placing equal emphasis on productivity and ethics, early advocates of the socio-technical method were able to elicit detail and dynamics of the elements of the system that previous (Taylor, 1967) oriented industrial engineers had missed. Furthermore, by utilizing a psychological perspective without class bias they were able analyze the system with a raw honesty that the earlier industrial researchers missed owing to their allegiance to using only an emic perspective (the story that the management tells about the institution’s operation) in contrast to the STEs more etic examination style (the actual behavior of the system as perceived by an outsider).
By explicitly concerning themselves with the human values, STE practitioners are included as part of evaluation of a work environment. Also by concerning themselves with the more operational aspects of work systems, such as proper location of boundaries where knowledge is shared and the need for minimal yet adequate design of the process advocates of the STE approach, argues that ethical concerns and efficiency were not mutually exclusive goals in work process design.
Implicit in this approach is a belief in the importance of humanistic principles, and that one of the main tasks of the system designer is to enhance the quality of working life and the job satisfaction of the employee. In turn, they felt that the achievement of these objectives will enhance productivity and yield added value to the organization. At the root, the contribution of socio-technical design was to place the rights and needs of the employee as high a priority as those of the non-human parts of the system. What resulted were diagrams like Figure 3.3, where the employees are equal parts of the study rather than instruments of connection or as another machine to transform raw material.
Classical dimensions of concern to STE analysis are job enrichment, job enlargement and rotation, motivation, process improvement, task analysis, and work design (PTG Global, 2009). Frey (2009) posits four aspects of STEs (or in his case he calls them socio-technical-systems (STS): (1) STE components are interrelated and interact so that a change in one component often produces changes in the other components and in the system as a whole; (2) STE have different components (i.e., hardware, surroundings, software, groups and roles), which interact with one another; (3) socio-technical systems embody values, and this embodiment can be identified in specific components in the system; and (4) STEs change, and this change, referred to as a trajectory, reveals internal relationships and must be accommodated and perhaps canalized for the system to succeed in all dimensions of measurement (i.e., efficiency and satisfaction). The elements of his analysis are: hardware, software, physical surroundings, people (groups and roles), procedures, laws, and data and data structures
The developers of “classical” socio-technical theory created a number of frameworks and tools for analyzing and designing socio-technical systems. Enid Mumford, one of the seminal workers in socio-technical analysis, developed an approach to system analysis and design called the ETHICS method (Mumford, 2009). ETHICS consists of a series of steps that progressively involve the various stakeholders in the design and adoption of a new system.
Figure 2.22: Elements of a socio-technical system.
Of particular use to analysis of large and seemingly amorphous systems is the STE Network Model approach. In this approach the researcher uses graph modeling methodology to map out elements and known relations between them, and in the process may find existing tacit connections and missing, but necessary for optimal performance, functional elements, and connections (Scott, 2000).
Enid Mumford (2003), in her book Redesigning Human Systems, presents several personal cases of STE analysis and STE-supported design. Following the guideline in the STE approach of understanding current work practices before attempting to re-engineer the system, Mumford closely observed the process getting to know the participants, environment, and procedures. Her techniques, in one case of getting a job of a canteen worker in a loading dock area, while not formally using ethnographic protocols, approximated classical ethnographic work. Ethnographic participant observation (Bernard, 2000, 2002) becomes highly relevant in understanding both the present process and the iterative design and implementation of new processes. Mumford’s approach to respecting and learning from the current practitioners of a work practice and environment is echoed in a quote from Mao Tse Tung that she presents in her book (Mumford, 2003):
Go to the practical people and learn from them: then synthesise their experience into principles and theories; and then return to the practical people and call upon them to put these principles and methods into practice so as to solve their problems and achieve freedom and happiness.
Working on AT systems with an STE perspective is an instance of a wicked problem (see Section 2.8), in that the final form of the solution is typically not specifiable at the time of the initial design. This is because complex computer systems that are designed for interaction with people (in contrast to, say, rocket guidance software) are only properly studied in place. Since the system, its context, and the human users form a socio-technical environment, this is not ready to be studied until at least a rudimentary environment is in existence. Because the STE is comprised of the system itself and the members (actors/stakeholders) of the system and the interactions and the changes in work practices, the introduction of the computational artifacts induces an analysis of the problem that is complex and time consuming. Compounding the problem is that they often have multiple stakeholders (some of which are very tacit, e.g., the legal system). An example of the consequences of not studying the whole range of stakeholders and the changes that a new system induce in work practices was the result of a complex, very user-friendly, system. It was carefully crafted with user input to work with minimal effort in a system that enabled special education teachers to share best practices with AT and approaches in the classroom. Upon test installation, the school administrators demanded the removal of features that made the system unusable by the very end-users, the teachers, who needed these fucntions the most. This was done by the school administration who, under HIPAA privacy legal constraints, were advised by legal staff to remove any identifying information about the students, the very information (even when anonymized) that the system was designed to share (dePaula, 2004). Interjection of computational artifacts changes the environment’s work practices, which may affect many more shareholders than expected.
Good AT design is then best approached from the STE perspective. The creation of AT for people with cognitive disabilities is particularly an STE issue due to the complexities of relationship and invasiveness of the technology. Following on this is the question of how best the STE approach can be formalized in AT design. Frameworks, such as the ETHICS method (Mumford, 2009) and the decomposition into facets as presented by Frey (2009), would be good places to start. Most useful is the emphasis on ethnographic study forming the basis for understanding the potential and requirements of the situation. An explicit drawing out of stakeholders’ roles and investments in the whole system is illustrated in several ethnographic studies in Mumford’s works. Illustrating a similar approach is MAPS inclusion of the caregiver’s role and interface design, which naturally flows out of approaching the problem from an STE perspective, a concern that is often missing in other high-functioning but low-adoption systems.
Developing an explicit checklist set of heuristics to incorporate STE perspective in AT design is another approach that may roll back the tremendous problem of abandonment. Finally, STE’s systemic approach of acknowledging the dynamic interaction between user, artifact, environment, and tasks is critical for good AT design.
2.6.1CANONICAL PAPER
Mumford, E. (1987). Sociotechnical systems design: Evolving theory and practice, in computers and democracy, (Mumford, 1987)
2.6.2AT EXAMPLES
AT Design and Use Example: MAPS
Ethnographic Design: Because MAPS was intended to be used by a population with abilities and needs very different from the designer and programmer on the project, the initial steps of the design process were to gain understanding of the end-users and their tasks. Various ethnographic approaches were used to support this. Initially, spent time was spent with professional caregivers and teachers who specialized in working with young adults with cognitive disabilities. The object was to understand the young adult with cognitive disabilities and their caregivers, both in day-to-day life and in contexts where they might use MAPS. By doing ethnographic participant observation (LeCompte and Schensul, 1999) the MAPS designers gained an understanding of the young adult with cognitive disabilities abilities and preferences that informed the prompter design. The observation process also provided example tasks that MAPS could support, as well as provide details about the environment the tasks would be accomplished in. By observing and analyzing the observations of the interactions between the end users and caregivers in daily life the relationship dynamics were exposed in a way the simply gathering formal requirements for a software project could not do (LeCompte and Schensul, 1999).
In practice, what this meant was spending about 20 contact hours with each dyad (i.e., pairs of end-users and caregivers; see Section 3.1), 10–15 hours with just the young adult with cognitive disabilities, a lesser amount with just the caregiver, and the balance with the caregiver and the young adult with cognitive disabilities. The goal was to learn how each of them interacts with the world and with each other. One way this paid off was the discovery that sometimes the verbal prompts should not be recorded by the caregiver, especially so in the dyad consisting of a mother and 16-year-old daughter who had typical adolescent power issues; this highlighted the fact that although these end-users were developmentally disabled they would often go through the same changes in relationships within their family as their non-developmentally disabled siblings. Spending time with the caregiver gave the designer a better idea of the level of PC expertise that would be available and also helped to give the caregiver realistic expectations for the MAPS system. Most importantly, participant observation gave a window into how task accomplishment was currently being supported and provided a reference against which MAPS could be measured.
Having produced a prototype, the second part of the MAPS project was to study the initial use, iterative design changes, and final adoption of the system in a real world context. With each dyad there was a process of generation and use increasingly challenging set of scripts. The content and environment of the scripts was typically from simplest to most complex:
•controlled environment (e.g., a housekeeping chore), in which neither the task nor the environment is dynamic and the environment is familiar;
•less controlled script (e.g., cooking), in which the task doesn’t change and the environment is dynamic but familiar; and
•least controlled script (e.g., shopping), in which the task and the environment are unfamiliar and the environment changes.
The first script was also used to familiarize the caregiver with the script design and composition process and the person with cognitive disabilities with the use of the prompter, its controls, and how to follow a script. The final two were designed to simulate, in a real context, increasingly autonomous use of the system. The three-script process allowed changes to be introduced into the system to make it better fit the needs of the task and user. One example was the addition of multiple scripts that could be chosen by the young adult with cognitive disabilities depending on the state of the tasks. Another dyad changed the script between iterations several times to accommodate learned sub-tasks that were initially cued in detailed prompts and now could be called up with a single prompt.
Intersection between AT & STE
The most obvious connection between the STE approach and successful AT design is the equal focus on the mechanical/technical aspects and the human aspects of the whole system. This section will map some of the more specific topics and schemas from the STE literature and AT design.
Mapping the specific parts of the ETHICS method (Mumford, 2009) to the process of designing MAPS shows a congruence that might make for a good framework for AT design in general, as well as describing the process of choosing and adopting a new piece of AT. Starting with ETHICS first step (why change?), which corresponds to the AT professional determining not whether a change is needed, but what form the intervention will take, the disability itself is the motivator for wanting to change. ETHICS next step (system boundaries) corresponds to the caregiver and AT expert determining what the person with cognitive disabilities might be able to do with technological assistance, and what they should not attempt to do. ETHICS step 3 (description of existing system), which in the STE process describes the procedure of learning the current system (often from the inside out), corresponds to the AT designer studying the work practices and day-to-day details of the person with cognitive disabilities and the dyad’s life, as well as the occupational therapist or special education professional detailing the end-users existing problem solving personal adaptations. All too often high-functioning AT in the form of a cognitive prosthetic fails in functionality and/or adoption because the designers naively felt that they understood the needs of the end users, an insight best gained through personal involvement with the person with cognitive disabilities. Similarly, both the AT designer and the OT or AT expert that recommends and guides the end-user in obtaining AT need to perform the definition of key tasks which includes both teaching task segmentation and the use of a prompter.
The future step analysis corresponds to the AT designer explicitly making the AT re-configurable to support future needs, in some cases providing less support to the end-user—as in the case of acquisition of skills—and in other cases providing more support for greater levels of disability. This step also corresponds to the notion of an STE system accommodating system change, the trajectory in STS terms (see Socio-Technical Systems in Professional Decision Making Module (Frey, 2009)). This lack of support for co-evolution causes much of the abandonment (Phillips and Zhao, 1993) of AT tools. Caregivers, who have the most intimate knowledge of the client, need to become the “programmer/end-user developer” of the application for that person by creating the needed scripts. Similarly, the other steps in ETHICS can be mapped to stages in AT design or adoption.
Successful AT design and STE analysis are both based on worker (end-user) participation and management (caregiver and AT professionals) support. It is only with a base of both of these that the use of, and appropriate AT aid, can continue past initial use. Suchman (1987) describes the actual trajectory of attaining a goal as being heavily dependent on situated action—on accommodating changes in the context and requirements to attain a given goal. Therefore, both STE systems and successful AT must be able to accommodate change in use, what Frey and others (Geels and Kemp, 2007; Frey, 2009) refer to as trajectory. For an STE to continue to be relevant and appropriate it must adapt to coordinated sets of changes within the socio-technical system. This coordinated series of changes in an STS is called a trajectory. Trajectory must fit situational needs if STNs can support/withstand change. It is also a quality of successful AT for people with cognitive disabilities such that it must reflect change as the system does not need to fit into “artificially” created (and thus stable) work practices but into the world as it is, which is constantly changing. In the case of MAPS, the ability of the caregiver to edit the MAPS scripts, enlarging them if there is greater need, compressing them in the case where sub-tasks become learned.
Decomposing AT into STE Grid
Frey (2009) uses a schema to analyze socio-technical systems that divides the system into seven components. These parts—hardware, software, physical surroundings, people (groups or roles), procedures, laws, data (and data structures)—give a good starting point for comparing different STEs and also for clarification of the interaction between the technical and social dimensions of an STE. Following is the decomposition of the MAPS system.
•Hardware: MAPS uses a PC for its MAPS-DE script-designing tool, feeding the script composition are recorded voice prompts and images collected by a digital recorder and camera, respectively. MAPS scripts are played on by the MAPS-PR on a PDA or smart phone that runs one of the mobile versions of the windows operating system.
•Software: The MAPS system software consists of the MAPS-PR script player and the MAPS-DE script editor. Optionally, in addition to these (and in support of them) are an image editor for the pictures illustrating the prompt, and an audio editor for the verbal prompts. Behind these are the Windows desktop and small device operating systems. Because one of the functions not disabled in the PDA was the MP3 player (to motivate retention of the PDA), the MP3 player application was also sometimes used. Additionally, some caregivers used a text editor (like MS Word) for preliminary script design. The scripts themselves were stored in a Sybase database on the PC and PDA, as well in as in a MAPS script template server that held pre-outlined typical scripts accessible in the Internet.
•Physical surroundings: MAPS was used in two kinds of environments. The MAPS-RP was used wherever the end-user was performing tasks with the aid of the scripts prompting. In the initial trials of MAPS these ranged from the end-users home to a school to employment (i.e., in a used clothing store). As well as the MAPS-PR being used in these spaces, the caregiver would photograph prompt visuals in them for script creation. The prompts were most often recoded in the home, or in the case of the job coach, the office of the caregiver. In the case where the MAPS-PR PDA was being used as a MP3 player, the location varied with the path of the end-user through the day.
•People: The list of people includes not just individuals (roles) but also groups of people (groups). These include the designers of the MAPS system and the end-user co-designers. Central to the socio-technical system are the end-user (also referred to as the MAPS-PR-users, a person with cognitive disabilities, and the client) and the day-to-day caregiver, who may be a family member or a professional caregiver, paid for by insurance, the family, or the state. Influencing the system at a remove are AT experts, special-education experts and teachers (in the case of a young adult with cognitive disabilities), insurance personnel, and state funding staff. At a further remove, but still very much affecting the system, are school administrators and employers. Finally, intimate influencers of the system are the end-user’s immediate family and friends, as well as their peer groups (either in school or employment).
•Procedures: There are several kinds of procedures in MAPS, in setting up the system and in using it to guide task performance. Included in the tasks required to setup the system are task segmentation (i.e., breaking the task down into sections that are of the correct cognitive level), task rehearsal (i.e., performing the task yourself to ensure no tacit steps are left out), and script building. The construction of scripts (after the outlining has been done) requires collecting photographs of the task with a digital camera and recoding the verbal prompts with a computer and mike. The caregiver must master the art of using the MAPS-DE using the provided tutorial. Script assembly requires using the MAPS-DE editor and the operating system to identify and insert script steps into the script database.
Next, the caregiver has to transfer the script to the MAPS-PR from the caregiver’s PC. The end-user has to initially learn how to use the prompter by working with the caregiver and perhaps the MAPS design-support personnel. The young adult with cognitive disabilities is then ready to use the script on the MAPS-PR to accomplish the task, which is embedded in the larger set of ADL and IADL tasks that she can do without external support.
Finally, the caregiver has to review the script log to see if the script needs to have certain steps collapsed into a trigger step (collapsing scaffolding) or expanded into several additional steps because the end-user found performing them too difficult (expanding scaffolding).
Laws, statutes, and regulations: The MAPS system is not impacted by laws and regulations except inasmuch as it’s purchase can aided by state funding.
•Data and data structures: MAPS stores external wave files (for the recording prompts) and jpg files (for the prompt images) in the caregiver’s PC. Completed scripts are stored in a Sybase database on the caregiver’s PC and scripts on the MAPS-PR are stored on a mobile lightweight version of the Sybase database. Additionally, a Sybase database of template scripts is stored on a networked server, accessible through the Internet. Design documents used in creating scripts (i.e., task segmentation notes) may be stored in text documents. MAPS-PR stores a log produced of the use of a given script in a text file for later analysis.
Finally, in this comparison between AT design and adoption and traditional STE analysis (Table 2.3) there is in both a concern of studying work practices or in the case of AT, the activities of ADL and IADL, as they are. Therefore both disciplines use varying forms of ethnography and base their theoretical analysis on both foreground (worker/person with cognitive disabilities) and background (environments, rules, and technologies).
STE and AT design particularly impacts “workplace practices” in the sense that introduction of new AT technology may change a “stable” family dynamic in very unanticipated ways. The in-house special education consultant we had in our lab at L3D told us her experiences (Kintsch, 2002) with the introduction of AAC (assistive and augmentative communication devices) into a family that had worked for years to create a family dynamic that worked for them based on a member being the communication helper for the child with a fairly severe communication disability that required the family member to be the one that was the only way to support communication for the child with the disability within the family and with the outside world. This stable dynamic was threatened by the introduction of the support AT and she had seen several situations where the caregiver subtly sabotaged the new configuration, presumably threatened by the new and unfamiliar family dynamic. By being aware of all the stakeholders and anticipating the sorts of unintended changes AT introduction can cause, a painful cause for abandonment can be circumvented (Kintsch and dePaula, 2002). Similar to this situation is the use of AT in school and transitioning into the use of the same AT in the home environment.
2.6.3CONCLUSION
Every introduction of technology is a socio-technical event. But every AT system does not necessarily benefit (e.g., in a cost/benefit way) from basing the design with STE. It’s important to evaluate how much the work practices will change, how many “shadow” stakeholders are involved, and the intent of the AT system. For the systems that this book describes the STE approach is recommended, but this needs to be a decision not an assumption. Doing STE-based research properly is not trivial; it can be quite time, and resource, consuming.
2.7UNIVERSE OF ONE
Short Definition: People who could most benefit from high-functioning AT are also often afflicted with co-morbidity, resulting in users that are not effectively generalizable as a design target.
Longer Description: Abandonment Based on the “Universe of One.” People with cognitive disabilities represent a “universe of one” problem (Fischer, 2009): a solution for one person will rarely work for another. Accessing and addressing these unexpected variations in skills and needs, particularly with respect to creating task support, requires an intimate knowledge of the client that only caregivers can provide (Cole, 2006). The consequence of the high rate of device abandonment (Phillips and Zhao, 1993) is that the very population that could benefit most from technology pays for expensive devices that end up in the back of closets after a short time.
AT for cognitive disabilities (and to a lesser degree all AT) presents unique design challenges stemming from being in the intersection of AT and cognitive science. One aspect of this is that due to the distance between the experience of the designer and the end-user’s systems, which are often inappropriate or ineffective in real context of use. In attempting to understand the needs of the user and the task to be performed, the system designer can take one of two naive approaches. One could label these two problems “I’ve got a cousin” and “I’ve got a theory” (Section 5.2).
Disabilities are often complex mixtures of separate needs and lacks. Medical papers often refer to this as co-morbidity (Jakovljević and Ostojić, 2013), a useful word as well as sounding like what it describes. Especially in the domains of people with cognitive disabilities these multipliers of barriers make design of support systems and extensions of existing abilities very difficult. One size never (or very rarely) fits all. So the CLever group discussed what we were doing was designing for a universe of one, in contrast to making a single system that would properly fit many.
Reduced intellectual ability is often combined with sensory impairments such as visual and hearing impairments; with motoric impairments making input to the device require special adaptations; and psychological/developmental impairments affecting comprehension of the system and its outputs as well as problems with losing or not properly caring for devices and becoming impatient with incomprehensible or delayed outputs.
Compounding this problem, it is typical for novices to this kind of population to rely on diagnosis as a basis for design, starting with problem specification. This is exactly what we in CLever did, initially, until we had the good fortune (and excellent planning by Gerhard Fischer) to hire a formally trained special education professional with many years’ experience. What resulted was a focus on functional disability, which allowed more effective persona building and problem specification. One of advantages to focusing on functional needs, in contrast to diagnosis, is the curb cut10 effect that produces benefits for unintended populations (e.g., curb cuts for wheelchairs and baby carriages), as well as benefiting those with temporary conditions that produced disabilities. Examples of this are that driving a car lowers the cognitive and motoric abilities of the end-user, and being confined to a wheelchair while recuperating is not fundamentally different than using one on a day-to-day basis.
Universe of one dilemma is one end of an axis and the other end is “truly one size fits all.” All (interface) design problems fall on this axis. By designing too tightly to one person, others cannot benefit; by only designing generic solutions, often those who do not fall into the typical range are excluded. One solution to this dilemma is deep personalization (see Section 4.4). There is validity to the of repeated motto “design for all just good design in the first place,” even in some cases for the problems discussed here, but unfortunately the seven-point slogans of the Design for All movement (Center for Universal Design, 2011) really don’t cover many of the multi-layered and interactive needs of intelligence augmentation.
2.7.1CANONICAL PAPERS
There really is no canonical paper on this topic, but this is a good medical overview:
Jakovljević, M. and L. Ostojić (2013). “Comorbidity and multimorbidity in medicine today: challenges and opportunities for bringing separated branches of medicine closer to each other.” (Jakovljević and Ostojić, 2013).
Also, Cole’s position paper in a CHI workshop is very useful:
Cole, E. “Patient-centered design as a research strategy for cognitive prosthetics: Lessons learned from working with patients and clinicians for 2 decades.” (Cole, 2006).
2.7.2AT EXAMPLE
Deep personalization in ASSISTANT. The project assumed, from the initial requirements stage, that the target population would be quite varied. Notwithstanding the “bandwidth of need and ability,” discussed in the Section 2.3, making the target group much smaller, within this group there was a fairly wide variation in what the focus groups expressed as needed features. One elderly woman said that she would be anxious having to hold the PND equipped smartphone out in public, fearing theft. So ASSISTANT provided a voice only guidance feature, allowing them to use a mono ear bud or a Bluetooth earpiece. Some of the elders had a slight tremor in touching the screen, so the personalization tab provided an option to specify ignoring multiple rapid taps, while allowing more agile users to interact more rapidly. Some had vision problems, so there was an option of inverting the screen; some wanted to contact emergency help immediately, some wanted to have the system contact their relatives, so this was configurable by them in the route editor. Many more options were possible, or none, if the user chose.
In doing research for the optimal representation on the screen for image/audio prompts for the MAPS system and in talking with professionals in this field (special education and rehabilitation experts) I ran into comments that the more challenged the person is the more difficulty they have in using icons and generic symbols of objects and actions. Not finding satisfactory answers in a broad literature review, spanning studies on AT, rehabilitation and cognitive psychology, our team did an experiment (discussed in detail in the Section 3.2). As a result, the MAPS system used actual images of the prompts (e.g., vegetables bin at the real store where the user was instructed to pick a head of lettuce, the familiar cashier at the store the end-user and her mother went to regularly). To do this, a much more complex script editor was provided to the caregiver to assemble scripts to use photos that were for her alone. The resultant script was then tailored to the young lady and her mom. If you look at the resultant script as an application, then the problem of universe of one was accommodated. Interestingly enough, the mother in this case had me record all the prompts to accommodate her teenage daughter’s resistance to being told what to do, something that was commonly unexpected on my part. Autistic teenagers, even not-so-high functioning ones, are still teenagers with all the emotions of any teenager.
2.7.3CONCLUSION
Design for one is not as much particularly a design framework as a cautionary perspective prescription. Certainly deep personalization is part of the answer, as is a clear design problem statement. Bounding the problem to specific functional disabilities helps a lot. Best is interaction and study of the end-user and stakeholder population. These are sometimes quite different activities; when doing ethnographic research for the MAPS project, about ¼ of the time I was with the young adults and caregivers was spent just hanging out rather than focusing on the task at hand, producing unexpected insights and a motivational “push” that greatly improved the system.
2.8WICKED PROBLEMS
Short Definition: A wicked problem is a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often initially difficult to recognize.
Longer Description: Ill-defined design and planning problems can be labeled “wicked” (Rittel and Webber, 1984; Simon, 1984) in contrast against the relatively “tame” problems of mathematics, chess, or puzzle solving. Wicked problems have incomplete, contradictory, and changing requirements, and solutions to them are often difficult to recognize as such due to the complex interdependencies. Typically, wicked problems have the following characteristics.
•The problem is not understood until after formulation of a solution.
•Stakeholders have radically different worldviews and different frames for understanding the problem.
•Constraints and resources to solve the problem change over time.
•The problem is never solved.
•Solutions to wicked problems are typically better, worse, or good enough (satisficing).
In the domain of developing AT as IA the wicked problem space requires the two primary symmetrical holders of knowledge, the client and caregiver, to both contribute to the solution—the caregivers contributing the finished scripts and the client contributing the existing internal scripts (see Section 4.1). Wicked problems are not statically solved; rather, the on-going solution is a process. An example of this is the plotting of a bus route through a residential neighborhood, where the trade-offs include local passengers, property owners, traffic managers, and urban planners. Additionally, the route as planned may be good for only several years or less.
Wicked problems are most often “solved” (here the notion of satisficing emerges) through group efforts. Satisficing solutions are not true or false, but better, worse, or good enough. Task support through computationally based multimedia prompting is such a problem. The starting-off point for a designed solution is to do a stakeholder analysis of the problem space (Overseas Development Administration, 1995). Several types of stakeholders are involved in prompted task-support:
1.End-users: those with cognitive disabilities who will use the system to support doing the task;
2.Key stakeholders: those who can significantly influence or who are key to the success of the activity (in this case caregivers and clients);
3.Primary stakeholders: people who are directly affected by the solution; in this case, parents, employers, group home staff; and
4.Secondary stakeholders: all others with an interest in the activity; in this case, members of state and federal organizations who concern themselves with AT, insurance companies, and HIPAA bureaucrats.
Having identified the stakeholders, the designer then can proceed with a better assurance that the system can be adopted as a solution that must satisfice each class of stakeholder. Additionally, this analysis identifies the (sometimes orthogonal) requirements and identifies the symmetry of ignorance members of the problem space.
Some of the characteristics of wicked problems are:
•incomplete,
•contradictory, and
•have dynamically changing requirements.
Understanding this provides both solution constraints and also, perhaps more importantly, a tool for identifying the presence of this type of problem. Typically, the actual parameters of a wicked problem are not understood until after formulation of an initial solution, thus prototypes (tested by stakeholders) and models become important tools in the process. Because stakeholders have radically different worldviews and different frames for understanding the problem, stakeholder analysis and broad participation of different stakeholders in requirements gathering, like focus groups, becomes quite important. Because constraints and resources to solve the problem change over time, the solution must be adaptable, sometimes radically so. A well-functioning example of a satisficing solution is the U.S. Constitution and its amendments.
The problem is never optimally solved but solutions to wicked problems are typically better, worse, or good enough, hence the motion of a satisficing solution, one that disappoints all equally (Simon, 1996).
Does this sound familiar? What to do? Explicitly involve all stakeholders; underdesign solutions; allow for the seed, evolve, reseed process (Fischer, 1998); design for change (see Section 4.3)? Just knowing that the problem is wicked helps.
2.8.1CANONICAL PAPER
Rittel, H. and M.M. Webber (1984). Planning problems are wicked problems. (Rittel and Webber, 1984)
2.8.2AT EXAMPLES
Interestingly, it is difficult to find examples in AT/IA that are not successful due to the tacit implementation of the wicked problem design approach. In AT, the notion of iterative design is standard, however the need to involve all stakeholders to produce a satisficing solution is not always obvious from the beginning. An example of this is the rollout failure of DePaula’s innovative special educators collaboration system (DePaula, 2002) due to legal blocks (see discussion in Section 2.6)
2.8.3CONCLUSION
Like distributed cognition (see Section 2.3), wicked problems are everywhere. It may be, in the domain of AT/ IA, that it is better to start assuming that the problem is wicked. Contemporary software engineering, in the form of the agile approach (Cockburn 2001) does, to some degree, acknowledge this truth in the conventional application world.
5Interestingly, on the top of the machine is a sign alerting shoppers that there is a video camera on. This is an excellent example of what happens when you do not do due-diligence in shareholder analysis. I am guessing that the application was complete and tested well enough to deploy when a legal expert from another part of the company forced them to add this laminated cardboard sign to comply with EU privacy laws.
6This is actually a very relevant point. One of the common mistakes that beginning developers of AT make is to start with diagnosis as the ordering principle in generating requirements and eventual design. This is not as effective or generalizable as the more common functional approach, i.e., designing to compensate for a missing facility rather than for people with a disease that may cause that missing ability.
7Interestingly, enough reading silently was not a skill that many had until the early Middle Ages, at least in the west. Augustine’s confessions mentions that he was surprised when he came upon Saint Ambrose reading silently one afternoon in 384 AD, Manguel (1996).
8http://www.nctm.org/Standards-and-Positions/Position-Statements/Calculator-Use-in-Elementary-Grades/
9Fortunately the typical course of Alzheimer’s tends to shelve out for several years (Reisberg et al., 1996) when both the skills and the need for a system like MAPS to have some use.
10The curb cut effect: technology intended to benefit people with disabilities sometimes wind up benefiting everyone. Curb cuts, which are intended for wheelchair users to be able to get onto sidewalks, help bicyclists, parents with strollers, delivery people, and a dozen other nondisabled groups. Similarly, closed captioning, which was originally meant to benefit deaf people, helps people who have trouble with auditory information processing, people who like talking during films, and people trying to watch TV in noisy bars.