CHAPTER4
Technique
This chapter describes a set of approaches that help make an AT/IA system work with a tractable design to implement.
•Plans and actions
•Low-hanging fruit
•MetaDesign
•Personalization
•Symmetry of ignorance
4.1PLANS AND ACTIONS
Short Definition: A script is a description of how a goal is attained, and scripts can be broken down into sub-scripts and eventually into “atomic” steps. Attempting to follow a script can be unsuccessful in many ways, which are called errors.
Longer Description: Early in the development of cognitive science’s discussion of human behavior in describing day-to-day behavior, a model emerged of scripts, goals, and actions. Basically this breaks down a set of behaviors into steps that, combined, are called scripts. Scripts are aimed at a goal, and can be broken down into sets of sub-scripts with intermediate goals until “atomic” actions are reached. These ideas came out of both studies in education (Weinberger et al., 2005) and early AI and cognitive psychology (Schank and Abelson, 1975). First some definitions:
Goals are the desired end point of the script. Important to defining and discussing goals is that they be measurable, that there is an endpoint. Also important, especially from an AI perspective is the use of goals as part of a utility function in deciding the best action (Russell and Norvig, 2009).
Plans are the description of how to get from the current state to the goal state. As with goals, AI literature has much to say about this topic. One of the critical issues with plans is how they are represented. The plan could be an algorithm, a set of cards, or computer-supported prompts. Plans are typically comprised of steps of appropriate granularity.
Actions are the constituent part of plans that lead to the goal. Actions can be sub plans, can be cued externally, and can be an inventory of internal abilities.
Errors are typically included in the canonical literature is the notion of a breakdown in the process. This can be referred to as an error. Other sections of this book discuss this in some detail (see Section 2.2).
Schank and Abelson (1977) described human behavior in terms of goals and scripts. A script is a description of how a goal is attained. These scripts are typically internal to the person acting on them. However they can be looked at another way. In training and rehabilitation of a person with cognitive disabilities, professionals often train people with cognitive disabilities to accomplish ADLs by splitting ADL tasks into discrete steps that the client can easily accomplish and then teaching the task as a set of these kinds of tasks sequentially linked together (Saskatchewan Learning—Special Education Unit, 2003). The client then memorizes this sequence of steps and then has the ability to do a task that she could not do previous to the training. Such a set of steps is called a script and the individual steps are called prompts. By moving this process out of internal representation into a computationally based system the learning is moved from that to learning the task scripts one by one to learning how to use the script player (implemented on a PDA); such a transfer of necessary skills is part of the process of distributing of cognition (Solomon, 1993; Hollan et al., 2001). Distributed cognition is also in play in the change from memorizing long texts (i.e., the Qur’an, The Iliad) to acquiring the ability to read these texts in written form; thus acquiring the distributed cognition skill of reading effectively enables a person to “memorize” as many texts as there are books. In this case the steps are presented, as prompts constituted from an image and a verbal instruction. Tailored exactly to each person with cognitive disabilities, some tasks are represented with many more prompts (corresponding to the clients lack of internal subtask knowledge) or less prompting steps (corresponding to a client having more internal sub tasks) (Carmien, 2006b). So the client’s limited internal scripts are complemented by powerful external scripts (Carmien, 2006b).
One advantage to looking at task support in this way is that it becomes easier to create guides and tutorials for thecreation of scripts using task segmentation. Task segmentation is the process, taken from the domain of vocational rehabilitation, of breaking a specific task, with granularity of prompts derived from intimate knowledge of the intended end-user’s needs and abilities, into a set of steps, each of which can be done as an “atomic” task. Here, atomic means that the sub task is itself an internal script that the end-user can do without support. For some this is larger, like get the ingredients for the sandwich from the refrigerator, for others smaller, like go to the refrigerator, open the door, remove the mayonnaise.
The practical implementation of the notion of plans using scripts for defining a task and supporting the successful accomplishment of the task comes to this domain from the world of rehabilitation. There is a plethora of practical best practices and already prepared templates for tasks for people with cognitive disabilities. These templates become very useful for showing how to segment a task as well as a stepping off point for creating individualized versions. The LRE for LIFE Project (LRE for LIFE Project, 2001) provided a set of scripts to be used by professionals in teaching activities to people who cannot currently do them, ranging from simple to more complex (see Figures 4.1 and 4.2)
Figure 4.1: Script to teach undressing. From LRE for LIFE Project (2001).
Figure 4.2: Script to teach cooking breakfast. From LRE for LIFE Project (2001).
With these as a guide and some simple short exercises the caregivers in the MAPS project became proficient in segmenting tasks for their end-user participant.
In many practical AI problems, a hard problem can be made tractable by constraining some of the parameters of the problem. Even after making the problem less “wild” this can still be a daunting task. As an example, the solution of a constrained problem, supporting the washing of hands in an instrumented bathroom by Alzheimer’s patients with error correction and plan identification, took five years and many researchers from different disciplines
There are two interesting problems associated with task support. The first one is to infer a plan and its goal (or a goal and its plan) on the basis of perceived actions of the end user. This problem is a classic AI issue and has been extraordinarily difficult to solve in a general sense. When this topic was discussed in L3D, one of the guest scholars commented on the difficulty of inferring a goal from a set of observed actions saying many times he himself did not really know what he was trying to do. This is also a common dead-end that computer scientists make when approaching dynamic and complex task support, starting with the assumption that it was possible to dynamically ascertain what the user was trying to do by observation and sensor inputs. This is a hard problem and, while a worthy goal, is really not an AT/AI project that will succeed in the near future. As discussed above, one way to make such a problem tractable is to constrain the problem space, which was what allowed the COACH project to succeed (i.e., in an instrumented bathroom, one simple task).
4.1.1CANONICAL PAPER
Schank, R.C. and R.P. Abelson (1977). Scripts, Plans, Goals, and Understanding. (Schank and Abelson, 1977).
4.1.2AT DESIGN EXAMPLES
Script Segments and Plan in MAPS
To use MAPS support for performing a task, first someone needs to make a set of prompts, but to do this several things need to be prepared: (1) the elements of the task need to be separated into individual sub-tasks, and (2) these subtasks need to be based on the existing abilities of the end-users. In some cases the subtasks will be large (e.g., a script to make a sandwich consists of three prompts—get the bread and sandwich ingredients from the refrigerator, make the sandwich, clean up); in some cases the prompts may be many more (e.g., go to the refrigerator, get the bread, put the bread on the countertop, go back to the refrigerator and get the package of ham and the mayonnaise, put them on the counter . . . and on). All of this is dependant on the available set of “internal” scripts that the end-user has available. In this case the element of intelligence augmentation is the fit to the user’s support needs, where simple AI might simply provide the script that breaks the talks into as many steps as possible, one size fits all, the IA approach tailors the script to fit the user. Table 4.1 is a segmented task that was created to evaluate the usability of the hand-held prompter.
The COACH project, as discussed in Section 2.2, particularly in Figure 2.15, used the plans and actions model to base its handwashing support system.
4.1.3CONCLUSION
Plans, scripts, and actions are all related to discrete tasks. When designing supports for accomplishing a specific and discrete task, the process of segmenting the task into steps can be useful, even if you are not specifically making a system similar to the above task support. For instance, in the ASSISTANT project, informally looking at the elder’s route guidance needs in this way lead the design group to a useful set of modalities that, put into a petri net-like structure (e.g., a finite-state machine), helped a lot in working our the Design for Failure systems in the project.
4.2LOW-HANGING FRUIT
Short Definition:
“Le meglio è l’inimico del bene”
“The best is the enemy of the good” (Voltaire; Peter Gay (1962)).
Approaching complex systems, by attempting to solve the whole problem, a designer may end up solving no problem. Complex AT should be carefully evaluated with respect to the actual needs of the user and available resources of the situation and often it is more successful if the easier and more common parts are addressed initially.
Longer Description: The standard way of developing a system starts with an idea and proceeds with requirements gathering. Based on interviews and focus groups with end-users, experts, and other stakeholders, technologists attempt to come up with a design that will surpass the effectivity of existing work practices. This is a good thing, however in our training as professionals we had drilled into us the need to conceder edge situations in design and implementation. A rough way to evaluate code as ready for the real world is counting cases of “if-then” statements that do not have other cases. We think robust; we seek coverage of the problem in beautifully complex and complete systems. However, this may work against us in real complex AT projects, as the solution may be too complex to use or may depend on somewhat fragile infrastructure that is efficacious in the lab and not so much in the “real world.” Attempting to do it all can often in complex applications lead to failure of the whole system. Furthermore, most projects of this kind are wicked in nature (see Section 2.8), with the consequence that the real requirements may not come to light until an initial prototype deployment. From this insight, we often referred in the L3D lab as “low-hanging fruit” as an initial goal, trying to keep the complete solution in mind so as not to prevent, by the architecture, later passes at the system that might implement more covering solution.
So, how to prevent the perfect preventing completion of good in high-functioning AT? One of the tools I have brought to this domain from a previous career as a material control manager in the 1970’s and 1980’s was the idea of controlling inventory with what is called in production and inventory control management as “ABC” analysis. At the time I was working in this discipline we were trying to emulate the Japanese, who had become experts in inventory control and just-in-time manufacturing following the approach of W. Edwards Deming (1950), whose ideas were rejected in the U.S., leading him to present them to the Japanese manufacturers. Coming out of his statistical approachs were sets of tools that supported efficient work practices to raise inventory accuracy in factories from typically 75% to the low 90% range. One of the tools that came out of the (eventually successful) attempts at total quality control and just-in-time manufacturing management, was striving to get very high accuracy of the raw and intermediate materials that comprised a cumulatively high cost in the finished product. Attempting to do this with all the parts and assemblies in the factory at once was too expensive to implement for all parts as a result of diminishing returns (i.e., there were items that were so relatively inexpensive that driving the records to 95% accuracy was more expensive in manpower than just buying more than you think you need). You don’t count individual paperclips—you make sure you always have more than enough—you do count automobile engines.
Based on Pareto’s11 observations that, in many fields of human endeavour, 80% of the value of the items being inspected is accounted for by 20% of the item types. It’s easier to see this in the context of the current concern about almost half of the world’s wealth is now owned by just 1% of the population. So inventory management broke this further onto 10/20/70%,12 which substantially supported the production gains in U.S. manufacturing in the 1990’s. For AT design, based on the gathered and imagined requirements, I have used the more approximate 80/20 split for planning implementation of complex AT. By producing robust solution to the 20% of the events and desires that end-users encounter in use of the applications and providing functions that get human help for those 80% of types of events that are only 20% of cases (see the two-basket approach in Section 3.2), it is more likely that the limited resources are utilized in the most effective manner.
The most difficult part of using this approach is to get realistic, real-world estimation of what those 20% of cases are. Domain experts are the most useful, especially for applications that are, to some degree, emulating existing work practices. Another way to approach this is to make some assumptions about the most frequent events and also the most difficult to capture/mitigate events. Then design and architecture that will accommodate both in a modular fashion categorize the estimated event and contact a human support for the more difficult one (see Section 2.2). By doing this you can test the general concept and integration of the system while making it easier for the implementation of the more difficult/rare events.
Several things can make this easier. One, error studies have shown that the absolute count of error event classes is typically a handful (see Section 4.1), and, two, the problems can be made more tractable by constraining the parameters like the COACH project did successfully.
An “inverse” example of the low-hanging fruit approach was Ritchie and Thompson’s (1974) decision not to implement in kernel code provision for breaking certain kinds or quite rare deadlocks in their UNIX operating system. Their decision not to do this was the trade-off between the cost of including this (the overhead of this running would slow down the operation of all processes) and the frequency of the event occurring (I can’t find a reference to this but in my undergraduate OS class it was estimated to be something like every 100,000 hours).
4.2.1CANONICAL PAPER
Interestingly enough, J.M Juran the evangelist for quality control and management consultant, later recanted his widespread promulgation of Pareto’s assumption for industrial management (Juran, 1975). However, before he did so, use of it influenced the practices of many practitioners of materials and production management in the U.S. and elsewhere.13
Juran, J.M. (1950). “Pareto. Lorenz, Cournot Bernoulli, Juran and others” (Juran, 1950).
4.2.2AT EXAMPLE
The principle that kept MAPS from becoming an intractably difficult problem involved reconsideration of the original goal: to support independence for a realistically large population, a subset of all of the people with cognitive disabilities. Domain experts considered “low-hanging fruit” of this subset of the population to be large enough (Braddock 2006). This subset population needed to be bound further by the limits of “could use the system” and “needed to use the system” (see Section 2.7). Given that, an existence proof could be made to show that it was possible to create scripts that would guide the user to do real (not trivial) tasks for ADL support. Assumptions had to be made about partial order plans and the resilience of this band of the population of people with cognitive disabilities as a basis for the design of such a system.
ASSISTANT Pedestrian Guidance
In the ASSISTANT project, preliminary trials of using smartphone pedestrian GPS were a failure (see Figure 4.3). There are many personal and research papers on the urban canyon effect (Martinek and emlicka, 2010), and antenna size that affirm this. The reason I think that so many projects and products rely on this technology is due to the success of the automobile systems on which so many drivers rely. One of the reasons that the automobile applications work is that they can make assumptions that if you are anywhere on or sometimes near the road it snaps the display to the road. However, pedestrians are not so predictable and due to their size need tighter granularity of location. GPS pedestrian navigation guidance has been quite successful in non-urban environments, due to the lack of the aforementioned urban canyon effect.
Figure 4.3: Actual urban route taken vs. smartphone GPS. From VTT (2013).
In an earlier project, HAPTIMAP (2012a), that Tecnalia worked on, there were similar problems. The toolkit that HAPTIMAP project (2012b) provided for smartphones supported traveling a route also. With the toolkit HAPTIMAP consortium members made applications that successfully guided typically able users to locate each other at huge open air music festivals and hike on trails well outside of the city. However, testing the very well done interface for blind and blind-deaf users in Madrid’s city streets required using a partner with a smartphone “spoofing” the GPS coordinates. It was a little better in the city park but still smartphone GPS was unusable.
Given that ASSISTANT’s specification for the system, when it was funded (ASSISTANT, 2012a), had the requirement of providing guidance for the “last kilometre” from the terminal vehicle stop to the door of the route goal this could not be just abandoned without further exploration. At a technical meeting with the consortium members we explored the boundaries of what was reliable about both the system state and the GPS signal. We knew when they get off the bus, where they were, and where the goal was, so an appropriately scaled and easy-to-use level of detail map of the walking route was provided. Also, averaging the GPS signal over time and using the built-in compass in the smartphone, the absolute direction of the goal was known and the system provided a compass arrow for the direction they should be going. This took the users a bit of getting used to because the arrow pointed directly at the goal (i.e., through buildings, and across the middle of streets); but doing this was modeled on the answer to the common question asked of others at the bus stop—“Where is the Brill Building?”—and the success that resulted with a finger point of just the “as the crow flies” direction. Finally, we provided a warning and compass if the user had walked in the wrong direction more than could be accounted for by zig-zagging through waypoints. This final support was a bit tricky because of the end-user tendency to abandon applications that provided too many false positives (type 1 errors).
The resultant tests on the application (13 users in 3 cities for 1 week with no support other than an emergency phone) were successful (ASSISTANT Consortium, 2015).
4.2.3CONCLUSION
The concept of first doing what can be done can be a very usefull strategy for finding out really what the answer to the problem will resemble. However it’s very important to make sure that in doing this (which may be the only way to make a wicked problem more tractable) that you are not painting yourself into a corner, and also making maintenance a nightmare. Draw the big picture and if low-hanging fruit is applicable, make sure that your solution is malleable to suite the eventual larger system. Sometimes the low-hanging fruit is all you need; sometimes it’s impossible to solve just the easy bits without a bigger covering solution. Like programming problems, design work, at its most effective, is 70% thinking and planning then the 30% of actual work.
4.3METADESIGN
Short Definition: MetaDesign is a theoretical framework for supporting open systems to allow end-users (and other stakeholders) to become designers in dynamic use contexts.
Longer Description: Of all the concepts discussed here, MetaDesign is perhaps the most difficult to pin down. This is for several reasons. First, it is used in pretty specific ways in disparate domains that do not completely overlap. So in the original context of Industrial Design MetaDesign is quite different in description and action than in Maturana and Varela’s biological approach (Maturana and Varela, 1987; Maturana, 1997), and Giaccardi’s and Fischer’s socio-technical approach (Fischer and Giaccardi, 2006). Additionally, there are a variety of ways to even spell MetaDesign: MetaDesign, Meta-design, Meta Design. In this book, I will use the spelling MetaDesign.
In task support AT tools and more generalized AT/IA applications MetaDesign is a way to talk about how to design tools that allow dyads (see Section 3.1) to together create and use supports for doing that which the end-user cannot do by themselves. This is a version of what Fischer and Giaccardi discuss as design for designers. This design approach refers to making artifacts that can be used to design further artifacts. To make sense of this, one needs to break the end-users into two groups, the (secondary) tool maker, in most cases a caregiver or more technically savvy member of the dyad, and the end-user herself, the person who used the created tool. There is a third person in the MetaDesign constellation, the MetaDesign tool designer/creator. Table 4.2 illustrates this for the example of the MAPS system.
Here is the same table for the ASSISTANT project.
Table 4.3: Overview of MetaDesign attributes for MAPS
Among the attributes of this kind of MetaDesign for AT that are relevant to AT designers are the different artifacts to be generated, the target user population, and the need for and danger of underdesign. Typically a MetaDesign project has at least two interfaces and one shared database (or persistent data of some sort). The first interface is for the person who will configure the final tool for task support; in this is a response to the universe of one problem and, in some ways, an ultimate personalization tool. The second interface is for the end user that will use it as support for task completion. The first tool will often not just create content to be played on the end-users interface but also may produce configuration commands for the second tool’s user interface. In this way the first tool becomes an example of end user programming (Nardi, 1993; Lieberman et al., 2006), and as such the designer may well profit from studying work on this topic.
Underdesign of the first tool needs to be guided by the position on the axis of, at one end, only supporting a very limited set of results to be used in the second tool, and on the other end of the continuum, producing a very general first tool but on that takes deep programming skills to do anything. A MetaDesign system at the first end may be very easy to use for the initial intended audience, but does not generalize well; at the other end it may be completely unworkable due to complexity. This is another example of the Turing tarpit (Perlis, 1982) where the tool may be able to do anything but anything of interest is impossible to do.
One solution to the underdesign problem is to clearly understand the internal scripts and skills of the intended end users, and to constrain the solution domain tightly enough so that appropriate support elements for that domain are built into the first tool.
4.3.1CANONICAL PAPER
There are many relevant publications on MetaDesign (which can be seen in Part 3), however the paper below represents the seminal use of MetaDesign in this domain. While not specifically pointed at AT it does capture all the elements that became the attributes in MetaDesign for AT.
Fischer, G. and E. Scharff (2000). “Meta-design—Design for designers.” (Fischer and Scharff, 2000).
4.3.2AT EXAMPLE
The section above on describing and designing MetaDesign in the MAPS system has said much about an AT example, there are some other dimensions worthy of discussion. First is the issue of MetaDesign over time, second is the idea of reusable and personalized components.
Figure 4.4: MAPS over time. From Carmien (2006b).
There are two timelines in MAPS MetaDesign, the design process, and the use practice (Figure 4.4). The MAPS system designer actually designed prototypes (corresponding to wicked problem iteration) of the MAPS script editor, hand-held prompter, and the database schema and functionality to bridge data between them. Once several iterations were tested and refined the realistic testing were done in a time span related to system use. One of the most difficult and important parts of the first tool, the scripts editor, was making it easy for the caregiver to use digital photographs that they had taken for the script; although the system could accommodate icons, that were provided, almost all the images were personal images collected by the caregiver. Similar care was taken with the recorded verbal prompts. Both of these were stored in a database rather than taking the easier route of using the Microsoft file systems and something like XML (as AbleLink’s commercial offering (AbleLink, 2002, 2007) and proposed standard (Technologies does) to ensure atomicity and robust reliability in an already fragile environment (i.e., IPAQ running winCE and windows in 2004).
4.3.3CONCLUSION
Ignoring the need for a MetaDesign approach to system design can lead to a lack of ecological validity of the solution. I have seen quite a few projects and proposals that thought a lot about the end user’s artifacts, and did not pay enough attention to the creation of the content in the artifact. Examples include work on learning material accessibility systems (see the discussion od EU4ALL) and scripting systems for the IoT, how are the scripts created and edited?
4.4PERSONALIZATION
Short Definition: Personalization is the process of structuring a system, both the user interface and the systems functionalities, to more precisely fit the abilities, needs, and desires of the end-user.
Longer Description: Practically every user interface since the Teletype has had some facilities for customization, from terminals where you could invert the screen background and foreground, to changing the voice accent in text to speech interfaces. The axis of customization runs from convenience to necessity. I can change the font type in my browser, which is nice. But my almost blind son-in-law needs a magnified screen to read his assigned text in engineering school. Here I am discussing the need end of the continuum.
The literature on personalization typically focuses on automatic interface personalization (Mourlas and Germanakos, 2009), and the first example below shows how, laboriously, such automatic service works, as well as the hidden drawbacks that are not often discussed. I have gone through the personalization workflow in (excruciating) detail not so much to provide a model to follow but as an example of how complex and interconnected doing adaptation can be in the practice. Personalization of the interface by providing the tools for a caregiver or end-user to configure their user environment has been the bulk of my work.
To simplify the issues involved in automatic and self controlled personalization I introduce the notion of the two kinds of personalization, Ser and Estar personalization Models. Whether explicit or tacit, behind all personalization systems lies a model of the user. This abstraction allows computational personalization to be implemented.
One common design problem encountered by user model systems is not distinguishing between user model attributes that represent the users “being” and the users state. This leads to unnecessary data duplication and can migrate up to the user interface, causing confusion in the system’s end-user and unnecessary duplication in the system’s architecture and persistent storage. Looking at the whole of user modeling data, it basically falls into two kinds of models, which I have called the ser and estar types of user models.
I have chosen to use two Spanish verbs that express essential and acquired qualities because their use roughly corresponds to the two kinds of data: one uses Ser to refer to the essential characteristics of things that you are, your name, your gender; Estar is used to refer to descriptive, potentially temporary attributes, the condition, like I am walking. The Ser user model contains the information about the users—these essential characteristics form the static user model. In Ser models changes occur very infrequently over time and they are typically independent of context. The Estar user models, in contrast, contain changing values that model the current condition of user; this dynamic user model may change over time, or there may be multiple copies that are linked one at a time to the ser model of the user. Estar models are often context dependent or reflect the different devices that are used at different times. Estar data type is used to keep histories of user actions.
This novel user model framework is driven by a relational database approach to users and contexts (i.e., normalization (Codd, 1990)), and inspired by several experiences with projects and models where the two were conflated, with problematic consequences. By keeping data that never changed in the same “table” as attribute values that change frequently, the system is forced to either keep multiple copies of a user model for a single person, or to constantly change the model as the context of use changes. Also, conflation of the two types makes keeping histories complicated and causes redundant data to be stored. This section will discuss the ramifications of these approaches. This is not a “new” approach per se, but by explicitly creating a framework that holds these two models design and performance advantages can be accrued.
Another advantage of discriminating between ser and estar models is that this provides a nice way to separate the actual data, so that applications with large amounts of sensor data can store it locally, making a mobile system less dependent on reliable high bandwidth connectivity.
Looking at these two model types, most of the effort to do automatic personalizing should be applied to estar data. And, in most cases, the Ser model data can be quite simply setup once and not need further attention. It is these Ser preferences that are on the preferences page of ASSISTANT (see Figure 4.5).
Figure 4.5: Preferences choices in ASSISTANT. From Assistant Project (2015).
4.4.1CANONICAL PAPERS
There really are not any “canonical” publications about personalization, especially not for static personalization. Below is a good start at dynamic personalization overview, and the human interface apple guidelines do a good job of taking about the interface for the end-user.
Mourlas, C. and P. Germanakos (2009). “Intelligent user interfaces: Adaptation and personalization systems and technologies.” (Mourlas and Germanakos, 2009).
Apple, Inc. (2009). Apple Human Interface Guidelines (Apple, 2009).
4.4.2AT DESIGN EXAMPLES
A current topic in European research is systems supporting universal access to distance learning,14,15,16 the specifics described here came out of initial design work in the EU4ALL project (EU4ALL, 2007). These projects emphasize the use of content fitted to stereotypical sensory, and to a lesser degree, cognitive disabilities. Adding to the complexity of the problem is the rising popularity of mobile platforms; where previously a reasonable assumption was that the material would be accessed through “typical” PCs, now material may be presented on screens as small as a smartphone and in the pad format. The rise of ubiquitous and embodied computational services adds to the complexity of delivery of the right content, in the right way, at the right time, and via the right medium (Fischer, 2001b).
This kind of personalization is a special case of a broad range of applications and systems that will be tailored to a user’s needs and capabilities and with respect to the context at the time and place of use. Scenarios of use may include searching and accessing schedules and real-time location of buses in transit services non-visually on a smartphone (Sullivan, 2004); reading medical records on a wall display closest to the users current location and printing them on the nearby printer; locating a friend in a crowded shopping district, locating a film center nearby and purchasing tickets, and using a mobile pad to present a text version of a lecture given as part of a distance learning university course.
This kind of deep personalization is dependent on three things: the end user, the context (which includes the device used for input and output), and the content. Each end-user presents a unique set of abilities and needs; sometimes this may be as broad as a preferred language or as deep as sensory and motoric disabilities with preferred alterative presentation mode (e.g., low visual acuity and synthesized text) and preferred input mode (i.e., voice commands, scan and select input). The context of the application has two dimensions: the actual environment, including the history of the user’s interactions with the computer (in general) and earlier sessions with this application; and the device with which the user interacts with the system, including the device constraints (i.e., small screened smartphones) and the device capabilities (i.e., speakers, text synthesizer). The third part of the personalization is the content that is presented to the user; this may be one of a set of identical “content” expressed in different modalities, or a server that adapts the content to the needs and capabilities of the user/device.
It is difficult to discuss a system that uses user models without considering the context in which the user models provide leverage. Start by considering the user herself, the User Model (UM). While the specifics discussed here are concerned with the needs and abilities that differ from “typically” abled users, looking at the problem with this lens makes it easier to highlight the salient issues for all users. Each person brings to the problem space a unique set of abilities and needs, and it is important to consider both the disability and the preferred mode of adaptation that this user has. Because of the unique set of attributes that the user brings to the application the use of simple stereotypes, while initially appealing, may cause a poor fit (due to the too large granularity of the stereotype) between document, applications and user. For these reasons these user models have attributes that discretely describe individuals to the adaptation system. Examples of these discrete attributes may include various levels of visual disability, where describing their visual sense with a binary blind/not blind may not capture the large percentage of partially sighted people who could gain some advantage from a custom form of visual interface. Similarly, motoric and cognitive disabilities both require a wide range of attribute-value pairs to describe them. Further, a given person may have a combination of sensory/motoric/cognitive abilities that makes them unique. Ontologies and hierarchical description schemas may be useful, both as a support for attribute names but also in that nodes that are not leafs may also provide goals for places to base accessibility support. Finally, this part of the user modeling system should only concern itself with those attributed to the user, which never change, or change very slowly monotonically down an axis of ability. This means that if a user condition pertinent to the application waxes and wanes, in contrast to only decreasing over time, this should be captured as a part of the context /device model as described below.
The next element in this user modeling system is the device model (DM), which in a larger sense is the context. This means several things: (1) the actual context which includes the current time and place and various other details that constitute the environment; (2) the device used to access the material; and (3) the changing attributes of the user. To describe the actual physical context existing ontologies and frameworks may prove both useful and a way to access existing data sources. Part of the captured context includes resources as well as static descriptions, i.e., network accessibility, temperature, light.
Device as context makes sense if you talk about context as everything on the outside of the user that affects, or is pertinent to, the user. In the case of the user’s device this provides an inventory of capabilities to the system. The inventory of device abilities can include local availability of resources (i.e., printers, Java, browser) I/O capabilities such as a list of compatible mime types (IANA (Internet Assigned Numbers Authority), 1996) and input affordances (touch screen, speech synthesizer, and verbal recognition). These qualities can be represented in existing schemas (frameworks or ontologies) such as UAProf and CC/PP (Open Mobile Alliance, 2007; W3C, 2007). A more dynamic approach for device description can be taken with systems like the examples of V2 (trace.wisc.edu, 2005) and URC (URC Consortium, 2007), or the Aria initiative (W3C, 2009a) where the device can self configure to match the needs of the applications user controls.
Finally, context can be considered as the history of the user and application. History as context can be built from system history, application history, and user’s history (as captured for example, in bookmarks and previous preferred configurations). The user modeling system can use these as both local support for making inferences, i.e., he is doing this task and has just finished this subtask, and as a basis for inferring conditions of importance to the successful completion of the goal-at-hand. An example of this could come from a task support application (Carmien, 2007) where the user may have paused for a long time at this point in previous trials (and might trigger additional help from the system) or where the user is going over the sub task prompts very rapidly, from which the application may infer that the user has these sub-tasks Memorized in a chunk and therefore might flag compression of many prompts for a set of sub-tasks into one single prompt for a larger sub-task.
The last part of this user modeling system is the content itself. It may be easier to think of the content as a participant in the accessibility process to talk about it in process terms. This example uses the term Digital Resource Description (DRD), a set of metadata associated with documents. In the educational example being discussed every document or “chunk” of digital material in an educational process has a DRD record. This approach comes out of the work that has been done in learning object metadata (LOM—IEEE 1484.12.1-2002 Standard for Learning Object Metadata) (IMS Global Learning Consortium, 2007). A LOM is a data model, usually encoded in XML, used to describe a learning object (a chunk of content). In this case the specifics were drawn from a pre-release copy of the ISO 24751 standard (ISO/IEC 2007a, ISO/IEC 2007b, ISO/IEC, 2007c).
The provision of appropriate material results from this set of attributes and values fitting together with the needs of the user as expressed in her user model and the capabilities of the user current device as express in the DM. This approach can take two forms: (1) selection and delivery, and (2) adaption. Both of these have network bandwidth requirements for successful use, which are part of the information on the DRD record. The selection and delivery approach finds the existing right content based on the user and device/context and presents this to the user. The adaption approach takes a Meta document representing the content and creates the accessible content on the fly.
Select and deliver requires the system to create and store different versions of the same material, and this leads to authoring and on-going revision problems. The advantage of replacement it that it is relatively easy to implement, after the matching infrastructure in the UM and DM are in place. The problem with this approach is that content authors have the burden of many adaptations and later modifications of material will make sets of the same content out of synch. Adaptation forces the author to write in a kind of markup language, which may be mitigated by authoring support tools. Adaptation also requires another layer of software to take the marked up content as input and produce the appropriate content; this may either be done on the server or at the client. Also adaptation may support interface adaptation like ARIA (W3C, 2009a) and URC/V2 (INCTS, 2007), taking motoric issues in accessibility into account.
Adaption, while a complete solution, requires standardizing and built-in functionality like cascading style sheets (CSS), which have been used in this fashion with some success. Select and deliver can use existing systems but adaptation requires a whole new infrastructure, and more relevantly, the adoption of this infrastructure by all content developers.
Having described all the parts of the user modeling for accessibility system, it can be now described it in shorthand as:
UM + DM + DRD = CP, which can be called the Content Personalization formula.
Where UM is the user model. DM refers here to the Device Model, remembering that it includes context and history in the DM. The DRD refers to the metadata that describes the content at hand, which may take many different forms while pointing to the same concepts. Finally the CP stands for content personalized for the user and context. In order for this to work there must exist a tightly controlled vocabulary, which is often domain specific, so that UM, DM, and DRD must have matching attributes and ranges of values. It is important to ensure adoption of such systems by building on existing standards, such as shown in Table 4.4.
Table 4.4: Existing personalized distance learning standards |
LOM (IEEE) |
Dublin Core (DC-education extensions) |
IMS |
ACCLIP |
ACCMD |
ISO 24751-1,2,3 |
DRD, PNP |
Here are existing standards used in the following example.
•User Modeling (UM): PNP (personal needs and preferences from part 2 ISO 24751 draft 2007).
•Device modeling (DM): CC/PP (from W3C Composite Capabilities/Preference Profiles: Structure and Vocabularies 2.0 (CC/PP 2.0).
•Content Metadata: DRD (from Part 3: Access for All ISO 24751 draft 2007).
So the content personalization formula then becomes:
PNP + CC/PP + DRD = CP
Here is an example of the process, in our prototypical remote learning accessibility system (Figure 4.6). First, there is a request for content object (CO) in learning process. To do this, the user agent (browser) uses a proxy that inserts device model ID into the header of http request containing the request for a CO in the form of the device identifier. Then the virtual learning environment (VLE) passes these (the content ID, the UM-ID, and the DM-ID) to a content personalization module (using web service calls from here to the return to the VLE). Content personalization module then gets:
1.user profile from user modeling subsystem;
2.device profile from device modeling system;
3.CO accessibility metadata-digital resource description (DRD) from Learning Object Metadata Repository; and
4.it matches them and returns the right one (if there is one).
Figure 4.6: Content personalization flow in select and deliver model. From Carmien and Cantara (2012).
An alternative process flow, using the adaptation model is seen in Figure 4.7.
Following that are examples of both the select and deliver approach and the adapt approach; to deliver accessible educational materials to students, matched to their needs and the capabilities of the device that they are using. To effect this the three parts of the CP equation need to be considered. This can be expressed as shown in Table 4.5.
This process is further broken down in Tables 4.5–4.13. Table 4.6 showing the Adaption Preference from PNP; Table 4.7 containing Preferences from PNP; Table 4.8 giving an example of DM Attributes from CC/PP; Table 4.9 laying out the parts of a Media Meta Record from DRM; Table 4.10 explicating how Table 4.6 is comprised of the Access Mode Record Template; Table 4.11 expands on Table 4.10 into an Adaptation Template; Table 4.12 illustrates the use of the Adaptation Template; and finally Table 4.13 shows the whole process in the Content Selection Matrix.
Figure 4.7: Content personalization flow in adaptation process. From Carmien and Cantara (2012).
The PNP record in the user model stores the user’s preferences of content presentation. The index into the preferences table:
The preferences table itself.
Having gotten the user needs from the above, the personalizing process continues with the Device Model (in this simple case there are no context or historical data, but this is where they would go). Here the DM is based on the CC/PP record from UAProf (Wireless Application Protocol Forum, 2001):
Where the mime types are:
•.aif audio/aiff;
•.aos application/x-nokia-9000;
•.asm text/x-asm;
•.eps application/postscript;
•.gif image/gif;
•.html text/html;
•.java text/plain;
•.jpg image/jpeg;
and the transformation type could be:
•Text -> audio;
•Audio-> text; and
•HTML-> correct color contrast.
Finally, there is the metadata attached to each chunk of content. This is taken from the ISO 24751 Part 3, the DRD record template. The media META record:
An access mode record:
An “is adaptation” record:
And finally, the adaptation statement (AS):
Putting them altogether:
Here is a more complex example requiring multiple adaptations (Figure 4.8). In this example there is a video of a physics lecture and a demonstration. The video has two parts: audio and visual.
Figure 4.8: A more complex example of selecting content. From Carmien and Cantara (2012).
Finally, an example (Table 4.14) using device-based transformation.
Disability |
Visually impaired |
Example |
Text (a book, etc.) |
Orig. access mode |
Screen reader |
Adaptation type |
Orig. access. Mode = Text Adaptation type = Audio representation Adaptation_ preference _ranking = 1 |
PNP attribute/values |
In the adaptation stanza Orig. access. Mode = Text No matching adaptation type |
DRD Attribute/values |
In the adaptation stanza Orig. access. Mode = Text No matching adaptation type |
DM Attribute/values |
DM has transformation attribute that maps from text to audio (i.e., has jaws or the like) |
The above brief introduction to the complex domain of personalized content delivery is but one of many possible implementations of the UM + DM + DRD = CP formula. In the author’s opinion, the content selection and delivery approach is seriously flawed due to the authoring and updating requirement. By this I mean that although a given “chunk” of content may exist in many different media, covering most possible needs and disabilities, any updating of the content requires simultaneous updating of all variations, a condition that has a very low probability of happening. On the other hand, the adaptation approach, while complete, requires the adoption of a meta-language and adaption engines across the domain, which requires an integrated system to be designed, implemented, and widely adopted. Because of the difficulty of doing this current efforts are typically focused on the select and deliver approach.
The other approach widely implemented is writing educational content in a markup language. The wide implementation is, of course, the World Wide Web. A browser can take as in input an html/css file set and produce very different output depending on the settings and add-ons in the browser. The drawbacks to this approach are two-fold:
1.All education content authors and consumers must agree on the markup language and the produced file interpreter. This worked well for the WWW and there is work being done in the W3 consortium (W3C, 2009b), but the LOM community has demonstrated needs that HTML/CSS may not apply to.
2.Education content authors now must either master the markup language or authoring tools need to be implemented and widely accepted.
So this automatic problem is very real and not solved in a fashion amenable to wide adoption. Either endless updates of many, many different expressions of the same content, including the problems with orphan files and enslaved authors; or making all content to be written in a markup language with near universal adoption of a markup language reader.
4.4.3CONCLUSION
So, there are two types of personalization: (1) static that can be fairly easily implemented and is mostly dependent on the perception of the needs of the various end-users and proposed functionality of the application, and (2) dynamic that reflects the changing context of the user in action. It is important not to confuse static and dynamic; the conflation can cause both problems with data and determining which user is the user. It is important that, as much as possible, these static preferences are scrutible, that the end-user or caregiver can see and change them as needed. The second type, personalized dynamic content delivery, is much more complex (see the length of the illustration above) and has non-trivial problems with authoring tools and updating.
Some very promising work is being done in dynamic personalization and content representation in the Global Public Inclusive Infrastructure (GPII)17 consortium and the work of Gregg Vanderheiden and Jutta Treviranus. This is the current result of a long process stating in the URC (trace.wisc.edu, 2005; Zimmermann et al., 2006) work in the 1990s leading to the Raising the Floor initiative.18 This group has done some very pertinent work ion addressing the “this will solve the problem perfectly, BUT everyone must adopt XXX framework/language/notation” dilemma. As of 2016 this is a project to keep an eye on, and perhaps participate in.
4.5SYMMETRY OF IGNORANCE
Short Definition: Since the Renaissance it has not been possible for any one person, no matter how talented or educated, to have all the knowledge required for design or creating a non-trivial system.
Longer Description: The symmetry of ignorance (or asymmetry of knowledge) is a way of describing situations in which several participants or roles in an endeavour each individually have parts of the knowledge needed to accomplish the task, but none has enough to accomplish the task independently. Symmetry of ignorance or asymmetry of knowledge (Ostwald, 2003) refers to the fact that for many “real-world” problems no one person or role has all the knowledge to design a satisficing solution. A satisficing solution (Simon, 1996) refers to a solution that is “good enough” for all the stakeholders to achieve some minimum of desired functionality. This is a useful notion, because any attempt to reach an optimal solution is very difficult due to limits on resources and time.
As Gerhard Fischer pointed out (Fischer, 1999):
The Renaissance scholar does not exist anymore. Human beings have a bounded rationality—making satisfying instead of optimizing a necessity. There is only so much we can remember and there is only so much we can learn. Talented people require approximately a decade to reach top professional proficiency. When a domain reaches a point where the knowledge for skillful professional practice cannot be acquired in a decade, specialization will increase, collaboration will become a necessity, and practitioners will make increasing use of reference aids, such as printed and computational media supporting external cognition.
Much of our intelligence and creativity results from the collective memory of communities of practice and of the artifacts and technology surrounding them. Though creative individuals are often thought of working in isolation, the role of interaction and collaboration with other individuals is critical. Creative activity grows out of the relationship between an individual and the world of his or her work, and out of the ties between an individual and other human beings. The basic capacities . . . are then differentially organized and elaborated into complex systems of higher psychological functions, depending on the actual activities in which people engage. These activities depend crucially on the historical and cultural circumstances in which people live (Resnick et al., 1991).
There are two primary components to this need for a community of interest (Wenger, 1998) approach to larger design and implementation projects. First, is the mastery of skills, where there is simply not enough time for any one person to obtain all the relevant skills, especially up to date at the same time. Second, is the need for many perspectives, similar to the need for stakeholder analysis (Overseas Development Administration, 1995) in the early stages of the project. Again, not enough time to gain the “observational” experience to make the design effective, and in the case of AT successfully adoptable.
How can one expect an individual to maintain the requisite specialist knowledge in their technological discipline, while at the same time have the needed competence in industrial design, sociology, anthropology, psychology, etc.? The solution to this dilemma lies in the notion of Renaissance team (Buxton, 2002): a social network of specialists from different disciplines working as a team with a common language. One way of seeing this is the exploitation of the symmetry of ignorance, by exploiting my expertise in an area you are lacking in (and vice versa, over a group). This requires support by CSCW tools and structured participation to be more than inconsequential. In a balanced team the deficiency of knowledge in one member will be balanced by the expertise being held by another member, and this symmetry is spread across the research team
An example of not attending to the need for using the perspective of symmetry of ignorance that I experienced as a MIS manager for a hotel reservation company (Express Reservations, Carmien (2002)) was shopping for a new hotel reservation system. Those we looked at that were designed by programmers alone ran and were quite robust and extendable to our needs (a phone-based reservation system based in Boulder Colorado for hotels in a 33-square block area in midtown Manhattan), but were virtually useless for making actual hotel reservations and the data needed behind the front end. The offerings designed by hotel reservationists had a totally fitting interface, supporting existing work practices (i.e., understood what a reservationist needed and how and what way to store the information about hotels, reservations and rates) that was needed and in the way that fit best—but continuously crashed
Symmetry of ignorance is part of why domain-spanning initiatives can be so successful. An acknowledgement of symmetry of ignorance is the concept of a community of interest. Communities of interest are groups of people (typically coming from different disciplines) that engage in a joint activity. Instead of being obstructed by lack of expertise one gets a synergistic coverage of the domain
4.5.1CANONICAL PAPER
Rittel, H.W.J. (1972). “On the planning crisis: Systems analysis of the first and second generations. (Rittel, 1972)
4.5.2AT EXAMPLES
MAPS and the role of Anja in CLever set of projects. Leveraging the Symmetry of Ignorance. There were two sets of members in the symmetry of ignorance in the MAPS system: (1) caregivers and clients; and (2) tool (AT) designers and caregivers. Here was a further iteration of the notion of symmetry of ignorance at a higher design level: the caregiver and the script editor together have the solution to an adoptable script editor, but neither has all the relevant skills and knowledge to construct an editor on their own.
Requirement |
Partner |
Knowledge of older people and technology |
ESENIORS |
Assistive Technology design and design for all |
TECN |
Knowledge of existing State of the Art |
UNIVIE, TECN |
System Architecture design |
VTT |
Route editor design |
UNIVIE, FARA |
Transport data analysis |
VTT, FARA, UNIVIE |
User modeling |
TECN |
Error detection |
TECN |
Error recovery |
TECN |
PND design |
TECN |
GIS and telematics expertise |
UNIVIE |
GPS |
UNIVIE |
System integration |
VTT TECN |
Field trials |
ESENIORS, TECN, VTT, CIT |
System validation |
VTT |
Commercial exploitation |
TTR |
Dissemination |
TTR |
Project management |
TECN |
Service Oriented Architecture |
Subcontracted by TECN to IN2 |
Map and transportation data |
Local transport authorities, mapping organizations |
Develop mobile application |
CITRUNA |
Another example of the way that using the symmetry of ignorance can help to avoid system killing approaches. The ASSISTANT project (Table 4.15) put the way-point and map-based guidance information for elders who found exiting smartphone based transport navigation support too complex or with an interface too difficult to use. This Personal Navigation Device (PND) application was to be installed on medium sized smartphones for the targeted population. The technical partners of the ASSISTANT consortium, experts in mobile phone interface design and implementation of complex functionality at the server, phone and data supplier (i.e., schedule, routes, and real-time data from local transit companies) level, brought an enormous amount of skills and knowledge to the problem. What they did not bring to the problem was knowledge of the specific sensory and cognitive needs of the target elder population.
Figure 4.9: The PND GUI: left—initial designs; right—final design. From Assistant Project (2015).
Fortunately, the consortium was brought together initially for the Symmetry of Ignorance perspective, as a result other consortium members had intimate and deep knowledge of elders and elders adoption of new technology, expertise in human computer interaction with emphasis on AT design, and knowledge of research literature on navigation guidance and design of interfaces for elders. So, in the end the interface for the phone was well accepted and there were no requests for UI change by the elders after a week long trial in “the wild.” However, there were many detailed and almost acrimonious discussion about the elements to include or put focus on (Figure 4.9). The positive result, besides the PND itself, was a meta study (Carmien and Garzo, 2014) of appropriate design guidelines for these types of systems, motivated by a need to demonstrate that the GUI design for elders is quite different from GUI design for 20-35 year-old programmers.
4.5.3CONCLUSION
Taking the Symmetry of Ignorance approach may be somewhat tricky due to power issues: who has the vision and who decides the relative priority of different requirements that need to be coordinated in a satisficing solution. In ASSISTANT the solution was a well-defined set of UI specifications, general enough so that the implementation was not forced to use specific elements and controls. Additionally face-to-face meetings allow everyone’s ideas to be expressed and rationale for each part to be agreed on, something that just exchanging images of deign alone does not support well.
11Vilfredo Federico Damaso Pareto (1848–1923), Italian engineer, sociologist, economist, political scientist, and philosopher. He made several important contributions to economics, most meaningful in this book was his study of income distribution.
12In materials management, the ABC analysis (or Selective Inventory Control) is an inventory categorization technique. ABC analysis divides an inventory into three categories- “A items” with very tight control and accurate records, “B items” with less tightly controlled and good records, and “C items” with the simplest controls possible and minimal records. The selecting function is typically total yearly cost and loss cost due to missing inventory.
14http://www.aegis-project.eu.
15http://adenu.ia.uned.es/alpe/index.tcl.