Goals of Performance Assessments

Performance assessments can measure students’ cognitive thinking and reasoning skills and their ability to apply knowledge to solve realistic, meaningful problems. They are designed to more closely reflect the performance of interest, allow students to construct or perform an original response, and use predetermined criteria to evaluate student work. The close similarity between the performance that is assessed and the performance of interest is the defining characteristic of a performance assessment (Kane, Crooks, & Cohen, 1999). As stated by the Standards for Educational and Psychological Testing, performance assessments attempt to “emulate the context or conditions in which the intended knowledge or skills are actually applied” (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999, p. 137).

As this definition indicates, performance assessments do not have to assess complex reasoning and problem-solving skills. For example, if the targeted domain is the speed and accuracy at which students can keyboard, a measure that captures the accuracy and speed of students’ keyboarding would be considered a performance assessment. Clearly keyboarding is not a high-level thinking skill but a learned, automated procedural skill. However, the performance assessments we treat in this chapter are those designed to assess complex reasoning and problem-solving skills in academic disciplines and that can be used for large-scale assessments.

The Maryland School Performance Assessment Program (MSPAP) was an excellent example of a performance assessment that consisted of interdisciplinary tasks that assessed problem-solving, reasoning, and evaluation skills (Maryland State Board of Education, 1995). As an example for a grade 5 science MSPAP task, students were asked to investigate how a hydrometer can be used to measure different levels of saltiness (salinity) in various water samples, predict how the hydrometer might float in mixtures of freshwater and saltwater, and determine how the hydrometer could be used to establish the correct salinity for an aquarium. This hands-on task allowed students to conduct several investigations, make predictions, evaluate their work, and provide explanations for their responses.

When working on well-designed performance tasks, students may be engaged in applying their knowledge to real-world problems, evaluating different approaches to solving problems, and providing reasoning for their solutions. A prevailing assumption underlying performance assessments is that they serve as motivators in improving student achievement and learning and that they encourage instructional strategies that foster reasoning, problem solving, and communication (Frederiksen & Collins, 1989; National Council on Education Standards and Testing, 1992; Resnick & Resnick, 1982). Performance assessments can not only link school activities to real-world experiences (Darling-Hammond, Ancess, & Falk, 1995); they can include opportunities for self-reflection and collaboration as well as student choice, such as choosing a particular topic for a writing assignment (Baker, O’Neil, & Linn, 1993; Baron, 1991).

Collaborative efforts were required on the MSPAP (Maryland State Board of Education, 1995) in that students worked together on conducting science investigations and evaluated each other’s essays. Collaboration is required on many performance assessments in other countries (see chapter 4, this volume). It can be argued that these types of collaborations on performance assessments better reflect skills required in the twenty-first century.

Performance assessments may also allow a particular task to yield multiple scores in different content domains, which has practical as well as pedagogical appeal. Tasks that are designed to elicit scores in more than one content domain may not only reflect a more integrated approach in instruction, but also motivate a more integrated approach to learning. As an example, MSPAP tasks were integrated and would yield scores in two or more domains (Maryland State Board of Education, 1995). Practical implications for tasks that afford multiple scores may be reduced time and cost for task development, test administration, and scoring by raters (Goldberg & Roswell, 2001). It is important, however, to provide evidence that each score represents the construct it is designed to measure and does not include construct-irrelevant variance (Messick, 1989).

Furthermore, Mislevy (1996) pointed out that performance assessments may enable better measurements of change, both quantitative and qualitative. A hypothetical example of a quantitative measure of change for a mathematics performance assessment would be that a student used a recursive strategy only two out of ten times during the first administration of a math assessment, but six out of ten times during the second administration. An example of a qualitative evaluation of change would be that the student switched from a less effective strategy to the more sophisticated recursive strategy after instruction.

Performance assessments, in particular, writing assessments, have been included in some large-scale assessment programs in the United States for monitoring students’ progress toward meeting national or state content standards, promoting educational reform, and holding schools accountable for student learning. High-stakes decisions are typically required, as well as an evaluation of changes in performance over time, which requires a level of standardization of the content to be assessed, the administration of the assessment, and the scoring of student performances over time. Thus, extended time periods, collaborative work, choice of task, and use of ancillary material may challenge the standardization of the assessment and, consequently, the accuracy of score interpretations. Large-scale performance assessment programs such as MSPAP, however, have included these attractive features of performance assessments while ensuring the quality and validity of the score interpretations at the school level.

Another consideration in the design of assessments of complex skills is whether a portfolio approach will be used. The Advanced Placement (AP) Studio Art portfolios provide an excellent example of a large-scale portfolio assessment that has been sustained over time (Myford & Mislevy, 1995). As an example, in the three-dimensional design portfolio, students are required to submit a specified series of images of their three-dimensional artworks, which are evaluated independently according to their quality (demonstration of form, technique, and content), breadth (demonstration of visual principles and material techniques), and concentration (demonstration of depth of investigation and process of discovery). Using a well-delineated scoring rubric for each of these three areas, from three to seven artist-educators evaluate the submitted images. The portfolios that are submitted are standardized in that specific instructions are provided to students that specify what type of artwork is appropriate, along with detailed scoring rubrics that delineate what is expected for each of the dimensions being assessed.

Performance assessments that are aligned with curriculum and instruction can provide valuable information to guide the instructional process. Thus, it is imperative to ensure that both classroom and large-scale assessments are aligned to the curriculum and instruction. A rich, contextualized curriculum-embedded performance assessment that is used for classroom purposes does not require the same level of standardization as the typical large-scale performance assessment. These curriculum-embedded assessments allow teachers to more fully discern the ways in which students understand the subject matter and can help guide day-to-day instruction. Large-scale assessments can also inform instruction, but at a broader level for both individual students and groups of students (e.g., classrooms).

Cognitive Theories in the Design of Performance Assessments

The need for models of cognition and learning and quantitative psychometric models to be used together to develop and interpret achievement measures has been widely recognized (Embretson, 1985; Glaser, Lesgold, & Lajoie, 1987; National Research Council, 2001). When we understand how individuals acquire and structure knowledge and cognitive skills and how they perform cognitive tasks, we are better able to assess their reasoning and obtain information that will lead to improved learning. Substantial theories of knowledge acquisition are needed in order to design assessments that can be used in meaningful ways to guide instruction and monitor student learning.

Several early research programs that have had a direct impact on the assessment of achievement studied the difference between experts’ and novices’ knowledge structures (Simon & Chase, 1973; Chi, Feltovich, & Glaser, 1981). Much of what is known in the development of expertise is based on studies of students’ acquisition of knowledge and skills in content domains. For example, Chi and her colleagues (1981) demonstrated that an expert’s knowledge in physics is organized around central principles of physics, whereas a novice’s knowledge is organized around the surface features represented in the problem description.

Other early approaches that link cognitive models of learning and psychometrics have drawn on work in the area of artificial intelligence (e.g., Brown & Burton, 1978). As an example, Brown and Burton (1978) represented the complex procedures underlying addition and subtraction as a set of component procedural skills, and described proficiency in terms of these procedural skills. Using artificial intelligence, they were able to uncover procedural errors or bugs in students’ performance that represented misconceptions in their understanding.

Cognitive task analysis using experts’ talk-alouds (Ericsson & Smith, 1991) has also been used to design performance assessments in the medical domain (Mislevy, Steinberg, Breyer, Almond, & Johnson, 2002). Features of the experts’ thinking, knowledge, procedures, and problem posing are considered to be indicators of developing expertise in the domain (Glaser et al., 1987) and can be used systematically in the design of assessment tasks. These features can then be used in the design of the scoring rubrics by embedding them in the criteria at each score level. Experts need not be professionals in the field. Instead, in the design of K–12 assessments, experts are typically considered students who have attained competency within the content domain.

While many researchers recognize that theories of cognition and learning should ground the design and interpretation of assessments, widespread use of cognitive models of learning in assessment design has not been realized. As summarized by Bennett and Gitomer (2009), there are three primary reasons for this: (1) the disciplines of psychometrics and of cognition and learning that have developed separately are just beginning to merge, (2) theories of the nature of proficiency and learning progressions are not fully developed, and (3) economic and practical constraints affect what is possible.

Some promising assessment design efforts, however, are taking advantage of what has been learned about the acquisition of student proficiency. A systematic approach to designing assessments that reflect theories of cognition and learning is embodied in Mislevy and his colleagues’ (Mislevy, Steinberg, and Almond, 2003) evidence-centered design (ECD), in which evidence observed in student performances on complex problem-solving tasks (that have clearly articulated cognitive demands) is used to make inferences about student proficiency. Some of these design efforts are discussed in this chapter.

A Conceptual Framework for Design

A well-designed performance assessment begins with the delineation of the conceptual framework. The extent to which the conceptual framework considers cognitive theories of student proficiency and is closely aligned to the relevant curriculum will affect the validity of score interpretations. The delineation of the conceptual framework includes a description of the construct to be assessed, the purpose of the assessment, and the intended inferences to be drawn from the assessment results (Lane & Stone, 2006). Construct theory as a guide to the development of an assessment provides a rational basis for specifying features of assessment tasks and scoring rubrics, as well as for expecting certain empirical evidence, such as the extent of homogeneity of item responses and the relationship between scores with other measures (Messick, 1994; Mislevy, 1996; National Research Council, 2001).

Two general approaches to designing performance assessments have been proposed: a construct-centered approach and a task-centered approach (Messick, 1994). We focus here on the construct-centered approach, in which developers start by identifying the set of knowledge and skills that are valued in instruction and need to be assessed and then identify the performances or responses that should be elicited by the assessment. Thus, the construct guides the development of the tasks as well as the specification of the scoring criteria.

This process allows the designer to pay attention to both construct underrepresentation and construct-irrelevant variance, which may have an impact on the validity of score inferences (Messick, 1994). Construct underrepresentation occurs when the assessment does not fully capture the targeted construct, and therefore the score inferences may not be generalizable to the larger domain of interest. Construct-irrelevant variance occurs when one or more irrelevant constructs is being assessed in addition to the intended construct. For example, students’ writing ability may have an unwanted impact on performance on a mathematics assessment. (This is further treated in the section in this chapter on validity and fairness.) Scores derived from a construct-centered approach may be more generalizable across tasks, settings, and examinee groups than scores derived from a task-centered approach because of the attention to reducing construct-irrelevant variance and increasing the representation of the construct (Messick, 1994).

As an example, a construct-centered approach was taken in the design of a mathematics performance assessment that required students to show their solution processes and explain their reasoning (Lane, 1993; Lane et al., 1995). Cognitive theories of student mathematical proficiency provided a foundation for defining the construct domain. The conceptual framework that Lane and her colleagues proposed then guided the design of the performance tasks and scoring rubrics. Four components were specified for the task design: cognitive processes, mathematical content, mode of representation, and task context. To reflect the construct domain of mathematical problem solving, reasoning, and communication, for example, a range of cognitive processes was specified, including discerning mathematical relations, using and discovering strategies and heuristics, formulating conjectures, and evaluating the reasonableness of answers. Performance tasks were then developed to assess one or more of these skills.

Test specifications need to clearly articulate the cognitive demands of the tasks, problem-solving skills, and strategies that can be employed and criteria to judge performance. This includes the specification of knowledge and strategies that are not only linked closely to the content domain but also are content domain independent (Baker, 2007). Carefully crafted and detailed test specifications are even more important for performance assessments than multiple-choice tests because there are fewer performance tasks and typically each is designed to measure something relatively unique (Haertel & Linn, 1996). The use of detailed test specifications can also help ensure that the content of the assessment is comparable across forms within a year and across years so as to support measurement of change over time. The performance tasks and scoring rubrics are then developed iteratively based on a well-delineated conceptual framework and test specifications (Lane & Stone, 2006).

The use of conceptual frameworks in designing performance assessments leads to assessments that are linked to educational outcomes and provide meaningful information to guide curriculum and instructional reform. For large-scale educational assessments, the conceptual framework is typically defined by content standards delineated at the state or national level. The grain size at which the content standards are specified affects whether narrow bits of information or broader, more contextualized understanding of the content domain will be assessed. This is because the content standards guide the development of the test specifications that include the content, cognitive processes and skills, and psychometric characteristics of the tasks.

Degree of Structure in Task Design

Underlying performance assessments is a continuum that represents different degrees of structure versus open-endedness in the response (Messick, 1996). The degree of structure for the problem posed and the response expected should be considered in the design of performance assessments. Baxter and Glaser (1998) characterized performance assessments along two continua with respect to their task demands. One continuum represents the task demand for cognitive processes, ranging from open to constrained, and the other continuum represents the task demand for content knowledge, from rich to lean. A task is process open if it promotes opportunities for students to develop their own procedures and strategies, and a task is content rich if it requires substantial content knowledge for successful performance. These two continua are crossed to form four quadrants so that tasks can be designed to fit one or more of these quadrants or considered along a progression within a content area.

This model allows cognitive and content targets to be clearly articulated in task design and for the evaluation of tasks in terms of their alignment with these targets (Baxter & Glazer, 1998). In the design of performance assessments that assess complex cognitive thinking skills, design efforts can be aimed primarily in the quadrant that reflects tasks that are process open and content rich; however, familiarity with these types of tasks in instruction and the age of the student need to be considered in design efforts.

Task Models

Task models, sometimes called templates or task shells, help ensure that the cognitive skills that are of interest are assessed. These models can enable the design of tasks that assess the same cognitive processes and skills, and a scoring rubric can then be designed for the tasks that can be generated from a particular task model. The use of task models for task design allows an explicit delineation of the cognitive skills to be assessed and can improve the generalizability of the score inferences.

Baker (2007) proposed a model-based assessment approach that uses task models. The major components of the model are the cognitive demands of the task, criteria to judge performance derived by competent performance, and a content map that describes the subject matter, including the interrelationships among concepts and the most salient features of the content. The cognitive demands of the tasks can then be represented in terms of families of tasks such as reasoning, problem solving, and knowledge representation tasks (Baker, 2007).

As an example, the explanation task model asks students to read one or more texts that require some prior knowledge of the subject domain, including concepts, principles, and declarative knowledge, in order to understand them and to evaluate and explain important issues introduced in the text (Niemi, Baker, & Sylvester, 2007). Following is a task from the explanation family that was developed for assessing student proficiency in Hawaii:

Imagine you are in a class that has been studying Hawaiian history. One of your friends, who is a new student in the class, has missed all the classes. Recently, your class began studying the Bayonet Constitution. Your friend is very interested in this topic and asks you to write an essay to explain everything that you have learned about it.

Write an essay explaining the most important ideas you want your friend to understand. Include what you have already learned in class about Hawaiian history and what you have learned from the texts you have just read. While you write, think about what Thurston and Liliuokalani said about the Bayonet Constitution, and what is shown in the other materials.

Your essay should be based on two major sources:

1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of Bayonet Constitution
2. What you have learned from the readings yesterday. (Niemi et al., 2007, p. 199)

Prior to receiving this task, students were required to read the primary source documents referred to in the prompt. This task requires students to not only make sense of the material from multiple sources, but to integrate material from these sources in their explanations. This is just one example of a task that can be generated from the explanation task model. Task models can also be used to design computer-based simulation tasks.

Design of Computer-Based Simulation Tasks

Computer-based simulations have made it possible to assess complex thinking skills that cannot be measured well by more traditional assessment methods. Using extended, integrated tasks, a large problem-solving space with various levels of complexity can be provided in an assessment (Vendlinski, Baker, & Niemi, 2008). Computer-based simulation tasks can assess students’ competency in formulating, testing, and evaluating hypotheses; selecting an appropriate solution strategy; and, when necessary, adapting strategies based on the degree to which a solution has been successful. An attractive feature of computer-based simulation tasks is that they can include some form of immediate feedback to the student based on the course of actions he or she takes.

Other important features of computer-based simulations are the variety of the types of interactions that a student has with tools in the problem-solving space and the monitoring and recording of how a student uses these tools (Vendlinski et al., 2008). Technology used in computer-based simulations allows assessments to provide more meaningful information by capturing students’ processes and strategies, as well as their products. Information on how a student arrived at an answer or conclusion can be valuable in guiding instruction and monitoring the progression of student learning (Bennett, Persky, Weiss, & Jenkins, 2007). The use of automated scoring procedures for evaluating student performances to computer-based simulation tasks can sometimes address the cost and time demands of human scoring.

Designing Assessments That Measure Learning Progressions

Some recent advances in assessment design efforts reflect learning progressions, defined as “descriptions of successively more sophisticated ways of reasoning within a content domain based on research syntheses and conceptual analyses” (Smith, Wiser, Anderson, & Krajcik, 2006, p. 1). Such progressions should be organized around central concepts or big ideas within a content domain.

Assessments that reflect learning progressions can identify where students are on a continuum of knowledge and skill development within a particular domain and the knowledge they need to acquire to become more competent. Empirically validated models of cognition and learning can be used to design assessments that monitor students’ learning as they develop understanding and competency in the content domain. These models of student cognition and learning across grade levels can be reflected in a coherent set of content standards across grade levels. This can help ensure the continuity of student assessment across grades, supporting monitoring of student understanding and competency and informing instruction and learning. Furthermore, this may lead to more meaningful scaling of assessments that span grade levels and thus more valid score interpretations regarding student growth.

An issue in the design of learning progressions is that there may be multiple paths to proficiency; however, some paths typically are followed by students more often than others (Bennett & Gitomer, 2009; National Research Council, 2006). Learning progressions that inform the design of assessments should be based on cognitive models of learning supplemented by teacher knowledge of student learning within content domains. Assessments can then be designed to elicit evidence that can support inferences about student achievement at different points along the learning progression (National Research Council, 2006).

Wilson and his colleagues have designed an assessment system that incorporates information from learning progressions and advances in both technology and measurement referred to as the BEAR Assessment System (Wilson, 2005; Wilson & Sloane, 2000). One application of this assessment system is for measuring a student’s progression for one of the three “big ideas” in the domain of chemistry. One is matter, which is concerned with describing molecular and atomic views of matter. The two other big ideas are change and stability, the former concerned with kinetic views of change and the conservation of matter during chemical change, and the latter concerned with the system of relationships in conservation of energy. Table 5.1 illustrates the construct map for the matter big idea for two of its substrands, visualizing and measuring.

Level 1 in the table is the lowest level of proficiency and reflects students’ lack of understanding of atomic views of matter, reflecting only their ability to describe some characteristics of matter, such as differentiating between a solid and a gas (Wilson, 2005). At level 2, students begin to use a definition or simple representation to interpret chemical phenomena, and at level 3 they begin to combine and relate patterns to account for chemical phenomena. Items are designed to reflect the differing achievement levels of the learning progression, or construct map, and empirical evidence is then collected to validate the construct map.

A task designed to assess the lower levels of the construct map depicted in table 5.1 asks students to explain why two solutions with the same molecular formula have two very different smells. The task presents students with the two solutions, butyric acid and ethyl acetate; their common molecular formula, C₄H₈O₄; and a pictorial representation depicting that one smells good and the other bad. The students are required to respond in writing to the following prompt: “Both of the solutions have the same molecular formulas, but butyric acid smells bad and putrid while ethyl acetate smells good and sweet. Explain why these two solutions smell different” (Wilson, 2005, p. 11).

By delineating the learning progressions within each of the “big ideas” of chemistry based on models of cognition and learning, assessments can be designed so as to provide evidence to support inferences about student competency at different achievement levels along the learning progressions. Performance assessments are well suited for capturing student understanding and thinking along these learning progressions. Smith and her colleagues (2006) proposed a learning progression around three key questions and six big ideas within the scientific topic of matter and atomic-molecular theory and provided examples of performance tasks that can assess different points along the continuum of understanding and inquiry within this domain.

BioKIDS Project

Learning progressions are also considered in the BioKIDS project, based on the Principled Assessment Designs for Inquiry (PADI) system. Within this system, three main design patterns for assessing scientific inquiry were identified: formulating scientific explanations from evidence, interpreting data, and making hypotheses and predictions (Gotwals & Songer, 2006).

Tasks based on a specific design pattern have many features in common. As an example, the Formulating Scientific Explanations from Evidence design pattern has two dimensions that are crossed: the level of inquiry skill required for the task and the level of content knowledge required for the task. This allows the design of assessment tasks in nine cells, each cell representing a task model. There are three inquiry skill steps:

• Students match relevant evidence to a given claim.
• Students choose a relevant claim and construct a simple explanation based on given evidence (construction is scaffolded).
• Students construct a claim and explanation that justifies the claim using relevant evidence (construction is unscaffolded). (Gotwals & Songer, 2006, p. 13)

The level of content knowledge required for the task is classified as simple, moderate, or complex, ranging from requiring minimal content knowledge and no interpretation to applying extra content knowledge and interpretation of evidence.

This is similar to Baxter and Glaser’s (1998) conceptualization of four quadrants that differ in terms of content richness and level of inquiry skills and then divides these two dimensions into nine quadrants. To better reflect scientific inquiry, Gotwals and Songer (2006) have proposed a matrix for each of the three design patterns (formulating scientific explanations from evidence, interpreting data, and making hypotheses and predictions).

In designing performance tasks, the amount of scaffolding needs to be considered. Scaffolding is a task feature that is manipulated explicitly in the BioKIDS design patterns (Gotwals & Songer, 2006; Mislevy & Haertel, 2006). For example, figure 5.1 presents two scientific inquiry assessment tasks that require scientific explanations from the BioKIDS project.

The first task requires more scaffolding than the second one. The amount of scaffolding built into a task depends on the age of the students and the extent to which they have had the opportunity in instruction to solve tasks that require explanations and complex reasoning skills. The first task in the figure represents the second step in the level of inquiry skills and the second level of content knowledge (moderate) in that evidence is provided (pictures of invertebrates that must be grouped together based on their characteristics), but the students need to choose a claim and construct the explanation. They must also interpret evidence or apply additional content knowledge, or both (need to know which characteristics are relevant for classifying animals).

The second task is at step 3 of the level of inquiry skill and the third level of content knowledge (complex): students need to construct a claim and an explanation that require the interpretation of evidence and application of additional content knowledge (Gotwals & Songer, 2006). More specifically, in this second task, “Students are provided a scenario, and they must construct (rather than choose) a claim and then, using their knowledge of food web interactions, provide evidence to back up their claim” (Gotwals & Songer, 2006, p. 16).

Review and Field Testing of Performance Assessments

Systems of appraisal are needed to evaluate the quality and comprehensiveness of the content and processes being assessed, as well as to guard against potential bias in task content, language, and context. The review process is an iterative one: when tasks are developed, they may be reviewed by experts and modified a number of times prior to and after being field-tested. This involves logical analyses of the tasks to help evaluate whether they are assessing the intended content and processes, worded clearly and concisely, and free from anticipated sources of bias. The development process also includes field-testing the tasks and scoring rubrics to ensure they elicit the processes and skills intended.

Scoring Performance Assessments

As in the design of performance tasks, the design of scoring rubrics is an iterative process and involves coordination across grades as well as across content domains to ensure a cohesive approach to student assessment (Lane & Stone, 2006). Much has been learned about the design of quality scoring rubrics for performance assessments.

First, it is critical that criteria embedded in rubrics are aligned to the processes and skills that are intended to be measured by the assessment tasks. Unfortunately, it is not uncommon for performance assessments to be accompanied by scoring rubrics that focus on lower levels of thinking rather than on the more complex reasoning and thinking skills that the tasks are intended to measure; therefore, the benefits of the performance tasks are not fully realized. Typically scoring rubrics should not be developed to be unique to specific tasks or generic to the entire construct domain; rather, they should be reflective of the “classes of tasks that the construct empirically generalizes or transfers to” (Messick, 1994, p. 17). Thus, a scoring rubric can be designed for a family of tasks or a particular task model. The underlying performance on a task is a continuum that represents different degrees of structure versus open-endedness in the response, and this needs to be considered in the design of the scoring rubric and criteria (Messick, 1996).

The design of scoring rubrics requires the specification of the criteria for judging the quality of performances, the choice of a scoring procedure (e.g., analytic or holistic), ways for developing criteria, and procedures used to apply the criteria (Clauser, 2000). The ways for developing criteria include the process used for specifying the criteria and who should be involved in developing them. For large-scale assessments in K–12 education, the scoring criteria typically are developed by a group of experts as defined by their knowledge of the content domain and experience as educators. Often these experienced educators have been involved in the design of the performance tasks and have knowledge of how students of differing levels of proficiency would perform on the task.

Criteria may also be developed by analyzing experts’ thinking and reasoning when solving tasks. Cognitive task analysis using experts’ talk-alouds (Ericsson & Smith, 1991) has been used to design performance tasks and scoring criteria in the medical domain (Mislevy et al., 2002). Features of experts’ thinking, knowledge, procedures, and problem posing are considered to be indicators of developing expertise in a domain (Glaser et al., 1987) and can be used systematically in the design of assessment tasks and scoring criteria.

Two ways in which the criteria can be applied rely on the use of trained raters and computer-automated scoring procedures (Clauser, 2000). The following sections discuss the specification of the criteria, different scoring procedures, research on scoring procedures, and computer-automated scoring systems.

Content Representativeness

An analysis of the relationship between the content of the assessment and the construct it is intended to measure provides important validity evidence. Test content refers to the skills, knowledge, and processes that are intended to be assessed by tasks, as well as the task formats and scoring procedures. Performance tasks can be designed so as to emulate the skills and processes reflected in the targeted construct.

Although the performance tasks may be assessing students’ understanding of some concepts or set of concepts at a deeper level, the content of the domain may not be well represented by a relatively small subset of performance tasks. This can be addressed by including other item formats that can appropriately assess certain skills and using performance tasks to assess complex thinking skills that cannot be assessed by the other item formats.

Methods are currently being investigated that will produce accurate student-level scores derived from mathematics and language arts performance assessments that are administered on different occasions throughout the year (Bennett & Gitomer, 2009). This will not only permit content representation across the performance assessments, but the assessments can be administered in closer proximity to the relevant instruction and information from any one administration can be used to inform future instructional efforts. If school-level scores are of primary interest, matrix-sampling procedures can be used to ensure content representation on the performance assessment, as was done on the MSPAP (Maryland State Board of Education, 1995).

The coherence among the assessment tasks, scoring rubrics and procedures, and the target domain, as well as their representativeness, are other aspects of validity evidence for score interpretations. It is important to ensure that the cognitive skills and content of the target domain are systematically represented in the tasks and scoring procedures. Both logical and empirical evidence can support the validity of the method used for transforming performance to a score.

For a performance demonstration, such as a major project that illustrates the ability to perform a type of inquiry, for example, we are not interested in generalizing the student performance on the demonstration to the broader domain, so the content domain does not need to be represented fully. The content and skills being assessed by the performance demonstration should be meaningful and relevant within the content domain. Performance demonstrations provide the opportunity for students to show what they know and can do on a real-world task, similar to a driver’s license test or the design of a scientific investigation.

Cognitive Complexity

One of the most attractive aspects of performance assessments is that they can be designed to assess complex thinking and problem-solving skills. As Linn and his colleagues (1991) have cautioned, however, it should not be assumed that a performance assessment measures complex thinking skills; evidence is needed to examine the extent to which tasks and scoring rubrics are capturing the intended cognitive skills and processes. The alignment between the cognitive processes underlying task responses and the construct domain needs to be made explicit, because typically the goal is to generalize interpretations of scores to the construct domain (Messick, 1989). The validity of the score interpretations will be affected by the extent to which the design of performance assessments is guided by cognitive theories of student achievement and learning within academic disciplines. Furthermore, the use of task models will support the explicit delineation of the cognitive skills required to perform particular task types.

Several methods have been used to examine whether tasks are assessing the intended cognitive skills and processes (Messick, 1989), and they are particularly appropriate for performance assessments that are designed to tap complex thinking skills. These methods include protocol analysis, analysis of reasons, and analysis of errors. In protocol analysis, students are asked to think aloud as they solve a problem or describe retrospectively how they solved the problem. In the analysis-of-reasons method, students are asked to provide rationales, typically written, to their responses to the tasks. The analysis-of-errors method requires an examination of procedures, concepts, or representations of the problems in order to make inferences about students’ misconceptions or errors in their understanding.

As an example, in the design of a science performance assessment, Shavelson and Ruiz-Primo (1998) used Baxter and Glaser’s 1998 analytic framework, which reflects a content-process space depicting the necessary content knowledge and process skills for successful performance. Using protocol analysis, Shavelson and Ruiz-Primo (1998) compared expert and novice reasoning on the science performance tasks that were content rich and process open. Their results from the protocol analysis confirmed some of their hypotheses regarding the different reasoning skills that tasks were intended to elicit from examinees. Furthermore, the results elucidated the complexity of experts’ reasoning as compared to the novices and informed the design of the tasks and interpretation of the scores.

Meaningfulness and Transparency

An important validity criterion for performance assessments is their meaningfulness (Linn et al., 1991), which refers to the extent to which students, teachers, and other interested parties find value in the tasks at hand. Meaningfulness is inherent in the idea that performance assessments are intended to measure more directly the types of reasoning and problem-solving skills that educators value. A related criterion is transparency (Frederiksen & Collins, 1989), that is, students and teacher need to know what is being assessed, by what methods, the criteria used to evaluate performance, and what constitutes quality performance. It is important to ensure that all students are familiar with the task format and scoring criteria for both large-scale and classroom assessments. Teachers can use performance tasks with their students and engage them in discussions about what the tasks are assessing and the nature of the criteria used for evaluating student work. Teachers can also engage students in using scoring rubrics to evaluate their own work and the work of their peers.

Generalizability of Score Inferences

For many large-scale assessments, the intent is to draw inferences about student achievement in the domain of interest based on scores derived from the assessment. While this is the case for some performance tasks, it is not the intent for performance demonstrations. There are other aspects of generalizability, however, that are relevant.

Generalizability theory provides both a conceptual and statistical framework to examine the extent to which scores derived from an assessment can be generalized to the domain of interest (Brennan, 1996, 2000, 2001; Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It is particularly relevant in evaluating performance assessments that assess complex thinking skills because it examines multiple sources of errors that can limit the generalizability of the scores, such as error due to tasks, raters, and occasions. Error due to tasks occurs because only a small number of tasks typically are included in a performance assessment. As Haertel and Linn (1996) explained, students’ individual reactions to specific tasks tend to average out on multiple-choice tests because of the relatively large number of items, but such individual reactions to specific items have more of an effect on scores from performance assessments that are composed of relatively few items. Thus, it is important to consider the sampling of tasks; by increasing the number of tasks on an assessment the validity and generalizability of the assessment results is enhanced. Furthermore, this concern with task specificity is consistent with research in cognition and learning that underscores the context-specific nature of problem solving and reasoning in subject matter domains (Greeno, 1989). The use of different item formats, including performance tasks, can improve the generalizability of the scores.

Error due to raters can also affect the generalizability of the scores in that raters may differ in their evaluation of the quality of students’ responses to a particular performance task and across performance tasks. Raters may differ in their overall leniency, or they may differ in their judgments about whether one student’s response is better than another’s response, resulting in an interaction between the student and rater facets (Hieronymus & Hoover, 1987; Lane, Liu, Ankenmann, & Stone, 1996; Shavelson, Baxter, & Gao, 1993). Typically the occasion is an important hidden source of error because performance assessments are given on only one occasion and the occasion is not typically considered in generalizability studies (Cronbach, Linn, Brennan, & Haertel, 1997).

Generalizability theory estimates variance components for the object of measurement (e.g., student, class, school) and for the sources of error in measurement, such as task and rater. The estimated variance components provide information about the relative contribution of each source of measurement error. The variance estimates are then used to design measurement procedures that enable more accurate score interpretations. For example, the researcher can examine the effects of increasing the number of items or number of raters on the generalizability of the scores. Generalizability coefficients estimate the extent to which the scores generalize to the larger construct domain for relative or absolute decisions, or both.

Generalizability studies have shown that error due to raters for science hands-on performance tasks (e.g., Shavelson et al., 1993) and mathematics-constructed response items (Lane, Liu, et al., 1996) tends to be smaller than for writing assessments (Dunbar, Koretz, & Hoover, 1991). To help achieve consistency among raters, attention is needed in the design of well-articulated scoring rubrics, selection and training of raters, and evaluation of rater performance prior to and throughout operational scoring of student responses (Lane & Stone, 2006).

Researchers have also shown that task-sampling variability is a greater source of measurement error than rater-sampling variability in students’ scores in science, mathematics, and writing performance assessments (Baxter et al., 1993; Gao, Shavelson, & Baxter, 1994; Hieronymus & Hoover, 1987; Lane, Liu, Ankenmann, & Stone, 1996; Shavelson et al., 1993). This indicates that students were responding differently across the performance tasks, whereas error due to raters was negligible. Thus, increasing the number of tasks in an assessment has a greater positive effect on the generalizability of the scores than increasing the number of raters scoring student responses.

Shavelson and his colleagues (Shavelson et al., 1993; Shavelson, Ruiz-Primo, & Wiley, 1999) found that the large task sampling variability in science performance assessments was due to variability in both the person × task interaction and the person × task × occasion interaction. They conducted a generalizability study using data from a science-performance assessment (Shavelson et al., 1993). The person × task variance component accounted for 32 percent of the total variability, whereas the person × task × occasion variance component accounted for 59 percent of the total variability. In this study and another, Shavelson and colleagues (1999) found that students performed differently on each task from occasion to occasion. Interestingly, the variance component for the person × occasion effect was close to zero, suggesting that “even though students approached the tasks differently each time they were tested, the aggregate level of their performance, averaged over the tasks, did not vary from one occasion to another” (Shavelson et al., 1999, pp. 64–65).

In sum, the results from generalizability studies indicate that scoring rubrics and training can be designed so as to minimize rater error. Increasing the number of performance tasks will increase the generalizability of the scores across tasks, and including other item formats on performance assessments will aid in the generalizability of scores to the broader content domain.

Fairness of Assessments

The evaluation of the fairness of an assessment is inherently related to all sources of validity evidence. Bias can be conceptualized “as differential validity of a given interpretation of a test score for any definable, relevant subgroup of test takers” (Cole & Moss, 1989, p. 205). A fair assessment therefore requires evidence to support the meaningfulness, appropriateness, and usefulness of the test score inferences for all relevant subgroups of examinees. Validity evidence for assessments that are intended for students from various cultural, ethnic, and linguistic backgrounds needs to be collected continuously and systematically as the assessment is being developed, administered, and refined. The linguistic demands on items can be simplified to help ensure that ELLs are able to access the task as well as other students.

As Abedi and Lord (2001) have demonstrated through their language modification approach, simplifying the linguistic demands on items can narrow the gap between ELLs and other students. The contexts used in mathematics tasks can be evaluated to ensure that they are familiar to various subgroups and will not have a negative effect on the performance on the task for one or more subgroups. The amount of writing required on mathematics, reading, and science assessments, for example, can be examined to help ensure that writing ability will not unduly influence the ability of the students to demonstrate what they know and can do on these assessments. Scoring rubrics can be designed to ensure that the relevant math, reading, or science skills, not students’ writing ability, are the focus. The use of other response formats, such as graphic organizers, on reading assessments may alleviate the concerns of writing ability confounding student performance on reading assessments (O’Reilly & Sheehan, 2009).

Although researchers have argued that performance assessments offer the potential for more equitable assessments, it is unlikely they could eliminate disparities in achievement among groups. As Linn and his colleagues (1991) note, differences among subgroups most likely occur because of differences in learning opportunities, familiarity, and motivation, and are not necessarily due to item format.

Research that has examined subgroup differences has focused on the impact of an assessment on subgroups by examining mean differences or differential group performance on individual items when groups are matched with respect to ability, that is, differential item functioning (DIF; Lane & Stone, 2006). The presence of DIF may suggest that inferences based on the test score may be less valid for a particular group or groups.

This could occur if tasks measure construct-irrelevant features that contribute to DIF. Gender or ethnic bias could be introduced by the typical contextualized nature of performance tasks or the amount of writing and reading required. In addition, the use of raters to score responses to performance assessments could introduce another possible source of differential item functioning (see, for example, Gyagenda & Engelhard, 2010). Results from DIF studies can be used to inform the design of assessment tasks and scoring rubrics so as to help minimize any potential bias.

Some researchers have supplemented differential item functioning methods with cognitive analyses of student performances designed to uncover reasons that items behave differently across subgroups of students of approximately equal ability. In a study to detect DIF in a mathematics performance assessment consisting of constructed-response items that required students to show their solution processes and explain their reasoning, using the analyses of reasons method. Lane, Wang, and Magone (1996) examined differences in students’ solution strategies, mathematical explanations, and mathematical errors as a potential source of differential item functioning. They reported that for items that exhibited DIF and favored females, females performed better than their matched males because females tended to provide more comprehensive conceptual explanations and displayed their solution strategies more fully. They suggest that increasing the opportunities in instruction for students to provide explanations and show their solution strategies may help alleviate these differences.

Ericikan (2002) examined differential item response performances among different language groups. In her research, she conducted linguistic comparisons across different language test versions to identify potential sources of DIF. Her results suggest that care in item writing is needed so as to minimize linguistic demands of items. As Wilson (2005) has suggested, the inclusion of DIF parameters into measurement models would allow a direct measurement of different construct effects such as using different solution strategies and different types of explanations or to capture linguistic differences.

Some research studies have shown both gender and ethnic mean differences on performance assessments that measure complex thinking skill. As an example, Gabrielson, Gordon, and Engelhard (1995) observed ethnic and gender differences in persuasive writing. Their results indicated that high school female students wrote higher-quality persuasive essays than male students, and white students wrote higher-rated essays than black students. The scores for conventions and sentence formation were more affected by gender and ethnic characteristics than the scores in content, organization, and style—which were consistent with results from Engelhard, Gordon, Walker, and Gabrielson (1994). These differences may be more reflective of differences in learning opportunities and home or community language use patterns than true differences in ability, suggesting the need for targeted instruction.

Studies have used advances in statistical models to examine subgroup differences so as to better control for student demographic and school-level variables. One study examined the extent to which the potentially heavy linguistic demands of a performance assessment might interfere with the performance of students who have English as a second language (Goldschmidt, Martinez, Niemi, & Baker, 2007). The results obtained by Goldschmidt and his colleagues (2007) revealed that subgroup differences on student written essays to a writing prompt were less affected by student background variables than a language arts commercially developed test consisting of multiple-choice items and some constructed-response items. The performance gaps between white students, English-only students, and traditionally disadvantaged students (e.g., ELLs) were smaller for the writing performance assessment than for the commercially developed test (Goldschmidt et al., 2007). Thus, in a context where students had opportunities in instruction to craft written essays, the performance of students on the writing assessment used in this study was stronger than for traditional selected- and constructed-response items.

Consequential Evidence

The evaluation of both intended and unintended consequences of any assessment is fundamental to the validation of score interpretation and use (Messick, 1989). Because a major goal of performance assessments is to improve teaching and student learning, it is essential to obtain evidence of any such positive consequences and any potentially negative consequences (Messick, 1994). As Linn (1993) stated, the need to obtain evidence about consequences is “especially compelling for performance-based assessments . . . because particular intended consequences are an explicit part of the assessment systems’ rationale” (p. 6). Furthermore, adverse consequences bearing on issues of fairness are particularly relevant because it should not be assumed that a contextualized performance task is equally appropriate for all students because “. . . contextual features that engage and motivate one student and facilitate his or her effective task performances may alienate and confuse another student and bias or distort task performance may alienate and confuse another student and bias or distort task performance” (Messick, 1994, p. 29).

This concern can be addressed by a thoughtful design process in which fairness issues are addressed, including expert analyses of the tasks and rubrics, as well as analyses of student thinking as they solve performance tasks with special attention to examining potential subgroup differences and features of tasks that may contribute to these differences.

Large-scale performance assessments that measure complex thinking skills have been shown to have a positive impact on instruction and student learning (Lane, Parke, & Stone, 2002; Stecher, Barron, Chun, & Ross, 2000; Stein & Lane, 1996; Stone & Lane, 2003. See also chapters 1 to 3, this volume). In a study examining the consequences of Washington’s state assessment, Stecher and his colleagues (2000) indicated that approximately two-thirds of fourth- and seventh-grade teachers reported that the state standards and the state assessment short-answer and extended-response items were influential in promoting better instruction and student learning.

An important aspect of consequential evidence for performance assessments is the examination of the relationship between changes in instructional practice and improved student performance on the assessments. A series of studies examined the relationship between changes in instructional practice and improved performance on the MSPAP, which consisted entirely of performance tasks that were integrated across content domains (Lane et al., 2002; Parke, Lane, & Stone, 2006; Stone & Lane, 2003). The results revealed that teachers’ reports about features of their instruction accounted for differences in school performance on MSPAP in reading, writing, mathematics, and science—with higher performance among schools with more reform-oriented practices. Furthermore, increases in these practices over time accounted for differences in the rate of change in MSPAP school performance in reading and writing over a five-year period. Furthermore, Linn, Baker, and Betebenner (2002) demonstrated that the slopes of the trend lines for the math assessments on both NAEP and MSPAP were similar, suggesting that the performance gains in Maryland were due to deepened mathematical understanding on the part of the students rather than merely teaching to the content and format of the specific state test.

When using test scores to make inferences regarding the quality of education, contextual information is needed to inform the inferences and actions (Haertel, 1999). Stone and Lane (2003) indicated, for example, that whereas the percentage of students receiving free or reduced lunch (a proxy for socioeconomic) was significantly related to school-level performance on MSPAP in all areas, there was no significant relationship between this measure and school-level growth on MSPAP in mathematics, writing, science, and social studies.

Instructional Sensitivity

An assessment concept that can help inform the consequential aspect of validity is instructional sensitivity, which refers to the degree to which tasks are sensitive to improvements in instruction (Popham, 2003; Black & Wiliam, 2007). Performance assessments are considered to be vehicles that can help shape sound instructional practice by modeling to teachers what is important to teach and to students what is important to learn. In this regard, it is important to evaluate the extent to which improved performance on an assessment is linked to improved instructional practices. To accomplish this, the assessments need to be sensitive to instructional improvements. Assessments that may not be sensitive to well-designed instruction may be measuring something other than instruction, such as irrelevant domain constructs or learning that may occur outside of the school.

Two methods have been used to examine whether assessments are instructionally sensitive: studies have either examined whether students have had the opportunity to learn the material, or they have examined the extent to which differences in instruction affect performance on the assessment. For example, study using a model-based approach to assessment design Baker (2007) found that student performance on a language arts performance assessment was sensitive to different types of language instruction and captured improvement in instruction (Niemi, Wang, Steinberg, Baker, & Wang, 2007).

This study examined the effects of three types of instruction (literary analysis, organization of writing, and teacher-selected instruction) on student responses to an essay about conflict in literary work. The results indicated that students who received instruction on literary analysis were significantly better able to analyze and describe conflict in literature than students in the other two instructional groups and students who had direct instruction on organization of writing performed significantly better on measures of writing coherency and organization. These results provide evidence that performance assessments can be instructionally sensitive with respect to different types of instruction and suggest the need to ensure alignment and coherency among curriculum, instruction, and assessment.

Measurement Models and Performance Assessments

Item response theory (IRT) models are typically used to scale assessments that consist of performance tasks only and assessments that consist of both performance tasks and multiple-choice items. IRT involves a class of mathematical models that are used to estimate test performance based on characteristics of the items and characteristics of the examinees that are presumed to underlie performance. The models use one or more ability parameters and various item parameters to predict item responses (Embretson & Reise, 2000; Hambleton & Swaminathan, 1985). These parameters, in conjunction with a mathematical function, are used to model the probability of a score response as a function of ability.

The more commonly applied models assume one underlying ability dimension determines item performance (Allen & Yen, 1979) and accommodate ordinal response scales that are typical of performance assessments. They include the graded-response model (Samejima, 1969, 1996), the partial-credit model (Masters, 1982), and the generalized partial-credit model (Muraki, 1992). As an example, Lane, Stone, Ankenmann, and Liu (1995) demonstrated the use of the graded-response model with a mathematics performance assessment, and Allen, Johnson, Mislevy, and Thomas (1994) discussed the application of the generalized, partial-credit model to NAEP, which consists of both selected- and constructed-response items.

Performance assessment data may be best modeled by multidimensional item response theory (MIRT), which allows the estimation of student proficiency on more than one skill area (Reckase, 1997). The application of MIRT models to assessments that are intended to measure student proficiency on multiple skills can provide a set of scores that profile student proficiency across the skills. These scores can then be used to guide the instructional process so as to narrow gaps in student understanding. Profiles can be developed at the student level or the group level (e.g., class) to inform instruction and student learning. MIRT models are particularly relevant for performance assessments because these assessments can capture complex performances that draw on multiple dimensions within the content domain, such as procedural skills, conceptual skills, and reasoning skills.

Modeling Rater Effects Using IRT Models

Performance assessments require either human scorers or automated scoring procedures in evaluating student work. When human scorers are used, performance assessments are considered to be “rater mediated,” since information is mediated through interpretations by raters (Engelhard, 2002). Engelhard (2002) provided a conceptual model for performance assessments in which the obtained score is a function not only of the domain of interest (e.g., writing ability), but also of rater severity, difficulty of the task, and the structure of the rating scale (e.g., analytic versus holistic, number of score levels). Test developers exert control over the task difficulty and the nature of the rating scale; however, other potential sources of construct-irrelevant variance are introduced by the raters, including differential interpretation of score scales, halo effects, and bias (Engelhard, 2002).

Models have been developed that account for rater variability in scoring performance assessments. As an example, Patz and his colleagues (Patz, 1996; Patz, Junker, Johnson, & Mariano, 2002) developed a hierarchical rating model to account for the dependencies between rater judgments. A parameter was introduced into the model that could be considered an “ideal rating” or expected score for an individual, and raters could vary with respect to how close their rating is to this ideal rating. This variability reflects random error (e.g., lack of consistency) as well as systematic error (e.g., rater tendencies such as leniency). According to Bejar et al. (2006), this modeling of rater variability may reflect an accurate modeling of rater cognition in that under operational scoring conditions, raters may try to predict the score an expert rater would assign based on the scoring rubric and benchmark papers. In addition, covariates can be introduced into the model to predict rater behaviors such as rater features (e.g., hours of scoring) and item features. The modeling of rater variability is a way to account for error in the scores obtained when evaluating performance assessments and enables more valid interpretations of the scores.

Equating and Linking Issues

Equating helps ensure comparability of interpretations of assessment results from assessment forms administered at one time or over time; however, equating an assessment that consists of only performance tasks is complex (Kolen & Brennan, 2004). One way to equate forms so that they can be used interchangeably is to have a common set of items, typically called anchor items, on each of the forms. The anchor items are then used to adjust for any differences in difficulty across forms. An important issue that needs to be addressed in using performance tasks as anchor items in the equating procedure is that rater teams could change their scoring standards over time and the application of standard equating practices would lead to bias in the equating process and, consequently, inaccurate scores (Bock, 1995; Kim, Walker, & McHale, 2008a; Tate, 1999). As a solution to this problem, Tate (1999, 2000) suggested an initial linking study in which any changes in rater severity and discrimination across years could be identified.

To accomplish the equating, a large representative sample of anchor item papers (i.e., trend papers) from year 1 are rescored by raters in year 2. These raters in year 2 are the same raters who score the new constructed responses in year 2. These trend papers now have a set of scores from the old raters in year 1 and a set of scores from the new raters in year 2. This allows examining the extent to which the two rater teams across years differ in severity in assigning scores, and then adjustments can be made to ensure the two tests are on the same scale. Tate contends that instead of having item parameters, there are item and rating team parameters that reflect the notion that if the rating team changes across years, any differences due to the change in rating teams will be reflected in the item parameters. Another way to conceptualize this is that the item parameters are confounded by rater-team effects so the rating team needs to be considered in the equating.

The effectiveness of this IRT linking method using trend score papers was established by Tate and his colleague (Tate, 2003; Kamata & Tate, 2005). The use of trend score papers in non-IRT equating methods has also proven effective by Kim, Walker, and McHale (2008a, 2008b). They compared the effectiveness of equating for a design that required anchor items and a design that did not require anchor items with and without trend score papers. The design that does not incorporate anchor items alleviates the concern of content representativeness of anchor items. Their results indicated that both designs using trend score papers were more effective in equating the scores as compared to designs that did not use the trend score papers.

More important, their results suggest that changes in rater severity can be examined and the equating of test forms across years should adjust for differences in rater severity if the trend scoring indicates that a rater shift has occurred (Kim et al., 2008a, 2008b). Trend scoring should be implemented for any assessment program that uses constructed-response items for equating across years to control for equating bias caused by a scoring shift over years. Kim and colleagues (2008b) point out that the trend-scoring method requires additional rating of student papers, which increases cost, and it may be a bit cumbersome to implement. The use of image and online scoring methods, however, can ease the complexities of the implementation of the trend-scoring method.