Constructed Response versus Selected Response

For many educators, performance assessment is most easily defined by what it is not; specifically, it is not multiple-choice testing. In performance assessment, rather than choosing among predetermined options, the examinee must either construct or supply an answer, produce a product, or perform an activity (Madaus & O’Dwyer, 1999). From this perspective, performance assessment encompasses a wide range of activities, from completing a sentence with a few words (short answer), to writing a thorough analysis (essay), to conducting a laboratory investigation and writing a descriptive analysis of the process (hands-on). Given this range, it is surprising how often people make general claims about performance assessment without differentiating among types.

Elements of Performance Assessment

In the literature, different authors use the term performance assessment to mean different things. Some emphasize the cognitive processes demanded of the students, some the format of the desired response, and others the nature and content of the actual response (Palm, 2008). These differences in emphasis underscore one of the lingering problems facing performance assessment, which is that different educators and policymakers have different implicit meanings for the term.

For the purposes of this chapter, I define a performance assessment primarily in terms of the performances required of test takers involved in large-scale assessments. A performance task is a structured situation in which stimulus materials and a request for information or action are presented to an individual, who generates a response that can be rated for quality using explicit standards. The standards may apply to the final product or the process of creating it. A performance assessment is a collection of performance tasks.

This definition has four important elements. First, each task must occur in a structured situation, meaning the task is constrained with respect to time, space, access to materials, and so on. The standardized structure makes it possible to replicate the conditions, so the same assessment can be administered to different people and their performances can be compared. The requirement that there be structure with respect to administrative conditions does not exclude from consideration complex, extended tasks, such as conducting a scientific experiment and reporting the results; instead, structure ensures that tasks can be replicated at different times and in different places. The requirement for structure does exclude from consideration a number of familiar incidents of assessment, including oral examinations in which the examiner asks a different question of each student; ratings of behaviors in natural settings, such as ratings of athletes during competition (Nadeau, Richard, & Godbout, 2008); or observations of students’ literacy behaviors (Meisels, Xue, & Shamblott, 2008); and assessments that attempt to compare change in performance over time (Wentworth et al., 2009).

Second, each performance task contains some kind of stimulus material or information that serves as the basis for the response. In this respect, performance tasks can be very similar to multiple-choice items. They might begin with a mathematical problem, a text passage, a graph, or a figure. However, because the response options are unconstrained, the stimulus material does not have to lead to four specific choices. That freedom permits performance tasks to include more varied, complex, and novel stimuli than are typically used in multiple-choice assessments.²

Third, the task must have directions indicating the nature of the desired response. The directions can be part of the stimulus materials (e.g., “How many 3-inch by 4-inch tiles would it take to cover a circular dance floor 8 feet in diameter?”) or separate from them (e.g., “Read these descriptions of the Battle of the Bulge written by soldiers on opposing sides and write an essay to support or refute the claim that the Allied and Axis soldiers were more similar than different”). The key feature of these directions is that they be explicit enough so two test takers would have similar understanding of what they were asked to do. Because the responses can be much broader than the responses for multiple-choice tests, vague queries (e.g., “What do you think about that?”) that can be widely interpreted must be avoided.

Fourth, the task must prompt responses that can be scored according to a clear set of standards. It is usually the case that the standards are fully developed before the task is given. If the task developer does not know what constitutes a good response, it is unlikely that the task is measuring something clearly enough to be useful. In some cases, however, the scoring rubrics will need to be elaborated on the basis of responses received. For example, students may come up with legitimate responses the developers did not anticipate.

Classifying Based on Stimulus Materials and Response Options

I suggest a two-way classification scheme based on the structural characteristics of the task, particularly the nature of the stimulus materials and the nature of the response options. (This scheme is inspired by the work of Baxter & Glaser, 1998, discussed subsequently.) The stimulus materials can be classified in terms of complexity along a dimension that runs from simple to complex. (See table 2.1.)

A math task that asks the student to solve an equation for x represents a relatively simple stimulus. In contrast, a language arts task that asks the student to read an essay and a poem and to look at a reproduction of a painting as the basis for a comparative task presents a relatively complex set of stimulus materials. Similarly, the response options can be classified in terms of freedom along a dimension that runs from constrained to open. A short-answer history question requiring the test taker to fill in a short phrase to correctly place an event offers a relatively constrained response space. In comparison, a science task in which students are given a set of leaves to observe and are asked to create at least two different classification schemes and arrange the leaves into groups based on each scheme offers a relatively open range of responses.

By crossing the stimulus and response dimensions, we create four quadrants that can be used to classify all performance tasks. A written, short-answer (fill-in-the-blank) question is an example of a relatively simple, relatively constrained task. A math word problem that requires setting up equations, using a graphing calculator, and other calculations is an example of a relatively simple, relatively open task. Such tasks can also be found in many international examinations, such as the A-level examinations used in England, in which relatively simple prompts require extended, open-ended responses demonstrating the application of knowledge in real-world settings (see Darling-Hammond, chapter 4, this volume).

The language arts task in which an essay, a poem, and a piece of art serve as prompts for a comparative essay on the similarities and differences in the artists’ visions of the subject is an example of a relatively complex, relatively open task. The College and Work Readiness Assessment is another example; it includes complex, open tasks in which students must review a set of materials that might include graphs, interviews, testimonials, or memoranda; extract information from each; assess the credibility of the information; and synthesize it to respond to problem situation (Hersh, 2009). An interesting feature of this assessment is that it can be administered online with both prompts and responses transmitted over the Internet.

At this point, a savvy reader should object that the stimulus-response classification scheme focuses on surface features of the task and ignores more important cognitive and performance aspects of the activity. I agree and discuss these aspects below. Yet this simple classification will be useful when thinking about some of the practical aspects of performance assessment, including feasibility, burden, and costs.

Classifying Based on Content Knowledge and Process Skills

Baxter and Glaser (1998) suggest a way to classify science performance tasks by their cognitive complexity, and this approach could be used more generally. They divide the science assessment space into four quadrants depending on whether the process skills demanded are open or constrained and whether the content knowledge demanded is lean or rich. They provide examples of science tasks corresponding to each quadrant. For example, “Exploring the Maple Copter” is an example of a content-rich and open-process task. In this task, high school physics students are asked to design and conduct experiments with a maple seed and develop an explanation of its flight for someone who does not know physics. The two-way content-process classification is helpful for characterizing the cognitive complexity of performance tasks. This distinction is useful when thinking about the inferences that are appropriate to make from scores on performance assessments and the kinds of information that would be needed to validate those inferences.

Classifying Based on Subject Field

The examples I have given suggest that subject field may be another useful way to classify performance tasks. Expertise in one subject is demonstrated very differently from expertise in another, and performance assessment lets these distinctive styles of thinking and performing come to the fore. For example, “doing science” involves observing events, designing experiments, imposing controls, collecting information, analyzing data, and building theories, and rich performance tasks in the sciences incorporate these kinds of skills and behaviors. In contrast, “doing mathematics” is somewhat more abstract. Although mathematics begins with observations of objects in space, the discipline focuses more on manipulating numbers, building facility with operations, translating situations into representations, explicating assumptions, and testing theories. Performance assessment in mathematics is more likely to involve solving problems with pencil, paper, and calculators, representing relationships in different forms such as graphs, and so on.

Similarly, performance assessment in the arts involves performing—music, dance, drawing, and more. In language arts, performance assessments are likely to focus on the written word: reading; comprehending; interpreting; comparing; and producing text that describes, explains, and persuades, for example. Because of these disciplinary ways of thinking and acting, educators in different fields may be thinking about very different kinds of activities when they refer to performance assessments.

Finally, some performance assessments are designed to be less discipline bound. For example, the Collegiate Learning Assessment includes tasks that are designed to measure the integration of skills across disciplines (Klein, Benjamin, Shavelson, & Bolus, 2007). Similarly, Queensland, Australia, has developed a bank of rich tasks that call for students to demonstrate “transdisciplinary learnings” (Darling-Hammond & Wentworth, 2010; see chapter 4 in this volume). Other performance tasks are designed to measure transferable reasoning skills that can be applied in many contexts.

Portfolio Assessments

A portfolio is a collection of work, often with personal commentary or self-analysis, assembled by an individual over time as a cumulative record of accomplishment. In most cases, the individual selects the work that goes into the portfolio, making each collection unique. This was the case with the Vermont and Kentucky portfolio assessments in the early 1990s (described later in this chapter). Although both became more standardized in the definition of tasks over time, students and teachers could choose the work within each genre requested to place in the portfolio that would be scored. This is also the case with most of the currently popular portfolios in teacher education programs.

According to the definition of performance assessment I gave above, nonstandardized individual collections of work are not performance assessments because each portfolio contains different performance tasks. One student might include a persuasive essay, while another includes a set of poetry. One might include the solution to a set of math word problems that are completely omitted from the portfolio of another. It is not just narrow-mindedness that leads us to exclude free-choice portfolios from the realm of performance assessment; rather, the lack of standardization undermines the value of such collections of work as assessment tools.

The initial Vermont experience is a case in point. Teachers and students jointly selected student work to include in each student’s mathematics and writing portfolios. Thus, two portfolios selected from a given teacher’s class contained some common pieces and some unique choices. This variation was exacerbated across teachers and made the portfolios difficult to score reliably (Koretz, Stecher, Klein, & McCaffrey, 1994). Studies have reported much higher rates of scoring consistency for more standardized portfolios featuring common task expectations and analytical rubrics, like those that evolved in Vermont and were ultimately developed in Kentucky (Measured Progress, 2009). These portfolios evolved rules about how the tasks were produced to ensure additional commonality and that they could be verified as the work of a particular student.

The Appeal of Performance Assessment

The use of performance assessment can be traced back at least two millennia to the Han dynasty in China, and its history makes fascinating reading (Madaus & O’Dwyer, 1999). For our purposes, it is sufficient to look back two or three decades to the educational reform movement of the 1980s. This period was marked by an increased use of standardized tests for the purposes of accountability with consequences for schools and students (Hamilton & Koretz, 2002). Minimum competency testing programs in the states gave way to accountability systems in which test results variously determined student grade-to-grade promotion or graduation, teacher pay, and school rankings, financial rewards, or interventions. In this period, educators also began to subscribe to the idea that tests could be used to drive educational reform. The term measurement-driven instruction was coined to describe the purposeful use of high-stakes testing to try to change school and classroom behaviors (Popham, Cruse, Rankin, Sandifer, & Williams, 1985).

By the end of the decade, educators began to recognize a number of problems associated with high-stakes multiple-choice testing, including degraded instruction, for example, narrowing of the curriculum to tested topics and excessive class time devoted to test preparation, that led to inflated test scores (Hamilton & Koretz, 2002). They worried about persistent differences in performance between demographic groups, which many incorrectly attributed to the multiple-choice testing format. There were also pointed criticisms from specific content fields. Science educators, for example, complained that multiple-choice tests emphasized factual rather than procedural knowledge (Frederiksen, 1984).

Not wanting to give up the power of measurement-driven instruction to shape teacher and student behavior, many educators began to call for a new generation of “tests worth teaching to.” Under the banner of “what you test is what you get,” translated into the catchy acronym WYTIWYG (Resnick & Resnick, 1992), advocates of performance assessment thought they could bring about improvements in curriculum, instruction, and outcomes by incorporating more performance assessments into state testing programs. They also recognized that performance assessments could more easily be designed to tap higher-order skills, including problem solving and critical thinking (Raizen et al., 1989). This enthusiasm led many states to incorporate forms of performance assessment into their large-scale testing programs. We briefly summarize some of the more notable efforts in the following sections.

Vermont Portfolio Assessment Program

Educators in Vermont began to develop the Vermont Portfolio Assessment Program in 1988. They had twin goals: to provide high-quality data about student achievement (sufficient to permit comparisons of schools or districts) and to improve instruction. The centerpiece of the program was portfolios of student work in writing and mathematics collected jointly by students and teachers over the course of the school year. Teachers and students had nearly unconstrained choice in selecting tasks to be included in the portfolios. In writing, students were expected to identify a single best piece and a number of other pieces of specified types. In mathematics, students and teachers reviewed each student’s work and submitted the five to seven best pieces. The portfolios were complemented by on-demand “uniform tests” in writing (a single, standardized prompt) and mathematics (primarily multiple choice). The program was implemented in grades 4 and 8 as a pilot in 1990–1991 and statewide in 1991–1992 and 1992–1993. Early evaluation studies, however, raised concerns with the reliability of the scoring and the overall validity of the portfolio system (Koretz, Klein, McCaffrey, & Stecher, 1993).

As the tasks were more standardized, reliability did begin to increase (Koretz et al., 1994). In the late 1990s, though, the portfolio assessment program was replaced for accountability purposes with the New Standards Reference Exam (Rohten, Carnoy, Chabran, & Elmore, 2003), which included some on-demand performance tasks but not portfolios. Most local districts continued to use the portfolios for their own purposes, but they are no longer used for state-level reporting. More recently, Vermont joined with New Hampshire and Rhode Island to develop the New England Common Assessment Program, which includes multiple-choice and short constructed-response items in reading, math, and science, as well as a writing assessment. (See chapter 3 for more on this program and Vermont’s current assessment program.)

In addition, as part of the Vermont Developmental Reading Assessment (DRA) for second grade, students are asked to read short books and retell the story in their own words.³ Teachers score the student’s oral reading for accuracy and his or her retelling for comprehension. To ensure reliability, teachers who administer the assessment first have their scoring calibrated through an online process. The results of the DRA are reviewed annually at the Summer Auditing Institute. Note that the Vermont accountability system does not have high stakes for students; student promotion and high school graduation do not depend on test scores (Rohten et al., 2003).

Kentucky Instructional Results Information System

In response to a 1989 decision by the Kentucky Supreme Court declaring the state’s education system to be unconstitutional, the state legislature passed the Kentucky Education Reform Act of 1990. This law brought about sweeping changes to Kentucky’s public school system, including changes to school and district accountability for student performance. The Kentucky Instructional Results Information System (KIRIS) was a performance-based assessment system implemented for the first time in the spring of 1992.⁴ KIRIS tested students in grades 4, 8, and 11 in a three-part assessment that included multiple-choice and short-essay questions, performance “events” requiring students to solve practical and applied problems, and portfolios in writing and mathematics in which students presented the “best” examples of their classroom work collected throughout the school year. Students were assessed in seven areas: reading, writing, social science, science, mathematics, arts and humanities, and practical living/vocational studies (US Department of Education, 1995).

KIRIS was designed as a school-level accountability system, and schools received rewards or sanctions based on the aggregate performance of all their students.⁵ School ratings were based on a combination of cognitive and noncognitive indicators (including dropout rates, retention rates, and attendance rates). A school accountability index combined cognitive and noncognitive indicators and was reported in biennial cycles. Schools were expected to have all students at the proficient level, on average, within twenty years, and their annual improvement target was based on a straight-line projection toward this goal. Every two years, schools that exceeded their improvement goals received funds that could be used for salary bonuses, professional development, or as school improvement funds. In 1994–1995, about $26 million was awarded, with awards of about $2,000 per teacher in eligible schools. The state also devoted resources to support and improve low-performing schools, including assigning “distinguished educators” to advise on school operations.

Concerns raised by researchers (Hambleton, Jaeger, Koretz, Linn, Millman, & Phillips, 1995; Catterall, Mehrens, Flores, & Rubin, 1998) and the public (Fenster, 1996) about some aspects of the assessments contributed to changes in the program. In 1998, the Kentucky legislature replaced KIRIS with the Commonwealth Accountability Testing System (White, 1999), which incorporated some of the components (performance tasks, the writing portfolio) from KIRIS but replaced the mathematics portfolios with more structured performance tasks. Many factors contributed to this decision, including philosophical disagreements over the “valued outcomes” adopted for education, disputes about the correct way to teach mathematics and literacy, and a switch in the political balance in the legislature (Gong, 2009). More recently, Kentucky switched to a criterion-referenced test for NCLB reporting, the Kentucky Core Content Test (KCCT) for math (grades 3 through 8 and 11), English language arts (grades 3 through 8 and 10), and sciences (grades 4, 7, and 11). These include a fifty-fifty weighting of multiple-choice and constructed response items.

An on-demand writing assessment is part of the KCCT and provides students in grades 5 and 8 with the choice of two writing tasks that include a narrative writing prompt and a persuasive writing prompt; students in grade 12 are given one common writing task and the choice of one of two additional writing tasks (Kentucky Department of Education, 2009). Until 2012, the KCCT assessed student achievement in writing using the writing portfolio in grades 4, 7, and 12 and an on-demand writing assessment in grades 5, 8, and 12. A four-piece portfolio was required in grade 12 and a three-piece portfolio in grades 4 and 7. The required content includes samples of reflective writing, personal expressive writing/literary writing, transactive writing, and (in grade 12 only) transactive writing with an analytical or technical focus. The writing portfolio was discontinued in 2012, while the on-demand writing assessment remains in place. As of this writing, the Kentucky Department of Education was deliberating about what kind of performance assessment will take the place of the portfolio.

Maryland School Performance Assessment Program

The Maryland School Performance Assessment Program (MSPAP) was created in the late 1980s and early 1990s to assess progress toward the state’s educational reform goals. The MSPAP, first administered in 1991, assessed reading, writing, language use, mathematics, science, and social science in grades 3, 5, and 8. All of the tasks were performance based, ranging from short-answer responses to more complex, multistage responses to data, experiences, or text. Human raters scored all responses.

MSPAP tasks were innovative in several ways. Activities frequently integrated skills from several subject areas, some tasks were administered as group activities, some were hands-on tasks involving the use of equipment, and some tasks had preassessment activities that were not scored. MSPAP items were matrix sampled: every student took a portion of the exam in each subject. As a result, there was insufficient representation of content on each test form to permit reporting of student-level scores. MSPAP was designed to measure school performance, and standards-based scores (percentage achieving various levels) were reported at the school and district levels. Schools were rewarded or sanctioned depending on their performance on the MSPAP (Pearson, Calfee, Walker Webb, & Fleischer, 2002).

Many of the features of MSPAP were unusual for state testing programs, and some stakeholders raised concerns about the quality of its school results. A technical review committee commissioned by the Abell Foundation in 2000 reported generally positive findings with respect to the psychometric aspects of MSPAP (Hambleton, Impara, Mehrens, & Plake, 2000). The foundation also suggested changes to remove some aspects of the program, like the group-based, preassessment activities. The technical review committee criticized the content of the tests, and some objected to the Maryland Learning Outcomes on which the test was based (Ferrara, 2009). According to the Washington Post (Shulte, 2002), MSPAP school-level scores fluctuated widely from year to year, leading the superintendent of one of Maryland’s largest districts to demand the delay of the release of the test scores until the fluctuations could be explained. Partially due to concerns with scoring and partially due to a desire (and an NCLB requirement) to have individual student scores, the Maryland School Assessment replaced MSPAP in 2002 (Hoff, 2002). The Maryland School Assessment tests reading and mathematics in grades 3 through 8 and science in grades 5 and 8 using both multiple-choice and brief constructed-response items.

Washington Assessment of Student Learning

In 1993, the Washington legislature passed the Basic Education Reform Act, including the Essential Academic Learning Requirements for Washington State students. The learning requirements defined learning goals in reading; writing; communication; mathematics; social, physical, and life sciences; civics and history; geography; arts; and health and fitness. The Washington Assessment of Student Learning (WASL) was developed to assess student mastery of these standards. WASL included a combination of multiple-choice, short-answer, essay, and problem-solving tasks. In addition, the system included classroom-based assessments in subjects not included in WASL.

WASL was implemented in fourth grade in 1996 and in other grades subsequently. Eventually it was administered in reading (grades 3 through 8 and 10), writing (grades 4, 7, and 10), mathematics (grades 3 through 8 and 10), and science (grades 5, 8, and 10). Initially listening was assessed as part of the WASL, but this test was discontinued in 2004 as part of a legislative package of changes in anticipation of WASL’s use as the high school exit exam starting with the class of 2008.

The use of WASL as the state’s high school exit exam was controversial because of the low pass rates of tenth graders, especially in mathematics (Queary, 2004). Other concerns included considerable variation in student performance (percent proficient) in reading and mathematics from year to year, and even greater variation at the strand level (Washington State Institute for Public Policy, 2006). In 2007, the governor delayed the use of the math and science sections, and in 2008 he mandated that scores for the math portion of the WASL not be used.

The WASL was replaced in 2009–2010 with the Measurements of Student Progress (MSP) in grades 3 through 8 and the High School Proficiency Exam (HSPE) in grades 10 through 12. The MSP and HSPE tests include multiple-choice and short-answer questions; the essay questions have been eliminated from the reading, math, and science tests.

Interestingly, Washington uses classroom-based assessments, including performance assessments, to gauge student understanding of the EALR learning standards in social studies, the arts, and health and fitness. Districts must report to the state that they are implementing the assessments and strategies in those content areas, but individual student scores are not reported. (For examples of some of these assessments, see appendix A.)

California Learning Assessment System

The California Learning Assessment System (CLAS) was designed in 1991 to align the testing program with the state’s curricular content, measure students’ attainment of that content using performance-based assessment, and provide performance assessments for both students and schools (Kirst & Mazzeo, 1996). First administered in 1993, CLAS assessed students’ learning abilities in reading, writing, and mathematics in grades 4, 8, and 10. In reading and writing, CLAS used group activities, essays, and short stories to measure students’ critical thinking. In math, students were asked to show how they arrived at their answers. The performance assessment was based not only on the annual exams but also on portfolios of student work.

Controversy over CLAS arose shortly after the first round of testing, when some school groups and parents claimed that the test items were too subjective, they encouraged children to think about controversial topics, or they asked about the students’ feelings, which some parents said was a violation of their student’s civil rights (McDonnell, 2004; Kirst & Mazzeo, 1996). In addition, the debate in California highlighted fundamental conflict about the role of assessment in education, with policymakers, testing experts, and the public often voicing very different expectations and standards of judgment (McDonnell, 1994). The California Department of Education did not help matters when it initially declined to release sample items from the exams, citing the cost of developing new items. A series of newspaper articles and state-level committee reports were critical of the test’s sampling procedures and of the objectivity of the scoring. In 1994, the legislature reauthorized CLAS in a bill that increased the number of multiple-choice and short-answer questions to complement the performance tasks, but this change came too late to save the program; CLAS was administered for the last time later that year.

After a four-year hiatus from statewide achievement testing, the Standardized Testing and Reporting (STAR) exams began in 1998. STAR uses multiple-choice questions to measure the achievement of California content standards in English language arts, mathematics, science, and history/social science (in grades 2 through 11). Initially the STAR program used the Stanford Achievement Test (ninth edition); however, beginning in 2001 the state began to substitute the California Standards Tests, which are largely multiple-choice tests aligned to the California standards, with a writing component at specific grade levels.

NAEP Higher-Order Thinking Skills Assessment Pilot

In 1985–1986, the National Science Foundation funded the National Assessment of Educational Progress (NAEP) to conduct a pilot test of techniques to study higher-order thinking skills in mathematics and science. Adapting tasks that had been used in the United Kingdom, NAEP developed prototype assessment activities in a variety of formats, including pencil-and-paper tasks, demonstrations, computer-administered tasks, and hands-on tasks. In all, thirty tasks were developed and piloted with about one thousand students in grades 3, 7, and 11. The hands-on tasks were designed to assess classifying, observing and making inferences, formulating hypotheses, interpreting data, designing an experiment, and conducting a complete experiment.

For example, in Classifying Vertebrae, an individual hands-on task, students were asked to sort eleven small animal vertebrae into three groups based on similarities they observed, record their groups on paper, and provide written descriptions of the features of each group. In Triathlon, a group pencil-and-paper activity, students were given information about the performances of five children on three events (Frisbee toss, weight lift, and fifty-yard dash), asked to decide which child would be the all-around winner, and write an explanation of their reasoning.

According to NAEP, the results were promising: students “responded well to the tasks and in some cases, did quite well” (NAEP, 1987, p. 7). Older students did better than younger students, and across grade levels, students did better on tasks involving sorting and classifying than those that required determining relationships and conducting experiments. The researchers also concluded that conducting hands-on assessments was both feasible and worthwhile, although they found it to be “costly, time-consuming, and demanding on the schools and exercise administrators” (Blumberg, Epstein, MacDonald, & Mullis, 1986). Perhaps for these reasons, the hands-on items were not used in the 1990 NAEP assessment in science.

Summary

These six examples were among the more ambitious attempts to use performance assessment on a large scale in the United States during the past two decades, but many other states incorporated performance assessments in some form in their testing programs, and many continue to use performance assessments today. (See chapter 3, this volume.) The examples above were selected because they were pioneering efforts, because performance assessment played such a prominent role in each system, and because they offer lessons regarding technical quality, impact, and burden associated with performance assessment that continue to be relevant today. This brief history should be not be interpreted to mean that performance assessment has no future. The demise of some of these assessments was the result of a confluence of factors unique to each time and setting. Although there are lessons to be learned from these histories (as the rest of the chapter will discuss), it would be incorrect to infer from these cases that large-scale performance assessment is infeasible or impractical. Many states are using performance assessment successfully today for classroom assessment, end-of-course testing, and on-demand assessment. Instead, the history highlights the kinds of challenges that have to be addressed if performance assessments are to be used successfully on a large scale.

Technical Quality

Research on the technical quality of performance assessments provides information about agreement among raters (reliability of the rating process), the reliability of student scores, the fairness of performance assessment for different population groups, and the validity of scores for particular inferences. In reviewing the evidence, it is important to remember that the research was conducted in many different contexts—mathematics portfolios, hands-on science investigations, writing tasks, music performances—and the body of evidence may not be complete with respect to technical quality for any specific type of performance assessment.

Impact of Assessment Initiatives

Tests used for standards-based accountability send signals to educators (as well as students and parents) about the specific content, styles of learning, and styles of performing that are valued. An abundance of research suggests that teachers respond accordingly, emphasizing in their lessons the content, styles of learning, and performing that are manifest on the tests.¹⁰ In reviewing this literature, Stecher (2002) concluded that “large-scale high-stakes testing has been a relatively potent policy in terms of bringing about changes within schools and classrooms.” On the positive side, high-stakes testing is associated with more content-focused instruction and greater effort on the part of teachers and students. Performance assessment, in particular, has been found to lead to greater emphasis on problem solving and communication in mathematics and to more extended writing in language arts. For example, researchers in Vermont reported that the portfolio assessment program had a powerful positive effect on instruction, leading to changes that were consistent with the goals of the developers. Mathematics teachers reported devoting more time to problem solving and communication in mathematics; similarly they spent more time having students work in pairs or small groups (Stecher & Mitchell, 1995).

Likewise, researchers studying the Kentucky reforms found considerable evidence that teachers were changing their classroom practices to support the reform, for example, to support problem solving and communicating in mathematics and writing (Koretz, Barron, Mitchell, & Stecher, 1996). Similarly, researchers in Maryland found that statewide, most mathematics teaching activities were aligned with the state standards and performance assessments, although classroom assessments were less consistent with state assessments (Parke & Lane, 2008). Teachers in Maryland reported making positive changes in instruction as a result of MSPAP, and schools in which teachers reported the most changes saw the greatest score gains (Lane, Parke, & Stone, 2002). In general, these instructional effects are not a function of the format of the test; they occur with both multiple-choice tests and performance assessments—but of attaching consequence to measured student outcomes. However, Kentucky teachers were more likely to report that open-response items and portfolios had an effect on practice, adding credence to the impact of tests worth teaching to.

On the negative side, Stecher (2002) concluded, “Many of these changes appear to diminish students’ exposure to curriculum.” This conclusion was drawn primarily from research in which teachers reported that they changed instruction in ways that narrowed the curriculum to topics covered by the tests (Shepard & Dougherty, 1991). In addition, researchers documented substantial shifts in instructional time from nontested to tested subjects and, within subjects, from nontested to tested topics. For example, teachers increased coverage of basic math skills, paper-and-pencil computations, and topics included in the tests and decreased coverage of extended project work, work with calculators, and topics not included in the test (Romberg, Zarinia, & Williams, 1989).

More recently, researchers have documented the phenomenon of educational triage, where teachers focus resources on students near the cutoff point for proficient at the expense of other students (Booher-Jennings, 2005). Although these studies were conducted in the context of multiple-choice testing, it seems fair to predict that similar effects would be observed with high-stakes performance assessments if they focused on some parts of the curriculum or some students more than others. In fact, the curriculum-narrowing problem might be exacerbated with the use of performance assessments because each task is more “memorable” than a corresponding multiple-choice item, increasing the likelihood that teachers might focus on task-specific features rather than broader skills.

Finally, the research documents instructional changes that can be associated with the format of the high-stakes test. For example, teachers engaged in excessive test preparation, in which students practiced taking multiple-choice tests between one and four weeks per year and up to one hundred hours per class (Herman & Golan, n.d.). Others noted instances of coaching that focused on incidental aspects of the test (e.g., the orientation of the polygons) that were irrelevant to the skills that were supposed to be measured (Koretz et al., 2001). Other researchers found that teachers had students engage in activities that mimicked the format of the tests. For example, teachers had students find mistakes in written work rather than producing writing of their own (Shepard & Dougherty, 1991).

Using performance assessments rather than multiple-choice tests might reduce the prevalence of these effects because the tasks are more representative of the reasoning embodied in the standards. But performance assessment is not immune from negative effects when used in a high-stakes context. For example, teachers in Vermont were found to engage in “rubric-driven instruction,” in which they emphasized the aspects of problem solving that led to higher scores on the state rubric rather than problem solving in a larger sense (Stecher & Mitchell, 1995). Performance assessments are also more costly than multiple-choice tests (see chapters 8 and 9, in this volume), and they take more time to complete (Stecher & Klein, 1997).

Ultimately policymakers would like to know whether the benefits of performance assessment (in terms of more valid measurement of student performance and positive impact on classroom practice, for example) justify the burdens (in terms of development costs, classroom time, scoring costs, and so on). The expenditures and administrative burdens associated with performance assessments may be viewed as high relative to multiple-choice tests. Yet that is not the end of the story.

First, the benefits may justify the burdens from the perspective of education. Vermont teachers and principals thought their state’s portfolio assessment program was a “worthwhile burden.” In fact, in the first years, many schools expanded their use of portfolios to include other subjects (Koretz et al., 1994), and even in recent years, most Vermont districts have continued the use of the writing and mathematics portfolios, even though they are not used for state accountability purposes. Similarly, Kentucky principals reported that although they found KIRIS to be burdensome, the benefits outweighed the burdens (Koretz et al., 1996).

Second, as described later in this book, the costs associated with performance assessments have declined in recent years, making it more attractive to incorporate some degree of performance assessment into state testing programs. In spite of the complicated history of performance assessment, its future looks promising.

Collegiate Learning Assessment

The Council for Aid to Education created the Collegiate Learning Assessment (CLA) in 2000 to help postsecondary faculty improve teaching and learning in higher education institutions (Benjamin et al., 2009). The intent of the test is to provide an assessment of the value added by the school’s instructional and other programs with respect to desired learning outcomes (Klein et al., 2007).

CLA is entirely performance based and uses two types of tasks: performance tasks and analytic writing tasks. The performance tasks present students with problems that simulate real-world issues and give an assortment of relevant documents, including letters, memos, summaries of research reports, newspaper articles, maps, photographs, diagrams, tables, charts, interview notes, or transcripts. Students have ninety minutes to review the materials and prepare their answers. These tasks often require students to marshal evidence from different sources; distinguish rational from emotional arguments and fact from opinion; understand data in tables and figures; deal with inadequate, ambiguous, or conflicting information; spot deception and holes in the arguments made by others; recognize information that is and is not relevant to the task at hand; identify additional information that would help to resolve issues; and weigh, organize, and synthesize information from several sources. Students’ written responses to the problems are evaluated to assess their abilities to think critically, reason analytically, solve problems, and communicate clearly and cogently.

The analytical writing tasks ask students to write answers to two types of essay prompts: a make-an-argument question that asks them to support or reject a position on some issue and a critique-an-argument question that asks them to evaluate the validity of an argument made by someone else.

The CLA is computer administered over the Internet, with all supporting documents contained in an online document library. The online delivery permits the performance assessments to be administered, scored, analyzed, and reported to the students and their institutions more quickly and inexpensively. Initially, trained readers scored the tasks using standardized scoring rubrics. Starting in fall 2008, a combination of machine and human scoring was used. Scores are aggregated by institution and are not reported at the individual student level (Collegiate Learning Assessment, 2009). A recent study of machine scoring has suggested that the correlations between hand and machine scoring are so high that when the institution is the unit of analysis, machine scores alone can be relied on (Klein, 2008). Even at the student level, the correlation between machine scoring and human scorers is 0.86 (Klein et al., 2007).

National Assessment of Educational Progress

The NAEP conducts periodic assessments in mathematics, reading, science, writing, the arts, civics, economics, geography, and US history. NAEP results are based on representative samples of students at grades 4, 8, and 12 for the main assessments or samples of students at ages 9, 13, or 17 for the long-term trend assessments (National Assessment of Educational Progress, 2009a). In all of the subject areas but writing, the NAEP items are a combination of multiple-choice and constructed-response items, which require short or extended written responses. In the three subject areas discussed next, NAEP also uses other forms of performance-based tasks.

National Occupational Competency Testing Institute

The National Occupational Competency Testing Institute (NOCTI) is a nonprofit organization founded in the early 1970s to coordinate and lead the efforts of the states in developing competency tests for occupational programs (NOCTI, 2009). Today NOCTI provides “job-ready” assessments to measure the skills of an entry-level worker or a student in secondary or postsecondary career and technical programs. Most job-ready assessments include both multiple-choice and performance components. For example the Culinary Arts Cook II Assessment includes as a performance assessment preparing a recipe for chicken with sauce. The task is scored on organization, knife skills, use of tools and equipment, preparation of chicken and sauce, safety and sanitation procedures, appearance, and taste of finished product. The Business Information Processing Assessment includes a performance task requiring the student to create a spreadsheet; the task includes header and placement, spreadsheet and column headings, data entry, formula entry, computation of totals, use of functions, formatting, creating a pie chart, saving spreadsheet, printing material, and overall timeliness of job completion. More than seventy job-ready assessments are available in sixteen industry and occupational categories. The performance assessments may be scheduled over one to three days (NOCTI, 2009). Depending on the subject area and the test site, they range in cost from less than one hundred dollars to as much as seven hundred dollars per student.

Performance Assessments of Teachers

The previous examples all have low stakes for students, but performance assessments are also being used in cases where student performance has important consequences. The National Board for Professional Teaching Standards uses a variety of assessment center exercises as well as an annotated portfolio to certify teachers with advanced teaching skills, which often qualifies them for salary bonuses (National Research Council, 2008). The portfolios include teachers’ plans, videotapes of their instruction, and samples of student work illustrating their learning over time, along with teachers’ commentary explaining their decisions and interpreting the outcomes.

California requires teacher preparation programs to use a similar teaching performance assessment for beginning teachers as one component in their credentialing decision. The Performance Assessment for California Teachers (PACT), modeled on the National Board portfolio, has been found to be a valid measure for this purpose (Pecheone & Chung, 2006). In addition, teachers’ scores on both the National Board assessment and the PACT have been found to predict teachers’ effectiveness in supporting gains in student achievement (Darling-Hammond, Newton, & Wei, 2013). A similar assessment, the edTPA, developed from the PACT model, is now beginning to be used in many states across the country (Darling-Hammond, 2012).

How Performance Assessments May Be Best Used

The appropriate role for performance assessments should be determined in part by an analysis of content standards. Such an analysis should reveal which standards are served well by which types of assessments. To the extent that the standards call for mastery of higher-order, strategic skills, they may favor the use of performance assessment. Perhaps more important, to the extent that standards are silent about the nature of performances expected of students, they abrogate responsibility to others for these decisions. Thus, it may be important to revisit standards documents to make sure they provide adequate guidance with respect to desired student performance.

The research reminds us that subject domains are different, and mastery of each domain is manifest in unique ways. Rich, thoughtful, integrated writing can be observed under different circumstances and in different ways from rich, thoughtful, integrated scientific inquiry or rich, thoughtful, integrated musical performance. When the Mikado sings, “Let the punishment fit the crime” (Gilbert & Sullivan, 1885), the educator should reply, “Let the assessment fit the domain.”

There are costs and benefits associated with testing for accountability in whatever form that testing takes. We are used to the current high-stakes, multiple-choice model, but that does not mean it is cost free or benefit rich. Adopting performance assessments for some or all accountability testing will have trade-offs, and we are more likely to make wise decisions if we understand these trade-offs better. Performance assessments themselves differ; the costs and benefits associated with short-answer, fill-in-the-blank items are not the same as those associated with prompted writing tasks, equipment-rich investigations, or judged real-time performances.

At the same time, computer-based scoring of tasks with rich responses (like the Collegiate Learning Assessment described earlier) has reached levels of reliability with human scoring that make a greater range of options both possible and affordable. (See chapter 5 for further discussion of computer scoring of open-ended tasks.) In addition, costs could be controlled by deciding to use the scores from performance tasks as indicators of achievement at the school level rather than the individual level. This approach is consistent with current NCLB and state accountability systems in which the primary unit of accountability is the school.¹¹

If this situation changes—for example, if incentives are assigned to individual teachers as they are in pay-for-performance schemes—it would still be possible to use a matrix sampling approach within classrooms to offset some of the cost increases associated with performance assessment. Student-level accountability policies (such as testing for promotion or graduation) would probably require universal use of performance tasks; this would be more costly overall, although not more complicated logistically. Further analysis will be necessary to estimate the magnitude of the additional costs under different performance assessment scenarios. As states adopt Common Core State Standards, as more than forty have recently done, this could reduce costs by permitting the wider use of tasks. (See chapters 8 and 9, this volume, for discussion of the costs and benefits of multistate assessment development.)

Messick (1994) distinguishes between task-centered performance assessment, which begins with a specific activity that may be valued in its own right (e.g., an artistic performance) or from which one can score particular knowledge or skills, and construct-centered performance assessment, which begins with a particular construct or competency to be measured and creates a task in which it can be revealed. Research suggests it could be more productive to concentrate performance assessment for accountability on construct-oriented tasks derived from academic content standards and leave for classroom use more task-defined activities that may be engaging and stimulate student learning but do not represent a clear, state-adopted learning expectation. In addition, it would be wise to eschew the use of unstructured portfolios in large-scale assessment because they are difficult to score reliably and because it is difficult to interpret the scores once obtained.

The Problem of High Stakes

It would also be wise to remember that Campbell’s Law applies to performance assessments as well as multiple-choice tests (Campbell, 1979). When stakes are attached to the scores, teachers will feel pressure to focus narrowly on improving performance on specific tasks, which will undermine the interpretability of scores from those tasks. While performance assessments may be more “worth teaching to” than multiple-choice tests, performance tasks still represent just a sample of behaviors from a larger domain. Strategies that are used to reduce corruption with multiple-choice tests, including changing test forms regularly and varying the format of tasks and the representation of topics, will be equally useful to reduce corruption with performance assessments.

The recent history of performance assessment at the state level raises some concerns about using these tasks in high-stakes contexts. States at the forefront of the performance assessment movement sometimes found that it was difficult to garner and sustain public support for these “new” forms of testing. While some of the problems that states encountered were due to difficulties with scoring, reliability, and validity, others came from energized stakeholder groups that objected to the manner in which they were implemented. In some states, people objected because the assessments were unfamiliar and stretched the boundaries of traditional testing. In others, the assessments were implemented in ways that did not adequately answer parents’ questions or address their concerns. For example, some parents worried that if students were asked their views on a topic, they would be graded on the substance of their opinions rather than on their reasoning or writing skills.

McDonnell (2009) characterized these problems as disputes about “the cultural and curricular values embodied in the standards and assessments” (p. 422). Conflicts over values are not easily resolved, but better communication and dissemination of information might help to forestall them. Educators and policymakers may underestimate the need for efforts to inform the public. For example, despite the endless discussion of NCLB in the education community since 2001, a majority of the general public reports that it is not very familiar with the law (Bushaw & Gallup, 2008). History suggests that educators would be wise to clearly describe the role of performance assessments and make extra efforts to educate parents and the general public about changes in the testing program before they are adopted.

In their 1999 review, Madaus and O’Dwyer concluded that “the prognosis for the feasibility of deploying a predominantly performance-assessment oriented system for high-stakes decisions about large numbers of individual students is not very promising in light of the historical, practical, technical, ideological, instructional and financial issues described above” (p. 694). While it is still the case that a standards-based accountability system relying primarily on performance assessment seems unlikely in the United States, we can be more optimistic about the possibility of using performance assessment as part of a large-scale testing program in combination with less open-ended item formats. The successful use of structured performance assessments on a large scale in low-stakes contexts such as the Program in International Student Assessment and NAEP suggests that practical and logistical problems can be overcome and that performance tasks can enhance our understanding of student learning.

If such progress is to be made, more research will be needed to establish a “foundation for assessment design that is grounded in a theory of cognition and learning and methodological strategies for establishing the validity of score use and interpretation in terms of the quality and nature of the observed performance” (Baxter & Glaser, 1994, p. 44). Such work may need to be done in each of the major disciplines to be sensitive to discipline-specific ways of knowing, thinking, and acting.