Connecticut

Connecticut has two assessment programs: the Connecticut Mastery Test (CMT), which assesses reading, writing, and mathematics of students in grades 3 through 8, and the Connecticut Academic Performance Test (CAPT), which assesses reading, writing, mathematics, and science in grade 10 (Connecticut State Department of Education, 2009). The CAPT test includes performance-based components in each subject area.

By scoring student results relative to established state goals in each content area, CAPT is designed to measure progress toward the educational goals reflected in the Connecticut curriculum frameworks. Moreover, by statute, CAPT must be included as one indicator of performance to support a graduation decision but cannot be used as the sole criterion for graduation. Specifically, by statute, CAPT scores must be combined with other “measures of successful course completion” (Connecticut State Department of Education, 2009, p. 26).

The CAPT assessment was designed as a balanced assessment with multiple-choice, constructed-response, and curriculum-embedded performance tasks to assess student content knowledge. Performance-based components are scored by teachers who are trained for the task and include a variety of item types and tasks:

• Reading. The Reading across the Disciplines section of CAPT has two tests that assess students’ reading abilities: Response to Literature and Reading for Information. The Response to Literature test asks students to read a short story and respond to a series of essay questions requiring them to describe, interpret, connect to, and evaluate the story. The Reading for Information test “assesses a student’s ability to independently read, thoroughly comprehend, and thoughtfully respond to three authentic nonfiction texts” (Connecticut State Department of Education, 2006, p. 11), which are drawn from magazines, newspapers, and journals. Students answer a combination of multiple-choice and short-answer questions about the meaning of the article and the way the author wrote the article. All constructed responses are scored by trained scorers who have met calibration standards using a 0-to-2-point rubric (Connecticut State Department of Education, 2006).
• Response to Literature. The Response to Literature subtest assesses students on their ability to “independently read, thoroughly comprehend, and thoughtfully respond to one authentic fictional text through four constructed-response questions” (Connecticut State Department of Education, 2006, p. 8). Two independent readers score each of the four written responses on a six-point rubric.
• Writing across the Disciplines. The writing assessment contains two tests that assess students’ writing abilities: Interdisciplinary Writing and Editing and Revising. In the Interdisciplinary Writing test, students are given a set of source materials (e.g., newspaper and magazine articles, editorials, charts, and graphs) representing different perspectives on an issue. They are asked to read the materials and use the information to write a persuasive piece, such as a letter to a congressperson or a letter to the editor of their local newspaper, and cite given sources as support for their argument. Students are required to take two interdisciplinary writing tests about separate issues. Each response is scored holistically. Two independent readers score responses using a six-point scale (Connecticut State Department of Education, 2006). In the Editing and Revising test, students read passages with embedded errors and answer questions to indicate corrections.
• Mathematics. The mathematics section uses questions requiring written responses to assess students’ abilities to solve problems, communicate, compute, and estimate in four major content areas (number quantity; measurement and geometry; statistics, probability, and discrete mathematics; and algebra and functions). Half the total points on the test draw from these performance-based questions; the remainder are based on multiple-choice questions (Connecticut State Board of Education, 2009).
• Curriculum-Embedded Science Tasks. The CAPT science section currently uses a combination of multiple-choice questions and open-ended written responses. Prior to 2007, this section also included a performance task that assessed experimentation using a hands-on laboratory activity; starting in 2007, the laboratory activity was shifted to a curriculum-embedded classroom task. Currently two real-world tasks—a laboratory activity and a science, technology, and society investigation aligned to science standards and curriculum frameworks—are given to schools in each of the science content strands for grades 9 and 10. Students are required to complete these tasks in class; they are asked to formulate a hypothesis, conduct experiments, analyze data, and write a lab report to demonstrate their ability to engage in scientific reasoning. In the on-demand component of the CAPT science assessment, the specific scientific skills and processes needed to complete the embedded assessment are independently tested through the use of constructed-response items aligned to the locally embedded performance tasks (Connecticut State Department of Education, 2007a).

This testing methodology presents an innovative approach to high-stakes assessment in science using both curriculum-embedded performance tasks scored by classroom teachers and an on-demand assessment of student knowledge using a constructed-response methodology to independently measure student learning (Connecticut State Department of Education, 2007a). The approach takes advantage of the power of performance assessment to transform classroom practice and the need to ensure that measures of student learning are comparable and objective.

Given the recent development of Common Core Standards in English language arts and mathematics and Next Generation Standards in Science, Connecticut’s CAPT assessment serves as one proof point of an assessment that is designed to predict college and career success. To examine the impact of student performance on CAPT and its relationship to college and workplace success, Connecticut funded a major study tracking five cohorts of tenth-grade students between 1996 and 2000 over eight years beyond high school. The study found that students scoring higher on CAPT were more likely to attend and graduate from college and were more successful in the workplace as well (Coelen, Rende, & Fulton, 2008). (An example of a Connecticut science assessment task is shown in appendix A.)

New Jersey

In New Jersey, all students must pass the High School Proficiency Assessment (HSPA) or an approved alternative in order to graduate (New Jersey Department of Education, 2005). The state has by statute developed an alternative pathway to graduation through the use of a performance-based assessment system. If students do not pass the HSPA in March of their junior year during the first testing, they can retake the assessment in their senior year. In addition, students failing to meet state standards must begin remediation instruction in preparing for the performance-based Special Review Assessment (SRA).

The SRA is an “individually, locally administered, untimed, state-developed, locally scored assessment” (New Jersey Department of Education, 2008b, p. 7). Students must participate in a school-designed SRA instructional program for the content areas in which they did not meet proficiency on the HSPA prior to being administered SRA performance assessment tasks (PATs). The SRA Instructional Program is continued until the SRA teacher decides that students are successful on the required PATs. The PATs are open-ended tasks in each of the key competency areas to which students respond in writing, showing and explaining their ideas and solutions. (For an example, see appendix A.)

For content areas in which they have not scored at least 200 on the HSPA, students must successfully complete two PATs in each content area cluster or standard. Language arts literacy has two clusters, and mathematics has four standards. Selection of PATs “is based solely on the results of the student’s first HSPA administration” (New Jersey Department of Education, 2008b, p. 6). “If a student is not successful on a specific PAT, additional PATs may be administered until the student successfully completes the required number of PATs for that content area” (p. 7). In addition, to earn a diploma, “Students with disabilities who are in grade 11 . . . must participate in the HSPA or the APA [Alternate Proficiency Assessment]” (p. 12). “The Individualized Education Program (IEP) team for each student determines which assessment (HSPA or APA) the student will take for each content area addressed” and “must also determine if the student who is taking the HSPA in one or both content areas will be required to pass the HSPA in those content areas in order to graduate” (pp. 12–13).

New Jersey has been a leading state in developing and validating alternative assessment pathways to graduation. The state, through developing the SRA and APA, allows all students (including special needs students) alternative pathways to obtain a high school diploma. More important, the SRA alternatives offer diverse learners greater access to college through embedded performance measures that assess academic progress by using testing formats more sensitive to various learning modalities. As a result of implementing these policies, New Jersey reports one of the highest graduation rates (83 percent) in this country, comprising all racial and ethnic groups.

Although concerns were recently raised about the reliability and comparability of the locally managed and scored assessment results, the performance assessments were maintained after a formal state review. The review reinforced the value of this approach to supporting and evaluating learning. Based on the review recommendations, a more carefully controlled process for administering and scoring the PATs was put in place (New Jersey Department of Education, 2013).

New York State

New York has a 135-year history of state-level assessment that includes both on-demand and classroom-based performance tasks (New York State Education Department, 1987, p. 18). To earn a New York State diploma, students must pass commencement-level Regents Examinations in comprehensive English, global history and geography, US history and government, mathematics, and science, in addition to completing course requirements. Different cut scores on these syllabus-based, end-of-course tests are used for a local diploma and a state Regents diploma. Alternative assessments, approved by the state, can also be used for these diplomas.

Performance-based components of the Regents Examinations include a variety of tasks. In English, students write responses to both spoken and written texts. In addition, they are asked to write an essay discussing a controlling idea within two literary texts and the authors’ use of literary elements and techniques, and, in a separate essay, “interpret a statement provided to them about some aspect of literature and write an essay using two works they have read to support their interpretation of the statement” (Shyer, 2009, p. 3). The Regents Science Examination includes a laboratory performance test completed near the end of the course and a written test with a large number of open-ended questions (Shyer, 2009). In history and social studies, students complete essays about document-based questions that require analysis of a set of documents and artifacts. (See appendix A for an example.)

Generally at least two teachers must independently rate all Regents Examinations that lead to a Regents diploma, except mathematics, which requires at least three scorers. All teachers rate exams according to the scoring key and rubrics provided by the Department of Education, which have directions for scoring multiple-choice and constructed-response questions and, if applicable, guidelines for rating the essay or performance components (New York State Education Department, Office of State Assessment, 2008). Teachers are trained to score all extended writing tasks using benchmark performances and rubrics (University of the State of New York State Education Department, 2009a, 2009b). (See the New York State history assessment task in appendix A.)

The Regents have also approved a local diploma option that allows development of equivalent academic tasks, often part of a portfolio-based system, that can be substituted for the Regents Examination. All local options must be reviewed and approved by the state department of education. For example, the New York Performance Standards Consortium, a group of twenty-seven secondary schools, has received a state-approved waiver allowing their students to complete a graduation portfolio in lieu of the Regents Examinations in all subjects but English. This portfolio includes a set of ambitious performance tasks—a scientific investigation, a mathematical model, a literary essay, a history/social science research paper, an arts demonstration, and a reflection on a community service or internship experience—that meet a set of common standards and are scored through social moderation processes using common scoring rubrics (Performance Standards Consortium, 2013).

Washington

Washington’s standards, developed in the 1990s, feature ambitious goals in a wide range of areas, including reading; writing; communication; mathematics; social, physical, and life sciences; civics and history; geography; arts; and health and fitness. The Washington Assessment of Student Learning (WASL) was developed to assess student mastery of these standards. WASL included a combination of multiple-choice, short-answer, essay, and problem-solving tasks. In addition, the Washington assessment system included classroom-based assessments in subjects not included in WASL. Although the WASL was replaced in 2009–2010 with more traditional tests (the Measurements of Student Progress in grades 3 to 8 and a High School Proficiency Exam in grades 10 to 12), it expanded the use of classroom-based assessments, including performance assessments, to gauge student understanding of the learning standards in social studies, the arts, health/fitness, and educational technology.

Districts may use the state-designed assessments or may substitute assessments they have designed locally. They must report to the state that they are implementing the assessments or strategies in those content areas, but individual student scores are not reported. Surveys show that districts agree that the assessments support continuing focus on these areas of study; identify intentional models of instruction for students; help teachers identify areas of strength and weakness and modify instruction accordingly; and allow a more consistent standard of assessment across the state as they show what student performance should look like (Office of the Superintendent of Public Instruction, 2012). (For examples of some of these assessments in civics, economics, health and fitness, and the arts, see appendix A.)

New England Common Assessment Program

The New England Common Assessment Program (NECAP), an unprecedented state collaboration around development and design of common reference examinations, was launched by New Hampshire, Rhode Island, and Vermont, and recently expanded to include Maine. NECAP is offered in grades 3 to 8 plus grade 11 (Measured Progress, 2009). It is a hybrid assessment comprising both multiple-choice and constructed-response items, including performance tasks. In addition, each of these states supports more extended performance assessments that are designed and administered at the local level. We describe both below.

Performance-based components of NECAP include a range of standards-based constructed-response items and performance tasks. In writing, students respond to two writing prompts that are scored using a common set of rubrics. (See appendix A for an example.) In science, both constructed-response and performance items are included. The science constructed-response items “require students to respond to a question by using words, pictures, diagrams, charts, or tables to fully explain their response,” and an inquiry task asks students “to hypothesize, plan, and critique [scientific] investigations, analyze data, and develop explanations” (New England Common Assessment Program, 2009, pp. 1, 15).

All NECAP scorers are trained and go through a calibration process prior to scoring. The writing prompt is “scored by two independent readers both on quality of the stylistic and rhetorical aspects of the writing and on the use of standard English conventions” (Measured Progress, 2009, p. 8). The other constructed-response answers are scored using an item-specific rubric with score point descriptions. Common cut scores were established through representative expert committees in the NECAP states to ensure comparability across states. Professional development materials to support the NECAP assessment were developed by content specialists at the New Hampshire, Rhode Island, and Vermont departments of education in partnership with the Education Development Center and the National Center for the Improvement of Educational Assessment.

Through this collaboration, the NECAP states were able to maintain common content standards for the region; this fosters sharing of instructional and curriculum resources within and across state borders. In addition, NECAP states significantly lowered the cost of assessment while maintaining high standards of quality; these cost savings enabled the consortium to develop a more balanced assessment, including performance-based, constructed-response items, which would have strained the testing budgets of the individual states.

Collaboration among states in test design and administration could become a model for the country in developing innovative assessments that meet high standards of test quality, measure a broad range of skills and abilities, and are administratively feasible and cost-effective. By pooling resources, states are better able to afford the development of richer performance measures designed to address the skills and abilities needed to be college and workplace ready in the twenty-first century. Finally, state-based consortia can promote and support development of regional learning networks, which enable teachers and administrators to share promising practices and resources across states. These increasingly include networks that are designed to strengthen the local performance assessment systems that each state has previously nurtured individually. The Quality Performance Assessment (QPA) network, for example, is enabling schools, districts, and states across New England to strengthen local assessment systems by introducing common tasks that evaluate critical thinking, problem-solving, and communication skills and engaging school faculties in moderated-scoring processes (Center for Collaborative Education, 2012). (See chapter 7 in this volume.)

College and Work Readiness Assessment

The CWRA is a high school senior version of the Collegiate Learning Assessment (CLA), described in chapter 2 (Klein, Benjamin, Shavelson, & Bolus, 2007; Klein, Freedman, Shavelson, & Bolus, 2008; Shavelson, 2008). Both assessments were developed at the Council for Aid to Education.

The CLA was developed to measure undergraduates’ learning—in particular, their ability to think critically, reason analytically, solve problems, and communicate clearly. The assessment comprises performance tasks and critical writing components. The performance task component presents students with a real-world problem and an in-basket of information and asks them to either solve the problem or recommend a course of action based on the furnished evidence. The analytic writing prompts ask students either to take a position on a topic or to critique an argument. A ninety-minute, entirely constructed-response exam, the CLA is delivered over the Internet. The assessment focuses on performance at the institution level or on performance at the program level within an institution. Institution or program-level scores are reported as observed performance and as value added beyond what would be expected from entering student SAT scores.

Both the CLA and its high school counterpart, the CWRA, differ substantially from most other standardized tests, which are based on an empiricist philosophy and a psychometric/behavioral tradition. From this stance, everyday complex tasks are divided into components, and each component is analyzed to identify the abilities required for successful performance. For example, suppose that components such as critical thinking, problem solving, analytical reasoning, and written communication are identified. A separate measure of each component would then be constructed, and students would take each test. At the end of testing, student scores would be added up to construct a total score to describe their performance—not only on the assessment at hand but also generalizing to a universe of complex tasks similar to those the tests were intended to measure.

The conceptual underpinnings of the CLA and CWRA are embodied in what has been called a criterion sampling approach to measurement. This approach assumes that the whole is greater than the sum of its parts and that complex tasks require an integration of abilities that cannot be captured if divided into and measured as individual components. The criterion-sampling notion is straightforward: if you want to know what a person knows and can do, sample tasks from the domain in which she is to act, observe her performance, and infer competence and learning. For example, if you want to know whether a person not only knows the laws that govern driving a car but can also actually drive a car, do not just give her a multiple-choice test. Also administer a driving test with a sample of tasks from the general driving domain (starting the car, pulling into traffic, turning right and left in traffic, backing up, parking). On the basis of this sample of performance, it is possible to draw more general, valid inferences about driving performance.

The CLA/CWRA follows the criterion-sampling approach by defining a domain of real-world tasks that are holistic and drawn from life situations. It samples tasks and collects student-generated responses that are modified with feedback as the task is carried out. (For an example of a CLA task, see appendix B.) The skills tested by these assessments do indeed appear to be related to college success. Research shows that high school grade point average (GPA) combined with scores on CWRA performance tasks offer as good a measure as high school GPA combined with SAT scores for predicting a student’s first year GPA and a better measure for predicting the cumulative college GPA (Klein et al., 2009).

The CWRA follows a similar approach. Its twofold mission is to (1) improve teaching and learning by using performance tasks to connect classroom practice with authentic institutional assessment and (2) evaluate student readiness to do college work. It is based on the same structure as the CLA but adapted to test high school seniors’ critical thinking, analytical reasoning, problem solving, and communication abilities. The CWRA is designed to provide greater access to the test content by adjusting the reading level to make it appropriate for a range of high school students. The adapted, open-ended performance tasks are administered in conjunction with other standardized tests of critical thinking to produce reliable individual student scores.

If these assessments are to encourage productive teaching, it is crucial to work with teachers to offer a model for how curricular and pedagogical interventions can help students develop these higher-order skills. CWRA offers workshops called “Performance Task Academies” to provide hands-on training grounded in the literature on learning theory, critical thinking, and authentic assessment. The academies show teachers how to create classroom projects that are a hybrid of case studies and performance-based learning, with a special focus on higher-order skills. The Performance Tasks Library is a teacher-created teaching resource.

The College Readiness Performance Assessment System

C-PAS is another tool designed to evaluate the college readiness of high school students. Developed by David T. Conley, director of the Education Policy and Improvement Center at the University of Oregon, CPAS is a response to the problem that current tests do not effectively gauge students’ ability to apply learning strategies they will be expected to demonstrate in entry-level college courses and beyond. As American secondary school classrooms attempt to become more data driven and as more high school students set college as their goal, it is critical that teachers have the right set of data to enable them to make instructional decisions that prepare their students for postsecondary education and that students have a clearer picture of their readiness for college courses. C-PAS is designed to supply this type of information to teachers and students to ensure high school instruction leads to college readiness for all students.

C-PAS is a series of curriculum-embedded performance tasks that teachers administer within the context of their curriculum and score with a common scoring guide. The result is a performance profile for each task composed of scores from up to five key cognitive skills measuring students’ ability to reason, solve problems, interpret information, conduct research, and generate work with precision and accuracy. The teacher separately grades the task for inclusion as a component in the course grade, thereby increasing student engagement in the task. The tasks are carefully designed to encourage student development of key cognitive strategies that research identifies as being important elements of entry-level college courses. (For an example of a C-PAS task see appendix B.)

C-PAS development is rooted in psychometric principles and practices in order to achieve a high degree of technical adequacy, which helps ensure that the scores generated are valid and accurate indicators of student development of key cognitive strategies associated with college success. This is achieved in a number of ways. The five key cognitive strategies are carefully analyzed using item response theory to determine the degree of interaction among them and establish task difficulty. Scoring guides are refined so that they focus on the key attributes of each cognitive strategy. All task writers are carefully selected and then trained to use task shells to ensure the structural similarity of all tasks and minimize task variance on extraneous dimensions. Teachers must follow common conditions of administration when introducing the tasks in class. Finally, after teachers score their students’ tasks using the common scoring guides, master scorers rescore a subset of the tasks to ensure consistency of teacher scoring.

Ohio Performance Assessment Pilot Project

The goal of the Ohio Performance Assessment Pilot Project (OPAPP) is to contribute to developing a world-class assessment that raises learning expectations for all students, is balanced, and uses a multiple measures approach to assessment and accountability. One important purpose of this approach is to support improvement in instructional practice and align student achievement to college readiness standards and international benchmarks of student performance. The initial phase of the project focused on developing curriculum-embedded, teacher-managed, rich performance tasks that are content focused, skills driven, and aligned to college and workplace readiness standards.

Course-embedded performance assessment tasks measure the content knowledge and skills learned in eleventh- and twelfth-grade courses. Students complete tasks in and out of class over a period of one to four weeks as an embedded part of course curriculum. Teachers administer the tasks under the supervision of their districts and state coaches. Content-area and state department curriculum experts develop each task in consultation with teachers, higher education faculty, and national content experts.

Performance outcomes are the content-specific knowledge and skills described by content experts that are needed for college and career success in the twenty-first century. Ohio teachers, higher education faculty, and state curricular experts arrive at a consensus as to the relative importance and validity of the performance outcomes. The performance outcomes are aligned with state content standards, national content standards (like those of the National Councils of Teachers of Mathematics and English), college readiness standards (Conley, 2007), and international benchmarks. Performance outcomes are a set of high-leverage content and skills aligned to the Ohio standards and the national Common Core standards. Explication of performance outcomes serves as the blueprint for designing the course-embedded performance assessment tasks.

Common scoring rubrics are a set of evaluative criteria aligned with the performance outcomes, designed to assess all performance tasks within a specific disciplinary focus (e.g., scientific inquiry and investigation, mathematics problem solving, English language arts inquiry, and communication).

The scoring system is based on a set of training protocols and benchmarks designed to ensure high scorer reliability. In addition, the project has developed moderation procedures to audit local teacher scores and a data feedback loop to give districts and schools formative information to improve teaching and learning.

OPAPP is designed to support consequential decisions; the design of the scoring is therefore grounded in psychometric principles and practices to support the validity of the measure. A high degree of technical adequacy is needed to ensure that the scores generated are credible and defensible and that they measure important dimensions of learning associated with college and career success. This is accomplished in part through using item response theory to examine the degree of interaction among them and establish estimates of student ability and task difficulties.

The performance assessments offer a rich, authentic measure of higher-order thinking and focus on discipline-specific thinking skills deemed necessary for college and career success. Assessment results supply formative information to teachers about student achievement of key performance outcomes, as well as credible, defensible evidence that can be used as part of a summative high-stakes accountability decision. In addition, the approach is designed specifically to create a two-way flow of information and engagement from the classroom level to the school and district, and from the state and systems level back into the classroom. These performance tasks are aligned with state curriculum frameworks and course syllabi and modeled after proven assessment practices that are already in place in high-performing countries worldwide.

Development of the course-embedded content knowledge and skills assessments is designed to fit the redesign of the Ohio accountability system as a component of an end-of-course examination system. Evidence and scores from course-embedded, performance-based assessments are specifically designed for use in combination with state-developed end-of-course exams. The reference exams include both constructed-response and on-demand multiple-choice items—essays and problem solutions, as well as curriculum-embedded, extended-performance tasks that may require more extended writing, research, and inquiry tasks. The tasks will be constructed by high school faculty and college faculty under the leadership of the state department of education and are intended to inform the grade given to the student in the course as well as combine with end-of-course exams to support high school graduation decisions.

The assessments may also be used to satisfy Ohio’s senior project requirement. The senior project is defined to include a variety of formats. One format might be a single project in an area of deep interest to students; a second could be a graduation portfolio that includes performance assessments in a selected number of content areas—for example, subject areas chosen by the student, as is common in other countries. (See chapter 4, this volume.) Students demonstrating mastery in additional content areas could receive diploma endorsements (“merit badges”) recognizing their outstanding achievement. These endorsements of accomplishment could be taken into account as part of a student’s application for college or in conjunction with a placement exam used by colleges to determine course eligibility. (See the Ohio Performance Assessment Task in appendix B.)

Task Quality

Traditional state assessments typically emerge from a development process that involves multiple reviews by curriculum and assessment experts, field testing, psychometric analyses, and further review and revision. But several programs with performance components rely on local development and selection of tasks. As expected, there is varying quality in locally developed tasks. The next sections examine task quality in terms of several critical attributes: alignment to standards; rich, scorable products; and technical quality, including scoring reliability and informative score scales.

Technical (Measurement) Quality

Issues of technical quality—sometimes real and sometimes perceived—contributed to the demise of the authentic assessment movement more than a decade ago. The challenges of performance assessment were identified and exposed, but not all states made the effort to work through the technical problems of reliability and validity. Given the demands of continuous development of new tests, coordination and administration of tests statewide, and analysis and reporting, the assessment and accountability staff in the state departments of education and their contractors had more than enough on their plates. Few had time to publish and disseminate information about what they were learning—including evidence of strong technical quality, when it existed—beyond the requirements of the programs and contracts. Unsubstantiated criticism was common and often left unanswered. For example, this was clearly the case with respect to human scoring, which is relied on heavily in performance assessment.

On-Demand Component

It is important that both assessment components model good classroom practices. As discussed earlier, sole reliance on multiple-choice tests can and will narrow curriculum and drive instruction toward tested skills (Amrein & Berliner, 2002; Shepard et al., 1995). These authors recommend that an on-demand component (either paper and pencil or computer delivered) include a significant number of constructed-response questions carrying substantial weight toward the total score. Teachers tend to model the state tests in their classroom testing; as Kentucky showed, on-demand, constructed-response testing can indeed lead teachers to place greater emphasis on this format, using rubrics for scoring and gaining the benefit of seeing more actual student work while focusing more on higher-order thinking skills.

Over the past few decades, several states have used a test with common and matrix-sampled questions. Matrix sampling is the practice of generating multiple forms of an assessment, each with a different set of items (though some items overlap across forms), and administering these different forms to different students within classrooms and schools. Common questions are the same across all forms of a test at a grade level and are the basis for individual scores. Matrix-sampled questions differ across forms and serve several purposes. If included for successive years, they can be used for test-equating purposes. Also, matrix sampling is a means of field-testing items for use in future years, replacing common items that are released or held for reuse in later years. Embedding field-test items in operational tests constitutes the most effective means of field-testing items because students do not know which items are operational and which are being field-tested. Consequently, student motivation is the same for both. Matrix-sampled items can also be used to bolster measurement in subtest areas for which school results are produced.

The NECAP uses a common/matrix-sampled design similar to the one used previously by several consortium states. This design allows the equating items to approximate the proportions of numerous types of items in the common test. The highly cited Massachusetts Comprehensive Assessment System has used a test design similar to the NECAP since the late 1990s. Matrix sampling of multiple item types is an efficient way to develop multiple test forms specific within grade levels, as well as to support the development of a vertical score scale across grade levels (K–12).

Matrix sampling is a cost-efficient and technically sound practice that allows sampling across a content domain to measure the full range of learning outcomes within a school or district. During the 1990s, matrix sampling allowed school-level reporting in a policy context in which schools were the focus of accountability policies in many states (e.g., Maryland, Nebraska, and Kentucky). By the early 2000s, however, with the authorization of NCLB, student-level reporting on state tests had become a policy imperative and matrix sampling approaches are now rare in state-level summative assessment.

Curriculum-Embedded Performance Component

There is growing belief that to support education of students who can compete effectively in the digital age, assessment systems must be broadened to include locally administered, curriculum-embedded performance assessments (Darling-Hammond, 2010; Popham, 1999). Much of what is considered core knowledge can be assessed by traditional summative tests, but higher-order skills that traditional tests address either inadequately or not at all should be the focus of curriculum-embedded performance tasks designed to measure higher-order cognitive ability. A performance assessment component that capitalizes on the valuable lessons from the past could be built in the following way:

1. The state posts models online, tried-and-true tasks and projects calling for individual, scorable products closely aligned to standards, with a total score range of at least twenty points for each task or project. The tasks use materials and other resources readily available in schools, homes, or online. The posting also includes sample student work, scoring rubrics, and specifications for tasks and projects.
2. Teachers use the state-provided tasks and projects in their own instruction and as models for tasks and projects they develop themselves to submit to the state for review. The state also conducts professional development training sessions using both online and train-the-trainer or coaching models.
3. The state reviews, selects, rejects, revises, and furnishes feedback to teachers for their submissions.
4. The state selects high-quality tasks or projects for pilot testing, collects associated student work, and then posts the tasks and projects, rubrics, and sample student work online for local use. This development, vetting, field testing, and posting sequence is ongoing.
5. The state holds back (does not post) selected tasks or projects, saving them for later use in the local performance assessment component of accountability testing.
6. The state posts a set of tasks or projects for schools to administer within a specified time frame. Teachers score the resulting student work and submit the scores to the state.
7. Each school identifies a low-, middle-, and high-performing student for each task or project and submits the work of those students to the state using electronic portfolio platforms. The teachers’ scoring for those students is audited (rescored) by content specialists.
8. Audit scores are sent back to the schools, and local personnel adjust the scores of their students to be consistent with the benchmarks obtained through the audit process.
9. The next year, steps 6, 7, and 8 are repeated three times, with the tasks and projects for each round chosen to coincide as closely as possible to the time during the year when relevant instruction is given.
10. The results of the performance component are combined with those of the on-demand assessment component, thereby contributing to both student- and school-level results.
11. States support and supply resources for creating learning networks that build and spread educator capacity to strengthen instructional practice by means of effective use of performance assessments.

This approach, which builds on successful models of balanced assessment systems in the United States and abroad, could help create high-quality, reliable, and feasible assessments that are also educationally valuable. Creating systems that support the teaching and learning process also renders assessment more consequentially valid, and more validly consequential, as we describe next.

A New Approach to Validity

The external validity argument for performance assessments begins with the fact that to meet the new demands of the Common Core standards (aimed at ensuring that all students are college and career ready), measures of higher-order thinking need to be a core component of the next generation of assessment and accountability in this country. Performance tasks are necessary to evaluate whether students can think critically, reason analytically, solve problems, and communicate effectively.

Achieving these goals requires a more balanced view of assessment and accountability that includes both formative information—how students develop and access learning resources to complete challenging tasks—and summative judgments of student learning, based on performance tasks that are aligned to national standards and can be used for district and state accountability purposes. Because these metacognitive tasks are situated in the learning process in classrooms, they change the nature of the validity argument supporting use and interpretation of embedded performance tasks. A broader conception of a validity argument is suggested—one that begins with a detailed description of the constructs being measured, takes into account multiple types of evidence at different levels of the scale, and is sensitive to the dynamic interaction between the student and the task, as the act of inquiry continuously shapes student learning.

Validity theory is centered around claims about the appropriateness of the interpretation of data in relationship to student performance on a test or performance task (Cronbach, 1971; Fredericksen & Collins, 1996; Mislevy, Steinberg, & Almond, 2003; Pellegrino, Chudowsky, & Glaser, 2001; Moss, Girard, & Haniford, 2006). According to Campbell and Stanley (1963), to claim test validity means that evidence obtained from the assessment provides support for interpretation of the evidence to the extent that the interpretation is stronger than any other alternative explanation (e.g., internal validity).

Applying this paradigm to performance assessment focuses on the scorer’s collecting and presenting the evidence used to make judgments about the knowledge and skills the student exhibits. Teachers or expert scorers who have been trained and calibrated to score consistently using a scoring protocol or rubric often make judgments rooted in cognitive theory about student learning. As they analyze student work more completely and accurately, they are better able to understand the abilities that have been developed and are being exhibited by students. (See chapter 7, this volume.)

This chain of validity evidence to support curriculum-embedded work requires clear understanding of the skills to be measured and rich description of the performance to support interpretation of a student’s metacognitive ability to carry out complex performance tasks. In addition, teachers can use this chain of evidence to inform their practice by evaluating the impact of specific instructional strategies on student learning, including subgroup analyses (Moss et al., 2006).

Teachers and students need to know the learning demands of the tasks the students are expected to master, and teachers need to create instructional opportunities for students to complete the tasks successfully. This interaction between standards and tasks in the classroom is based on teachers and students developing common understandings of the skills that will be measured and a clear description of the performance indicators to interpret the students’ performance on the basis of standards, student work samples, benchmarks, and rubrics. To this end, the goal of creating more transparency in assessment is to signal to the students the knowledge and skills necessary for success in the classroom.

In addition, students who are engaged in self-assessment and peer assessment of student work cause a significant shift in how we think about test administration and validation. That is, using standardized administration standards for on-demand tests to ensure the objectivity and comparability of assessment is confounded when students have the opportunity to collaborate on the performance tasks. Research suggests that opportunities for peer collaboration, coupled with formative feedback to students, are a leading indicator of student knowledge of the subject matter and a strong predictor of future success (Black & Wiliam, 1998; Bransford & Schwartz, 2001).

Finally, performance assessment data go well beyond supplying scores. They are designed to inform students and teachers about what is important to learn, what learning looks like, and how learning is shaped by the context of the learning environment and its learners (Engeström, 1999). As a result, broadening the conception of validity to address richer and more complex performance tasks requires considering how assessment functions in various instructional and school-based contexts and how the learner is influenced (shaped) by the learning environment.