Understand the Difference Between What Legally Must Be Included in Your Reports and What Must Be True About Your Assessment

A distinction we often discuss in our workshops is between what must be true about your evaluations and what must be explicitly included in your reports. This distinction grew out of a conversation between one of the authors and a special education attorney who was an early collaborator on this project (J. Riel, personal communication, 2009). For what must be true about your evaluation, ultimately the evaluator should be prepared to testify to the truth and accuracy of those points. To the extent possible, these “truths” about the evaluation should be evident to the readers of your reports. In other words, you should show the truth of these legal mandates rather than simply tell the reader they are true.

Another title for this section might be “Quoting the law in your report does not make it or your evaluation more legally defensible,” or perhaps even more straightforwardly, “Saying it does not make it true.” Many reports we read are full of legal-sounding boilerplate language that tells the reader that the evaluation has followed legal guidelines. Often, school psychologists have a standard set of paragraphs that they cut and paste into all their reports stating that their evaluation has met all legal guidelines, as if these boilerplate statements were protective talismans that could ward off the evil questioning of attorneys and advocates. For the most part, we believe this muddles the focus of reports, making them harder to read, and does little if anything to make them more legally defensible. For example, the following statement (or something similar) regarding assessment procedure decisions is in many of the reports we read:

This paragraph has two parts. One makes a claim about the evaluation process and the other offers advice to the reader. When reading the claim that the evaluator took the students’ primary language, ethnicity, and cultural background into consideration prior to the selection of the assessment procedures, the first question that comes to mind is how was this done? In other words, what evidence supports the truth of this claim? Frequently, there is little actual evidence included in the report to support the claim. As for the validity advice, it is not clear how this would be helpful to a reader since the task of understanding validity and interpreting assessment results is the writer’s responsibility, not that of the parent or teacher reading the report. In other words, the burden to choose valid assessments and interpret them accurately is on the author of the report and not the reader.

Contrast the previous boilerplate language example with information about how assessment procedure decisions were made in Marie’s evaluation:

This statement is not a claim but rather an explanation of how the legal and ethical mandate to consider language and culture in the evaluation process was met. Marie’s example suggests that the writer has made a reasonable good-faith effort to consider these issues in designing and conducting the evaluation. Another important difference is that this statement is about a particular child, Marie, while the first statement is generic and could be about any child. The truthfulness and accuracy of any legal claim about the evaluation process can be communicated only by information about an individual child, not by vague general statements. In contrast, the statement in the boilerplate example expects the reader to accept the statement as fact. It does not provide us with any information that allows us to judge the truthfulness of the claim. Consider also the following information, which occurred later in Marie’s report:

Although you may have made a different professional judgment about what instruments to use or not use in this case, this selection of assessment procedures seems logical given the information provided earlier in Marie’s report. Taken together, the two paragraphs provide reasonable evidence for the truth of the legal claim, “The student’s primary language, ethnicity, and cultural background were taken into consideration prior to the selection of the assessment procedures,” rather than simply saying it was true or presenting a boilerplate statement as fact.

The second part of the boilerplate-language statement contains a statement that is not a claim but rather seems more like a reminder: “The tests chosen should be interpreted within the limits of their measured validity.” As stated earlier, since the burden of interpretation is on the writer of the report, not the reader, it is not clear for whom this advice is written. In addition, it is not clear what the writer means by “measured validity,” other than you should use the tests and procedures chosen for your evaluation for the purposes they were intended. Again, since it is the job of the evaluator rather than the reader to understand validity and use tests and procedures for their intended use, the statement simply confuses the reader and provides no legal confirmation that it is true.

Let us look at another commonly used boilerplate statement:

Ask yourself, “What is the author claiming?” and then, “What information would I be looking for to assess the truth of this statement?” Our guess is that the author is claiming that she conducted a comprehensive evaluation and used best practices in interpreting the data and making decisions. As we have argued, simply making a claim that something is true does not make it so. Ahead are different sections from one of our reports that, taken together, support the truth that the evaluation was comprehensive (i.e., the evaluation assessed all areas of suspected and related needs). They also demonstrate that data from multiple sources were integrated and synthesized for the reader to help support the Individualized Education Plan (IEP) team in their decisions for special education eligibility, services, and placements. Contrast the boilerplate-language statement with the following statements from the academic section from Max’s report:

Within the same section, we provide detailed information about Max’s reading skills to help the reader better understand his academic needs. Taken together, these statements clarify the reasoning behind our assessment procedure choices and demonstrate a comprehensive approach to assessing Max’s reading skills.

What Must Be True About Your Evaluations (and Therefore Reflected in Your Reports) According to Federal Legal Mandate?

IDEA’s guidelines form the legal basis for our practice. Although these guidelines do not specify what must be included in your report, IDEA has quite a bit to say about evaluations and much of this is contained in Sec. 300.304, Evaluation Procedures (IDEA, 2004). This section contains several important points that must be “true” about your evaluations. Remember, simply placing these statements in your report does not make them true. We have included the IDEA section 300.304 for your reference. In the remainder of the chapter, we highlight aspects of the law we believe are important to consider. We have organized these into themes, which include our recommendations of how to meet these legal mandates. These broad recommendations represent our thinking about how to approach these issues. Each of these topics is complex and could be its own book. For the sake of simplicity, we have divided these points into the following themes:

1. The evaluation should be comprehensive.
2. The evaluator should use a variety of assessment tools or approaches that gather functional and relevant data.
3. The evaluation should be fair.
4. The evaluator should be competent.
5. The procedures used should be valid and reliable.

The Evaluation Should Be Comprehensive

In several places, the federal law mandates that evaluations should be comprehensive. We believe this mandate can be divided into two parts. The first part focuses on assessing in all areas of suspected disability. The second part focuses on assessing in all areas of related need.

If we are required to assess in all areas of suspected disability and need, it is in no way best practice to assess everything imaginable to cover our bases. Instead, we must first determine what disabilities and areas of need are suspected. Determining this is largely a pre-assessment activity, which then drives the evaluation process. This seems self-evident but our experience is that many psychologists do not spend the time up front to establish these hypotheses, often writing an Assessment Plan with minimal knowledge of the concerns that led to the referral.

Evidence of this can be found in reports that state the reason for referral in the broadest terms, such as, The student was referred for an evaluation by the SST team due to academic concerns, or by practitioners who use the same set of assessment procedures for every student, regardless of referral reason or concern. We will further discuss this in Chapter 3, though for now it is important to plant the seed that the best ways to identify suspected areas of disabilities are through (a) good communication with the referring party, be it a parent or teacher, and (b) a solid review of existing information, including the students’ records, before the assessment plan is signed and the evaluation begins.

Legally, your obligation does not stop with suspected disabilities. We are also required to assess in all areas of related need. One way to conceptualize this is to imagine creating a list of the child’s challenges and needs. Some of these needs and challenges will be directly linked to the definition of a disability and thus inform what the suspected disabilities are. Others, although part of this individual child’s profile, may not be directly linked to a diagnosis or eligibility category. For example, take the disability of autism. The definition of autism in the federal statute is:

Autism means a developmental disability significantly affecting verbal and nonverbal communication and social interaction, generally evident before age three that adversely affects a child’s educational performance. Other characteristics often associated with autism are engagement in repetitive activities and stereotyped movements, resistance to environmental change or change in daily routines, and unusual responses to sensory experiences (Assistance to States for Education of Children with Disabilities Program and the Preschool Grants for Children with Disabilities Program, 2006).

If you are assessing a child and the concerns are that he has difficulty making and keeping friends and trouble adapting to changes in routine, these would perhaps suggest autism as a possible disability. This same child might have a great deal of anxiety or difficulty attending to academic tasks. These challenges, although commonly found in children with autism, are not part of the federal or most state definitions, though they would also need to be addressed in the evaluation. The bottom line, legally and professionally, is that you are responsible to assess in all areas on this hypothetical list.

It is important to develop an accurate list of concerns early in the evaluation process. This provides a focus for your evaluation and informs your choice of assessment procedures. Although it is always possible to discover something about a child you did not suspect going into an evaluation, it is our experience that most often it is feasible and practical to narrow the suspected disabilities to a small number of possibilities. In essence, these become our working hypotheses about the child’s disability classification. This sharpening of focus clarifies the evaluation process and prevents us from over-assessing children, making the assessment process less intrusive for children and teachers. It simultaneously saves us time and makes for a more coherent assessment. Surely this is a win-win for everyone involved.

This process of problem clarification and narrowing down of the suspected disabilities and relevant domains of need begins with parents and teachers, who are the primary source of referrals. It is our experience that the time invested early in the evaluation process clarifying consumers’ concerns saves time later and prevents the assessment from being a fishing expedition in search of a disability.

There are two broad types of data available to practitioners to help them identify suspected disabilities and areas of need. One is the hard data of group test scores (e.g., state, district, school), academic benchmarks, office referrals for discipline violations, screenings, and so forth. Indeed, given the federal mandates for data collection and progress monitoring of academic progress across different groups of students, schools often seem awash with these types of data. Unfortunately, they are frequently underutilized.

The second type of data is the soft data that come from teachers, parents, and even students in the form of their perceptions of the referral concerns. These perceptions are most often communicated in their responses to interview questions or written questionnaires. Interviewing teachers and parents about their perceptions of the problems can help practitioners frame the problem and develop hypotheses about the diagnoses (i.e., What IDEA disability categories are suspected in this situation?) and areas of need (i.e., What areas of functioning are of most concern for this child, even if not part of identifying the disability?). It is beyond the scope of this book to discuss all the instruments and possible interview protocols that might assist practitioners in doing this. We have successfully used the interview guide in Appendix III with children, parents, and teachers. It reflects our bias toward developing an understanding of strengths as well as possible diagnoses and areas of need, which, while not strictly speaking a legal requirement, provides a more comprehensive picture of children’s functioning.

The softer data of teacher or parent referrals are subject to the biases that arise from differing performance expectations. There is evidence that more objective data, such as curriculum-based measurement (CBM) or systematic behavioral screening tools, can help reduce bias in referrals and identify children for early intervention (Kamphaus, DiStefano, Dowdy, Eklund, & Dunn, 2010; Minneapolis Public Schools, 2001; Raines, Dever, Kamphaus, & Roach, 2012; Severson, Walker, Hope-Doolittle, Krotochwill, & Gresham, 2007). Yet, the soft data of consumers’ concerns expressed in their own terms help us understand the problem in the context of a particular classroom or family. Both forms of data have value and can be helpful in assisting us in meeting the legal mandate of assessing in all areas of suspected disabilities and domains of need.

The Evaluator Should Use a Variety of Assessment Tools or Approaches That Gather Functional and Relevant Data

The second theme in meeting legal assessment mandates again has two parts and focuses on (1) using a variety of means to gather data, and (2) gathering data that are both functional and relevant. The law does not provide guidance on how we might interpret variety, but we are certain it does not simply mean more tests. More is not always better. One way to understand what constitutes variation as part of a comprehensive assessment is the acronym RIOT (Leung, 1993). In this model, the mnemonic RIOT or (a) record review/history, (b) interviews, (c) observations, and (d) tests, guides the assessor through a comprehensive assessment by focusing on a variety of assessment procedures that provide a distinct range of information. The evaluation entails gathering and interpreting information about the student in a variety of settings and under varied conditions. Merrell suggests another rubric for judging the comprehensiveness of evaluations with the Rule of Two (Levitt & Merrell, 2009). Like RIOT, the Rule of Two focuses on the principle that a comprehensive assessment must have multiple elements, but as Levitt and Merrell put it, “acknowledges the reality that resources and time are often limited” (2009, p. 19). The Rule of Two suggests that a comprehensive assessment should include information from a minimum of two settings, two informants, and two assessment methods. We suggest the integration of the Rule of Two and the framework of RIOT as a way to incorporate a variety of assessment tools and data sources into your evaluation.

Besides using a variety of assessment tools, we must gather data that are both functional and relevant. The notion of functional is not defined in IDEA but the subsequent regulations include the following comment:

It is not necessary to include a definition of “functional” in these regulations because we believe it is a term that is generally understood to refer to skills or activities that are not considered academic or related to a child’s academic achievement. Instead, “functional” is often used in the context of routine activities of everyday living. (Federal Register/Vol. 71, No. 156/Monday, August 14, 2006/Rules and Regulations, p. 4661)

Given this position, we believe it is reasonable to interpret functional as similar to what is often described as adaptive behaviors. Although often used narrowly in the context of the diagnosis of intellectual disabilities, adaptive behavior is a wider construct that is important for all children. Sternberg (1985) provides another perspective on the definition of functional in his description of practical or contextual intelligence, which he describes as the ability to adapt to the demands of an environment. According to Sternberg (1985), practical intelligence “deals with the mental activity involved in attaining fit to context” (p. 45).

Adaptive behavior has been described in different ways but a consensus has emerged that it can be divided into three broad areas: (1) practical skills such as self-care, use of transportation, and so on; (2) conceptual skills such as the application of academic skills to everyday situations; and (3) social and interpersonal skills (Tassé et al., 2012). We would add two more areas to provide a fuller picture of functional skills: (4) the planning and organizational skills commonly associated with the neuropsychological construct of executive functioning; and (5) intrapersonal skills such as emotional self-regulation.

The implication of the mandate to gather functional information is clear. We must extend the focus of our evaluations beyond the purely academic and cognitive to include life skills that are related to success in all school environments, including outside of the classroom (i.e., playground, cafeteria, auditorium, etc.) and, in some cases, the home environment. Given the everyday nature of these life skills, we believe it is not sufficient to describe a child’s functioning in these areas purely in terms of performance on standardized tests. We view this legal mandate as requiring us to include data about performance in the classroom, on the playground, and in the home, where functional skills would be called upon. At a minimum, we recommend that the evaluation, including the pre-referral process we describe on page 19 include a screening for these important functional skills to determine if they are potential areas of need.

We suggest the use of Table 2.1 as a way of thinking about your assessments and understanding if you have gathered functional information as part of your assessments. The table is designed to help you decide whether the assessment procedures you have chosen are providing you with functional data. Each assessment procedure is given a score from 0 to 2, representing the quality of the data the assessment procedure provides.

In Table 2.2, we provide an example of an analysis of the information collected during a learning disabilities evaluation for Lisa, a second-grade student whose main academic concern is reading and reading-related anxiety that is impacting school attendance. During this evaluation, we used the WISC-V as part of the cognitive assessment. The WISC-V data provide extensive standardized information about Lisa’s overall cognitive ability and perhaps support for underlying hypotheses, though it provides no functional data regarding her reading difficulties or anxiety. The WIAT-III, a standardized achievement assessment, provides some general standardized academic data that could begin to identify areas of reading need; however, the WIAT-III was not designed as a curriculum-based assessment and Lisa’s performance on this measure may not align with the reading strengths and weaknesses she demonstrates in the classroom. The Behavior Assessment System for Children–3: Parent & Teacher Rating Systems provides us with standardized information regarding the parents’ and teacher’s perspective of Lisa’s problem and adaptive behaviors, including anxiety. If this information is further used during an interview, we can gather very specific information about Lisa’s anxiety, as well as potential strengths. The next two data sources we used, the CBM Reading Fluency Rate and Open Court Unit Test scores, will provide us some comparison data with her classmates, but also with clear functional data on how Lisa is performing in the classroom with the language arts curriculum. Lastly, the observations during Guided Reading and interviews with the teacher, parent, and student will provide us with very specific functional data about both the reading skills and the impact anxiety is having on this student. The goal is not to reach a specific total or to have ratings of 2 across all assessment procedures. Rather, through integrating these multiple sources of information, we are able to meet the challenge of incorporating functional data into our evaluation.

The concept of gathering relevant information is more straightforward. The information we gather must be helpful in identifying educational needs and assisting those who work with the child in meeting those needs. It is important to note that although determining a disability classification or diagnosis can be helpful, it is often of limited value in identifying specific educational needs or planning interventions. For example, when a parent or teacher learns that a child has a learning disability, the diagnosis can help organize their thinking about the child but does not necessarily help in determining specific goals or strategies. We believe that to meet the letter and spirit of the law, our assessments (and the reports we write) must do two things to be relevant: (1) Help consumers better understand children and their strengths and challenges and (2) help those who work with children understand their educational and functional needs and how to better meet those needs. Table 2.3 is designed to help you determine if you have gathered relevant information as part of your assessment. Each assessment procedure is given a score from 0 to 2, representing the relevance of the data the procedure provides. There is no specific number we are trying to achieve when using this table. The table simply helps us reflect on our evaluation and ensure we have used a variety of data sources and have gathered relevant data.

In Table 2.4, we take the example of Lisa’s assessment further to review the relevance of the collected data. Using our standardized and functional data, Lisa was identified as having weak phonics-based decoding skills. Although we could probably identify this concern with data from the standardized measures used, such as the Wechsler Individual Achievement Test–III and the Comprehensive Test of Phonological Processing, neither of these measures is precise enough to provide the relevant data needed to write academic goals or create interventions. Think about the results you get from a standardized achievement test. Is a standard score from an index or subtest, such as, Lisa’s score on the Word Reading subtest was 88, enough information to write a quality IEP goal or help the teacher create an intervention? No, it is not. Through the use of curriculum-based tools, such as a district-created phonemic awareness test, we can identify specific phonemic awareness or phonics-based decoding skills that Lisa is missing, such as consonant sounds, long vowel sounds, short vowel sounds, consonant blends, sound blending, segmenting, syllabication, rhyming, and so forth. These relevant data can help us write a clear and measurable goal as well as give us insight into appropriate interventions.

The Evaluation Should Be Fair

Assessment should be designed to yield the best results for a child. Specifically, the law says two things: The tools used as part of the assessment (1) should not be racially or culturally biased and (2) should be chosen to yield accurate information about a child, given the child’s language and culture but also limitations in his or her sensory (vision and hearing), motor, and speaking skills.

Although the mandate that the assessment be fair is clear, the process of how one does this is less straightforward. Much of the literature has focused on the complicated issue of fairness in the assessment of bilingual and bicultural youth (e.g., Figueroa & Newsome, 2006; Lau & Blatchey, 2009; Olvera & Gomez-Cerrillo, 2011; Ortiz, 2008; Ortiz, Flanagan, & Dynda, 2008; Rhodes, Ochoa, & Ortiz, 2005). Two models of how to approach this issue systematically have been developed by Olvera and Gomez-Cerrillo (2011) and Rhodes, Ochoa, and Ortiz (2005). Olvera and Gomez-Cerrillo’s MODEL approach (2011) shares many of the attributes we discussed in the section on how to ensure assessments are comprehensive. MODEL stands for (a) multiple sources of information, (b) observation, (c) data-driven hypothesis, (d) English-language development, and (e) language of assessment. Like RIOT (Leung, 1993), MODEL gives practitioners guidance on how to judge the comprehensiveness of an assessment, but does this in the context of working with bilingual and bicultural youth.

Ochoa and Ortiz’s (2005) model helps practitioners determine the best language(s) to use in the assessment to yield the most accurate results. They recommend that practitioners consider: (a) the grade of student; (b) the mode of assessment (i.e., reduced culture and language “load,” native language, English, or both languages); (c) the current and previous types of educational program (e.g., native language, bilingual education, sheltered English, pullout English-language development, etc.); and (d) the child’s proficiency in both English and the native language (Rhodes, Ochoa, & Ortiz, 2005, p. 169).

In terms of making your assessments and the reports you write legally defensible, regardless of the model you use, it is important for practitioners to document a systematic process by which they have determined the fairest way to approach an assessment. Rather than making the unsupported generic boilerplate statements we discussed earlier in the chapter, reports should provide specific information about how these decisions were made for a particular child in a specific context.

The following are examples of what this might look like in reports. We do not believe they are perfect but rather demonstrate a good-faith effort to make our thinking about evaluation fairness transparent to the reader.

The Evaluator Should Be Competent

How can someone make a legal judgment regarding your competence? There is no professional or official consensus for a legal standard of expertise, although several attributes can be taken into consideration. These include: (a) years of experience; (b) accreditations (e.g., licenses, certificates, etc.); (c) peer identification (i.e., Is this person well known in the field?); (d) between-experts reliability (i.e., Are this person’s judgments similar to other experts?); (e) within-person reliability (i.e., Are this person’s judgments reliable over time?); (f) subject matter expertise (i.e., using an acknowledged “super-expert” as a reference point); and (g) factual knowledge (e.g., how well you can respond to questions) (Shanteau, Weiss, Thomas, & Pounds, 2003).

Considering these attributes, we believe there are two general answers to how our competence is judged. One is professional experience and the other is scope of practice. Professional experience incorporates years of experience and knowledge of types of assessment and assessment tools. Professional experience also can encompass experience with a particular population, including special education eligibility categories, mental health or medical diagnoses, school placements, ethnicity, or socioeconomic status. In general, it represents our acquired experience. For example, both of us are competent in all areas of school psychology (i.e., basic expertise in our scope of practice), yet Michael has extensive experience with mental health problems and assessment related to the special education category of Emotional Disturbance. In contrast, Jeanne Anne has considerable experience with children from diverse cultural and socioeconomic backgrounds, the disability category of Autism, and the functional assessment of academic skills. In a legal context, all of our experiences would bolster the argument for general competence and expertise in specific areas.

The second aspect of legal competence is scope of practice and is reflected by our qualifications, licenses, and certifications. School psychologists’ range of practice can be broad. We can be engaged in psychoeducational evaluations, suicide or threat assessments, parent–teacher consultations, group counseling, and providing professional development. It is common for some of us to participate in all of these activities in the course of a day. Our scope of practice is directly related to, but not limited by, our professional experience. All school psychologists must meet state certification requirements to practice in the schools, though many school psychologists have extra certifications, licenses, or trainings that expand their scope of practice.

We think it is important to review the scope of practice of school psychologists because we often have conversations with practitioners where they have set artificial limits on their ability to address problems in the schools. These include things like, “We can’t identify children with that because it is a medical diagnosis,” or “You have to be licensed as a therapist or clinical psychologist to provide that service.” Of course, we should always practice within our experience and training, but it is also important to know that in schools our roles are broadly defined to include a wide range of skills and services. For example, the National Association of School Psychologists (NASP) describes the work of school psychologists thus:

School psychologists help children and youth succeed academically, socially, behaviorally, and emotionally. They collaborate with educators, parents, and other professionals to create safe, healthy, and supportive learning environments that strengthen connections between home, school, and the community for all students. (National Association of School Psychologists, n.d.)

The NASP Blue Print III (2006) goes on to define four functional competencies that school psychologists should have:

1. Data-based decision making and accountability
2. Systems-based service delivery
3. Enhancing the development of cognitive and academic skills
4. Enhancing the development of wellness, social skills, mental health, and life competencies

These competencies are a required part of the training students receive in any NASP-approved graduate program. We can assume from the breadth of these standards that the national organization that represents school psychologists and determines our ethical standards sees the scope of practice as including systems and individuals as well as encompassing cognitive, academic, and social-emotional development.

Scope of practice will differ somewhat from state to state but will have similarities. For example, in the California Education Code, school psychologists and their services are described in Section 49424:

A school psychologist is a credentialed professional whose primary objective is the application of scientific principles of learning and behavior to ameliorate school-related problems and to facilitate the learning and development of children in the public schools of California. To accomplish this objective the school psychologist provides services to children, teachers, parents, community agencies, and the school system itself. These services include:

1. Consultation with school administrators concerning appropriate learning objectives for children, planning of developmental and remedial programs for pupils in regular and special school programs, and the development of educational experimentation and evaluation.
2. Consultation with teachers in the development and implementation of classroom methods and procedures designed to facilitate pupil learning and to overcome learning and behavior disorders.
3. Consultation with parents to assist in understanding the learning and adjustment processes of children.
4. Consultation with community agencies, such as probation departments, mental health clinics, and welfare departments, concerning pupils who are being served by such community agencies.
5. Consultation and supervision of pupil personnel services workers.
6. Psychoeducational assessment and diagnosis of specific learning and behavioral disabilities, including, but not limited to, case study evaluation, recommendations for remediation or placement, and periodic reevaluation of such children.
7. Psychological counseling of and other therapeutic techniques with children and parents, including parent education.

We have reviewed several other states’ descriptions of what services school psychologists provide and it is clear that, like California, their scope of practice allows them to provide a wide range of evaluation, consultation, and treatment services in schools. There is no doubt that when it comes to assessment for special education purposes, school psychologists would be judged as legally competent.

The Procedures Used Should Be Valid and Reliable

The final theme involves using assessment procedures that are both valid and reliable. Validity and reliability are complex issues and it often intimidates school psychologists when they find out that the requirement to use valid and reliable measures falls on them. In other words, it is our responsibility to establish whether a tool we are using is valid and reliable. It is not the responsibility of our school district, which may very well purchase and own tools that do not meet the standards for validity and reliability described in the American Psychological Association’s (APA) Standards for Educational and Psychological Testing (American Educational Research Association [AERA], APA, & National Council on Measurement in Education, 2013).

We will begin with reliability, because a test cannot be valid if it is not reliable. Reliability refers to the consistency of a measure. Does the measure provide stable results when the testing procedure is repeated? For example, if a test is designed to measure cognitive ability, then each time the test is administered to a student, the results should be approximately the same. We use the word approximately, because even under standardized conditions, variations in performance are going to be reflected in the student’s scores on the measure. We call this measurement error. Measurement error is the difference between a student’s observed or obtained score and his or her hypothetical “true” score. Reliability provides a measure of the extent to which an examinee’s score on a test reflects random measurement error.

When a student is given a standardized measure, measurement error is caused by factors specific to the student (e.g., motivation, hunger, fatigue), factors specific to the test or measure (e.g., unclear directions, poor choice discrimination), and factors specific to scoring (e.g., vague scoring criteria, computational errors). In a more reliable test, these errors are minimalized, though they cannot be eliminated. In a less reliable test, a child’s performance can vary significantly and thus be less dependable, especially when making high-stakes decisions (e.g., does this score represent evidence of a significant weakness, suggesting a disability?). Lastly, composite and index scores are often more reliable (i.e., have less measurement error) than individual subtest scores, which is why interpretation at the subtest level can be inappropriate.

In the end, it is our responsibility to choose and use reliable measures (remember, not the readers of your reports!). Fortunately, reliability evidence for standardized measures typically can be found in the test manual (consider it a red flag if you cannot find it!). This allows us, as practitioners, to compare measures and choose appropriate instruments.

According to the Standards for Educational and Psychological Testing, when scores are to be interpreted and, we would add, described in a report, relevant reliability and measurement error should also be reported. We believe using confidence intervals instead of reporting obtained single scores is an easy-to-understand approach to conveying this information (e.g., not “Rina earned a 100” but rather “There is a 90% chance that Rina’s score falls between 90 and 110”). Confidence intervals are an estimated range or band of scores that contain the obtained score plus or minus a margin of error. Test manuals routinely provide information on confidence intervals at the 68%, 90%, and 95% levels. For example, when a 95% confidence interval is used, we are providing a statement of reliability that we are 95% sure that the student’s true score falls within this range of scores (AERA, APA, National Council on Measurement in Education, 2013). There are, of course, different arguments to be made for different levels of confidence, but we have generally settled on 90% as a minimal level of acceptance.

For a test to be valid, evidence and theory need to support or demonstrate a relationship between the test and the behavior it is reported to measure. Valid measures are effective because they provide accurate information on the construct the measure was designed to assess. Simply put, validity is the extent to which a test measures what it says it measures. For example, when getting a driver’s license, the formal driving test given through the Department of Motor Vehicles is probably a more valid measure of your actual ability to drive on city streets than the written test that is also given. Conversely, the written test may be a more valid measure of your knowledge of your state’s driving laws than the formal driving test. Unfortunately, the concept of validity is not as clear in psychological testing.

Validity is an important concept because it is fundamental to appropriate test interpretation. Test performance must be interpreted within the construct the test proposes to assess. Many test developers provide claims of validity in their manuals, though they do not provide evidence to support that their measure is more valid than another method for gathering similar information. For example, a standardized test of written language ability might focus on having students write complete sentences in response to prompts. This test does not measure the ability to compose a multi-paragraph essay, or use persuasive language in writing, which we could argue are equally valid aspects of the construct of written language ability.

Test interpretation is also influenced by extraneous circumstances that are not related to the measure itself or the construct assessed. Perhaps the student had an emotional response to the writing prompt, the testing environment was loud, the student misunderstood the directions, or the evaluator made an error in test administration. In summary, validation is the mutual responsibility of both the test developer and the test user. The test developer should present evidence that his or her measure is appropriate for its intended use. The test user is responsible for evaluating the test developer’s claims and then using and interpreting the test correctly.

In this chapter, we have reviewed the federal guidelines set forth by IDEA and the subsequent regulations regarding the legal aspects of assessment and report writing. Unfortunately, most of the legal guidelines for assessment are stated very generally. We are left to define many of these mandates ourselves. In this chapter, we have tried to provide brief descriptions of best practice to draw upon to fill in these blanks. One of the takeaway messages is that there is no great separation between best practices and the notion of legal defensibility. Another important message from this chapter is that our reports serve as evidence that we have met the legal obligations of our evaluations. The critical word is evidence. It is not enough to tell the reader that we have met these legal mandates but we are obligated to show the reader how this was accomplished.

It is our view that the answer to how to make a report legally defensible is to conduct an assessment that is fair and comprehensive and uses a variety of procedures designed to gather functional and relevant information, so we can respond directly to the concerns raised by the parents and teachers who initiated the assessment. In the following chapter, we will answer the question, “How do I make my reports more useful to consumers?” We will introduce the concept of useful and review our main strategies for how to make your reports more understandable and accessible for consumers, specifically parents and teachers.