Why Is Assessment of Service-Learning Important?

What Are the Different Types of Service-Learning Assessment? What Purposes Do They Serve?

What Are the Elements of a Comprehensive Assessment Plan?

Assessment of service-learning enables its practitioners, participants, supporters, advocates, and funders to gain an understanding of its value to students, faculty, community leaders and members, the institution, and to higher education and society. It can also reveal its effectiveness in achieving its desired outcomes, its cost-benefit ratio for all its constituencies, and what is required to enhance its value and outcomes.

The term assessment can be used as an umbrella for other terms—such as counting, evaluation, benchmarking, outcomes measures, and research—that refer to the systematic gathering of information and processing it to reach conclusions (Hatcher & Bringle, 2010). In the context of student learning and development, assessment also describes the process of determining the extent to which a particular outcome or set of outcomes has been achieved by an individual or a group. In this book, I have used both Hatcher and Bringle's broad definition of assessment and the more specific definition of assessment related primarily to learning outcomes.

Because service-learning is a complex process and involves multiple stakeholders, several forms and levels of assessment are required. An overview of the potential components of a comprehensive plan for the assessment of service-learning follows.

Counting in service-learning helps to answer questions related to numbers of service-learning courses, community partner organizations, student participants, service hours, tons of trash hauled out of a river during a cleanup, children tutored, and the like. While numbers do not reflect the impact of service-learning on students, communities, and institutions, they are one measure of output. Numbers are required in grant applications and other requests for funding, annual reports, national surveys like Campus Compact's Member Survey, and applications for recognition, such as the President's Community Service Honor Roll and the Carnegie Elective Community Engagement Classification.

Evaluation, like counting, is also about outputs. It can measure the quality and effectiveness of a program or course, as well as participant satisfaction. Its focus is on a specific intervention, and it is primarily used internally to improve design and practice. Evaluation can be used to measure how consistent the implementation of a course or program was with its design, whether it kept to its timeline, whether its actual costs matched the budget, and how well it met its goals (Hatcher & Bringle, 2010). Along with counting, evaluation is the most common type of assessment required in reports on grant-funded programs. The findings of evaluations are usually not generalizable.

Benchmarking is often used to determine how one or more aspects of an institution's service-learning program compare with an established standard, programs at other institutions nationally, or programs at peer or aspirational-peer institutions. It is fairly easy to gather and compare data regarding numbers of courses, participants, or staff members; types of curricular or cocurricular offerings; and practices employed. It is more challenging to compare results, such as impacts on students and communities, although there are some multi-institutional groups that are doing this work.

Service-learning outcomes assessment, which is a more specific definition of assessment, measures the extent to which desired outcomes are achieved for students, communities, faculty, and institutions. Its purpose is to gather, analyze, and interpret various forms of evidence to increase outcome attainment by improving practice.

In the context of service-learning, research is systematic, scientific inquiry that is designed to collect, analyze, interpret, and use data to understand, describe, predict, or control an educational phenomenon (Mertens, 2005). It produces information about why specific outcomes do or do not occur or why a program produces or does not produce a particular result. Research both uses and guides theoretical and conceptual frameworks. High-quality research can generalize findings so that they can be used beyond the specific study population.

Assessment, generally speaking, can occur on multiple levels. When the individual is the unit of analysis, assessment seeks to determine changes in individuals or their circumstances. In the case of service-learning student participants, one could examine a student's achievement of learning outcomes in a particular course or cocurricular experience, in the major, as a result of several service-learning experiences, or across the entire college experience. For individual clients of a community partner organization, one could measure, for example, improvement in a child's reading comprehension, a recent immigrant's language skills, or an elderly person's ability to use small motor skills.

Individuals' data can be sampled and aggregated for analysis at course, major, program, experience, or institutional levels. For example, sampled, aggregated, and summarized data related to achievement of particular learning outcomes in a first-year service-learning seminar in a specific major could be compared with sampled, aggregated, and summarized data from a senior capstone course in the same major to determine the extent to which students have improved in critical thinking. Aggregated student data from assessment of achievement of learning outcomes is also required by regional accrediting associations. Disaggregated data can reveal how well subgroups of students are succeeding. As far as the community is concerned, aggregated data can be used to examine whether engaging with service-learning has had an overall positive impact on the lives of individuals in the community. For example, in an ongoing partnership between a faculty member who teaches a service-learning course in public health and a community where there are high numbers of people with obesity and diabetes, the students have provided health education and screening to residents over six semesters. Aggregated data from a door-to-door survey of community residents could be used to determine whether residents' health has improved during the period of the students' interventions. Disaggregated data could show, for instance, that women's health improved more than men's, enabling the faculty member and students to consider alternative ways to deliver their message to male residents.

Assessment at programmatic and institutional levels can also examine such areas as infrastructure for service-learning, environmental factors that act as supports and barriers for service-learning, and cost-effectiveness. Assessment of campus-community service-learning partnerships can address these areas as well as whether the partnership has yielded changes in the broad structures that affect the life of the community.

A comprehensive assessment plan for service-learning includes a variety of types and levels of assessment. The development of an assessment plan can be approached in several ways, ranging from reactive to proactive. For example, a simple plan could begin with a list of requirements for data on an annual basis, such as institutional reports, national surveys, and recognition and award applications, together with a list of sources of existing data from which information can be drawn, such as course catalogs, faculty assessments of learning in service-learning courses, and routine program evaluations. By comparing the requirements with existing data, it is possible to determine what assessments need to be added. A more proactive and useful assessment plan might be based on a grid that identifies desired outcomes—for students, faculty, community partners, and the institution—and the service-learning experiences that are designed to address these outcomes (Hatcher & Bringle, 2010). An assessment team can then determine which types and methods of assessment would provide information about the extent to which the outcomes are achieved. It is unrealistic to assess every outcome annually, but the plan can be used to develop a schedule of which assessments will be conducted in a particular year.

Why Assess the Impact of Service-Learning on Students?

In addition to understanding the effects of service-learning on student participants in individual courses or cocurricular experiences, there are many other reasons to assess student impacts. Faculty and staff members who engage students in service-learning naturally want to know whether their desired learning outcomes were achieved and what it was about the experiences that led to their achievement or lack of achievement. They can then use the data to improve their practice. Service-learning educators also find themselves in the position of explaining to students why they should participate and, thus, find it useful to be able to call upon documentation of how service-learning contributes to student learning and development. Faculty members also can use assessment data to demonstrate to colleagues the academic rigor of service-learning for inclusion in their teaching portfolios and for encouraging others to adopt the pedagogy. As 7.7 indicates, assessment data can also go a long way to demonstrate the value of service-learning in the pursuit of institutional support and in fundraising.

There are various ways to categorize potential effects of service-learning on students. Janet Eyler and Dwight E. Giles, Jr., identify six broad categories of student impact: personal and interpersonal development; understanding and applying knowledge; engagement, curiosity, and reflective practice; critical thinking; perspective transformation; and citizenship (1999). Gelmon, Holland, Driscoll, Spring, and Kerrigan answer the question, “What do we want to know?” with this list of potential student outcomes: awareness of community issues, assets, and needs; quality and quantity of interactions with the community; present and future commitment to service; career decision making and professional skill development; awareness of personal strengths and limits and changes in preconceived beliefs; understanding and applying course content; self-confidence and comfort in diverse settings; sense of ownership of responsibility for outcome of community project and role as learner; demonstrated abilities in oral and written communication and recognition of their importance; and valuing the pedagogy of multiple teachers, including peers, community partners, and faculty (2001, p. 28). Question 6.2 describes various methods that can be utilized in assessing the impacts of service-learning on students. Questions 2.3 and 4.6 address assessment of student learning through reflection and in academic courses, respectively. Question 5.7 covers assessment of cocurricular service-learning experiences. An extensive compilation of scales that can be used to measure a wide range of student outcomes can be found in The Measure of Service Learning: Research Scales to Assess Student Experiences (Bringle, Phillips, & Hudson, 2004).

Although they were not designed specifically for service-learning, the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics developed by AAC&U are being used by more than 1,000 colleges and universities. The sixteen rubrics are designed for institution-level assessment of AAC&U's Liberal Education and America's Promise (LEAP) Essential Learning Outcomes: civic engagement, creative thinking, critical thinking, ethical reasoning, foundations and skills for lifelong learning, information literacy, inquiry and analysis, integrative and applied learning, intercultural knowledge and competence, oral communication, problem solving, quantitative literacy, reading, teamwork, written communication, and global learning (Rhodes & Finley, 2013). These outcomes, which represent a consensus among educators and employers about the preparation students need to be successful in “civic life and the global economy,” also represent the desired outcomes of curricular service-learning (Rhodes & Finley, 2013, p. 5).

All the regional accrediting bodies have embraced the VALUE rubrics as an acceptable approach for assessment of student learning. The rubrics are useful for both formative and summative assessment within individual disciplines and across general education programs. While they were not developed as grading rubrics, they “can be translated into grading rubrics for specific courses, using the same criteria or dimensions for learning, but the performance descriptors would need to be modified to reflect the course content and assignments being examined, while still preserving the dimensions of learning in the original rubric” (Rhodes & Finley, 2013, pp. 6–7; italics in original). Thus, the rubrics can be adapted to assess either individual service-learning courses or all service-learning courses at a particular institution. Terrel L. Rhodes and Ashley Finley (2013) offer guidance and examples of how the rubrics can be modified for specific institution-wide and course-based assessment of student learning.

How Can We Tell Whether Service-Learning Is Worth the Cost-Benefit Ratio to the Community Partner?

How Can We Measure Changes in Individuals or Systems?

There is no doubt that there is far more evidence about the effects of service-learning on students than about its effects on the community. Service-learning educators are challenged by such issues as: What community impacts should we measure? Should we burden our community partner with unwieldy assessment responsibilities? Where do we find the time and expertise to do high-quality assessment beyond simple counts?

As discussed in Chapter Three, potential campus and community partners should hold early conversations about the outcomes each seeks from the relationship and what criteria and measures will be used to assess the extent to which the desired outcomes are achieved. Randy Stoecker and Elizabeth A. Tryon rightly point out that “turning the ship of service learning to point to community outcomes rather than primarily student outcomes actually requires agency staff to involve themselves in steering that ship” (2009, p. 180). However, while campus resources for assessment are often limited, they are likely to be even more limited on the part of the community. As a result, it is important to be realistic and to carefully prioritize what specific information is most needed at what point in time and for what purposes. If, for example, the partnership is funded by a grant, then the information required by the grantor for reports or applications for additional funding may be primary. It is also worthwhile for the partners to discuss what information is easiest to collect that would also be useful in assessing the extent to which community outcomes were achieved. For instance, counting and use of self-report surveys are simple, objective methods that can yield valuable data. How many meals were packaged and delivered? Were more clients of the community partner organization served? Were there shorter wait times by clients for services? Were there more hits on the organization's website? Brief one-time surveys administered orally or in writing to the organization's on-site clients can assess whether there is increased satisfaction with the quality of information or service.

Most service-learning centers conduct routine evaluations of the benefits community partners believe they received and the level of their satisfaction with their participation in service-learning. Subjective questions related to satisfaction of community partners might address whether the students had sufficient knowledge and skills to complete their responsibilities, whether the number of service hours was adequate, and whether there was sufficient communication with campus partners. Formative assessment, even if done informally, is helpful in identifying areas of concern and making adjustments in them before they become insurmountable problems.

As a partnership progresses, additional outcomes are generally sought and more joint activities are planned. Further assessment of the extent to which those outcomes are achieved requires the use of more complex assessment measures. This is especially true if the desired outcomes include positive effects on individuals, groups, and systems. These assessments can be undertaken by students in service-learning courses or independent studies that involve community-based research, graduate students seeking topics for theses and dissertations, and teams of students in courses in measurement and statistics, sociology, education, business, or other fields related to the work of the community partner. A simple but excellent worksheet that can be used to analyze the cost-benefit ratio of participation in service-learning for a community organization is included as Exhibit 6.1. It can enable individuals without extensive knowledge or experience to conduct this assessment.

In summary, evaluation and assessment of service-learning from the community perspective should make results meaningful for the community audience, report at least some results as quickly as possible, determine the organization's cost-benefit ratio, and be sure that the evaluation reveals something that is not already known (Dewar, 1997). Hopefully, a review of the assessment results will enable both partners to make adjustments that will increase the likelihood of positive results in future assessments. It is important to note that assessment results may indicate that the campus-community partnership, like any other partnership, simply is not working and that the partners should consider whether it would be best to terminate it.

How Should We Assess the Quality of Service-Learning Teaching?

What Do We Need to Know About the Impact of Service-Learning on Faculty?

How Should We Assess the Benefits and Costs of Service-Learning for Faculty?

In the context of service-learning, the focus of faculty assessment is generally construed as examining the quality of the teaching of service-learning courses. However, it is also important to assess how service-learning affects teaching, learning, and scholarship. Assessment of faculty perceptions of service-learning can inform efforts to increase the satisfaction of service-learning faculty and the number of faculty members who integrate it into their teaching.

Several of the methods described in 6.2 apply to the assessment of the quality of service-learning teaching. One of the simplest is using document review to analyze service-learning syllabi or faculty curriculum vitae. Syllabus analysis is straightforward and is necessary at institutions where there are criteria for service-learning course designation, as described in 4.10. A checklist can be devised for syllabus analysis based on the course designation criteria or, if there is no official service-learning designation, on the Principles of Good Practice for Service-Learning Pedagogy (Howard, 2001) in Exhibit 4.1 or on the list of unique elements of a service-learning syllabus that can be found in 4.7. Analysis of curriculum vitae can be used to learn the extent of faculty engagement with service-learning and, when compared with tenure and promotion results, the effects of service-learning on professional advancement. Criteria for analysis of curriculum vitae might include evidence of integration of service-learning into courses and independent studies, grants based on community-based research or teaching, professional presentations and publications, and community projects, presentations, or reports.

Classroom observation can be an effective means of learning about teaching techniques that can be used to improve service-learning teaching, both for the individual observed and for others. The use of trained peer evaluators is common practice and is readily applicable to service-learning. Experienced service-learning faculty members, equipped with a checklist and questions to use in preparing a narrative report, can gather information related to such topics as use of class time devoted to lecture, group work, reflection, and discussion; relationships between the faculty members and the students; quality of student participation and engagement; and connections between academic content and community experiences.

Student evaluations of teaching can also be useful indicators both to individual faculty members for use in improving their courses and in aggregate form to assess the effectiveness of service-learning teaching in general. Because standard faculty evaluation surveys do not usually provide enough specific feedback about the unique aspects of service-learning, it is wise to supplement these evaluations with questions specifically related to teaching in the context of service-learning. Such questions can either be in the form of a survey or incorporated into reflection.

Because faculty members are such key players in the service-learning enterprise, it is essential to assess the impacts of service-learning on faculty. Understanding their perceived benefits and challenges can facilitate efforts to motivate faculty members to practice service-learning and to sustain them so that they remain involved in the work. Given the time and effort a comprehensive assessment would entail, it is appropriate to select one or two most salient areas to assess at a time. Areas to consider for assessment of the effects of service-learning on faculty, together with sample questions, include:

• Student learning. Do students learn more academic content? Does service-learning take too much time away from content learning? Do students improve skills? Do they increase their understanding of community issues? Do they learn how the discipline can address social issues? Do you find it too challenging to assess student learning through service-learning?
• Enhancement of teaching. Does service-learning improve your teaching? Does it encourage you to try other new pedagogies? Do you enjoy teaching more? Less? Does it improve your relationship with students? Does it help you understand your professional strengths and weaknesses? Does it take too much of your time?
• Effects on scholarship. Does service-learning help clarify focus areas for your scholarship? Does it open up possibilities for developing research partnerships with the community? What have you learned by working with the community? Does service-learning distract you from your research?
• Effects on career. How do colleagues perceive your work with service-learning? Do your colleagues advise you to concentrate on traditional teaching and scholarship? Do you expect service-learning to enhance or detract from your professional portfolio?
• Professional development and support. What topics need to be addressed in faculty development? What kinds of support have you found helpful? What additional supports are necessary?

As far as methods, most of those mentioned in 6.2 would be useful in assessing the impacts of service-learning on faculty. It is common practice for the service-learning center to, at a minimum, administer a brief survey at the end of each semester to service-learning faculty. It is beneficial to supplement such surveys with interviews, focus groups, or faculty members' reflective journals to address particular areas of concern that may be identified by analyzing the survey results. It is also worthwhile to consider assessing faculty who are not involved in service-learning to learn what they know about service-learning, what might motivate them to consider adopting it for one of their courses, and what has deterred them from using it.

How Can We Assess the Benefits to the Institution of Service-Learning?

How Should We Assess Our Institutional Infrastructure to Support Service-Learning? Our Institutional Commitment?

How Can We Identify the Internal Obstacles to Service-Learning?

There are several reasons why assessment of service-learning from the institutional perspective is a critical aspect of the overall assessment plan. Service-learning is always strongly influenced by the institutional environment (Gelmon, Holland, Driscoll, Spring, & Kerrigan, 2001). It cannot achieve its substantial potential benefits for students, faculty, the community, and the institution, as enumerated in 1.4, without institution-level commitment and support. Because broad institutional support is required, administrators, board members, and funders need to understand its benefits. They also need to know what barriers have to be addressed. In addition, service-learning initiatives cannot survive and thrive if they are isolated in one corner or pocket of the campus. As 3.6 illustrates, successful, sustainable service-learning should be institutionalized so that it becomes an ongoing, legitimate, and valued element of the institution's organizational culture.

Several frameworks exist that can be used for assessment of the degree of institutionalization of service-learning. The Self-Assessment Rubric for the Institutionalization of Service-Learning in Higher Education (Furco, 2002) is comprehensive and straightforward. It is based on five dimensions, each of which has several components: philosophy and mission of service-learning, faculty support and involvement, student support and involvement, community participation and partnerships, and institutional support. The rubric comprises three stages of institutionalization: critical mass building, quality building, and sustained institutionalization. The components of each of the five dimensions are listed in Exhibit 6.2. The entire rubric, together with instructions for its use, is available at http://talloiresnetwork.tufts.edu/wp-content/uploads/Self-AssessmentRubricfortheInstitutionalizationofService-LearninginHigherEducation.pdf.

Based on the work of Barbara A. Holland, another rubric for assessing institutional levels of commitment to service-learning appears as Exhibit 6.3. It includes seven aspects of commitment—mission; promotion, tenure, and hiring; organizational structure; student involvement and curriculum; faculty involvement; community involvement; and campus publications. The rubric is scored using four levels of relevance to institutional mission: low relevance, medium relevance, high relevance, and full integration (Gelmon, Holland, Driscoll, Spring, & Kerrigan, 2001).

The Council for the Advancement of Standards in Higher Education (CAS) promulgates national standards for use in institutional self-assessment of programs and services, including service-learning. The CAS framework consists of standards and guidelines for the assessment of the overall service-learning program's mission, program, leadership, human resources, ethics, legal responsibilities, equity and access, diversity, organization and management, campus and external relations, financial resources, technology, facilities and equipment, and assessment and evaluation. CAS also offers self-assessment guides that include a comprehensive self-study process for program evaluation (Council for the Advancement of Standards, 2013).

Institutions wishing to conduct assessment from the broad perspective of community engagement, which includes but goes well beyond service-learning, should consider using the framework promulgated by the Carnegie Foundation for the Advancement of Teaching for their Elective Community Engagement Classification. Even if an institution decides not to apply to be recognized under the classification, the framework provides an excellent tool for the purpose of self-assessment. Although it does not include a rubric, responding to the open-ended questions, which requires both data and narrative, would create a thorough case study of the institution's commitments, activities, and outcomes related to community engagement (2013b).

Another instrument useful for institutional self-assessment is the annual Campus Compact Member Survey. Conducted each year since 1986, the survey collects data on student and faculty engagement in service and service-learning, campus infrastructure, faculty roles and rewards, and alumni engagement. The executive summaries published by Campus Compact based on annual survey data are helpful for benchmarking an individual institution's results against national data.

In conducting institution-level assessment of the benefits of service-learning to various stakeholders, obstacles to its further development, and degree of institutionalization, most of the methods described in 6.2 are applicable. It is important to note again here that there are multiple sources of existing data to tap in the process of completing the instruments and rubrics described above. Among these are institutional publications and online presence, mission and goal statements, annual and accreditation reports, strategic plans, student application essays, media reporting, catalogs and course schedules, budget documents, and institutional policies.

What Are the Logistical Considerations?

There is no doubt that adding assessment to the many responsibilities that service-learning entails can seem overwhelming. Some of the common challenges in assessing service-learning are: