Phase II Final Report
July 1998
This report was prepared by Daniel Weiler, Amy LaGoy, Eric Crane, and Abby Rovner of RPP International under contract with the California Department of Education.
Service-learning engages students in community service linked to school curriculum as a strategy to improve learning, help students become good citizens, and advance their personal and social development. Service-learning goals also include the promotion of school reform and the provision of needed services to communities.
Service-learning typically relies on partnerships between schools and public or non-profit agencies that provide service opportunities for students in the community. In California, grants to K-12 service-learning partnerships are made through the CalServe Initiative of the California Department of Education (CDE). The Initiative is funded by the federal Learn and Serve America Program, which was created by the National and Community Service Trust Act of 1993. Under this Act, federal funds are allocated to state education agencies for awards to local service-learning partnerships and for state-level capacity-building. Since the CalServe Initiative began in 1992, CDE has funded some forty service-learning partnerships throughout the state.
In June 1995, CDE contracted with RPP International to conduct an evaluation of the CalServe partnerships. The evaluation goals were to:
- document the potential of well-implemented service-learning
to enhance student academic achievement and interest in schoolwork,
and strengthen students' sense of civic, social, and personal
responsibility;
- describe the impact of service-learning on teachers and schools;
- describe how well-implemented service-learning impacts community
partners; and
- assess the relationship between service-learning implementation and its impacts on students, teachers, schools, and communities.
In order to determine what range of impacts could be expected from high quality service-learning programs, the evaluation concentrated on a study of well-designed and well-implemented partnerships with potentially high impacts that were representative of a range of key partnership characteristics. In the first phase of the evaluation, RPP conducted a preliminary fieldwork investigation and documentary analysis of 27 service-learning partnerships that were being funded through CalServe in spring 1995. This investigation identified 12 partnerships where service-learning was well-implemented and might be expected to have a reasonably high impact, and that also represented a wide range of variation in program scope, location, and design. The results of that work were published in December 1995 (An Evaluation of K-12 Service-Learning in California: Phase I Evaluability Report).
The Phase I work determined that service-learning was most often practiced by one or a few committed teachers in a school, that teachers were having difficulty implementing service-learning effectively, and that few schools were embracing service-learning as a way to realize school-wide goals. In the first year of Phase II, therefore, the evaluation focused on how schools and teachers used service-learning, in order to identify schools and classrooms where student and other impact measures might subsequently be employed. This work was also aimed at providing policy makers with a more thorough understanding of how service-learning was operating at the classroom level and making a preliminary assessment of the possible impact of service-learning on teachers and students. A report on the findings of that investigation was published in February 1997 (An Evaluation of K-12 Service-Learning in California: Phase II Status Report).
Utilizing information gathered during the first year of Phase II, evaluation staff identified 15 classes in 14 schools (in 11 CalServe partnerships) where service-learning appeared to be well designed and well implemented. Eight comparison classes in eight schools were also selected to be part of the evaluation. These classes and schools became the focus of evaluation efforts during the 1996-97 school year the final year of the evaluation when the evaluation collected data on the impact of service-learning on students, schools, teachers, and community partners. Thus, the findings summarized below are based on data from selected classes where evidence suggested that teachers were implementing service-learning effectively; they do not represent a portrait of the average impacts of service-learning at all CalServe-funded partnerships.
Of the 15 classroom sites participating in the evaluation, seven were urban, three rural and five suburban or suburban/rural. There were five elementary school classes, three at middle schools, three at the high school level, and four in continuation high schools. The smallest site had 12 students; the largest included two classes totaling 60 students, taught by the same teacher. School enrollments ranged from as few as 140 to as many as 3,000 students. Data were collected on approximately 775 students participating in service-learning in grades 6 through 12. The eight comparison classes had a total of some 310 students. Three of the comparison classes were in the same schools as their "matched" classes that used service-learning; five were in different schools. The data collected by the evaluation included student achievement tests, student attitude surveys, school record data, interviews with teachers, principals, students, program coordinators, and community partners, and observations of student service and classes where service-learning was being used.
Impacts on Students
Data gathered during the 1996-97 school year and in the fall of 1997 indicate that service-learning benefits student learning, attitudes toward school and citizenship, volunteer service behavior, and student personal and social growth. Quantitative findings were available from 12 of the 15 evaluation sites; field work data were collected from all 15 sites. The overall results were quite positive, though the specific measures for which impacts were found, as well as the magnitude of those impacts, varied from site to site.
Quantitative Findings
- Students in six of the 12 classes from which quantitative
data were collected had statistically significant, moderate
to large positive effects on various measures of educational
impacts, including achievements tests in language arts or reading
and/or measures of engagement in school, students' sense of
their educational competence, homework completion, and educational
aspirations. Positive academic achievement results were found
in an elementary school class, a middle school class, and a
high school class.
- With two exceptions, large majorities of students engaged
in service-learning at each site said they had learned more
in the class that had included service-learning than in a typical
class in their school. Most said that service-learning had heightened
their interest in the class where it was used, and many indicated
that it had also made them more interested in their other classes.
- Positive impacts were recorded for measures of student civic
and social attitudes at nine sites. Service-learning appeared
to have positive impacts on five types of student civic and
social attitudes: personal and social responsibility, work orientation,
communication, voluntary service participation, and community
service leadership. For one other attitudinal measure "student
acceptance of cultural diversity" there were negative impacts
at two sites and no statistically significant findings at the
remaining 13 sites.
- Four out of five elementary school sites had a variety of strong positive results; one elementary school had mostly negative impacts. Results were weaker at the middle school level, with one site showing mostly negative impacts, but two out of three high school sites had a range of strong positive educational and attitudinal impacts. (The third high school had positive findings on an achievement test in reading, a negative finding on a measure of work orientation, and no other statistically significant findings.) The quantitative findings for continuation high school students show that they did not do as well as comparison group students on measures of their educational aspirations, sense of engagement in school, sense of educational competence, hours of homework completed, and sense of personal and social responsibility.
Field Work Findings
- Field work findings indicate that service-learning had positive
impacts on students across a wide range of educational, attitudinal,
and behavioral dimensions. At virtually every evaluation site,
field work interviews and observations determined that students
received substantial benefits from their participation in service-learning,
including improved understanding of the curriculum, a greater
sense of civic responsibility, increased interest in school,
and greater academic self-confidence.
- Even at sites where the quantitative findings were either
not statistically significant or suggested negative impacts,
field work interviews and observations consistently recorded
student, teacher, community partner and other testimony pointing
to these educational and attitudinal benefits of service-learning.
- In particular, while the quantitative results from continuation high schools in the evaluation were uniformly negative, field work interviews and observations, as well as other evidence, suggest that service-learning had a strong positive impact on continuation school students at the time they were busy with service-learning activities but was not long-lasting or powerful enough to change student attitudes or behavior once they were no longer engaged in service-learning.
Impacts on Schools and Teachers
At schools where service-learning was well-designed and well-implemented, it had substantial benefits for both schools and teachers. The most substantial impacts occurred at schools where service-learning was most widespread, i.e., where more than 20 percent of the school's teachers were using service-learning. Service-learning:
- built group cohesiveness, mutual respect, and more positive
peer interactions among students, by helping them to appreciate
and respect one another as doers, knowers, and learners.
- fostered greater mutual respect between students and teachers,
who often began to view one another as colleagues in the pursuit
of learning.
- improved overall school climate by helping students feel more
connected to their schools and by making expertise, rather than
authority, the "medium of exchange" between principals
and teachers. At some schools, the evidence suggests that service-learning
also contributed to reduced teacher turnover.
- contributed to schools' achievement of schoolwide educational
and attitudinal goals for students. At some schools, service-learning
provided a strong impetus for reform, including the restructuring
of school operations and a move away from traditional instructional
methods toward more student-centered approaches.
- required both teacher buy-in and principal support in order
to expand within schools. Teachers were opposed to having service-learning
mandated, but responded positively to principal support for
service-learning training and the provision of other school
resources as a way of fostering the growth of service-learning.
- reinforced and stimulated innovative instructional practices.
Teachers using service-learning made their classes more student-centered,
incorporated more hands-on activities into the curricula, and/or
made better or more frequent use of community resources.
- promoted teacher collegiality within and across schools and
districts, and gave teachers valued opportunities to have professional
discussions with peers about teaching and learning. Increased
collegiality among teachers in turn both enabled and expedited
the spread of service-learning.
- encouraged teachers to extend themselves professionally (e.g., cultivating community partnerships, trying new instructional approaches, orchestrating off-site service activities), but teachers using service-learning rarely perceived their extra work load as a serious burden.
Impacts on Community Partners
Data from field work interviews and observations indicate that service conducted by students from the evaluation sites addressed real community needs and enhanced relationships between schools and communities.
- Service-learning fostered positive changes in community views
of students and their schools. Community agencies viewed students'
conduct during service as responsible and helpful. Responsible
and conscientious behavior during service helped students at
continuation schools foster very positive changes in community
attitudes toward both the students and their schools, which
had previously been seen as institutions for young people who
were difficult to teach and who had little or no interest in
helping their communities.
- Positive changes in community perceptions of students and
their schools promoted increased community involvement in and
support for the schools. Increased numbers of community groups
and/or individuals became interested in forming service-learning
partnerships, volunteering at the schools, and offering other
forms of support.
- Service-learning activities met real community needs and generated increased community demand for student service. Students often filled a gap between the need for and availability of service volunteers, were sometimes able to meet multiple community needs, and often identified and met a community need that had not previously been addressed.
Service-Learning Implementation and Service-Learning Impacts
The evaluation assessed the relationship of service-learning implementation to the spread of service-learning within schools and to service-learning's impacts on students.
Implementation and the Spread of Service-Learning Within Schools
The findings point to a number of factors that were associated with school-wide adoption of service-learning:
- Teacher control with active principal support, encouragement,
and vision. Teacher buy-in and control of the process was
crucial to schools' adoption of service-learning. And service-learning
was most likely to be adopted by a sizable proportion of the
faculty when principals provided teachers with opportunities
to share their experiences with service-learning, removed scheduling
and logistical barriers to planning and conducting service,
and made service-learning adoption a school-wide goal.
- High-quality professional development for service-learning
implementation. Most teachers who were fully implementing
service-learning had at some point attended a two- or three-day
service-learning institute. Incorporation of service-learning
into regular, school-site staff development activities often
led to higher levels of service-learning adoption among faculty
and the infusion of service-learning into the school curriculum.
- Actively involved program coordinators. Hands-on
program coordinators (or a teacher or administrator at a school
who filled this role) provided essential support for the adoption
of service-learning beyond one or two classrooms. They spread
the news about service-learning at the school and in the community,
offered teachers and principals ideas for service projects and
partners, cultivated community partnerships, addressed logistical
issues, and ensured that the model of service-learning adopted
at the school adhered to standards supported by CDE.
- Supportive school climates. Supportive
schools were those where teachers felt respected and empowered
to implement curricula. At every school that was in the process
of taking service-learning school-wide, the field work data
reveal a climate of collegiality among teachers and administrators.
- Active community partners. Partners who participated in service-learning planning and implementation or collaborated with teachers to ensure that students were well prepared for their service projects supported service-learning sustainability and institutionalization. They helped to sustain a demand for service-learning, brought resources to the schools, and fostered positive school-community relations.
Implementation and Student Impacts
The evaluation did not identify any particular pattern of service-learning implementation that could "guarantee" positive student impacts. However, there were a number of factors which, taken together, increased the likelihood that service-learning would promote improved student educational performance and civic responsibility:
- Teacher professional development. Regular training
opportunities helped teachers move toward a standard model of
service-learning, but the quality of teacher training
was more important than its frequency. Teachers needed to understand
how to integrate the various key components of service-learning
(pre-service preparation, reflection, student voice, need-based
service, collaboration with service partners, etc.) in order
to realize the full potential of this strategy.
- Pre-service activities. Pre-service orientation
to a service-learning activity provided students with invaluable
information about the service site and recipients and contributed
to students' sense that the work they were doing was valuable
and necessary. Pre-service at sites with positive student impacts
included service-specific, practical/logistical, affective,
and curricular preparation.
- Service emphasis. Service activities that engaged
students in hands-on learning offering opportunities to learn
both course content and new skills promoted improvements in
students' academic performance, behavior, and attitudes about
contributing to society. Service activities that offered students
an opportunity to establish personal relationships with service
recipients promoted students' sense of responsibility for contributing
to their communities. A tight integration of service and the
curriculum appeared to support improvements in student academic
achievement. Positive impacts also appeared to be associated
with regular (weekly or bi-weekly) opportunities to conduct
service, with student perceptions that their service made a
difference, and with student "ownership" of their
service projects.
- Reflection. Reflection activities at sites with positive student impacts offered students regular opportunities to discuss and write about their service experiences. Reflection encouraged students to think critically about their service performance. Sites where students were required during reflection to suggest probable causes of and solutions for problems that arose during service were somewhat more likely to have positive student impacts on measures related to both educational performance and civic responsibility. Effective reflection varied among oral discussion, guided and free writing, and other strategies, and was integrated into the course curriculum.
Conclusions
The findings of this evaluation suggest cautious optimism. These findings indicate that service-learning led to a variety of positive results for students, schools, teachers, and communities. It would appear that well-designed, well-implemented service-learning programs can make an important contribution to student learning, civic responsibility, and personal growth; improve school climate; strengthen teacher professionalism and collegiality; and provide valuable services to communities.
What are the "right circumstances?" The evidence shows that effective service-learning implementation at the classroom level is a necessary (if not sufficient) condition for obtaining positive student impacts. Effective service-learning implementation in the hands of talented teachers has the greatest likelihood of achieving positive results for students, but good results were possible even with teachers of more modest talents, where service-learning was well-implemented. And yet, one of the most important lessons of this evaluation is that effective implementation is not easy. It requires teachers to understand how to link service to the curriculum, arrange pre-service and reflection activities that support learning, and involve community partners in service activities designed to empower and help teach students in a variety of ways.
The evaluation also shows that while service-learning can have beneficial impacts on teachers and schools, these impacts are likely to be much greater at schools where service-learning has been widely adopted. These schools typically welcomed innovation and supported and empowered teachers. Service-learning is most likely to be adopted and institutionalized in schools when teachers, principals, and community groups can see that it does have positive impacts on students.
The evaluation studied service-learning sites that were among the most well-implemented of the classes receiving CalServe support, together with a number of comparison classrooms. At these "high quality" service-learning sites, the evaluation nevertheless revealed considerable variation in the magnitude and breadth of service-learning impacts, as well as great variation in the quality of service-learning implementation. These results suggest that the majority of CalServe-funded programs probably still have much work to do in order to improve the effectiveness of service-learning implementation at the classroom level or achieve school-wide adoption of service-learning in a manner that will garner widespread teacher support.
At the same time, the evaluation has shown that the promise of service-learning can be fulfilled, provided that its classroom implementation challenges are understood and dealt with effectively.
Copies of the full evaluation report can be obtained from the California Department of Education, Counseling, Student Support and Service-Learning Office, 1430 N Street, Suite 6408, Sacramento, CA 95814, 916-323-2183.
