Outcomes and Evaluation Subcommittee Meeting Minutes
May 9, 2007
Members Present
Steven Amick
Michael Funk
Renee Newton (Chair)
Paula Wenzl
Louis Fernandez
Frank Pisi, CDE staff to the Advisory Committee
Members Absent
Gary Moody
Amy Christianson
Commencement
Renee Newton, Subcommittee Chair, opened the April meeting at 9:37 a.m.
Approval of Meeting Minutes
Michael Funk motioned to approve the minutes as presented and Paula Wenzl seconded the motion. The minutes were unanimously approved as presented to the Subcommittee
The Chair reordered the agenda to accommodate presenters’ schedules
CDE Staff Report: Legal Opinion on the Definition of Teacher in After School Education and Safety and 21st Century Community Learning Centers Education Code
Frank Pisi reported to the Subcommittee that he forwarded the memorandum received from Senator Tom Torlakson's office to CDE legal counsel. In subsequent conversations with legal counsel, it was determined that the clarification of author’s intent did not cause a change in the original legal opinion: Since the CDE has no statutory authority to define teacher in a way inconsistent with the definition of the Commission on Teacher Credentialing, the body responsible for defining teacher requirements, teacher must be interpreted to mean any person credentialed to teach in California. CDE staff has requested this opinion, with any statutory authority cited, in writing. To date, CDE legal counsel has been in court and unable to provide such document.
Discussion was held on this topic. Subcommittee member Amick offered that it appears that the remedy to this situation is to change legislation to remove the term “after school teacher” and replace it with the term "after school staff."
Subcommittee member Funk raised an issue with the notion of having a student’s regular day teacher report on their progress in the after school program, given the fact that skill development and positive behavioral change would most likely occur in the program. To that point, CDE staff Frank Pisi offered that one way to view program effectiveness is to measure how much a student is able to transfer skills obtained in the after school program to the regular day. Mr. Funk relied that it would be difficult to measure this, as many other factors play a role in a student’s conduct in the regular day.
Mr. Funk then made a motion that the Subcommittee recommends to the full Advisory Committee that it supports the legislative intent around the definition of after school teacher, as clarified in Senator Torlakson’s memorandum. Clarifying discussion was held on this topic. The motion was seconded by Subcommittee member Fernandez and unanimously approved by the Subcommittee.
CDE staff informed the Subcommittee that he would include this specific item on the next Advisory Committee agenda as an action item so that the full Advisory may vote on this recommendation.
Testimony from the Field: Evaluating After School Programs
Carla Sanger and Dr. Jim Sass from LA’s BEST presented testimony to the Subcommittee. Ms. Sanger provided an overview of LA's BEST (its development, philosophical framework, and current evaluation efforts) for the Subcommittee. As to her views on evaluation, Ms. Sanger believes that 50 percent of any evaluation effort must be dedicated to satisfying the needs of the evaluator and the other 50 percent must be dedicated to supporting the program’s continuous improvement efforts. If this is not the case, then, in Ms. Sanger’s view, half of the evaluation effort is wasted time. LA’s BEST receives numerous requests to participate in evaluation studies, many times with significant funding attached, and every request is put to this "50 percent test." Evaluation proposals that will not be able to support LA's BEST in their improvement efforts are denied. Ms. Sanger asked that the Subcommittee and Advisory Committee recommend to the CDE and its independent evaluator that it be a requirement that any statewide evaluation effort follow this model- do not have programs collect or report data only for the sake of collecting data; the evaluation effort must be a benefit to the program. Along with that, Ms. Sanger noted the importance of “selling” the benefits of any evaluation or data collection effort to the staff that will be doing the work. Without their buy in and belief that the effort is important, the validity of the data may be suspect.
Dr. James Sass continued the testimony with a PowerPoint presentation describing LA's BEST’s internal and selected external evaluation processes. LA's Best has identified four primary outcome areas: Academics, Social Development, Family Enrichment, and Crime Prevention and Safety.
In the area of academics, data in the following areas are obtained and evaluated:
- Homework
- Attitudes toward school
- Future aspirations
- Attendance (regular school day)
- Classroom behavior (regular school day)
- Test scores
In the area of social development, data in the following areas are obtained and evaluated:
- Leadership
- Relationships with adults
- Empathy
- Self-esteem
- Classroom behavior (regular school day)
In the area of family enrichment, data in the following areas are obtained and evaluated:
- Family knowledge of school work
- Parents talking about school/homework
In the area of crime prevention and safety, data in the following areas are obtained and evaluated:
- Decision making
- Conflict resolution
- Feelings of safety
The Subcommittee had discussion about the methods of collecting such data and Dr. Sass informed the group that they employ a variety of methods: parent, teacher, and student surveys, data obtained from schools and districts (STAR data, attendance data, report card data, etc.), and other means. Mr. Sass made the point that it is important to match the collection method with the type of data you’re planning on collecting.
Additionally, the following data is collected to provide the perspective of parents and children:
- Program attendance and retention
- Parent satisfaction (overall, homework support, staff relations, safety)
- Reasons children participate in LA's BEST
- "What do you want to be when you grow up?"
- Children’s "like most" and "like least" about LA’s BEST
- Child and parent health-related learning and behavior
Dr. Sass reported to the group that just as important as evaluating the program from the perspective of parents and students, it is important to survey after school program staff. LA's BEST collects the following data from program staff:
- Retention and tenure
- Demographics
- Program-relevant characteristics (plans to teach, living near school, working as classroom aide, college enrollment)
- Workshop and meeting attendance
- Program-related learning and attitudes
- Support from traveling staff and central office
Reinforcing the point made by Ms. Sanger, Dr. Sass presented to the group how all of the data collected serves two purposes: External purposes for funders and other stakeholders (to support continuation of funding, statewide policy development, etc.) and internal purposes including resources for briefings for managers and local boards, and summary information for program clusters and sites, reports to foster learning among staff. Using the data collected and evaluated, LA's BEST engages in the following processes:
- Integrating staff support planning into monthly activity reports.
- Conversations about how to spend time in activities and observe program quality.
- Focusing on the four outcome areas.
- Creating logic models with program managers.
- Developing evaluation questions, procedures, and tools with program managers.
Dr. Sass then presented information about various external evaluations of LA's BEST. Two such studies are the "Keeping Kids in School" study which found that LA's BEST students are significantly less likely to drop out of school as well as a reduction in risk behaviors for students who attended the program for at least three years. Another study on Social Capital found that LA’s BEST staff members quickly internalized the mission and values of the program and staff members’ positive interactions with children lead to higher engagement in the regular school day.
A third study presented looks at LA's BEST long term effects on juvenile crime. This study was conducted by the Department of Justice (DOJ) and, at meeting time, had not been released publicly. Specific information related to this study has been embargoed until it is publicly released by the DOJ.
Dr. Sass concluded his presentation with a set of what’s been learned as a result of the internal and external evaluations conducted by LA's BEST:
- Need to balance funders’ requirements and internal priorities.
- Importance of identifying realistic outcomes and objectives.
- Necessity of developing a data collection/ management system that is both reliable and flexible.
- Value of making evaluation useful to stakeholders.
- Relationships are at least as important as technical expertise.
Chair Report
Subcommittee Chair Renee Newton informed the subcommittee that she had received two memoranda regarding recommendations for the independent evaluation of before and after school programs: one from Bay Area Partnership and another from California Tomorrow. The Bay Area Partnership memorandum presented the following recommendations
- Clarify the purpose of the evaluation
- Define a list of evaluation questions to be addressed by the independent evaluation
- Study implementation first
- Compensate teachers and after school staff for their time
- Encourage study of local evaluation practices as part of the independent evaluation
- Make sure the committee’s recommendations are reflected in CDE's RFP
Discussion was had around the third recommendation, with questions about the use of the term "experimental study.". Subcommittee Funk clarified that the recommendation was against undertaking a study such as that before an evaluation of implementation and program quality was conducted. All Subcommittee members agreed with that recommendation.
The memorandum from California Tomorrow proposed recommendations in two categories: Evaluation content and evaluation design and methods. The broad recommendation related to evaluation content was for the inclusion of content elements and evaluation questions that will help the state monitor access and equity goals. Design and methodology recommendations were centered on including design and methodology practices to help ensure attention to access, equity and diversity issues. Subcommittee members expressed support for the recommendations presented and Ms. Newton offered the opinion that inclusion and equity issues must be included in the evaluation design.
Drafting the Set of Final Recommendations for the Independent Statewide Evaluation of After School Programs
Frank Pisi, CDE staff, presented a final draft of this document. Subcommittee member Wenzl suggested a reordering of the document that might more clearly present the recommendations to the reader. Subcommittee members had no recommendations or suggestions to revise the current content of the documents, but asked for inclusion of the following items from the Bay Area Partnership and California Tomorrow memoranda:
- Include a recommendation regarding a clarification of the purpose of the evaluation. Subcommittee member Funk proposed the following purposes: (1) To improve program quality and (2) To support the equity goals presented in the California Tomorrow memorandum.
- Specifically include recommendations 4 and 5 from the Bay Area Partnership memorandum
- Include Recommendations 1,2, and 5 from the California Tomorrow memorandum
CDE staff will incorporate these items and work with the Chair to finalize the document for presentation at the Advisory Committee meeting on May 23.
Drafting the Summary Report of Consultation with the CDE on Outcome Measures
Frank Pisi presented the first draft of this document to the Subcommittee. This document is intended to provide a summary of the testimony received over the past year regarding outcome measures as well as a summary of the Subcommittee's consultation with the CDE around the development of standardized tools and procedures for the outcome measures. Subcommittee members had no revisions to the content as presented, but did request that the testimony received at this meeting (from LA's BEST) be incorporated into the document. Chairperson Newton suggested that, for the purposes of presenting information to the full Advisory Committee, a companion document listing only a summary of the findings and recommendations related to the work completed on outcome measures. CDE staff will work with the chair to develop this additional document. Both documents will be presented at the next (May 31) Subcommittee meeting.
Public Comment
Corey Newhouse, Senior Policy Associate, Children Now, addressed the Subcommittee on the issue of the definition of teacher. She reiterated the previous request for the legal basis of CDE legal counsel’s opinion that "teacher" in an after school program means a certificated or credentialed individual. Frank Pisi stated that he will bring this request to CDE legal counsel and reminded the subcommittee that legal counsel has been in court and unable to provide the written document as previously requested.
Determine Additional Items to Report at the April Advisory Committee Meeting
The Subcommittee Chair will work with CDE staff to develop the report.
Determine Additional Topics for Next Meeting
Beyond the standing items, the Subcommittee identified the following topics:
- Discussion on the final recommendations for the statewide evaluation of programs
- Discussion of the draft summary of work with the CDE on outcome measures
Adjournment
The Subcommittee meeting was adjourned at 12:15 p.m.
