Outcomes and Evaluation Subcommittee Meeting Minutes
April 3, 2007
Members Present
Steven Amick
Michael Funk
Renee Newton (Chair)
John Malloy
Louis Fernandez
Amy Christianson
Members Absent
Gary Moody
Frank Pisi, CDE staff to the Advisory Committee
Commencement
Renee Newton, Subcommittee Chair, opened the April meeting at 1:35 p.m.
Approval of Meeting Minutes
Amy Christianson motioned to approve the minutes as presented and Lou Fernandez seconded the motion. The minutes were unanimously approved as presented to the Subcommittee
Testimony from the Field: Evaluating After School Programs
Christy Hurt from Center for Evaluation and Research, an independent evaluation firm in rural northern California, presented information regarding the unique issues facing programs in rural areas. Most challenges are around issues of data collection, namely lack of consistency in reporting formats for attendance, suspensions, and other data, and limited numbers of students taking Content Standards Tests (CSTs). Ms Hurt then described her organization’s process for evaluating after school programs including annual focus groups of administrators, after school staff, teachers, and parents, surveys of randomly selected students, as well as a variety of other surveys of parents, students, and teachers.
Ms. Hurt then presented concerns regarding any new evaluation (outcome measure) requirements including the fact that many rural programs have made cuts in evaluation services and may not have the expertise to complete additional evaluation requirements proposed. Also there is a concern about the ability of program staff (and teachers) to complete state mandated surveys or checklists at a high rate.
Recommendations put forth were as follows: Make schools’ responses to any statewide evaluation effort available to them for the purposes of continuous improvement. With reference to sample size, have all programs submit the required data, regardless of size, and leave it up to the independent evaluator to decide which is “usable” for the statewide evaluation.
Measuring the Positive Behavioral Changes Outcome Measure Using Elements from the Strengths and Difficulties Questionnaire (SDQ) and Social Skills Rating System
Frank Pisi, CDE staff to the Subcommittee, presented information regarding this topic. Copies of the SDQ were presented to the Subcommittee and it was made known that this tool was created as an assessment tool for clinical Psychologists. It would be possible for select questions or prompts to be extracted from the SDQ for use in an after school program, but it would not be a perfect application. Mr. Pisi then presented to the Subcommittee the After School Program Survey (ASPS) of the California Healthy Kids Survey (CHKS). This survey was created expressly for CDE-funded after school programs, and sections of this might be of better use for measuring both positive behavioral changes and skill development. The Subcommittee was walked through the various items included in the ASPS and pertinent questions relating to the appropriate outcome measures were pointed out. There was some discussion about the feasibility of using this tool and of the issues of responses being elicited from students, which is apparently in conflict with Education Code. The possibility of rewriting the questions to be student focused and reported by teachers or staff was discussed.
Mr. Pisi also proposed that the CDE is interested in creating a workgroup that will focus on creating tools and procedures for measuring outcome measures. This workgroup would include CDE staff, individuals with experience in after school program evaluation, and one or two Subcommittee members. The workgroup would take the advice and recommendations put forth by this Subcommittee and engage in this work over the summer (2007). The completed draft tools and procedures would then be presented to the Subcommittee at an August or September meeting. All Subcommittee members expressed great interest in this idea and offered their support.
Memoranda received from the Bay Area Partnership and Children Now
Katie Brackenridge from the Bay Area Partnership (BAP) presented information regarding her organization’s recommendations to the CDE for implementing the outcome measure provisions of Education Code (EC) . Ms. Brackenridge provided the following recommendations: Honor the intent of the authors, including the notion that “teacher” be interpreted to include after school staff, that multiple outcomes be allowed for accountability, and promote the fact that programs should link activities to outcomes. With reference to the tools and procedures for outcome measures, simple tools that utilize technology would be best. Also technical assistance from the CDE to support successful use of the tools is imperative. It was suggested that the CDE consider program attendance as the critical measure in the first years. Drawing on existing measurement tools would enable the CDE to create tools that are field tested and successful. Ms. Brackenridge further suggested that the CDE consider issuing waivers to programs with high quality evaluation systems that are measuring the same outcomes as are required in law. Finally, it was suggested that the CDE consider these first years as a trial period and use them as a means to fine tune the processes and tools.
Subcommittee members engaged in discussion and expressed general agreement with BAP’s recommendations. Concern was expressed, however on the issue of waivers for programs. CDE staff cautioned against this as the process of granting waivers might include involving the State Board or other CDE offices responsible for granting waivers for other programs. Likewise, Subcommittee members discussed the idea that not requiring all grantees to report the same data on the outcome measures might not provide a common ground to make funding decisions.
Theresa Garcia-Araya from Children NOW addressed the Subcommittee on issues raised in her organization’s memorandum to the Subcommittee. Many of her comments echoed those put forth by Ms. Brackenridge. Ms. Garcia-Araya, however, voiced concern over the notion of issuing waivers to programs, as her organization strongly recommends that any processes or tools must produce comparable measures of change. To that end, it was recommended that all programs be made to report standardized data via standardized tools and procedures. Additionally, Ms. Garcia-Araya made the point that the value of the data to be collected should be weighed against the cost of collection and analysis. While it is imperative that the data collected be sound and comparable, the CDE should consider the cost of obtaining such data when developing the tools and procedures. Support for the need of increased technical assistance for programs, as was support for a broader interpretation of the term “teacher” was voiced.
The Subcommittee engaged in discussion around Ms. Garcia-Araya's comments, and, with regard to the issue of programs with more robust evaluation systems, CDE staff proposed the following to the Subcommittee and Ms. Brackenridge and Garcia-Araya: While all programs must use the tools and procedures developed by the CDE for the purpose of measuring the selected outcome measures, programs may also request that the CDE allow for the submittal of supplemental data related to the measures. The CDE would first approve the methods used to collect the data in question, but if approved, supplemental data could be considered when making any funding decisions. This was met with general agreement with some Subcommittee members expressing the concern that allowing supplemental data might render the mandatory data meaningless and simply impose an additional data collection burden on programs. After some discussion on this concern, it was agreed that the possibility of allowing for supplemental data should be addressed by the CDE created workgroup discussed earlier in the meeting.
Measuring the Skill Building outcome measure via the California Content Standards
Michael Funk, Subcommittee member, presented a document summarizing his findings to the Subcommittee. In summary, Mr. Funk found that it would not be feasible to measure skill building using the California Content Standards. With the complexity and wide variety of standards, there is no simple or reliable way to have programs report on outcomes using the standards. With this determined, Mr. Funk engaged in conversation with a variety of experts in after school evaluation from across the nation. Based on these conversations, the following recommendations were presented:
- Consider the first year as a development year. This would allow the CDE to pilot difference data collection tools and procedures and decide upon the best format to use.
- Focus on anchor outcomes related to attendance in the first year
- Ensure access to technical assistance to enable all programs to be successful in their data collection.
- Development of instruments
- Include measures of more broadly outcomes in the tools, unless doing so greatly increases the size of such tools
- Build on existing tools
- If requiring teachers or other to report on student progress, shorter surveys will foster a greater return rate
- Use technology whenever possible to reduce the burden on programs.
Drafting the Set of Final Recommendations for the Independent Statewide Evaluation of After School Programs
Renee Newton presented the following procedure/timeline for drafting the set of final recommendations: CDE staff will create the first draft based upon discussions had at all Subcommittee meeting and any testimony presented by the field. This set will be submitted to the Subcommittee in advance of the early May Subcommittee meeting, and discussed, noting any further suggestions or edits, at that meeting. Another draft of recommendations will be created and submitted to the full Advisory Committee in advance of the May meeting for revision. Any recommendations or suggestions from the Advisory Committee will be incorporated into a final draft will be presented to the Subcommittee at the late May Subcommittee meeting. This version will be discussed and up for approval by the Subcommittee. The final, Subcommittee approved recommendations will be submitted to the full Advisory Committee at its June Meeting.
Similarly, a summary of work completed regarding consultation with the CDE on the outcome measures will be drafted by CDE staff. A preliminary summary will be presented to the Subcommittee at the Early May meeting. Any necessary revisions to this document will be made in advance of the late May meeting at which the Subcommittee will review and approve the final version for presentation at the June Advisory Committee meeting.
Definition of Teacher in After School Education and Safety and 21st Century Community Learning Centers EC
Renee Newton reported to the Subcommittee that she had received a memorandum from Senator Tom Torlakson regarding the definition of “after school teacher” in EC. According to the memo, the authors intended for the term “after school teacher” to be interpreted to include all staff working in an after school program, not just credentialed staff. Frank Pisi informed the Subcommittee that this memo would be sent to CDE legal counsel for review. Legal counsel has previously rendered the opinion that teacher in EC is interpreted to refer only to persons credentialed to teach in California. Mr. Pisi will consult with legal counsel to determine whether or not the memorandum from Senator Torlakson will change their opinion and report back to the Subcommittee at its next meeting.
Public Comment
Theresa Garcia, Vice President, Children Now, addressed the Subcommittee on the issue of the definition of teacher. She requested the legal basis for the opinion in writing, if CDE legal counsel determines that teacher in an after school program means a certificated or credentialed individual.
Determine Additional Items to Report at the April Advisory Committee Meeting
The Subcommittee Chair will work with CDE staff to develop the report.
Determine additional topics for next meeting
Beyond the standing items, the Subcommittee identified the following topics:
- Testimony from representatives of LA's BEST after school program regarding program evaluation
- Discussion of the draft final recommendations for the statewide evaluation of programs
- Discussion of the draft summary of work with the CDE on outcome measures
- Report of inquiry to CDE legal counsel regarding the definition of teacher in after school programs
Adjournment
The Subcommittee meeting was adjourned at 4:05 p.m.
