Outcomes and Evaluation Subcommittee Meeting Minutes
February 27, 2007
Members Present
Steven Amick
Amy Christianson
Louis Fernandez
Renee Newton
Michael Funk
Frank Pisi, CDE staff to the Advisory Committee
Members Absent
Gary Moody
John Malloy
Note: All handouts and documents referenced in these minutes are available to the public. Please contact the California Department of Education's (CDE) After School Programs Office at 916-319-0923.
Commencement
Renee Newton, Subcommittee Chair, convened the February 27, 2007 Outcomes and Evaluation Subcommittee meeting at 1:15 p.m.
Introduction of Members
All members introduced themselves to the invited experts and members of the public in attendance.
Approval of Meeting Minutes
Minutes from the December 11, 2006 Subcommittee meeting were reviewed. One revision was suggested. Minutes were approved as amended.
Chair Report/CDE Staff Report
Renee Newton informed the Subcommittee that she had received a memorandum from the Bay Area Partnership. A copy of the memorandum was provided to all present. The purpose of the memorandum was to request that the Subcommittee consider some proposed core principles and practices when creating recommendations regarding local program evaluation to the CDE. Included in this document was a suggestion to consider measuring skill development through the skills present in the California Content Standards. Subcommittee members asked that a discussion item regarding this be placed on the next agenda.
Testimony From Representatives of Resource Development Associates (RDA)
Patricia Bennett and Kayce Garcia Rane from RDA presented information regarding various after school evaluation projects they have conducted. They presented a PowerPoint document entitled, "Measuring Skill Development in After School Programs." In the presentation the question of "What is skill development?" was explored. According to the presenters, skill development could be considered "developing and practicing practical skills and behaviors…" like public speaking.
Ms. Bennett suggested that the CDE consider adopting a definition of skill development that is more consistent with life skills. She then offered examples of ways that survey questions could be framed to demonstrate skill development through life skills.
Kayce Garcia Rane then concluded the presentation by discussing some of the lessons RDA has learned from engaging in many evaluation processes over the years. Lessons in the following areas were discussed (a sampling of the lessons learned follows):
- Evaluation Components and Tools
- Employ mixed methods and an inclusive design process, and tools should be simple and concise.
- School District and Program Sites
- Know the limitations of teacher involvement and maintain realistic expectations for staff.
- Data Collection
- Requires staff buy-in and training, trust, and technical assistance.
- Analysis and Findings
- Must be relevant to program management and shared with programs.
After discussion with the subcommittee, it was recommended that the CDE consider providing grantees with a set of mandatory reporting points related to the outcomes selected while allowing programs to choose others (beyond the mandatory) to suit their program goals.
Discussion of Previous Testimony Related to Outcome Measures
- Homework completion
- CDE Staff reviewed the previously presented draft conceptual framework for assessing the homework completion outcome measure. As currently drafted, after school programs would survey a sampling of classroom teachers about the homework completion rates of students in the program. These rates would be analyzed along a dosage continuum. From this review and discussion grew further discussion around the use of the term "teacher" in after school program related education code. CDE staff reiterated the position held by the CDE’s legal office that the term teacher connotes someone that is credentialed to teach in California. The conceptual framework was developed consistent with this distinction. Subcommittee members commented that they would like more discussion about that point and, depending on the outcome of such discussions, might suggest revisions to the homework completion framework.
- Skill development
- The subcommittee discussed the previous related testimony and suggested that the Strengths and Difficulties Questionnaire (SDQ) that Dr. Joe Durlak presented in December might be better suited to measure positive behavioral changes.
- Positive behavioral changes
- Renee Newton suggested that CDE staff research the SDQ further in the context of utilizing either questions from the document, or similar questions, to measure positive behavioral changes. The Subcommittee agreed that this would be something worth exploring.
Information About the Research of the Academy of Sciences on Critical Features of Programs That Support Healthy Youth Development
Sam Piha, Temescal Associates, delivered this presentation. Mr. Piha provided the subcommittee with background information about the origins of the Academy of Sciences research around critical features that support youth development. He also provided some information about the research itself. Mr. Piha referenced a document of the National Research Council and Institute of Medicine, Community Programs to Promote Youth Development that summarizes the research and critical features.
A main point stressed in the document is that young people must acquire knowledge and skills in multiple domains. Critical program features addressed in the document include:
- Safety
- Appropriate Structure
- Positive Social Norms (rules of behavior)
- Supportive Relationships
- Opportunities to Belong
- Support for Mattering (making a difference)
- Opportunities for Skill Building
- Integration of Family, School,
- Community Efforts (coordination)
Mr. Piha addressed these features in his presentation and discussed the importance of including such features in a program’s design. These features provide programs with the "gold standard" for quality in a program. When addressing program quality, three key questions should be addressed: "Do we know what it is?, Can we Measure it?, and Can we move it?"
Within the context of the independent statewide evaluation of after school programs, Mr. Piha suggested that the critical features that support healthy youth development be included and considered. These features are widely accepted by a variety of organizations and institutions and do have a research base to show that they do have an impact on youth development.
Public Comment
Nancy Chaires, Children Now, provided the subcommittee with a memorandum from Children Now. This memorandum provides suggestions and points to consider related to the development of tools and procedures related to the outcome measures identified in Education Code. Some suggestions include the maintenance of flexibility in measures; the possibility of considering mastery of academic content standards to demonstrate success, any process should produce comparable measures of change, the value of the data should be commensurate to the cost of collection, and broad support to the system must be provided to ensure success.
Katie Brackenridge, Bay Area Partnership, commented about the discussion around the definition of teacher in after school related Education Code. She suggested that the CDE or Subcommittee contact the author of SB 638 Senator Tom Torlakson to better understand his intent.
Determine Additional Items to Report at the March Advisory Committee Meeting
No items beyond what was discussed on the agenda were identified. CDE staff will work with the Subcommittee chair on this report.
Determine Additional Topics for Next Meeting – Discussion and Action
The following items were suggested:
- The term "teacher" in after school related education code
- Measuring skill building via the California content standards (Michael Funk will work with CDE staff on drafting a sample document to present and discuss at the next meeting)
- Using the SDQ to measure positive behavioral changes (CDE staff will research this further and present sample items)
- Memoranda received from the Bay Area Partnership and Children Now
- How to draft final recommendations for the independent statewide evaluation
- How to draft the summary document for the Subcommittee’s work with the CDE on outcome measures.
Adjournment
Renee Newton adjourned the February meeting at 4:15 p.m.
