Superintendent’s Advisory Committee
Public Schools Accountability Act (PSAA) of 1999
Minutes
Wednesday, July 18, 2007
9:30 a.m.
Members Present:
Holly Jacobson, Chair, Assistant Executive Director, Policy Analysis and Research, California School Boards Association
Vicki Barber, Superintendent, El Dorado County Office of Education
Marilyn Buchi, School Board Member, Fullerton Joint Union High School District
Nancy Carroll, Superintendent, Ocean View Elementary School District
James S. Catterall, Professor, School of Education, University of California, Los Angeles
Marc Ecker, Superintendent, Fountain Valley Unified School District
Jerry Hayward, Consultant, Management, Analysis and Planning, Inc.
Ginger Hovenic, President/CEO, San Diego Regional Chamber of Commerce
Charles Weis, Superintendent, Ventura County Office of Education
Beverly L. Young, Assistant Vice Chancellor, California State University
Members Absent:
Ronald Garcia, Superintendent, Delano Union Elementary School District
Edward Haertel, Professor, School of Education, Stanford University
Pam Kinsley, Member, California Teachers Association, Harding School
Kelvin Lee, Superintendent, Dry Creek Joint Elementary School District (Retired)
Cecelia Mansfield, Vice President for Education, California State Parent-Teacher Association
Maria Medina, Parent, Parent Advisory Council
Lynette Nyaggah, Teacher, Rio Hondo College
Bernice Stafford, Vice President, School Marketing and Evaluation, Lightspan, Inc.
Ting Sun, Educational Programs Director, Natomas Charter School
Principal Staff to the PSAA Advisory Committee:
Bill Padia, Deputy Superintendent, Assessment and Accountability Branch
Patrick McCabe, Director, Policy and Evaluation Division
Robert Bernstein, Manager, Academic Accountability Unit
Rachel Perry, Manager, Evaluation, Research and Analysis Unit
Holly Jacobson called the meeting to order at 9:40 a.m.
Introduction of Committee Members, California Department of Education (CDE) staff, and audience members - Holly Jacobson
Oath of Office – Bill Padia
- Bill Padia administered the oath of office for three committee members: James Catterall, Jerry Hayward, and Beverly Young. The oath documents were signed and collected.
Agenda for the Day and Review of API History – Pat McCabe
- Pat McCabe introduced the agenda topics for the day and reviewed the legal requirements for Academic Performance Index (API) indicators, the tests that have been added to the API, and the key changes that have occurred in the calculation of the API since it began in 1999.
API and AYP Reports – Rachel Perry
- Rachel Perry described the current formats of the API and Adequate Yearly Progress (AYP) reports, which were revised and streamlined beginning in 2005. The Committee suggested that the name, "Comparable Improvement," which refers to performance by numerically significant subgroups, be changed for future API reporting.
API-Related Proposed Legislation – Richard Fattig
- Richard Fattig provided a review of current state legislation related to the API. Some bills would require changes in the API calculation while others would require changes in the assessment system, which in turn may impact the API:
- Assembly Bill 400 (Nunez) would require the API to include non-assessment indicators, including graduation rates, college readiness, and career readiness, commencing with the 2009-10 fiscal year. These indicators would constitute 50 percent of the value of the API by June 30, 2014, and be computed using the best available data.
- Mr. Fattig also summarized Assembly Bills 494 (Huffman) and 519 (Mendoza) and Senate Bills 219 (Steinberg and Romero), 405 (Steinberg), 123 (Romero), and 305 (Ducheny)
Assessment Indicators – Rachel Perry
- Rachel Perry described what tests could be added to the API and what tests might need to be dropped from the API in the future. Assessments not currently in the API include the Standards Test in Spanish (STS), California English Language Development Test (CELDT), California Modified Assessment (CMA), California Alternate Performance Assessment (CAPA) in science, and Physical Fitness Test (PFT). Assessments currently in the API but that may need to be dropped include the California Achievement Test, Sixth Edition Survey (CAT/6 Survey) at grades three and seven and grade two of the California Standards Test (CST), CAPA, and CMA. Decisions to incorporate or not incorporate tests in the API would involve decisions about current API test weights. Test weights also could be changed at any time, regardless of whether tests are added or deleted.
Non-Assessment Indicators - Pat McCabe
- Pat McCabe described possible non-assessment indicators that might be added to the API in the future as a result of current legislative efforts. These indicators included attendance rates, graduation rates, college readiness, workplace readiness, and dropout rates. Except for the grade eight dropout rates, non-assessment indicators would largely impact high schools only. Mr. McCabe provided the committee with an example of adding graduation rates to the API and some of the decisions that would need to be made to do so. The advantages of adding non-assessment indicators are they would introduce multiple measures to the API and enhance the validity of the API as a reflection of school performance. The disadvantages of adding non-assessment indicators to the API are they would raise concerns about data quality, reduce the API’s credibility as an academic measure, and introduce complexities into calculating growth targets and subgroup APIs. Reporting graduation rates would not be possible until 2011 or 2012 to show the real graduation difference for the API. Therefore, the committee may need to consider how to phase-in the graduation rate. James Catterall suggested that it may be beneficial to wait until there is four years of data but to notify people of the progress often.
Similar Schools Ranks – Robert Bernstein
- Robert Bernstein described the background and calculation of the similar schools ranks (SSRs). He showed the comparison of SSR changes between 2005 and 2006. Some of the concerns of districts about the SSRs are that many schools do not receive SSRs, schools may not really be similar, and the current method is difficult to understand. The range of the API scores of most 100 similar schools lists is about 250 points while the range of the Schools Characteristics Index (SCI) scores is more compact.
Statewide Performance Target – Jan Volkoff
- Jan Volkoff described the origins of the statewide performance target of 800. Alternative levels of a statewide performance target were presented as well as the percentage of schools that would have made their schoolwide targets in the 2006 Base API under each alternative. As the statewide performance target increases, the percentage of schools making their schoolwide targets decreases. This same effect also would occur for subgroups because the subgroup targets now use the same calculation method as the schoolwide targets. The committee concluded that changing the statewide performance target would create communication issues, and it should not be changed at this time.
Education Data System: Current and Future – Keric Ashley
- Keric Ashley described the current education data system, the California Longitudinal Pupil Achievement Data System (CALPADS), and the California Longitudinal Teacher Integrated Data Education System (CALTIDES). Mr. Ashley discussed the three primary tasks that need to be accomplished for CALPADS to be established. Task 1, completed already, requires all statewide student identifiers (SSIDs) to have been assigned statewide by the California School Information Services (CSIS). Task 2 requires the annual Information Day reports and annual statewide student identifier (SSID) maintenance during the year. Some small high schools and charter schools have been late in data submission, and this causes inaccurate data. Also, it is recommended that local educational agencies (LEAs) correct data during the year on an ongoing basis rather than only once a year. Task 3 requires a vendor to be selected, report approval by Department of Finance, development by September or October, piloting by 2008 or 2009, and system completion by 2009 or 2010. CALTIDES will collect teacher data, including information about teacher development. The system will have unique identification numbers for teachers. It will enable LEAs and county offices of education (COEs) to query the system for high quality teacher status and assignment monitoring, reduce the number of data submissions from LEAs to COEs, and eliminate the need for LEAs to submit teacher credential information to the CDE.
Growth Model – Bill Padia
- Bill Padia commented that the legislature and others have great anticipation and expectations for a value-added growth model system in California since SSIDs were assigned to all students two years ago. The CDE has been doing internal analyses and working with Educational Testing Service (ETS) to determine the feasibility for such a system that would match test scores from year to year. Because the CSTs are not vertically scaled tests, the feasibility for a value-added system is unknown. The CDE continues to investigate options in this area.
- In 2003, Senate Bill 257 provided funds to study matched longitudinal scores compared with cross sectional data for an accountability system. The resulting paper by Edward Haertel suggested that scores on statewide tests would need to be vertically scaled in order to compare results across years. There may be other possible ways to use current data, but the results are not optimal. ETS also conducted a feasibility study to determine alternatives using existing tests. This study produced some alternatives that were very complicated. The CDE has presented to the State Board of Education (SBE) a simplified alternative method for a value-added system using Z scores, but the SBE did not support this approach. All except one of the states that have growth models approved by the U.S. Department of Education (ED) have vertically-scaled tests that are used. The exception state uses a Z score methodology.
Committee Comments
- Evaluation
Chuck Weis asked about the evaluation of the PSAA and whether the evaluation findings included unintended consequences or other issues that could serve to guide the recommendations of the PSAA Advisory Committee. Bill Padia said a first evaluation of PSAA focused on the Immediate Intervention/Underperforming Schools Program (II/USP), and a second evaluation focused on the High Priority Schools Grant Program (HPSGP). Evaluation findings have tended to suggest only small changes if a school participates in these intervention programs and that district involvement in the intervention process for a school is critical. On average, HPSGP schools showed gains in student performance during the period of program implementation. However, the effect on student performance of participating in the program was negligible and not statistically different from the gains of the comparison schools in the analyses. Vicki Barber mentioned that the Alternative Schools Accountability Model (ASAM) accountability requirements are currently being revised. Chuck Weis suggested that steps be taken to obtain funding for an additional formal evaluation of the PSAA.
- Relationship of API and AYP
The committee suggested it would like to review the relationship of API and AYP and to discuss future priorities. Pat McCabe said that the aspects of API and AYP are hard to describe as one cohesive system. For example, some schools that have very high API growth have not made AYP. Bill Padia said that California has submitted a request to the ED to use a growth model for No Child Left Behind (NCLB) requirements, but that request has been denied.
Future Meetings
Future meetings of the PSAA Advisory Committee will be held on August 22, 2007, and October 30, 2007.
Holly Jacobson adjourned the meeting at 3:32 p.m.
