Most of the questions are organized by criteria category. Under each category, the criterion referenced in the question(s) is stated first, followed by the question(s) and the answer(s). Questions not specific to the criteria are included at the end of this attachment.
Category 1: Science Content/Alignment with Standards
Category 1, Criterion 1
Content that is scientifically accurate.
Question 1
How many citations are expected for Category 1, Criterion 1, if the judgment is “yes”? (So, if it is “accurate,” how much proof is asked for?)
Answer
For each criteria category on the Report of Findings, you will be required to list several exemplary citations of where you as a panel found that the program met the criteria in that category. You may not have a citation for each criterion in that category, but you should have a representative number of citations that support your recommendation.
Category 1, Criterion 2
Comprehensive teaching of all California Science Standards at the intended grade level(s) as discussed and prioritized in the California Science Framework, chapters 3 and 4. The only standards that may be referenced are the California Science Standards. There should be no reference to national standards or benchmarks or to any standards other than the California Science Standards.
Question 2
Category 1, Criterion 2, all California science standards, does this include Investigation and Experimentation Standards?
Answer
Yes, the instructional materials must cover all the Investigation and Experimentation Standards.
Question 3
If the standards are covered in the teacher’s edition but not in the student’s edition, is that considered covering the standard?
Answer
If the standards are covered in the teacher’s edition but, according to the way the program is to be implemented, that material is not accessible to the students, then the standards are not being covered.
Question 4
How is it assured that textbooks are accurate on pages not referenced in the standards map?
Answer
It is assured because the Instructional Materials Advisory Panel (MAP) and Content Review Panel (CRP) members will be reviewing the whole program for content accuracy.
Question 5
Grade 5, Standard 1c, do publishers need to include the element name and chemical symbol: aluminum (Al)?
Answer
Grade 5, Standard 1c, states: “Students know metals have properties in common, such as high electrical and thermal conductivity. Some metals, such as aluminum (Al), iron (FE), nickel (Ni), copper (Cu), silver (Ag), and gold (Au), are pure elements; others, such as steel and brass, are composed of a combination of elemental metals.” When pure elements or a combination of elements are discussed in the program, the criterion requires that the publisher include examples such as those listed in the standard but would not have to include the elemental symbol.
Category 1, Criterion 3
Multiple exposures to the California Science Standards (introductory, reinforcing, and summative), leading to student mastery of each standard through sustained effort.
Question 6
“Multiple exposure,” is this for each standard or in general (Category 1, Criterion 3)?
Answer
The criterion states: “Multiple exposures to the California Science Standards (introductory, reinforcing, and summative), leading to student mastery of each standard through sustained effort.” Science programs must provide multiple opportunities for students to master each content standard at the appropriate grade level.
Category 1, Criterion 4
A checklist of California Science Standards in the teacher edition, with page number citations or other references that demonstrate multiple points of student exposure, and a reasonable and judicious allotment of instructional time for learning the content of each standard. Extraneous lessons or topics that are not directly focused on the standards are minimal, certainly composing no more than 10 percent of the science instructional time.
Category 1, Criterion 7
Evidence in the teacher edition that each hands-on activity directly covers one or more of the standards in the California Science Standards (in the grade-appropriate physical, life, or earth sciences strands); demonstrates scientific concepts, principles, and theories outlined in the California Science Framework; and produces scientifically meaningful data in practice. All hands-on activities must be safe and age appropriate.
Question 7
Can a hands-on activity ever be considered extraneous? Category 1, Criterion 7, places a requirement on every hands-on activity, which appears to exclude them from the 10 percent extraneous material allowed by Category 1, Criterion 4. So if we find a hands-on activity which does not “directly [cover] one or more of the standards,” must the proposal be rejected, or is the term “every” in Criterion 7 to be understood implicitly to have the 10 percent flexibility given in Criterion 4?
Answer
Category 1, Criterion 7, requires: “Evidence in the teacher edition that each hands-on activity directly covers one or more of the standards in the California Science Standards…,” so hands-on activities are excluded from the Category 1, Criterion 4, that allows for 10 percent extraneous lessons or topics.
Question 8
Clarify “no more than 10 percent extraneous lessons or topics,” keeping in mind that teachers of grades four and five require scaffolding lessons for their students who have never been taught science in the lower grades.
Answer
Extraneous means lessons or topics that students do not have to master in order to master the California Science Standards. Scaffold activities that are essential to understanding the science standards are, in fact, leading to mastery of the content standards and are not considered extraneous.
Question 9
Is material that appears out of grade level considered extraneous (e.g., grade-level three material that appears in the other grade-level text)? At what point is an extension to a topic (which now covers content at another grade level) considered extraneous?
Answer
If, for example, a grade three program includes topics or lessons that cover some standards in grade four, it would be considered building toward mastery of the content standards, so would not be considered extraneous.
Category 1, Criterion 5
A table of evidence in the teacher edition, demonstrating that the California Science Standards can be comprehensively taught from the submitted materials with hands-on activities composing at least 20 to 25 percent of the science instructional program. Hands-on activities must be cohesive, be connected, and build on each other to lead students to a comprehensive understanding of the California Science Standards.
Question 10
How do you define a “hands-on” activity specifically for the 20-25 percent criteria?
Answer
Criteria Category 5, Criterion 9, states: “Suggestions for how to adapt each hands-on activity provided to other methods of teaching, including teacher modeling, teacher demonstration, direct instruction, or reading, as specified in the California Science Framework.” Based on this criterion, teacher modeling, teacher demonstration, direct instruction, and reading are not hands-on activities. Generally, a hands-on activity can be defined as any educational experience where students are actively engaged in manipulating materials to gain knowledge or understanding of the science content standards.
Question 11
In the criteria for evaluating instructional materials in science, kindergarten (K) through grade 8, Category 1, Criterion 5, states hands-on activities at least 20-25 percent; yet the Science Framework, Chapter 1, page 11, states a maximum of 20-25 percent. Which do we go by?
Answer
The criteria governing this adoption are in the addendum included with your Science Framework (framework) identified as: “This reprint of Chapter 9 replaces the chapter printed in this framework (2004 edition).” The criteria are also included in the Invitation to Submit document (beginning on p. 49) and as a handout behind the “Evaluation Criteria” tab included in your training binder.
Question 12
“…20 to 25 percent of the science instructional program…,” is that 20 to 25 percent by time spent or grade outcome e.g., lab grade?
Answer
It is 20-25 percent of the instructional program.
Category 1, Criterion 6
Investigations and experiments that are integral to and supportive of the grade-appropriate physical, life, and earth sciences standards so that investigative and experimental skills are learned in the context of those content standards. The instructional materials must include clear procedures and explanations, in the teacher and student materials, of the science content embedded in hands-on activities.
Question 13
Does “clear procedures” imply a cookbook approach to experiments? Is there any room for some inquiry/exploration approach?
Answer
No, “clear procedures” does not imply a cookbook approach to experiments, and yes, there is opportunity for the inquiry/exploration approach. The framework provides guidance in Chapter 1, where it states: “Instructional materials need to provide teachers with a variety of options for implementation that are based on the science standards.” (Science Framework, Chapter 1, p. 6)
Category 1, Criterion 9
Extensive, grade-level-appropriate reading and writing of expository text and practice in the use of mathematics, aligned with the Reading/Language Arts Framework for California Public Schools and the Mathematics Framework for California Public Schools, respectively.
Question 14
Writing of expository text – should it be a direct lesson to students or stated in the teacher edition as a suggested follow-up?
Answer
If there are suggested activities for writing expository text in the teacher’s edition but, according to the way the program is to be implemented, that material is not accessible to the students, then this criterion is not met.
Category 1, Criterion 10
Examples, when directly supportive of the California Science Standards, of the historical development of science and its impact on technology and society. The contributions of minority persons, particularly those individuals who are recognized as prominent in their respective fields, should be included and discussed when it is historically accurate to do so.
Question 15
Examples, how much is enough?
Answer
There is no specific number of examples required. The absence of examples of the historical development of science and its impact on technology and society that directly support the standards would suggest that this criterion is not met.
Category 2: Program Organization
Category 2, Criterion 3
Clearly stated student outcomes and goals that are measurable and are based on standards.
Question 16
If the criterion states “…based on standards,” do the standards have to be explicitly stated?
Answer
No, if the criterion states “based on the standards,” the standards do not have to be explicitly stated.
Category 2, Criterion 5
A program organization that provides the option of preparing for or pre-teaching the science content embedded in any hands-on activities.
Question 17
Does preparing for or pre-teaching the science content refer to materials for the teacher to set up the activity? (“Discovery” approach option for pre-teaching)
Answer
This is not referring to the materials to set-up the activity. It requires that publishers provide the option of preparing for or pre-teaching the science content embedded in any hands-on activities. If publishers use the “discovery” approach, they must provide a way of preparing for or pre-teaching the science content embedded in the activity.
Category 2, Criterion 6
A program organization that supports various lengths of instructional time and helps make efficient use of small blocks of time (that may be available during the instructional day) in kindergarten through grade three.
Question 18
Does support various lengths of instructional time require a listing of approximate time to complete? Does this pertain only to Kindergarten through grade three?
Answer
No listing is explicitly required under this criterion, though it is a requirement in Category 5, Criterion 2, which states: “A checklist of program lessons in the teacher edition, with cross-references to the standards covered, and details regarding the instructional time necessary for all instruction and hands-on activities.” Category 2, Criterion 6, applies only to Kindergarten through grade three.
Category 2, Criterion 8
Support materials that are an integral part of the instructional program. These may include video and audio materials, software, and student workbooks.
Question 19
Program Organization #8, support materials that are an integral part of the program… Define “integral.” Clarify the difference between outcome and goal as perceived by the publisher.
Answer
Integral means support materials that are clearly connected to the instructional program.
In Publishers Bulletin 1, question 11, publishers were given the following guidance regarding the distinction between outcomes and goals.
Publishers Bulletin 1, Question 11
What constitutes “student outcomes and goals”? Are these specific questions that students should be able to answer if they have mastered material? How are outcomes different from goals?
Publishers Bulletin 1, Answer 11
Both the criteria and the framework provide the answer to this question. Category 2, Criterion 3, calls for materials that have “Clearly stated student outcomes and goals that are measurable and are based on standards.”
The Science Framework states: “Effective science programs include continual assessment of students’ knowledge and understanding, with appropriate adjustments being made during the academic year. Effective assessment (on a continuing basis through the academic year) is a key ingredient of standards-based instruction.” (Science Framework, Chapter 1, p. 11)
The Science Framework further states: “The science standards are set forth in terms of what students know. Therefore, mastery of an individual standard is achieved when students have actually learned the fact, skill, concept, principle, or theory specified. Mastery does not occur simply because students have received a particular explanation or participated in a particular activity.” (Science Framework, Chapter 3, p. 24)
Later the Science Framework discusses what the assessments should measure:
“The measures need to:
- Reveal the student’s knowledge and skill in science and the ability to apply that knowledge and skill as a foundation for future learning.
- Document the student’s progress (or lack of progress) toward mastering the content standards.
- Provide information useful to planning and modifying future instruction in ways that will help all students master (or exceed) the content standards.
- Help identify and reinforce effective instructional practices.”
(Science Framework, Chapter 6, p. 284)
Category 3: Assessment
Category 3, Criterion 2
Multiple measures of the individual student’s progress at regular intervals and at strategic points of instruction, such as lesson, chapter, and unit tests or laboratory reports
Question 20
With regard to Assessment Criterion 2, does “progress” in the phrase “Multiple measures of the individual student’s progress…” mean progress toward mastery of the standards? If not, where do we evaluate whether the publisher assesses the essence of the standards and not the trivial details.
Answer
“Progress” means that the program must provide multiple measures of students’ progress toward mastery of the science content standards.
Category 3, Criterion 3
Suggestions on how to use assessment data to guide decisions about instructional practices and to help teachers determine the effectiveness of their instruction.
Question 21
How should publishers show me the methods of using the information gathered in assessments? Do they need to specifically tell me what to do with my data or do I rely on my teaching experience and are they going to tell me in the teacher’s edition how to determine the effectiveness of my teaching instruction?
Answer
Publishers should provide suggestions on how to use assessment data to help teachers determine the effectiveness of their instruction.
Question 22
Where do you record within the criteria if the assessment tool and instructional materials are not in alignment?
Answer
There isn’t a specific criterion that requires that the assessments be aligned to the instructional material. You should use your Note-Taking Guide to determine if the program meets the Category 3, Assessment, criteria.
Category 4: Universal Access
Category 4, Criterion 1
Suggestions, based on current and confirmed research, for strategies to adapt the curriculum and the instruction to meet students’ identified special needs.
Question 23
What would current or confirmed research for special needs students mean, look like, and where would it be located? Who evaluates this research requirement? What criteria are we to use for “confirmed research” – federal definition or some other criteria?
Answer
Whenever the criteria state current and confirmed research, they are referencing the current and confirmed research that the framework is based on. So use the framework as your guide for the instructional strategies to adapt the curriculum and the instruction to meet students’ identified special needs. You are not expected to consult resources other than the framework or the content standards to evaluate the instructional materials.
Question 24
Does “special needs” in Category 4, Criterion 1, mean individualized education program (IEP) or 504 only, or does it include gifted and talented education (GATE) or a broader group?
Answer
“Special needs” means the following:
- Students who are below grade level in science learning
- Advanced learners
- Students whose reading, writing, listening, and speaking skills are below grade level
- Students with disabilities – those students with a 504 accommodation plan or IEP.
Category 4, Criterion 4
Suggestions to help teachers pre-teach and reinforce science vocabulary and concepts with English learners.
Question 25
If the publishers label a “suggestion” as appropriate for “English learners,” but IMAP members don’t feel it is English learner appropriate, must we accept it as a suggestion for English learners?
Answer
Your judgment is whether or not the suggestions help teachers to pre-teach and reinforce science vocabulary and concepts with English learners.
Category 5: Instructional Planning and Support
Category 5, Criterion 6
Clear, grade-appropriate explanations of science concepts, principles, and theories that are presented in a form that teachers can easily adapt for classroom use.
Question 26
Category 5, Criterion 6, Are these explanations to be included in the teacher edition, student edition, or both? (It doesn’t say where to include wording for the teacher to follow when planning instruction.)
Answer
The criterion does not state where the explanations must be placed; therefore, the publisher can put them anywhere in the teacher or student editions.
General Criteria Questions
Question 27
When the criteria refer to suggestions/strategies – how much is enough?
Answer
Use your professional judgment as to whether or not there are sufficient suggestions or strategies to meet the criteria.
Framework Questions
Question 28
Should vocabulary support the framework and standards, or can any terminology be selected? Is vocabulary not related to the framework part of the 10 percent extraneous material?
Answer
In general, the framework is considered a model curriculum and a guide for developing instructional materials, so publishers can adopt a different terminology for their instructional materials. The 10 percent extraneous refers to lessons or topics that are not directly focused on the standards, not vocabulary.
Question 29
In view of the sentence, …teaching of all California Science Standards…as discussed and prioritized in the California Science Framework…, just how should we incorporate the framework into our evaluation?
Answer
The framework is the state’s model curriculum, so it shows one way of weaving the California Science Standards into a coherent educational program; but it’s not the only way they can be so woven, and publishers can make different choices. You should use the framework as a guideline for the kind of instructional materials that might be expected in supporting the standards.
Edits and Corrections
Question 30
Will the publishers have a chance to correct specific pieces of materials according to reviewer’s recommendations? Inaccurate illustrations – is this enough to call content inaccurate, or can this be addressed in the report and changes made in illustrations?
Answer
In Publishers Bulletin 5, question 11, publishers were given the following guidance regarding the correction of errors:
Publishers Bulletin 5 Question 11
Is there a way for the correction of errors? For example, if a content error is found, can the publisher correct it?
Publishers Bulletin 5, Answer 11
The Edits and Corrections policy on page 31 of the Invitation To Submit (ITS) states: “Only the corrections and edits defined below will be permissible during the review of instructional materials for adoption.
Edits and Corrections are defined as:
- Inexact language and imprecise definitions
- Mistaken notations
- Mislabeling of pictures, objects, animals, plants, and so on
- Misspellings or grammatical errors
- Computational errors and examples
Changes as defined below will not be allowed for instructional materials submitted for this adoption.
- Revising the program to meet the criteria and standards
- Rewriting of a chapter or section
- Adding new content
- Moving materials from one grade level to another
- Incorrect data, including definitions and factual errors that require content experts to review the materials prior to approval”
Other Questions
Questions 31
Do Content Review Panels (CRPs) review materials of Category 1 – content alignment only – or do they review materials for Categories 1-5?
Answer
IMAP and CRP members review all the instructional materials; however, the CRP members’ primary responsibility is to check the science content. Instructional Materials Advisory Panel (IMAP) members also have a responsibility to review the materials for science content if they have the content expertise.
Question 32
A chart or other organizer to show how much of a publisher’s materials need to be
purchased to cover the complete curriculum, if important parts of the curriculum are in “extras” materials. When judging the materials are we looking at the fact that not all materials may be part of the “basic” overall price – but may require additional purchase? Or are we assuming that all schools would purchase the complete materials?
Answer
IMAP and CRP members should not take potential district purchase decisions into account when conducting their review. The sole determination used by the IMAP and CRP members is whether the submitted programs meet the evaluation criteria adopted by the State Board of Education.
Question 33
Is there/has there been any encouragement for publisher materials to reflect California environments (life science, earth science)?
Answer
The publishers have not been given direction to specifically reference California environments, they are only held to the evaluation criteria. Category 1, Criterion 11, states: “Examples, when directly supportive of the California Science Standards, of the principles of environmental science, such as conservation of natural resources and pollution prevention. These examples should give direct attention to the responsibilities of all people to create and maintain a healthy environment and to use resources wisely.”