To demonstrate achievement in the California English-language-arts academic content standards from the Writing Applications strand, students must successfully respond to one on-demand writing task. The writing task will be a response to a reading passage or a response to a writing prompt. With a response to literature, students are asked to analyze the passage and write a text-based response. With a response to a writing prompt, students are asked to write a response based on their own knowledge and viewpoints about a given topic. The California High School Exit Examination (CAHSEE) 4-point Response to Literary/Expository Text Scoring Guide and Response to Writing Prompt Scoring Guide, which are based on the Writing Applications content standards, are used to score the writing task beginning on page 115 of the Preparing for the California High School (CAHSEE): An English–Language Arts Study Guide.
Scorers Selection and Training
- Who scores the writing task and how are the readers selected?
- Are California teachers involved in the scoring process?
- How does a potential reader qualify to be a CAHSEE reader?
- How are the readers trained to score the writing task?
- How are scoring leaders selected?
- How are the training papers chosen?
- What scoring guide do readers use to evaluate the writing task?
- Is the writing task scored on computers?
- How many times is each student’s writing task scored?
- How is the accuracy of scoring maintained throughout the scoring process?
- What percent of the total CAHSEE score is the writing task worth and how are the scores reported?
- What are the guidelines for sensitive topics on the CAHSEE?
The people who score the writing tasks are trained and experienced in the scoring of the California High School Exit Examination (CAHSEE) writing tasks. Most are educators who hold full-time teaching positions. Minimum qualifications require that readers have a bachelor’s degree and complete a rigorous training and certification process. Preference is given to those with a bachelor’s degree in English or a related field.
Yes, California teachers are involved in several aspects of the scoring process. They may participate in the field testing of new California High School Exit Examination (CAHSEE) writing prompts, help select sample student compositions to be used for training, serve as readers, and serve as scoring leaders.
After completing the CAHSEE Online Scoring Network (OSN™) Tutorial, readers must pass an official certification test (a total of 40 writing tasks, 20 from “Response to Literary/Expository Text” and 20 from “Response to Writing Prompt”) before being eligible for consideration as readers for the CAHSEE. To pass, readers need to score at least 70 percent of the papers with the exact score and have no discrepancies (more than one point apart from the predetermined score).
Before each scoring session, each reader must pass a calibration test of ten papers (from the item type he or she is scheduled to read during that session) with seven exact scores and with no discrepancies (more than one point apart from the predetermined score). A reader has up to two opportunities on three discrete calibration tests, with retraining by the scoring leader between each set. If a reader does not pass one of these sets, he or she is not allowed to score.
Each prospective reader is required to take an extensive online tutorial. The tutorial consists of the following:
- General information about the Online Scoring Network (OSN™) system
- Background information about the California English-language arts standards
- Extensive information on the different composition types
- Explanations of CAHSEE scoring guides and scoring principles
- Detailed topic notes on selected topics
- Sets of annotated training papers for each selected topic. The training materials include Benchmarks and Rangefinders. Benchmarks are solid samples of student writing for each score point (1–4). Rangefinders are samples of student writing which demonstrate the “high” and “low” end of each score point.
The tutorial also includes a practice “certification test.”
The most experienced readers with the highest accuracy rate on the certification test are selected to be scoring leaders. Scoring leaders monitor and mentor readers during operational scoring. There are approximately six readers assigned to one scoring leader; this allows scoring leaders to work closely with each reader. The Online Scoring Network (OSN™) also allows scoring leaders to read essays simultaneously with their readers, so they can easily work with readers on individual papers.
The majority of scoring leaders participate in a one-day training session. This training covers the California English-language arts content standards, the scoring guides, the operational topics, and a wide variety of sample papers. The training also includes detailed information on how to work with readers and how to use the (OSN™) system. During operational scoring, additional newly-selected scoring leaders are individually trained by an experienced scoring leader.
Working with experienced readers, content specialists review student writing tasks from the field tests to look for representations of each of the score points and the variety of approaches used to respond to the prompt. They choose samples that illustrate the criteria in the scoring guides and the diverse ways in which students may have responded to the topic. The responses are then put into sets used for training and qualifying readers.
The CAHSEE writing task can either be a “Response to Literary/Expository Text” composition or a “Response to Writing Prompt” composition. Readers use a specific scoring guide for each of these writing tasks.
The scoring guide used for a “Response to Literary/Expository Text” writing task includes the criteria in the grades nine and ten California English-language arts (ELA) content standard, “Writing Applications 2.2, Write Responses to Literature.” The literature (text) may include drama, a short story, poetry, or nonfiction (e.g., essays, autobiographies, biographies, or memoirs) written in a literary style. The text also may be informational, persuasive, or functional texts (e.g., consumer materials or workplace documents). The scoring guide reflects the need for textual support in order to earn the higher scores.
The scoring guide used for a “Response to Writing Prompt” task includes the criteria in the grades nine and ten California ELA content standard “Writing Applications:”
- 2.1–write biographical compositions
- 2.3–write expository compositions
- 2.4–write persuasive compositions
The scoring guide reflects the skills needed to write a composition that is not related to a given text. The writing tasks and sample student responses from previous CAHSEE administrations can be found in the English-Language Arts Study Guide (see page 99 for writing tasks and page 115 for scoring guide).
Yes. All student writing responses are scanned into the state-of-the-art OSN™ system. (The OSN™ system is currently used to score such high-stakes writing assessments as the Graduate Management Admission Test-Analytical Writing Assessment, the Graduate Record Examination Writing Measure, and such programs as the Test of English as a Foreign Language and College Level Examination Program.) The readers view each response via the Internet on individual home computer screens where they also assign the scores. The screen does NOT display any background information about the student; the reader sees only the actual essay. Scoring leaders monitor readers from their individual home computers.
Scores are assigned independently by each reader. Each student response is scored twice, once by one reader and once by another reader; the average of the two scores constitutes the reported score.
If the two scores are non-adjacent (more than one score point apart), the response is assigned a third score by a scoring leader. In these cases, the reported score is determined by averaging the two closest scores. If the third score falls in between the first two scores, then all three scores are averaged.
If a paper is difficult to score and needs the expertise of the scoring leader to determine the reported score, the reader defers the paper to the scoring leader to be scored.
The quality of all scoring is monitored on a regular basis.
First, the consistency between readers is calculated based on whether the first and second readers assigned identical, adjacent, or non-adjacent scores. Scoring leaders and test developers constantly monitor agreement percentages. If a reader’s rate of agreement begins to get too low, the reader is retrained by the scoring leader and closely monitored thereafter. If the reader’s performance does not improve, the reader is released.
Second, ten percent of the papers that are read by each reader have been previously scored by content specialists and scoring leaders. The consistency of the reader’s ratings with those of the predetermined scores is checked several times a day to validate that each reader applies the criteria in the scoring guides accurately to each student response.
Third, the scoring leaders back-read and formally record the scores for a percentage of the scored papers to ensure reliability throughout the scoring process. “Backreading” is the process of reading papers that have already been scored by a reader to verify his or her consistency.
Fourth, to help keep readers focused, readers must successfully score a calibration set of papers before each scoring session. Failure to do so locks them out of scoring for that session.
A student’s writing task (essay) can receive a 1, 2, 3, or 4 score. Since each essay is scored by two readers, the scores are averaged. The averaged score for each essay is reported to the student and the schools. For example, if a student’s essay received a 4 from one reader and a 3 from another reader, the scores are averaged, and the reported score is 3.5.
CAHSEE English-language arts (ELA) scores are reported on a scale ranging from 250 to 450. Scores on the writing task are “weighted” to account for 20 percent of the ELA scale score. Multiple-choice scores for reading and writing are summed and then weighted to account for the remaining 80 percent of each student’s scale score.
To keep the California High School Exit Examination (CAHSEE) free from potentially biased, sensitive, or controversial content, the following topics are avoided on the examination:
- Violence (including guns, other weapons, and graphic animal violence)
- Dying, death, disease, hunger, famine
- Natural disasters with loss of life
- Drugs (including prescription drugs), alcohol, tobacco, smoking
- Junk food
- Abuse, poverty, running away
- Socio-economic advantages (e.g., video games, swimming pools, computers in the home, expensive vacations)
- Complex discussions of sports
- Evolution, prehistoric times, age of solar system, dinosaurs
- Rap music, rock concerts
- Extrasensory perception, witchcraft
- Halloween, religious holidays
- Anything disrespectful, demeaning, moralistic, chauvinistic
- Children coping with adult situations or decisions; young people challenging or questioning authority
- Mention of individuals who may be associated with drug use or with advertising of substances such as cigarettes or alcohol
- Losing a job, home, or pets
- Rats, roaches, lice, spiders
- Dieting, other concerns with self-image
- Political issues
- Any topic that is likely to upset students and affect their performance on the rest of the test
It is important to note that these guidelines are applied in the context of the purpose of the test as well as the overall passage or item. For example, some topics (e.g., the socio-economic advantages) may be mentioned in a text, although an entire passage would not focus on these topics.