Legislative Analyst's Office Analysis of the 2001-02 Budget Bill

Assessments

The Governor's budget proposes $221 million to develop, administer, and analyze state assessments, an increase of $97 million over estimated current-year expenditures and more than three times the spending level for this purpose in 1999-00. Figure 1 shows the increase in funding of state assessments over the last two years. Figure 2 provides background on the major state assessments.

The Governor's budget includes five major assessment expansions as shown in Figure 1 and described below.

• High School Exit Exam (HSEE). The largest expansion—$45.6 million—is for the administration of the HSEE even though that exam will not be phased in fully for several years. Ninth graders will take the test for the first time in spring 2001. These same students will be required to pass the exam as a condition of graduation in June 2004. The budget also funds three additional administrations for 10^th grade students in 2001-02.
• English Language Development Test (ELD). The ELD test was field-tested in fall 2000 and will be administered for the first time to the state's 1.5 million English language learner (ELL) pupils in spring 2001. The ELD test assesses how well ELL pupils are learning skills to transition to fluency in English. Over time, the results of this test will allow the state to determine how well schools are doing in achieving this goal.
• Standards-Aligned Standardized Testing and Reporting (STAR). The State Board of Education (SBE) selected an "off-the-shelf" test—the Stanford, ninth edition, form-T (Stanford-9) to be administered by schools to all students in grades 2 through 11 annually. The State Department of Education (SDE) developed the Standards-Aligned STAR to augment the Stanford-9 with questions aligned to the California Academic Content Standards. The spring 2001 administration of the Standards-Aligned STAR will include math, English/language arts, science, social science, and a writing "prompt" (essay test) for grades 4 and 7. In addition, SDE has developed performance standards for each of the standards-aligned tests which will be used to provide student-level results this spring.
• The STAR and HSEE Workbooks. The budget provides $27.5 million to develop, publish, and distribute workbooks to assist students in preparing for the Standards-Aligned STAR and the HSEE.
• The STAR Data Analysis. The budget provides a total of $20 million for incentive funding to districts to purchase or develop software to analyze STAR test data.

Assessment Reauthorization

The Leroy Greene California Assessment of Academic Achievement Act (Chapter 975, Statutes of 1995 [AB 265, Alpert]), reenacted the Golden State Exam (GSE), and created the STAR program, which has been evolving ever since. Figure 2 shows the three components of the STAR program that are currently operating. STAR expansions include:

• Spring 1998. The Stanford-9 was administered for the first time.
• Spring 1999. Standards-Aligned STAR tests in English Language Arts and Math, and the Spanish Assessment of Basic Skills
• (SABE 2) were added to the Stanford-9.
• Spring 2001. Standards-Aligned STAR tests in science and social science for high school grades, and writing prompts in grades 4 and 7.

The provisions of Chapter 975 will sunset in January 2002. As part of any reauthorization, the Legislature will have the opportunity to reexamine the testing system that has evolved. There are five major assessments authorized by Chapter 975. The Legislature will need to determine the role each assessment will play in the future state assessment system, as discussed below.

Standards-Aligned STAR. In the future, we believe this assessment should be the backbone of the state's assessment system. The state has made a significant investment in the development of academic content standards, alignment of textbooks to those standards, and retraining of teachers around the standards. The full implementation of this test, and its inclusion in the Academic Performance Index (API) is an essential step to ensuring that students are learning what the state has determined is needed.

National Norm-Referenced Test (Currently the Stanford-9). Chapter 828, Statutes of 1997 (SB 376, Alpert), required the SBE to purchase an "off-the-shelf" test in order to start administering a statewide assessment of students, even though the academic content standards still were under development. Now that test, the Stanford-9, has become the only measure used for individual and school comparisons, and the only outcome measure used to calculate the API. (See "Accountability" section for discussion of the role of the Stanford-9 in the API). As the Standards-Aligned STAR is integrated into the API, the Legislature will have the option to reconsider the value of the Stanford-9. The Legislature will need answers to three questions about the nationally norm-referenced test:

• Will California public schools need to administer both the Standards-Aligned STAR and a nationally norm-referenced test?
• If the Legislature chooses to continue a nationally norm-referenced test, will it be necessary to administer the test for each grade 2 through 11?
• Is the Stanford-9 the best nationally norm-referenced test, or should the state review other alternatives?

Spanish Assessment of Basic Education, Second Edition (SABE 2). The SABE 2 is a basic skills test of math, language arts, reading and spelling that is administered in Spanish to ELL students in their first year in a school district. The Legislature will want to reconsider the value of this primary language test, and whether its administration is consistent with the intent of Proposition 227 (see discussion of SABE 2 below).

Golden State Exams. The GSE are 13 end-of-course exams for college-track students to determine their achievement level compared to state academic content standards. There may continue to be a role for the GSE to provide end-of-course exams for California's higher-achieving students. This is because the tests (1) provide students with information on how their achievement compares to what the state requires that they should learn in a specific subject, and (2) provides feedback to teachers on how well their students are learning the standards for that course.

Assessment of Applied Academic Skills or "Matrix Test." The matrix test, as required by Chapter 975, was originally planned to be the foundation of the state's assessment system. The test was to be administered in grades 4, 5, 8, and 10, and provide an in-depth school level assessment of how well students had learned each of the subject matter areas covered by the content standards. Although this test has never been developed, it also has never been eliminated from the Education Code. With the Standards-Aligned STAR almost fully implemented, and the HSEE being administered soon, there may no longer be a need for the matrix test.

Eliminate Primary Language Test Requirement

We recommend that legislation be enacted to eliminate the requirement that districts test limited-English-proficient pupils in their primary language because Proposition 227 generally requires schools to transition children to English fluency rapidly. We further recommend redirection of the $1.7 million for the Spanish Assessment of Basic Education Exam—the only one of these primary language exams in use—to other education priorities. (Reduce Item 6110-113-0001 by $1.7 million.)

Under the STAR program, school districts are required to test limited-English-proficient pupils in grades 2 through 11 in their primary language if such a test is available and the pupils have been enrolled in a California public school for less than 12 months. In addition, districts may administer a primary language test to pupils with limited English proficiency who have been enrolled for longer than 12 months. Existing law requires the SBE to designate the single primary language test in each language for which such a test is available for pupils in grades 2 through 11. Currently, the SBE has only designated one such test—the SABE 2, to test Spanish-speaking pupils in reading, language, mathematics, and spelling. The test currently is administered to about 8 percent of the state's 1.5 million ELL pupils because most schools do not administer the test to pupils beyond the required first year. The Governor's budget includes $1.7 million in Proposition 98 funds under Item 6110-113-0001 for the SABE 2, in order to (1) pay the test publisher for materials and exam scoring and (2) reimburse school districts, county offices of education, and charter schools for administering the exam.

Proposition 227, passed by the voters in June 1998, in essence requires that children in California's public schools learn English by being taught in all subjects predominantly in English. Generally, Proposition 227 requires schools to place ELL pupils in special "sheltered English immersion" classes taught predominantly in English, and to limit bilingual education classes to specified exceptional circumstances. The proposition states that (1) ELL pupils should move from special classes to regular classes when they have acquired a good working knowledge of English and (2) normally these classes should not last longer than one year.

Given the changes made by the proposition and its objective to help ELL pupils learn English as quickly as possible, we believe there is no longer a reason for the state to require certain ELLs to be tested in Spanish. For instance, the results from the SABE 2 are not included in the state's current accountability system and are not used to make comparisons of achievement across schools. Moreover, the SABE 2 is rarely of use for assessing a pupil's academic growth, because existing law requires districts to administer the primary language test only once to any individual ELL pupil.

In view of the above, we recommend that legislation be enacted to eliminate the requirement that districts test limited-English-proficient pupils in their primary language if the pupils have been enrolled in a California public school for less than 12 months. We further recommend redirection of the $1.7 million for the SABE 2 to other education priorities. (Reduce Item 6110-113-0001 by $1.7 million.) Under our proposal, districts, county offices of education, and charter schools still would have the option of allocating a portion of their general purpose funds to test ELL pupils in their primary language if local school officials deemed such a test to be useful for local purposes. In addition, the ELD test would still be assessed annually in each grade, providing a comprehensive assessment over time of an ELL's English proficiency based on state standards.

The HSEE Changes Would Create Savings

We recommend that the Legislature (1) reduce funding for administration of the High School Exit Exam (HSEE) by $35 million and (2) require the State Department of Education to report on the costs of developing and administering the HSEE. (Reduce Item 6110-113-0001 by $35 million.)

The Governor's budget provides $61 million to administer the HSEE three times during the 2001-02 school year. Since developing the Governor's budget, the administration has been advised by testing and legal experts to change the way in which the state administers the HSEE. Senate Bill 84 (O'Connell) includes these recommended changes to HSEE administration.

Under current law:

• Ninth graders could voluntarily take HSEE in spring 2001.
• In 2001-02 SDE will offer the HSEE three times to 10^th graders, and 10^th graders would be required to take it at least once.
• In 2001-02, 9^th graders would be able to take the spring 2002 administration of the HSEE.

Senate Bill 84 would change HSEE to:

• Make the spring 2001 test a "practice" for 9^th graders.
• Eliminate administration of the test thereafter to 9^th graders.
• Administer one "census" test each spring to all 10^th graders beginning in 2001-02.

Senate Bill 84 would not change the schedule for 11^th and 12^th graders. Beginning in 2002-03, students in in 11^th grade would have three changes to take the test. Beginning in 2003-04, students in each of these grades would have three chances to take the test.

Assessment experts have advised SDE and SBE that it is important to offer a "census" administration of the test to allow test experts to determine the appropriate "cut score" for passing/not passing the HSEE. In a census administration, all students in a specific grade are required to take the test at the same time. The Governor now proposes that the first census administration for 10^th grade students occur in the spring of 2002. Legal experts have advised the SDE not to offer the census administration to 9^th grade students. Since the test covers material that in some cases meets 10^th grade academic content standards, 9^th graders would not necessarily have had adequate opportunity to learn the material.

Changed Policy Creates Opportunity for Budget Savings. The proposed changes discussed above in the test administration would, if implemented, reduce costs of HSEE administration in 2001-02. Instead of the $61 million proposed in the budget, the state could provide the single administration of the HSEE in 2001-02 for considerably less. We estimate these savings would be at least $35 million. Actual savings would depend on other factors—such as the costs of test item development and funding school district administration of the HSEE. Accordingly, we recommend the Legislature reduce funding for the HSEE by $35 million, and require the SDE to report at budget hearings on the costs of developing and administering the HSEE. Based upon the SDE cost estimates, the Legislature may be able to reduce the HSEE funding level even further.

Should the State Release the Questions on the HSEE?

We recommend that the State Department of Education report on the costs of publically releasing High School Exit Exam (HSEE) test questions, and that the Legislature codify a policy about releasing HSEE test questions.

Currently, the SDE plans to publically release questions on the HSEE to be administered in spring 2001 after the test is given. The Education Code is silent on whether the SDE legally can release the test. The SDE argues that releasing the test questions will help the public see the level of material that is on the test, and may help 10^th grade students prepare for the spring 2002 test. This approach may have merit but releasing the test would increase the cost of administration because test questions would have to be changed each time. We believe the Legislature should determine if the benefits are worth the cost. We recommend that the SDE report prior to budget hearings on the cost impact of releasing test questions. We further recommend that the Legislature codify if and how test questions can be released to the public. Three potential options are: (1) release all questions from all administrations, (2) only release questions from the main (spring) administration, (3) allow the SDE to selectively release test questions.

On-Line Assessments

We recommend the Legislature appropriate $250,000 from the General Fund (non-Proposition 98) to the State Department of Education to contract for a study of the feasibility of administering state assessments on-line because of potential long-run benefits and cost savings. (Augment Item 6110-001-0001 by $250,000.)

Recently, major testing companies have begun offering assessments on-line. There are several advantages to offering tests on-line:

• Instant Results. Since a computer grades the test at the same time that a student takes it, the results can be provided immediately.
• Reduced Grading and Administration Costs. On-line tests eliminate the need for hard copies of the test, which would dramatically reduce costs of printing, shipping, distributing materials, overseeing the collection of materials, and grading the tests. Assuming that the school has a computer lab, the cost to the school district to administer the test also would be less due to fewer logistical requirements.
• Potentially More Accurate. Offering tests on the computer has the potential of increasing the accuracy of reported results if dynamic testing methods are used. A dynamic test offers a different set of questions to a student depending on the student's ability and thus, allows the possibility of "homing in" on a precise assessment of ability. For example, if a student were taking a math test on-line, the testing program could ask ten general math questions at the start of the test. If a student got most of the questions correct, then the testing program would ask the student more difficult questions. If the student missed most of these questions, then the test would ask easier questions. In the same number of questions, the computerized test is able to more accurately determine each student's real skill level.

Based upon the advantages of on-line testing, we believe that it is worthwhile for California to study the feasibility of administering tests on-line. Of course not all tests can administered on-line. For example, offering the STAR test on-line would not be feasible because over four million students need to take the STAR test during a relatively small testing "window." We believe, however, that the following three California tests potentially could be offered on-line.

• English Language Development Test.
• Golden State Exam.
• High School Exit Exam, summer and fall administrations.

These three assessments could be offered on-line because they are offered to small subsets of a school's enrollment at one time. The ELD test has an additional advantage of helping the school make a placement decision in a more timely fashion. The ELD test is administered to ELL pupils when they first move into a school district. Under the current SDE plan, the test takes between six to eight weeks to grade. This forces schools to place the students without the benefit of the ELD test results. Administering the test on-line would eliminate this problem and allow the ELD results to assist schools in placing ELL pupils in the best learning environments.

Based on the significant potential for educational benefits and monetary savings, we recommend the Legislature appropriate $250,000 non-Proposition 98 funds to the SDE to contract for a study of the feasibility of administering California assessments on-line.

The STAR and HSEE Workbooks

We recommend the Legislature approve $27.5 million requested for the initial distribution of assessment workbooks, but do so as a one-time appropriation and provide $5 million annually in future years for annual workbook replacement costs. (Redesignate $22.5 million in Item 6110-113-0001 as one-time spending.)

The Governor's budget proposes $27.5 million for the development and distribution of student study workbooks for the HSEE and STAR. We agree with the need for such workbooks. Providing students with an HSEE workbook will help give them the adequate opportunity to learn the material that is essential for the test to be fair and legally defensible. For the STAR test, workbooks would ensure a level playing field for test preparation. With the most recent administration of the STAR test, the SBE had to review several incidents of possibly illegal test preparations. There seems to be a lot of gray area in what is appropriate test preparation and what is not. The workbooks would ensure equal access to test preparation materials and help reduce this ambiguity.

The Governor's budget funds this program with an ongoing appropriation, and plans to provide new workbooks to each student each year. However, we believe that once distributed to schools, most student workbooks could be reused from year to year. We suggest adopting a five-year replacement schedule, and provide funding to replace 20 percent of the workbooks each year. Therefore, we recommend the Legislature provide $5 million in ongoing funding for the replacement costs of workbooks, and use $22.5 million in one-time Proposition 98 funding for the remaining first-year cost of development and the initial distribution of the workbooks.

Consultant to Align State Tests to Academic Content Standards

We recommend the Legislature move $3 million of one-time non-Proposition 98 funds requested for the State Board of Education's support budget to the State Department of Education's (SDE) support budget for the purpose of hiring consultants to assist in aligning state tests with academic content standards. (Redesignate $3 million appropriation to SDE.)

The Governor's budget proposes $3 million from the General Fund (non-Proposition 98) on a one-time basis to the SBE to hire a consultant to (1) ensure that the STAR and HSEE are aligned to the academic content standards and (2) develop performance measures for the two tests. The funding for this proposal would be provided in an as yet unidentified bill. Although the SBE has oversight authority for the state testing system, the SDE is charged by the Legislature with developing and administering both tests. In addition, the Legislature has authorized SDE to administer all contracting requirements related to the state testing system. Accordingly, we recommend the Legislature redirect $3 million in non-Proposition 98 funding from the SBE's support budget to the SDE's support budget.