Crosscutting

Issues

General Government

The Year 2000 ("Y2K")

Computer Problem

Most governmental agencies will need to make changes to their computer systems to accommodate the year 2000 change. California has embarked on a major effort to fix its computers, headed by the Department of Information Technology (DOIT). The DOIT estimates that it will cost the state at least $240 million over several years to fix the problem, but we believe that this estimate is understated. We have identified a number of problems with the state's effort, including optimistic time lines, lack of contingency planning, and insufficient oversight.

We recommend that the Legislature (1) require departments to provide, during budget hearings, a status of their efforts to modify computers to accommodate the year 2000, (2) delay funding for new projects in those departments which have not modified their computers for year 2000, (3) approve DOIT's budget to provide additional oversight and assistance to departments, and (4) establish a reserve fund from which departments can request monies for yet-to-be identified efforts to address the problems related to the year 2000 conversion.

The budget proposes augmentations totaling $19.6 million by state agencies to fund efforts to modify computers to accommodate the year 2000 (Y2K). This does not include hundreds of millions of dollars in re-directions within departments' budgets to address these efforts. In this analysis, we outline the nature of the problem, what the state is doing about it, the potential costs and risks, and how the Legislature can ensure that the problem is addressed adequately.

What Is the Y2K Problem?

In the 1960s and 1970s when mainframe computer capacity was expensive, programmers established a standard for identification of dates in order to reduce the amount of space needed. By using the last two digits to represent the year (for example, 1973 was designated as 73), computing costs were reduced. With the new millennium approaching, these computer systems must now distinguish between dates in the 1900s and the 2000s. Computers, both in the public and private sectors, are unable to distinguish between these dates and must be modified to accommodate the change to the year 2000.

Failure to make the Y2K change will for some systems simply produce undetectable erroneous calculations, but some systems will completely fail. The "failure date," as it is known, is not necessarily January 1, 2000. For some systems in California, it has been as early as 1995, because these systems were required to provide dates into the future (for example, licenses that were granted in the mid- to late 1990s that will expire after the year 2000).

The potential risk for the state if computer systems are not converted is significant. For example, the Department of Motor Vehicles reports that the failure to convert its computer systems could result in $3 billion in lost revenue annually. The Department of Consumer Affairs reports a potential annual revenue loss of $241 million.

The DOIT identified almost 3,000 state computer systems; of these 1,100 are either fixed already or do not need to be fixed. Of the remaining 1,900 systems, 650 are mission critical, meaning that they enable the department to carry out its primary responsibilities such as issuing drivers' licenses, collecting taxes, et cetera. The remaining 1,250 systems were identified as essential but not mission critical and will also need to be fixed.

How Can the Problem Be Fixed?

In an effort to correct the problem, hundreds of millions of dollars are being spent on replacing, retiring, or modifying computer systems in state government. Each department must decide whether it is going to eliminate the system if it is no longer necessary to meet the department's business needs, replace a system which is essential but not currently meeting the business needs, or modify an existing system to become Y2K compliant. These efforts are collectively known as "remediation."

Several Methods of Y2K Remediation Available

In addition to simply replacing an existing system, a department can fix an existing system through several different methods. First, the department must determine how to identify within all of its lines of codes the dates that need to be changed. In some cases, an automated tool can read the lines of code and determine where the dates are and change them with the department's chosen new date standard. Other cases will require all the work to be done manually, taking much longer. Many systems will require a combination of the two approaches.

Once identified, there are several ways to fix the code. The code can be changed to accept dates with the entire year included (1973 rather than 73, 1910 or 2010 rather than 10). An alternative method is referred to as "windowing" in which the system is told that any date between 00-10 should be considered in the 2000s and any date from 11 through 99 should be considered in the 1900s. A department can choose one of these or many other solutions, depending upon what type of code it has and how much the existing code has been customized over the years. The more customized the code, the more manual intervention will be necessary.

Some of the methods of remediation, such as windowing, should be considered short-term fixes because dates will once again become a problem as the calendar nears the end of the window. Those departments that use a short-term fix may find that they still have to replace an entire system in the near future.

The State's Y2K Program

Shortly after DOIT was established in 1995, it began to make other departments and agencies aware of the impact the year 2000 would have on computer systems. Departments received a "white paper" on the issue in October 1996. One month later, DOIT distributed to departments its official Y2K Program Guide.

The DOIT's Y2K Program Guide lists the steps a department should go through to establish a plan for modifying the affected systems. It requires departments to create an inventory of existing systems; identify those systems that are critical to the overall mission of the department (referred to as "mission critical systems"); assess the impact the century change will have on these systems; and develop a plan to fix the systems. Departments had to identify the impact its computer systems have on the department, outside entities with whom it exchanges information, and the public. In addition, DOIT required departments to report this information as well as budget and Y2K conversion schedule in June 1997 with quarterly updates thereafter. Executive Order W-163-97, issued in October 1997, requires all state departments to complete their Y2K remediation efforts by December 31, 1998.

What Will it Cost?

In January 1998, DOIT estimated that remediation of the mission critical computer systems in state government will cost at least $240 million over several years. As we discuss below, we believe this estimate understates the total costs to the state because it does not include a number of project components and costs.

Current-Year Funding. The 1997-98 Budget Act (Item 9899) appropriated $55 million for Y2K conversion costs. This included $25 million from the General Fund, $25 million from special funds, and $5 million from non-governmental cost funds. As of early January 1998, 14 departments have requested and received about $44 million.

Budget-Year Funding. The Governor's budget does not propose to include funding for Y2K efforts in Item 9899 for 1998-99, but rather includes funding for Y2K projects in individual departmental budgets. The budget proposes total augmentations of $19.6 million in 12 departments for 1998-99. This amount does not include funds in the baseline budgets. The Department of Finance has not been able to provide information on how much departments have in their baseline budgets for Y2K efforts.

Analyst's Assessment of the Administration's Program

As we have indicated previously, DOIT has made the Y2K Program a top priority and has taken a number of positive steps to address the issue. It started work on the problem early, developed the Y2K Program Guide, conducted education and training seminars, and provided information and assistance to state departments. However, we have identified a number of shortfalls and missing elements in the program.

Essential, But Nonmission Critical, Systems Need to Be Fixed, Too

In addition to performing tasks on mission critical computer systems, each department will need to review its essential, but nonmission critical computer systems. To date, the focus has been on the mission critical systems so that departments can continue to provide programmatic services such as issuing business licenses, registering vehicles, and collecting taxes. Such a focus on mission critical systems is appropriate, however, there are many essential computer systems which the departments will need to fix in order to administer the department. Systems such as human resources, accounting, inventory, and others are not currently included in many department's plans to become Y2K compliant, yet will need to be fixed. Such efforts will add to the state's current cost estimate for remediation.

Embedded-Chip Technology. Nontraditional information technology, known as "embedded chip" technology, also needs to be fixed to accommodate Y2K. This includes elevators, voice mail systems, security systems, heating and air conditioning systems, as well as many other aspects of state government's daily functions. At this time, few vendors have indicated an ability or willingness to perform remediation efforts on the nontraditional systems. Although DOIT's program requires departments to indicate whether they have created plans and embarked on these efforts, the program does not require this to be included in the department's cost estimate, and DOIT does not review these plans. Thus, the state's planning and budgeting for these efforts is in the preliminary stages.

Costs Are Underestimated

As we indicated earlier, the DOIT estimates that the total costs of Y2K remediation will be $240 million over several years. However, this estimate does not include a number of direct and indirect costs to the state, including:

Who Pays?--Contracts Need to Be Reviewed

State departments have entered into thousands of contracts over the years to purchase hardware, software, and services. It was not until recently that these contracts contained language that required the vendor to provide hardware and software that was year 2000 compliant. Thus, in addition to fixing mission critical systems, departments must review existing contracts to determine whether the vendor or the state is responsible for paying for these remediation efforts. In most cases, this review is not currently being done. Some contracts require the vendor to ensure the system is maintained and "bug" free. Other contracts may require the state to accept this fiscal responsibility.

Additionally, each contract needs to be reviewed to determine responsibility for liability for noncompliance. If a major system is not remediated in time, the state runs the risk of being sued by someone who alleges harm as a result of the system's failure. As we indicated above, industry experts predict that legal expenses could add 40 percent to projects costs. Forty percent of the existing $240 million estimate is approximately $100 million for state government. Determining which failures pose the greatest risk to the state should help to prioritize remediation efforts throughout the state and thus reduce potential liability. Review of existing contracts by qualified legal counsel is an important component to the total effort. The DOIT proposes in its 1998-99 budget to provide these services for all state departments that request it.

Contingency Plans Lacking

Although DOIT required departments to provide contingency plans in their initial reports, our review indicates that few departments have such plans. For departments whose systems have already failed (because they did not meet their conversion date), it is unclear what has been done to enable the department to continue to provide its core business services.

Time Lines Are Too Optimistic

As we indicated above, the executive order provides that all departments complete remediation efforts by December 31, 1998. This is probably an optimistic time line for three reasons.

Testing. The time lines outlined by the departments and reported by DOIT do not provide sufficient time for testing. After the computer code is fixed, the system must be tested. It is generally accepted in the information technology industry that 40 percent of total project time and cost should be devoted to testing. Testing is a critical component of a project because it is at this phase that problems are identified. If adequate time is not set aside for testing, and significant problems are identified when the system is tested, the project may not be completed on time. Once testing has proven successful, the system is put into "production." This is the process in which the system is actually implemented and workers are trained to use it.

Contingency Planning. The time lines do not incorporate time to activate or invoke contingency plans in the event a remediation effort fails to meet project milestones.

Entities With Whom the State Exchanges Data. The state exchanges data with many entities including the federal government, welfare offices, car dealerships, hospitals, schools, and emergency response services. The time lines do not always include enough time for the state department to develop a "bridge" with these partners in the event the partner is unable to modify its systems in time. Without a bridge, even if the state modifies its system on time, it will be unable to exchange data with the confidence that the data are accurate.

What Should the Legislature Do?

The Need for Oversight

The consequences can be significant if a state department does not make the Y2K conversion on time. Departments which rely on computer systems to carry out their business, whether it be to collect revenues, ensure public safety, or regulate an industry, need close legislative and executive branch scrutiny. We are concerned that DOIT is currently unable to provide the necessary level of oversight with its limited staff (we discuss this issue below and in our analysis of DOIT contained in this chapter).

As a result of the lack of adequate oversight, the Legislature will not be apprised of the full extent of the problem, efforts being undertaken to fix it, the budget required to complete these efforts, and the fiscal effects if systems are not fixed on time. Since this is the largest information technology project the state has ever undertaken and it has significant ramifications to all citizens, closer monitoring by both the administration and Legislature is warranted.

Require Departments to Report on Progress During Budget Hearings

We recommend that the budget subcommittees require departments to provide a status report on efforts to become Year 2000 compliant during hearings on the 1998-99 budget.

During budget hearings, the budget subcommittees should require each department to provide a status report on its progress in making its systems Y2K compliant. Some departments which submitted plans to DOIT did not indicate how they were going to fix embedded chip problems; what the impact to the General Fund and special funds would be if its systems failed; potential legal exposure; or what they intended to do should their plans fail. The DOIT does not know whether these departments have a backup plan if initial efforts to fix the computer systems are unsuccessful. If a large computer system which issues billings or tracks collections is not remediated before its failure date, the state could stand to lose hundreds of millions of dollars. Likewise, should a computer system which keeps records for law enforcement not be expected to be remediated on time, the Legislature should know of the potential problems beforehand.

According to DOIT's Y2K project time line, all departments should now be done with developing Y2K solutions and should be in the testing phase. If departments are not in the testing phase, it may be a sign that the state will be at risk of not meeting the deadline. Many department's plans contain a minimum number of days between the time at which the first failure date occurs and the time at which the system will be fixed, known as the "cushion" time. This is a key factor in determining risk. A cushion is needed to allow for resolving unanticipated problems due to the complexity of the Y2K modifications. The less cushion time, the higher the risk to achieving success.

Here are some of the questions that the budget subcommittees should ask of departments during budget hearings:

We recommend that the Legislature require every department, during budget hearings, to provide an update on the status of its efforts to complete remediation efforts. This would allow the Legislature to direct the departments to improve their efforts, develop contingency plans, or take other steps to ensure that the negative impacts of failure to remediate systems are minimized.

Deny Funding for New Information Technology Projects Until a Department Has Completed Y2K Projects

We recommend that the Legislature deny the budget request for a new information technology project in any department that has not completed its year 2000 projects.

In order to encourage departments to focus their resources on Y2K remediation efforts, the Governor issued an executive order (W-163-97), which requires the deferral of new computer systems unless the proposed new system is legislatively mandated or is necessary to help the department become Y2K compliant. A preliminary review of approval letters indicates 28 new projects (totaling over $200 million) were approved in the three months since issuance of the executive order. Of those, only 11 could be identified as being mandated to meet a department's business needs or necessary for a department to become Y2K compliant.

Some of these nonmandated projects are being requested by departments which will not have their systems compliant before they fail or before the Governor's deadline of December 31, 1998.

Due to the nature of the impact of the century change on nearly every aspect of an organization's mission and the immovable deadline, any new project will likely deflect resources from Y2K efforts. For this reason, we recommend that the Legislature deny funding new projects until the department has its mission critical systems working correctly or the system is mandated by law. In addition to mission critical systems, departments still need to make their essential but nonmission critical systems Y2K compliant. The Legislature may consider requiring a department to have all of its systems substantially compliant before approving funding requests for new systems. Failure of the systems that are essential, but not mission critical, may nonetheless significantly affect the ability of an entity to do business effectively.

Approve the DOIT's Y2K Program Funding Request for 1998-99

We recommend that the Legislature approve the Department of Information Technology's request for additional resources for its Year 2000 office.

In our analysis of the DOIT budget (later in this chapter), we recommend approval of the department's request for additional resources for its Y2K office. Currently, DOIT is unable to review many department's plans for, and monitor progress toward, remediation. We believe that close monitoring of departmental Y2K efforts by the state's information technology office is necessary, and recommend that they be provided with the tools to do so. We believe that the funding request is the minimum necessary to accomplish the task.

Establish a Reserve Fund

We recommend that the Legislature consider setting aside a reserve fund in the 1998-99 Budget Bill for additional Year 2000 projects.

The state is involved in a multiyear effort to be Y2K compliant. As we indicated earlier, the state's estimate of Y2K remediation costs is probably understated. This is because not all of the plans filed with DOIT include remediation of all systems (especially essential, but not mission critical systems, and embedded chip), implementation of contingency plans, and legal expenses. We believe that the Legislature is likely to receive additional funding requests for these efforts in addition to what is in the Governor's proposed budget.

Departments are refining their cost estimates as they further define the scope of their Y2K remediation efforts. Departments will need to perform the same remediation efforts for nonmission critical systems, potentially invoke contingency plans, review existing contract terms, and defend themselves in court. Many of these expenses have not been identified yet and are not accounted for in the proposed budget.

The Department of Finance is only tracking the augmentation requests made by departments. It is not tracking how much departments are actually spending on Y2K efforts. Nor does it plan to track the Y2K efforts in a way to be able to plan for revenue shortfalls in the General Fund or special funds should these systems not be fixed on time. This will be problematic if a computer system at a department that is charged with collecting revenue does not function properly.

Since departments will not be able to wait until the 1999-00 Budget Actfor appropriations to resolve these yet-to-be identified efforts, and shortfalls in revenue collected poses a potential problem to the General Fund, the Legislature should consider setting aside funds to resolve these problems. Although some level of redirection of resources by departments is appropriate, we believe that additional resources will be necessary. In the 1997-98 Budget Act, the Legislature appropriated $55 million (all funds) for Y2K remediation efforts. We believe that a reserve fund of at least that magnitude is warranted for 1998-99 above the amounts proposed for the individual departmental budgets.




 
Return to 1998-99 Budget Analysis Table of Contents
Return to LAO Home Page