Oversight of State Information Technology Projects. The Legislature has devoted considerable attention in recent years to developing, consolidating, and refining the state’s processes for managing the significant risks associated with developing major new information technology (IT) systems, including the 21st Century (TFC) Project sponsored by the State Controller’s Office (SCO). Currently, two state departments have official oversight roles concerning IT projects.
The California Technology Agency (Technology Agency)—to be renamed the Department of Technology effective on July 1—has lead responsibility for various IT functions, including project oversight and risk management (through its Program and Portfolio Management Office) concerning major IT projects sponsored by other departments. The Department of General Services (DGS) has key responsibilities concerning procurement of IT goods and services. However, Chapter 139, Statutes of 2012 (AB 1498, Buchanan), requested that the administration develop a plan for moving IT procurement authority from DGS to the Department of Technology. Chapter 139 continues the Legislature’s efforts to consolidate responsibility for major IT project oversight and risk management in a single place—the Technology Agency.
Human Resources and Payroll Management System. In 2004, SCO proposed the TFC Project, the IT effort to replace the existing statewide human resources management and payroll systems used to pay approximately 294,000 state employees. The new system, also called MyCalPAYS, was intended to allow the state to improve management processes such as payroll, benefits administration, and timekeeping and include self-service access for employees and managers, among other capabilities. The existing systems, commonly referred to as “legacy systems,” were developed more than 30 years ago and are inflexible, fragmented, and costly to maintain. In 2005, the Legislature approved the project with an estimated total cost of $130 million with full implementation scheduled for July 2009.
Two-Phase Procurement. In conjunction with state IT oversight officials, SCO decided to pursue a two-phase, or “unbundled,” procurement approach. This meant the state sought two vendors and undertook two procurements. The first vendor was to supply the software package, and the second vendor (the primary vendor) was to integrate the software to the state’s business requirements. In April 2005, SAP Public Services, Inc. was selected to supply the software package. The system integrator contract was awarded to BearingPoint in June 2006.
Early Issues Delayed Project Development. During 2006 and 2007, SCO asserted that multiple problems had emerged with the work of BearingPoint, the primary vendor hired to integrate the human resources software to the state’s business needs. The vendor asserted that issues with the software package and with SCO caused delays. In October 2007, following multiple schedule delays, SCO issued a breach-of-contract notice to BearingPoint. The vendor and SCO then reached a plan to address project failures and integration continued. These delays extended the schedule by two years and raised estimated total costs to about $180 million.
Vendor Contract Terminated. After several months, BearingPoint once again fell behind schedule, unable to complete project activities and provide deliverables on time. With the project’s schedule and development in jeopardy, DGS issued a default notice to the vendor on December 3, 2008. The notice stated that the vendor failed to (1) properly manage the project, (2) complete designs in a timely manner, and (3) make progress toward development. On January 6, 2009, SCO formally terminated the contract, and primary work on the TFC Project stopped.
Strategy to Move Project Forward Developed. Following the termination of the primary vendor contract, SCO developed a new strategy. In particular, the project scope was narrowed by excluding California State University (CSU) from the project. The CSU has different payroll requirements from those applicable to state civil service employees. The legacy system will continue to process payroll for CSU until a revised system for CSU employees is completed as a separate project. The SCO also decided to select a new system integrator using a two-stage procurement approach, discussed below. The new strategy was documented in Special Project Report (SPR) 3.
New Contract Procured. In March 2009, DGS released a request for proposal for a new system integrator. The procurement was conducted as a two-stage procurement approach. Stage I was the selection of contractors to evaluate the work completed to date and its possible reuse, and to better understand the requirements of the project. Accenture, LLP and SAP were selected in Stage I. Both companies submitted Stage II proposals, which detailed the approach, cost, and schedule for completing the project. In February 2010, SAP was awarded the contract, at which point the project costs and schedule were revised. The project schedule was extended to October 2012, and estimated total costs rose to about $283 million. Implementation was to occur in five phases—known as pilots and waves—where Pilots 1 and 2 would bring a small number of employees into the new system in order to test it prior to Waves 3, 4, and 5, which would fully implement the system in three large and roughly proportional stages (SPR 4 documents these changes).
Subsequent challenges occurred when the project began converting data from the legacy system to the new system. Project management issued a cure notice to the primary vendor, which then subcontracted with a data migration vendor, BackOffice Associates, to improve the data conversion process. SPR 5, the most recently approved project plan, accounted for these delays, and increased the estimated total project costs to $373 million and extended the final wave of the project, Wave 5, by one year—from October 2012 to September 2013.
Pilot 1 Test. On June 11, 2012, the first major test of the state’s new payroll system took place. The test, known as Pilot 1, produced payroll, benefits, timekeeping, and position management activities for about 1,400 SCO employees. This pilot program tested the new system’s functionality with a small number of employees in an effort to identify and correct potential problems before expanding the number of employees covered by the new system.
Problems Encountered During Pilot 1. Although SCO expected minor discrepancies during Pilot 1, significant errors surfaced during the first payroll cycle and persisted through each of the subsequent seven monthly payroll cycles. In particular, incorrect paycheck deductions were made, payroll and pension wages were erroneously calculated, and medical benefits were denied for some employees and their dependents. In one case, employees that took vacation time during the first payroll cycle received compensation in addition to their base salary. Attempts to correct these errors created further problems in the following payroll cycle. In early August, project staff determined that the severity for these issues warranted the delay of Pilot 2, an expansion of the new system to 15,000 employees across numerous departments. Pilot 2 was initially delayed from September 2012 to March 2013. As a result of subsequent challenges, discussed below, Pilot 2 was not implemented.
Cure Notice Issued to SAP.On October 25, 2012, SCO issued a cure notice to the system integrator, SAP, expressing serious concern regarding SAP’s ability to successfully implement the new system. According to SCO, SAP’s lack of expertise and strategic planning lead to inadequate scheduling, staffing, knowledge transfer, deliverable management, and quality assurance. SCO also identified concerns regarding design, testing, organizational change management, and training weaknesses. In total, the cure notice cited 13 grievances and prompted SAP to correct these problems by November 30, 2012, so that the project could move forward.
SAP Responds to Notice.SAP submitted a response to the cure notice on November 30, 2012.In its response, SAP did not assume responsibility for the grievances outlined by SCO and took no action to resolve the issues.
Contract Terminated in Early 2013
On February 8, 2013, SCO terminated its vendor contract with SAP, citing inaction regarding issues listed in the cure notice and a lack of confidence in the vendor to implement the project successfully. At the same time, the Technology Agency suspended further work on the project until a new plan could be established to address the state’s aged payroll systems. SCO states that it will attempt to recover payments made to SAP for system integration costs prior to project termination through the legal process. The primary vendor payments made to SAP total $50 million of the $90 million contract. (A separate payment was made to SAP for the software used in the project, which the state now owns.) At the conclusion of this analysis, Figure 2 shows the timeline of major events from the start of the project in 2004 through the termination of the SAP contract.
State Payroll Reverted to Legacy Systems for Now. Beginning this month, SCO returned the payroll processing for its Pilot 1 employees to the existing legacy systems. According to project staff, SCO had begun to run parallel payrolls on the legacy systems in order to identify inaccuracies and ensure that no pay or benefits discrepancies were left unresolved. As a result of this precaution, the return to the legacy payroll systems for these employees should not pose a problem.
Ballooning Project Costs
As of January 1, 2013, the state had spent $262 million on the TFC Project—of the $373 million estimated total project cost. Total project expenditures were approximately double the entire project cost as estimated when the project began. Estimated General Fund expenditures for the entire project under SPR 5 (prior to the project’s suspension) were $180 million, well over three times as much as the original plan in the 2004 Feasibility Study Report (FSR). Estimated total contract costs with project vendors also increased significantly since 2004, from $59 million to $193 million. To date, the state has spent $134 million for contracting work completed by various project vendors. Figure 1 shows how total project costs and timeline have increased with each successive project plan. Due to the project’s current state, no estimate is possible concerning the eventual costs required to complete the TFC Project, as currently conceived.
2013-14 Governor’s Budget Proposal
Due to uncertainty about the TFC Project, including its primary vendor, project schedule, and near-term costs, the administration did not include a budget proposal for this item in the 2013-14 Governor’s Budget. Instead, the administration included a $38 million ($12 million General Fund) set-aside for the project in Item 9901 of the Governor’s Budget plan, an “off-budget item” used for various fiscal contingencies of this type. (This amount roughly matches the 2013-14 project costs estimated in the most recently approved project plan, SPR 5.) In the coming months, SCO project staff plan to submit an interim SPR for approval by the Technology Agency and review by the Legislature. The details included in the SPR reportedly will inform the SCO’s 2013-14 budget request.
Issues for Legislative Consideration
State Now Faces Difficult Decisions. After the termination of the project’s contract with its primary vendor, the state has spent roughly $260 million on the TFC Project with few tangible deliverables. The Legislature now faces very difficult decisions concerning this project. The Legislature may determine that continuing the project risks “throwing good money after bad” and wastes additional state resources that could be appropriated towards other purposes. The Legislature, on the other hand, must weigh this concern against the importance of updating the state’s aged payroll systems, a critical component of the state’s personnel and financial management. Accurate and timely management of pay and other employee compensation activities is critical. Failure to administer these functions due to a technology failure could result in significant economic disruptions for thousands of state employees and vendors, as well as substantial financial penalties for the state.
Legacy Systems Only Alternative in the Near Term. The state’s legacy payroll systems are comprised of numerous individual components, known as “stove pipe” applications, which were created to address individual payroll needs—such as leave balance management, basic payroll, and warrants to state vendors—as those needs arose. Although the systems lack a central architecture, including consistent data methodology and programming languages, they have been running the state’s payroll for decades. As noted above, the legacy systems are expected to continue managing CSU payroll for the foreseeable future. In general, the legacy systems—though outdated, inflexible, costly to maintain, and at some risk of eventual failure—seem a credible (and, in practice, the only) alternative for the state to use over the next few years. Given the availability of these still-functioning legacy systems, we suggest that the Legislature prioritize thoroughness over timeliness as it evaluates the shortcomings of the TFC Project and considers new alternatives for replacing California’s legacy payroll systems.
Updating the State’s Payroll Processes and Systems Still Worthwhile.The current suspension of the TFC Project means that the state’s need for an updated human resources management system remains unmet. There are numerous functionality and stability issues that justify the continued pursuit of an updated payroll configuration, including the capabilities to respond quickly to payroll changes, issue reports to other state agencies and stakeholders, and allow employees online access to payroll and tax information. Updating the state’s payroll systems also likely would reduce ongoing costs and the likelihood of significant payroll disruptions in future years. In addition, several IT projects are to be integrated with the TFC Project as aspects of their own development, including FI$Cal (the state’s financial management project) and the California Public Employees’ Retirement System’s ITintegration project.
Accountability for Oversight Organizations
The Legislature vested the Technology Agency and DGS with responsibilities to provide greater assurances that large IT projects of this type are implemented successfully. Managing major projects of this type is inherently difficult. Both public and private entities often experience major delays, cost overruns, and operational problems when implementing these efforts. Nevertheless, this is now the second time that the TFC Project has terminated its primary vendor contract, and preventing subsequent delays to this specific project was one of the key objectives behind recent efforts to focus and consolidate the state’s technology oversight. Accordingly, we advise the Legislature to scrutinize the performance of both the Technology Agency and DGS concerning this major project.
Understanding the shortcomings of these oversight entities and engaging with the administration about its technology oversight functions are important activities if costly failures of this kind are to be avoided in the future. We recommend that the Legislature initiate a review of the performance of both the Technology Agency and DGS in overseeing and managing the risk of the TFC Project. Specifically, we recommend that the Legislature require the administration to undertake a self-assessment of the deficiencies of its own oversight functions. The lessons derived from the self-assessment have broad implications for the success of the state’s other large and complex IT projects—including projects currently under development and future projects.
Recommend Self-Assessments by Oversight Entities. We recommend that the Legislature require the Technology Agency and DGS to each present later this year a detailed self-assessment—in a straightforward style with conversational language—of their activities with regard to the TFC Project. Each Agency’s assessment should (1) describe its current oversight policies and practices, (2) evaluate how these policies were applied to the TFC Project, and (3) identify impediments—statutory or otherwise—faced by it that limit its ability to exercise effective oversight in this and other cases. These impediments might include limits to the agency’s statutory authority that constrain its ability to mitigate project risks or influence project direction as needed. In their assessments, the Technology Agency and DGS should each present recommendations that address their own institutional policy and process deficiencies. The assessments should suggest revisions to current oversight practices and, if necessary, statutory authority to better support other departments’ IT projects.
Recommend Assessment of Tools and Authority Technology Agency Has to Exercise Its Oversight Role. In reviewing the oversight entities’ self-assessment, we advise the Legislature to consider—in light of the recent issues concerning the TFC Project and other projects—whether the Technology Agency has the authority, staffing, budget and expertise to be reasonably held accountable for project shortcomings of this type. Under current law, the Technology Agency has the authority to approve, suspend, terminate, and reinstate IT projects. This broad authority, however, may not allow the Technology Agency to make enforceable recommendations as project issues arise on a day-to-day basis. Instead, the Agency seems able to exercise its oversight authority only at intermittent junctures (when approving FSRs, SPRs, and collaborating on procurement plans), or under extreme circumstances (when suspending or terminating a project is called for). In addition, the intermittent oversight opportunities available to Technology Agency appear to be concentrated at the front-end of IT projects, during the planning phase. Once a vendor contract is signed and development of the project begins, however, relatively few opportunities exist for Technology Agency to exercise enforceable authority regarding project direction. In our view, the Legislature should consider (with the benefit of the self-assessments described above) whether the Technology Agency has effective tools within its existing authority to prevent project outcomes such as what happened in the case of the TFC Project.
The Future of the State’s Payroll System
Recommend Project Pause. There are serious risks associated with moving the TFC Project forward prematurely. Given the potential to repeat previous project shortcomings, we recommend that the Legislature not approve a full-scale project plan—one that includes re-procurement of a new system integrator—until after a thorough evaluation of the work completed by the primary vendor and a cost-benefit analysis of project alternatives is conducted, as discussed further below. We expect that such an evaluation could take a year or more to produce.
Recommend Independent Assessment. Given that the SCO has experienced numerous challenges while managing the TFC project, we recommend that the Legislature require the Technology Agency to contract with a firm or entity independent of the state to conduct an assessment of the following two issues that will help inform the path forward for the state’s legacy payroll systems.
Quality and Value of SAP Software and Work Completed by SAP.We recommend the evaluator determine if the SAP software solution and the work completed by SAP can be reused should the state continue the TFC Project in something like its prior form.
Potential Simplification of State Payroll Practices. We recommend that the evaluator complete an assessment evaluating the state’s current payroll practices to determine if these practices can by simplified. The applicability of commercial payroll practices to the state payroll procedures should also be included in this assessment. Simplifying the state’s payroll practices may reduce needed software customizations, thereby facilitating integration and enhancing the prospects of successfully implementing a statewide payroll system.
We recommend that the report of the independent evaluator be required to be provided to the Legislature and the Governor unedited by any state official.
Recommend Analysis of Alternatives and Evaluation of Payroll Process. Due to recent events, it is unclear to our office that integrating the state’s payroll systems, in their current structure, is feasible. Amid this uncertainty, we recommend that the Legislature require SCO and the administration to conduct a cost–benefit and feasibility analysis (informed by results from the independent assessment described above) that compares the costs and feasibility of several alternatives, including:
Maintaining the state’s legacy payroll systems.
Updating the existing legacy systems using a decentralized model. (This option could require changing state law to provide authority to departments to perform many payroll functions currently assigned to SCO in statute.)
Hiring a vendor to finish the configuration work developed by SAP using the existing SAP software solution that the state owns.
Hiring a vendor to restart configuration work, but using the existing underlying SAP software solution.
Hiring a vendor to restart configuration with a new underlying software solution.
The cost-benefit analysis should also document lessons learned and strategies to prevent a repeat of previous problems as well as respond to the payroll practices recommended in the independent assessment.