INFRASTRUCTURE ONTARIO—ALTERNATIVE FINANCING AND PROCUREMENT (PDF)

Legislative Assembly of Ontario

Assemblée législative de l'Ontario

STANDING COMMITTEE ON
PUBLIC ACCOUNTS

INFRASTRUCTURE ONTARIO-ALTERNATIVE FINANCING AND PROCUREMENT
(Section 3.05, 2014 Annual Report of the Auditor General of Ontario)

1^st Session, 41^st Parliament
64 Elizabeth II

ISBN 978-1-4606-5864-2 (Print)
ISBN 978-1-4606-5866-6 [English] (PDF)
ISBN 978-1-4606-5868-0 [French] (PDF)
ISBN 978-1-4606-5865-9 [English] (HTML)
ISBN 978-1-4606-5867-3 [French] (HTML)

Legislative Assembly of Ontario

Assemblée législative de l'Ontario

The Honourable Dave Levac, MPP

Speaker of the Legislative Assembly

Sir,

Your Standing Committee on Public Accounts has the honour to present its Report and commends it to the House.

Ernie Hardeman, MPP
Chair of the Committee

Queen's Park
May 2015

STANDING COMMITTEE ON PUBLIC ACCOUNTS

Comité permanent des comptes publics

Toronto, Ontario M7A 1A2

STANDING COMMITTEE ON PUBLIC ACCOUNTS

MEMBERSHIP LIST

1^st Session, 41^st Parliament

Ernie Hardeman

Chair

Lisa Macleod

Vice-Chair

Han Dong	John Fraser
Percy Hatfield	Harinder Malhi
Julia Munro	Arthur Potts
Lou Rinaldi

CATHERINE FIFE regularly served as a substitute member of the Committee.

William Short

Clerk of the Committee

Ian Morris
Research Officer

Contents

Preamble 1

Acknowledgments 1

Overview 1

Auditor's Objective 1

Background 1

Issues Raised in the Audit and Before the Committee 2

Value-for-money (VFM) Assessment 2

Procurement of AFP Contractor 7

Evaluation of Bidders for AFP Projects 8

Monitoring of AFP Projects 9

Consolidated List of Committee Recommendations 11

Preamble

On March 25, 2015, the Standing Committee on Public Accounts held public hearings on the Auditor General's 2014 audit of Infrastructure Ontario-Alternative Financing and Procurement, Section 3.05 of the Auditor's 2014 Annual Report. Senior officials from the Ministry of Economic Development, Employment and Infrastructure (the Ministry) and Infrastructure Ontario participated in the hearings. (For a transcript of the Committee proceedings, please see Committee Hansard, March 25, 2015.)

The Committee endorses the Auditor's findings and recommendations and presents its own findings, views, and recommendations in this report. The Committee requests that Infrastructure Ontario provide the Committee Clerk with written responses to the recommendations within 120 calendar days of the tabling of this report with the Speaker of the Legislative Assembly, unless otherwise specified in a recommendation.

Acknowledgments

The Standing Committee on Public Accounts extends its appreciation to officials from the Ministry and Infrastructure Ontario for their attendance at the hearings. The Committee also acknowledges the assistance provided during the hearings and report writing deliberations by the Office of the Auditor General of Ontario, the Clerk of the Committee, and staff in the Legislative Research Service.

Overview

Auditor's Objective

The Auditor's objective in conducting this value-for-money audit was to assess whether Infrastructure Ontario had effective systems and processes in place to ensure that

· the decision to use the alternative financing and procurement (AFP) model is suitably supported by a competent analysis of alternatives;

· all significant risks and issues are considered and appropriately addressed in the final agreement; and

· public expenditures are incurred with due regard for economy.

Background

AFP is the name given to forms of public-private partnerships (P3s) frequently used in Ontario. Under the AFP model, project sponsors (provincial ministries, agencies, or broader public sector entities such as hospitals) establish the scope and purpose of a project, while design and construction work is financed and carried out by the private sector. The private sector company is paid by the province only after the project is substantially complete. In some cases, the private sector will also be responsible for the maintenance and/or operation of the project.

The Ontario Infrastructure and Lands Corporation-commonly referred to as Infrastructure Ontario-is a Crown corporation established in 2005, initially to deliver large scale infrastructure projects using the AFP model. In addition to AFP project delivery, Infrastructure Ontario now has three other main lines of business: Real Estate Services, Infrastructure Lending, and Commercial Projects. As of March 2014, approximately 160 of the 493 full-time employees at Infrastructure Ontario supported the delivery of AFP projects. Infrastructure Ontario's primary sources of revenue from the province derive from project delivery fees, management fees, and recoverable costs.[1]

In June 2011 the Ontario government introduced, through the Ministry, a 10-year strategic framework titled Building Together: Jobs and Prosperity for Ontarians, to guide investments in infrastructure in Ontario. Among other things, the framework proposed that Infrastructure Ontario have a greater role in procuring infrastructure, including engaging in traditional public-sector forms of procurement as well as AFPs when appropriate.

Generally, ministries evaluate and prioritize their infrastructure needs and their proposals are submitted to the Treasury Board for approval. After being approved for AFP delivery, the project is assigned to Infrastructure Ontario. For large-scale projects, Infrastructure Ontario assesses AFP feasibility and recommends AFP based on an initial assessment that considers, among other things, the size and complexity of the project and the potential to transfer risk to the private sector contractor. A key principle that guides Infrastructure Ontario in delivering projects using the AFP approach is that value-for-money (VFM) must be demonstrable under the VFM assessment model they use. The Treasury Board's funding approvals for AFP projects are contingent on continued demonstration of positive VFM.

As of May 2014, Infrastructure Ontario was involved to various degrees in the delivery of 75 AFP projects, ranging from hospitals, courthouses, highways, and transit projects. Infrastructure Ontario has conducted over 200 VFM assessments for 74 infrastructure projects that, based on an initial assessment, it had deemed suitable for AFP delivery. None of these VFM assessments has shown a negative VFM result from using the AFP assessment model.

Issues Raised in the Audit and Before the Committee

Value-for-money (VFM) Assessment

The Auditor recommended that Infrastructure Ontario should, in conjunction with the Ministry,

· gather data on actual cost experience from recent public-sector infrastructure procurements and AFPs; and

· revise its VFM assessment methodology to ensure that the valuation of risks assumed to be retained under both the AFP and public-sector delivery models are well justified.

The Auditor found that the total of the tangible components of project cost (such as base construction costs, financing costs, and ancillary costs) for the 74 AFP projects that Infrastructure Ontario had managed or was managing at the time of the audit was $8 billion higher ($6.5 billion related to higher financing costs) under the AFP delivery model than in the public sector comparator (PSC), assuming they were completed on budget and on time. However, the estimated value of the risks retained by the public sector calculated by Infrastructure Ontario when the public sector delivers a project ($14.6 billion higher in the PSC) is shown to offset the higher costs of the AFP delivery model.

The Auditor indicated that she had concerns regarding how the $14.6 billion was calculated in that there was no empirical evidence to support that risks under a public sector project are estimated to be about five times higher than if the project was done through AFP delivery. The Auditor noted that while there are examples of recent projects delivered by the public sector that have experienced cost overruns, there is no empirical data supporting the key assumptions used by Infrastructure Ontario to assign costs to specific risks. Instead, the agency relies on the professional judgment and experience of external advisers to make these cost assignments, making them difficult to verify. In this regard, often the delivery of projects by the public sector was cast in a negative light, resulting in significant differences in the assumptions used to value risks between the public sector delivering projects and the AFP approach. This is important to understand because costing of risks tips the assessment of whether AFPs or public sector project delivery will result in more value for money assessments in favour of using AFPs.

Infrastructure Ontario explained that it is continuing to track cost and performance data on all of its projects as a regular course of business and will be expanding its annual track record report in 2015 to include performance measures on non-AFP projects delivered by Infrastructure Ontario.

Infrastructure Ontario also indicated that it continues to gather data on traditionally-delivered projects from other jurisdictions, including the UK and Australia. Infrastructure Ontario points out that this review supports the higher levels of risk associated with traditional projects in the Infrastructure Ontario VFM methodology. Infrastructure Ontario also noted that Deloitte recently published a review of major project results across Canada delivered by both public-private partnerships (P3s) and traditional methods; Infrastructure Ontario suggested that this review also supports the levels of retained risk assumed in its VFM approach. However, Deloitte acknowledged in their review that currently there is no systematic performance data tracking of traditionally delivered projects in Canada. Based on publicly available information, 16 of 20 transit projects in Canada informally reviewed by Deloitte appeared to be behind schedule and over budget, but Deloitte could not verify these findings.[2]

Infrastructure Ontario agreed with the Auditor's report which found that two of the risks that Infrastructure Ontario included in its VFM assessments needed to be removed. Their combined cost over 74 AFP projects was almost $6 billion (about a third of the overall total of risk costs for public-sector project delivery), and if they had not been included in the VFM assessments, public-sector delivery for 18 of these projects would have been assessed as $350 million cheaper than delivery under AFP (taking into account both estimated tangible costs and the remaining estimated risk costs).

Infrastructure Ontario stated that it is undertaking extensive work on revising its VFM methodology. It is engaging "independent professional cost consulting firms" to update the assessment of risks assumed under both traditional and AFP delivery. Infrastructure Ontario also indicated that it is consulting with key ministries that have experience under various delivery approaches.

Infrastructure Ontario committed to working closely over the coming year with the Ministry to build "a more comprehensive, externally reviewed dataset of public-sector infrastructure procurements by both traditional and AFP methods." Infrastructure Ontario indicated that an updated VFM methodology guide will be published in spring 2015.

The Committee supports the recent actions being taken by Infrastructure Ontario to update its VFM assessment methodology and work on building a comprehensive dataset of both traditional and AFP delivered projects. The Committee believes that it is important for Infrastructure Ontario to continue to adjust its model to ensure that its methodology is based on empirical data that supports the valuation of the risks.

Committee Recommendations

The Standing Committee on Public Accounts recommends that:

1. Infrastructure Ontario shall report back to the Standing Committee on Public Accounts a summary of its analysis into the collection of data on actual cost experience from recent public sector procurements and AFPs in Ontario.

2. Infrastructure Ontario shall report back to the Standing Committee on Public Accounts on details of its revised VFM assessment methodology, including what was changed, when, and why. This response must also include the extent to which the valuation of risks assumed to be retained under the AFP and public sector delivery models are supported by actual cost experience from recent public sector procurements and AFPs in Ontario.

The Auditor recommended that to ensure that VFM assessments in procuring large-scale infrastructure projects are valid and objective, Infrastructure Ontario should confirm

· that all risks assumed to be transferred to the AFP contractor are supported by relevant provisions of the project agreement;

· that the costs assigned to retained risks in the public-sector comparator are not accounted for elsewhere in the assessments; and

· that the threshold for what is considered a large-scale project is useful in screening projects that should be procured using the AFP approach versus the public-sector delivering the project.

In response to the Auditor's recommendations, Infrastructure Ontario noted that it reviews how major risks assumed in the VFM assessment are reflected in the project agreement (PA) to ensure that all risks are effectively transferred through its procurement and PA documents. Infrastructure Ontario also committed to using the same mapping process as the Auditor General when conducting periodic audits of the PAs.

In terms of the two specific risks that were flagged by the Auditor-"asset residual" risk and "planning, process and allocation practices" risk-Infrastructure Ontario indicated that these are being addressed in the updated VFM methodology. The lifecycle cost under the public sector comparator has been reduced to recognize the historically observed under-spending by the public sector. This reduced cost is offset by the increased risk associated with the quality of the asset at the end of its lifecycle (i.e., asset residual risk). Also, Infrastructure Ontario explained to the Committee that in its risk matrix there are three different risks about planning that will collapse into one with the release of its updated VFM methodology. Infrastructure Ontario suggested that this consolidation is expected to eliminate the double-counting identified by the Auditor.

The Ministry indicated that it is undertaking, in cooperation with Infrastructure Ontario, a review to assess a potential increase in the screening threshold (currently at $50 million) for AFP projects, with assessment of projects below $100 million on a case-by-case basis.

Committee Recommendation

The Standing Committee on Public Accounts recommends that:

3. The Ministry of Economic Development, Employment and Infrastructure shall report back to the Standing Committee on Public Accounts on its plan for the screening threshold for AFP projects. This response must

· state the amount of the new screening threshold, if applicable, and provide a rationale for the change; and

· provide an anticipated implementation date for any proposed changes to the screening threshold.

The Auditor recommended that Infrastructure Ontario should ensure that all proposed changes to its VFM assessment methodology, including its plan to increase the base cost on the public-sector comparator side by up to 13.3% to reflect value-added innovations that the private sector may be bringing to projects, can be and are fully supported and can sustain scrutiny.

Infrastructure Ontario noted that as part of the update to its VFM methodology, it has conducted research into the levels of innovation in its projects. Infrastructure Ontario asserted that the introduction of an innovation factor recognizes that the use of performance-based specifications in AFP projects allows contractors to deliver projects that exceed technical compliance requirements while minimizing costs compared to traditional delivery. Infrastructure Ontario noted that to ensure this innovation factor is well supported by both external and internal sources, it has taken a number of steps, including the following:

· Reviewed approaches in other jurisdictions: The proposed introduction of an innovation factor is consistent with the VFM approach used in a number of other jurisdictions.

· Reviewed third-party research: MMM Group conducted its own review and concluded that contractors estimated innovation on transit and highway projects delivered through performance-based specifications like the AFP model would result in capital cost savings.

· Reviewed internal data: Infrastructure Ontario's own review of bid data-both comparing between the original project budget and the average bid, and analyzing the dispersion between bids received-also validates the introduction of an innovation factor.

However, Infrastructure Ontario also noted it is currently participating in a third party study, developed and conducted by academics, to review and assess the extent of innovation on AFP projects. The results of this study will be used to further validate and refine the application of an innovation factor in Infrastructure Ontario's VFM methodology going forward.

The Auditor recommended that the Ministry should also engage Infrastructure Ontario in traditional forms of procurement that utilize the agency's experience in delivering AFPs, for the most part, on time and on budget, in order to achieve cost benefits and to be consistent with the government's June 2011 strategic framework to guide investments in infrastructure in the province.

In its response, Infrastructure Ontario noted that it continues to deliver a wide range of infrastructure projects through traditional delivery methods and the Ministry retains the discretion to have Infrastructure Ontario deliver additional infrastructure projects using a variety of procurement methods. The Ministry indicated that it will continue to consider the opportunity to have Infrastructure Ontario deliver additional infrastructure projects using a variety of procurement methods.

Committee Recommendation

The Standing Committee on Public Accounts recommends that:

4. Infrastructure Ontario shall report back to the Standing Committee on Public Accounts with a summary of its traditionally delivered projects. This summary must

· include statistics about cost overruns and project delays for these projects; and

· highlight any trends in these statistics over the period from when Infrastructure Ontario first began overseeing traditionally delivered projects to the most recent projects.

Procurement of AFP Contractor

The Auditor recommended that in order to have a good estimate of project costs before seeking Treasury Board approval, as well as to better evaluate the reasonableness of future bids, Infrastructure Ontario should

· identify the reasons for the significant differences between actual contract values and its estimates of project cost, especially for projects that have long-term financing, maintenance and life-cycle costs; and

· accordingly review and update its processes for arriving at these estimates.

Infrastructure Ontario indicated that it has completed a review of its historical bid data to better understand the underlying drivers of differences between initial project budgets approved by the government before the request-for-proposal (RFP) is issued and actual contract values. Infrastructure Ontario explained that the largest driver of the difference is that the designs, when initial project budgets are developed, are high level and conceptual (i.e., the designs are typically 20-30% complete). Infrastructure Ontario highlighted that under these circumstances, industry best practice is for cost estimates to be within 20-25% of final project costs.

Infrastructure Ontario noted that it is in the process of retaining a third-party cost consultant to review its budget process and estimating methodology and assess how this compares to industry best practices. Infrastructure Ontario has committed to "implement any recommendations coming out of the review." Moreover, Infrastructure Ontario has committed to working to ensure budgeting for future infrastructure projects incorporates the best available information.

Committee Recommendation

The Standing Committee on Public Accounts recommends that:

5. Infrastructure Ontario shall report back to the Standing Committee of Public Accounts on the results of its third-party review of its budget process and estimating methodology.

Evaluation of Bidders for AFP Projects

The Auditor recommended that Infrastructure Ontario should review and update its system of scoring bidders' submissions to ensure that due consideration is afforded to both the technical merits of the submissions and to price.

Infrastructure Ontario noted that it has conducted a review of its evaluation methodology. Infrastructure Ontario further highlighted that based on a review of all design-build-finance-maintain (DBFM) and design-build-finance (DBF) projects that have reached financial close to date, approximately two-thirds of the lowest bids were also the highest ranked in design quality. Infrastructure Ontario suggested that the current scoring methodology appears to result in the right balance by encouraging bid submissions towards a high-quality design that meets or exceeds all output specifications at the lowest possible cost. Moreover, Infrastructure Ontario added that "anything that doesn't meet the output specification and anything that doesn't achieve a high score from a design and technical perspective can't win."

Committee Recommendation

The Standing Committee on Public Accounts recommends that:

6. Infrastructure Ontario shall report back to the Standing Committee on Public Accounts on its competitive bidding process for AFP projects. This response must

· explain how the remaining one-third of projects were chosen and how the system of scoring bidders' submissions would be modified to ensure appropriate weighting is given to both technical merits of the submission and price;

· include steps taken to ensure the bidding process is a competitive one; and

· include a summary of the average number of qualified bids received for AFP projects.

The Auditor recommended that Infrastructure Ontario should ensure that participants involved in evaluating the submissions sign the required conflict of interest declaration that discloses any relationships with entities identified in the submissions.

In its response to the Auditor's recommendation, Infrastructure Ontario indicated that it will continue to ensure that all evaluation participants sign the required conflict of interest declarations and explained that the difficulty in being able to locate all such declarations reflected a gap in records management and not process.

Infrastructure Ontario further noted that when this issue was brought to its attention through the audit process, a memorandum was circulated to the Procurement team in May 2014 reminding them of the obligation to retain copies of the signed conflict of interest declarations. Since then, Infrastructure Ontario's Procurement team has further streamlined the required documentation by consolidating the conflict of interest and confidentiality agreements into one form. In addition, Infrastructure Ontario committed to conducting periodic audits of the conflict of interest records to ensure compliance with respect to retention of records.

The Auditor recommended that, consistent with the March 2012 letter from the Minister of Economic Development, Employment and Infrastructure, Infrastructure Ontario should develop a formal process for managing the intellectual property rights acquired in exchange for the bid fees paid to unsuccessful bidders to ensure that the province receives any benefits from these rights in planning new projects.

Infrastructure Ontario indicated that it has centralized all electronic design submissions in its document management system. Infrastructure Ontario also noted that access to the design submissions is available to its project teams for review to inform the planning of future projects, thereby ensuring that intellectual property rights acquired in the bidding process are being properly leveraged.

Monitoring of AFP Projects

The Auditor recommended that Infrastructure Ontario should review the amount of the payments that it holds back at substantial completion of the projects it delivers to help ensure that minor deficiencies are corrected on a timely basis.

In its response, Infrastructure Ontario indicated that an assessment of the substantial completion payments has been completed and determined that an opportunity exists to increase the amount of the payment held back for some projects-in particular social infrastructure projects (e.g., hospitals, courts, jails, colleges). Infrastructure Ontario indicated that such an increase in the hold back payment would decrease the cost of long-term financing on future projects.

Further, Infrastructure Ontario noted that it has introduced a monthly report of substantially complete projects in order to actively monitor the duration between substantial and final completion. Infrastructure Ontario has also committed to conducting further review and due diligence in spring 2015 to ensure timely completion of minor deficiencies.

The Auditor recommended that in order to properly monitor the construction phase of projects, Infrastructure Ontario should ensure that information on individual projects is stored in a centralized database using a consistent structure, and that its construction status reports are accurate and complete.

Infrastructure Ontario indicated that it has made changes to ensure that its project management information system, eBuilder, is being used more consistently to track all project information in a centralized database using a consistent structure. Infrastructure Ontario also noted that modifications to the system to make it more user-friendly and comprehensive were implemented in fall 2014, and training on the use of eBuilder and reporting templates has been provided to all staff.

With respect to construction status reports, Infrastructure Ontario noted that it has recently revised and expanded them. Infrastructure Ontario also explained that additional training on these revised reports will be provided to staff. In order to further ensure completeness and accuracy, Infrastructure Ontario has committed to conducting periodic audits of its construction status reports.

Consolidated List of Committee Recommendations