8.2 Evaluation Criteria and Evaluation Process Regulations

Evaluation criteria

Broadly speaking, there are two main types of processes in terms of evaluation criteria.

• Processes based only on price (also referred to as least cost selection) in which the technical factors are evaluated on a pass/fail basis. This is sometimes called an auction; however it is more appropriate to use the term “auction” for asset monetization PPPs only, that is, PPPs that are concessions out of existing revenue-making infrastructure, or greenfield or yellowfield user-pays projects that are likely to generate an excess of revenue over costs. In this circumstance, the price criteria is based on who offers the highest value share to the procuring authority.
• Processes based on price in combination with qualitative factors, basically related to the quality of the technical offer (approach to construction and project design, and approach to O&M). This may be referred to as Quality and Cost Based Selection (QCBS) or more frequently as Most Economically Advantageous Tender (MEAT).

A price only evaluation is sometimes undertaken by evaluating the quantity/product offered for a certain price of the total budget allocated by the contract, for example, the number of homes supplied for a water distribution project or the number of kilometres of road offered for a certain price. This quantitative approach is not advisable for most PPPs, as the project solution defined by the government should reflect the total need assessed in project selection and/or the scope should be clearly defined as the required solution (with the scope adapted if necessary for affordability constraints).

In all cases, objectivity and transparency should be an essential driver when structuring and defining the evaluation methodology, even for pass/fail considerations, as the criteria are the basis on which the awardee will be selected and called for contract signature.

The criteria will usually reflect requirements within the procurement framework (for example, inclusion [or not] of qualitative criteria, maximum or minimum weightings for price versus qualitative criteria, and so on). However, many sub-criteria (especially those of a qualitative nature) will need to be adapted or defined ad hoc for the specific project.

Price only (or only objective or numerical) evaluation

When a selection is based only on price (for example, the lowest size of availability payment required by the bidder, the lowest size of a grant financing in a user-pays PPP that is not commercially feasible on a stand-alone basis, or the highest concession fee offered by the bidder), the technical criteria will be pass/fail. This means that only bids that meet the minimum bar of the technical criteria will be assessed in terms of price, but among the technical qualifying bids, price will be the only factor considered.

A price-only evaluation is dangerous for the authority as it may create an auction situation with overly aggressive bids that may compromise either quality or reliability. When using a price-only approach in greenfield projects, the procuring authority should be sure about what is wanted. A price-only evaluation is typically suited to simpler projects or infrastructure with no particular complexities or technical challenges.

A way to limit the potential dangers of “price only” criteria is not only to set out an overall technical pass/fail, but to also include specific and relative high qualitative scoring for key quality/technical sub-criteria. For example, a weighted average score of 6 out of 10 may be required for a technical pass, but a score of 7 or 8 or more may be required for key elements of the technical offer.

However, it should be noted that sometimes price may be in the form of more than one factor in addition to the basic price of the contract, that is, the payment requested by the bidder (or the price to be paid by the bidder in a high return user-pays PPPs). The RFP may request that bidders also present and bid for other quantitative aspects (for example, the percentage of certain revenues to be shared with the government above a revenue baseline, and so on).

In some cases, the additional quantitative factors may relate to something other than strictly price. They may also be referred to as “numerical criteria” or “criteria scored under numerical formulas”, but usually referring to cost or measurable efficiency rather than quality. For example, bidders could be asked to propose the construction term required. However, an evaluation based on a bidders’ ability to shorten the construction period required for the project is undesirable, as discussed earlier. This is especially the case if the revenue regime or the payment mechanism already creates an incentive to accelerate construction — and this of itself may result in a more competitive price.

In each of these cases, with a technical pass/fail evaluation and multiple quantitative factors, each numerical criteria should have its specific weighting clearly set out in the RFP as well as the scoring formula. Potential redundancies in some of these evaluation factors should be carefully considered (for example, scoring on the basis of the lower availability payment, and scoring also on the basis of shorter construction terms is redundant in some projects, as the former evaluation factor also naturally incentivizes the bidder to consider a shorter construction term).

In government-pays PPPs, there are various forms in which the procuring authority can ask bidders to submit the price. The most common alternatives are to quote a single price (for example, the shadow toll to be applied in the first year, the size of the availability payment to be made during the first year, or the maximum toll in a road project) or to require bidders to present (or have the authority calculate) an NPV of the revenues or payments.

Generally, for government-pays projects where all bidders must base their payments on the same payment profile and indexation over the term of the contract, it may be more appropriate to submit one single price for the payment requested by the bidder, rather than relying on NPV calculations. If bidders can propose different payment profiles or different indexation factors, NPV calculations are necessary for comparison purposes. The particular case of government-pays PPPs where the government is co-financing has already been discussed in section 4.2.

An exception in some more sophisticated approaches (for example, the Chilean approach to evaluate real toll PPP roads) is to offer and evaluate on the basis of the NPV of revenues. This is not only a price-evaluation mechanism, but also a risk-sharing mechanism, as the concession term will be variable to meet the NPV offered[75].

Price and quality

The other most often used approach to evaluation (probably more common than price-only) is to evaluate based on both price and quality. Sometimes this approach is referred to as “criteria subject to qualitative assessment”, while the price-only approach is referred to as “criteria not subject to qualitative assessment” or scored under numerical formulas. Price and quality evaluation is the most typical approach in the EU where the procurement regime allows for some flexibility regarding the criteria under the “most economically advantageous tender” concept[76].

When technical and economic criteria coexist, the appropriate weighting of each category of criteria (technical/quality versus economic) should be expected to depend upon the type of project. For example, for projects regarded as very innovative, the proposed approach, means, and methods involved will be much more important than in a more conventional project.

Evaluation approaches with a significant weight on price or other objective/numerical factors are most common. In a number of countries/jurisdictions, it is common that the economic/objective or numerically assessable criteria have a weighting of at least 50 percent (for example, typically in Spain and some Latin American countries) to provide a significant weighting to pure objective criteria.

In some countries, when price or other quantitative criteria represent less than 50 percent, it is customary to constitute an expert committee or an expert evaluation panel with robust checks and balances in the evaluation process (for example, in Australia and New Zealand).Qualitative criteria should be objective to the maximum extent possible and clearly defined or explained. Transparency and clarity are essential and are a general legal requirement in jurisdictions such as the EU.

In general terms, in addition to informing the weighting split between “price” versus “technical/quality”, each criteria (within these two groups) should have a specific weighting reflecting the relative importance of the different objectives and informing the bidders of the government’s priorities. However, some projects may present a second layer of criteria (sub-criteria), and in some cases there are no weightings for this second layer. When a weighting is not provided, it is customary to at least provide a “list of factors” that will be considered when assessing and scoring the respective criteria or sub-criteria. Figure 5.11 represents an approach to defining a long list of criteria with the specific weightings. These are presented as grouped into “main criteria” (while some could describe it as a list of criteria and sub-criteria). There is no universal approach for the organization of the criteria and the information on weightings and factors to be considered. For example, some projects directly present a long list of criteria (see example BOX 5.26), while others create categories (see example in BOX 5.27)

There are some cases/countries where the process relies significantly on qualitative criteria, Australia being one principal case. There, the RFP typically provides the evaluation criteria but does not provide significant information on the scoring methodology or evaluation procedures. Relying significantly on qualitative assessment may be possible when there is significant confidence in the equality and fairness of the evaluation process, which will only be possible in countries/markets with a high recognition in the investor community.

FIGURE 5.11: Categories, Criteria, and Sub-Criteria

Note: QA= Quality Assurance; QC= Quality Control; QMS= Quality Management System.

Responsiveness and minimum scoring/minimum quality

The RFP should state clearly how proposals may be rejected or disqualified when not respecting the prescribed requirements that are clearly set out in the RFP documents, that is, proposals not regarded as responsive to quality/technical matters. Beyond that, it is also customary to provide a minimum score to achieve a minimum quality bar, that is, a score above what would represent a mere pass For example, requiring an average score in technical overall criteria of 7 out of 10 and/or specific minimum scores requested for some specific sub-criteria.

Other technical or qualitative criteria and sub-criteria issues

It is essential to develop a clear list of technical and/or quality criteria.

It is good practice to assign to each of the criteria a specific weighting in the overall scoring, or the maximum number of points that will be allocated, out of the total scoring considered for the technical criterion. For some projects, a weighting for the second layer of criteria or “sub-criteria” may be provided.

It is customary and good practice that for every criteria, the RFP describes a definition or explanation for transparency purposes — and even a description of the main factors that will be considered when assessing the respective sub-criteria. However, to split the scoring/weighting into too many factors may be dangerous since that may introduce excessive rigidity and place too much emphasis on individual sub-components.

Typical qualitative and/or technical criteria include[77]:

• Construction matters, which may include criteria such as quality and reliability of the project design offered (usually under a pre-design format), the reliability of the construction period estimated, the quality assurance methods proposed for construction oversight, and so on.
• Operational matters, which may include criteria such as quality and reliability of operating procedures and manuals, commitment of means, service or O&M quality management systems or plans, and so on.
• Maintenance matters, including criteria such as quality and reliability of the maintenance plans and programs, renewal/major maintenance programs, specific plan for hand-back, and so on.
• Evaluation criteria for environmental compliance and environmental sustainability (for example, landscape factors)
• Evaluation criteria on health and safety plans
• Other qualitative criteria such as criteria related to benefits for minority or disadvantaged populations (usually set as a minimum bar or condition, for example, the number of members of the minority community provided with employment by the project or the consortium), financing reliability, and so on.

The list of criteria and sub-criteria should not be so large as to create undue complexity in the evaluation and make it difficult for bidders to focus on the fundamental objectives of the government (see examples in boxes 5.25 and 5.26).

The financial package is sometimes subject to evaluation in terms of reliability, such as the commitment level shown by the equity investor, the level of confidence in the financing availability, and the degree of robustness of the project finance structure. In processes with staged evaluation, it can be difficult to manage the evaluation of the financial package without information on the reliability and robustness of the financial structure that would allow the appraiser of the offers to know in advance or infer the price offered. Strict instructions should be provided to bidders, emphasizing that the financial package documentation should not disclose the overall price offered.

Another controversial issue is the potential role of experience as an evaluation factor. In a number of jurisdictions, including experience as an evaluation factor is prohibited by law because using experience as a selection criteria creates a risk of perpetuating the status quo where the most experienced bidders frequently win the projects. If experience has already been used to qualify bidders (or even to short list), it should not be necessary to also use experience as a criteria in evaluating the bids.

As PPPs focus on performance and PPP specifications are mostly based on outputs rather than inputs, the technical requirements in the RFP should not be prescriptive. It should provide only a reference design in a pre-design form or a “functional design”, but the service requirements should be focused on the results/quality of the service through key performance indicators (KPIs), rather than the amount of inputs or activities.

Consequently, the technical proposal evaluation should not be based on inputs committed (for example, the number of workers or professionals to develop certain functions), but should check that the means proposed by the bidder respond to the minimum requirements established in the RFP. It should also evaluate (using reasonable judgment) the extent to which the means and methods proposed by the bidder will result in quality and reliability of the output.