Highways Agency

Safe roads, Reliable journeys, Informed travellers

Developing performance specifications

• Background
• Broad Objectives
• Summary Of Consultation Findings
• Analysis Of Responses
  • Issues For The Suppliers
  • Issues For The Agency
  • Issues Common To Both The Agency And Suppliers
• Results Of Discussions With Key Organisations
• Consultation Respondents

Alternatively download the complete report as a single file (400KB PDF)

Background

The Highways Agency (HA), in April 2003, issued a document entitled "Developing Performance Specifications: Consultation Document" to organisations and individuals currently and previously involved in the management of the motorways and trunk roads in England and Wales.

This document was the third of the three linked consultations in respect of developing new and/or improving the existing policies for procurement and forms of contracts.

The first consultation released in December 2002 aimed at making improvements to the DBFO contract and introducing the principles of early contractor involvement into the delivery process. The second consultation relates to introducing private finance into operation & maintenance contracts such as the managing agent contracts (PFMAC).

The Developing Performance Specifications document, the third in the series, put forward options for widening the use of performance specifications, not only in private finance contracts but across all forms of procurement and sought to obtain views on the proposals presented and any other ideas to improve the procurement process. This document provides detailed analysis of the responses received.

The consultation process also included follow-up interviews with a number of respondents and a series of internal HA workshops to consider the results and identify a way forward.

Broad Objectives

The HA is committed to improving its procurement and contract management processes. The intention is to develop a strategy to foster an environment of trust and accountability in which the service providers are motivated to innovate and provide an excellent services whilst optimising value for money.

Six streams of measurement have been identified that support the key objectives of the HA:

1. Safety
2. Journey Time Reliability
3. Environment
4. Customer Satisfaction
5. Network Best Value
6. Supplier Capability of Performance

The adoption of an output/outcome base performance specification will enable a service provider to be rewarded for achieving the Agency's stated objectives, rather than simply by reference to the amount of work done, thereby promoting better value and improved price certainty against service delivery.

Summary Of Consultation Findings

The Highways Agency, in April 2003, issued a document entitled "Developing Performance Specifications: Consultation Document" to put forward options for widening the use of performance specifications, not only in private finance contracts but across all forms of procurement. The document was circulated to a range of stakeholders both within and outside the HA.

The 54 responses that were received provided a wealth of information and opinions about the options being presented and practical advice about the implementation of performance specifications. Respondents were generally positive about the move towards outcome based measurement and performance specifications although a number of concerns and reservations were expressed on a range of related issues.

Key Messages

• There are significant and worthwhile benefits in working under Performance Specifications including flexibility and a boost to innovation.
• The link between 'Outcomes' and 'Outputs' is not always clear or established. The large number of external influences may mean that in some instances that link is weak. Providers want to be measured by outcomes they can control.
• There are real problems of measurement in relation to outcome specifications. Time is needed to research and develop the measurement process, the HA should proceed slowly in introducing outcome measures.
• There is scope for industry to take on more risk but industry feels that risks should be shared with the HA and that there would be a cost associated with taking on more risk which would need to be reflected in the reward offered. Some risks should remain with HA. Risks should be fully understood and quantified and providers feel that there should be a cap on the element of risk they are required to bear.
• There is support for technical standards/specifications being made more flexible with some possible risk sharing with suppliers. However, there is no support for the HA fulfilling its Technical Governance role by totally passing responsibility for such standards to suppliers.
• There is support for industry engagement in setting standards/specifications. Joint working to speed up the approval of 'innovations' (rather than 'departures') was seen as the way forward.
• Short term and some other contracts were not considered suitable for outcome specifications.
• Consistency across the HA network may be compromised if there is wide flexibility in specifications.

Issues for Suppliers

• Respondents tended to agree that performance specifications would encourage innovation, improve efficiency and increase value within contracts although there was concern that small and medium contractors and the supply chain would be at a disadvantage due to the high insurance costs associated with performance specifications.
• Respondents were split over which elements of performance should be measured on an outcome basis. Some contractors appeared to welcome the application of performance specifications for all elements of work, although they also pointed to a number of potential problems. A number of client bodies and suppliers questioned the appropriateness of using outcome specifications. The majority of respondents agreed that routine and cyclic maintenance would be appropriate areas for outcome measurement and that outcomes outside the providers' control should not be measured using performance specifications (examples mentioned included safety and congestion). A specific concern raised was that the means of determining the long term durability of pavement structures do not currently exist and that this will constrain the associated performance measurement.
• When discussing the ability of industry in general to assume the risks inherent in outcome based performance specifications all respondents agreed that risks should only be transferred if they are manageable by the party to whom they are being allocated. Suppliers raised a particular problem about their ability to obtain insurance and the impact that this would have on their ability to apply for and fund projects. Clients noted a concern that unnecessary risks might be taken by contractors either due to commercial pressures or through a lack of adequate understanding.
• In order to develop the issue of risk management consultees were asked whether there were any risks that should not be transferred. The majority of respondents believed that the risk of political and client objectives changing could not be transferred to providers due to their lack of influence over the management of such risks. Clients preferred not to transfer the highway geometry and structural integrity risks.
• Respondents were also asked what risks they perceive in working under performance specifications; 44% of respondents believed that there would be misunderstandings and incorrect interpretations and that tender stage assurances may not be met. Again possible problems obtaining funding were raised as a major risk, especially for small and medium organisations. When asked how these risks could be managed the respondents were split almost equally between 'implement risk analysis and management', 'work in partnership' and 'link performance measurement to risk transfer'.
• The final question specifically relating to suppliers asked for suggestions on the cultural changes that might be required to facilitate the implementation of performance specification based contracts. The majority of respondents agreed with the sentiments within the consultation document that; "An atmosphere of trust and unity will need to be established and maintained by development of a partnership culture among suppliers as well as between the suppliers and the Agency". Respondents also felt that a change of mindsets would be required both within the Highways Agency and the industry itself. Again it was noted that cultural changes would affect the smallest companies hardest.

Issues for the Agency

• The first issue for the HA was concerned with whether performance specifications should be extended to all Agency contracts. A high proportion (83%) of the organisations consulted felt that performance specifications should not be extended to all, if any HA contracts; although 17% of respondents stated that they should be extended to all contracts, this view coming predominantly from contractors. Specific areas of procurement where traditional prescriptive types of specification should be retained included short term contracts, sensitive locations and improvement schemes. It was also noted by a large proportion of respondents that existing specifications should be retained as guidance and as a benchmark for best practice.
• In response to the question of how the HA's technical governance role could be supported by suppliers, respondents stressed that this role should remain with the HA with support from suppliers and not transferred. Tools that could be applied to assist the support process included quality management systems, working in partnership and innovation forums for continuous improvement.
• Taking this issue forward, the implications for suppliers if they were allocated the responsibility for standards was discussed. Respondents did not appear to welcome the transfer of responsibility and were concerned about inconsistency, uncertainty, inefficiency and increased costs. However there was also a view that such a transfer of responsibilities could result in improved professionalism with the emergence of specialists.
• The final question in this section of the consultation document asked for suggestions in relation to the use of future looking indicators. The responses showed that there were differences of interpretation amongst the respondents as to what 'future looking indicators' meant, what the benefits would be and how they would be utilized. Generally, it was felt that if such indicators are to be used they must be robust. Some contractors and consultants were of the view that future looking indicators are of limited value, apparently due to the problems associated with measuring the future performance of assets.

Issues common to both the Agency and Suppliers

• Respondents were asked to give their opinions on how suppliers will guarantee continued quality and best value without a detailed specification to follow. Feedback included a majority response that the SHW and DMRB should be retained as baseline guides to requirements and to provide a consistent approach. In addition it was commented that demonstrable outputs and benchmarking of costs should be established and quality management systems employed to monitor the processes.
• Similarly, the issue of how the HA will assure itself that the services being provided are to acceptable standards and meeting its objectives provided a split of responses between specifying performance standards, implementing an audit process, measuring outcomes and using quality management systems. Respondents generally agreed that a transparent and robust audit system would provide the necessary assurance and that the extension of PRIDe could be of value.
• Mechanisms that should be included to ensure that specifications and standards are updated as required to reflect best practice and to deliver continuous improvement were discussed. Continuous Improvement Groups were seen as beneficial, as was the retention of a custodian role by the HA for the specifications and standards. The importance of setting up a mechanism to make best practice available to all and therefore encouraging continuous improvement through the achievement of defined outcomes was noted, as were discrete reviews through external panels. There was also a comment that in an outcome specification based contract, achieving continuous improvement may require the contract to be renegotiated.
• Within performance specifications it was agreed that there would be considerable benefits if suppliers were prepared to share information; the majority of consultants, contractors and suppliers stated that they would be prepared to share information although clients were unsure that this would be the case.
• It was noted that the implementation of incentivisation mechanisms would be appropriate, especially to retain smaller organisations within the process. Innovation forums appear to be the most acceptable method of sharing information and it was noted that trade organisations already share information successfully and that therefore this could be extended subject to the protection of intellectual property rights.
• The final question within the document considered the impact on the bidding process if performance specifications were introduced. All the respondent types agreed that the key impact would be a more costly tender process and a requirement for longer tender periods and negotiations. Other key issues were the increased quality of data that would be required by the tenderers and its timely provision at the initiation of the tender stage and the requirement to better develop and understand the asset inventory for the network.

Other Key Issues

• One point of note made by a number of respondents was that higher prices should be expected at the beginning of the implementation of performance specification since this is the beginning of a learning curve, however in the long term this should level out.
• Other key issues raised through the responses were concerns that the transfer of responsibility for standards and specifications to suppliers would result in more conservative designs, in order to reduce the risk to the respective organisations. It was also pointed out that reducing costs is not necessarily conducive to better performance and that performance specification may lead to higher standards coupled with higher costs.
• There is a specific requirement for clearly identifiable reward systems and incentives for suppliers who accept the risks associated with performance specifications, especially at the early stages. Furthermore, it was mentioned that the introduction of outcome measures should not be rushed; they need to be thoroughly investigated and clearly understood by all the parties involved before they are rolled out; more research is required and trials conducted.
• It was noted that the recent issue of the new European Standard has moved the road markings industry from a purely "recipe" specification into one that allows the engineer to specify which standard of performance he requires. So the tools are coming into place. Many other areas of highway construction and maintenance are either introducing new standards or already committed to them.
• It was commented that the performance specified must be attainable, that standard methods of measurement are essential and that KPI's must be transparent and meaningful.

Analysis Of Responses

In April 2003 the Highways Agency issued a consultation document entitled "Developing Performance Specifications: Consultation Document" to put forward options for widening the use of performance specifications, not only in private finance contracts but across all forms of procurement. The document was circulated to a range of stakeholders both within and outside the HA.

A total of 54 organisations and individuals responded to the consultation document. The range of organizations and individuals who submitted is shown in the pie chart below. As can be seen the respondents were divided into the following categories:

• Consultants
• Contractors
• Suppliers
• Client Bodies
• Others (eg organisations representing particular types of road user)

The responses that were received to each of the consultation paper questions were input into a database utilising Microsoft Excel software such that they could be analysed. Following an initial review of the responses the key themes were identified, this formed the basis for the detailed analysis shown on the following pages.

The results of the analysis of responses to each question are presented below in chart form and where appropriate some of the comments made by respondents are included (on a non-attributable basis) in order to illustrate and expand the views expressed. It is important to emphasise that the sample of organizations surveyed, whilst intended to cover the range of views and interests likely to be relevant, does not constitute a statistically representative sample of stakeholders. Accordingly, the significant element of the results is the identification of the main views expressed and issues raised rather than the precise proportions of alternative viewpoints. It should also be noted that some respondents chose not to answer all of the questions posed whilst others provided more than one comment in response to each question, accordingly the aggregate percentage of responses may not add up to 100%.

Issues for the Suppliers

Question One - What are the benefits in working under a performance specification?

This question was answered by 42 of the respondents to the consultation who suggested a range of benefits that they believed would be achieved in working under a performance specification. As shown below, 61% of the full sample of respondents that participated in the response to the survey stated that the major benefit of performance specifications would be to encourage of innovation; 44% felt that efficiency would be improved and a third of organizations mentioned that increased value would be achieved.

There were only small differences of emphasis between the different types of organizations. Contractors tended to stress increased value and improving efficiency whilst consultants emphasised improved efficiency and flexibility. Client bodies saw encouraging innovation as a key benefit.

Question Two (a) - What elements of performance should be measured on an outcome basis?

A total of 34 respondents to the consultation document answered this specific question, many of whom identified more than one element of performance that should be measured on an outcome basis. As can be seen below, 26% of the full sample agreed that both routine and cyclic maintenance should be measured on an outcome basis, in fact these were the most common responses by the contractors, clients and others.

A further 17% believed that all elements should be measured on an outcome basis, however this was in many cases qualified elsewhere in the response by reference to the need for robust measures. There were few differences between the responses from the different types of organization, although none of the small sample of client organizations appeared to support the view that all elements of performance should be measured on an outcome basis.

Question Two (b) - Are there any which should not be so measured?

Following on from the initial question the consultation document went on to ask if there were any elements of performance that should not be measured on an outcome basis. A range of responses was given by the 34 organisations which provided a response on this issue, including safety, technical standards, customer service and elements outside the suppliers' control. Some responses indicated that no elements of performance should be excluded.

As can be seen in the chart below, 37% of the full sample stated that elements that were outside the control of the suppliers should not be measured on an outcome basis, citing as reasons the additional risk for the supplier, the difficulty of measurement and the need for continuous improvement. More specifically, 17% believed that safety should not be included within a performance specification due to the influence of factors outside the contractor's control such as driver behaviour.

It was stated by one contractor that "the specification should be focused more towards outcome in line with the greater degree to which the requirement is definable, quantifiable and understood".

Suppliers and the client bodies tended to support the view that technical parameters (e.g. the performance of pavement layers) should not be measured on an outcome basis and this was further investigated in questions 7 and 8 of the consultation document. A number of contractors and consultants stated that no elements of performance should be excluded as all are appropriate for performance specifications, however, in many cases, this view was qualified elsewhere in the organisations' response

About 13% of contractors stated that customer satisfaction should not be measured on an outcome basis. This was further developed within the contractors' responses by stating that customer satisfaction is largely outside the contractor's control and also difficult to measure on an objective basis and to relate to current performance. Safety and the physical performance of assets were also the most commonly mentioned specific elements by client organizations.

Question Three (a) - What are your views on the ability of industry to assume the risks inherent in outcome-based performance specifications?

Again this question was divided into two elements during the analysis. The first part of the question asked for respondents' views on the ability of industry to assume the risks associated with performance specifications and 41 respondents provided specific answers to this question.

As can be seen in the bar chart above, 57% of the full sample expressed the view that only those risks that can be managed by the contractors and/or suppliers should be transferred to them. This view was held by all respondent types (87% of contractors, 85% of consultants, 40% of suppliers and 31% of clients). There was also concern that risks would be passed down the supply chain; as one respondent put it: "The smallest player in the game carries the entire risk; and yet the risk he carries is unknown to the Agency which only deals with the main contractor"

Clients were particularly concerned that unnecessary risks would be taken by contractors either for commercial reasons or through a lack of adequate understanding.

It was noted by one consultant that "There will be little enthusiasm on the part of contractors to assume risks where they cannot be quantified at time of tender. As an example, risks associated with the improvement of safety performance would be problematic because the safety performance of the road is partly dependent on factors outside their control including driver behaviour and traffic volumes."

Question Three (b) - Are there any risks that should not be transferred?

Only 31 of the respondents answered this question with the majority of clients and suppliers feeling that they were not able to identify specific risks. Those organizations that did respond generally agreed that risks associated with political and client objectives should not be transferred to providers due to the lack of influence that they have over changes in policy and objectives.

As can be seen in the bar chart below, providing the breakdown of responses by contractors and consultants; 54% of the respondents believed that political and client objective risks should not be transferred with other key risks including safety identified by 21% and traffic forecasts by 14%.

In contrast to the responses by the consultants, contractors and suppliers; relatively more client bodies tended to feel that technical issues such as structural integrity and highway geometry were not appropriate for transfer, this may be a reflection of the particular specialisms of the client representatives who replied.

One Contractor stated that "The Highways Agency is best placed to deal with risks that will have a major impact if they occur. If the industry is asked to take on these risks in a competitive situation, the HA faces the additional risk that the supplier will not be able to fulfill its responsibilities. The ability to insure such risks is limited, and given the current state of the market is not likely to be cost-effective for Highways Agency"

Whilst another said that "The risk transfer will be affected by the extent and quality of the condition and inventory data available and his ability to control the service"

Question Four (a) - What risks do you perceive in working under a performance specification?

Question four asked the respondents to list the risks that they perceive are associated with working under an outcome based measurement specification. Four main responses were provided by the 38 respondents who answered this question, as shown in the bar chart below Misunderstandings and incorrect interpretation of performance standards were mentioned by 44% of the full sample.

Consultants, suppliers and clients considered that the risk of tender stage assurances not being met was the next most significant risk whilst contractors and others mentioned problems obtaining funding as a major risk.

It was noted by a consultant that the source of risks is the key to understanding how they will impact and how they can be mitigated. They noted that risks may occur for a number of reasons, including:

• The supply chain understands individual parts of the overall process, but not the interactions
• The designers, who convert performance specifications to method statements, do not fully understand the ramifications of their decisions

Question Four (b) - How would you propose to manage those risks?

Once the respondents had identified the risks associated with performance specifications they were asked to state how they would manage them. As can be seen in the chart below, 31% of the full sample stated that implementing risk analysis and management would be the key to managing the risks; 28% suggested that organisations should work in partnership and 17% stated that performance measurement should be linked to risk transfer to appropriately allocate the risks.

For contractors and suppliers the approach most frequently mentioned was working in partnership.

A number of specific problems were raised including the view that only the major contractors and larger suppliers would be capable of accepting the risks involved. It was also noted that some smaller specialists could make valuable contributions, but would be unlikely to be able to underwrite potential mistakes and would therefore be reliant on larger contractors.

Question Five - What cultural changes do you think will be necessary within industry as a whole to facilitate the implementation of performance specification based contracts across a wider spectrum of work?

This question threw up a number of suggestions from the 40 respondents that identified specific cultural changes that would be required within the industry to facilitate the implementation of performance specifications.

Respondents suggested six main changes, these are listed below in descending order with the percentage of the full sample identifying each change:

• Openness and Trust (37%)
• Change of Industry mindset (35%)
• Change of HA mindset (19%)
• Focus on solutions rather than services (19%)
• Delivery driven by value not cost (15%)
• Long term stability (9%)

The majority of the respondents agreed with the sentiments within the consultation document that;

"An atmosphere of trust and unity will need to be established and maintained by development of a partnership culture among suppliers as well as between the suppliers and the Agency"

Suppliers tended to more frequently mention 'change in HA mindset' as a required change

One organisation commented that the cultural changes would hit small companies hardest.

"The change from a standard specification to a performance one measured against key performance indicators will commit companies to seek innovative approaches and also remove the low price low quality approach still prevalent in the industry. It is the larger committed companies that will be able to survive in that environment since the idea of anticipating the risks of long term performance is alien to many small contractors. Perhaps a stepped approach would be appropriate."

Issues for the Agency

Question Six (a) - Do you think performance specifications should be extended to all Agency contracts?

Overall 83% of the full sample felt that performance specifications should not be extended to all, if any HA contracts, as shown in the chart below.

Percentage of All Organisations Identifying Extent of Performance Specifications (Base: 54 Organisations)
Note: Percentages do not add up to 100% due to multiple and nil responses.

It is worth noting that the majority of the 17% of respondents who supported the extension of performance specifications to all contracts were contractors.

Question Six (b) - Are there specific areas of procurement where the traditional prescriptive type of specification should be retained?

It was generally felt by the 44 respondents who answered this question that traditional prescriptive types of specification should be retained for short term contracts, for improvement schemes and for schemes in sensitive locations. This can be seen in the bar chart below which shows support for the retention of the existing specifications as guidance when performance specifications are introduced.

Percentage of All Respondents Identifying Specific Areas Where Traditional Specifications Should be Retained (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Overall it appears that the message from the response to this question is that performance specifications should not be extended to all HA contracts. This was indicated by a number of comments within the responses as shown below.

"Contractors have not delivered the quality of environmental work expected. It is believed that the construction industry hasn't the experience or the commercial will to take the client's responsibilities."

However a large number of positive suggestions were put forward as to how a performance specification could be implemented in HA contracts, including the following.

"Consider also the production of 'A step by Step Checklist for Performance Specification Work' to indicate typical activities associated with the procuring of performance specification work"

"Developments of the existing specification system could enable the HA to continue to provide minimum specification requirements but encourage contractors to challenge/develop the existing base specifications. There are potential analogies with the aircraft industry with minimum standards for aircraft design managed by the CAA with performance outputs determined by the airlines"

Question Seven - How best can the Agency's technical governance role be supported by suppliers?

The 37 organisations which replied to the question suggested a number of ways for the HA's technical governance role to be supported by suppliers. Working in partnership was stated by 31% of respondents with innovation forums suggested by 28%.

Percentage of All Respondents Identifying Various Ways of Supporting HA Technical Governance Role (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

When analysing the responses from the different respondent types the following key differences were noted. Client bodies rated Quality Management Systems above the other supporting roles followed by working in partnership and guarantees and insurances. Suppliers also emphasised working in partnership and innovation forums but also stated that objectives and responsibilities should be clearly defined. Contractors rated working in partnership as the key approach, followed by quality management systems. Consultants suggested 'innovation forums' followed equally by 'working in partnership' and 'clearly defined objectives and responsibilities'

It is important to note that a considerable number of respondents expressed concerns over the possibility of the HA fulfilling its technical governance role by passing responsibility for standards and/or specifications to suppliers. It was generally felt that this role should remain with the HA and not transferred, however there was support for suppliers to become involved as illustrated by the following quote.

"This is a supporting role that may be best done by close liaison with suppliers or as HA technical governors for example seconding civil servant and their agents into the supply chain industry."

Question Eight - What impact and implications do you consider there will be on suppliers if the Agency were to fulfil its technical governance role by passing responsibility for standards and/or specifications to suppliers?

As stated in the section above analysing the previous question, generally the 43 respondents who answered this question did not support the idea of the HA fulfilling its technical governance role by passing responsibility for standards and/or specifications to suppliers, this is shown in the chart below.

Percentage of All Respondents Identifying Various Implications (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Only 4% of respondents felt there would be no impact with a further 4% stating that less disputes would be encountered and that this would be a positive impact. The rest of the responses stated that the implications of passing responsibility for standards and specifications to suppliers would be negative with impacts including inconsistency, uncertainty, inefficiency and increased costs.

The key point to note is that many respondents, particularly among the consultants, felt that there is a lack of professional and technical resources within the industry and therefore the transfer of responsibility would not be possible. Specific comments to illustrate this included:

"The need for additional technical support/testing could result in increased prices. Furthermore, there could be a duplication of resources where major suppliers maintain parallel organisations and facilities"

By contrast, a few positive comments were received including:

"The impact on suppliers will be one of improved professionalism. It will certainly restrict "small operators" who will not have the financial resources to provide all the necessary know-how and technical expertise. It is anticipated that a new breed of specialist companies will emerge."

Question Nine - What views and suggestions do you have in relation to the use of future looking indicators as a measure of performance?

Although the information gleaned from these responses is of interest it was felt that there were differences of interpretation amongst the respondents as to what 'future looking indicators' meant, what the benefits would be and how they would be utilized. There was also a low rate of response to this question. As can be seen in the bar chart below the most popular response was that if future looking indicators are going to be introduced into performance specifications then they must be robust.

Percentage of All Respondents Giving Various Views on the Use of Future Looking Indicators (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Interestingly 13% of contractors and 23% of consultants provided the response that future looking indicators are of a limited value due to perceived problems associated with measuring the future performance of assets. The majority of suppliers and others did not respond to this question.

Organisations with experience of performance specifications made some interesting points including:

"Gainsharing of efficiencies benefits both parties, as the Agency achieves continuous improvement in value for money, and both parties are able to share in the resulting savings."

"Future looking indicators should take the form of a 'health check', the essential components being serviceability, sustainability, premature failure, future fitness and political and environmental issues."

"Targets are best set with stepped increments in performance after a given time. Suppliers are then incentivised to deliver continuous improvement to meet the new targets".

Issues common to both the Agency and Suppliers

Question Ten - Without a detailed specification to follow, how will suppliers guarantee continued quality and best value for the Agency

"It is essential for the Agency to retain SHW and DMRB as a baseline guide to its requirements and to provide a consistency of approach."

"Ultimately there must be a detailed specification albeit one which has been produced by the Supplier or his supplier in turn. Best value must be assessed during the tender adjudication process by the Agency but will only be finally assured after delivery of the project and its maintenance."

These quotes from responses received, clearly summarise the issues raised, which are analysed in the chart below. The 45 respondents who provided opinions stated that by providing demonstrable outputs and benchmarking of costs, quality and best value could be guaranteed. Other approaches mentioned included quality management systems and minimum performance levels to meet current standards.

Percentage of All Respondents Identifying Various Ways of Guaranteeing Continued Quality (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

However the provision of guarantees and insurances was lower down the list, particularly for contractors and suppliers. One respondent stated:

"Guarantees will be less soundly based and may not be sustainable. To provide the level of assurance asked for and necessary to obtain the required insurance, prices will rise and conservatism will set in. A guarantee is only as good as the paper it is written on!"

Question Eleven - How will the Agency assure itself that the service is being provided to an acceptable standard and that the Agency's objectives are being met?

The key question of how the HA will assure itself that the services being provided are to acceptable standards gave rise to a broad range of responses from the various respondent groups.

Overall the respondents were split between specifying performance standards, implementing an audit process followed by measuring outcomes and quality management systems, as shown in the chart below.

Percentage of All Respondents Identifying Various Ways of Ensuring an Acceptable Standard and That HA Objectives Are Met (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Consultants emphasised 'specify performance standards' and 'implement an audit process' with 54% of the consultancy organisations mentioning these approaches. Contractors stressed 'specify performance standards' followed by 'quality management systems' and 'implementing an audit process'.

Suppliers stated that 'implementing an audit process' was the key to ensuring that standards are being met followed by 'working in partnership'. Clients however rated 'implement an audit process' as the key approach then 'measure outcomes' followed by 'extending PRIDe' and 'Quality Management Systems'.

In general, there was broad agreement that a transparent and robust audit system will assure the HA that the services being provided are to acceptable standards and meet their objectives.

Question Twelve - What mechanisms should be included in contracts to ensure that specifications and standards are updated as required to reflect current best practice and to deliver continuous improvement?

Again a large number of varied responses were received in response to this question. These are summarised in the bar chart below.

Percentage of All Respondents Identifying Various Mechanisms (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Continuous improvement groups were seen as the most beneficial mechanism by both contractors and consultants; clients and suppliers felt that the key mechanism should be the HA retaining the custodian role for specifications and standards, followed by the use of continuous improvement groups.

Some of the other mechanisms mentioned include ensuring that the current specification is the basis for best practice with the HA taking ownership of it and also the subsequent method statement. It was also considered important that the HA sets up a mechanism to make best practice available to all and therefore encourage continuous improvement through the achievement of defined outcomes and that periodic and discrete reviews of contracts and performance be implemented, the results of which could be reviewed by an external panel.

There was also a view expressed that continuous improvement may be difficult to secure in a long term performance led contract without some renegotiation of contract terms.

Question Thirteen (a) - Will suppliers be prepared to share information with other suppliers in relation to performance specification issues?

Overall, as shown in the bar chart below, 43% of the full sample (about two thirds of those who gave a response to this question) stated yes they would be prepared to share information with other suppliers in relation to performance specification issues if it was of benefit to all suppliers; 6% stated no since it is an issue of competitive advantage. Two of the three suppliers who replied also said they would share information.

A further 19% of all organisations stated that it was unlikely unless there were incentivisation mechanisms in place. In particular, half the client bodies which replied were doubtful that information would be shared.

Percentage of All Respondents Giving Alternative Responses (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Question Thirteen (b) - On what terms will suppliers be prepared to share information with other suppliers?

The respondent types generally agreed in relation to this issue that the major requirement is the provision of innovation forums. A number of respondents stated that sharing research costs and providing innovation fees could be a way forward but it was generally agreed that this would be complex and could reduce the cost effectiveness and value of the process.

The analysis of the key responses is shown below.

Percentage of All Respondents Identifying Various Terms (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Specific issues that were raised included Intellectual Property Rights (IPR) specifically for material suppliers and how the HA could compensate IPR owners adequately.

It was suggested that the role of PRIDe and the Network Boards be extended to deal with sharing information and the associated benefits.

One point which was noted was that the trade organisations already share information successfully and that this could be extended as long as competition legislation allows such action.

Question Fourteen- How will the use of performance specification based contracts impact on the bidding process?

The final question in the consultation document attempted to identify the impact on the bidding process if performance specifications were introduced.

All the respondent types agreed that the key impact will be a more costly tender process and a requirement for a longer tender period and negotiations.

Percentage of All Respondents Identifying Various Impacts (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

As can be seen above other key issues were the increased quality of data that would be required by the tenderers and its timely provision at the initiation of the tender stage (a problem that is currently perceived as not being overcome). There is a requirement to understand the asset inventory at the commencement of the tender stage and this requires some considerable resources to be allocated. Open dialogue was stated as a key requirement of the bidding process and this would require additional HA resource.

A number of issues were raised by the respondents including:

"No one should go out of pocket. That should be the bottom line"

"The industry is currently experiencing difficulties in attracting Contractors to tender for Design and Build Projects because of the level of risk the Contractors are being asked to accept."

One point of note made by a number of respondents was that higher prices should be expected at the beginning of the implementation of performance specification since this is the beginning of a learning curve, however in the long term this should level out.

Results Of Discussions With Key Organisations

Introduction

Following the analysis of the responses to the consultation document, a number of individual meetings were held with a broad cross section of the organisations that had provided responses. The purpose of these discussions was to probe the responses and explore the issues raised in more detail. In addition to the face to face meetings, discussions were also held by telephone with a further sample of organisations including some that had not provided a response to the initial consultation document and whose opinions we felt could be valuable.

This section of the report summarises the additional information and insights gained from these interviews. For ease of reference, the results are set out broadly in the order in which the issues were raised in the consultation document. However it was clear from the meetings that many of the issues are inter-linked and do not always neatly fit into the categories set out.

Definition of Performance Specifications

There is an issue of definition that underlies any consideration of performance specifications. The boundary between output and outcome specifications is not always fixed or clear. For example, a specification that requires grass in highway verges to be cut to always be below a height of three inches, rather than a specific number of times in the season, might be regarded as an outcome specification. However a higher level outcome specification might be to require grass to be cut to a standard which is consistent with the safety and environmental requirements of the contract. If the latter were also specified at the outcome level, there would be considerable scope for interpretation and innovation by providers with potential benefits in terms of effectiveness, but also with an inherent risk that the provider's interpretation of acceptable standards may not coincide with the client's.

An example of such a difference of interpretation where the actions taken by the provider might have potentially undesirable consequences would be for the provider to seek to reduce traffic delays caused by roadworks on part of his network by diverting traffic to local authority roads. Another example might be the use of concrete barriers instead of safety fencing on central reservations to reduce cost.

A further distinction which became evident is that whilst measurement of outcomes was acceptable within contracts, possibly as a basis for benchmarking; payment based on outcomes was far less so, particularly if there was a risk that costs incurred by the provider would not be met if outcomes are not achieved.

Benefits of Performance Specifications

Respondents generally felt that there were real and worthwhile benefits in working under a performance specification and more particularly under an outcome specification. These benefits include flexibility, innovation and better integration of design and construction processes.

As one respondent put it, a performance specification 'allows the resourcefulness, experience, imagination and ingenuity of the design and build team to be fully realised'.

A number of examples were given of measures or actions that providers might consider introducing under performance regimes:

• More use of traffic modelling to improve the design of roadworks so as to minimise traffic delays.
• If gully cleaning was subject to an outcome measure, contractors may target gullies prone to flooding with more frequent maintenance and reduce cleaning frequency where not critical.
• Emergency response using motorbikes (trialled in one area)
• Possible use of vehicle-actuated speed warning signs to encourage driver awareness of speeding and improve road safety
• More pro-active approach to customer information, for example keeping databases of people who have commented or complained in the past and providing them (if requested) with better information
• Road markings could be specified on an output basis with the contractor responsible for maintaining them at a specified standard for a number of years
• More careful consideration of maintenance options, an example was given of a failing concrete culvert which was successfully dealt with through a regime of repairs and inspection which the provider considered to have been more cost-effective than reconstruction

However, the impression gained from the interviews was that most providers have not as yet developed clear and well thought out strategies for re-engineering their approach or their processes under an outcome specification based contract. In particular, the possibility of a whole life costing approach leading to significant changes in the way providers deliver the service did not feature significantly in the responses.

It is likely that providers have as yet not fully considered their commercial response to a different (possibly radically different) contract basis and may not do so properly until they see the details of any proposed new contract forms. An additional factor is that many respondents did not support a full outcome based contract (the option 3 scenario) and felt that this is unlikely to be introduced for some time.

Elements Suitable for Performance Specification Contracts

Generally it was felt that performance specifications were more suitable for routine maintenance activities and more workable for long term contracts of 20 years or more duration. Most providers are as yet unconvinced of the feasibility of some of the high level measures put forward in terms of their suitability for use as a basis for payment.

The provider's ability to determine changes in traffic accident levels was particularly questioned and there were also concerns about providers' ability to influence journey time reliability and customer satisfaction. However this view may change as prototyping of these measures proceeds and providers gain more confidence in the measures proposed and in their ability to influence outcomes.

There was also a widely held view that the long term durability of pavements was difficult to predict and was not a suitable element for performance measurement. This was partly because of the limitations of existing knowledge of the performance of pavement materials and structures under different site conditions and partly a concern about the effects of external factors which are not under the provider's control, such as the type of traffic using the road.

It was also considered that there is a difficulty in reliably quantifying the residual life of the network at the handback stage. This would need to be overcome in order to enable suitable handback provisions to be included in contracts to ensure that providers do not reduce the level of maintenance towards the end of the contract.

Measuring Outcomes

There was significant support in principle for moving forward with the introduction of outcome measures into Highways Agency contracts, although there was no clear agreement as to which outcomes should reside with the HA and which with the providers. It was noted that where the responsibility for the delivery of outcome measures rests with the providers, they must be achievable, measurable and deliverable. An example of where factors not under the control of the provider could reduce the validity of the measurement of the provider's performance is where congestion in adjacent HA areas caused knock-on congestion on the provider's network. The possible development of the current DBFO contract KPIs was suggested as a way forward with an initial emphasis on outputs rather than outcomes. Other approaches such as the provision of detailed method statements at tender stage, the use of integrated management systems and operation within agreed quality management plans were also mentioned.

Time is needed to understand and develop the measurement process and this will require additional research and development and testing over and above that being undertaken currently by the HA. Many interviewees noted that the majority of organisations involved in HA commissions do not undertake or invest in any major R&D and therefore a process of piloting and testing may be the most appropriate way forward.

Risk Transfer Issues

The assumption of greater risk was approached with caution by most respondents and particularly by consultants who generally felt that the level of risk they were required to carry should be capped. The key message was that providers should only be required to carry risks which they had the ability to manage.

Latent defects were held up as an example of risks which are outside the provider's control and would make a contract based on an asset condition performance specification unworkable. However it was acknowledged that the problem may be less significant where contracts cover a wide network. Other risks mentioned as not suitable for transferring to providers or would impede the introduction of performance contracts included the following:

• changes in client objectives and changes in legislation (eg higher permitted axle weights). A further example given was the possible introduction of motorway tolls which might encourage night time driving and make it more difficult to carry out night time maintenance.
• Third party interaction including:
  • damage to the network
  • traffic generating development schemes affecting network performance
  • disruption of the network (eg because of accidents or statutory undertakers' work)
• Changes in road use, particularly in terms of HGV volumes and weights.
• Numbers of accidents occurring on the network
• Network inventory inaccuracies and omissions
• Funding changes
• Changes in standards
• Outcome based payment mechanism not covering costs.
• Mismatches between contract performance requirements and client objectives. As one contractor put it 'The risks of performance specification is that design and build team builders under specify, which results in poor operational performance, excessive maintenance and premature replacement.'
• Providers' ability to control network performance
• Lack of management or engineering expertise and lack of skilled resources generally within the highways sector

Client respondents expressed concern at the feasibility of transferring risk from a practical legal point of view of demonstrating the provider's responsibility (eg concrete spalling occurring decades after construction) as well as the risk of providers' inability to meet their obligations should the need arise (long term financial viability of commercial enterprises).

The transfer of risk was perceived to be particularly problematic for small organisations which may not have the financial strength to absorb substantial levels of risk. There was a perception that the cost of insuring the risks related to performance specifications would be high, although none of the firms interviewed had as yet tested the insurance market in this respect. By contrast, some materials suppliers were concerned that risks would be passed down the supply chain and that as large companies (in comparison to most contractors) they feel that they 'have more to lose' if inappropriate risks are taken.

A further issue in relation to risk is that providers will look at risk in commercial terms, which may not coincide with the political perception of risk. It would be difficult to frame commercially acceptable performance contracts that realistically reflect political risk.

A number of interviewees stated that providers prefer to have steady assurance of turnover and profit and therefore will not take unnecessary risks for the opportunity of possible large profits. If innovations require high risks, they believed that these should be developed by the HA in partnership with providers to ensure that continuous improvements are assured and risks minimised without detriment to providers' businesses.

There was also a concern that acceptance of risk by providers may lead to more conservative design decisions. Sub contractors and suppliers may not be able to fund their risk allocations or obtain the necessary insurances and insurance companies are likely to require client approval for decisions, new techniques and innovations in order to reduce the risk to the provider and themselves

It was considered that when risks are appropriately allocated and shared where necessary between the HA and the providers, pricing will be more accurate and transparent, providing greater value for money. It was additionally noted that the benefits of partnering and risk sharing would be increased if the HA was sharing both the pain and the gain in an equity or other arrangement.

A number of methods for managing risks appropriately were suggested, some of which are summarised in the following list.

• Effective and thorough selection of providers based on both their own risk management techniques and those of their supply chain
• Early involvement of supply chain in the procurement process
• Partnership principles applied to the management of the whole supply chain
• Supply chain agreements that have clearly defined risk allocation linked to payment
• Retention by the public sector of risks that can not be appropriately allocated within the supply chain or insured against
• Incentivised supply chain arrangements, including back-to-back risk sharing limited to the particular activity that is being subcontracted
• Robust risk mitigation plans

Cultural Change

There was a general view that a change of mindset towards more openness and trust would be required for both the HA and providers under a performance specified contract. Providers would have to be prepared to take on more risk and this might be less easy for consultancy firms than for contractors who are more used to operating in a risk environment.

It will also be necessary for the HA to concentrate on monitoring outcomes and accept a more hands-off role in terms of monitoring detailed specifications. As an example, it was pointed out by one of the providers that Sector Schemes which were intended to lead to less client supervision, by providing some quality assurance, including effective training of operatives, have not yet resulted in a reduction in client supervision.

There is an understandable reluctance within some parts of the HA to move in the direction of reducing direct supervision of processes until there is clear evidence that performance contracts are resulting in providers delivering the required quality and that the measures used are sufficiently robust to ensure the right result.

Technical Governance

Some providers identified potential benefits if the HA were to pass responsibility for standards to suppliers, as illustrated by the following comments: 'The impact on suppliers will be one of improved professionalism' and 'Engineers should be encouraged to engineer solutions instead of the prescriptive "easy option"'

However, the concept of the HA exercising its technical governance role by passing responsibility for standards to providers was generally not supported. The reasons for the lack of support seem partly cultural because the industry has not been used to 'guaranteeing' its products in terms of either durability or fitness for purpose.

A second and related reason is the lack of adequate technical knowledge to enable such 'guarantees' to be responsibly given, this in turn is a reflection of the fact that the industry currently looks to the HA for much of the R & D required. The scale of change required is considerable and is likely to take some time. Some clients and providers questioned whether it would be in the HA's interests to become a less technically involved organisation. These issues have to be weighed up against the benefits which were set out earlier. It was also noted that the HA is in a better position to justify expenditure on R&D given its responsibility for the entire network whereas providers may find it difficult to justify such expenditure on the basis of a single contract. A contractor noted that 'If the responsibility for full technical compliance was passed to the supplier each would have to devise R&D plans along with long term monitoring recording and feedback systems…'

Client staff were generally concerned at the potential loss of quality if only outputs or outcomes were specified with no detailed technical specification. A number of clients pointed to examples of poor compliance with existing detailed specifications as evidence that standards are likely to fall further in the absence of such specifications. However, there was also a view that some of the non-technical requirements of existing specifications (eg the parts of DMRB and MCHW concerned with the management of maintenance programmes) could be removed if robust performance specifications were in place. There was also an HA client view that the existing departures process did not inhibit innovation and perhaps needed to be better communicated to providers. This view was supported by a local authority client who commented 'Desired benefits could be achieved without the perceived risks by maintaining the existing procedure but adopting a more flexible approach to accepting suppliers/contractors proposed alternatives'

Some providers felt that the geometric standards within DMRB should be retained but there may be less necessity for detailed construction related standards.

A further issue mentioned was the potential impact on third party claims if there were no generally agreed technical standards in place.

There is support for more involvement of industry in the setting and approval of standards and a number of suggested ways of achieving this were put forward, including an expanded role for the Roads Board or some form of new technical standards approvals body. All respondents agreed that the HA should continue to be involved in the setting of technical standards at the very least in a coordinating role and a number of both clients and providers believed that the HA should maintain its leadership role in terms of technical expertise. It was noted that 'the Steel Construction Institute is an example where quality has been monitored / improved by a suppliers representative body'.

A number of organisations pointed to the wide use of HA standards such as SHW and DMRB by public and private sector clients outside the HA. Some public sector clients feel that they should be more involved in formulating the standards and a number of providers would favour such a development particularly if it led to nationally agreed standards which can be used on all UK highway projects, as this would reduce their 'learning' costs in dealing with different standards.

Ensuring Quality and Continuous Improvement

Some providers believed that suppliers would develop their own specifications over time and would have to update these in order to remain competitive. One commented that 'With an outcome specification for new contracts there is no need to ensure that specifications and standards are updated. Updating will occur naturally as suppliers seek to maintain competitiveness'.

The need for some form of change mechanism within contracts to enable requirements to be updated in line with changes in technology or for other reasons was referred to in some of the interviews.

Impact on Tendering

A concern expressed by a number of respondents, particularly clients, was that the introduction of performance specifications may lead to a reduction in the number of firms prepared to tender with a risk of increased costs. There is some evidence from local authority clients and from contractors that this has already happened in some instances. One company had experienced having to incur a large loss as a result of agreeing to enter a performance contract for pavement durability (for which the measurement basis was limited to only one type of durability test). The firm commented that they would be very unlikely to bid for such a contract in future as the risks are considered too high.

Providers commented on the high cost and risk associated with tendering and suggested that the HA should consider reimbursing a proportion of tendering costs incurred by unsuccessful bidders. The complexity of tenders was commented on. One company has adopted the practice of deferring detailed due diligence checking of the contract until after success at the bidding stage, this was due to the high cost of such vetting of tenders.

A number of providers emphasised that the availability of robust network data is particularly important in tendering for a performance based contract and providers felt that the quality of asset data would need to be considerably enhanced to enable them to tender for a performance contract. A number of respondents commented that the evaluation of tenders under a performance specified contract may become more complex and less transparent as it would be difficult to compare tenders which are prepared on the basis of differing standards.

Payment Mechanisms

The need to cover costs and secure a reasonable return on investment is a key concern for providers when considering the linking of payment to performance measurement. If too high a proportion of contract payments are linked to performance, potential tenderers may be discouraged from bidding or bid prices may increase significantly to cover the risk.

Some providers expressed a preference for an open book system with basic costs being met and with additional performance payments linked to achieving specific outcomes.

Consultation Respondents

The following table provides the list of respondents to the Developing Performance Specifications Consultation Document, whose responses have been analysed in this report.

Consultants

• Scott Wilson Pavement Engineering
• WSP Group
• Scott Wilson
• Mouchel
• Atkins
• Parsons Brinckerhoff
• Gifford and Partners Ltd
• Pell Frischmann Consultants Ltd
• KBR Infrastructure
• EC Harris
• URS
• Optima Infrastructure Management
• Rex Perkins Ltd

Contractors

• Aon Ltd
• Amey Highways
• Carillion - URS
• Sir Robert McAlpine Ltd
• Alfred McAlpine Civil Engineering
• Carillion Plc
• Balfour Beatty Civil Engineering Limited
• Laing O'Rourke
• Civil Engineering Contractors Association
• May Gurney Limited
• Jarvis Highway Maintenance
• Keir Construction Limited
• Ringway
• AMScott
• AMEC

Suppliers

• Golden River Traffic
• Quarry Products Association
• British Precast Concrete Federation
• RSMA
• Aggregate Industries UK Ltd

Client Bodies

• Highways Agency, SSR SDM
• South Ayrshire Council
• Herefordshire Council
• Society of Chief engineers for Transportation in Scotland (SCOTS)
• BBA
• Highways Agency, Technical Appraisal
• South Lanarkshire Council
• DfT
• Devon County Council
• Highways Agency - Highways Infrastructure Group SSR
• CSS Soils and Materials (Design and Specification) Group
• Welsh Assembly
• Highways Agency - Sydney Xavier

Others

• Confederation of Passenger Transport UK
• Michael Graham Consultancy
• Colin Cleverly
• Environment Agency
• Landscape Institute
• The British Horse Society
• KPMG
• David Cooper