Highways Agency

Analysis Of Responses

In April 2003 the Highways Agency issued a consultation document entitled "Developing Performance Specifications: Consultation Document" to put forward options for widening the use of performance specifications, not only in private finance contracts but across all forms of procurement. The document was circulated to a range of stakeholders both within and outside the HA.

A total of 54 organisations and individuals responded to the consultation document. The range of organizations and individuals who submitted is shown in the pie chart below. As can be seen the respondents were divided into the following categories:

• Consultants
• Contractors
• Suppliers
• Client Bodies
• Others (eg organisations representing particular types of road user)

The responses that were received to each of the consultation paper questions were input into a database utilising Microsoft Excel software such that they could be analysed. Following an initial review of the responses the key themes were identified, this formed the basis for the detailed analysis shown on the following pages.

The results of the analysis of responses to each question are presented below in chart form and where appropriate some of the comments made by respondents are included (on a non-attributable basis) in order to illustrate and expand the views expressed. It is important to emphasise that the sample of organizations surveyed, whilst intended to cover the range of views and interests likely to be relevant, does not constitute a statistically representative sample of stakeholders. Accordingly, the significant element of the results is the identification of the main views expressed and issues raised rather than the precise proportions of alternative viewpoints. It should also be noted that some respondents chose not to answer all of the questions posed whilst others provided more than one comment in response to each question, accordingly the aggregate percentage of responses may not add up to 100%.

Issues for the Suppliers

Question One - What are the benefits in working under a performance specification?

This question was answered by 42 of the respondents to the consultation who suggested a range of benefits that they believed would be achieved in working under a performance specification. As shown below, 61% of the full sample of respondents that participated in the response to the survey stated that the major benefit of performance specifications would be to encourage of innovation; 44% felt that efficiency would be improved and a third of organizations mentioned that increased value would be achieved.

There were only small differences of emphasis between the different types of organizations. Contractors tended to stress increased value and improving efficiency whilst consultants emphasised improved efficiency and flexibility. Client bodies saw encouraging innovation as a key benefit.

Question Two (a) - What elements of performance should be measured on an outcome basis?

A total of 34 respondents to the consultation document answered this specific question, many of whom identified more than one element of performance that should be measured on an outcome basis. As can be seen below, 26% of the full sample agreed that both routine and cyclic maintenance should be measured on an outcome basis, in fact these were the most common responses by the contractors, clients and others.

A further 17% believed that all elements should be measured on an outcome basis, however this was in many cases qualified elsewhere in the response by reference to the need for robust measures. There were few differences between the responses from the different types of organization, although none of the small sample of client organizations appeared to support the view that all elements of performance should be measured on an outcome basis.

Question Two (b) - Are there any which should not be so measured?

Following on from the initial question the consultation document went on to ask if there were any elements of performance that should not be measured on an outcome basis. A range of responses was given by the 34 organisations which provided a response on this issue, including safety, technical standards, customer service and elements outside the suppliers' control. Some responses indicated that no elements of performance should be excluded.

As can be seen in the chart below, 37% of the full sample stated that elements that were outside the control of the suppliers should not be measured on an outcome basis, citing as reasons the additional risk for the supplier, the difficulty of measurement and the need for continuous improvement. More specifically, 17% believed that safety should not be included within a performance specification due to the influence of factors outside the contractor's control such as driver behaviour.

It was stated by one contractor that "the specification should be focused more towards outcome in line with the greater degree to which the requirement is definable, quantifiable and understood".

Suppliers and the client bodies tended to support the view that technical parameters (e.g. the performance of pavement layers) should not be measured on an outcome basis and this was further investigated in questions 7 and 8 of the consultation document. A number of contractors and consultants stated that no elements of performance should be excluded as all are appropriate for performance specifications, however, in many cases, this view was qualified elsewhere in the organisations' response

About 13% of contractors stated that customer satisfaction should not be measured on an outcome basis. This was further developed within the contractors' responses by stating that customer satisfaction is largely outside the contractor's control and also difficult to measure on an objective basis and to relate to current performance. Safety and the physical performance of assets were also the most commonly mentioned specific elements by client organizations.

Question Three (a) - What are your views on the ability of industry to assume the risks inherent in outcome-based performance specifications?

Again this question was divided into two elements during the analysis. The first part of the question asked for respondents' views on the ability of industry to assume the risks associated with performance specifications and 41 respondents provided specific answers to this question.

As can be seen in the bar chart above, 57% of the full sample expressed the view that only those risks that can be managed by the contractors and/or suppliers should be transferred to them. This view was held by all respondent types (87% of contractors, 85% of consultants, 40% of suppliers and 31% of clients). There was also concern that risks would be passed down the supply chain; as one respondent put it: "The smallest player in the game carries the entire risk; and yet the risk he carries is unknown to the Agency which only deals with the main contractor"

Clients were particularly concerned that unnecessary risks would be taken by contractors either for commercial reasons or through a lack of adequate understanding.

It was noted by one consultant that "There will be little enthusiasm on the part of contractors to assume risks where they cannot be quantified at time of tender. As an example, risks associated with the improvement of safety performance would be problematic because the safety performance of the road is partly dependent on factors outside their control including driver behaviour and traffic volumes."

Question Three (b) - Are there any risks that should not be transferred?

Only 31 of the respondents answered this question with the majority of clients and suppliers feeling that they were not able to identify specific risks. Those organizations that did respond generally agreed that risks associated with political and client objectives should not be transferred to providers due to the lack of influence that they have over changes in policy and objectives.

As can be seen in the bar chart below, providing the breakdown of responses by contractors and consultants; 54% of the respondents believed that political and client objective risks should not be transferred with other key risks including safety identified by 21% and traffic forecasts by 14%.

In contrast to the responses by the consultants, contractors and suppliers; relatively more client bodies tended to feel that technical issues such as structural integrity and highway geometry were not appropriate for transfer, this may be a reflection of the particular specialisms of the client representatives who replied.

One Contractor stated that "The Highways Agency is best placed to deal with risks that will have a major impact if they occur. If the industry is asked to take on these risks in a competitive situation, the HA faces the additional risk that the supplier will not be able to fulfill its responsibilities. The ability to insure such risks is limited, and given the current state of the market is not likely to be cost-effective for Highways Agency"

Whilst another said that "The risk transfer will be affected by the extent and quality of the condition and inventory data available and his ability to control the service"

Question Four (a) - What risks do you perceive in working under a performance specification?

Question four asked the respondents to list the risks that they perceive are associated with working under an outcome based measurement specification. Four main responses were provided by the 38 respondents who answered this question, as shown in the bar chart below Misunderstandings and incorrect interpretation of performance standards were mentioned by 44% of the full sample.

Consultants, suppliers and clients considered that the risk of tender stage assurances not being met was the next most significant risk whilst contractors and others mentioned problems obtaining funding as a major risk.

It was noted by a consultant that the source of risks is the key to understanding how they will impact and how they can be mitigated. They noted that risks may occur for a number of reasons, including:

• The supply chain understands individual parts of the overall process, but not the interactions
• The designers, who convert performance specifications to method statements, do not fully understand the ramifications of their decisions

Question Four (b) - How would you propose to manage those risks?

Once the respondents had identified the risks associated with performance specifications they were asked to state how they would manage them. As can be seen in the chart below, 31% of the full sample stated that implementing risk analysis and management would be the key to managing the risks; 28% suggested that organisations should work in partnership and 17% stated that performance measurement should be linked to risk transfer to appropriately allocate the risks.

For contractors and suppliers the approach most frequently mentioned was working in partnership.

A number of specific problems were raised including the view that only the major contractors and larger suppliers would be capable of accepting the risks involved. It was also noted that some smaller specialists could make valuable contributions, but would be unlikely to be able to underwrite potential mistakes and would therefore be reliant on larger contractors.

Question Five - What cultural changes do you think will be necessary within industry as a whole to facilitate the implementation of performance specification based contracts across a wider spectrum of work?

This question threw up a number of suggestions from the 40 respondents that identified specific cultural changes that would be required within the industry to facilitate the implementation of performance specifications.

Respondents suggested six main changes, these are listed below in descending order with the percentage of the full sample identifying each change:

• Openness and Trust (37%)
• Change of Industry mindset (35%)
• Change of HA mindset (19%)
• Focus on solutions rather than services (19%)
• Delivery driven by value not cost (15%)
• Long term stability (9%)

The majority of the respondents agreed with the sentiments within the consultation document that;

"An atmosphere of trust and unity will need to be established and maintained by development of a partnership culture among suppliers as well as between the suppliers and the Agency"

Suppliers tended to more frequently mention 'change in HA mindset' as a required change

One organisation commented that the cultural changes would hit small companies hardest.

"The change from a standard specification to a performance one measured against key performance indicators will commit companies to seek innovative approaches and also remove the low price low quality approach still prevalent in the industry. It is the larger committed companies that will be able to survive in that environment since the idea of anticipating the risks of long term performance is alien to many small contractors. Perhaps a stepped approach would be appropriate."

Issues for the Agency

Question Six (a) - Do you think performance specifications should be extended to all Agency contracts?

Overall 83% of the full sample felt that performance specifications should not be extended to all, if any HA contracts, as shown in the chart below.

Percentage of All Organisations Identifying Extent of Performance Specifications (Base: 54 Organisations)
Note: Percentages do not add up to 100% due to multiple and nil responses.

It is worth noting that the majority of the 17% of respondents who supported the extension of performance specifications to all contracts were contractors.

Question Six (b) - Are there specific areas of procurement where the traditional prescriptive type of specification should be retained?

It was generally felt by the 44 respondents who answered this question that traditional prescriptive types of specification should be retained for short term contracts, for improvement schemes and for schemes in sensitive locations. This can be seen in the bar chart below which shows support for the retention of the existing specifications as guidance when performance specifications are introduced.

Percentage of All Respondents Identifying Specific Areas Where Traditional Specifications Should be Retained (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Overall it appears that the message from the response to this question is that performance specifications should not be extended to all HA contracts. This was indicated by a number of comments within the responses as shown below.

"Contractors have not delivered the quality of environmental work expected. It is believed that the construction industry hasn't the experience or the commercial will to take the client's responsibilities."

However a large number of positive suggestions were put forward as to how a performance specification could be implemented in HA contracts, including the following.

"Consider also the production of 'A step by Step Checklist for Performance Specification Work' to indicate typical activities associated with the procuring of performance specification work"

"Developments of the existing specification system could enable the HA to continue to provide minimum specification requirements but encourage contractors to challenge/develop the existing base specifications. There are potential analogies with the aircraft industry with minimum standards for aircraft design managed by the CAA with performance outputs determined by the airlines"

Question Seven - How best can the Agency's technical governance role be supported by suppliers?

The 37 organisations which replied to the question suggested a number of ways for the HA's technical governance role to be supported by suppliers. Working in partnership was stated by 31% of respondents with innovation forums suggested by 28%.

Percentage of All Respondents Identifying Various Ways of Supporting HA Technical Governance Role (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

When analysing the responses from the different respondent types the following key differences were noted. Client bodies rated Quality Management Systems above the other supporting roles followed by working in partnership and guarantees and insurances. Suppliers also emphasised working in partnership and innovation forums but also stated that objectives and responsibilities should be clearly defined. Contractors rated working in partnership as the key approach, followed by quality management systems. Consultants suggested 'innovation forums' followed equally by 'working in partnership' and 'clearly defined objectives and responsibilities'

It is important to note that a considerable number of respondents expressed concerns over the possibility of the HA fulfilling its technical governance role by passing responsibility for standards and/or specifications to suppliers. It was generally felt that this role should remain with the HA and not transferred, however there was support for suppliers to become involved as illustrated by the following quote.

"This is a supporting role that may be best done by close liaison with suppliers or as HA technical governors for example seconding civil servant and their agents into the supply chain industry."

Question Eight - What impact and implications do you consider there will be on suppliers if the Agency were to fulfil its technical governance role by passing responsibility for standards and/or specifications to suppliers?

As stated in the section above analysing the previous question, generally the 43 respondents who answered this question did not support the idea of the HA fulfilling its technical governance role by passing responsibility for standards and/or specifications to suppliers, this is shown in the chart below.

Percentage of All Respondents Identifying Various Implications (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Only 4% of respondents felt there would be no impact with a further 4% stating that less disputes would be encountered and that this would be a positive impact. The rest of the responses stated that the implications of passing responsibility for standards and specifications to suppliers would be negative with impacts including inconsistency, uncertainty, inefficiency and increased costs.

The key point to note is that many respondents, particularly among the consultants, felt that there is a lack of professional and technical resources within the industry and therefore the transfer of responsibility would not be possible. Specific comments to illustrate this included:

"The need for additional technical support/testing could result in increased prices. Furthermore, there could be a duplication of resources where major suppliers maintain parallel organisations and facilities"

By contrast, a few positive comments were received including:

"The impact on suppliers will be one of improved professionalism. It will certainly restrict "small operators" who will not have the financial resources to provide all the necessary know-how and technical expertise. It is anticipated that a new breed of specialist companies will emerge."

Question Nine - What views and suggestions do you have in relation to the use of future looking indicators as a measure of performance?

Although the information gleaned from these responses is of interest it was felt that there were differences of interpretation amongst the respondents as to what 'future looking indicators' meant, what the benefits would be and how they would be utilized. There was also a low rate of response to this question. As can be seen in the bar chart below the most popular response was that if future looking indicators are going to be introduced into performance specifications then they must be robust.

Percentage of All Respondents Giving Various Views on the Use of Future Looking Indicators (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Interestingly 13% of contractors and 23% of consultants provided the response that future looking indicators are of a limited value due to perceived problems associated with measuring the future performance of assets. The majority of suppliers and others did not respond to this question.

Organisations with experience of performance specifications made some interesting points including:

"Gainsharing of efficiencies benefits both parties, as the Agency achieves continuous improvement in value for money, and both parties are able to share in the resulting savings."

"Future looking indicators should take the form of a 'health check', the essential components being serviceability, sustainability, premature failure, future fitness and political and environmental issues."

"Targets are best set with stepped increments in performance after a given time. Suppliers are then incentivised to deliver continuous improvement to meet the new targets".

Issues common to both the Agency and Suppliers

Question Ten - Without a detailed specification to follow, how will suppliers guarantee continued quality and best value for the Agency

"It is essential for the Agency to retain SHW and DMRB as a baseline guide to its requirements and to provide a consistency of approach."

"Ultimately there must be a detailed specification albeit one which has been produced by the Supplier or his supplier in turn. Best value must be assessed during the tender adjudication process by the Agency but will only be finally assured after delivery of the project and its maintenance."

These quotes from responses received, clearly summarise the issues raised, which are analysed in the chart below. The 45 respondents who provided opinions stated that by providing demonstrable outputs and benchmarking of costs, quality and best value could be guaranteed. Other approaches mentioned included quality management systems and minimum performance levels to meet current standards.

Percentage of All Respondents Identifying Various Ways of Guaranteeing Continued Quality (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

However the provision of guarantees and insurances was lower down the list, particularly for contractors and suppliers. One respondent stated:

"Guarantees will be less soundly based and may not be sustainable. To provide the level of assurance asked for and necessary to obtain the required insurance, prices will rise and conservatism will set in. A guarantee is only as good as the paper it is written on!"

Question Eleven - How will the Agency assure itself that the service is being provided to an acceptable standard and that the Agency's objectives are being met?

The key question of how the HA will assure itself that the services being provided are to acceptable standards gave rise to a broad range of responses from the various respondent groups.

Overall the respondents were split between specifying performance standards, implementing an audit process followed by measuring outcomes and quality management systems, as shown in the chart below.

Percentage of All Respondents Identifying Various Ways of Ensuring an Acceptable Standard and That HA Objectives Are Met (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Consultants emphasised 'specify performance standards' and 'implement an audit process' with 54% of the consultancy organisations mentioning these approaches. Contractors stressed 'specify performance standards' followed by 'quality management systems' and 'implementing an audit process'.

Suppliers stated that 'implementing an audit process' was the key to ensuring that standards are being met followed by 'working in partnership'. Clients however rated 'implement an audit process' as the key approach then 'measure outcomes' followed by 'extending PRIDe' and 'Quality Management Systems'.

In general, there was broad agreement that a transparent and robust audit system will assure the HA that the services being provided are to acceptable standards and meet their objectives.

Question Twelve - What mechanisms should be included in contracts to ensure that specifications and standards are updated as required to reflect current best practice and to deliver continuous improvement?

Again a large number of varied responses were received in response to this question. These are summarised in the bar chart below.

Percentage of All Respondents Identifying Various Mechanisms (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Continuous improvement groups were seen as the most beneficial mechanism by both contractors and consultants; clients and suppliers felt that the key mechanism should be the HA retaining the custodian role for specifications and standards, followed by the use of continuous improvement groups.

Some of the other mechanisms mentioned include ensuring that the current specification is the basis for best practice with the HA taking ownership of it and also the subsequent method statement. It was also considered important that the HA sets up a mechanism to make best practice available to all and therefore encourage continuous improvement through the achievement of defined outcomes and that periodic and discrete reviews of contracts and performance be implemented, the results of which could be reviewed by an external panel.

There was also a view expressed that continuous improvement may be difficult to secure in a long term performance led contract without some renegotiation of contract terms.

Question Thirteen (a) - Will suppliers be prepared to share information with other suppliers in relation to performance specification issues?

Overall, as shown in the bar chart below, 43% of the full sample (about two thirds of those who gave a response to this question) stated yes they would be prepared to share information with other suppliers in relation to performance specification issues if it was of benefit to all suppliers; 6% stated no since it is an issue of competitive advantage. Two of the three suppliers who replied also said they would share information.

A further 19% of all organisations stated that it was unlikely unless there were incentivisation mechanisms in place. In particular, half the client bodies which replied were doubtful that information would be shared.

Percentage of All Respondents Giving Alternative Responses (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Question Thirteen (b) - On what terms will suppliers be prepared to share information with other suppliers?

The respondent types generally agreed in relation to this issue that the major requirement is the provision of innovation forums. A number of respondents stated that sharing research costs and providing innovation fees could be a way forward but it was generally agreed that this would be complex and could reduce the cost effectiveness and value of the process.

The analysis of the key responses is shown below.

Percentage of All Respondents Identifying Various Terms (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

Specific issues that were raised included Intellectual Property Rights (IPR) specifically for material suppliers and how the HA could compensate IPR owners adequately.

It was suggested that the role of PRIDe and the Network Boards be extended to deal with sharing information and the associated benefits.

One point which was noted was that the trade organisations already share information successfully and that this could be extended as long as competition legislation allows such action.

Question Fourteen- How will the use of performance specification based contracts impact on the bidding process?

The final question in the consultation document attempted to identify the impact on the bidding process if performance specifications were introduced.

All the respondent types agreed that the key impact will be a more costly tender process and a requirement for a longer tender period and negotiations.

Percentage of All Respondents Identifying Various Impacts (Base: 54 Respondents)
Note: Percentages do not add up to 100% due to multiple and nil responses.

As can be seen above other key issues were the increased quality of data that would be required by the tenderers and its timely provision at the initiation of the tender stage (a problem that is currently perceived as not being overcome). There is a requirement to understand the asset inventory at the commencement of the tender stage and this requires some considerable resources to be allocated. Open dialogue was stated as a key requirement of the bidding process and this would require additional HA resource.

A number of issues were raised by the respondents including:

"No one should go out of pocket. That should be the bottom line"

"The industry is currently experiencing difficulties in attracting Contractors to tender for Design and Build Projects because of the level of risk the Contractors are being asked to accept."

One point of note made by a number of respondents was that higher prices should be expected at the beginning of the implementation of performance specification since this is the beginning of a learning curve, however in the long term this should level out.