Review of IPPC 5 in Seattle: Europe can learn from new international procurement research
7 September 2012
Cosy consensus is a rare thing in the public procurement world. I have sat on fractious evaluation panels and listened to passionate attacks and defences of scoring methodologies, not always as an innocent bystander. Disagreement can stimulate a more rigorous approach to procurement, as decisions must be defended internally as well as externally. But it can also lead to compromises which are 'suboptimal', to borrow a term from economics. It is interesting then to see some of the same frictions which afflict evaluation panels played out in the growing academic discourse on public procurement.
The Fifth International Public Procurement Conference (IPPC5) took place in Seattle, Washington in August. Delegates from over thirty countries gathered amid news of local forest fires, soaring petrol prices and, further south in Texas, outbreaks of the West Nile virus. Perhaps no surprise then that fifteen of the papers focused on green or sustainable procurement.These ranged from a Swedish economic analysis of the "Objective Effectiveness of GPP" to a study of social value in Ugandan public procurement. One of the themes of the conference was the development of procurement as a profession, with a set of internationally shared principles. This is trickier than it sounds, due in part to the lack of interaction between academia, practitioners and policy-makers. Amongst the SPP contributions (including my own) there was a notable lack of common definitions, let alone techniques or findings. Perhaps this is not surprising for a relatively young field of study - but is it a good thing?
The papers divided roughly evenly between those which presented empirical findings or case studies and those which focused on the theoretical/legal/policy framework. It would have been nice if more did both, for example by carrying out field research which tests the precepts upon which law and policy are based. This is particularly valuable for SPP, as the policy discourse is developing rapidly while high-quality empirical studies are still few and far between. I have written previously about some of the challenges which face such studies in an EU context, such as low response rates for questionnaires and the limited information available from OJEU notices. Meanwhile, our colleagues in Israel and Russia have pushed ahead with innovative research methods.
Dr. Omer Dekel of the Ramat-Gan Center of Law and Business presented draft findings on "Cognitive Biases in Government Contracting" - a fascinating study of the 'low-bid bias' in which evaluators who have been exposed to bids’ prices give low-priced bids higher marks for quality than they would otherwise. The effects of this bias were observed in experiments conducted with experienced public procurement officials on professional training courses in Israel. Procurers were given two bids to analyse and score, one of which was clearly of lower quality. In the first experiment, they were asked to score the qualitative criteria without being aware of the price. In the second, conducted with the same subjects but after a four-week delay, they were asked to reevaluate the same bids, this time with knowledge of their prices. The result was a 40% decrease in the gap between the two scores, in favour of the lower-priced, lower-quality bid. This included a 17% decrease in the difference between the qualitative scores. Being exposed to the price at the same time as the qualitative aspects changed the outcome in 20% of the evaluations.
The authors (Dekel and Amos Schurr) make links between these findings and policy arguments for two-stage bid analysis. What is striking about their work is that it manages to gather information about procurement decision making which is both more nuanced and convincing than many large-scale surveys. Extensions to their work would include repeating the experiments in other jurisdictions or under different conditions (e.g. more than two bids, different spreads of quality and cost) to see if the results are replicated. Such work in the field of SPP, for example to examine the use of different life-cycle costing methods, could provide welcome insight into policy and legal developments. It is suggested as a complement, rather than replacement for large-scale studies. The ongoing role of these is evident in two papers by Russian economists Olga Demidova and Andrei Yakovlev, from the Higher School of Economics, Moscow.
Their work analyses empirical data on public procurement in Russia. In an article published in the most recent Journal of Public Procurement, they focus on the relationship between participation in public procurement and receipt of other forms of government support, based on a survey of nearly one thousand manufacturing companies. While not finding a strong correlation at all levels of government, they highlight the potential for both complementarity and substitution effects. The second paper, co-authored by Olga Balaeva, examines the effect of different procurement procedures on price reductions and contract performance problems. This was based on analysis of two thousand purchases made by a single large public entity in the period 2008-2010 - data which the authors turned to in frustration at the unavailability of comprehensive data from the existing national e-procurement portal. The results of both studies, and their methodologies, are relevant for a range of procurement topics from support of SMEs to use of auctions and detection of corruption.
In the SPP world, at least in Europe, it sometimes seems we are all singing from the same hymn sheet. Everyone agrees that it should be possible to take social and environmental criteria into account in procurement decisions. Disagreement focuses on how this can be done and to what extent it should be governed by EU law. Questions about the efficacy of social and environmental criteria are seldom to the fore. Instead we get a high-level policy discourse about the need for SPP, and the (often unconnected) experience of individual public authorities putting it into practice. In between there are vast pools of indifference and multiple constraints and demands on public spending. Attempts to quantify the extent of SPP in the EU are welcome, but do not answer basic questions such as whether the criteria included lead to any measurable improvement in social or environmental outcomes - which is the whole point.
So can we benefit from the divergence in approaches evident at IPPC5? That probably depends on whether we have the patience to examine the scholarship and relate it to our EU experience. The efforts of the NIGP and other bodies to advance international exchange and develop common principles should assist in this, especially if they remain both open and rigorous. EU countries were strongly represented amongst the delegates and we can expect to see more research on sustainable procurement and related topics, perhaps building upon the above methods. There is plenty to look forward to for IPPC6, which will be held in Dublin in 2014.
Copyright 2012-2016 Abby Semple