Sreeja Nair and Michael Howlett argue for the value of policy experiments and pilot projects in a study on 15 pilot initiatives in multiple sectors, including the water sector.
“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong,” wrote Richard P. Feynman.
Experiments are considered to be the bedrock of science and often a beacon of guiding light in uncharted territory. Specifically, policy experiments can provide useful insights for policymakers dealing with water resource management.
Managing water resources is a major challenge for policymakers and water managers given the multiple stresses which adversely impact water resources worldwide. There is a range of issues that need handling and often inadequate resources to do so.
In the world of water, these concerns include conflicts over water allocation and use, the inadequacy of current water-distribution systems, the presence of multiple stakeholders with varied interests which lead to competing demands for resource use, and increasing numbers and severity of environmental stressors such as climate change which impact on the water cycle.
In this conflicted and complex landscape, policy experiments can be a useful tool for effective policy design. Pilot projects, in particular, are an increasingly common mode of policy experimentation and a widely-used method to introduce major government policies or programmes in a phased manner, allowing them to be tested, evaluated and adjusted beforehand.
What is a policy experiment?
Governments can initiate experiments in the water sector for a variety of reasons such as for testing potential policies, implementing policies that have difficulties in being implemented and evaluating new policies at an early stage.
Following the classification as described in the book, Development projects as policy experiments: an adaptive approach to development administration, by Dennis Rondinelli, an international development expert, at its simplest form, policy experiments can be need-based assessments such as willingness-to-pay surveys; these are often used as a proxy to assess the demand for water and sanitation services. For example, a willingness-to-pay experiment of 1800 households in Sri Lanka demonstrated that the demand for improvement in water and sanitation services is driven by a combination of several factors such as socioeconomic status, costs, location, means for self-provision and perceptions of stakeholders.
Another category of policy experiments involves projects that explore the most effective way of achieving pre-set policy goals. This modus operandi involves innovations and transition experiments. Literature defines transitions as ‘a gradual, continuous process of structural change within a society or culture’. Transitions in society are by their very nature complex, spread over long timeframes, involve many stakeholders and actors, and occur across multiple levels.
An example is a study that examined 11 local-scale experiments in Australian cities and found that sustainable transitions to urban-water management required changes in underlying societal culture and beliefs along with structural reforms.
Another category of policy experiments involves those that identify gaps in current policies. In this case policy experiments help fix the gaps before the programme is scaled up and avoid costly mistakes. A study on the privatization of urban-water services in Kenya concluded that an experiment towards privatization failed to accurately identify and fix the gaps in the current policy process owing to the intrusion of central and local government authorities in the privatisation process and resistance to any change to status quo.
The fourth category of policy experiments is natural experiments. While studying from history, i.e., natural experiments in the water sector is helpful however their applicability as a ‘blueprint’ for similar outcomes in the future is limited for dealing with policy issues such as climate change that face a high degree of uncertainty.
The final type of policy experiments focuses on problems that are partly or wholly undefined.
Challenge of replication and expanding policy experiments
However, the real challenge in policy experiments (or pilot projects) is expanding, replicating, adapting and sustaining successful policies, programs or projects.
The empirical research on the composition of the ‘perfect pilot’ is not adequate; many factors are thought to influence pilot dynamics such as, the number and type of stakeholders involved, the choice of scale of implementation and the choice for pilot sites, the mode of governance that influences the nature of stakeholder engagement and learning, the level of innovativeness of the pilot and how it converges or diverges from the current policy context.
The discourse on policy experiments has changed from one whereby the design of a policy experiment was considered to be a technical process to one where it is shaped by the interests, behaviour and attitudes of the stakeholders. For many experts, scaling up a programme is considered more a craft than a science.
Our research examines experiences with scaling up of different types of policy experiments through an analysis of 15 pilot initiatives in multiple sectors, including water policy. Some countries that were covered included Ghana, China, Thailand and Bangladesh.
Our research specifically followed the Arntraud Hartmann and Johannes F. Linn framework and considered the following factors as necessary for scaling up of pilot projects.
- Presence of leaders driving the scaling up with a clear vision
- Scaling up is supported by political constituencies. This entails the active engagement of political players in the scaling up process and guarded against elite capture
- Presence of a supportive policy framework (laws, regulations, norms and linkages with related policies, programs and projects) for scaling up
- Strong institutional support and capacities to facilitate change. Scaling up requires adequate training, development and institutional capacity building
- Incentives for stakeholders and accountability for scaling up is essential to ensure that incentives are in sync with some shared objectives of the stakeholders
- Effective monitoring & evaluation
- A systematic and gradual process, careful planning, and clear demarcation of roles and responsibilities of partners and strong communication channels are important factors for scaling up
The findings show that the presence of political support is found to be necessary for scaling up in 97 percent of the cases studied, followed closely by the need for synergies with ongoing policies and programmes. When in combination with effective pilot planning and robust monitoring and evaluation, both these factors create a sufficient condition for successful scaling up in nearly 60 percent of the cases studied.
Indeed, policy experiments can be a valuable tool to design new policies, undertake policy transitions and allocate financial resources while deciding which policy initiatives to expand and what it needs in order to transition successfully into a new policy.
Sreeja Nair is a postdoctoral fellow, School of Social Sciences, Nanyang Technological University and a former PhD candidate, Lee Kuan Yew School of Public Policy, National University of Singapore. Michael Howlett is Professor, Burnaby Mountain Chair in the Department of Political Science at Simon Fraser University and Professor at the Lee Kuan Yew School of Public Policy.
This op-ed is based on the research paper Scaling up of Policy Experiments and Pilots: A Qualitative Comparative Analysis and Lessons for the Water Sector.