If natural experiments don’t come naturally….
As RDS advisers we advise researchers on projects requiring a wide range of methods from RCTs to ethnography. The recent increased focus from NIHR on public health and social care research might have some of us turning to methodologies that we are less familiar with. Public health interventions, for example, are often best evaluated by natural experimental designs.

What is a natural experiment?
A natural experiment in the context of public health is the evaluation of an intervention or event that is outside the control of the research team. Variations in exposure to an ‘intervention’ or event offer an opportunity to evaluate the impact on health or related outcomes. Examples of interventions include a local authority providing free access to leisure facilities. We might also want to evaluate the impact of an event of interest, such as observed reductions in childhood vaccination rates. Events which are not directly related to health, may still have public health outcomes of interest. For example the 2007-2008 financial crisis which led to many people experiencing financial hardship, impacted on decisions about food purchases that could have adverse effects on health.

When is a natural experiment an appropriate method?
Natural experimental designs are most appropriate when there are ethical or practical reasons why the intervention can’t be evaluated using controlled experimental methods. For example the implementation of a clean air zone in a city. Their complexity and sometimes unpredictable nature can make these interventions difficult to evaluate with randomised controlled trials. A recent paper argued that natural experiments in public health should be embraced and not just tolerated.

Evaluating natural experiments
It can require creative thinking to evaluate a natural experiment robustly. Typically a combination of methods will be needed to ensure that any effects attributed to the intervention are truly a result of it and can’t be explained by other causes. Often a mixed methods approach will be needed and a focus on describing plausible pathways through which the intervention has exerted an effect.

Establishing the degree to which individuals have been exposed to the intervention/event can be difficult in natural experiments. It might be (relatively) straightforward to evaluate the impact of an intervention that is implemented on a known date and has a geographically wide effect, meaning that individuals have little influence over their level of exposure to the intervention. Where the effects of a natural experiment are likely to be more subtle and occur more gradually more sophisticated evaluation methods (for example interrupted time series) might be needed.

Another important consideration is the difficulty of accounting sufficiently for confounding factors in the analysis. In studies where participants are randomised to receive an exposure it is (often) assumed that confounding factors are taken care of by the randomisation procedure. In natural experiments this assumption cannot be made. To take the example of the introduction of a clean air zone in a city it is possible, or even likely, that there are systematic differences between groups of individuals who receive a greater exposure to the intervention and those that are less affected by it. Techniques exist to address this, for example the use of instrumental variables in the analysis.

Where can I find out more?

If your appetite for natural experiments has been whetted by this blog the MRC provide guidance on how to conduct them: https://mrc.ukri.org/documents/pdf/natural-experiments-guidance/

One final thing you should do: contact your local NIHR Research Design Service. We’re here to offer free and confidential advice on your research application and we can help natural experiments.