National Institute for Health Research

back to blog

Research Design Service Blog


Sample size - what difference matters?

Sample size - what difference matters?

01 November 2019

Nick Taub, Senior Adviser in Medical Statistics, RDS East Midlands

Suppose that you want to evaluate a healthcare intervention, and are designing a quantitative research study – such as a population survey or a follow-up cohort study or a clinical trial. You want to be as sure as reasonably possible that you are recruiting a large enough number of patients (or healthy people, or community centres, or whatever ‘unit’ you are studying) to answer your question well. The funding panel and their reviewers will also want to be sure! 

There are only a few exceptions to this, such as some feasibility studies and purely exploratory studies where little or no previous work has been done – and then you would justify the size of your study more informally.

The Research Question
Start by writing down your main research question, for example: ‘Following an acute cardiac event, does cardiac rehabilitation lead to an increase in physical activity, compared with usual care?’
This question will then be focussed so that it applies to the specific clinical situation to be studied. The PICO framework is a good way of doing this. 

Next you need to decide the ‘primary outcome measure’ to assess patient improvement. In this example it would be around the physical activity – for example, the number of minutes of physical activity that the patient carries out, on average, each week.
You will also need to determine what method of measurement will you use to collect this data:  a patient diary, a pedometer or maybe an accelerometer? 
How long will the patient be followed up before the principle assessment is made? In this example it might be 8 weeks after the end of a 6-week period of rehabilitation therapy.

Sample Size Calculations
The RDS EM Newsletter - Spring 2017 explains sample size calculations in greater detail and where you can go for further help and information. 
Now you are nearly ready to carry out the calculation – and it is best to consult with RDS staff or your own team’s statistician to help you with this process.
One of the key ingredients will be: The Minimum Clinically Important Difference (MCID) also known as the ‘effect size’. In this example - How many extra minutes per week of physical activity, averaged over patients, in the treatment group compared with the control group, counts as a success for the treatment?

The following are both crucial criteria
‘Can I believe that?’
The MCID must be a small enough average improvement to be plausible. 
In our example if patients without cardiac rehabilitation generally averaged around 90 minutes per week physical activity and you claimed that rehabilitation would bring this up to 3 hours, you would need to provide evidence to justify such a dramatic improvement.

Sources of evidence: 

  • Previous published studies, using similar treatments with similar patients (needn’t be exactly the same) – even better are systematic reviews, such as those in the Cochrane Library.
  • RDS Information Officers can offer advice on which databases to search to find the evidence.
  • A feasibility or pilot study is unlikely to be large enough to give useful evidence by itself, although it can help make the argument.
  • Data from well-maintained clinical or research databases.

and also 
‘So What?’
The MCID must also be a large enough average improvement to persuade clinicians and – even more important - the commissioners of healthcare that it is worth making the change in standard clinical practice.
If you were a commissioner of NHS services, would you be interested in an extra 10 minutes average physical activity per week by a patient, or 20 minutes, or 30 minutes?

This is a question of judgement which may take into account:

  • What this specific average improvement is likely to represent, in broader terms, for the patients’ benefit.
  • How common and how severe adverse events are.
  • The cost in money terms, and also logistic factors involved in introducing the new intervention - these might include the provision of staff training, medical equipment, floor space in wards or hospital outpatient areas, and so on.
  • RDS staff won’t usually have the medical and NHS management expertise to answer these questions, but we can
    • Help you find good clinical and NHS opinions.
    • Discuss with you and review how well we think the argument has been made.

The Minimum Clinically Important Difference is a key aspect of any study that evaluates a healthcare intervention. It is not only a number needed (too often, at the last minute) to ‘plug into’ a sample size calculation. It encourages you, as a researcher, to think about and try to cautiously estimate the impact that the intervention, and therefore your research, can make.

For more detail, when designing a clinical trial, see the DELTA2 guidance in the BMJ (Cook JA, Julious SA, Sones W et al.  DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ 2018; 363:k3750. doi: 101136.bmj.k3750)

back to blog