Trial data can’t tell us everything about how effective vaccines are: we need to wait to see the real world impact


Vaccinations against Covid-19 at Cwmbran Stadium, Wales.Vaccinations against Covid-19 at Cwmbran Stadium, Wales. Photograph: Matthew Horwood/Getty Images


The delivery of Covid-19 vaccines continues apace in Britain and around the world, and soon we will have a lot of data on their initial effectiveness. Vaccines are vital tools that will help to rescue us from the pandemic, and most people accept them as part of everyday life. But many people have differing opinions about how they should best be used, and there is concern that vaccines won’t be as effective against new variants. Indeed, one study (which has not yet been peer reviewed) suggests that the Oxford/AstraZeneca vaccine provides only minimal protection against illness caused by the South African strain of Covid-19, meaning people may need a third jab later in the year to protect them from other variants. In any case, we need to understand what vaccines are, how we measure how well they work, and what they can and cannot do.

1. Vaccines and the diseases they prevent are all different

Although there are many general principles we use to understand infections, each one is unique. They even change over time and vary from place to place. This creates a lot of uncertainty and means we constantly have to monitor what is going on. This work is called surveillance and without it, you can’t deploy vaccines effectively or adjust what you are doing to maximise their effects. This instability means that we should all expect advice to change frequently as new information arrives in the coming months about how best to use vaccines.

2. Vaccines don’t guarantee complete protection

Different people will have widely different responses and some people will remain susceptible. Vaccines usually prevent severe disease more efficiently than milder disease. In theory, this ought to make it easier to prove they work against serious illness but in many infections, Covid-19 included, serious cases are comparatively rarer and so larger studies run over longer periods of time would be needed to show how effective they were. In a pandemic you need results fast, so all the studies are measuring prevention of relatively mild but more common cases in order to get the numbers quickly. The efficacy estimates widely reported for the different vaccines from the phase 3 trials describe their ability to prevent relatively mild illness. So, if you receive a vaccine that is reported to be, say, 80% efficacious against symptomatic disease, your chance of getting really seriously ill should go down by more than 80%, possibly close to 100%, although no vaccine ever hits 100% guaranteed protection against anything.

3. You can’t compare the numbers from different trials

Each trial will involve different people in different places, and the definitions used to decide whether they have the disease or not are different too. Only when two vaccines are compared “head-to-head” in the same trial can you accurately judge if one is working better than another, and this is rarely done. It’s really hard to resist making comparisons between vaccines from the results of different phase 3 trials but such comparisons are misleading.

4. You can’t compare the results of randomised trials with the results of ‘real-world’ studies

Trials use mostly young healthy people, less prone to fall ill for different reasons, making the trials run more smoothly and quickly. They volunteer and are then screened and selected before receiving a vaccine or placebo control. Once vaccines are in general use, the people who accept and receive them are not screened in the same way. With Covid-19 vaccines, the people receiving them first will mostly be much older than those in the studies – and older people generally have weaker immune responses and are more prone to getting seriously ill from viral infections. In trials, chance decides who gets the vaccine and who doesn’t. In the real world, many other non-random factors drive what happens, so results comparing people who are vaccinated with others who aren’t can be biased. For all these reasons, estimates of real-world effectiveness generally come out a lot lower than trial efficacy estimates. The two cannot be compared because they are measuring different things in entirely different ways. When effectiveness figures come through for the different vaccines, we expect them to be lower than the efficacy numbers reported beforehand – but even so, we can still feel pleased about the numbers.

5. Most (but not all) vaccines reduce transmission, to create ‘herd immunity’

This really matters because it provides cover for the many imperfections of mass vaccination programmes. Some people don’t get the vaccine – either it doesn’t reach them or they refuse it. Some people get the vaccine and it doesn’t work or work well enough. The protection from the vaccine usually wears off over time, faster in some than others. All those things add up to leave you with a problem. But if vaccinated people – or at least some of them – really don’t get the infection at all, or if they get it more mildly and are less infectious to others – then the “R” value of the infection in the population can be driven down below 1, and the infection will eventually die out. In this way all the people without immunity are indirectly protected by the programme. This is why we are anxiously waiting to find out how well Covid-19 vaccines do this and whether some do it better than others.

The bottom line here is that we need to see the bigger picture to understand how vaccines really work. Thinking about Covid-19 vaccines at the level of individuals, particular occupations, localities or even whole countries obscures how immunisation affects entire populations.

When your vaccine supplies are limited, accurately targeting the doses you have towards those most likely to get seriously sick doesn’t just benefit them, it benefits everyone. By bringing down the number of dangerously ill people as fast as possible, we can relieve the pressure on hospitals, which is a threat to us all. The efficacy percentages we are now reading about from the reported trials of the different vaccines are not by themselves good predictors of the impact these vaccines will have as they become more and more widely used. That will depend much more on how soon and how well they work in the real world, in the high-risk people receiving them, and on how quickly and comprehensively we deliver the doses we have.

But the most important question is how well the vaccines reduce transmission. If they can do that reasonably well, and we should be optimistic that they will, then getting people across the world immunised in sufficient numbers to achieve herd immunity really does become a pathway back to normality.

  • Adam Finn is professor of paediatrics at the Bristol Children’s Vaccine Centre, University of Bristol