1 INTRODUCTION

Vaccines have been an important topic of late. Our public health strategy to deal with the COVID-19 pandemic is highly reliant not only on the effectiveness of those vaccines but also on how the benefits and harms are communicated to the public. Communicating the effects of vaccines to the public presents several challenges. For example, the benefits and harms are typically only known in a probabilistic sense—statistical information can be presented in several ways, each of which can have different effects on how it is interpreted. This point is something that has been well established in the literature but seems to have been overlooked in much of the public discourse on vaccine effectiveness and communication to the public (as we will discuss later). Furthermore, the decision to be vaccinated in the current pandemic may be in a large part informed through public channels rather than direct discussion with a trusted healthcare professional.

Olliaro et al.¹ suggest that ‘fully understanding the efficacy and effectiveness of vaccines is less straightforward than it may seem’ (p. 1). It is likely, in part, for that reason that communication of trial and post surveillance study results to the public were not always clear and understandable, and often misleading. This is reflected in several tools, such as decision aids, information sheets and statements to the media, that were used in delivering the message to the public. In an era where patients are encouraged to actively participate in treatment decisions (e.g., whether to get the vaccine or not), providing clear, easy to understand, and correct information is a necessary condition for an informed decision-making process. Much can be learned by taking some time to reflect on some examples of how vaccine benefits and harms have been communicated to the public. Let us begin by highlighting what the trials show and how that is presented in published studies.

2 PRESENTATION OF VACCINE EFFECTS IN CLINICAL TRIALS: WHAT THEY MEAN AND DO NOT MEAN

The objective of vaccine trials was to estimate their effects against COVID-19 infection. For the AstraZeneca vaccine, ‘efficacy was calculated as 1—adjusted relative risk’² (p. 103), which yielded an estimate of 70.4% in the initial interim analysis. The Pfizer/BioNTech and Moderna vaccine trials used a similar approach. For example, Polock et al.³ report an effect estimate of 95% for the Pfizer/BioNTech vaccine, where ‘vaccine efficacy was estimated by 100× (1-IRR), where IRR is the calculated ratio of confirmed cases of Covid-19 illness per 1000 person-years of follow-up in the active vaccine group to the corresponding illness rate in the placebo group’ (p. 2605). For the Moderna vaccine, ‘vaccine efficacy was defined as the percentage reduction in the hazard ratios for the primary end point’⁴ (p. 406), and was estimated at 94.1% in the trial. Immediately, we encounter a problem of communication. Vaccine trials estimate ‘vaccine efficacy’. However, in real-world practice, we speak of effectiveness, which cannot be assumed from the trial results for any individual or population that might acquire the vaccine.⁵

While trial estimates of vaccine efficacy are important, their interpretation and presentation at the individual and community level is potentially unclear to the audience and can be misleading. One reason is different ways of expressing the effect estimate can give different impressions of the benefits and harms.¹ Vaccine trials describe reductions in relative risk of infection, which is a metric that is likely not understood among those untrained in epidemiology or biostatistics, and is not sensitive to the baseline risk.⁶ Stegenga and Kenna⁷ have argued that the use of relative measures ‘contributes to pervaisive misinterpretations of data from medical research’ (p. 40). To illustrate this point, consider a population where baseline infection risk is 50%. Suppose a vaccine reduced infection risk to 40%. In that case, the reduction in absolute risk of infection (ARR) is 10%, while the reduction in relative risk (RRR) is calculated at 20% (ARR is the difference in risk [0.5–0.4 = 0.1] and the RRR is percent reduction in the risk in the experiment group compared to the control group [1–0.4/0.5 = 0.2]). It is important to note that the 20% reduction does not mean that for every 100 people who take the vaccine, 20 will not get the virus—the reduction does not refer to the whole population in the trial, only those who were exposed in the trial (in contrast, ARR considers the whole population).¹ Now suppose the baseline infection risk was 5%. If the vaccine reduced that risk of infection to 4%, the ARR is now 1%, while the RRR is still 20%. The benefit to the population is not the same in both scenarios, a fact that is not captured by the RRR. Furthermore, as Naylor et al.⁸ have shown, how the effect is reported can also shape one's (i.e., clinicians in their study) view. That is, ‘effects were perceived to increase when treatment data were summarized using percentage reductions in relative as opposed to absolute rates’⁸ (p. 918).

The vaccine trial reports mentioned above include absolute event counts, either in a table, figure, or text, from which the ARR can be calculated. However, formal calculation and/or explicit mention of that is absent in each report. Given the low baseline risk of infection, the calculated ARR for the Pfizer/BioNTech, Moderna, and AstraZeneca vaccine candidates are 0.7%, 1.2%, and 1.2%, respectively. Remarkably, these values are all much lower than the reported RRR.¹⁰ As suggested above, there is reason to believe that reporting these values, rather than the relative values, would give the reader a less optimistic impression of vaccine benefit. It is important to note that our goal is not to challenge the use of RRR in vaccine trials or communication with the public, nor are we suggesting the vaccines are less effective than believed by the public health community. We are only illustrating the challenges in communicating such information to the public who are making decisions about the vaccine. Again, as already illustrated by Naylor et al.,⁸ clinicians (who are often asked to explain these findings to their patients and the public) have difficulties understanding these metrics. Hence, how should we expect that members of the general public will understand them?

Vaccines have the potential to help the individual directly, in that acquiring the vaccine prevents infection and/or illness from infection for that person. Vaccines also have the potential to help individuals indirectly, whereby the acquisition of the vaccine by others reduces the opportunity for the individual to become infected. Clinical trials present the proportion of participants that experienced some measured event. From a frequentist statistics perspective, which underwrites effect estimates in the presented vaccine trials, proportions are thought to approximate probabilities (in large populations identical to trial participants) of that event. That might be useful when inferring, from study data, what might happen in a population if the intervention were to be used, assuming important differences, for example, in both the context of care and features of the population (known and unknown) are not present (i.e., the study sample is representative). Aggregate estimates of effect derived from trials (or posttrial surveillance studies) do not apply in the same way to an individual as they do to populations. That is, one cannot know from the clinical trial data, before accepting the vaccine, if she will be among those who will directly (as opposed to indirectly) benefit or those who will not, or what is her probability to benefit or not.⁵

Consider that for the individual accepting the vaccine, there are, for simplicity, four possible outcomes: (1) infection prevented and no side effects, (2) infection prevented and side effects, (3) infection not prevented and no side effects, and (4) infection not prevented and side effects. Eschewing the vaccine also presents several possible outcomes: (1) no infection and no side effects and (2) infection not prevented and no side effects. Again, the individual cannot choose from among the possible outcomes, nor can one predict (i.e., even probabilistically from the trials results and/or post surveillance studies) which outcome will occur for her depending on her choice to be vaccinated or not. Thus, it is possible that the individual can choose to not accept the vaccine and still achieve a desirable outcome (i.e., never being infected and no side effects). Likewise, in both options (vaccine vs. no vaccine), the individual does not necessarily need to have a side effect to obtain a benefit. In other words, a desirable outcome can be obtained that does not result in a ‘trade-off’ between the potential protection from the disease and potential side effect from the intervention.¹¹ Given these difficulties, it is important to assess how the information reported in the vaccine trials and post surveillance studies is being communicated to individuals (the apparent target of the public communication) about their own potential benefits and harms.

3 PRESENTATION OF VACCINE EFFECTS TO THE PUBLIC: SOME EXAMPLES FOR PROBLEMS IN COMMUNICATION

In many jurisdictions, COVID-19 vaccines can be acquired without prescription, and information regarding benefits and harms is often acquired through public channels, rather than in consultation with a clinician. Communication by public health and government officials, clinicians, and so forth, has leveraged the Internet and popular/social media. The message has been presented in the form of decision aids, infographics, press conferences, and media statements. As we will show, the information presented to the public in many cases included a description of what was done in the trial, including the effect estimate. However, the way this information was conveyed to the public was at best not clear and easy to understand, and in some cases misleading. Clear and understandable, for example, means that the public will know what is meant by ‘vaccine efficacy’, what is meant by a risk reduction, and that it is measured in relative terms (i.e., RRR). By misleading, we mean providing the wrong interpretation of the information. Furthermore, much of the communication is framed around ‘personal risk’ (i.e., the risk to a specific individual), which, as mentioned above, cannot be informed from vaccine trial data. Let us look at a few examples to illustrate some issues with communication.

Decision aids and information sheets are widely used tools to communicate the potential benefits and harms of healthcare interventions (and when known, their probabilities). A brief look at some examples of decision aids and information sheets developed for COVID-19 vaccines highlights some of the concerns we raised above. Consider the information sheet on COVID-19 vaccination and blood clotting presented by Public Health England/National Healh Service (NHS).¹² Risk of harm from COVID-19 and from vaccination are presented using nonnumerical terms, such as ‘low’, ‘very high’, ‘uncommon’ and so forth. One cannot assume that all these terms are understood and interpreted in the same way by individuals (e.g., does 20% fall under ‘low’, ‘medum’ or ‘high’ risk?). For those ‘risks’ that are ‘extremely rare’, a probability (i.e., population frequency) is presented (e.g., ‘clotting problems [around 1 in 100,000 first doses]’). The benefit of vaccination is presented, for example, as ‘one dose more than 80% reduction’ in ‘deaths, hospitalization, intensive care’ (The 80% is the RRR measured in the trial). Note that the potential benefits are described using RRR and the potential harms are described using absolute risk (i.e., population frequency). We will discuss the problem with such presentation a bit later. Importantly, the document is framed around ‘protecting you’ and is aimed at providing ‘information about your vaccination’, which might imply that the presented information pertains to individual probability—something that trial data and post surveillance studies cannot provide. Also important is that the baseline risk of infection is not presented.

When discussing the benefits of the vaccine relative to the potential harm of a blood clot for people aged 40 or older or with underyling medical conditions, the NHS information sheet¹² states that ‘the benefits of vaccination in protecting you against the serious consequences of COVID-19 outweigh any risk of this extremely rare condition’. Again, trials do not provide information about the individual probability of benefits or harms. Furthermore, as we explained earlier, because the individual cannot know (even probabilistically) or choose the outcomes resulting from taking the vaccine (or not), there is no ‘trade-off’ between the increased benefit of protection against infection from being vaccinated and harms (e.g., blood clot) stemming from vaccination at the individual level. As such, it cannot be known whether the benefits outweigh the harms for the individual, making the statement misleading for the individual. The statement is at best about populations, which is what the trial data provides for the sample population that participated in the trial. It may be the case that the benefits of wide uptake of the vaccine for a population outweigh the harms to the population, in which case that statement would be appropriate information to include in communication about the need for public uptake—not for a document about making a personal decision about accepting the vaccine, which is what the language in the information sheet implies.

Also consider the decision aid for vaccination during pregnancy developed by the Ontario Ministry of Health.¹³ Again, the risks are framed around the individual. For example, one of the prompts to frame the decision process asks the reader to ‘think about your own personal risk’. When providing information on ‘the benefits of getting the COVID-19 vaccine’, the document states that ‘vaccines are 94%–95% effective (e.g., for every 100 people who get the vaccine, 94 to 95 will not get a COVID-19 infection)’. This is a misrepresentation of the RRR presented in the trials, even at a population level. The decision aid uses the RRR estimate presented in the Moderna and Pfizer/BioNTech vaccine trials, respectively, and provided an interpretation of them as if they were a probability for a group or the individual. Again, the RRR does not tell us what is the actual reduction in the risk of infection. That is described by the ARR. Nor does the RRR describe what is the probability of being infected with or without a vaccine. The interpretation of the RRR as a probability as described in the bracketed part of the statement also misrepresents what was learned from the trial. Consider the Moderna vaccine trial, where 185/14,073 (1.3%) and 11/14,134 (0.08%) tested positive for COVID-19 in the control and vaccine groups, respectively. Based on those numbers, for every 100 people not vaccinated in the trial, 98.7 did not get a COVID-19 infection. If we consider that observation in light of a claim that for every 100 people vaccinated 94 will not be infected, one might incorrectly conclude that getting the vaccine increases the number of infected by 4.7 cases per 100 people! The numbers reported reflect the fact that only a small proportion of people in the trial were exposed to the virus. The trial does not provide sufficient information to extrapolate what would be the benefit if the entire population were to be exposed—those exposed in the trial might be different in important ways compared to those not exposed (or in the general public), and there is no guarantee everyone can be exposed (e.g., herd immunity might develop once a critical mass of people are exposed, some people might take themselves out of the population to avoid exposure, etc.). Providing the baseline risk of infection might give important context to help avoid such errors of interpretation but does so at the risk of giving a less optimistic impression of effectiveness, which may undermine the uptake of an important public health intervention.

The issues we raise are not limited to decision aids and information sheets—they also surface in popular media (a common source of information for much of the public deciding on if to acquire a vaccine). Stakeholders have used media to communicate information about the vaccines. A brief look at some media reports will illustrate confusion in the interpretation of reported trial data. Consider a 9 March 2021, article discussing why Canadians should get the first vaccine offered to them rather than wait for one where trials reported a higher efficacy rate, reported by the Canadian Broadcasting Corporation (Canada's publicly funded broadcaster).¹⁴ The article reports that ‘Pfizer-BioNTech and Moderna both have been determined by Health Canada to have efficacy rates of around 95 per cent. AstraZeneca/Oxford has an efficacy rate of 62 per cent’. The meaning of ‘efficacy rates of around 95 per cent’ is not explained. No context is given regarding baseline risk of infection, nor is it stated that these are relative measures. An infectious disease physician is quoted as saying ‘you don't want to be that person with zero per cent protection going into COVID-19 when you could be someone with at least 60 to 70 percent protection, if not higher’ (presumably referring to the benefit of acquiring the AstraZeneca vaccine rather than waiting for one of the other candidates). Here, we see again an interpretation of the reported vaccine effect estimate as an individual probability. Later in the article, we are provided some hints at these values as being something different than an individual probability. For example, the article reports a claim by the Chief Medical Advisor for Health Canada that ‘a vaccination with a vaccine with 66 per cent efficacy does not mean a person will have a 34 per cent chance of contracting COVID-19’. Unfortunately, the article does not provide information about how we should interpret these ‘efficacy rates’ beyond a plea to the public to get vaccinated with whichever approved vaccine is offered first. What these numbers mean for the individual weighing the potential benefits and harms in deciding to acquire (or not) the vaccine (and which one) is not clear, nor is it clear what impact they will have on her interpretation of the probability of an outcome.

Also noteworthy is the language used when comparing different vaccines. Consider in the previous quote the point about 95% efficacy for the Moderna and Pfizer/BioNTech vaccines, and 62% efficacy for the AstraZeneca vaccine. The difference in efficacy sounds impressive. However, owing to the higher baseline rate of infection in the trial, the ARR for the AstraZeneca vaccine is similar to that reported in the trials for the other two vaccines. Failure to provide context for the RRR estimate might give the public an impression that those getting the AstraZeneca vaccine are accepting an inferior product, which cannot be determined from comparing the trials' data.

Finally, there seems to be an asymmetry in the way benefits and harms are reported and communicated. As we have shown, benefits are reported using relative metrics, which can give a more optimistic impression when compared to absolute metrics. Harms, on the other hand, are shown using absolute metrics. Consider again the NHS information sheet discussed above, which states ‘clotting problems (around 1 in 100,000 first doses)’.¹² Also consider a report on the risk of Vaccine-Induced Thrombotic Thrombocytopenia (VITT) related to the AstraZeneca vaccine that was produced by the Ontario Science Table, a group of scientists that provides advice on COVID-19 to the government of the largest province in Canada.¹⁵ That report outlines the absolute number of cases of VITT among people receiving the vaccine as well as the incidence of the clot (which they calculated as 1 in 55,000). How is such information on the rate of VITT or clot reconciled with claims that ‘one dose more than 80% reduction’ in ‘deaths, hospitalization, intensive care’, as stated in the NHS information sheet,¹² or the 62% efficacy of the AstraZeneca vaccine that was reported by government officials in a news article (mentioned earlier)? The incidence of a harm and RRR are not directly comparable to each other without additional calculation using the frequency data. Asking the individual to (i) know the difference between them (i.e., that they are comparing apples and oranges) and (ii) compare them when making a decision does not seem to be a reasonable expectation.

4 COMMUNICATION IS A COMPLEX TASK

It is not our intention to persuade the reader on the potential benefits and harms of the vaccines and whether one should (or not) be vaccinated. We simply seek to highlight that the way one presents the benefits and harms of the vaccine might have an impact on the choice of an individual to acquire the vaccine and how they act once it is acquired. If the goal of public health and government officials, physicians, and so forth, is to get people to accept the vaccine (which might be suggested in some of the statements in the examples we provide above), then it is vitally important that the information is not misleading and that purpose is clearly communicated. Otherwise, there is chance that the attempt to persuade may backfire when individuals find out that the information presented was not accurate and/or was presented in a potentially biased way.

In responding to the pandemic, the public is often in a position where they must rely on what they are told by experts. Those experts must be careful not to assume that the public knows the meaning of terms used in the trials (e.g., RRR, vaccine efficacy), nor can we assume the public has access to the trial data, even if they had the expertise to interpret the findings. It is also important that the experts are correct in their own understanding—in our examples, we see cases where even the people who are trained on how to interpret trial results ran into difficulty in translating that in a way that the general public might understand without misrepresenting what is learned from the trials.

Communication of evidence from clinical trials is a complex task and to the best of our knowledge there are no simple solutions. Unfortunately, we cannot provide definitive solutions—we only describe the problem. In our opinion recognizing the problem(s) is the first step in the long road to finding solutions. Ignoring the problem is a prescription to falling into the trap described by Bernard Shaw.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.