Resize text 
Resize text
16.2.1. Vaccine safety
16.2.1.1. General considerations
16.2.1.2. Signal detection and validation
16.2.1.3. Study designs for vaccine safety assessment
16.2.2. Vaccine effectiveness
16.2.2.1. General considerations
16.2.2.2. Sources of exposure and outcome data
16.2.2.3. Study designs for vaccine effectiveness assessment
16.2.3. Specific aspects of pharmacoepidemiological vaccine studies
16.2.3.1. Studies in special populations
16.2.3.2. Meta-analyses
16.2.3.3. Pharmacogenetic studies
16.2.3.4. Generic protocols
16.2.1.1. General considerations
The book Vaccination Programmes | Epidemiology, Monitoring, Evaluation (Hahné, S., Bollaerts, K., & Farrington, P., Routledge, 2021) is a comprehensive textbook addressing most of the concepts presented in this Chapter. For contents related to safety monitoring of vaccines, it further builds on the 2014 ADVANCE Report on appraisal of vaccine safety methods that described a wide range of direct and indirect methods for vaccine safety assessment. Specific aspects related to vaccine safety and effectiveness are discussed in several documents:
The Report of the CIOMS/WHO Working Group on Definition and Application of Terms for Vaccine Pharmacovigilance (2012) provides definitions and explanatory notes for the terms ‘vaccine pharmacovigilance’, ‘vaccination failure’ and ‘adverse event following immunisation (AEFI)’.
The Guide to active vaccine safety surveillance: Report of CIOMS working group on vaccine safety – executive summary (Vaccine 2017;35(32):3917-21) describes the process for selecting the best approach to active surveillance considering key implementation issues, including in resource-limited countries.
The CIOMS Guide to Vaccine Safety Communication (2018) addresses vaccine safety communication aspects for regulators, vaccination policy-makers, and other stakeholders, when introducing vaccines in populations, based on selected examples.
The Brighton Collaboration provide a resource to facilitate and harmonise collection, analysis, and presentation of vaccine safety data, including case definitions for outcomes of interest, including adverse events of special interest (AESIs).
Module 4 (Surveillance) of the e-learning training course Vaccine Safety Basics of the World Health Organization (WHO) describes pharmacovigilance principles, causality assessment procedures, surveillance systems, and places safety in the context of the vaccine benefit/risk profile.
Recommendations on vaccine-specific aspects of the EU Pharmacovigilance System, including on risk management, signal detection and post-authorisation safety studies (PASS) are presented in Module P.I: Vaccines for prophylaxis against infectious diseases (EMA, 2013) of the Good Pharmacovigilance Practices (GVP).
The WHO Covid-19 vaccines: safety surveillance manual (WHO, 2020) was developed upon recommendation of the WHO Global Advisory Committee on Vaccine Safety (GACVS) and describes categories of surveillance strategies: passive, active, cohort event monitoring, and sentinel surveillance. While developed for COVID-19 vaccines, it can be used to guide pandemic preparedness activities for the monitoring of novel vaccines.
A vaccine study design selection framework for the postlicensure rapid immunization safety monitoring program (Am J Epidemiol. 2015;181(8):608-18) describes in a tabular form strengths and weaknesses of study designs and can be broadly applied to vaccine research questions beyond safety assessment.
16.2.1.2. Signal detection and validation
Besides a qualitative analysis of spontaneous case reports or case series, quantitative methods such as disproportionality analyses (described in Chapter 11) and observed-to-expected (O/E) analyses are routinely employed in signal detection and validation for vaccines. Several documents discuss the merits and review the methods of these approaches for vaccines.
Disproportionality analyses
GVP Module P.I: Vaccines for prophylaxis against infectious diseases describes aspects to be considered when applying methods for vaccine disproportionality analyses, including choice of the comparator group and use of stratification. Effects of stratification on data mining in the US Vaccine Adverse Event Reporting System (VAERS) (Drug Saf. 2008;31(8):667-74) demonstrates that stratification can reveal and reduce confounding and unmask some vaccine-event pairs not found by crude analyses. However, Stratification for Spontaneous Report Databases (Drug Saf. 2008;31(11):1049-52) highlights that extensive use of stratification in signal detection algorithms should be avoided, as it can mask true signals. Vaccine-Based Subgroup Analysis in VigiBase: Effect on Sensitivity in Paediatric Signal Detection (Drug Saf. 2012;35(4):335-46) further examines the effects of subgroup analyses based on the relative distribution of vaccine/non-vaccine reports in paediatric adverse drug reaction data (ADR) data. In Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases (Drug Saf. 2016;39(4):355-64), subgrouping by vaccines/non-vaccines resulted in a decrease in both precision and sensitivity in all spontaneous report databases that contributed data.
Optimization of a quantitative signal detection algorithm for spontaneous reports of adverse events post immunization (Pharmacoepidemiol Drug Saf. 2013;22(5): 477–87) explores various ways of improving performance of signal detection algorithms for vaccines.
Adverse events associated with pandemic influenza vaccines: comparison of the results of a follow-up study with those coming from spontaneous reporting (Vaccine 2011;29(3):519-22) reported a more complete pattern of reactions when using two complementary methods for first characterisation of the post-marketing safety profile of a new vaccine, which may impact on signal detection.
In Review of the initial post-marketing safety surveillance for the recombinant zoster vaccine (Vaccine 2020;38(18):3489-500), the time-to-onset distribution of zoster vaccine-adverse event pairs was used to generate a quantitative signal of unexpected temporal relationship.
Bayesian disproportionality methods have also been developed to generate disproportionality signals. In Association of Facial Paralysis With mRNA COVID-19 Vaccines: A Disproportionality Analysis Using the World Health Organization Pharmacovigilance Database (JAMA Intern Med. 2021;e212219), a potential safety signal for facial paralysis was explored using the Bayesian neural network method.
In Disproportionality analysis of anaphylactic reactions after vaccination with messenger RNA coronavirus disease 2019 vaccines in the United States (Ann Allergy Asthma Immunol. 2021; S1081-1206(21)00267-2) the CDC Wide-ranging Online Data for Epidemiologic Research (CDC WONDER) system was used in conjunction with proportional reporting ratios to evaluate whether rates of anaphylaxis cases reported in the VAERS database following administration of mRNA COVID-19 vaccines is disproportionately different from all other vaccines.
Signaling COVID-19 Vaccine Adverse Events (Drug Saf. 2022 Jun 23:1–16) discusses the extent, direction, impact, and causes of masking, an issue associated with signal detection methodologies, in which signals for a product of interest are hidden by the presence of other reported products, which may limit the understanding of the risks associated with COVID-19 and other vaccines, and delay their identification.
Observed-to-expected analyses and background incidence rates
In vaccine vigilance, O/E analyses compare the ‘observed’ number of cases of an adverse event occurring in vaccinated individuals and recorded in a data collection system (e.g. a spontaneous reporting system or an electronic healthcare database) and the ‘expected’ number of cases that would have naturally occurred in the same population without vaccination, estimated from available incidence rates in a non-vaccinated population. O/E analyses constitute a first step in the continuum of safety signal evaluation, and can guide further steps such as a formal pharmacoepidemiological study. GVP Module P.I: Vaccines for prophylaxis against infectious diseases (EMA, 2013) suggests conducting O/E analyses for signal validation and preliminary signal evaluation when prompt decision-making is required, and there is insufficient time to review a large number of individual cases. It discusses key requirements of O/E analyses: an observed number of cases detected in a passive or active surveillance system, near real-time exposure data, appropriately stratified background incidence rates calculated on a population similar to the vaccinated population (for the expected number of cases), the definition of appropriate risk periods (where there is suspicion and/or biological plausibility that there is a vaccine‐associated increased risk of the event) and sensitivity analyses around these measures. O/E analyses may require some adjustments for continuous monitoring due to inflation of type 1 error rates when multiple tests are performed. The method is further discussed in Pharmacoepidemiological considerations in observed‐to‐expected analyses for vaccines (Pharmacoepidemiol Drug Saf. 2016;25(2):215-22) and the review Near real‐time vaccine safety surveillance using electronic health records - a systematic review of the application of statistical methods (Pharmacoepidemiol Drug Saf. 2016;25(3):225-37).
O/E analyses require several pre-defined assumptions based on the requirements listed above. Each of these assumptions can be associated with uncertainties. How to manage these uncertainties is also addressed in Pharmacoepidemiological considerations in observed-to-expected analyses for vaccines (Pharmacoepidemiol Drug Saf. 2016;25(2):215–22). Observed-over-Expected analysis as additional method for pharmacovigilance signal detection in large-scaled spontaneous adverse event reporting (Pharmacoepidemiol Drug Saf. 2023;32(7):783-794) uses two examples of events of interest (idiopathic peripheral facial paralysis and Bell's palsy) in the context of the COVID-19 immunisation campaigns, when very large numbers of case safety reports (ICSRs) had to be timely handled.
Use of population based background rates of disease to assess vaccine safety in childhood and mass immunisation in Denmark: nationwide population based cohort study (BMJ. 2012;345:e5823) illustrates the importance of collecting background rates by estimating risks of coincident associations of emergency consultations, hospitalisations and outpatients consultations, with vaccination. Rates of selected disease events for several countries may vary by age, sex, method of ascertainment, and geography, as shown in Incidence Rates of Autoimmune Diseases in European Healthcare Databases: A Contribution of the ADVANCE Project (Drug Saf. 2021;44(3):383-95), where age-, gender-, and calendar-year stratified incidence rates of nine autoimmune diseases in seven European healthcare databases from four countries were generated to support O/E analyses. Guillain-Barré syndrome and influenza vaccines: A meta-analysis (Vaccine 2015; 33(31):3773-8) suggests that a trend observed between different geographical areas would be consistent with a different susceptibility of developing a particular adverse reaction among different populations. In addition, comparisons with background rates may be invalid if conditions are unmasked at vaccination visits (see Human papillomavirus vaccination of adult women and risk of autoimmune and neurological diseases, J Intern Med. 2018;283:154-65)).
Several studies have generated background incidence rates of AESIs for COVID-19 vaccines and discuss methodological challenges related to identifying AESIs in electronic health records (EHRs) (see The critical role of background rates of possible adverse events in the assessment of COVID-19 vaccine safety, Vaccine 2021;39(19):2712-18).
In Arterial events, venous thromboembolism, thrombocytopenia, and bleeding after vaccination with Oxford-AstraZeneca ChAdOx1-S in Denmark and Norway: population based cohort study (BMJ. 2021;373:n1114), observed age- and sex-specific rates of events among vaccinated people were compared with expected rates in the general population calculated from the same databases, thereby removing a source of variability between observed and expected rates. Where this is not possible, rates from multiple data sources have shown to be heterogeneous, and the choice of relevant data should take into account differences in database and population characteristics related to different diagnoses, recording and coding practices, source populations (e.g., inclusion of subjects from general practitioners and/or hospitals), healthcare systems, and linkage ability (e.g., to hospital records). This is further discussed in Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: multinational network cohort study (BMJ. 2021;373:n1435) and Background rates of five thrombosis with thrombocytopenia syndromes of special interest for COVID-19 vaccine safety surveillance: Incidence between 2017 and 2019 and patient profiles from 38.6 million people in six European countries (Pharmacoepidemiol Drug Saf. 2022;31(5):495-510).
Contextualising adverse events of special interest to characterise the baseline incidence rates in 24 million patients with COVID-19 across 26 databases: a multinational retrospective cohort study (EClinicalMedicine. 2023;58:101932) used data from primary care, electronic health records, and insurance claims mapped to a common data model to characterise the incidence rates of AESIs, also following SARS-CoV-2 infection (considered a confounder), compared them to historical rates in the general population, and addressed issues of heterogeneity.
Historical comparator designs may generate false positives, as discussed in Bias, Precision and Timeliness of Historical (Background) Rate Comparison Methods for Vaccine Safety Monitoring: An Empirical Multi-Database Analysis (Front Pharmacol. 2021;12:773875), which explores the effect of empirical calibration on type 1 and 2 errors using outcomes presumed to be unrelated to vaccines (negative control outcomes) as well as positive controls (outcomes simulated to be caused by the vaccines).
Factors Influencing Background Incidence Rate Calculation: Systematic Empirical Evaluation Across an International Network of Observational Databases (Front Pharmacol. 2022;13:814198) examined the sensitivity of rates to the choice of design parameters using 12 data sources to systematically examine their influence on incidence rates using 15 AESIs for COVID-19 vaccines. Rates were highly influenced by the choice of the database (varying by up to a factor of 100), the choice of anchoring (e.g., health visit, vaccination, or arbitrary date) for the time-at-risk start, the choice of clean window and time-at-risk duration, but less so by secular or seasonal trends. It concluded that results should be interpreted in the context of study parameter choices.
Sequential methods
Sequential methods, as described in Early detection of adverse drug events within population-based health networks: application of sequential methods (Pharmacoepidemiol Drug Saf. 2007;16(12):1275-84), allow O/E analyses to be performed on a routine (e.g., weekly) basis using cumulative data with adjustment for multiplicity. Such methods are routinely used for near-real time surveillance in the Vaccine Safety Datalink (VSD) (see Near real-time surveillance for influenza vaccine safety: proof-of-concept in the Vaccine Safety Datalink Project, Am J Epidemiol 2010;171(2):177-88). Potential issues are described in Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data (Pharmacoepidemiol Drug Saf. 2012;21(S1):62-71). A review of signals detected over 3 years in the VSD concluded that care with data quality, outcome definitions, comparator groups, and duration of surveillance, is required to enable detection of true safety issues while controlling for error (Active surveillance for adverse events: the experience of the Vaccine Safety Datalink Project, Pediatrics 2011;127(S1):S54-S64).
A new self-controlled case series method for analyzing spontaneous reports of adverse events after vaccination (Am J Epidemiol. 2013;178(9):1496-504) extends the self-controlled case series approach (see Chapter 4.2.3, and 16.2.2.2 in this Chapter) to explore and quantify vaccine safety signals from spontaneous reports using different assumptions (e.g., considering large amount of underreporting, and variation of reporting with time since vaccination). The method should be seen as a signal strengthening approach for quickly exploring a signal prior to a pharmacoepidemiological study (see for example, Kawasaki disease and 13-valent pneumococcal conjugate vaccination among young children: A self-controlled risk interval and cohort study with null results, PLoS Med. 2019;16(7):e100284).
The tree-based scan statistic (TreeScan) is a statistical data mining method that can be used for the detection of vaccine safety signals from large health insurance claims and electronic health records (Drug safety data mining with a tree-based scan statistic, Pharmacoepidemiol Drug Saf. 2013;22(5):517-23). A Broad Safety Assessment of the 9-Valent Human Papillomavirus Vaccine (Am J Epidemiol. 2021;kwab022) and A broad assessment of covid-19 vaccine safety using tree-based data-mining in the vaccine safety datalink (Vaccine. 2023;41(3):826-835) used the self-controlled tree-temporal scan statistic which does not require pre-specified outcomes or specific post-exposure risk periods. The method requires further evaluation of its utility for routine vaccine surveillance in terms of requirements for large databases and computer resources, as well as predictive value of the signals detected.
16.2.1.3. Study designs for vaccine safety assessment
A complete review of vaccine safety study designs and methods for hypothesis-testing studies is included in the ADVANCE Report on appraisal of vaccine safety methods (2014) and in Part IV of the book Vaccination Programmes | Epidemiology, Monitoring, Evaluation (Hahné, S., Bollaerts, K., & Farrington, P., Routledge, 2021).
Current Approaches to Vaccine Safety Using Observational Data: A Rationale for the EUMAEUS (Evaluating Use of Methods for Adverse Events Under Surveillance-for Vaccines) Study Design (Front Pharmacol. 2022;13:837632) provides an overview of strengths and limitations of study designs for vaccine safety monitoring and discusses the assumptions made to mitigate bias in such studies.
Methodological frontiers in vaccine safety: qualifying available evidence for rare events, use of distributed data networks to monitor vaccine safety issues, and monitoring the safety of pregnancy interventions (BMJ Glob Health. 2021;6(Suppl 2):e003540) addresses multiple aspects of pharmacoepidemiological vaccine safety studies, including study designs.
Cohort and case-control studies
There is a large body of published literature reporting on the use of the cohort design (and to a lesser extent, the case-control design) for the assessment of vaccine safety. Aspects of these designs presented in Chapters 4.2.1 and 4.2.2 are applicable to vaccine studies (for the cohort design, see also the examples of studies on background incidence rates in paragraph 16.2.2.1 of this Chapter). A recent illustration of the cohort design is provided in Clinical outcomes of myocarditis after SARS-CoV-2 mRNA vaccination in four Nordic countries: population based cohort study (BMJ Med. 2023 Feb 1;2(1):e000373) which used nationwide register data to compare clinical outcomes of myocarditis associated with vaccination, with COVID-19 disease, and with conventional myocarditis, with respect to readmission to hospital, heart failure, and death, using the Kaplan-Meier estimator approach.
Cohort-event monitoring
Prospective cohort-event monitoring (CEM) including active surveillance of vaccinated subjects using smartphone applications and/or web-based tools has been extensively used to monitor the safety of COVID-19 vaccines, as primary data collection was the only means to rapidly identify safety concerns when the vaccines started to be used at large scale. A definition of cohort-event monitoring is provided in The safety of medicines in public health programmes : pharmacovigilance, an essential tool (WHO, 2006, Chapter 6.5, Cohort event monitoring, pp 40-41). Specialist Cohort Event Monitoring studies: a new study method for risk management in pharmacovigilance (Drug Saf. 2015;38(2):153-63) discusses the rationale and features to address possible bias, and some applications of this design. COVID-19 vaccine waning and effectiveness and side-effects of boosters: a prospective community study from the ZOE COVID Study (Lancet Infect Dis. 2022:S1473-3099(22)00146-3) is a longitudinal, prospective, community-based study to assess self-reported systemic and localised adverse reactions of COVID-19 booster doses, in addition to effectiveness against infection (a confounder). Self-reported data may introduce information bias, as some participants might be more likely to report symptoms and some may drop out; however, multi-country CEM studies allow to include large populations, as shown in Cohort Event Monitoring of Adverse Reactions to COVID-19 Vaccines in Seven European Countries: Pooled Results on First Dose (Drug Saf. 2023;46(4):391-404).
Case-only designs
Traditional designs such as the cohort and case-control designs (see Chapters 4.2.1 and 4.2.2) may be difficult to implement in circumstances of high vaccine coverage (for example, in mass immunisation campaigns such as for COVID-19), a lack of an appropriate comparator group (e.g., unvaccinated), or a lack of adequate covariate information at the individual level. Frequent sources of confounding are underlying health status and factors influencing the likelihood of being vaccinated, such as access to healthcare or belonging to a high-risk group (see paragraph 16.2.4.1 on Studies in special populations in this Chapter). In such situations, case-only designs may provide stronger evidence than large cohort studies as they control for fixed individual-level confounders (such as demographics, genetics, or social deprivation) and have similar, sometimes higher, power (see Control without separate controls: evaluation of vaccine safety using case-only methods, Vaccine 2004;22(15-16):2064-70). Case-only designs are discussed in Chapter 4.2.3.
Several publications have compared traditional and case-only study designs for vaccine studies:
Epidemiological designs for vaccine safety assessment: methods and pitfalls (Biologicals 2012;40(5):389-92) used three designs (cohort, case-control, and self-controlled case-series (SCCS)) to illustrate aspects such as case definition, limitations of data sources, uncontrolled confounding, and interpretation of findings.
Comparison of epidemiologic methods for active surveillance of vaccine safety (Vaccine 2008; 26(26):3341-45) performed simulations to compare four designs (matched cohort, vaccinated-only (risk interval) cohort, case-control, and SCCS). The cohort design allowed for the most rapid signal detection, less false-positive error and highest statistical power in sequential analyses. However, one limitation of this simulation was the lack of case validation.
The simulation study Four different study designs to evaluate vaccine safety were equally validated with contrasting limitations (J Clin Epidemiol. 2006; 59(8):808-18) compared four designs (cohort, case-control, risk-interval and SCCS) and concluded that all were valid, however, with contrasting strengths and weaknesses. The SCCS, in particular, proved to be an efficient and valid alternative to the cohort design.
Hepatitis B vaccination and first central nervous system demyelinating events: Reanalysis of a case-control study using the self-controlled case series method (Vaccine 2007;25(31):5938-43) describes how the SCCS found similar results as the case-control design but with greater precision, based on the assumption that exposures are independent of earlier events, and recommended that case-series analyses should be conducted in parallel to case-control analyses.
It is increasingly considered good practice to use combined approaches, such as a cohort design and sensitivity analyses using a self-controlled method, as this provides an opportunity for minimising some biases that cannot be taken into account in the primary design (see for example, Myocarditis and pericarditis associated with SARS-CoV-2 vaccines: A population-based descriptive cohort and a nested self-controlled risk interval study using electronic health care data from four European countries; Front Pharmacol. 2022;13:1038043).
While the SCCS is suited to secondary use of data, it may not always be appropriate in situations where rapid evidence generation is needed, since follow-up time needs to be accrued. In such instances, design approaches include the SCRI method that can be used to shorten observation time (see The risk of Guillain-Barre Syndrome associated with influenza A (H1N1) 2009 monovalent vaccine and 2009-2010 seasonal influenza vaccines: Results from self-controlled analyses, Pharmacoepidemiol. Drug Saf 2012;21(5):546-52; and Chapter 4.2.3); O/E analyses using historical background rates (see Near real-time surveillance for influenza vaccine safety: proof-of-concept in the Vaccine Safety Datalink Project, Am J Epidemiol 2010;171(2):177-88); or traditional case-control studies (see Guillain-Barré syndrome and adjuvanted pandemic influenza A (H1N1) 2009 vaccine: multinational case-control study in Europe, BMJ 2011;343:d3908).
Nevertheless, the SCCS design is an adequate method to study vaccine safety, provided the main requirements of the method are taken into account (see Chapter 4.2.3). An illustrative example is shown in Bell's palsy and influenza(H1N1)pdm09 containing vaccines: A self-controlled case series (PLoS One. 2017;12(5):e0175539). In First dose ChAdOx1 and BNT162b2 COVID-19 vaccinations and cerebral venous sinus thrombosis: A pooled self-controlled case series study of 11.6 million individuals in England, Scotland, and Wales (PLoS Med. 2022;19(2):e1003927), pooled primary care, secondary care, mortality, and virological data were used. The authors discuss the possibility that the SCCS assumption of event-independent exposure may not have been satisfied in the case of cerebral venous sinus thrombosis (CVST) since vaccination prioritised risk groups, which may have caused a selection effect where individuals more likely to have an event were less likely to be vaccinated and thus less likely to be included in the analyses. In First-dose ChAdOx1 and BNT162b2 COVID-19 vaccines and thrombocytopenic, thromboembolic and hemorrhagic events in Scotland (Nat Med. 2021; 27(7):1290-7), potential residual confounding by indication in the primary analysis (a nested case-control design) was addressed by a SCCS to adjust for time-invariant confounders. Risk of acute myocardial infarction and ischaemic stroke following COVID-19 in Sweden: a self-controlled case series and matched cohort study (Lancet 2021;398(10300):599-607) showed that a COVID-19 diagnosis is an independent risk factor for the events, using two complementary designs in Swedish healthcare data: a SCCS to calculate incidence rate ratios in temporal risk periods following COVID-19 onset, and a matched cohort study to compare risks within 2 weeks following COVID-19 to the risk in the background population.
A modified self-controlled case series method for event-dependent exposures and high event-related mortality, with application to COVID-19 vaccine safety (Stat Med. 2022;41(10):1735-50) used data from a study of the risk of cardiovascular events, together with simulated data, to illustrate how to handle event-dependent exposures and high event-related mortality, and proposes a newly developed test to determine whether a vaccine has the same effect (or lack of effect) at different doses.
Estimating the attributable risk
The attributable risk of a given safety outcome (assuming a causal effect attributable to vaccination) is an important estimate to support public health decision-making in the context of vaccination campaigns. In the population-based cohort study Investigation of an association between onset of narcolepsy and vaccination with pandemic influenza vaccine, Ireland April 2009-December 2010 (Euro Surveill. 2014;19(17):15-25), the relative risk was calculated as the ratio of the incidence rates for vaccinated and unvaccinated subjects, while the absolute attributable risk was calculated as the difference in incidence rates. Safety of COVID-19 vaccination and acute neurological events: A self-controlled case series in England using the OpenSAFELY platform (Vaccine. 2022;40(32):4479-4487) used primary care, hospital admission, emergency care, mortality, vaccination, and infection surveillance data linked through a dedicated data analytics platform, and calculated the absolute risk of selected AESIs.
Case-coverage design
The case-coverage design is a type of ecological design using exposure information on cases, and population data on vaccination coverage to serve as control. It compares odds of exposure in cases to odds of exposure in the general population, similar to the screening method used in vaccine effectiveness studies (see below paragraph16.2.3.3 in this Chapter). However, it does not control for residual confounding and is prone to selection bias introduced by propensity to seek care (and vaccination) and by awareness of possible occurrence of a specific outcome, and does not consider underlying medical conditions, with limited comparability between cases and controls. In addition, it requires reliable and granular vaccine coverage data corresponding to the population from which cases are drawn, to allow control of confounding by stratified analyses (see for example, Risk of narcolepsy in children and young people receiving AS03 adjuvanted pandemic A/H1N1 2009 influenza vaccine: retrospective analysis, BMJ. 2013; 346:f794).
16.2.2.1 General considerations
The book Vaccination Programmes | Epidemiology, Monitoring, Evaluation (Hahné, S., Bollaerts, K., & Farrington, P., Routledge, 2021) discusses the concept of vaccine effectiveness and provides further insight into the methods discussed in this section. The book Design and Analysis of Vaccine Studies (ME Halloran, IM Longini Jr., CJ Struchiner, Ed., Springer, 2010) presents methods and a conceptual framework of the different effects of vaccination at the individual and population level, and includes methods for evaluating indirect, total and overall effects of vaccination in populations.
A key reference is Vaccine effects and impact of vaccination programmes in post-licensure studies (Vaccine 2013;31(48):5634-42), which reviews methods for the evaluation of the effectiveness of vaccines and vaccination programmes and discusses design assumptions and biases to consider. A framework for research on vaccine effectiveness (Vaccine 2018;36(48):7286-93) proposes standardised definitions, considers models of vaccine failure, and provides methodological considerations for different designs.
Evaluation of influenza vaccine effectiveness: a guide to the design and interpretation of observational studies (WHO, 2017) provides an overview of methods to study influenza vaccine effectiveness, also relevant for other vaccines. Evaluation of COVID-19 vaccine effectiveness (WHO, 2021) provides guidance on how to monitor COVID-19 vaccine effectiveness using observational study designs, including considerations relevant to low- and middle-income countries. Methods for measuring vaccine effectiveness and a discussion of strengths and limitations are presented in Exploring the Feasibility of Conducting Vaccine Effectiveness Studies in Sentinel’s PRISM Program (CBER, 2018). Although focusing on the planning, evaluation, and modelling of vaccine efficacy trials, Challenges of evaluating and modelling vaccination in emerging infectious diseases (Epidemics 2021:100506) includes a useful summary of references for the estimation of indirect, total, and overall effects of vaccines.
16.2.2.2. Sources of exposure and outcome data
Data sources for vaccine studies largely rely on vaccine-preventable infectious disease surveillance (for effectiveness studies) and vaccine registries or vaccination data recorded in healthcare databases (for both safety and effectiveness studies). Considerations on validation of exposure and outcome data are provided in Chapter 5.
Infectious disease surveillance is a population-based, routine public health activity involving systematic data collection to monitor epidemiological trends over time in a defined catchment population, and can use various indicators. Data can be obtained from reference laboratories, outbreak reports, hospital records or sentinel systems, and use consistent case definitions and reporting methods. There is usually no known population denominator, thus surveillance data cannot be used to measure disease incidence. Limitations include under-detection/under-reporting (if passive surveillance) or over-reporting (e.g., due to improvements in case detection or introduction of new vaccines with increased disease awareness). Changes/delays in case counting or reporting can artificially reduce the number of reported cases, thus artificially increasing vaccine effectiveness. Infectious Disease Surveillance (International Encyclopedia of Public Health 2017;222-9) is a comprehensive review including definitions, methods, and considerations on use of surveillance data in vaccine studies. The chapter on Routine Surveillance of Infectious Diseases in Modern Infectious Disease Epidemiology (J. Giesecke. 3rd Ed. CRC Press, 2017) discusses how surveillance data are collected and interpreted, and identifies sources of potential bias. Chapter 8 of Vaccination Programmes | Epidemiology, Monitoring, Evaluation outlines the main methods of vaccine-preventable disease surveillance, considering data sources, case definitions, biases and methods for descriptive analyses.
Granular epidemiological surveillance data (e.g., by age, gender, pathogen strain) are of particular importance for vaccine effectiveness studies. Such data were available from the European Centre for Disease Prevention and Control and the WHO Coronavirus (COVID-19) Dashboard during the COVID-19 pandemic and, importantly, also included vaccine coverage data.
EHRs and claims-based databases constitute an alternative to epidemiological surveillance data held by national public health bodies, as illustrated in Using EHR data to identify coronavirus infections in hospitalized patients: Impact of case definitions on disease surveillance (Int J Med Inform. 2022;166:104842), which also recommends using sensitivity analyses to assess the impact of variations in case definitions.
Examples of vaccination registries, and challenges in developing such registries, are discussed in Vaccine registers-experiences from Europe and elsewhere (Euro Surveill. 2012;17(17):20159), Validation of the new Swedish vaccination register - Accuracy and completeness of register data (Vaccine 2020; 38(25):4104-10), and Establishing and maintaining the National Vaccination Register in Finland (Euro Surveill. 2017;22(17):30520). Developed by WHO, Public health surveillance for COVID-19: interim guidance describes key aspects of the implementation of SARS-CoV-2 surveillance, including a section on vaccine effectiveness monitoring in relation to surveillance systems.
16.2.2.3. Study designs for vaccine effectiveness assessment
Traditional cohort and case-control designs
The case-control design has been used to evaluate vaccine effectiveness, but the likelihood of bias and confounding is a potential important limitation. The articles Case-control vaccine effectiveness studies: Preparation, design, and enrollment of cases and controls (Vaccine 2017; 35(25):3295-302) and Case-control vaccine effectiveness studies: Data collection, analysis and reporting results (Vaccine 2017; 35(25):3303-8) provide recommendations on best practices for their design, analysis and reporting. Based on a meta-analysis of 49 cohort studies and 10 case-control studies, Efficacy and effectiveness of influenza vaccines in elderly people: a systematic review (Lancet 2005;366(9492):1165-74) highlights the heterogeneity of outcomes and study populations included in such studies and the high likelihood of selection bias. In A Dynamic Model for Evaluation of the Bias of Influenza Vaccine Effectiveness Estimates From Observational Studies (Am J Epidemiol. 2019;188(2):451-60), a dynamic probability model was developed to evaluate biases in passive surveillance cohort, test-negative, and traditional case-control studies.
Non-specific effects of vaccines, such as a decrease of mortality, have been claimed in observational studies but can be affected by bias and confounding. Epidemiological studies of the 'non-specific effects' of vaccines: I--data collection in observational studies (Trop Med Int Health 2009;14(9):969-76.) and Epidemiological studies of the non-specific effects of vaccines: II--methodological issues in the design and analysis of cohort studies (Trop Med Int Health 2009;14(9):977-85) provide recommendations for observational studies conducted in high mortality settings; however, these recommendations have wider relevance.
The cohort design has been widely used to monitor the effectiveness of COVID-19 vaccines; the following two examples reflect early times of the pandemic, and its later phase when several vaccines were used, reaching wider population groups and used according to different types of vaccination schedule depending on national policies: BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting (N Engl J Med. 2021;384(15):1412-23) used data from a nationwide healthcare organisation to match vaccinated and unvaccinated subjects according to demographic and clinical characteristics, to assess effectiveness against infection, COVID-19 related hospitalisation, severe illness, and death. Vaccine effectiveness against SARS-CoV-2 infection, hospitalization, and death when combining a first dose ChAdOx1 vaccine with a subsequent mRNA vaccine in Denmark: A nationwide population-based cohort study (PLoS Med. 2021;18(12):e1003874) used nationwide linked registries to estimate VE against several outcomes of interest of a heterologous vaccination schedule, compared to unvaccinated individuals. As vaccination coverage increased, using a non-vaccinated comparator group became no longer feasible or suitable, and alternative comparators were needed (see paragraph below on comparative effectiveness).
More recently, pharmacoepidemiological studies have assessed the effectiveness of COVID-19 booster vaccination, which uncovered new methodological challenges, such as the need to account for time-varying confounding. Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data (Ann Intern Med. 2023;176(5):685-693) describes two approaches to target trial emulation to overcome limitations due to confounding or designs not considering the evolution of the pandemic over time and the rapid uptake of vaccination. Comparative effectiveness of different primary vaccination courses on mRNA-based booster vaccines against SARs-COV-2 infections: a time-varying cohort analysis using trial emulation in the Virus Watch community cohort (Int J Epidemiol. 2023 Apr 19;52(2):342-354) conducted trial emulation by meta-analysing eight cohort results to reduce time-varying confounding-by-indication.
Test-negative case-control design
The test-negative case-control design aims to reduce bias associated with misclassification of infection and confounding by healthcare-seeking behaviour, at the cost of sometimes difficult-to-test assumptions. The test-negative design for estimating influenza vaccine effectiveness (Vaccine 2013;31(17):2165-8) explains the rationale, assumptions and analysis of this design, originally developed for influenza vaccines. Study subjects were all persons seeking care for an acute respiratory illness, and influenza VE was estimated from the ratio of the odds of vaccination among subjects testing positive for influenza to the odds of vaccination among subject testing negative. Test-Negative Designs: Differences and Commonalities with Other Case-Control Studies with "Other Patient" Controls (Epidemiology. 2019 Nov;30(6):838-44) discusses advantages and disadvantages of the design in various circumstances. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology (Epidemiology 2020;31(1):43-64) discusses challenges of this design for various vaccines and pathogens, also providing a list of recommendations.
In Effectiveness of rotavirus vaccines in preventing cases and hospitalizations due to rotavirus gastroenteritis in Navarre, Spain (Vaccine 2012;30(3):539-43), electronic clinical reports were used to select cases (children with confirmed rotavirus infection) and test-negative controls (children who tested negative for rotavirus in all samples), under the assumption that the rate of gastroenteritis caused by pathogens other than rotavirus is the same in both vaccinated and unvaccinated subjects. A limitation is sensitivity of the laboratory test, which may underestimate vaccine effectiveness. In addition, if the viral type is not available, it is not possible to study the association between vaccine failure and a possible mismatch between vaccine strains and circulating strains. These learnings still apply today in the context of COVId-19 vaccines.
The article Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness (Am J Epidemiol. 2016;184(5):345-53; see also the related Comments) uses directed acyclic graphs to characterise potential biases and shows how they can be avoided or minimised. In Estimands and Estimation of COVID-19 Vaccine Effectiveness Under the Test-Negative Design: Connections to Causal Inference (Epidemiology 2022;33(3):325-33), an unbiased estimator for vaccine effectiveness using the test-negative design is proposed under the scenario of different vaccine effectiveness estimates across patient subgroups.
In the multicentre study in 18 hospitals 2012/13 influenza vaccine effectiveness against hospitalised influenza A(H1N1)pdm09, A(H3N2) and B: estimates from a European network of hospitals (EuroSurveill 2015;20(2):pii=21011), influenza VE was estimated based on the assumption that confounding due to health-seeking behaviour is minimised since all individuals needing hospitalisation are likely to be hospitalised.
Postlicensure Evaluation of COVID-19 Vaccines (JAMA. 2020;324(19):1939-40) describes methodological challenges of the test-negative design applied to COVID-19 vaccines and discusses solutions to minimise bias. Covid-19 Vaccine Effectiveness and the Test-Negative Design (N Engl J Med. 2021;385(15):1431-33) uses the example of a published study in a large hospital network to provide considerations on how to report findings and assess their sensitivity to biases specific to the test-negative design. The study Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study (BMJ 2021;373:n1088) linked routine community testing and vaccination data to estimate effectiveness against confirmed symptomatic infection, COVID-19 related hospital admissions and case fatality, and estimated the odds ratios for testing positive to SARS-CoV-2 in vaccinated compared to unvaccinated subjects with compatible symptoms. The study also provides considerations on strengths and limitations of the test-negative design.
Case-population, case-coverage, and screening methods
These methods are related, and all include, to some extent, an ecological component such as vaccine coverage or epidemiological surveillance data at population level. Terms to refer to these designs are sometimes used interchangeably. The case-coverage design is discussed above in paragraph 16.2.2.2. Case-population studies are described in Chapter 4.2.5 and in Vaccine Case-Population: A New Method for Vaccine Safety Surveillance (Drug Saf. 2016;39(12):1197-209).
The screening method estimates vaccine effectiveness by comparing vaccination coverage in positive (usually laboratory confirmed) cases of a disease (e.g., influenza) with the vaccination coverage in the population from which the cases are derived (e.g., in the same age group). If representative data on cases and vaccination coverage are available, it can provide an inexpensive and rapid method to provide early estimates or identify changes in effectiveness over time. However, Application of the screening method to monitor influenza vaccine effectiveness among the elderly in Germany (BMC Infect Dis. 2015;15(1):137) emphasises that accurate and age-specific vaccine coverage data are crucial to provide valid estimates. Since adjusting for important confounders and assessing product-specific effectiveness is generally challenging, this method should be considered mainly as a supplementary tool to assess crude effectiveness. COVID-19 vaccine effectiveness estimation using the screening method – operational tool for countries (2022) also provides a good introduction to the method and its strengths and limitations.
Indirect cohort (Broome) method
The indirect cohort method is a case-control type design which uses cases caused by non-vaccine serotypes as controls, and uses surveillance data, instead of vaccination coverage data. Use of surveillance data to estimate the effectiveness of the 7-valent conjugate pneumococcal vaccine in children less than 5 years of age over a 9 year period (Vaccine 2012;30(27):4067-72) evaluated the effectiveness of a pneumococcal conjugate vaccine against invasive pneumococcal disease and compared to the results of a standard case-control design conducted during the same time period. The authors consider the method most useful shortly after vaccine introduction, and less useful in a setting of very high vaccine coverage and fewer cases. Using the indirect cohort design to estimate the effectiveness of the seven valent pneumococcal conjugate vaccine in England and Wales (PLoS One 2011;6(12):e28435) and Effectiveness of the seven-valent and thirteen-valent pneumococcal conjugate vaccines in England: The indirect cohort design, 2006-2018 (Vaccine 2019;37(32):4491-98) describe how the method was used to estimate effectiveness of various vaccine schedules as well as for each vaccine serotype.
Density case-control design
Effectiveness of live-attenuated Japanese encephalitis vaccine (SA14-14-2): a case-control study (Lancet 1996;347(9015):1583-6) describes a case-control study of incident cases in which the control group consisted of all village-matched children of a given age who were at risk of developing disease at the time that the case occurred (density sampling). The effect measured is an incidence density rate ratio. In Vaccine Effectiveness of Polysaccharide Vaccines Against Clinical Meningitis - Niamey, Niger, June 2015 (PLoS Curr. 2016;8), a case-control study compared the odds of vaccination among suspected meningitis cases to controls enrolled in a vaccine coverage survey performed at the end of the epidemic. A simulated density case-control design randomly attributing recruitment dates to controls based on case dates of onset was used to compute vaccine effectiveness. In Surveillance of COVID-19 vaccine effectiveness: a real-time case–control study in southern Sweden (Epidemiol Infect. 2022;150:1-15) a continuous density case-control sampling was performed, with the control group randomly selected from the complete study cohort as individuals without a positive test the same week as the case or 12 weeks prior.
Waning immunity
Studying how immunity conferred by vaccination wanes over time requires consideration of within-host dynamics of the pathogen and immune system, as well as the associated population-level transmission dynamics. Implications of vaccination and waning immunity (Proc Biol Sci. 2009; 276(1664):2071-80) combined immunological and epidemiological models of measles infection to examine the interplay between disease incidence, waning immunity and boosting.
Global Varicella Vaccine Effectiveness: A Meta-analysis (Pediatrics 2016; 137(3):e20153741) highlights the challenges to reliably measure effectiveness when some confounders cannot be controlled for, force of infection may be high, degree of exposure in study participants may be variable, and data may originate from settings where there is evidence of vaccine failure. Several estimates or studies may therefore be needed to accurately conclude in waning immunity. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression (Lancet 2022;399(10328):924-944) reviews evidence of changes in efficacy or effectiveness with time since full vaccination for various clinical outcomes; biases in evaluating changes in effectiveness over time, and how to minimise them, are presented in a tabular format. Effectiveness of Covid-19 Vaccines over a 9-Month Period in North Carolina (N Engl J Med. 2022;386(10):933-941) linked COVID-19 surveillance and vaccination data to estimate reduction in current risks of infection, hospitalisation and death as a function of time elapsed since vaccination, and demonstrated durable effectiveness against hospitalisation and death while waning protection against infection over time was shown to be due to both declining immunity and emergence of the delta variant.
Vaccine effectiveness estimates over time are subject to bias from differential depletion of susceptibles (persons at risk of infection) between vaccinated and unvaccinated groups, which can lead to biased estimates of waning effectiveness. Depletion-of-susceptibles bias in influenza vaccine waning studies: how to ensure robust results (Epidemiol Infect. 2019;147:e306) recommends to study only vaccinated persons, and compare for each day the incidence in persons with earlier or later dates of vaccination, to assess waning as a function of vaccination time. Identifying and Alleviating Bias Due to Differential Depletion of Susceptible People in Postmarketing Evaluations of COVID-19 Vaccines (Am J Epidemiol. 2022;191(5):800-11) outlines scenarios under which bias can arise and identifies approaches to minimise these biases.
Comparative vaccine effectiveness
Comparing vaccine benefits has traditionally been performed using head-to-head immunogenicity studies, while comparative effectiveness designs have been used mostly to compare vaccination schedules, vaccine formulations, or administration routes (e.g., for measles, mumps and rubella (MMR), influenza, or pneumococcal vaccines; see for example, Analysis of relative effectiveness of high-dose versus standard-dose influenza vaccines using an instrumental variable method (Vaccine 2019;37(11):1484-90). Methods to account for measured and unmeasured confounders in influenza relative vaccine effectiveness studies: A brief review of the literature (Influenza Other Respir. Viruses 2022;16(5):846-850) discusses methods to account for confounding in such studies. In The risk of non-specific hospitalised infections following MMR vaccination given with and without inactivated vaccines in the second year of life. Comparative self-controlled case-series study in England (Vaccine 2019;37(36):5211-17) the SCCS design was used to compare the effectiveness of the MMR vaccine alone with the MMR vaccine in combination with PCV7 or with both PCV7 and the combined Hib-MenC vaccine. Comparative effectiveness of pneumococcal vaccination with PPV23 and PCV13 in COPD patients over a 5-year follow-up cohort study, (Sci Rep 2021;11(1):15948.) used a prospective cohort design to compare effectiveness between the 23-valent vaccine, the 13-valent vaccine, and no vaccination.
The COVID-19 vaccination campaigns increased the interest in, and triggered, comparative effectiveness studies. Postmarketing studies: can they provide a safety net for COVID-19 vaccines in the UK? (BMJ Evid Based Med. 2020:bmjebm-2020-111507) discusses methodological and operational aspects and provides considerations on head-to-head vaccine comparisons. Assessment of Effectiveness of 1 Dose of BNT162b2 Vaccine for SARS-CoV-2 Infection 13 to 24 Days After Immunization (JAMA Netw Open. 2021;4(6):e2115985) compared the effectiveness of the first vaccine dose between two post-immunisation periods. Comparative effectiveness of the BNT162b2 and ChAdOx1 vaccines against Covid-19 in people over 50 (Nat Commun. 2022;13(1):1519) used data from the UK Biobank linked to primary care, hospital admissions, and COVID-19 testing data, to compare the effectiveness of BNT162b2 vs. ChAdOx1s against COVID-19 infection and hospitalisation, using propensity score modelling. Comparative Effectiveness of BNT162b2 and mRNA-1273 Vaccines in U.S. Veterans (N Engl J Med. 2022;386(2):105-15) and Comparative effectiveness of BNT162b2 versus mRNA-1273 covid-19 vaccine boosting in England: matched cohort study in OpenSAFELY-TPP used a target trial emulation design.
Comparative vaccine effectiveness studies may require larger sample sizes, as they aim to detect smaller effect sizes as opposed to effectiveness studies for a single vaccine, where an unvaccinated group is used as a comparator. Various sources of confounding (such as self-seeking testing behaviour) should be considered, and appropriate methods used, such as (propensity score) matching, instrumental variable analysis, inverse probability of treatment weighting, use of negative control outcomes, off-season outcomes (for influenza vaccines) and positive control outcomes. For some vaccines (e.g., COVID-19 vaccines), variant-specific comparative effectiveness data are important, taking into consideration the correlation between vaccine schedules and calendar periods, and therefore with variants in circulation at a given time.
Impact studies
Vaccine impact studies estimate disease reduction in a community. These studies are typically ecological or modelling analyses that compare disease outcomes pre- and post-vaccine introduction. Reductions in disease outcomes are observed through direct effects of vaccination in vaccinated people, and indirect effects due to reduced transmission within a community. Other concurrent interventions or phenomena unrelated to vaccine effects, such as changes in risk behaviours or healthcare practices, may reduce disease outcomes and confound the assessment of vaccine impact (see The value of vaccine programme impact monitoring during the COVID-19 pandemic, Lancet 2022;399(10320):119-21). For example, for a paediatric vaccine, the impact of vaccination can be quantified in the targeted age group (overall effect) or in other age groups (indirect effect). For an overview, see Vaccine effects and impact of vaccination programmes in post-licensure studies (Vaccine 2013;31(48):5634-42).
Direct and indirect effects in vaccine efficacy and effectiveness (Am J Epidemiol. 1991;133(4):323-31) describes how parameters intended to measure direct effects must be robust and interpretable in the midst of complex indirect effects of vaccine intervention programmes. Lack of impact of rotavirus vaccination on childhood seizure hospitalizations in England - An interrupted time series analysis (Vaccine 2018; 36(31):4589-92) discusses possible reasons for negative findings compared to previous studies. In a review of 65 articles, Population-level impact and herd effects following the introduction of human papillomavirus vaccination programmes: updated systematic review and meta-analysis (Lancet. 2019;394(10197):497–509) compared the prevalence or incidence of several HPV-related endpoints between the pre- and post-vaccination periods with stratification by sex, age, and years since introduction of HPV vaccination.
Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data (Lancet. 2021;397(10287):1819-29) evaluated the public health impact of vaccination using national surveillance and vaccine uptake data. Although such population-level data are ecological, and teasing apart the impact of the vaccination programme from the impact of non-pharmaceutical interventions is complex, declines in incident cases by age group were shown to be aligned with high vaccine coverage rather than initiation of the nationwide lockdown.
Accumulated effectiveness data has suggested the potential for a population-level effect of COVID-19 vaccination, which has been critical to control the pandemic. Community-level evidence for SARS-CoV-2 vaccine protection of unvaccinated individuals (Nat Med. 2021;27(8):1367-9) analysed vaccination records and test results in a large population, while mitigating the confounding effect of natural immunity and the spatiotemporally dynamic nature of the epidemic, and showed that vaccination provided cross-protection to unvaccinated individuals in the community.
Transmission studies
Vaccination programmes have indirect effects at the population-level, also called herd immunity, as a result of reduced transmission. Besides measuring the direct effect of vaccination in vaccine effectiveness studies, it is important to assess whether vaccination has an effect on transmission. As a high-risk setting, households can provide evidence of such impact.
Among the first studies of the impact of COVID-19 vaccination on transmission, Effect of Vaccination on Household Transmission of SARS-CoV-2 in England (N Engl J Med. 2021;385(8):759-60) was a nested case-control study estimating odds ratios for household members becoming secondary cases if the case was vaccinated within 21 days or more before testing positive, vs. household members where the case was not vaccinated. Vaccination with BNT162b2 reduces transmission of SARS-CoV-2 to household contacts in Israel (Science. 2022;375(6585):1151-54) assessed the effectiveness of BNT162b2 against susceptibility to infection and infectiousness, comparing pre- and post-Delta periods, using a chain binomial model applied to data from a large healthcare organisation. Community transmission and viral load kinetics of the SARS-CoV-2 delta (B.1.617.2) variant in vaccinated and unvaccinated individuals in the UK: a prospective, longitudinal, cohort study (Lancet Infect Dis. 2022;22(2):183-95) ascertained secondary transmission by longitudinally following index cases and their contacts (regardless of symptoms) early after exposure to the Delta variant, and highlights the importance of community studies to characterise transmission in highly vaccinated populations.
Specific limitations of transmission studies such as likelihood of information bias (misclassification) and selection bias, should be considered when interpreting findings and are discussed in the above references.
Cluster design
A cluster is a group of subjects sharing common characteristics: geographical (community, administrative area), health-related (hospital), educational (schools), or social (household). In cluster randomised trials, clusters instead of individual subjects are randomly allocated to an intervention, whereas in infectious disease epidemiology studies, clusters are sampled based on aspects of transmission (e.g., within a community) or a vaccination programme. This design is often used in low and middle income settings and can measure vaccination interventions naturally applied at the cluster level or when the study objectives require a cluster design (e.g., to estimate herd immunity).
Meningococcal B Vaccine and Meningococcal Carriage in Adolescents in Australia (N Engl J Med. 2020;382(4):318-27) used cluster randomisation to assign students, according to school, to receive 4CMenB vaccination, either at baseline or at 12 months (as a control) to measure oropharyngeal carriage.
In The ring vaccination trial: a novel cluster randomised controlled trial design to evaluate vaccine efficacy and effectiveness during outbreaks, with special reference to Ebola (BMJ. 2015;351:h3740), a newly diagnosed Ebola case served as the index case to form a “ring”, which was then randomised to immediate or delayed vaccination with inclusion based on tracing cases using active surveillance instead of randomisation. In Assessing the safety, impact and effectiveness of RTS,S/AS01 E malaria vaccine following its introduction in three sub-Saharan African countries: methodological approaches and study set-up (Malar J. 2022;21(1):132), active surveillance was used to enrol large numbers of children in vaccinated and unvaccinated clusters as part of the WHO Malaria Vaccine Implementation Programme, to conduct temporal (before vs. after) and concurrent (exposed vs. unexposed) cluster comparisons. Clusters were selected based on geographically limited areas with demographic surveillance in place and infrastructure to monitor population health and vaccination programmes.
Misclassification in studies of vaccine effectiveness
Like vaccine safety studies, studies of vaccine effectiveness rely on accurate identification of vaccine exposure and of cases of the targeted vaccine-preventable disease/infection, but in practice, diagnostic tests, clinical case definitions and vaccination records often present inaccuracies. Bias due to differential and non-differential disease- and exposure misclassification in studies of vaccine effectiveness (PLoS One 2018;15;13(6):e0199180) explores through simulations the impact of non-differential and differential disease- and exposure-misclassification when estimating vaccine effectiveness using cohort, case-control, test-negative case-control and case-cohort designs. This can also be applied to safety outcomes, especially those with a complex natural history such as neurological or potential immune mediated diseases, and is particularly relevant for secondary use of data, where validation studies may be needed in a first step. Misclassification can lead to significant bias and its impact strongly depends on the vaccination scenarios. A web application designed in the ADVANCE project is publicly available to assess the potential (joint) impact of possibly differential disease- and exposure misclassification.
16.2.3. Specific aspects of pharmacoepidemiological vaccine studies
16.2.3.1. Studies in special populations
Special populations include pregnant and breastfeeding persons, immunocompromised patients (including transplanted patients), paediatric populations, older adults/the elderly, and patients with rare disorders. Post-authorisation studies are often required for these populations, which are usually not included in the clinical development of vaccines. In real-world settings, special populations are often the subject of specific vaccination recommendations, which may impact study designs and choice of an appropriate comparator. This was the case, for example, of COVID-19 vaccines which initially targeted high-risk priority groups. The article Vaccine safety in special populations (Hum Vaccin. 2011;7(2):269-71) highlights design issues when evaluating vaccine safety in these populations. Methodological challenges include defining the study population (particularly for immunocompromised populations), low sample size due to rare outcomes, accounting for comorbidities and other types of confounders, or difficulty in identifying cases or disease duration and severity in immunocompromised patients.
Influenza vaccination for immunocompromised patients: systematic review and meta-analysis by etiology (J Infect Dis. 2012;206(8):1250-9) illustrates the importance of performing stratified analyses by aetiology of immunocompromised status and limitations due to residual confounding, differences within and between etiological groups and small sample size in some subgroups. In anticipation of the design of post-authorisation vaccine effectiveness and safety studies, the study Burden of herpes zoster in 16 selected immunocompromised populations in England: a cohort study in the Clinical Practice Research Datalink 2000–2012 (BMJ Open 2018;8(6): e020528) illustrated the challenges of defining an immunocompromised cohort and a relevant comparator cohort in a primary healthcare database. Validation of a Method to Identify Immunocompromised Patients with Severe Sepsis in Administrative Databases (Ann Am Thorac Soc. 2016;13(2):253-8) provides considerations on identifying this group of patients in large administrative databases.
Pregnant and breastfeeding persons represent an important group to be addressed when monitoring vaccine use; Annex 2 of this Guide provides guidance on methods to evaluate medicines in pregnancy and breastfeeding, including for vaccine studies. The Guidance for design and analysis of observational studies of foetal and newborn outcomes following COVID-19 vaccination during pregnancy (Vaccine 2021;39(14):1882-6) provides useful insights on study design, data collection, and analytical issues in COVID-19 vaccine safety studies in pregnant people, and can be applied to other vaccines.
The guidance on conducting meta-analyses of pharmacoepidemiological studies of safety outcomes (Annex 1 of this Guide) is also applicable to vaccines. A systematic review evaluating the potential for bias and the methodological quality of meta-analyses in vaccinology (Vaccine 2007;25(52):8794-806) provides a comprehensive overview of quality and limitations of meta-analyses. Meta-analysis of the risk of autoimmune thyroiditis, Guillain-Barré syndrome, and inflammatory bowel disease following vaccination with AS04-adjuvanted human papillomavirus 16/18 vaccine (Pharmacoepidemiol Drug Saf. 2020;29(9):1159-67) combined data from 18 randomised controlled trials, one cluster-randomised trial, two large observational retrospective cohort studies, and one case-control study, resulting in a large sample size for these rare events. The Systematic review and meta-analysis of the effectiveness and perinatal outcomes of COVID-19 vaccination in pregnancy (Nat Commun. 2022;13(1):2414) generated evidence on a large number of adverse pregnancy and perinatal outcomes.
Meta-analytical methods are increasingly used in multi-database studies (see Chapter 9) to combine data generated at country level to obtain pooled risk estimates in large populations. In SARS-CoV-2 Vaccination and Myocarditis in a Nordic Cohort Study of 23 Million Residents (JAMA Cardiol. 2022;7(6):600-12), four cohort studies were conducted in linked nationwide health registers in Denmark, Finland, Norway, and Sweden according to a common protocol; the results were combined using meta-analysis and the homogeneity of country-specific estimates was tested.
16.2.3.3. Pharmacogenetic studies
There is increasing interest in the role of genomics in pharmacoepidemiology (see Chapter 16.3), including for the study of vaccine safety outcomes (see Adversomics: a new paradigm for vaccine safety and design, Expert Rev Vaccines 2015; 14(7): 935–47). Vaccinomics and Adversomics in the Era of Precision Medicine: A Review Based on HBV, MMR, HPV, and COVID-19 Vaccines (J Clin Med. 2020;9(11):3561) highlights that knowledge of genetic factors modulating responses to vaccination could contribute to the evaluation of vaccine safety and effectiveness. In State-wide genomic epidemiology investigations of COVID-19 in healthcare workers in 2020 Victoria, Australia: Qualitative thematic analysis to provide insights for future pandemic preparedness (Lancet Reg Health West Pac. 2022;25:100487), a large SARS-CoV-2 genomic epidemiological investigation identified transmission dynamics using a newly developed set of metadata. Genetic risk and incident venous thromboembolism in middle-aged and older adults following COVID-19 vaccination (J Thromb Haemost. 2022;20(12):2887-2895) used data from the UK Biobank to estimate hazard ratios of the associations between a polygenic risk score and post-vaccination incident veinous thromboembolism.
Generic protocols, also referred to as template or master protocols, provide a standardised structure to support study design and protocol development. Such protocols have supported the urgent need for COVID-19 vaccine monitoring, often based, in Europe, on the EMA Guidance for the format and content of the protocol of non-interventional post-authorisation safety studies (2012).
A protocol for generating background rates of AESIs for the monitoring of COVID-19 vaccines (2021) was developed by the vACcine Covid-19 monitoring readinESS (ACCESS) consortium, which also published Template study protocols (2021) to support the design of safety studies, based on both cohort-event monitoring and secondary use of data. The protocol Rapid assessment of COVID-19 vaccines safety concerns through electronic health records- a protocol template from the ACCESS project compares the suitability of the ecological design and the unadjusted self-controlled risk interval (SCRI) for rapid safety assessment, by type of AESI. Other published templates include FDA’s Background Rates of Adverse Events of Special Interest for COVID-19 Vaccine Safety Monitoring protocol, the COVID-19 Vaccine Safety Active Monitoring Protocol and the Master Protocol: Assessment of Risk of Safety Outcomes Following COVID-19 Vaccination (FDA BEST Initiative, 2021); and the Template for observational study protocols for sentinel surveillance of adverse events of special interest (AESIs) after vaccination with COVID-19 vaccines (WHO, 2021).
The ACCESS consortium also published template protocols (2021) for COVID-19 vaccine effectiveness studies using the cohort and test-negative case-control designs. The Core protocol for ECDC studies of COVID-19 vaccine effectiveness against hospitalisation with Severe Acute Respiratory Infection laboratory-confirmed with SARS-CoV-2 (ECDC, 2021) presents the main elements to consider to design multi-centre, multi-country hospital-based COVID-19 vaccine effectiveness studies in patients hospitalised with severe acute respiratory infections (SARI).
The DRIVE project developed a Core protocol for type/brand specific influenza vaccine effectiveness studies - Test-negative design studies and a Core protocol for population-based database cohort-studies, and the COVIDRIVE consortium a Brand-specific COVID-19 vaccine effectiveness protocol to assess effectiveness against severe COVID-19 disease.
Generic protocols for retrospective case-control studies and retrospective cohort studies to assess the effectiveness of rotavirus and influenza vaccination in EU Member States are published by ECDC and describe potential data sources to identify virological outcomes. The Protocol for Cluster Investigations to Measure Influenza Vaccine Effectiveness (ECDC, 2009) builds on the cluster design to generate rapid/early influenza season estimates in settings where investigation can take place at the same time as vaccination is carried out (e.g. schools, care homes). The generic study protocol to assess the impact of rotavirus vaccination (ECDC, 2013) lists the information to be collected to compare the incidence/proportion of rotavirus cases in the period before and after vaccine introduction.
Although developed for specific vaccines, all these protocols can be tailored to other vaccine exposures and outcomes, as they address the most important aspects to consider for the design of vaccine safety and effectiveness studies.
For the UK, as from 1.1.2021, EU Law applies only to the territory of Northern Ireland (NI) to the extent foreseen in the Protocol on Ireland/NI
