Print page Resize text Change font-size Change font-size Change font-size High contrast

Home > Standards & Guidances > Methodological Guide

ENCePP Guide on Methodological Standards in Pharmacoepidemiology Signal refinement

When prompt decision-making about a safety concern is required and there is insufficient time to review individual cases, the GVP Module P.I: Vaccines for prophylaxis against infectious diseases suggests the conduct of observed vs. expected (O/E) analyses for signal validation and preliminary signal evaluation. The module discusses key requirements of O/E analyses: the observed number of cases detected in a passive or active surveillance systems, near real-time exposure data, appropriately stratified background incidence rates (to calculate the expected number of cases) and sensitivity analyses around these measures.


Human papilloma virus immunization in adolescents and young adults: a cohort study to illustrate what events might be mistaken for adverse reactions (Pediatr Infect Dis J 2007;26(11):979-84) and Health problems most commonly diagnosed among young female patients during visits to general practitioners and gynecologists in France before the initiation of the human papillomavirus vaccination program (Pharmacoepidemiol Drug Saf 2012; 21(3):261-80) illustrate the importance of collecting background rates by estimating risks of coincident associations of emergency consultations, hospitalisations and outpatients consultations with vaccination. Rates of selected disease events for several countries also vary by age, sex, method of ascertainment and geography, as shown in Importance of background rates of disease in assessment of vaccine safety during mass immunisation with pandemic H1N1 influenza vaccines (Lancet 2009; 374(9707):2115-22).


Simple “snapshot” O/E analyses are easy to perform but may not be appropriate for continuous monitoring due to inflation of type 1 error rates when multiple tests are performed. Safety monitoring of Influenza A/H1N1 pandemic vaccines in EudraVigilance (Vaccine 2011;29(26):4378-87) illustrates that simple “snapshot” O/E analyses are also affected by uncertainties regarding the numbers of vaccinated individuals and age-specific background incidence rates.


Sequential methods, as described in Early detection of adverse drug events within population-based health networks: application of sequential methods (Pharmacoepidemiol Drug Saf 2007; 16(12):1275-1284), allow O/E analyses to be performed on a routine (e.g. weekly) basis using cumulative data with adjustment for multiplicity. Such methods are routinely used for near-real time surveillance in the Vaccine Safety Datalink (VSD) (Near real-time surveillance for influenza vaccine safety: proof-of-concept in the Vaccine Safety Datalink Project. Am J Epidemiol 2010;171(2):177-88). Potential issues are described in Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data (Pharmacoepidemiol Drug Saf 2012; 21(S1):62-71). A review of signals detected over 3 years with these methods in VSD concluded that care with data quality, outcome definitions, comparison groups and length of surveillance is required to enable detection of true safety problems while controlling error rates (Active surveillance for adverse events: the experience of the Vaccine Safety Datalink Project (Pediatrics 2011; 127(S1):S54-S64). Sequential methods are, therefore, more robust but also more complex to perform, understand and communicate to a non-statistical audience.

A new self-controlled case series method for analyzing spontaneous reports of adverse events after vaccination (Am J Epidemiol 2013;178(9):1496-504) extends the self-controlled case series approach to explore and quantify vaccine safety signals from spontaneous reports. It uses parametric and nonparametric versions with different assumptions to account for the specific features of the data (large amount of underreporting and variation of reporting with time since vaccination). The method should be seen as a signal strengthening approach for exploring a signal based on spontaneous reports prior to a pharmacoepidemiologic study, if any. It was used to document the risk of intussusception after rotavirus vaccines (see Intussusception after Rotavirus Vaccination — Spontaneous Reports; N Engl J Med 2011; 365:2139).


Individual Chapters:


1. General aspects of study protocol

2. Research question

3. Approaches to data collection

3.1. Primary data collection

3.2. Secondary use of data

3.3. Research networks

3.4. Spontaneous report database

3.5. Using data from social media and electronic devices as a data source

3.5.1. General considerations

4. Study design and methods

4.1. General considerations

4.2. Challenges and lessons learned

4.2.1. Definition and validation of drug exposure, outcomes and covariates Assessment of exposure Assessment of outcomes Assessment of covariates Validation

4.2.2. Bias and confounding Choice of exposure risk windows Time-related bias Immortal time bias Other forms of time-related bias Confounding by indication Protopathic bias Surveillance bias Unmeasured confounding

4.2.3. Methods to handle bias and confounding New-user designs Case-only designs Disease risk scores Propensity scores Instrumental variables Prior event rate ratios Handling time-dependent confounding in the analysis

4.2.4. Effect modification

4.3. Ecological analyses and case-population studies

4.4. Hybrid studies

4.4.1. Pragmatic trials

4.4.2. Large simple trials

4.4.3. Randomised database studies

4.5. Systematic review and meta-analysis

4.6. Signal detection methodology and application

5. The statistical analysis plan

5.1. General considerations

5.2. Statistical plan

5.3. Handling of missing data

6. Quality management

7. Communication

7.1. Principles of communication

7.2. Guidelines on communication of studies

8. Legal context

8.1. Ethical conduct, patient and data protection

8.2. Pharmacovigilance legislation

8.3. Reporting of adverse events/reactions

9. Specific topics

9.1. Comparative effectiveness research

9.1.1. Introduction

9.1.2. General aspects

9.1.3. Prominent issues in CER Randomised clinical trials vs. observational studies Use of electronic healthcare databases Bias and confounding in observational CER

9.2. Vaccine safety and effectiveness

9.2.1. Vaccine safety General aspects Signal detection Signal refinement Hypothesis testing studies Meta-analyses Studies on vaccine safety in special populations

9.2.2. Vaccine effectiveness Definitions Traditional cohort and case-control studies Screening method Indirect cohort (Broome) method Density case-control design Test negative design Case coverage design Impact assessment Methods to study waning immunity

9.3. Design and analysis of pharmacogenetic studies

9.3.1. Introduction

9.3.2. Identification of genetic variants

9.3.3. Study designs

9.3.4. Data collection

9.3.5. Data analysis

9.3.6. Reporting

9.3.7. Clinical practice guidelines

9.3.8. Resources

Annex 1. Guidance on conducting systematic revies and meta-analyses of completed comparative pharmacoepidemiological studies of safety outcomes