Print page Resize text Change font-size Change font-size Change font-size High contrast

Home > Standards & Guidances > Methodological Guide

ENCePP Guide on Methodological Standards in Pharmacoepidemiology


16.6. Real-world evidence and pharmacoepidemiology

16.6.1. Introduction

16.6.2. Definitions

16.6.3. Use of real-world evidence in medicines evaluation

16.6.4. Real-world evidence vs. clinical trials

16.6.5. Real-world evidence and pharmacoepidemiology


16.6.1. Introduction


The pharmacoepidemiology community has a long tradition of producing, evaluating, and interpreting observational data to provide evidence on the use, safety and effectiveness of medicines. The increasing ability to electronically capture and store data from routine healthcare systems and transform it into additional knowledge has opened up new opportunities for investigators to conduct studies. The terms real-world data (RWD) and real-world evidence (RWE) have been increasingly used since the early 2000’s to denote evidence generated from observational data collected during routine patient-level healthcare interactions. In medicines evaluation, evidence relying on RWD is now frequently submitted across the lifecycle of a product to complement and contextualise clinical trial knowledge with information from the routine healthcare setting, but the place of RWD in regulatory decision-making is still a subject of debate (see for example Replacing RCTs with real world data for regulatory decision making: a self-fulfilling prophecy? BMJ. 2023:380:e073100). Contribution of Real-World Evidence in European Medicines Agency’s Regulatory Decision Making (Clin Pharmacol Ther. 2023;113(1):136-51) reports that RWD/RWE was considered not supportive or was not further addressed in the regulatory evaluation report for 15 of 26 applications submitted to EMA in 2018-2019, where RWD/RWE was included to support efficacy pre-authorisation. Many issues discussed in the evaluation reports with respect to RWE were weaknesses related to methodological aspects, highlighting the need for adequate pharmacoepidemiological and statistical expertise in the generation of RWE.


There is currently no internationally agreed definition of RWD and RWE. Real World Evidence – Where Are We Now? (N Engl J Med. 2022;386(18):1680-2) emphasises that these terms are being used inconsistently and sometimes interchangeably across different health domains. Although evolving, a consistent terminology is yet to be established.


This chapter discusses ENCePP’s views on definitions of RWD and RWE, their role in medicines approval and evaluation, their relation to evidence generated by clinical trials, and why pharmacoepidemiological methods remain essential for the generation and assessment of RWD and RWE.


16.6.2. Definitions


The FDA’s Real-World Evidence website defines RWD as “the data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources” and RWE as “clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of RWD”. These definitions are widely used and have been adopted by other regulatory agencies. There is however a debate about the appropriateness of including both the nature of the data and the way it is collected (“routinely”) in the definition of RWD. RWD is commonly understood as observational data from various origins (e.g., electronic healthcare records, claims data, registries) but Marketing Authorization Applications Made to the European Medicines Agency in 2018-2019: What was the Contribution of Real-World Evidence? (Clin Pharmacol Ther. 2021;111(1):90-7) illustrates the difficulty of applying definitions of RWD in authorisation applications, notably when RWD is included as an element of clinical trials. Real-World Trends in the Evaluation of Medical Products (Am J Epidemiol. 2023;192(1):1-5) states that there is room for interpretation as to the data considered as RWD, for example data collected outside of health care settings for research purposes such as those collected through patient self-report outside of clinical encounters, or data collected through new technologies such as wearable biometric devices. This comment also applies to genetic data that are often collected outside the context of routine care or clinical trials but are generally considered as RWD. It is also noted that the data quality frameworks developed for RWD (see Chapter 13.2) examine how accurately the data represent the original information and how suitable they are but not how routinely they have been collected.


The view of ENCePP is that the specificity of RWD in comparison to any other observational data lies in the requirement for a true representation of the “real-world” patient characteristics (i.e., data with a high external validity) without influence of any specific study conditions. An assessment and validation of this real-world attribute, e.g., by external validation or benchmarking, is needed to provide assurance that it applies, or at least to evaluate and understand the deviation that may exist. A simpler definition of RWD could therefore only refer to patient data in contemporary clinical practice.


RWE is information derived from the analysis of RWD using sound epidemiological and statistical practices. The term RWE does not refer to specific methodologies and overlaps with pharmacoepidemiology, although it only partially overlaps with traditional classification of clinical research such as randomised vs. observational, prospective vs. retrospective or primary data collection vs. secondary use of data. The term RWE is nevertheless useful to state that the evidence originates from RWD, in the same way as the term experimental evidence is sometimes used to state that the evidence is based on experimental data.


16.6.3. Use of real-world evidence in medicines evaluation


There are many examples where RWD and RWE can be submitted to support medicines evaluation and regulatory decision-making. Three main objectives are identified in EMA’s DARWIN EU®: Multi-stakeholder information webinar (2022; slides 14-21):

  • to support the planning and validity of applicant studies, for example to inform the recruitment in pre- and post-authorisation studies, to examine the impact of planned inclusion/exclusion criteria, to measure the representativeness of the CT population (treatment and control arm) vs. the real-world target population and to evaluate whether the standard of care used in the control arm of a CT is comparable with the current real-word standard of care;

  • to understand the clinical context, for example to evaluate the incidence, prevalence and characteristics of diseases, to generate evidence on the actual clinical standards of care and compare them in different populations, and to characterise real-world drug use (incidence, prevalence, amount, duration, switching patterns);

  • to investigate associations and impact, for example to investigate the association between treatment exposure and either effectiveness or safety outcomes (including use of RWD as external control group), and to monitor the implementation and the effectiveness of risk minimisation measures.

Several studies have recently attempted to measure the frequency of use of RWD or RWE in marketing authorisation applications and the extent to which these data were actually utilised for decision-making, see, for example:

Due to variability in definitions, data sources, study designs and acceptability of RWD by regulatory decision-making bodies, very different estimates were found in these studies, with percentages of authorisation applications including RWE ranging from 39.9% to 100%.


How to enhance the suitability and acceptability of RWD/RWE to support authorisation applications is a matter of discussion and several publications have made proposals:

  • Contribution of Real-World Evidence in European Medicines Agency’s Regulatory Decision Making (Clin Pharmacol Ther. 2023;113(1):136-51) provides an in-depth analysis of the actual contribution of RWE in the decision-making on marketing authorisation approvals of applications submitted to EMA in 2018-2019, why such information was not considered supportive in some cases and how it contributed to the approval decision in other cases. It discusses suggestions to enable broader use of RWE in medicines development, including provision of data on mechanisms of action where RWE is used to extrapolate efficacy data from adults to children, previous experience with the medicinal product outside the EU application, description of the disease population and natural course of the disease, and early interactions (such as through scientific advice) between applicants and regulators to discuss the expected value of RWD to answer a specific research question, their limitations and how they could be minimised.

  • Harnessing Real-World Evidence to Advance Cancer Research (Curr. Oncol. 2023;30(2):1844-59) proposes a strategy with four steps: 1) to identify meaningful and well-defined clinical questions answerable with available RWD rather than scenarios for which RCTs are necessary and feasible; 2) to rely on high-quality RWD representative of the population of interest and contemporary clinical practice and with documented data completeness and provenance; 3) to use appropriate study designs accounting for data limitations, bias, confounding and sensitivity analyses; 4) to use clear, transparent and replicable study methodology to increase the confidence in the results.

  • Assessing and Interpreting Real-World Evidence Studies: Introductory Points for New Reviewers (Clin Pharmacol. 2022;111(1):145-9) details three aspects: the research question evaluated in the RWE study must align with the question of interest, with a recommendation to break it down according to the Population, Intervention, Comparator Outcome and Timing (PICOT) framework; the study design must use valid methods minimising selection bias, information bias and confounding, with a recommendation to use the target trial framework to help plan and design the RWE study; and the data must be suitable to address the research question, with elements of reliability (incl. plausibility and missingness) and relevance. 

  • When Can We Rely on Real‐World Evidence to Evaluate New Medical Treatments? (Clin Pharmacol Ther. 2021;111(1):30-4) recommends that decisions regarding use of RWE in the evaluation of new treatments should depend on the specific research question, characteristics of the potential study settings and characteristics of the settings where study results would be applied, and take into account three dimensions in which RWE studies might differ from traditional clinical trials: use of RWD, delivery of real-world treatment and real-world treatment assignment.

  • Real-world evidence to support regulatory decision making: New or expanded medical product indications (Pharmacoepidemiol Drug Saf. 2021;30(6):685-93) reviews more specifically study designs used to generate RWE, including pragmatic trials, externally controlled trials and non-randomised healthcare database studies, among others.

  • Real-World Data for Regulatory Decision Making: Challenges and Possible Solutions for Europe (Clin Pharmacol Ther. 2019; 106(1):36-9) specifies four criteria for acceptability of RWE for regulatory purposes: it should be derived from data sources of demonstrated good quality, valid (with both internal and external validity), consistent (or heterogeneity should be explained) and adequate in terms of amount of information provided.

  • When and How Can Real World Data Analyses Substitute for Randomized Controlled Trials? (Clin Pharmacol. Ther. 2017;102(6):924-33) suggests that RWE is likely to be preferred over RCTs when studying a highly promising treatment for a disease with no other available treatments, where ethical considerations may preclude randomising patients to placebo, particularly if the disease is likely to result in severely compromised quality of life or mortality. In these cases, RWE could support product regulation by providing evidence on the safety and effectiveness of the therapy against the typical disease progression observed in the absence of treatment.

  • Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0 (Pharmacoepidemiol Drug Saf. 2017;26(9):1018-32) highlights that substantial improvement in reproducibility, rigor and confidence in RWE generated from healthcare databases could be achieved with greater transparency about study parameters used to create analytic datasets from longitudinal healthcare databases and provides lists of specific parameters to be reported to increase reproducibility of studies.

Regulatory agencies have also published methodological recommendations to medicines developers on the submission of RWD/RWE within their applications to support their evaluation and acceptability:

16.6.4. Real-world evidence vs. clinical trials


The value of RWE to provide valid evidence on medicinal products as compared to clinical trials is a frequent subject of debate in the context of regulatory assessments, especially for medicines effectiveness where a departure from traditional clinical trials has been called on to speed-up their pace, reduce their cost and increase their generalisability. While RCTs are the gold standard for demonstrating the efficacy of medicinal products, they rarely measure the benefits and risks of an intervention when used in contemporary clinical practice and the current thinking is moving away from the long-held position that RWE is always inferior due to the likelihood of bias. Randomized Controlled Trials Versus Real World Evidence: Neither Magic Nor Myth (Clin Pharmacol Ther. 2021;109(5):1212–8) illustrates that the question is not about RCTs vs. RWE but about RCTs and RWE. In other words, use of observational evidence should generally not be considered to replace RCT information, except in specific circumstances, but both are complementary, as RWE may provide additional data, such as on longer follow-up of interventions and on treatment effects in populations not included in RCTs. Real World Evidence – Where Are We Now? (N Engl J Med. 2022;386(18):1680-2) suggests that randomised, non-randomised interventional and non-randomised non-interventional studies may rely on RWD for different objectives and therefore generate RWE, as illustrated by the following diagram:


Reliance on RWD in Representative Types of Study Design. RCT denotes randomized, controlled trial; RWD real-world data; and RWE real-world evidence. Source: Concato J, Corrigan-Curay JD. Real World Evidence – Where Are We Now? (N Engl J Med. 2022;386(18):1680-2).


Statistical Considerations When Using RWD and RWE in Clinical Studies for Regulatory Purposes: A Landscape Assessment (Statistics in Biopharmaceutical Research 2023;15:1,3-13) discusses examples of when RWD can be incorporated into the design of various study types, including RCTs and purely observational studies, and reviews biostatistical challenges and methods for the use of RWE for medicinal product development.


A current domain of research is the assessment of whether non-interventional RWE studies can provide the same results as RCTs performed for the same research question. Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials (JAMA 2023;329(16):1376-85) concludes that RWE studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.


16.6.5. Real-world evidence and pharmacoepidemiology


All the elements cited above to generate valid and reliable RWE using RWD are related to fundamental principles of pharmacoepidemiology. The widespread use of the concept of RWD/RWE has stimulated the use, accessibility and quality control of data sources as well as methodological developments to prevent and control bias and confounding, for example confounding by indication. Pharmacoepidemiologists should therefore take a leadership role and embrace this concept as a domain of research supporting regulatory decisions on medicinal products and public health in general. The following list includes areas of pharmacoepidemiological expertise that ENCePP considers important to develop and disseminate:


- Knowledge about RW data source metadata and its characteristics

  • Understanding of different data types (e.g., primary care, specialist care, hospital care, disease registries, claims data, longitudinal drug prescription, dispensing or other drug utilisation data).
  • Understanding of the context in which the data are collected, which should include – but not be limited to – local diagnostic criteria, local prescribing practices, local prescribing formularies, local coding practices, reimbursement policies, etc.
  • Understanding of real-world data sources, including:
    • common coding terminologies for drug exposure and clinical events,
    • common data models,
    • assessment of data quality (incl. data quality metrics, data quality frameworks, misclassification and missingness, benchmarking),
    • their limitations and the statistical approaches to address them

- Knowledge about appropriate methods to establish meaningful RW evidence


  • Expertise in epidemiological study designs, including traditional designs as well as case-only and case-population designs; studies with primary data collection vs. secondary use of data; prevalent-user vs. incident-user designs, positive and negative control exposures and outcomes; use of active exposure vs. non-exposure comparator groups.
  • Knowledge of mechanisms of bias in observational studies (information bias, selection bias, confounding) and methods to address them at the design and analytical stages (incl. restriction, matching, stratification, modelling, use of propensity score methods, multiple imputation); methods to address unmeasured confounding and time-dependent confounding.
  • Knowledge in handling effect modification, interaction and heterogeneity in observational studies.
  • Expertise in assessing and validating different exposures, outcomes and covariates in observational studies.
  • Knowledge in causal inference methods (incl. missing data handling, target trial emulation and interplay with ICH E9 (R1)).
  • Knowledge in evidence synthesis, meta-analysis and data pooling.
  • Experience in assessing a statistical analysis plan for an RWE study.



« Back