Annex 2 has not been updated for Revision 8
3. Data sources
This Annex provides recommendations on methods for measuring the impact of pharmacovigilance activities on patients and public health.
Pharmacovigilance activities aim to protect patients and promote public health by leading to changes in the knowledge and behaviour of individuals (e.g. patients, consumers, caregivers and healthcare professionals) and in healthcare practice. Impact research aims to generate evidence about the effect of these activities, the effect being direct or indirect, intended or unintended.
The European Medicines Agency’s Pharmacovigilance Risk Assessment Committee (PRAC) launched a strategy aiming to improve safety monitoring, communication and risk management practices for medicinal products and to determine which activities are the most effective (1). An important component of the strategy is a methodological approach for measuring the impact of pharmacovigilance activities by delivering data on the effectiveness of major regulatory actions targeted to specific products and therapeutic classes.
Pharmacovigilance activities have been so far examined predominantly for their impact on processes of healthcare delivery, such as prescription patterns following changes of the product information. A dual evidence-based approach to evaluating the effectiveness of risk minimisation measures (2) combines quantitative measures of the extent of implementation (process indicators) with measures of frequency of outcomes of interest (outcome indicators) before and after implementation. This approach is a key element of risk management to establish whether an intervention has been effective or not, and which corrective actions might be warranted in line with Good Pharmacovigilance Practices module XVI (3).
A qualitative review of effectiveness studies published in the EU PAS Register® (http://www.encepp.eu/encepp/studiesDatabase.jsp) showed that cross-sectional survey studies typically used process indicators whereas studies using secondary data sources focused on outcome indicators, but only about half of the indicators were reported as successful with conclusive results (4). There are few examples of assessment of the impact on patient-relevant health outcomes. If performed systematically, such measure would underpin the key role of pharmacovigilance systems and provide opportunities to focus resources on activities that make a difference to patients.
Measuring the impact of pharmacovigilance activities may be challenging as these activities may target stakeholder groups at different levels of the health care system, use several tools applied simultaneously or over time to deliver information and influence behaviour, and induce unintended effects. It is also challenging to differentiate between the effects of individual pharmacovigilance activities and other simultaneous events such as media attention, publications in scientific journals, changes in clinical practice, or secular trends in health outcomes. Figure 1 shows a model of how various pathways and effects of pharmacovigilance activities are intertwined and influence each other.
Figure 1: Model of pathways and effects of pharmacovigilance activities (PASS: post-authorisation safety study)
Outcomes to be studied in impact research are closely tied to the nature and objective of the pharmacovigilance activities. Because regulatory actions are mostly tailored to individual medicinal products, there is no standard outcome that could be measured for each activity and the concepts outlined in this chapter need to be applied on a case-by-case basis (5).
Outcome measures provide an overall indication of the level of risk reduction that has been achieved with a specific risk minimisation measure in place. This may also require measuring potential outcomes not linked to the specific medicinal product but representing possible unintended consequences of regulatory interventions e.g. change of drug use in a population leading to less favourable health outcomes (6).
Relevant outcomes may include: timelines and outreach of implementation of risk minimisation material; changes in knowledge, perception, behaviour or clinical practice; drug utilisation patterns (e.g. prescription or dispensing rates, contraindicated co-prescription, use of treatment alternatives); and health outcomes (6). Health outcomes should preferably be measured directly. They may include clinical outcomes such as all-cause mortality, congenital defects or other conditions that prompted the pharmacovigilance activity. Direct measurement of health outcomes is not always feasible or may not be necessary, for example when it can be replaced with indirect measures based on already collected health care data. Indirect surrogate measures may use data on hospitalisations, emergency department admissions or laboratory values e.g. blood pressure as a surrogate for cardiac risk (7). An example of use of a surrogate measure has been given by Shi et al. who measured glycaemic outcomes (HbA1C change from baseline) in patients with diabetes mellitus using the Veterans Integrated Services Network database. The results confirmed a 45% discontinuation of thiazolidinedione use in the veterans’ population and a worsening of glycaemic control following safety warning publicity in 2007, which may have driven the decline in usage of this class of medicines. One in ten patients stopping thiazolidinedione medication did not receive any other antidiabetic medication and had worse glycaemic control compared with those who had continued (8).
Depending on the nature of the safety concern and the regulatory intervention or when the assessment of outcomes indicators is unfeasible (e.g. inadequate number of exposed patients, very rare adverse events), process indicators can also be used for impact research (9).
The impact of pharmacovigilance activities can be measured using both primary and secondary data collection, although the literature shows that the latter is more commonly used (6). Chapter 4 of the ENCePP Guide provides a general description of the main characteristics, advantages and disadvantages of various data sources.
Primary data collection by means of qualitative surveys (see Chapter 4.1.1.) is useful to collect information about the impact of pharmacovigilance activities on knowledge, risk perceptions and behaviour of health care professionals as well as patients.
In a primary care setting, relevant information can be derived from extraction of secondary data captured in electronic medical records (EMR), health and social care databases, prescription databases and administrative insurance claims databases set up for the reimbursement of health care (10–12).
The usefulness of each dataset depends however on the original purpose of the data collection and its content. Drugs that can be purchased over-the-counter (OTC) without a medical prescription or are not reimbursed are rarely covered in secondary data sources. A relevant example is the measurement of the desired reduction of paracetamol use following concerns about the risks of death and liver damage mostly associated with high dosage, as paracetamol use is incompletely captured in most primary care databases. Another example concerns drug utilisation in hospitals, which is rarely captured in the most commonly used EMRs and only captured using secondary/tertiary care medical records such as the Clinical Record Interactive Search (CRIS) (13) or the Utrecht Patient Oriented Database (14).
The impact of pharmacovigilance activities should be interpreted in light of the limitations of the data sources used for the assessment (15). Researchers should have a clear view of the limitations of the different data sources when planning their research, and assess whether these limitations could impact the results in one direction or the other in a such a way that their interpretation may be significantly influenced, for example due to bias or unmeasured confounders. As for all observational studies, the evaluation of the usefulness and limitation of a given data source for the study requires a very good understanding of the research question.
Different databases are unlikely to capture all impact–relevant outcomes, even when they are linked to one another. Data of good quality may be available on hard outcomes such as death, hospital admission, emergency room visit or medical contacts. Conversely, claims databases rarely capture primary care diagnoses, symptoms, conditions or other events that do not lead to a claim, such as suicidal ideation, abuse or misuse. An accurate definition of the outcomes also often requires the development of algorithms that need validation in the database that will be used for impact measurement.
One study reported that only about 50% of the less serious drug-related problems listed in the product information are recorded in patient notes (16). If generalisable to electronic data sources, this result would indicate that incomplete recording of patient-reported outcomes of low severity may reduce the likelihood of identifying some benefits and unintended consequences of a pharmacovigilance activity, for example a change in the frequency of occurrence of an ADR. Combining different approaches integrating a patients’ survey would be necessary to overcome this situation.
Missing information on vulnerable populations such as pregnant women, and missing mother-child or father-child links is a significant barrier to measuring the impact of paternal/maternal exposure or behaviour. For example, the impact of pregnancy prevention programmes could not be accurately assessed using European databases that had been used to report prescribing in pregnancy (17). This was largely due to inadequate data on planned abortions and exposure to oral contraceptives.
Depending on the initial purpose of the data source used for impact research, information on potential confounders may be missing, such as indication of drug use, co-morbidities, co-medication, smoking, diet, body mass index, family history of disease or recreational drug use. Missing information may impair a valid assessment of risk factors for changes in health care practice but this limitation should be considered in light of the research question at stake. In some settings, record linkage between different types of data sources including different information could provide comprehensive data on the frequency of ADRs and potential confounders (18-20).
There are a few examples of randomised controlled trials evaluating the impact of pharmacovigilance activities (21, 22), and most studies are observational. In this context, the methodological considerations for study design and methods described in chapter 5 of the ENCePP Guide on Methodological Standards in Pharmacoepidemiology apply to impact research.
Important components of the research question for impact studies are the clinical setting, the type of safety concern (identified risk, potential risk or missing information) and the legal basis for the planned pharmacovigilance study (9). It is also important to account for the timing of implementation of the pharmacovigilance actions in the concerned countries (e.g. distribution of the risk minimisation material), the definition of the outcome(s) to be measured, the definition of the ‘target group’ and potential confounders such as changes in health care or other changes of risk factors in the population.
A design frequently used in impact studies is the comparison of the period before and the period after an intervention occurred. Before/after time series can be used to analyse changes in trends in incidence or prevalence of an outcome (e. g. number of new prescriptions for a specific drug or incidence of an ADR) at several time intervals before and after the intervention (6, 23). This design requires specifying the date (e.g. month) of the intervention. If the date is not precisely known, it should be defined arbitrarily, for example as the date at which the regulatory decision was published. It should be kept in mind that all time-series approaches may be affected by simultaneously occurring interventions or events (e.g. media coverage, pricing policy changes etc.) during the same time period. Other observational study designs such as cohort studies or case-control studies may also be appropriate, provided that the study designs are appropriately adapted to the research question.
A comparator group that did not receive the intervention may be used to facilitate the interpretation of any association found between the intervention and the change in trends, by discriminating between the regulatory intervention and simultaneous events such as media attention, publications, regulatory actions in similar therapeutic areas, guideline changes or secular trends. Despite this advantage, a systematic review showed that 13.7% (21/153) of impact studies still fail to use a comparator or trend analysis (6). In practice, a good comparator group is actually not easy to identify because, for obvious ethical reasons, a pharmacovigilance activity aiming to reduce risks should be applied simultaneously to the entire target population. Possible options for comparator groups may be countries, regions or institutions where the pharmacovigilance measure was not yet implemented at local healthcare level or where alternative medicinal products are prescribed and are not affected by the measure.
Within the boundaries of the required underlying assumptions, a self-controlled case-series design may also be used and does not require a comparator group (see Chapter 5.3.2.).
The analytical method to be applied in impact research depends on the study design and approach to data collection. Various descriptive analyses have been used to assess the impact of a regulatory guidance (6, 24, 25).
Before/after time series studies are usually analysed with regression techniques. An interrupted time series (ITS) regression is the strongest analytical tool to assess the impact of an intervention (26). Examples of studies which used the ITS approach include: the impact of FDA black box advisory on antipsychotic medication use (27); spill-over effects on treatment of adult depression in primary care after FDA advisory on risk of paediatric suicidality with SSRIs (28); long term effect of reduced pack sizes of paracetamol on poisoning deaths and liver transplant activity in England and Wales: interrupted time series analyses (29).
The use of ITS regression in impact research has increased over recent years and recommendations on methodological aspects such as autocorrelation and adjusting for seasonality, and on reporting have been published (30). ITS regression requires that the time point (or period) of the intervention is known prior to the analysis and sufficient data points are collected before and after the intervention for adequate power. If the date of the intervention is not known, Joinpoint regression models offer an alternative by calculating trend line changes at time points where the intervention led to changes in the outcome of interest (31, 32). For example, Joinpoint regression analysis was used to investigate the effect of scientific publications, FDA advisories and media exposure on glitazone use (33) and to measure the population health impact of the fall of hormone replacement therapy in England following the results of the women’s health initiative (WHI) (34).
Metrics such as “Population Impact Number of Eliminating a Risk factor over time t” (PIN-ER-t), and “Number of Events Prevented in a Population” (NEPP) have proven valuable in assessing the impact of removing a risk factor on public health, and may be useful in assessing impact of regulatory interventions. Illustrative examples for population impact analyses include: the potential population impact of changes in heroin treatment and smoking prevalence rates using population impact measures (35); a framework for assessing the population impact of a risk or intervention (36); assessing the population impact of low rates of vitamin D supplementation on type 1 diabetes using a new statistical method (37). Further, statistical analysis using impact metrics is possible where proxy measures are used to assess the impact that one event or resource has on another, as shown for communicating risks at the population level by applying population impact numbers (38), the benefit-risk case study report for rimonabant in IMI Work Package 5 (39), or with a framework for assessing the population impact of a risk or intervention (40).
Studies without regression modelling may be suitable for large immediate changes (e.g. product withdrawals), but they risk producing spurious results when the changes are more subtle or multiple confounders are present. Descriptive analysis alone may lead to misinterpretation of trends.
Predictive modelling techniques may provide an insight into future impacts of regulatory actions. Modelling the risk of adverse reactions leading to product withdrawal alongside drug utilisation data can assess the number of patients at risk of experiencing the adverse reactions per year, and provide an estimate of the number of patients per year which are protected from as a result of regulatory action (40, 41).
Pharmacovigilance activities may have both intended and unintended effects and impact studies should measure both. Unintended effects may not be expected at the design stage of an impact study and require specific analytical approaches to determine the net attributable impact of pharmacovigilance activities on the totality of patient outcomes. It may never be possible to identify all consequences, but such contingencies should be incorporated into the design of impact research.
Regulatory interventions such as withdrawals or restrictions of use of medicine can lead to therapeutic switching to alternative medicines which may have patient or public health implications not foreseen at the time of intervention. Examples of these have been reported in the USA, where black box warnings on psychotropic drug use in elderly patients with dementia resulted in increased prescribing of benzodiazepines and anti-dementia medicines, but resulted in little change in antipsychotic use (42). Another example concerns the unintended increased use of conventional antipsychotics in two European countries after the introduction of EU risk minimisation measures for the risk of stroke and all-cause mortality with atypical antipsychotic drug use (43). Further, prescribers may extrapolate warnings for one group of patients to other groups (spill-over effects), although they may not share the same risk factors. In 2003, the FDA warned of an association between SSRI prescription and suicidality in paediatric patients (<18). Subsequently, the number of prescriptions of SSRIs in newly diagnosed adult patients fell without compensation by alternative medicines or treatment (28).
Socio-economic factors may also play an important role in implementing regulatory interventions at local level. It has been suggested that practices in affluent communities are more likely to implement regulatory interventions faster than over-stretched or under-resourced practices in more deprived communities and that permanent changes in daily practice in these communities may take longer (44, 45).
Both health care service providers and users may circumvent or ‘work round’ restrictions. Where medicines are restricted or restrictions are perceived as inconvenient, patients may turn to buying medicines over the internet, self-medicating with over-the-counter medicines or using herbals or other complementary medicines. Healthcare professionals may subvert requirements for additional documentation by realigning diagnostic categories (46) or switch to medicines where patient monitoring is not mandated (47). The effects of progressive dextropropoxyphene withdrawal in the EU since 2007 on prescribing behaviour showed an increased use of same level analgesics but also an increased use of paracetamol as monotherapy. Aggregated dispensation data suggested that the choice of analgesics depended on physician speciality, healthcare setting, indication, patients’ comorbidities and age, underlining the complexity and international differences in pain management (48).
Other challenges will need to be addressed by impact research: the identification of long-term consequences of pharmacovigilance actions (49), the definition of thresholds for successful risk minimisation activities that may be translated into clinically meaningful information, or the identification of various types of data, advanced study designs and new methodologies that would be needed for a comprehensive evaluation of impact on patients and public health.
In addition, a more structured approach for how to measure impact of pharmacovigilance actions is needed. One way forward is to develop a framework for a standardised assessment of impact of pharmacovigilance activities. A framework for evaluating the effectiveness of risk minimisation measures has been developed and tested which is based on four domains: data, knowledge, behaviour and outcomes (50). A checklist for assessing the reporting and the quality of studies evaluating risk minimisation measures (RIMES) has been developed recently and its application might be a starting point for improving transparency of the methods applied and appraisal of the evidence (51). Further testing of this method is needed to ascertain its usefulness in regulatory practice.
The text of this Annex will be revised annually and as needed in parallel with new information on methods for impact research emerging from the literature and regulatory practice.
1. Pharmacovigilance Risk Assessment Committee (PRAC). PRAC Strategy on Measuring the Impact of Pharmacovigilance Activities (Rev 1) (EMA/165407/2017) [Internet]. 2017 [cited 2018 Apr 10]. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2016/01/WC500199756.pdf
2. Prieto L, Spooner A, Hidalgo-Simon A, Rubino A, Kurz X, Arlett P. Evaluation of the effectiveness of risk minimization measures: EFFECTIVENESS OF RISK MINIMIZATION. Pharmacoepidemiol Drug Saf. 2012 Aug; 21(8):896–9.
3. European Medicines Agency (EMA). Guideline on good pharmacovigilance practices (GVP) Module XVI – Risk minimisation measures: selection of tools and effectiveness indicators (Rev 2) [Internet]. 2017 [cited 2018 Feb 3]. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/02/WC500162051.pdf
4. Vora P, Artime E, Soriano-Gabarró M, Qizilbash N, Singh V, Asiimwe A. A review of studies evaluating the effectiveness of risk minimisation measures in Europe using the European Union electronic Register of Post-Authorization Studies. Pharmacoepidemiol Drug Saf [Internet]. 2018 Apr 16 [cited 2018 Apr 19]; Available from: http://doi.wiley.com/10.1002/pds.4434
6. Goedecke T, Morales DR, Pacurariu A, Kurz X. Measuring the impact of medicines regulatory interventions - Systematic review and methodological considerations: Methods for measuring impact of medicines regulatory interventions. Br J Clin Pharmacol. 2018 Mar; 84(3):419–33.
7. CIOMS Working Group IX. Practical approaches to risk minimisation for medicinal products. Geneva 2014. [Internet]. Available from: https://cioms.ch/working_groups/risk-minimisation/
8. Shi L, Zhao Y, Szymanski K, Yau L, Fonseca V. Impact of thiazolidinedione safety warnings on medication use patterns and glycemic control among veterans with diabetes mellitus. J Diabetes Complications. 2011 May; 25(3):143–50.
9. European Medicines Agency (EMA). Guideline on good pharmacovigilance practices (GVP) Module VIII – Post-authorisation safety studies (Rev 3) [Internet]. 2017 [cited 2018 Mar 2]. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2012/06/WC500129137.pdf
10. Sanfélix- G, Cervera-Casino P, Peiró S, López-Valcarcel BG, Blázquez A, Barbera T. Effectiveness of safety warnings in atypical antipsychotic drugs: an interrupted time-series analysis in Spain. Drug Saf. 2009; 32(11):1075–87.
11. Franchi C, Tettamanti M, Marengoni A, Bonometti F, Pasina L, Cortesi L, et al. Changes in trend of antipsychotics prescription in patients treated with cholinesterase inhibitors after warnings from Italian Medicines Agency. Results from the EPIFARM-Elderly Project. Eur Neuropsychopharmacol J Eur Coll Neuropsychopharmacol. 2012 Aug; 22(8):569–77.
12. Schulze J, van den Bussche H, Glaeske G, Kaduszkiewicz H, Wiese B, Hoffmann F. Impact of safety warnings on antipsychotic prescriptions in dementia: nothing has changed but the years and the substances. Eur Neuropsychopharmacol J Eur Coll Neuropsychopharmacol. 2013 Sep; 23(9):1034–42.
13. Jackson RG, Patel R, Jayatilleke N, Kolliakou A, Ball M, Gorrell G, et al. Natural language processing to extract symptoms of severe mental illness from clinical text: the Clinical Record Interactive Search Comprehensive Data Extraction (CRIS-CODE) project. BMJ Open. 2017 Jan; 7(1):e012012.
14. ten Berg MJ, Huisman A, van den Bemt PMLA, Schobben AFAM, Egberts ACG, van Solinge WW. Linking laboratory and medication data: new opportunities for pharmacoepidemiological research. Clin Chem Lab Med. 2007; 45(1):13–9.
16. Jordan S, Gabe-Walters ME, Watkins A, Humphreys I, Newson L, Snelgrove S, et al. Nurse-Led Medicines’ Monitoring for Patients with Dementia in Care Homes: A Pragmatic Cohort Stepped Wedge Cluster Randomised Trial. Briel M, editor. PLOS ONE. 2015 Oct 13; 10(10):e0140203.
17. Charlton RA, Bettoli V, Bos HJ, Engeland A, Garne E, Gini R, et al. The limitations of some European healthcare databases for monitoring the effectiveness of pregnancy prevention programmes as risk minimisation measures. Eur J Clin Pharmacol. 2018; 74(4):513–20.
18. Jordan S, Morris JK, Davies GI, Tucker D, Thayer DS, Luteijn JM, et al. Selective Serotonin Reuptake Inhibitor (SSRI) Antidepressants in Pregnancy and Congenital Anomalies: Analysis of Linked Databases in Wales, Norway and Funen, Denmark. PloS One. 2016; 11(12):e0165122.
21. Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, Davis B, et al. Comparison of Upper Gastrointestinal Toxicity of Rofecoxib and Naproxen in Patients with Rheumatoid Arthritis. N Engl J Med. 2000 Nov 23; 343(21):1520–8.
22. Home PD, Pocock SJ, Beck-Nielsen H, Curtis PS, Gomis R, Hanefeld M, et al. Rosiglitazone evaluated for cardiovascular outcomes in oral agent combination therapy for type 2 diabetes (RECORD): a multicentre, randomised, open-label trial. The Lancet. 2009 Jun 20; 373(9681):2125–35.
23. Rockers PC, Røttingen J-A, Shemilt I, Tugwell P, Bärnighausen T. Inclusion of quasi-experimental studies in systematic reviews of health systems research. Health Policy Amst Neth. 2015 Apr; 119(4):511–21.
24. Nkeng L, Cloutier A-M, Craig C, Lelorier J, Moride Y. Impact of regulatory guidances and drug regulation on risk minimization interventions in drug safety: a systematic review. Drug Saf. 2012 Jul 1; 35(7):535–46.
28. Valuck RJ, Libby AM, Orton HD, Morrato EH, Allen R, Baldessarini RJ. Spillover effects on treatment of adult depression in primary care after FDA advisory on risk of pediatric suicidality with SSRIs. Am J Psychiatry. 2007 Aug; 164(8):1198–205.
29. Hawton K, Bergen H, Simkin S, Dodd S, Pocock P, Bernal W, et al. Long term effect of reduced pack sizes of paracetamol on poisoning deaths and liver transplant activity in England and Wales: interrupted time series analyses. BMJ. 2013 Feb 7; 346:f403.
30. Jandoc R, Burden AM, Mamdani M, Lévesque LE, Cadarette SM. Interrupted time series analysis in drug utilization research is increasing: systematic review and recommendations. J Clin Epidemiol. 2015 Aug; 68(8):950–6.
34. Martin RM, Wheeler BW, Metcalfe C, Gunnell D. What was the immediate impact on population health of the recent fall in hormone replacement therapy prescribing in England? Ecological study. J Public Health Oxf Engl. 2010 Dec; 32(4):555–64.
36. Verma A, Torun P, Harris E, Edwards R, Gemmell I, Harrison RA, et al. Population Impact Analysis: a framework for assessing the population impact of a risk or intervention. J Public Health Oxf Engl. 2012 Mar; 34(1):83–9.
37. Zipitis CS, Mughal ZM, Clayton PE. Assessing the population impact of low rates of vitamin D supplementation on type 1 diabetes using a new statistical method. JRSM Open. 2016 Nov; 7(11):2054270416653522.
39. IMI Work Package 5. IMI Work Package 5: Report 1:b:i Benefit - Risk. Wave 1 Case Study Report: Rimonabant 11/10/2011 [Internet]. 2018 [cited 2018 Mar 2]. Available from: http://www.imi-protect.eu/documents/JuhaerietalBenefitRiskWave1CasestudyreportRimonabantOct2011.pdf
40. Lynn E, Lane S, Shakir S. Measuring the impact of product withdrawals and other major pharmacovigilance actions on public health burden in the EU: Methodological considerations. Drug Saf. 2017; 40(10):953–4.
41. Lane S, Lynn E, Shakir S. Investigation assessing the publicly available evidence supporting postmarketing withdrawals, revocations and suspensions of marketing authorisations in the EU since 2012. BMJ Open. 2018 23; 8(1):e019759.
42. Singh RR, Nayak R. Impact of FDA Black Box Warning on Psychotropic Drug Use in Noninstitutionalized Elderly Patients Diagnosed With Dementia: A Retrospective Study. J Pharm Pract. 2016 Oct; 29(5):495–502.
43. Sultana J, Fontana A, Giorgianni F, Pasqua A, Cricelli C, Spina E, et al. The Effect of Safety Warnings on Antipsychotic Drug Prescribing in Elderly Persons with Dementia in the United Kingdom and Italy: A Population-Based Study. CNS Drugs. 2016 Nov; 30(11):1097–109.
46. Kendrick T, Stuart B, Newell C, Geraghty AWA, Moore M. Changes in rates of recorded depression in English primary care 2003–2013: Time trend analyses of effects of the economic recession, and the GP contract quality outcomes framework (QOF). J Affect Disord. 2015 Jul; 180:68–78.
47. Mohammad I, Korkis B, Garwood CL. Incorporating Comprehensive Management of Direct Oral Anticoagulants into Anticoagulation Clinics. Pharmacother J Hum Pharmacol Drug Ther. 2017 Oct; 37(10):1284–97.
50. Nyeland ME, Laursen MV, Callréus T. Evaluating the effectiveness of risk minimisation measures: the application of a conceptual framework to Danish real-world dabigatran data: Evaluating the effectiveness of risk minimisation. Pharmacoepidemiol Drug Saf. 2017 Jun; 26(6):607–14.
51. Smith MY, Russell A, Bahri P, Mol PGM, Frise S, Freeman E, et al. The RIMES Statement: A Checklist to Assess the Quality of Studies Evaluating Risk Minimization Programs for Medicinal Products. Drug Saf. 2018 Apr; 41(4):389–401.