11.1. General considerations
There is a considerable body of literature explaining statistical methods for observational studies but little addressing the statistical analysis plan (SAP). A SAP is a document authored prior to the start of the observational study that presents significant details about how the data will be coded and analysed. While the study protocol will have specified the questions to be addressed by the study and will contain an overview of the statistical methods, the SAP is the document in which the statistics to be calculated, and expected tabular and graphical presentations of the results of the study, are fully described.
Guidance on general principles and justification for the need for a SAP are provided in Design of Observational Studies (P.R. Rosenbaum, Springer Series in Statistics, 2020).
The following objectives of a SAP apply to most studies, including observational studies:
Transparency as to how the analysis will proceed by specifying in advance the methodology that will be applied. A SAP should always be completed prior to start of data analysis. Revisions after the start of the analysis might be possible provided these changes are noted and justified in a revised SAP.
Communication to the study team, especially statisticians, involved in the study. It promotes good planning and efficiency for other stakeholders such as reviewers and the target audience of the study. Readers of observational research might dismiss important findings when they were not prespecified.
Replication so that in the future, for similar studies, the same analytical steps can be performed. The SAP should be sufficiently detailed so that it can be followed and reproduced by any statistician. Thus, it should provide clear and complete templates for each analysis.
A study is generally designed with the objective of addressing a set of research questions. A main component of the study is an initial raw dataset including a set of variables that do not usually provide a direct answer to the questions. The SAP details the statistical calculations that will be performed on these observed data and the patterns of results that will in turn be interpreted.
Pre-specification of statistical and epidemiological analyses can be challenging for data that are not collected specifically to answer the research questions. This is often the case in observational studies where secondary data are used (see Chapter 7.2 Secondary use of data). However, thoughtful specification of the way missing values will be handled or the use of a small part of the data as a pilot set to guide analysis can be useful techniques to overcome such problems. Handling of missing data is further discussed in Chapter 5.3.
Specific to observational studies, strong emphasis will be given to measures applied to control and possibly quantify bias. Avoiding bias in observational studies: part 8 in a series of articles on evaluation of scientific publications (Dtsch Arztebl Int. 2009;106(41):664-8) explains how these main methodological problems can be avoided by careful planning. Factors that may bias the results of observational studies are described in Chapter 5.1.
A feature common to most studies is that some analyses that are not pre-specified will be performed in response to observations in the data to help interpretation of results. It is important to distinguish between such data-driven analyses and pre-specified findings. Post-hoc modifications to the analytical strategy should be duly noted and justified. The SAP provides a confirmation of this process.
Since the decision criteria for the study are specified in terms of the observed values of these detailed statistics, it is worth formulating the SAP at an early stage and, in particular, before any informal inspection of aspects of the data or results that might influence opinions regarding the study hypotheses. Ideally the SAP will be developed as soon as the protocol is finalised.
A particular concern in retrospective studies is that decisions about the analysis should be made blinded to any knowledge of the results. This should be a consideration in the study design, particularly when feasibility assessments are to be performed to inform the design phase. Such feasibility assessments should be independent of the main study results (see Chapter 2).
At any cost, a SAP should always be completed before the data have been unblinded for the statistician. This contributes to the transparency of the study process and confirms that the set of analyses have not been influenced by the data. Making alterations to a planned statistical analysis after seeing the data increases the risk of bias and inflates the probability of type I errors.
A SAP is usually structured to reflect the protocol but will provide more granularity regarding the statistical methodology. Ideally it includes and addresses the following elements in detail:
Objectives and testable hypothesis to answer a well-framed question (see Chapter 2)Defining primary and secondary objectives is important to avoid 'data dredging'. A hypothesis is the product of deductive reasoning, going from general premises to specific results one would expect if those general premises are indeed true. This usually involves a set of possible relationships between a set of variables. It should be clearly stated how each outcome will be measured. Negative findings may be equally important as positive findings.
Formal definitions of study outcomesOutcome variables based on historical data may involve complex transformations to approximate clinical variables not explicitly measured in the dataset used. These transformations should be discriminated from those made to improve the fit of a statistical model. In either case the rationale should be given. In the latter case this will include which tests of fit will be used and under what conditions a transformation will be used. Next to the outcomes, also the variables used in the study need to be further formalised; formatting (e.g., categorisation, dichotomisation), modifications or derivations with a special attention to time-dependent variables (e.g., age, BMI). Another reason to carefully choose the primary outcome is to minimise multiplicity effects. These occur when there are multiple statistical tests needed to assess the primary outcome, which increases the likelihood of finding a false positive.
Study methods addressing the elements of study design (see Chapter 4) and sample sizeThe SAP should make explicit the data source(s) from which the expected variation of relevant quantities and the clinically relevant differences are derived. It should be noted that in observational studies on data that already exist and where no additional data can be collected, sample size is not preclusive and the ethical injunction against 'underpowered' studies has no obvious force provided the results, in particular the 'absence of effect' and 'insufficient evidence', are properly presented and interpreted.
Interim analysesIf considered, interim analyses can be beneficial. Criteria, circumstances and possible drawbacks for performing an interim analysis and possible actions (including stopping rules) that can be taken on the basis of such an analysis should be presented.
Study populationThis section includes a description of the data sources and linkage methods, inclusion and exclusion criteria, withdrawal/follow-up, baseline patient characteristics and potential confounding variables, and full analysis set. In addition, attention should be given to the creation of the study groups/cohorts and definition of subgroups, as applicable.
Analytical methodsThis section should describe effect measures and statistical methods used to address each objective; how the achieved study population will be characterised; handling of confounding, heterogeneity and assessing bias; statistical methods to handle missing data; assessing goodness of fit; sensitivity analyses considered.
Statistical principles including confidence intervals and level of significanceWhen false positives are a greater concern, a smaller confidence interval should be considered. Any planned adjustment of the significance level to control for type 1 error that can arise from comparisons across multiple subgroups or analysis of multiple predictors or outcomes (secondary analyses) should be presented.
Decision criteriaIf decisions are drawn from the study results, a section of the SAP should explain the different outcomes that might be selected for each decision, which statistics influence the decision-making process and which values of the statistics will be considered to support each outcome.
Often statistical analyses will employ standard procedures incorporated in statistical packages that provide outputs seen as implicit decision criteria – for instance default p-values (i.e., 5%) or confidence intervals (i.e., 95%). However, different objectives of the study may require lower or higher strength of evidence – for instance, policy recommendations regarding drug licensing may require a lower chance of false positive decisions than the classical one when deciding whether further investigation is needed for a product safety issue. Hence, consideration of decision-making criteria with explicit reference to the type of decision to be made is beneficial.
For further reading on how to draft a SAP tailored to observational studies, see DEBATE-statistical analysis plans for observational studies (BMC Med Res Methodol. 2019;19(1):233), Guide to the statistical analysis plan (Pediatric Anesthesia. 2019;29:237-42) which provides an exhaustive list of SAP items applicable to both prospective and retrospective observational studies, and The value of statistical analysis plans in observational research: defining high-quality research from the start (JAMA. 2012;308(8):773-4). A good example of a SAP where the main components are included can be found in Necrotizing soft tissue infections - a multicentre, prospective observational study (INFECT): protocol and statistical analysis plan - PubMed (nih.gov) (Acta Anaesthesiol Scand. 2018;62;272-79).