Evidence-based medicine and health services research in plastic surgery

Evaluating existing literature

Best evidence requires three things: (1) a study design that is suited to the research question; (2) an appropriate statistical analysis; and (3) clinically meaningful outcome measures. In a thorough review, Offer and Perks have enumerated the challenges that plastic surgeons face when trying to practice evidence-based medicine. They cite the lack of quality evidence in our literature as a major factor. Using Levels of Evidence (LOE) as a surrogate for quality, this finding was substantiated by Burns, Rohrich, and Chung, who in 2011 reported that the majority of articles published in Plastic and Reconstructive Surgery over a 20-year period were Level IV or V evidence (i.e., case series and case reports). The authors concluded that while the level of evidence was improving, there remained a long way to go. In 2019, Sugrue, Joyce, and Carroll confirmed that the quality of plastic surgery research continues to improve, as by their analysis, the percentage of higher LOE studies (Levels I and II) increased from 6.6% in 2008 to 15.7%. However, lower LOE studies (Levels IV and V) still made up half (49.6%) of publications. These Level IV and Level V studies often represent descriptions of procedures by individual surgeons and the “outcomes” of those procedures in that surgeon’s hands. Known as case reports or case series, they have limited applicability to other practices, and are not structured, not compared, and not randomized. This is not only weak evidence on which to make medical decisions, but it is not an acceptable way to make such decisions in this era of healthcare reform and pay-for-performance (P4P). As depicted in Fig. 11.1 , the stronger the evidence, the better the care one is able to deliver.

Literature search strategies

Practicing evidence-based medicine involves: (1) converting the need for information about a diagnosis, prognosis, or intervention into an answerable question, (2) searching for the best available research to answer that question, (3) critically appraising the evidence for its validity, (4) integrating critical appraisal with clinical expertise given a patient’s unique values and circumstances to guide management, and (5) evaluating our efficiency and effectiveness in steps 1 through 4 and seeking ways to improve. Many clinical questions remain unanswered because of problems formulating relevant questions, insufficient access to information resources, and a lack of search skills. Today, there are a variety of strategies and web-based resources that allow searches of the relevant literature to answer many clinical questions. Some examples are provided below.

Systematic reviews

Systematic reviews are evaluations of the literature conducted according to clearly stated, scientific research methods that are designed to minimize the risk of bias and the errors associated with traditional literature reviews. The statistical components of a systematic review (i.e., summary effect estimates, tests of heterogeneity and publication bias) are referred to as a meta-analysis. The review process is based on a comprehensive and unbiased search of the literature using defined criteria and includes a thorough evaluation of the quality and validity of the studies identified in the search process.

The best-known source for systematic reviews is the Cochrane Database of Systematic Reviews. The Cochrane Group also provides a handbook for systematic reviews of interventions which provides a breakdown of core methods for preparing a review. Systematic reviews are influential tools in supporting evidence-based practice and some consider them to provide stronger evidence than randomized controlled trials. Moreover, they are essential for summarizing existing data, thereby avoiding the wasted effort that would result from unnecessary duplication of previous studies. However, a drawback of systematic reviews is that they can produce evidence that appears reliable on first glance, however upon closer inspection if the systematic review is based on the results of individual studies that are of poor methodological quality, this evidence is at a high risk of bias. This can be particularly relevant in systematic reviews of observational data, where often times the results of studies are not adjusted for potential confounding variables. To see examples of rigorously conducted (high-quality) systematic reviews, the reader is referred to the online Cochrane reviews database.

Meta-analysis

Conceptually, a meta-analysis is the summation of multiple quantitative studies in order to increase the sample size, thus increasing the overall power and strengthening the conclusions over that which can be drawn from any individual article for a particular research question. This analysis allows researchers to reach a more reliable conclusion even when there are conflicting results from multiple studies. Ideally, a meta-analysis is conducted using the highest level of evidence, corresponding to the least amount of potential bias, such as randomized controlled clinical trials. Although meta-analysis can also be conducted on cohort studies and even case series, the quality of the evidence and, therefore, the conclusions are weaker. Methods to evaluate risk of bias within meta-analyses are highlighted in the Cochrane handbook for systematic reviews of interventions and include the Cochrane Risk of Bias 2 (RoB 2) tool and the ROBINS-I tool for randomized and non-randomized studies of interventions, respectively.

The meta-analysis has some disadvantages. First, meta-analyses have been criticized for both overly broad inclusion criteria and, conversely, for overly strict inclusion criteria. Either may result in degradation of the results. Inclusion criteria that are overly broad may lead to inclusion of studies of lesser quality or with less reliable results. Very strict inclusion criteria may mean that the studies that are included have limited generalizability. Another disadvantage is that meta-analyses require a significant time commitment from the research team.

The technique of meta-analysis may have limited applicability in plastic surgery, where there are few randomized trials and the outcomes from one study to another may vary considerably. An example of a meta-analysis can be seen in Fig. 11.2 . This figure demonstrates an analysis of randomized trials evaluating the impact of tamoxifen on survival. As the reader can observe, there are multiple studies, with the findings of each plotted on a central axis. The summary analysis, which incorporates data from these studies, appears at the bottom of the vertical axis and suggests a benefit for the use of tamoxifen in early breast cancer. As can be seen, the 95% confidence interval for that summary measure is quite narrow. This implies a better point estimate of the true benefit of tamoxifen. Also note that the quality of the individual study is used to determine its weight in computing the summary score.

More recently, Norman et al . conducted a Cochrane systematic review and meta-analysis of randomized controlled trials comparing the incidence of surgical site infections (SSIs) in patients who receive negative pressure wound therapy (NPWT) versus standard dressings for surgical wounds healing by primary intention. The authors identified 31 studies (6204 patients) that evaluated this outcome across various surgical incisions. When these studies were pooled in the form of a meta-analysis, the results demonstrated a moderate reduction in the incidence of SSI with NPWT (RR 0.66, 95% CI [0.55 to 0.80]) versus standard dressings; corresponding to 43 fewer SSIs per 1000 people. Overall, the authors concluded that “NPWT probably decreases the incidence of surgical site infection compared with standard dressings”. To see additional examples of good-quality meta-analyses, the reader is referred to the online Cochrane reviews.

Network meta-analysis

Traditional systematic reviews seek to compare two interventions (e.g. placebo vs. intervention, intervention A vs. intervention B) in the form of a pairwise meta-analysis. This technique is particularly useful when comparing a new intervention to placebo or to standard of care from direct head-to-head clinical trials. Notably, a limitation of this analysis is that it does not permit the comparison of several interventions to determine relative effectiveness. This is particularly relevant in the field of plastic surgery where several competing techniques may be used to address the same problem. For example, when comparing interventions for the treatment of keloid scarring a surgeon may compare the relative effectiveness of various interventions on the incidence of lesion recurrence, including: (1) surgical excision alone, (2) surgical excision with skin grafting, (3) surgical excision with pressure dressing, (4) surgical excision with radiation, and (5) surgical excision with adjuvant steroid. Traditional pairwise meta-analyses do not allow for conclusions to be made regarding the relative effectiveness of these treatments unless they have been compared directly in head-to-head trials. Unfortunately, direct comparisons of multiple interventions are infrequently reported within the surgical literature.

Another form of meta-analysis, known as the “network meta-analysis”, was developed to address this limitation. Specifically, network meta-analyses seek to provide an effect size estimate for all possible pairwise comparisons, irrespective of whether these interventions have been compared directly within traditional randomized or observational head-to-head study designs. This technique proposes that one can estimate the relative effect of two interventions (i.e., treatment A and treatment B), even if not compared directly, if both interventions have been directly compared to a similar third intervention (i.e., treatment A vs. treatment C; treatment B vs. treatment C). These techniques rely on the assumption that all relevant studies have similar key characteristics (i.e., patient inclusion criteria, outcome measurement and risk of bias) such that they can be combined to make both direct and indirect comparisons. For example, a network meta-analysis evaluating the effect of various interventions on keloid scar recurrence by Siotos et al . concluded that the odds ratios (OR) of keloid recurrence relative to no excision were as follows: excision with pressure (OR 0.18 [95% CI, 0.01–7.07]); excision with 2 adjuvants drugs (OR 0.47 [95% CI, 0.02–12.82]); excision with radiation (OR 0.39 [95% CI, 0.04–3.31]); excision and skin grafting (OR 0.58 [95% CI, 0.00–76.10]); excision with 1 adjuvant drug (OR 1.76 [95% CI, 0.17–21.35]); and excision alone (OR 2.17 [95% CI, 0.23–23.95]).

By including effect estimates from direct and indirect comparisons, network meta-analyses enable researchers and clinicians to simultaneously compare multiple interventions and rank treatments. While guidance on how to perform a network meta-analysis remains beyond the scope of this chapter, we refer readers to the “Undertaking Network Meta-analyses” chapter in the Cochrane Handbook for Systematic Reviews . We strongly recommend consultation with a health research methodologist prior to embarking on this study design as these techniques can be challenging to undertake and interpret without formal training.

Experimental studies: Randomized controlled trials

The randomized controlled trial (RCT) is perhaps the most complex of the experimental study designs and is viewed as the gold standard. The primary advantage of a trial over other study designs is “randomization”. Specifically, randomly assigning an intervention minimizes the influence of known and unknown confounding variables that may bias the study results. Furthermore, “blinding” functions to preserve the benefits of randomization and prevents bias in the adjudication of the intended outcome. The design of the RCT evolved over the course of many years and each component of the design attempts to minimize the influence of study bias and confounders on the results. The goal is to create a sample population that is truly representative of the whole, often referred to as the “external validity” of a study. In this way, the results of a limited trial can be generalized to the population at large with confidence.

According to the National Institutes of Health, clinical trials are conducted in four phases, each serving a different purpose and helping researchers answer different questions ( Table 11.2 ). In phase I, researchers evaluate an experimental drug or intervention in a small number of test subjects (20–80) to evaluate its safety, determine a dose–response curve, and identify potential side-effects. In phase II, the experimental drug or intervention is given to a larger group of subjects (100–300) to test its effectiveness and further evaluate its safety profile. In phase III, the experimental drug or intervention is given to a much larger group of subjects (1000–3000) in order to “confirm its effectiveness, monitor side effects, compare it to commonly used treatments, and collect information that will allow the experimental drug or treatment to be used safely”. In phase IV, post-marketing surveillance is continued to identify additional information on the risks, benefits, and optimal use of the intervention.

Observational studies

Given the limitations of RCTs to answer questions about harm, there are alternative strategies used to minimize bias, which involve observational study designs. The two main types of observational studies are cohort and case–control. The cohort design is similar to an RCT but without randomization. Instead, it is patient or physician preference, or sometimes happenstance, that determines whether the patient received the exposure of interest. Although there is widespread belief that observational studies are less valid, often overestimating the magnitude of treatment effects, there are at least several reports suggesting that good observational studies can provide the same level of internal validity as randomized trials.

Cohort studies

Cohort studies evaluate a group or cohort at risk for disease. The evaluation may occur over time or at a single point in time. The latter is further described as a cross-sectional cohort study and is often used to collect data on the prevalence of disease. The second common use of cohort studies is to analyze the exposures that put subjects at risk for disease or interventions that reduce that risk. This requires an evaluation of the cohort at a minimum of two points in time. The study may be designed prospectively ( Fig. 11.3 ) or retrospectively ( Fig. 11.4 ). In the prospective cohort study, the investigator develops a hypothesis about variables that may impact the outcome under investigation. The investigator then identifies exposed and nonexposed groups of patients, each being a cohort, collects data about their risk factors, and then follows the cohorts forward in time, monitoring for development of the outcome of interest. For example, a prospective cohort study was used to compare the incidence of lymphedema post mastectomy in women with or without breast reconstruction. The results support the conclusion that breast reconstruction does not increase the risk of lymphedema after 10 years follow-up.

While prospective studies lend credence to causal associations, they are expensive and generally require years of follow-up. Rare outcomes or those that take a long time to develop affect the feasibility of cohort studies. Retrospective studies have similar goals, but identify the cohort at risk in the present and investigate the risk factors or exposures that occurred in the past. This is often done by subject recall or chart analysis, both of which are flawed when compared to prospective data collection. Data collected in this manner are more likely to be incomplete, inaccurate, inconsistent, or subject to recall bias. On the other hand, the major advantages of retrospective studies are that they can be done in a relatively short timeframe and are much less costly than prospective studies.

Case–control studies

Case–control studies also assess associations between exposures and outcomes. Case–control studies ( Fig. 11.5 ) are an alternative design from cohort studies and rely on the initial identification of cases. The first group (cases) is selected by the presence of disease (or outcome) and the second (control) are individuals who do not have the disease (or outcome) of interest. The major strength of the case–control study is in the investigation of rare conditions or outcomes that take an extended duration to occur. Its weakness lies in the inability to assess incidence or prevalence of disease and its increased susceptibility to bias. Controls may be concurrent or historic, matched (on key variables such as age and sex) or unmatched.