Evidence-based medicine and intraoperative neurophysiology

43.2.1

All evidence is not created equal

Evidence for improved outcomes boils down to evidence for causation, or causal links, between ION and outcomes. Early EBM thought made rigid distinctions between randomized and nonrandomized studies as represented in the familiar evidence pyramid and classes (I, II, III, IV) of evidence quality ( Fig. 43.1 ). Current EBM thought has moved from exclusive consideration of research design (randomized vs. observational) to how a study is conducted and the specific sources of bias, which limit our confidence in treatment effects .

Sources (domains) of bias are different for randomized compared to nonrandomized studies. Furlan et al. . described a domain-based tool for RCTs that has been adopted by the Cochrane Collaboration. For observational studies the ROBINS-I tool was recently developed by a team with experts in causal science . Like the Cochrane tool for RCTs, ROBINS-I uses signaling questions for risk of bias judgments in each domain. ROBINS-I starts with a hypothetical “target” randomized trial for the intervention and outcomes of each observational study, not limited by ethical constraints or practical issues, and then defines risk of bias as a systematic difference between the results of the observational study and those expected from the target trial. Modern domain-based risk of bias assessments recognize that RCTs with important limitations may not support causal links between treatments and outcomes, while observational studies with adjustments for confounding can sometimes provide strong support. The GRADE method for assessing overall quality of evidence provides a useful illustration of how this works .

In GRADE, RCTs start as high-quality evidence. However, with serious limitations (bias, inconsistency, indirectness, imprecision, publication bias), they are downgraded to moderate or low quality ( Fig. 43.1 ). For example, a systematic survey of 40 RCTs of spine surgery interventions found the results were frequently fragile from the sample sizes and small number of outcome events . Adding two events to one of the treatment arms in many of these studies would have eliminated statistical significance! Many of these studies would be downgraded for imprecision by GRADE.

Observational studies in GRADE start as low-quality evidence, but if rigorously conducted with adjustments for confounding are upgraded to moderate or high quality for size of effects, dose–response relationships, or if residual confounding would be expected to decrease treatment effects . A frequently cited example is the use of Warfarin, a blood thinner, to reduce stroke during cardiac valve replacement. A meta-analysis of observational studies with adjustments for suspected confounders found a very strong effect (risk ratio=0.17), which has been considered high-quality evidence for Warfarin-reduced stroke .

43.2.2

The totality of evidence is considered

Science is cumulative, and investigations of causal links between treatments and outcomes consider the totality of evidence available from all studies. The essentials of a study are its patients, interventions, comparison, and outcomes (PICO) . Comparative studies (studies with a control group) are typically (but not always) required to support treatment effects. For ION, O utcomes for P atients with neuromonitoring I ntervention are C ompared to O utcomes for patients without neuromonitoring or without a neuromonitoring modality. Outcomes should be those that people experience (feel physically or mentally) and care about . For ION, these are new or worsened neurologic deficits in the postoperative period and not end of surgery monitoring results (e.g., somatosensory evoked potentials (SEPs) or motor evoked potentials (MEPs) at closing compared to preincision baselines).

Totality of evidence is considered by assessing the quality of evidence for studies with comparable PICOs in a systematic review that can be either in narrative form or quantitative with a meta-analysis. The overall quality of evidence (another way of saying this is “our overall confidence”) for an intervention effect in a systematic review is then often summarized for specific outcomes in summary of findings (SoF) tables. In GRADE the SoF are “outcome centric” where quality of evidence (high, moderate, low, very low) for several or many studies may be different for different outcomes. For ION, new neurologic deficits associated with spinal cord injury and those associated with nerve root injury could be separate outcomes in a SoF table. The quality of evidence in a SoF table might also be different for immediate postoperative outcomes compared to those assessed at follow-up. Missing data from unavailable neurologic exams at follow-up (e.g., 3 months) can limit confidence.

A meta-analysis or statistical synthesis is used when the interventions and outcomes of studies are not too diverse . Meta-analyses will typically obtain a weighted average of the treatment effect across studies, with a confidence interval and significance level. Measures of heterogeneity (differences in treatment effect from one study to the next) are also provided. Meta-analyses are useful as a summary of average treatment effects with increased power and precision compared to an individual study. An example of a meta-analysis forest plot for ION with typical statistics and their interpretation is provided in Fig. 43.2 .

43.2.3

Patient values and quality of evidence determine the value of a treatment or practice

The Institute of Medicine (United States), as directed by the Medicare Improvements for Patients and Providers Act of 2008 , addressed variations in methods for assessing quality of evidence, lack of transparency, and conflicts of interest in the proliferation of guideline statements by professional organizations. In Clinical Practice Guidelines We Can Trust the Institute of Medicine emphasized that the value of a practice depends on both systematic reviews of the quality of evidence and assessments of its benefits and harms. “Clinical practice guidelines (CPGs) fundamentally rest on appraisal of the quality of relevant evidence, comparison of the benefits and harms of particular clinical recommendations, and value judgments regarding the importance of specific benefits and harms .” Like other medical practices, the value of ION depends on the quality of evidence from systematic reviews and patient valuations of the risks, benefits, and economic costs of ION.

Quality of evidence and practice value are different concepts. A good example is bracing in adolescent idiopathic scoliosis. Bracing continues to be used to prevent progression of high-risk curves despite only low or very low quality of evidence for its effectiveness. Recent Cochrane reviews described the obstacles to better evidence from RCTs . Value judgments from both parents and physicians were that the benefits of bracing (less deformity, avoidance of surgery) outweighed any harms. Parents and physicians refused to randomize treatment in many instances and attempts at RCTs for bracing versus simple observation failed.

Like bracing, and as underscored by the Institute of Medicine report, the value of ION depends on an appraisal of its benefits and harms as well as the quality of evidence that supports improved outcomes ( Fig. 43.3 ). Opportunity costs as “benefits lost from the next best alternative” depend on patient diagnosis and type of surgical procedure and remind us to compare the benefits lost without ION to those with ION. The opportunity cost of no ION (benefits lost without ION) for aneurysm, spine intramedullary tumor, and spinal deformity procedures is an expected decreased risk of new, permanent, and severe neurologic deficits and long-term medical expenses . By comparison, the opportunity cost of ION (benefits lost with ION) is relatively small, and includes fewer harms specifically associated with ION (tongue bites, seizures, needle infections, unnecessary changes in surgical strategy from inaccurate ION tests), and no expense for the ION service . For surgical procedures with high opportunity costs, less certainty in evidence for improved outcomes with ION is needed to support its value. The big difference in expected opportunity costs for no ION (vs ION) tends to unbalance or ‘‘break’’ the clinical equipoise required for randomized studies ( Fig. 43.3 , point C). On the other side, clinical equipoise is tipped in favor of randomized studies when there is substantial uncertainty in the evidence estimating ION effects and the opportunity cost difference (no ION vs ION) is low ( Fig. 43.3 , point A).

High-quality evidence from RCTs for high-risk surgeries such as those for pediatric spine tumors and spine deformities will not be forthcoming. Informed parents as well as surgeons will refuse to randomize treatment. A chair of spine surgery at a major medical center recently stated, “As to RCT in AIS (adolescent idiopathic scoliosis) you would not get a single pediatric deformity surgeon in the US to agree to that.”(Peter Sturm, personal communication). On the other hand, for very frequently performed surgeries with lower risks of new neurologic deficits (e.g., cervical decompression for radiculopathy), a large RCT and the resources required for its conduct may be justified if uncertainty (equipoise) remains after well-conducted, comparative observational studies of ION. Uncertainty may be present in the absence of very large benefits (e.g., risk ratios >0.2 for ION), difficult to measure suspected confounders, or imprecision.

The three EBM principles are most fully developed for comparative studies of interventions and, next, will be applied to ION efficacy in improving outcomes. Support for ION efficacy requires demonstration of causal links between ION and outcomes, typically from systemic reviews of comparative studies. Then, these same EBM principles will be applied to ION diagnostic test accuracy (DTA) and postoperative outcomes.