Surgical and procedural risk assessment of patients with valvular heart disease

Goals

Outcomes data from medical procedures are commonly used to compare treatments or providers. Early databases were originally used to assess outcomes from cardiac surgical procedures, most commonly coronary artery bypass grafting (CABG). In the United States, these registries were first constructed from administrative claims data from the Health Care Financing Administration (HCFA), which was the precursor of the Center for Medicare and Medicaid Services (CMS). The purpose of these databases was to assess outcomes in various clinical programs, but they did not account for patient-specific factors that could influence outcomes. ^, The need for patient-specific predictions of procedural outcomes led to the development of several high-quality clinical databases and risk models for patients undergoing cardiac surgery. ^,

Patient outcomes are influenced by severity of illness, treatment effectiveness, and chance, and comparisons between groups must account for differences in prevalence of risk factors, a concept called case mi x. Outcome variations due to case mix can be reduced or eliminated by several methods, most rigorously by randomization, which balance known and unknown risk factors. However, outcomes from randomized, controlled clinical trials may not be generalizable to the larger, unselected population of patients.

Registry data are important for comparing outcomes among various treatments or providers with covariate matching or propensity score matching techniques to account for case mix. ^, With the use of statistical modeling techniques, most commonly multivariable regression analysis, the association between individual risk factors, known as predictor variables or covariates, and outcomes can be determined. After the impact of each risk factor is determined (called weighting ) from a given population sample, it becomes possible to estimate the probability of the outcome for patients having particular combinations of these risk factors.

Construction of risk models

Risk scores are predicted probabilities calculated from a multivariable logistic regression model calibrated on data from a fixed time. The first element in constructing a robust risk model is a clinical database with as complete and accurate data as possible. The second element is risk modeling by experienced statisticians to ensure development of a relevant multivariable model.

Logistic regression modeling has been used for development of the Society of Thoracic Surgeons (STS) risk score and for models constructed by New York State, the Veterans Administration, and the Northern New England Cardiovascular Disease Study Group. Other models, such as the Parsonnet score and the European System for Cardiac Operative Risk Evaluation (EuroSCORE), used simple additive scores with weights derived from logistic regression models. Some evidence indicates that logistic regression models perform better.

For development of a risk model, the study population is usually divided into a development or training sample and a validation or test sample. For the STS Isolated Valve Risk model, the study population was randomly divided into a 60% development sample and a 40% validation sample. The development sample was then used to identify predictor variables and estimate model coefficients. Data from the validation sample were used to assess model fit, discrimination, and calibration.

Discrimination refers to the model‘s ability to separate two groups studied, such as survivors and nonsurvivors. An area under the receiver operating characteristic curve (AUROC) is calculated using the concordance statistic (i.e., C-index), with ranges between 0.5 and 1.0. The higher the value of the C-index, the better the discrimination, whereas values closer to 0.5 indicate that the model’s ability to discriminate is no better than random chance or the flip of a coin. In most risk prediction models used for cardiac surgery, the AUROC is between 0.75 and 0.80.

Limitations

In the employment of risk adjustment, important limitations have to be taken into account to ensure that valid information rather than misinformation is obtained from the correction. First, risk algorithms are accurate only for the population and in the time frame in which they are developed and validated.

Second, risk adjustment loses accuracy at the extremes of the population studied, where there are too few patients on which to build a statistically valid model. This tail of the bell-shaped curve is where high-risk patients with aortic stenosis reside, accounting for some of the overestimation of risk seen with many models. ^,

Third, risk algorithms cannot reliably be applied directly to populations and treatments other than those in which they were developed. The implication is that although both surgical aortic valve replacement (SAVR) and transcatheter aortic valve implantation (TAVI) are used in treating patients with aortic stenosis, AVR risk algorithms are based only on SAVR outcomes and therefore may not to be directly applicable to TAVI.

Fourth, risk algorithms cannot account for variables not collected or analyzed. This lack of accounting has one of two causes: (1) the occurrence of the factor or condition (e.g., porcelain aorta, liver disease) is so infrequent that its impact cannot be measured, or (2) the factor might have not been previously known to be a factor that was causal or cannot be accurately measured or quantified. The role of frailty and its impact on outcomes of treatment is a case in point.

Fifth, all risk predictors fall prey to the phenomenon of “garbage in equals garbage out.” Unless the factors on which the algorithm is formulated are based on complete and accurate data, an inaccurate predictor will result. A corollary is that the risk predictor must be user friendly. The greater number of variables collected in formulation of the risk algorithm, the more accurate the prediction of risk; however, the more burdensome the collection of data required, the less complete and accurate will be the information. There must be a balance between including all information that is likely to be a factor in causing risk and user-friendliness by being least burdensome to facilitate complete and accurate collection and ensure that the tool is routinely employed in decision making. One risk algorithm for aortic stenosis, the Age, Creatinine, Ejection Fraction (ACEF) score, provides reasonable prediction using only the three factors in its name: age, serum creatinine level, and ejection fraction.

Clinical utility

Profiling risk for patients undergoing medical procedures serves many purposes. First, it allows outcomes prediction for individual patients, enabling the patient and caregiver to be better informed in making decisions regarding the advisability and risks of a specific medical procedure. Second, patients undergoing medical procedures frequently have comorbidities that cause various levels of risk, and they therefore can adversely affect the outcomes of a procedure. When different modalities of treatment or different caregivers are compared, risk adjustment allows a balanced analysis of outcomes (i.e., comparative effectiveness) by accounting for the risk factor variation among different patient cohorts. This correction allows for a more level playing field of outcomes assessment, and the ability to achieve an apples-to-apples comparison is one of the advantages of clinical outcomes databases over administrative databases, which have limited ability to adjust risk.

Risk adjustment allows a more meaningful analysis of hospitals or therapies for comparative safety and effectiveness of treatment ( Table 7.1 ). For example, it is possible to compare two standard procedures (e.g., CABG surgery compared with percutaneous coronary intervention) or a new procedure with an existing standard (e.g., TAVI compared with SAVR) for outcomes comparisons in different centers. Public reporting of surgical outcomes in the United States is done by risk-adjusted results, in which the observed outcome divided by the expected outcome is based on known patient risk factors. This approach creates an observed-to-expected ratio (O/E) that is a multiplier of the observed mortality. An O/E ratio of less than 1 indicates a better-than-expected outcome, whereas a ratio greater than 1 means the outcome is worse than expected on the basis of the patient’s existing comorbidities or risk factors. Without the risk adjustment that takes into account the patient-specific factors that may adversely affect outcomes, meaningful comparison is not possible.