Validation of the Mayo Clinic Risk Score for In-Hospital Mortality After Percutaneous Coronary Interventions Using the National Cardiovascular Data RegistryCLINICAL PERSPECTIVE
Background— We sought to validate the recently developed Mayo Clinic Risk Score model for in-hospital mortality after percutaneous coronary intervention using an independent data set. The Mayo Clinic Risk Score has 7 simple clinical and noninvasive variables, available before coronary angiography, for prediction of in-hospital mortality. External validation using an independent data set would support broader applicability of the model.
Methods and Results— In-hospital mortality after percutaneous coronary intervention on 309 351 patients from the National Cardiovascular Data Registry admitted from January 1, 2004, to March, 30, 2006, was studied. Using the Mayo Clinic Risk Score equation, we assigned predicted probabilities of death to each patient. The area under the receiver-operating characteristics curve was 0.884, indicating excellent discrimination overall as well as among subgroups, including gender, diabetes mellitus, renal failure, low ejection fraction, different age groups, and multivessel disease. Ninety-seven percent of patients undergoing percutaneous coronary intervention had a Mayo Clinic Risk Score <10, indicating low to intermediate risk. The Mayo Clinic Risk Score model initially slightly underpredicted event rates when applied in National Cardiovascular Data Registry data (observed 1.23% versus predicted 1.10%), but this underprediction was corrected after recalibration. The recalibrated risk score discriminated (c index=0.885) and calibrated well in an National Cardiovascular Data Registry validation data set consisting of procedures performed between April 1, 2006, and March 30, 2007.
Conclusions— Seven variables can be combined into a convenient risk scoring system before coronary angiography is performed to predict in-hospital mortality after percutaneous coronary intervention. This model may be useful for providing patients with individualized, evidence-based estimates of procedural risk as part of the informed consent process before percutaneous coronary intervention.
Received November 28, 2007; accepted June 12, 2008.
With increasing operator experience, refinements in technology, and the availability of improved stent designs, percutaneous coronary intervention (PCI) is now used for many high-risk subgroups in which it was previously contraindicated.1–3 In an effort to increase the efficiency of care, PCI is often conducted in an ad hoc fashion after a coronary angiogram is performed. As such, there is often a need to have obtained consent for the PCI before the angiogram and the use of sedatives during the diagnostic procedure. Predictive models can assist patients and clinicians by providing an empirical estimate of procedural risks required for decision making and informed consent. However, existing PCI risk models are complex and include subjective covariates and/or angiographic variables that limit their routine clinical use. To meet the need of estimating the risks of PCI before the coronary anatomy is known, we previously developed a new Mayo Clinic Risk Score (MCRS) for in-hospital mortality after PCI.4 This model is based solely on preprocedural clinical and noninvasive assessments and is less reliant on subjective assessment and angiographic data. It can potentially serve as a risk assessment aid to patients and physicians before patients undergo coronary angiography for PCI.
Clinical Perspective see p 36
This internally validated model includes 7 clinical and easily obtainable noninvasive laboratory variables. To date, however, the MCRS has not been validated against an external PCI data set. The National Cardiovascular Data Registry (NCDR) Cath PCI Registry is designed to measure and promote quality improvement for interventional cardiology with an emphasis on patient and lesion selection criteria, procedural performance, and in-hospital outcomes/complications. Thus, it presents an ideal opportunity to validate the MCRS for in-hospital mortality after PCI. Once the validity of the MCRS is confirmed, it may be used as a tool to inform patients of their estimated risk from PCI and become an integral part of the informed consent process.
Development and Validation of the MCRS
The original MCRS risk score was validated using Mayo Clinic PCI data from 7457 patients who underwent ad hoc PCI at the Mayo Clinic between January 2000 and April 2005.4 The model for in-hospital mortality demonstrated excellent ability to discriminate between high-risk and low-risk patients (c index=0.90). The probabilities estimated from the model also closely matched the observed data, as indicated by a nonsignificant Hosmer-Lemeshow goodness-of-fit test (P=0.44). The original MCRS equation is presented in Figure 1 (the recalibrated quadratic model is provided in the legend and described below).
The NCDR Cath PCI Registry, a partnership between the American College of Cardiology (ACC) and Society of Coronary Angiography and Intervention, is a prospective registry of consecutive PCI (77% ad hoc) patients treated throughout the United States. The strengths of the registry include the use of standardized data definitions, geographic and institutional diversity, and large sample size.5 The data collection process has been described previously.5 This study focused on index PCI procedures for 309 351 patients in an NCDR-participating hospital between January 1, 2004, and March 30, 2006. The outcome for this analysis was the incidence of in-hospital mortality during the hospital admission after PCI.
The definitions used by the NCDR are similar to those used in the original MCRS for in-hospital death and other baseline variables included in the model, except cardiogenic shock. Detailed baseline patient demographic, angiographic, and procedural variables used in NCDR have been previously described (Table 1).5
The authors had full access to and take full responsibility for the integrity of the data. All authors agree to the manuscript as written.
Crude in-hospital mortality and morbidity rates were calculated for the entire NCDR analysis population. In addition, raw mortality rates for each level of baseline demographic and laboratory characteristics were compared by use of Pearson’s χ2 test for all categorical variables and Wilcoxon’s test for age and ejection factor. Using the Mayo Clinic multivariable risk factor equation,4 we calculated predicted probabilities of death for each patient in the NCDR population. Patients with the same predicted mortality score were grouped together, and within each group, the observed mortality rate and 95% approximate binomial CI were calculated. The observed versus expected in-hospital mortality rates for these groups were plotted, and the calibration of the model was assessed with the Hosmer-Lemeshow method. The observed versus expected in-hospital mortality rates for these 23 groups were plotted, and the calibration of the model was assessed with the Hosmer-Lemeshow statistic with 23 df.
Model discrimination was assessed with the area under the receiver-operating characteristics curve, or c statistic, for the entire population and within prespecified patient subgroups. Approximate 95% CIs were constructed using variance estimates described by DeLong et al.6
Because the MCRS prediction equation did not calibrate as well as expected in the external NCDR population, we compared the regression coefficients from the MCRS and NCDR and refined the analysis to include recalibration of the equation using the NCDR population. Logistic regression was used to model mortality as a function of the MCRS to determine a new coefficient for an additional “recalibration” term in the prediction equation. To account for the nonlinear relationship between the Mayo Clinic integer risk score and mortality, both linear and quadratic terms were included in the new logistic model. Internal validation of this recalibrated quadratic MCRS equation was performed using additional NCDR data consisting of PCI procedures performed between May 22, 2004, and March 30, 2007 (primarily after April 1, 2006). Patients with the same predicted mortality score were again grouped together, and the observed versus expected in-hospital mortality rates were plotted. Discrimination was assessed once more with the c statistic for the entire population, and calibration was again confirmed with the Hosmer-Lemeshow test. For the NCDR population, we ran a logistic regression on mortality using the variables from the Mayo Clinic model, with continuous variables categorized according to the MCRS equation. The regression coefficients were then converted to integers by dividing by 0.37 and rounding. All statistical analyses were performed by the Duke Clinical Research Institute with SAS version 9.0.
Demographic, Clinical, and Procedural Characteristics
In the NCDR, 3790 patients (1.23%) died during hospital admission. The mean age of 309 351 patients was 64±12.2 years; 33.7% were female; 36.2% underwent urgent interventions; and 22.0% presented with acute myocardial infarction. At the time of the procedure, 2.0% were in cardiogenic shock. The prevalence of hypertension was 75.2%; 31.7% presented with diabetes; 11.8% had peripheral vascular disease; and 5.3% reported renal failure. More than 97% of patients had an MCRS ≤10, indicating that most PCI procedures were performed in patients at low to intermediate risk (Figure 2).
Validation of the MCRS With the NCDR
The area under the receiver-operating characteristics curve was 0.884, indicating an excellent ability to discriminate between patients who died during the hospitalization from a pair of patients undergoing PCI. The discriminatory ability of the prediction equation, as measured by the area under the receiver-operating characteristics curve (c statistic), was robust across most clinically important subgroups (0.797 to 0.938) (Table 2), including all age groups, those with and without normal left ventricle function, those with congestive heart failure, and patients with diabetes mellitus, multivessel disease, and renal dysfunction.
Model Calibration and Recalibration
The observed incident rates of in-hospital mortality were significantly higher for patient subgroups across the MCRS spectrum except MCRS <7 and between 22 and 25; the model overestimated mortality at these MCRS values. Using the MCRS prediction equation, the observed and predicted in-hospital mortality rates after PCI in the registry are shown in Table 3 and Figure 3. The model generally underpredicted mortality; overall observed in-hospital mortality was 1.23% (n=3790), whereas predicted mortality was 1.10%. The Hosmer-Lemeshow goodness-of-fit statistic was 540 with 8 df (P<0.0001), suggesting poor calibration. To better understand the differences in the calibration, we compared the regression coefficients of the 7 variables included in the original model and found that although the continuous variables (age, serum creatinine, and ejection fraction) had a linear effect in the original MCRS model, they were not linear in the larger NCDR data set (Table 4). Accordingly, the MCRS model was recalibrated to the more representative ACC population using logistic regression that included the components of the MCRS to obtain a new quadratic prediction equation.
We internally validated this new prediction equation using more recent NCDR data as shown in Table 5 and Figure 4. The validation data set included 433 045 PCI procedures performed between April 2006 and March 2007. There were 5177 in-hospital deaths; the model predicted 5310 deaths (difference, 0.03 deaths per 100 procedures). The new model demonstrated excellent discrimination (c index=0.88) and calibrated well across MCRS (Hosmer-Lemeshow goodness-of-fit statistic, 29.8 with 8 df; P=0.0002; Figure 4). The clinical application of the model using the final recalibrated quadratic prediction equation is shown in Table 6. We grouped the MCRS integers and depicted corresponding average predicted probability of in-hospital mortality, demonstrating higher mortality with higher score.
The goal of this study was to assess the validity of the MCRS for the prediction of in-hospital mortality after PCI using an external data set that could support its relevance as a useful clinical risk tool. Using the NCDR, we found robust model discrimination overall and across various prespecified subgroups. However, the MCRS prediction equation was not optimally calibrated for the NCDR population. Therefore, the prediction equation was recalibrated and revalidated (using both discrimination and calibration estimates) in a novel cohort of the NCDR. This step resulted in a more accurate version of the MCRS that is now ready for broader use in estimating mortality risks in contemporary clinical care.
Model Without Angiographic Variables
This study represents a significant advance over previous models that rely on angiographic variables known to predict outcomes after PCI.7–14 Even the recently published New York State model includes variables such as stent thrombosis and left main coronary artery disease, which can be obtained only after diagnostic coronary angiography.15 In our report, 7 clinical and laboratory variables can be summed to make a simple risk score that any healthcare provider can use at the time of initial contact with the patient to better comprehend the attendant risks from PCI. Because PCI procedures are often done in the same setting as diagnostic angiography, our model can allow an accurate assessment of periprocedural mortality before the initial angiogram at a time when the patient is not sedated and is in a better position to discuss the risks and benefits of treatment with the physician. Given that our model discriminated in-hospital mortality well despite not including angiographic variables, it may have an important role in informing patients of the risks of PCI should their coronary anatomy indicate that such a procedure is technically feasible and clinically indicated.
Role of Angiographic Variables in Risk Prediction
The need and utility of angiographic variables for risk prediction are not clear. We had previously reported our original Mayo Clinic risk adjustment model that had angiographic variables. That model was later validated in the National Heart, Lung, and Blood Institute Dynamic Registry.16 Even with the angiographic variables, the original Mayo Clinic risk model did not perform optimally at the extremes (high- and low-risk patients), with significant overprediction of procedural complications in patients with prior coronary artery bypass graft surgery. Underscoring the modest improvements of clinical risk stratification associated with angiographic data, Holmes et al7 forced pertinent angiographic variables (ACC/AHA type C lesion, calcification, bifurcation lesion) into the previous New York State model. Except for the ACC/AHA type C lesion, other angiographic variables did not significantly add to the prognostic information already obtained by the model. Most of the significant prognostic variables in that model were clinical and were similar to those of our new MCRS. To study the utility of the angiographic variables, we previously compared the ACC/AHA lesion classification with the original MCRS.17 The lesion classification did not predict procedural success and was inferior to the MCRS, further underscoring that the most relevant prognostic information can be gathered with patients’ demographic characteristics, presentation, and a few noninvasive laboratory tests. Angiographic variables, however, were determined to be useful in predicting elevation of creatine kinase-MB after a PCI procedure, an outcome of marginal clinical significance compared with mortality.18
We recently constructed a practical pre-PCI risk model based solely on a baseline clinical and noninvasive assessment that could serve as a simple risk assessment aid for patients and physicians before coronary angiography is performed.4 We undertook that study with the specific aim of excluding variables that, thus far, have limited the application of published PCI risk models in the process of informed consent, when a prediction of risk before the angiogram is needed. The original MCRS risk model discriminated well but had only modest calibration when applied to the NCDR data set. To improve the calibration of the model across the entire risk spectrum of patients, we included an additional, simple set of recalibration terms. Our refined model still needs only easily obtained clinical information so that healthcare providers can use the model for gathering pertinent prognostic information with which to obtain higher-quality, evidence-based informed consent.
Information about risks from a revascularization procedure is critical in the planning stages, before the patient is sent to the cardiac catheterization laboratory. This assumes even greater relevance given the excellent long-term outcomes achieved by aggressive medical therapy and the need for patients to have a clearer understanding of the risks they face with the conduct of a PCI.19 The MCRS model initially tended to underpredict event rates when applied in NCDR data (observed 1.23% versus predicted 1.10%), but this underprediction was corrected after recalibration. It is difficult to point to any specific reasons for overprediction for low-risk PCI cases and underprediction of risk for higher-risk categories. In general, models have a tendency to predict a higher risk of death than is observed for the patients with the least severe illness and a lower risk of death than is observed for the most ill patients. First, there may be some unmeasured covariates that can modify the risk in the model. Second, we observed higher risk in the original model at a very young age and low serum creatinine that may partly explain the higher risk offered to low-risk patients. Third, lower risk assigned to higher-risk patients can be due to noninclusion of angiographic variables. Fourth, the effects of the continuous variables included in the MCRS were different in the NCDR (linear versus nonlinear effects). Differences in operator and institutional volumes and disparities in other processes of care, including volume, experience, and availability of onsite surgery, can be surrogates of quality of care and can potentially sway the results in favor or against our model. Ivanov et al20 emphasized the need for periodic recalibration of an existing model. Among 3 strategies to assess mortality after coronary artery bypass graft surgery, recalibration of the original model was recommended for risk prediction. The use of a ready-made model or development of a new model worked well for temporal benchmarking and in instances of profound disparities in the case mix and event rates. It is likely that a different case mix, exclusion of angiographic (left main/multivessel disease, thrombus) or procedural variables (exigency), or improved reportage of risk factors (upcoding, eg, for ejection fraction) have contributed to modest calibration of the model. Recently, Hubacek et al21 externally validated the Michigan mortality model in the Alberta Provincial Project for Outcome Assessment in Coronary Heart Disease (APPROACH). The original Michigan model underestimated the in-hospital mortality for the highest-risk patients with acute coronary syndrome undergoing PCI. With the addition of indication for the procedure (myocardial infarction) and ejection fraction, a significant improvement in the calibration of the extended Michigan model was observed. The new, recalibrated MCRS model is ready for application in routine clinical practice and should be explicitly tested to quantify the impact of accurate periprocedural mortality estimates on the quality of the informed consent process, on patients’ comfort with medical decision making, and on clinicians’ practice of PCI.
Several features of the present study warrant consideration when our results are applied in clinical practice. First, validation of a prediction rule developed on a different database carries the risk that not all variables have similar definitions across data sources. As a result of differences in definitions across studies, it is possible that the calculated probabilities of procedural complication may be inflated in the registry. However, the NCDR is the largest national registry of patients undergoing PCI, and the data definitions used in this study have therefore become the de facto standard for classifying patients’ clinical status. Second, the ACC NCDR is a voluntary clinical quality assurance data repository with small and variable data audits and adjudication. This limitation is somewhat mitigated by the large sample size and the outcome of interest (in-hospital mortality) with minimal ascertainment bias. It is also possible that other variables such as operator volume could be significantly associated with adverse events after PCI and require further exploration in future studies.22,23 Finally, although the MCRS has high discrimination in predicting in-hospital mortality, no predictive model can ameliorate the effect of chance and unanticipated circumstances and complications inherently encountered in invasive treatments.
External validation of MCRS using the multicenter ACC NCDR confirms the broader potential applicability of this score to informed consent and medical decision making. The recalibrated MCRS had high discrimination for in-hospital mortality using 7 simple clinical and noninvasive variables. Most of the variables can be obtained at the time of first contact with the patient. Risk stratification may help the operator to individualize the risk of procedural death from PCI and to counsel patients at the time of PCI.
Sources of Funding
Drs Spertus and Peterson have a research contract with the American College of Cardiology.
Hochman JS, Sleeper LA, Webb JG, Sanborn TA, White HD, Talley JD, Buller CE, Jacobs AK, Slater JN, Col J, McKinlay SM, LeJemtel TH. Early revascularization in acute myocardial infarction complicated by cardiogenic shock: SHOCK Investigators: Should We Emergently Revascularize Occluded Coronaries for Cardiogenic Shock? N Engl J Med. 1999; 341: 625–634.
Shaw RE, Anderson HV, Brindis RG, Krone RJ, Klein LW, McKay CR, Block PC, Shaw LJ, Hewitt K, Weintraub WS. Development of a risk adjustment mortality model using the American College of Cardiology–National Cardiovascular Data Registry (ACC-NCDR) experience: 1998–2000. J Am Coll Cardiol. 2002; 39: 1104–1112.
Holmes DR Jr, Berger PB, Garratt KN, Mathew V, Bell MR, Barsness GW, Higano ST, Grill DE, Hammes LN, Rihal CS. Application of the New York State PTCA mortality model in patients undergoing stent implantation. Circulation. 2000; 102: 517–522.
Kimmel SE, Berlin JA, Strom BL, Laskey WK. Development and validation of simplified predictive index for major complications in contemporary percutaneous transluminal coronary angioplasty practice: the Registry Committee of the Society for Cardiac Angiography and Interventions. J Am Coll Cardiol. 1995; 26: 931–938.
Moscucci M, Kline-Rogers E, Share D, O'Donnell M, Maxwell-Eward A, Meengs WL, Kraft P, DeFranco AC, Chambers JL, Patel K, McGinnity JG, Eagle KA. Simple bedside additive tool for prediction of in-hospital mortality after percutaneous coronary interventions. Circulation. 2001; 104: 263–268.
O'Connor GT, Malenka DJ, Quinton H, Robb JF, Kellett MA Jr, Shubrooks S, Bradley WA, Hearne MJ, Watkins MW, Wennberg DE, Hettleman B, O'Rourke DJ, McGrath PD, Ryan T Jr, VerLee P. Multivariate prediction of in-hospital mortality after percutaneous coronary interventions in 1994-1996: Northern New England Cardiovascular Disease Study Group. J Am Coll Cardiol. 1999; 34: 681–691.
Singh M, Rihal CS, Selzer F, Kip KE, Detre K, Holmes DR. Validation of Mayo Clinic risk adjustment model for in-hospital complications after percutaneous coronary interventions, using the National Heart, Lung, and Blood Institute dynamic registry. J Am Coll Cardiol. 2003; 42: 1722–1728.
Singh M, Rihal CS, Lennon RJ, Garratt KN, Holmes DR Jr. Comparison of Mayo Clinic Risk Score and American College of Cardiology/American Heart Association lesion classification in the prediction of adverse cardiovascular outcome following percutaneous coronary interventions. J Am Coll Cardiol. 2004; 44: 357–361.
Boden WE, O'Rourke RA, Teo KK, Hartigan PM, Maron DJ, Kostuk WJ, Knudtson M, Dada M, Casperson P, Harris CL, Chaitman BR, Shaw L, Gosselin G, Nawaz S, Title LM, Gau G, Blaustein AS, Booth DC, Bates ER, Spertus JA, Berman DS, Mancini GB, Weintraub WS. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007; 356: 1503–1516.
Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation. 1999; 99: 2098–2104.
Malenka DJ, McGrath PD, Wennberg DE, Ryan TJ Jr, Kellett MA Jr, Shubrooks SJ Jr, Bradley WA, Hettlemen BD, Robb JF, Hearne MJ, Silver TM, Watkins MW, O'Meara JR, VerLee PN, O'Rourke DJ. The relationship between operator volume and outcomes after percutaneous coronary interventions in high volume hospitals in 1994-1996: the northern New England experience: Northern New England Cardiovascular Disease Study Group. J Am Coll Cardiol. 1999; 34: 1471–1480.
The Mayo Clinic Risk Score model is based solely on preprocedural clinical and noninvasive assessments and is less reliant on subjective assessment or angiographic data. In this study, we confirm the validity of the Mayo Clinic Risk Score and show that this model can be used as a tool to inform patients of their estimated risk from percutaneous coronary intervention and can become an integral part of the informed consent process. External validation of Mayo Clinic Risk Score using the multicenter National Cardiovascular Data Registry confirms the broader potential applicability of this score to informed consent and medical decision making. The recalibrated Mayo Clinic Risk Score had high discrimination for in-hospital mortality using 7 simple clinical and noninvasive variables. Most of the variables can be obtained at the time of first contact with the patient. Risk stratification may help the operator to individualize the risk of procedural death from percutaneous coronary intervention and to counsel patients at the time of percutaneous coronary intervention.