A disease-specific comorbidity index for predicting mortality in patients admitted to hospital with a cardiac condition
=======================================================================================================================

* Lorenzo Azzalini
* Malorie Chabot-Blanchet
* Danielle A. Southern
* Anna Nozza
* Stephen B. Wilton
* Michelle M. Graham
* Guillaume Marquis Gravel
* Jean-Pierre Bluteau
* Jean-Lucien Rouleau
* Marie-Claude Guertin
* E. Marc Jolicoeur

## Abstract

**BACKGROUND:** Comorbidity indexes derived from administrative databases are essential tools of research in global health. We sought to develop and validate a novel cardiac-specific comorbidity index, and to compare its accuracy with the generic Charlson–Deyo and Elixhauser comorbidity indexes.

**METHODS:** We derived the cardiac-specific comorbidity index from consecutive patients who were admitted to hospital at a tertiary-care cardiology hospital in Quebec. We used logistic regression analysis and incorporated age, sex and 22 clinically relevant comorbidities to build the index. We compared the cardiac-specific comorbidity index with refitted Charlson–Deyo and Elixhauser comorbidity indexes using the C-statistic and net reclassification improvement to predict in-hospital death, and the Akaike information criterion to predict length of stay. We validated our findings externally in an independent cohort obtained from a provincial registry of coronary disease in Alberta.

**RESULTS:** The novel cardiac-specific comorbidity index outperformed the refitted generic Charlson–Deyo and Elixhauser comorbidity indexes for predicting in-hospital mortality in the derivation population (*n* = 10 137): C-statistic 0.95 (95% confidence interval [CI] 0.94–0.9) v. 0.81 (95% CI 0.77–0.84) and 0.86 (95% CI 0.82–0.89), respectively. In the validation population (*n* = 17 877), the cardiac-specific comorbidity index was similarly better: C-statistic 0.92 (95% CI 0.89–0.94) v. 0.76 (95% CI 0.71–0.81) and 0.82 (95% CI 0.78–0.86), respectively, and also numerically outperformed the Charlson–Deyo and Elixhauser comorbidity indexes for predicting 1-year mortality (C-statistic 0.78 [95% CI 0.76–0.80] v. 0.75 [95% CI 0.73–0.77] and 0.77 [95% CI 0.75–0.79], respectively). Similarly, the cardiac-specific comorbidity index showed better fit for the prediction of length of stay. The net reclassification improvement using the cardiac-specific comorbidity index for the prediction of death was 0.290 compared with the Charlson–Deyo comorbidity index and 0.192 compared with the Elixhauser comorbidity index.

**INTERPRETATION:** The cardiac-specific comorbidity index predicted in-hospital and 1-year death and length of stay in cardiovascular populations better than existing generic models. This novel index may be useful for research of cardiology outcomes performed with large administrative databases.

The processing of large health administrative data sets has been facilitated by the implementation of national comprehensive registries. Successful examples of outcomes research based on these data include randomized trials nested in such registries,1 original reports elucidating important epidemiological aspects of the prevalence and natural history of diseases,2,3 and big data research based on large data sets of discharge diagnosis summaries.4,5 However, outcomes research based on health administrative databases remains limited by the poor specificity for case mix adjustment and the lack of systemic validation, which may lead to biased analyses.

Tools have been designed to minimize case mix bias when conducting research using health administrative databases, such as the Charlson–Deyo comorbidity index and the Elixhauser comorbidity index, which have been shown to predict in-hospital and long-term mortality, length of stay, and health care–related costs.6–9 These indexes take into account 17 (CDCI) and 31 (ECI) comorbidities that are coded according to the *International Classification of Diseases and Related Health Problems, 10th Revision* (ICD-10). Although both indexes have been validated in a variety of clinical settings,10–13 the Charlson–Deyo and Elixhauser comorbidity indexes are generic tools (i.e., they are not disease specific), and their accuracy in specific populations — including patients with cardiac conditions — has received little scrutiny.14

We aimed to derive and validate a novel prognostic comorbidity index specific for cardiac populations to predict death, both in-hospital and at 1-year, and length of stay, and to compare its prognostic accuracy with established generic indexes.

## Methods

### Study population

The derivation cohort was described previously.15 Between April 2006 and March 2008, we included consecutive patients undergoing cardiac catheterization at the Montreal Heart Institute, Montréal. The primary indications for catheterization represented the whole spectrum of contemporary cardiology and included diagnostic (coronary angiogram and heart catheterization) as well as therapeutic catheterizations (structural and coronary interventions).

We used an independent contemporary cohort to validate the use of the cardiac-specific comorbidity index to predict in-hospital and 1-year death. This cohort was derived from the Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) registry, and prospectively included all patients undergoing cardiac catheterization in the province of Alberta between April 2014 and March 2016. The registry contains detailed clinical assessment and is routinely linked to administrative data to assure data quality and ascertain clinical outcomes.16

### Data extraction and calculation of the comorbidity index

For the derivation cohort, we obtained the demographic characteristics, comorbid conditions, in-hospital death and length of stay from the discharge summary database, including the diagnostic and intervention codes from hospital discharge claims, as provided to the Régie de l’assurance maladie du Québec. The discharge database was populated by the discharging physician and adjudicated by professional medical archivists, following a comprehensive review of the medical records, and abstracted using the ICD-10 codes. The reliability and predictive capability of this approach in cardiology has been validated previously. 17–21 We computed the Charlson–Deyo comorbidity index6 and the Elixhauser comorbidity index8 using the scoring system we described previously.15

### Outcomes

Our primary outcome was the rate of in-hospital all-cause death. Secondary outcomes were length of stay and 1-year rate of all-cause death. To account for an anticipated skewed distribution in length of stay, we stratified admissions to hospital as lasting 2 days or less versus 3–7 days versus 8 days or longer, as a way to represent short (e.g., simple elective coronary or structural interventions), intermediate (e.g., moderate-risk acute coronary syndrome) and long (e.g., cardiogenic shock) hospital stays, respectively.

### Statistical analysis

Descriptive statistics for continuous variables are presented as means (standard deviations [SDs]) and categorical variables are presented as counts (%). The associations between predictors and outcomes are presented as odds ratios (ORs) with a 95% confidence interval (CI). Statistical significance was determined at the 2-sided, α = 0.05 level. We used SAS (version 9.4, SAS Institute) to perform the data analysis. In accordance with our institution’s internal reidentification policy for data access requests, values in small size cells (*n* ≤ 5) are not reported to protect confidentiality.

The primary step was to develop a comorbidity index specific for cardiac diseases. To assess the relative merit of this novel index, we subsequently refitted the generic Charlson–Deyo comorbidity index and Elixhauser comorbidity index in the derivation cohort and ascertained the relation of all 3 indexes with in-hospital death and length of stay. In our final step, we tested the performance of our novel index in predicting in-hospital and 1-year death through internal and independent external validation.

We derived the cardiac-specific comorbidity index using logistic regression modelling to assess the relation between the information available in health administrative databases and outcomes.

Stepwise selection of variables was applied to binary logistic regression to model death22 and to generalized logistic regression to model the multinomial length of stay. We prespecified candidate variables to include age and sex, as well as 22 independent comorbid conditions that we deemed relevant in patients with cardiac diseases, as informed by clinical judgment (Appendix 1, Supplemental Table 1, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181186/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181186/-/DC1)).23 For both models, we forced age and sex to be included in the final model, whereas the other candidate variables needed a *p* value of less than 0.20 to enter the model and a *p* value of less than 0.10 to stay in the final multivariable model.

We refitted and recalibrated the original coefficients for the Charlson–Deyo and Elixhauser comorbidity indexes for the derivation cohort using multivariable logistic regression analysis. We used the C-statistic to evaluate model discrimination for death, and we assessed calibration by comparing the predicted and observed death rates with the Hosmer–Lemeshow test. Subsequently, we calculated the category-based event, nonevent and overall net reclassification improvement to evaluate the improvement in the correct classification between the cardiac-specific comorbidity index and the recalibrated Charlson–Deyo and Elixhauser comorbidity indexes for the prediction of death.24,25 We used a value of 1% of predicted risk of death as a cut-off to discriminate between low and high risk.26 To compare the overall fit of the different models to predict length of stay, we used the Akaike Information Criterion.

We performed internal validation of the cardiac-specific comorbidity index for death with bootstrap resampling, which allowed estimation of the internally validated C-statistic (corrected for optimism) and the calibration slope27 (Appendix 2, Supplemental methods, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181186/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181186/-/DC1)). The cardiac-specific comorbidity index and refitted Charlson–Deyo and Elixhauser comorbidity indexes were validated externally for the prediction of in-hospital and 1-year all-cause death. We sampled the validation cohort to get at least 100 deaths at any time point.

### Ethics approval

The study was approved by the institutional review board at the Montreal Heart Institute, the Conjoint Health Research Ethics Board at the University of Calgary and the Health Research Ethics Board at the University of Alberta.

## Results

The derivation and validation cohorts are shown Figure 1. The baseline characteristics of the derivation cohort are outlined in Table 1. Mean age was 63.9 (SD 12.0) years, and two-thirds of the patients in the cohort were male. This cohort is representative of a contemporary population of patients referred for cardiac catheterization, with a high prevalence of cardiovascular risk factors. Overall comorbidity burden was high, as assessed with both the Charlson–Deyo and Elixhauser comorbidity indexes. In particular, one-third of patients had an Elixhauser comorbidity index of 3 or more. Ten percent presented with ST-elevation myocardial infarction (MI), and about half of the population underwent a therapeutic procedure. Overall, there were 130 in-hospital deaths (1.3%), and the proportions of patients with a length of stay of 2 days or less, 3–7 days and 8 days or longer were 65.9%, 21.3% and 12.8%, respectively.

![Figure 1:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/191/11/E299/F1.medium.gif)

[Figure 1:](http://www.cmaj.ca/content/191/11/E299/F1)

Figure 1: 
Derivation and validation cohorts for the cardiac-specific comorbidity index. Note: APPROACH = Alberta Provincial Project for Outcomes Assessment in Coronary Artery Disease, LOS = length of stay.

View this table:
[Table 1:](http://www.cmaj.ca/content/191/11/E299/T1)

Table 1: 
Baseline characteristics for the derivation cohort

### Derivation of the cardiac-specific comorbidity index

Univariable analysis for the candidate predictors of death are presented in Table 2. Most candidate comorbidities related to heart disease were strongly associated with death, except for sex, the presence of an implantable cardioverter-defibrillator, chronic pulmonary disease, diabetes, hypertension and obesity. For multivariable analysis (Table 3), 11 of the 24 candidate predictors were independently associated with death (age; ST-elevation and non–ST-elevation MI; cardiac arrest, ventricular tachycardia/fibrillation, cardiogenic shock; acute respiratory failure; acute kidney failure; pulmonary hypertension; heart failure; liver disease; cancer; neurologic disorders; and obesity) (Box 1).

View this table:
[Table 2:](http://www.cmaj.ca/content/191/11/E299/T2)

Table 2: 
Candidate predictors for the cardiac-specific comorbidity index and their univariate association with in-hospital all-cause death

View this table:
[Table 3:](http://www.cmaj.ca/content/191/11/E299/T3)

Table 3: 
Independent predictors of all-cause death in the cardiac-specific comorbidity index

**Box 1: List of factors used for the cardiac-specific comorbidity index** 

| All-cause death                                                                                                                                                                                                                                                                                                                               | Length of stay                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *   Age

*   Sex

*   STEMI and NSTEMI

*   Stroke and TIA

*   Atrial fibrillation

*   Cardiac arrest, VT/VF, cardiogenic shock

*   Acute pulmonary embolism

*   Acute respiratory failure

*   Acute kidney failure

*   Pulmonary hypertension

*   Heart failure

*   Liver disease

*   Cancer

*   Neurologic disorders

*   Obesity | *   Age

*   Sex

*   STEMI and NSTEMI

*   Atrial fibrillation

*   Atrioventricular block

*   Cardiac arrest, VT/VF, cardiogenic shock

*   Acute pulmonary embolism

*   Acute respiratory failure

*   Acute kidney failure

*   Pulmonary hypertension

*   Presence of an ICD

*   Heart failure

*   PAD and CVD

*   Chronic pulmonary disease

*   Hypertension

*   Chronic kidney disease

*   Neurologic disorders

*   Coagulopathy |

*   Note: CVD = cerebrovascular disease, ICD = implantable cardioverter-defibrillator, NSTEMI = non–ST-elevation myocardial infarction, PAD = peripheral artery disease, STEMI = ST-elevation myocardial infarction, TIA = transient ischemic attack, VT/VF = ventricular tachycardia/ventricular fibrillation.

Table 4 and Supplemental Table 2 in Appendix 1 present the univariable and multivariable analyses for the prediction of length of stay with the cardiac-specific comorbidity index. Most comorbidities showed a significant univariable association with length of stay. After multivariable adjustment, 18 of the 24 candidate variables were associated with length of stay (Box 1).

View this table:
[Table 4:](http://www.cmaj.ca/content/191/11/E299/T5)

Table 4: 
Independent predictors of length of stay in the cardiac-specific comorbidity index*

### Refitting the Charlson–Deyo and Elixhauser comorbidity indexes

The recalibrated coefficients of the Charlson–Deyo and Elixhauser comorbidity indexes for the prediction of death are shown in Supplemental Table 3 of Appendix 1. Using the Charlson–Deyo comorbidity index, only 5 comorbidities (MI, chronic heart failure, cerebrovascular disease, renal disease and cancer) showed significant association with death, whereas 9 comorbidities (chronic heart failure, cardiac arrhythmia, pulmonary circulation disorders, complicated hypertension, other neurologic disorders, liver disease, lymphoma, coagulopathy, and fluid and electrolyte disorders) were significantly associated with death using the Elixhauser comorbidity index.

Supplemental Table 4 in Appendix 1 shows the recalibrated coefficients of the Charlson–Deyo and Elixhauser comorbidity indexes for predicting length of stay. Using the Charlson–Deyo comorbidity index, 10 of 17 comorbidities showed significant association with length of stay, whereas 21 of 31 comorbidities were significantly associated with length of stay using the Elixhauser comorbidity index.

### Predicting all-cause death with all 3 comorbidity indexes

The cardiac-specific comorbidity index yielded superior discrimination for the prediction of all-cause death compared with the refitted versions of the Charlson–Deyo and Elixhauser comorbidity indexes: C-statistic 0.95 (95% CI 0.94–0.97) v. 0.81 (95% CI 0.77–0.84) and 0.86 (95% CI 0.82–0.89) (*p* < 0.001), respectively. Supplemental Figure 1 in Appendix 1 shows the comparison between observed and predicted probabilities of death using the cardiac-specific comorbidity index as the predictor, stratified by deciles of the predicted probabilities. We observed very good calibration, which we also confirmed using the Hosmer–Lemeshow test (*p* = 0.74). The Hosmer–Lemeshow test also indicated good calibration for both the refitted Charlson–Deyo and Elixhauser comorbidity indexes (*p* = 0.44 and *p* = 0.73, respectively). When the 2 original indexes were used (categorized as 0, 1, 2 and ≥ 3), the Charlson–Deyo comorbidity index provided a C-statistic of 0.72 for the prediction of death, whereas the Elixhauser comorbidity index provided a slightly lower value (C-statistic 0.67) (*p* < 0.001). Supplemental Figure 2 in Appendix 1 presents the incidence of in-hospital death stratified by the categories from the Charlson–Deyo and Elixhauser comorbidity indexes.

Table 5 presents the net reclassification improvement analysis. Using the cardiac-specific comorbidity index to predict death led to an overall net reclassification improvement of 0.290 compared with the refitted Charlson–Deyo comorbidity index, and an overall net reclassification improvement of 0.192 compared with the refitted Elixhauser comorbidity index. In both instances, the overall net reclassification improvement was driven by the nonevent net reclassification improvement, indicating that, for the most part, the cardiac-specific comorbidity index correctly decreased risk estimates for nonevents (i.e., nondeath: 34.4% for the Charlson–Deyo comorbidity index and 11.5% for the Elixhauser comorbidity index ).

View this table:
[Table 5:](http://www.cmaj.ca/content/191/11/E299/T6)

Table 5: 
Category-based net reclassification improvement of the cardiac-specific comorbidity index compared with the recalibrated Charlson–Deyo and Elixhauser comorbidity indexes for predicting all-cause death

### Predicting length of stay with the 3 comorbidity indexes

The Akaike Information Criterion score for the cardiac-specific comorbidity index for the prediction of length of stay was 15 880, which compared favourably with both the refitted Charlson–Deyo comorbidity index (16 778) and Elixhauser comorbidity index (16 134). Using the categorized versions of the original Charlson–Deyo and Elixhauser comorbidity indexes, this difference increased further (17 146 for the Charlson–Deyo comorbidity index and 16 978 for the Elixhauser comorbidity index; Appendix 1, Supplemental Figure 3).

### Internal and external validation

After internal bootstrap resampling, the optimism for the cardiac-specific comorbidity index for the prediction of death in our data set was estimated at 0.01. Therefore, we estimated the internally validated C-statistic to be 0.94. The average calibration slope of the cardiac-specific comorbidity index was 0.93, indicating excellent calibration.

We validated the cardiac-specific comorbidity index for in-hospital and 1-year death in a different cohort of 17 877 patients with cardiac conditions in the province of Alberta (Figure 1). For in-hospital death, the cardiac-specific comorbidity index also outperformed the Charlson–Deyo and the Elixhauser comorbidity indexes, in terms of discrimination (C-statistic 0.92 [95% CI 0.89–0.94], compared with 0.76 [95% CI 0.71–0.81] and 0.82 [95% CI 0.78–0.86] for the Charlson–Deyo and Elixhauser comorbidity indexes, respectively). For 1-year death, the cardiac-specific comorbidity index performed better than the Charlson–Deyo and the Elixhauser comorbidity indexes: C-statistic 0.78 (95% CI 0.76–0.80) v. 0.75 (95% CI 0.73–0.77) and 0.77 (95% CI 0.75–0.79), respectively.

## Interpretation

We assessed the performance of the novel cardiac-specific comorbidity index using administrative data in patients admitted to hospital for a wide range of heart conditions. The cardiac-specific comorbidity index outperformed the refitted generic comorbidity indexes (Charlson–Deyo and Elixhauser comorbidity indexes) in predicting in-hospital and 1-year death, and length of stay. These findings have important clinical implications given the paucity of tools designed for comparative risk adjustment when working with electronic medical records and health administrative databases.

Risk estimation and case-mix adjustment are important for several stakeholders: patients and their families (to make informed decisions), researchers (to facilitate comparative research) and policy-makers (to facilitate comparison of outcomes across institutions).28 With the emergence of large administrative data sets and big data analysis, disease-specific indexes will refine the prediction of adverse events, health care utilization and costs. Previous examples can be found in gastroenterology and intensive care medicine.10,29 However, despite the huge effect of cardiac diseases on society, the potential of health administrative databases to risk-stratify patient cohorts in this context has received little attention.

Several studies17–21 have shown a good correlation between ICD-derived diagnoses and traditional chart review, especially for the diagnosis of MI18,20 and heart failure,18,21 and for cardiac procedure coding.19 A single-centre cohort study involving 1471 patients with coronary artery disease who underwent cardiac catheterization between 1985 and 1989 evaluated the ability of the Charlson comorbidity index and a novel coronary artery disease–specific index to predict long-term mortality.30 Both indexes showed reasonable discrimination for the prediction of 5-year mortality (C-statistic 0.72 and 0.73, respectively). However, the comorbidities were derived from traditional chart review, and the novel coronary artery disease-specific index was not validated. Normand and colleagues derived a novel generic comorbidity index to predict 2-year mortality in a population of Medicare beneficiaries who were admitted to hospital for MI.31 Similar to our study, they used an approach based on a health administrative database and initially chose 46 clusters of comorbidities, from which a final model was derived. Importantly, their comorbidity index included generic and broad definitions of cardiac conditions compared with our study (e.g., “arrhythmias” as opposed to “atrial fibrillation”). When compared with the traditional Charlson comorbidity index, their model provided a modest improvement in C-statistics in the validation cohorts (0.73 and 0.72 v. 0.66 and 0.65).

To the best of our knowledge, only 1 study specifically assessed the role of the Charlson–Deyo and Elixhauser comorbidity indexes for the prediction of in-hospital death in patients with cardiac conditions using an approach based on health administrative databases. In a retrospective study using National Health Insurance Database records of diagnosis that included 61 815 patients undergoing percutaneous coronary intervention in South Korea, 1598 (2.6%) died in hospital.14 After augmenting the original Charlson–Deyo comorbidity index and Elixhauser comorbidity index models with age, sex and admission type, the C-statistics were 0.75 for the Charlson–Deyo comorbidity index and 0.81 for Elixhauser comorbidity index. The Charlson–Deyo comorbidity index was developed initially to predict 1-year mortality, and the Elixhauser comorbidity index to predict resource utilization and in-hospital mortality. Therefore, their inferior performance in some of the studies mentioned previously may be explained, in part, by their utilization in different settings. In addition, the population in our study was a cohort of patients with cardiac conditions, whereas both the Charlson–Deyo and Elixhauser comorbidity indexes are nondisease-specific tools derived in the general population.

We found some potential advantages for incorporating risk models derived from health administrative databases into both clinical medicine and research. First, given that modern computer workstations are powerful, such risk models could be calculated automatically using the software of electronic medical records. Patients at high risk for adverse outcomes could be “flagged” to the clinician, and appropriate measures (e.g., admission to an intensive care unit, aggressive pharmacological and interventional management, and intensive work-up) could be undertaken more promptly. Second, these risk models could be used as a research tool to adjust for case mix, thus providing a standardized comparison between different study populations, which is relevant for both researchers and health care administrators.32 However, such speculations are hypothesis generating only, and the specific merits of the application of the cardiac-specific comorbidity index should be evaluated by further research.

Comorbidity indexes are both necessary and convenient when working with administrative data sets but should be used cautiously in certain circumstances. Medical registries enriched with disease-specific variables (such as laboratory examinations and vital signs) appear to be superior but infrequently available. The variation in discrimination observed in our validation data sets suggests that superspecific comorbidity indexes computed around a single ICD code (e.g., heart failure) could be useful in certain fields where the volume of research is important.

### Limitations

Our study has limitations. Certain comorbidities for both the Charlson–Deyo and Elixhauser comorbidity indexes were uncommon, which prevented the assessment of the contribution of such conditions with those comorbidity indexes. This issue was also reported by other studies that specifically focused on a patient population of patients with cardiac conditions.14 Balancing these limitations, however, are the large sample size and the external validation in a different Canadian province. Some of the variables that we selected could be considered as in-hospital events rather than comorbidities (e.g., cardiogenic shock). This explains, in part, the good discrimination of the cardiac-specific comorbidity index to predict in-hospital mortality and reinforces the importance of a long-term validation.

### Conclusion

Our cardiac-specific comorbidity index outperformed generic comorbidity indexes to predict all-cause death and length of stay in patients admitted to hospital for a cardiac condition. The cardiac-specific comorbidity index is a refined tool that can be used to adjust for case mix in research on cardiology outcomes and to inform decision-makers.

## Footnotes

*   **Competing interests:** Stephen Wilton has received consultant fees from Arca Biopharma and research grants from Medtronic of Canada, Abbott and Boston Scientific. No other competing interests were declared. This article has been peer reviewed.

*   **Contributors:** All of the authors made substantial contributions to the conception and design of the work, revised it critically for important intellectual content, gave final approval of the version to be published and agreed to be accountable for all aspects of the work.

*   **Funding**: E.M. Jolicoeur is supported by research grants from the Canadian Institutes of Health Research, the Canada Foundation for Innovation, the AGE-WELL Networks of Centres of Excellence, les Fonds de recherche en Santé du Québec and la Fondation de l’Institut de cardiologie de Montréal.

*   **Data sharing:** The data are not currently available for use by other researchers.

*   Accepted February 5, 2019.

## References

1.  Lagerqvist B, Fröbert O, Olivecrona GK, et al. Outcomes 1 year after thrombus aspiration for myocardial infarction. N Engl J Med 2014;371:1111–20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMoa1405707&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25176395&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000341687300006&link_type=ISI) 

2.  Råmunddal T, Hoebers LP, Henriques JPS, et al. Prognostic impact of chronic total occlusions: a report from SCAAR (Swedish Coronary Angiography and Angioplasty Registry). JACC Cardiovasc Interv 2016;9:1535–44.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jcin.2016.04.031&link_type=DOI) 

3.  Szummer K, Lundman P, Jacobson SH, et al. Influence of renal function on the effects of early revascularization in non-ST-elevation myocardial infarction: data from the Swedish web-system for Enhancement and Development of Evidence-based care in HEART disease evaluated according to recommended therapies (SWEDEHEART). Circulation 2009;120:851–8.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjEwOiIxMjAvMTAvODUxIjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE5MS8xMS9FMjk5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

4.  Wilhelm-Leen E, Montez-Rath ME, Chertow G. Estimating the risk of radiocontrast-associated nephropathy. J Am Soc Nephrol 2017;28:653–9.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiam5lcGhyb2wiO3M6NToicmVzaWQiO3M6ODoiMjgvMi82NTMiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTkxLzExL0UyOTkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

5.  McDonald RJ, McDonald JS, Bida JP, et al. Intravenous contrast material–induced nephropathy: causal or coincident phenomenon? Radiology 2013;267:106–18.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1148/radiol.12121823&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=23360742&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

6.  Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987; 40:373–83.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0021-9681(87)90171-8&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3558716&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1987G855900002&link_type=ISI) 

7.  Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0895-4356(92)90133-8&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1607900&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1992JA22600006&link_type=ISI) 

8.  Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care 1998;36:8–27.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-199801000-00004&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9431328&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000071181500004&link_type=ISI) 

9.  Chu YT, Ng YY, Wu SC. Comparison of different comorbidity measures for use with administrative data in predicting short- and long-term mortality. BMC Health Serv Res 2010;10:140.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1472-6963-10-140&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20507593&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

10. Ladha KS, Zhao K, Quraishi SA, et al. The Deyo–Charlson and Elixhauser-van Walraven comorbidity indices as predictors of mortality in critically ill patients. BMJ Open 2015;5:e008990.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNS85L2UwMDg5OTAiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTkxLzExL0UyOTkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

11. Frenkel WJ, Jongerius EJ, Mandjes-Van Uitert MJ, et al. Validation of the Charlson comorbidity index in acutely hospitalized elderly adults: a prospective cohort study. J Am Geriatr Soc 2014;62:342–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/jgs.12635&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24521366&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

12. Goldstein LB, Samsa GP, Matchar DB, et al. Charlson Index comorbidity adjustment for ischemic stroke outcome studies. Stroke 2004;35:1941–5.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjM1LzgvMTk0MSI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xOTEvMTEvRTI5OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

13. Chou WC, Chang PH, Lu CH, et al. Effect of comorbidity on postoperative survival outcomes in patients with solid cancers: a 6-year multicenter study in Taiwan. J Cancer 2016;7:854–61.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7150/jca.14777&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

14. Kim KH, Ahn LS. A comparative study on comorbidity measurements with lookback period using health insurance database: focused on patients who underwent percutaneous coronary intervention [article in Korean]. J Prev Med Public Health 2009;42:267–73.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.3961/jpmph.2009.42.4.267&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19675404&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

15. Azzalini L, Tosin K, Chabot-Blanchet M, et al. The benefits conferred by radial access for cardiac catheterization are offset by a paradoxical increase in the rate of vascular access site complications with femoral access: the Campeau Radial Paradox. JACC Cardiovasc Interv 2015;8:1854–64.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jcin.2015.07.029&link_type=DOI) 

16. Southern DA, Norris CM, Quan H, et al. An administrative data merging solution for dealing with missing data in a clinical registry: adaptation from ICD-9 to ICD-10. BMC Med Res Methodol 2008;8:1.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1471-2288-8-1&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18215293&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

17. Lambert L, Blais C, Hamel D, et al. Evaluation of care and surveillance of cardiovascular disease: Can we trust medico-administrative hospital data? Can J Cardiol 2012;28:162–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.cjca.2011.10.005&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=22230034&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

18. Austin PC, Daly PA, Tu JV. A multicenter study of the coding accuracy of hospital discharge administrative data for patients admitted to cardiac care units in Ontario. Am Heart J 2002;144:290–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1067/mhj.2002.123839&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12177647&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000177501400017&link_type=ISI) 

19. Lee DS, Stitt A, Wang X, et al. Administrative hospitalization database validation of cardiac procedure codes. Med Care 2013;51:e22–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3182329778&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21979370&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

20. Levy AR, Tamblyn RM, Fitchett D, et al. Coding accuracy of hospital discharge data for elderly survivors of myocardial infarction. Can J Cardiol 1999; 15: 1277–82.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10579743&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000084009100014&link_type=ISI) 

21. Lee DS, Donovan L, Austin PC, et al. Comparison of coding of heart failure and comorbidities in administrative and clinical data for use in outcomes research. Med Care 2005;43:182–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-200502000-00012&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15655432&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000226654500011&link_type=ISI) 

22. Mehta HB, Mehta V, Girman CJ, et al. Regression coefficient–based scoring system should be used to assign weights to the risk index. J Clin Epidemiol 2016; 79: 22–8.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 

23. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014;35: 1925–31.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1093/eurheartj/ehu207&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24898551&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000342232100007&link_type=ISI) 

24. Leening MJ, Vedder MM, Witteman JC, et al. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med 2014;160:122–31.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/M13-1522&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24592497&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000330275600018&link_type=ISI) 

25. Pencina MJ, D’Agostino RBS, D’Agostino RBJ, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157–72, discussion 207–12.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1002/sim.2929&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17569110&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000253098900001&link_type=ISI) 

26. GUSTO Investigators. An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. N Engl J Med 1993;329: 673–82.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199309023291001&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8204123&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1993LU58300001&link_type=ISI) 

27. Steyerberg EW, Harrell FE, Borsboom GJJ, et al. Internal validation of predictive models. J Clin Epidemiol 2001;54:774–81.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0895-4356(01)00341-9&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11470385&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000170019900003&link_type=ISI) 

28. Peterson ED, Dai D, DeLong ER, et al. Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry. J Am Coll Cardiol 2010;55:1923–32.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjEwOiI1NS8xOC8xOTIzIjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE5MS8xMS9FMjk5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

29. Xu Y, Li N, Lu M, et al. Comparison of risk adjustment methods in patients with liver disease using electronic medical record data. BMC Gastroenterol 2017; 17:5.
    
    

30. Sachdev M, Sun JL, Tsiatis AA, et al. The prognostic importance of comorbidity for mortality in patients with stable coronary artery disease. J Am Coll Cardiol 2004;43:576–82.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJhY2NqIjtzOjU6InJlc2lkIjtzOjg6IjQzLzQvNTc2IjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE5MS8xMS9FMjk5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. Normand SL, Morris CN, Fung KS, et al. Development and validation of a claims-based index for adjusting for risk of mortality: the case of acute myocardial infarction. J Clin Epidemiol 1995;48:229–43.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0895-4356(94)00126-B&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7869069&link_type=MED&atom=%2Fcmaj%2F191%2F11%2FE299.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995QL24500007&link_type=ISI) 

32. Kolek MJ, Graves AJ, Xu M, et al. Evaluation of a prediction model for the development of atrial fibrillation in a repository of electronic medical records. JAMA Cardiol 2016;1:1007–13.