Abstract
Background: The Thrombolysis in Myocardial Infarction (TIMI) risk score uses clinical data to predict the short-term risk of acute myocardial infarction, coronary revascularization or death from any cause. It was originally developed for use in patients with unstable angina or non–ST-elevation myocardial infarction. We sought to expand the clinical application of the TIMI risk score by assessing its prognostic accuracy in patients in the emergency department with potential acute coronary syndromes.
Methods: We searched five electronic databases, hand-searched reference lists of included studies and contacted content experts to identify articles for review. We included prospective cohort studies that validated the TIMI risk score in emergency department patients. We performed a meta-regression to determine whether a linear relation exists between TIMI risk score and the cumulative incidence of cardiac events.
Results: We included 10 prospective cohort studies (with a total of 17 265 patients) in our systematic review. Data were available for meta-analysis in 8 of the 10 studies. Of patients with a score of zero, 1.8% had a cardiac event within 30 days (sensitivity 97.2%, 95% CI 96.4–97.8; specificity 25.0%, 95% CI 24.3–25.7; positive likelihood ratio 1.30, 95% CI 1.28–1.31; negative likelihood ratio 0.11, 95% CI 0.09–0.15). Meta-regression analysis revealed a strong linear relation between TIMI risk score (p < 0.001) and the cumulative incidence of cardiac events.
Interpretation: Although the TIMI risk score is an effective risk stratification tool for patients in the emergency department with potential acute coronary syndromes, it should not be used as the sole means of determining patient disposition.
Chest pain is a common presenting complaint in the emergency department that requires efficient risk stratification, timely initiation of treatment in high-risk patients and safe determination of patient disposition. Several studies have been published that stratify the risk of patients in the emergency department with chest pain. 1–5 However, only the Thrombolysis in Myocardial Infarction (TIMI) risk score, which was initially developed for use in patients with unstable angina or non–ST-segment elevation myocardial infarction or both, 6 has been broadly validated in several independent emergency department populations with chest pain and thus constitutes the highest level of evidence available.
The TIMI risk score assigns each of seven predictors a value of one point, allowing stratification of patients into one of eight prognostic categories (Box 1). 6 The clinical end points are acute myocardial infarction, coronary revascularization and death from any cause.
A robust estimate of the performance of the TIMI risk score obtained from a systematic review may prove useful to both clinicians and researchers. Clinicians would have a reliable quantitative estimate of a patient’s short-term risk of a cardiac event. This could be used as an adjunct to clinical acumen and as a tool to communicate risk to patients in a shared decision-making model of care. 7 Researchers would also have an estimate of the prognostic accuracy of the TIMI risk score derived from different practice settings and patient populations that represent a wide variety of ethnic backgrounds. This estimate may serve as a useful baseline for comparison as emerging clinical prediction rules and imaging modalities continue to refine our approach to diagnosis and risk stratification in patients in the emergency department with potential acute coronary syndromes.
We conducted a comprehensive systematic review and meta-analysis to assess the methodological quality and prognostic performance of studies that had prospectively validated the TIMI risk score in patients in the emergency department.
Methods
This systematic review and meta-analysis adheres to the reporting guidelines of Meta-analysis of Observational Studies in Epidemiology (MOOSE) 8 and the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) statement as applicable to meta-analyses of observational studies. 9
Eligibility criteria
We included studies that prospectively validated the TIMI risk score in patients in the emergency department with chest pain. We excluded retrospective studies, case reports, studies not based on original research and studies not involving patients in the emergency department (e.g., hospital ward, intensive care unit).
Search strategy
An expert librarian (P.J.E.) designed a comprehensive search strategy with input from the clinical lead (E.P.H.). Our electronic search included five databases (MEDLINE, EMBASE, Cochrane Database of Systematic Reviews, Web of Science and Scopus; Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). We applied no language restrictions to the search strategy. We also hand-searched conference proceedings from 2007 to 2009 from the Society for Academic Emergency Medicine, the Canadian Association of Emergency Physicians and the American College of Emergency Physicians. We reviewed the reference lists of eligible articles and consulted with content experts (J.E.H., I.G.S.) to identify additional published reports.
Study selection
Two investigators (E.P.H., D.A.) independently screened the titles and abstracts of all identified records (phase I). If either reviewer believed that the study might be eligible, we obtained the full report. The same two investigators then independently assessed the eligibility of each full report (phase II). We used the Cohen κ to measure the chance-corrected agreement between reviewers for each phase of study selection. Any disagreements were resolved by consensus.
Quality assessment
Two reviewers (D.A., S.C.) independently assessed the methodological quality of each eligible article. Because there is no single quality assessment tool that has been designed to assess the methodological quality of studies of risk scores, we selected criteria from assessment tools 10–12 that, based on a consensus among investigators, most closely indicated study quality. We selected the following seven criteria: patients selected in an unbiased fashion (consecutive or random sample); study sample representative of a wide spectrum of the severity of disease; predictor variables assessed without knowledge of the outcome; outcome assessed without knowledge of the predictor variables; outcomes accurately defined (particularly, acute myocardial infarction); explicit interpretation of the risk score by clinicians in practice without knowledge of the outcome; and adequate follow-up (arbitrarily defined as a lost-to-follow-up rate of < 10%). We assessed the degree of interobserver agreement for quality assessment using the Cohen κ. Any disagreements were resolved by consensus.
Data extraction and outcomes
Using a standardized data extraction sheet, two authors (D.A., S.C.) independently extracted data from each included report. Because most reports did not differentiate between patients with TIMI risk scores from 5 to 7, we classified patients with TIMI risk scores ≥ 5 into a single risk stratum. If the data were not available in the original report or were unclear, we contacted the corresponding author for clarification. Clinical outcomes were acute myocardial infarction, coronary revascularization or death from any cause. Disagreements were resolved by consensus.
Data synthesis and analysis
We present all continuous data as either means with standard deviations (SD) or medians with interquartile ranges as reported in the primary study. Categorical data are presented as per cent frequency of occurrence. We performed meta-regression to determine whether a linear relation exists between TIMI risk score (independent variable) and the logit event rate (dependent variable) using Comprehensive Meta-Analysis version 2 (Biostat Inc., Englewood, NJ). We assessed the prognostic performance of the TIMI risk score at potential decision thresholds using Meta-DiSc software (Unit of Clinical Biostatistics and the Ramon y Cajal Hospital).
We used the I2 statistic to quantify statistical heterogeneity between studies. We hypothesized that statistical heterogeneity identified by the I2 statistic could be explained by differences in the prevalence of cardiac events between studies. To test this hypothesis, we arbitrarily defined low prevalence as less than 10% and conducted an a priori subgroup analysis of studies with low (< 10%) versus high (≥ 10%) prevalence using a test of statistical interaction as defined by Altman. 13 Estimates from each subgroup are analyzed on a log scale. The test is defined as the ratio of the difference in the log estimates to the standard error of this difference. The associated p value is derived from the normal distribution.
Results
Study selection
Figure 1 shows the study selection process. The initial search strategy identified 383 records. After removing 68 duplicate records, 315 records remained for screening in phase I. We identified 27 potentially eligible records for further review based on review of the titles and abstracts (κ = 0.87, 95% CI 0.76–0.97). Review of the full articles in phase II identified 10 that met our inclusion criteria (κ = 0.78, 95% CI 0.54–1.00). All 10 studies were included in the qualitative analysis. We were able to contact authors for missing data and verification of data extraction for 8 of the 10 included studies. We were unable to obtain TIMI risk score data in two studies, 14,15 leaving eight studies for the quantitative synthesis.
Characteristics of the included studies
The 10 studies included in the review enrolled 17 265 emergency department patients with chest pain from 5 countries (Table 1). 16–23 Three studies were conducted in the United States, two in the United Kingdom, two in Spain, two in Italy and one in Canada. The mean age ranged from 53.6 to 71.3 years. The proportion of males to females enrolled in each study was approximately equal.
Quality assessment
Assessment of study quality for all 10 studies had an inter-observer agreement of 92.8% (κ = 0.85, 95% CI 0.71–0.98). Although the predictor variables were assessed without knowledge of the outcome in all studies, only one study 16 explicitly reported blinding outcome assessors to knowledge of the predictor variables (Appendix 2, available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). Two studies did not report how acute myocardial infarction was defined. 14,15 Physicians explicitly interpreted the TIMI risk score in practice without knowledge of the outcome in three studies. 17–19 All studies that reported the fidelity of patient follow-up had lost-to-follow-up rates of less than 10%.
Heterogeneity
Cardiac troponin or creatine kinase–muscle and brain (CK-MB) or both were used as cardiac biomarkers in all eight studies included in the quantitative analysis. However, only three studies 16,20,21 reported the specific assay used (including the manufacturer, platform, 99th percentile reference limit and coefficient of variation) as recommended by standardized reporting guidelines. 24 This makes it difficult to assess potential heterogeneity between studies because of the different cardiac biomarker thresholds used to define acute myocardial infarction. However, all studies validated the TIMI risk score in emergency department patients and assessed the same predictor variables and the same composite outcome. Thirty-day outcome data were available for all studies included in the quantitative synthesis except one 17 in which outcomes were assessed at the time of discharge from hospital or emergency department. The prevalence of cardiac events varied from 3.7% to 25.5% between studies.
Classification characteristics and diagnostic performance
The number of patients and the number with cardiac events in each risk stratum of the TIMI risk score for the eight studies included in the quantitative synthesis are shown in Appendix 3 (available at www.cmaj.ca/cgi/content/full/cmaj.092119/DC1). The prevalence of cardiac events in patients with a TIMI risk score of zero was 1.8%, and the prevalence in patients with a score of one was 4.0%. Although the total number of patients in each risk stratum progressively decreased with each successive increase in TIMI risk score, the proportion with adverse outcomes progressively increased up to a score of 5 or greater.
The results of the meta-regression of the TIMI risk score on the logit event rate are shown in Figure 2. The regression coefficient was significant (p < 0.001), indicating a strong linear relation between TIMI risk score and cumulative incidence of cardiac events.
Table 2 shows the classification performance of the TIMI risk score at potential decision thresholds. At a cut-off of > 0, the sensitivity was 97.2% (95% CI 96.4–97.8; I2 = 99.9%) and the specificity was 25.0% (95% CI 24.3–25.7; I2 = 100.0%). The negative likelihood ratio was 0.11 (95% CI 0.09–0.15; I2 = 83.3%). At a cut-off of > 1, the sensitivity decreased to 90.6% (95% CI 89.3–91.8; I2 = 97.9%) and the specificity increased to 50.9% (95% CI 50.0–51.7; I2 = 99.5%).
We also conducted a sensitivity analysis excluding the study by Pellicia and colleagues, 17 because this study reported in-hospital rather than 30-day outcomes. There were no substantial differences in the classification performance of the TIMI risk score when this study was excluded (data not shown).
Subgroup analyses
The test for interaction comparing studies with a low (< 10%) prevalence of cardiac events at a threshold of TIMI > 0 was significant (diagnostic odds ratio 5.58, 95% CI 4.21–7.39 v. (34.21, 95% CI 17.01–68.78; p < 0.001). The test for interaction at a threshold of TIMI > 1 was also significant (diagnostic odds ratio 5.12, 95% CI 4.27–6.15 v. 18.55, 95% CI 13.88–24.78; p < 0.001).
Interpretation
The results of our meta-regression analysis indicated a strong linear relation between TIMI risk score and the short-term incidence of cardiac events. The incidence of cardiac events in the lowest risk stratum (TIMI score of zero) was 1.8%; the sensitivity was 97.2% and specificity was 25.0% at this decision threshold. All potential decision thresholds were insufficiently sensitive or specific to recommend use of the TIMI risk score as the sole means of determining patient disposition. Although the analysis was associated with significant heterogeneity, interaction testing indicated that the heterogeneity could be explained by differences in the prevalence of cardiac events between studies.
Strengths and limitations
The strengths of this study include a search strategy that involved five electronic databases, searching the bibliographies of the included articles and contact with content experts and authors of the included studies. This minimized, but did not exclude, the potential for publication bias. We also used sound methods in conducting the review, including assessment of interrater reliability for study selection and quality assessment and dual data extraction. Our findings also represent the performance of the TIMI risk score in unselected “real life” patients. These patients are generally at increased risk of adverse events compared with patients in randomized controlled trials. Thus, the estimates of risk provided in this review may be more relevant to general cardiology practices.
The study is limited by the relatively small number of studies included in the review and the statistical heterogeneity between studies, which limits the quality of the evidence and our ability to assess publication bias. Although the Global Registry of Acute Coronary Events (GRACE) risk score has been shown to have greater discriminative ability than the TIMI risk score for mortality in-hospital and at six months in patients with confirmed acute coronary syndromes, 25 few studies have assessed the performance of the GRACE score in emergency department patients with potential acute coronary syndromes. One study reported a sensitivity of 100% and a specificity of 13%, with data needed to calculate a complete GRACE score missing for 24% of patients. 22 The focused nature of the research question, however, enabled careful assessment and meta-analysis of studies with limited clinical heterogeneity.
Our review is also limited by the lack of reporting of the characteristics of the cardiac biomarker assays and the thresholds used to define acute myocardial infarction. All of the studies included in the meta-analysis, however, used either CK-MB or cardiac troponin as recommend by the Joint European Society of Cardiology, American College of Cardiology, American Heart Association and World Health Federation Task Force for the Redefinition of Myocardial Infarction. 26
Reporting bias may also have affected the results of this review. Despite repeated attempts to contact the primary authors to obtain data about TIMI risk scores, we were unable to obtain data for two studies. 14,15 The large number of remaining patients included in the review and the strong linear relation between TIMI risk score and the cumulative incidence of cardiac events in the eight studies included in the quantitative synthesis suggests that the lack of data from these two studies may not substantially reduce the scientific value of this research.
External validity, clinical implications and future research
This review included 10 validation studies conducted in 5 countries, encompassing 17 265 patients. The broad sample of patients from which the statistical estimates were derived suggests a high degree of external validity of the findings. The 1.8% miss rate suggests that this degree of accuracy may be achieved using clinical, electrocardiography and laboratory data without need for urgent stress testing or complex rule-out protocols, which could result in important cost savings in practice settings without observation units. However, a miss rate of 1.8% may not be acceptable in all practice settings.
Conclusion
Our analysis supports the consistent performance of the TIMI risk score in the risk stratification of patients in the emergency department with potential acute coronary syndromes. For every 1000 patients in the lowest risk category (TIMI score of zero), 20 will experience a cardiac event within 30 days of the visit to the emergency department. Although this rate may render the TIMI risk score an unacceptable single reason to dismiss patients from the emergency department, this rate highlights the potential utility of this score as an adjunct to clinical acumen, as a source of information for patient decision aids and as a benchmark for researchers developing new prognostic and disposition tools.
Footnotes
-
Previously published at www.cmaj.ca
This article has been peer reviewed.
Competing interests: None declared.
Contributors: Erik Hess conceived study, analyzed the data and wrote the manuscript. Dipti Agarwal and Subhash Chandra contributed substantially to data acquisition. Mohammed Murad contributed to the design of the study and conducted the meta-regression analysis. Patricia Erwin designed and conducted the search. Judd Hollander and Ian Stiell contributed to the interpretation of the data. Victor Montori contributed to the design of the study and supervised the meta-analysis. All authors revised the manuscript for important intellectual content and approved the final version submitted for publication.
Funding: This study was jointly funded by a grant from the American Heart Association, the Society for Academic Emergency Medicine and the Emergency Medicine Foundation.
Acknowledgement: Erik Hess is the recipient of a Fellow-to-Faculty Transition award from the American Heart Association, the Emergency Medicine Foundation and the Society for Academic Emergency Medicine.