Ruling out coronary artery disease in primary care: development and validation of a simple prediction rule
==========================================================================================================

* Stefan Bösner
* Jörg Haasenritter
* Annette Becker
* Konstantinos Karatolios
* Paul Vaucher
* Baris Gencer
* Lilli Herzig
* Monika Heinzel-Gutenbrunner
* Juergen R. Schaefer
* Maren Abu Hani
* Heidi Keller
* Andreas C. Sönnichsen
* Erika Baum
* Norbert Donner-Banzhoff

## Abstract

**Background:** Chest pain can be caused by various conditions, with life-threatening cardiac disease being of greatest concern. Prediction scores to rule out coronary artery disease have been developed for use in emergency settings. We developed and validated a simple prediction rule for use in primary care.

**Methods:** We conducted a cross-sectional diagnostic study in 74 primary care practices in Germany. Primary care physicians recruited all consecutive patients who presented with chest pain (*n* = 1249) and recorded symptoms and findings for each patient (derivation cohort). An independent expert panel reviewed follow-up data obtained at six weeks and six months on symptoms, investigations, hospital admissions and medications to determine the presence or absence of coronary artery disease. Adjusted odds ratios of relevant variables were used to develop a prediction rule. We calculated measures of diagnostic accuracy for different cut-off values for the prediction scores using data derived from another prospective primary care study (validation cohort).

**Results:** The prediction rule contained five determinants (age/sex, known vascular disease, patient assumes pain is of cardiac origin, pain is worse during exercise, and pain is not reproducible by palpation), with the score ranging from 0 to 5 points. The area under the curve (receiver operating characteristic curve) was 0.87 (95% confidence interval [CI] 0.83–0.91) for the derivation cohort and 0.90 (95% CI 0.87–0.93) for the validation cohort. The best overall discrimination was with a cut-off value of 3 (positive result 3–5 points; negative result ≤ 2 points), which had a sensitivity of 87.1% (95% CI 79.9%–94.2%) and a specificity of 80.8% (77.6%–83.9%).

**Interpretation:** The prediction rule for coronary artery disease in primary care proved to be robust in the validation cohort. It can help to rule out coronary artery disease in patients presenting with chest pain in primary care.

See related commentary by Stevens and Lasserson, page [1281](http://www.cmaj.ca/lookup/volpage/182/1281?iss=12)

Chest pain is common. Studies have shown a lifetime prevalence of 20% to 40% in the general population. 1 Its prevalence in primary care ranges from 0.7% to 2.7% depending on inclusion criteria and country, 2–4 with coronary artery disease being the underlying cause in about 12% of primary care patients. 1,5 General practitioners are challenged to identify serious cardiac disease reliably and also protect patients from unnecessary investigations and hospital admissions. Because electrocardiography and the cardiac troponin test are of limited value in primary care, 6,7 history taking and physical examination remain the main diagnostic tools.

Most published studies on the diagnostic accuracy of signs and symptoms for acute coronary events have been conducted in high-prevalence settings such as hospital emergency departments. 8–10 Predictive scores have also been developed for use in emergency departments, mainly for the diagnosis of acute coronary syndromes. 11–13 To what degree these apply in primary care is unknown. 14–16 

A clinical prediction score to rule out coronary artery disease in general practice has been developed. 17 However, it did not perform well when validated externally. The aim of our study was to develop a simple, valid and usable prediction score based on signs and symptoms to help primary care physicians rule out coronary artery disease in patients presenting with chest pain.

## Methods

We conducted a cross-sectional diagnostic study with a delayed-type reference standard 18 in a primary care setting. Coronary artery disease was the reference condition and included stable coronary artery disease and acute coronary syndromes.

### Derivation cohort

#### Participating physicians and patients

We approached 209 primary care physicians in the State of Hesse, Germany, of whom 74 (35%) agreed to participate in the study. Participating physicians were asked to recruit all patients who had chest pain that was either the reason for the consultation or was described by the patient during the consultation when asked by the physician. The recruitment period lasted 12 weeks for each practice and for logistical reasons was staggered in four waves between October 2005 and July 2006.

Every patient above 35 years of age with pain localized on the anterior chest wall in the area between the clavicles, the lower costal margins and the posterior axillary lines was to be included. Patients whose chest pain had subsided for more than one month, whose chest pain had been investigated already or who saw their physician for follow-up of their chest pain were excluded.

#### Data collection

Primary care providers took a standardized history and performed a physical examination using a report form that had been pilot tested and modified accordingly. Patients were contacted by phone six weeks and six months after the index consultation. Study assistants blinded to the results of the index tests asked about the course of the patient’s chest pain and treatments, including hospital admissions and medications. Random audits were performed by the study assistants by searching routine documentation of participating practices to identify cases of chest pain not included in the study.

#### Reference standard

A reference panel of one cardiologist, one primary care physician and one research staff member of the Department of General Practice at the University of Marburg reviewed baseline and follow-up data for every patient. The panel decided on whether coronary artery disease was present or absent at the time of the index consultation (delayed-type reference standard 18). It based its decision on all of the results available after the follow-up period (index questionnaire, the attending physician’s provisional diagnosis, coronary angiography, if available, and results of noninvasive tests such as electrocardiography, exercise test and echocardiography). This design is based on the assumption that serious diseases such as coronary artery disease would manifest themselves within the mentioned period. A diagnosis of coronary artery disease was based on recommendations from the German Program for Disease Management Guidelines. 19 

#### Statistical analysis

For the univariable analysis, we calculated the sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and diagnostic odds ratio for each item covered by the case report form. To arrive at a smaller subset of criteria, we selected index-test items that had a *p* value of less than 0.05 and likelihood ratios indicating at least moderate diagnostic accuracy (a positive likelihood ratio greater than 2 or a negative ratio less than 0.5). Those items were included as independent variables in the multivariable logistic regression analysis. The dependent variable was coronary artery disease.

We used the variables found to be significant in the multivariable analysis to build an initial score. The odds of having coronary artery disease was compared between patients with and without each risk factor by calculating odds ratios (ORs) with 95% confidence intervals (CIs). We used the regression coefficient rounded to the next integer as a weighting factor for the different variables and calculated the area under the curve (receiver operating characteristic curve). To derive the best possible user-friendly version of the score, we used a backward stepwise procedure to exclude different variables and analyzed whether this would lead to a reduction of the area under the curve. 20 

In a sensitivity analysis, we simplified the preduction rule by excluding contributing variables through stepwise reduction. The guiding principle of this process was to prevent significant changes in the area under the curve and the false-negative rate.

The final prediction rule consisted of five variables (including one compound variable). The prognostic ability of the prediction rule to discriminate between patients with and without coronary artery disease was assessed by analyzing the area under the curve. We calculated measures of diagnostic accuracy (sensitivity, specificity, likelihood ratios and predictive values) for different cut-off points of the prediction scores.

Additional details of the methodology appear in Appendix 1 (available at [www.cmaj.ca/cgi/content/full/cmaj.100212/DC1](http://www.cmaj.ca/cgi/content/full/cmaj.100212/DC1)).

### Validation cohort

We performed external validation of the prediction rule using data from the TOPIC (Thoracic Pain in Community) study, a multicentre clinical study involving primary care patients with chest pain. 5 Patients in that study were recruited between March and June 2001 by general practitioners in 58 independent urban and rural practices and one primary care outpatient clinic in western Switzerland. Physicians consecutively enrolled all patients over 16 years of age who reported chest pain during their visits. The presence of chest pain was ascertained according to the usual practice of each physician. Additional follow-up information was obtained after 3 and 12 months. All final diagnoses at 12 months’ follow-up were retained and grouped into six categories, one of which was coronary artery disease.

Four of the five variables on which our prediction rule is based could be derived directly from the data for the validation cohort. For the fifth variable (patient assumes pain is of cardiac origin), we used induced anxiety (defined as a positive answer to the question “Are you feeling very worried about your chest pain?”) as a proxy variable.

Additional details appear in Appendix 1.

## Results

### Derivation cohort

The majority of participating physicians were male and had practices in urban areas (Table 1). Physicians approached 1355 consecutive patients with chest pain; 106 were excluded (99 refused to participate, and 7 did not meet the inclusion criteria). We obtained data on symptoms and signs for the reamining 1249 patients (Figure 1).

View this table:
[Table1](http://www.cmaj.ca/content/182/12/1295/T1)

**Table 1:** Characteristics of patients and primary care physicians in the derivation and validation cohorts


![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/182/12/1295/F1.medium.gif)

[Figure1](http://www.cmaj.ca/content/182/12/1295/F1)

**Figure 1:** Flow chart showing selection of patients whose data were used to develop the prediction rule (derivation cohort). *Includes 60 patients lost to follow-up and 11 who died; both groups provided enough data to be included in the cohort.

We excluded patients with a history of trauma (*n* = 39), those who dropped out (*n* = 3) and those in whom coronary artery disease could not be diagnosed by the reference panel (*n* = 8). Although 60 patients were lost to follow-up (did not answer phone or moved) and 11 died during the follow-up period, enough data were available for them to be included in the final sample. The final sample included 1199 patients for analysis (Figure 1). Of these, 180 (15.0%) were given the reference diagnosis of coronary artery disease by the panel, including 44 (3.7%) with acute coronary syndromes.

#### Development of the prediction rule

Fourteen variables were significantly associated with coronary artery disease in the univariable analysis. Ten of these variables were found to be significant in the multivariable regression analysis and were used to develop the prediction rule (Table 2).

View this table:
[Table2](http://www.cmaj.ca/content/182/12/1295/T2)

**Table 2:** Independent variables that contributed to the development of a prediction score to rule out coronary artery disease in primary practice

In the sensitivity analysis, we found that the removal of 5 of the 10 variables and weighting of the remaining 5 variables affected the area under the curve only minimally. The area under the curve of the 24 other, more complicated, score models ranged from 0.87 to 0.89.

The final prediction rule contained five determinants and ranged from 0 to 5 points (Table 3).

View this table:
[Table3](http://www.cmaj.ca/content/182/12/1295/T3)

**Table 3:** Components of the prediction rule used to determine the presence or absence of coronary artery disease in patients with chest pain in primary care

#### Goodness of fit

When applied to the patients in the derivation cohort, the area under the curve (Figure 2) was 0.87 (95% CI 0.83–0.91).

![Figure2](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/182/12/1295/F2.medium.gif)

[Figure2](http://www.cmaj.ca/content/182/12/1295/F2)

**Figure 2:** Sensitivity and specificity analysis (receiver operating characteristic curves) for the performance of the prediction rule in detecting coronary artery disease in primary care at various cut-off points in the derivation and validation cohorts. In the derivation cohort, the area under the curve was 0.87 (95% confidence interval [CI] 0.83–0.91). For the validation cohort, the area under the curve was 0.90 (95% CI 0.87–0.93). Higher values for the area under the curve indicate better overall performance.

Table 4 shows the number of patients with complete data who did and did not have coronary artery disease for each score value. For a cut-off value of 3 (positive result 3–5 points, negative result ≤ 2 points), the score had a sensitivity of 86.4% (95% CI 78.5%–91.7%), a specificity of 75.2% (95% CI 71.8%–78.3%), a positive predictive value of 34.9% (95% CI 29.3%–40.9%), a negative predictive value of 97.3% (95% CI 95.5%–98.4%) and a false-negative rate of 2.7%. Additional details about the sensitivity analysis of the reference standard, replacement of missing values and internal validation through bootstrapping appear in Appendix 2 (available at [www.cmaj.ca/cgi/content/full/cmaj.100212/DC1](http://www.cmaj.ca/cgi/content/full/cmaj.100212/DC1)).

View this table:
[Table4](http://www.cmaj.ca/content/182/12/1295/T4)

**Table 4:** Number of patients with and without coronary artery disease for each score value

### Validation cohort

The validation cohort comprised 672 patients with chest pain (mean age 55 years); 66 (9.8%) were new patients to the practice (Table 1). 5,17 A total of 85 patients (12.6%) had coronary artery disease, including 10 (1.5%) with acute coronary syndromes.

#### Goodness of fit

When applied to the validation cohort, our preduction rule showed consistent results. The area under the curve (Figure 2) was 0.90 (95% CI 0.87–0.93). Estimates of the diagnostic accuracy of three different cut-off values of the prediction rule are shown in Table 5. The use of a lower cut-off value of 1 or 2 (positive result 1–5 points or 2–5 points) maximized sensitivity up to 100%, albeit with a significant drop in specificity. A cut-off value of 3 (positive result 3–5 points) was associated with the best overall discrimination, with a sensitivity of 87.1% (95% CI 79.9%–94.2%), a specificity of 80.8% (95% CI 77.6%–83.9%) and a negative predictive value of 97.7% (95% CI 96.4%–99.1%).

View this table:
[Table5](http://www.cmaj.ca/content/182/12/1295/T5)

**Table 5:** Measures of diagnostic accuracy of three cut-off values of prediction score for coronary artery disease in validation cohort

## Interpretation

Using data from 1199 patients who presented with chest pain to 74 primary care physicians in Germany (derivation cohort), we developed a prediction rule as a diagnostic aid for primary care physicians to rule out coronary artery disease in patients presenting with chest pain. The rule consists of five determinants (age/sex, known clinical vascular disease, patient assumes pain is of cardiac origin, pain worse during exercise and pain not reproducible by palpation) that can be easily identified during clinical consultation. When applied to a validation cohort of patients with chest pain in 58 primary care practices in Switzerland, we found that a cut-off value of 3 (positive result 3–5 points, negative result ≤ 2 points) had the best overall discrimination, with a sensitivity of 87.1% (95% CI 79.9%–94.2%) and a specificity of 80.8% (95% CI 77.6%–83.9%).

The main purpose of the prediction rule is to offer primary care physicians an effective tool to rule out coronary artery disease in patients at low risk with sufficient confidence. This always involves a clinical trade-off: avoiding over-investigation on the one hand while ensuring that very few people with cardiac-related chest pain are sent home with false reassurance. With a cut-off value of 2 (positive result 2–5 points, negative result ≤ 1 point), the prediction rule had a high sensitivity (98.8%) for ruling out coronary artery disease, albeit with the above mentioned trade-off. A cut-off value of 3 (positive result 3–5 points) provided the best overall discrimination. With a sensitivity of 87.1%, this value would mean that about one in seven people with cardiac-related chest pain would be missed. However, because primary care physicians work in a low-prevalence setting, only 2% of patients presenting with chest pain would initially be classified wrongly as negative. Applying a strategy of “watchful waiting” would still allow for a diagnosis of coronary artery disease at a later point.

Three patients (0.6%) with acute coronary syndromes in the derivation cohort and two (0.4%) in the validation cohort had a false-negative result at the cut-off value of 3 points. At first glance, this might be a reason for concern. Given that chronic coronary artery disease is much more prevalent than acute disease, our analysis is dominated by subacute or chronic presentations; therefore, the reliability of results might be higher in patients with chronic disease. In acute emergency situations, vital signs and symptoms should also be taken into account.

Another clinical prediction rule to identify coronary artery disease in primary care has been developed. However, external validation showed a noticeable drop in the area under the curve when applied to our data. 17 Scores developed in emergency departments include a broad range of different variables, including age, sex, risk factors and pain characteristics. 12,21–24 One study determined the usefulness of pain reproduced by palpation of the chest wall for defining low-risk patients. 11 However, most of these studies used other reference diagnoses such as acute coronary syndromes, and all were performed in a setting where the prevalence of coronary artery disease is much higher than that in a primary care setting. 1,16,25 Sox and colleagues developed a logistic function for chest pain in a secondary care setting to estimate the probability of coronary artery disease. 14 Like our score, theirs also contained age, pain during exertion and history of myocardial infarction as variables. However, when applied to a low-prevalence setting, there was a strong reduction in the predictive power of their score.

Previous reviews could not define an important role for signs and symptoms in general in the diagnosis of coronary artery disease and acute coronary syndromes. 8–10 Among the individual variables that were evaluated, age and history of prior myocardial infarction were effective for diagnosing typical angina (coronary artery disease), 9 and the absence of chest-wall tenderness on palpation was effective for diagnosing acute coronary syndromes. 8,10 The variable “patient assumes pain is of cardiac origin” was not used in other studies and therefore remains a unique variable in our study population.

### Strengths and limitations

The diagnostic tests we evaluated had been chosen according to the results of an exploratory analysis of primary care physicians’ practice. 26 Our study procedures ensured recruitment of consecutive patients from a large number of urban and rural practices. An independent reference panel reviewed the diagnostic data, and we performed external validation of the prediction rule using data from a patient cohort derived from a similar setting, albeit in the context of a different country and health care system. 18,20 Therefore, this phase III diagnostic study followed a clearly defined design, and its results should have immediate relevance for clinical practice. 27,28 

Given the low prevalence of coronary artery disease, we considered the use of a delayed-type reference standard (i.e., an expert panel reviewing follow-up data for the reference diagnosis) to be most suitable. However, the panel often had to make its decision on the basis of limited data because there was no requirement for physicians to use defined investigations. As a result, the reference standard in our study cannot be regarded as perfect.

The data for the validation cohort were also obtained by a prospective and consecutive study design with a delayed type, unblinded reference standard. One of the variables in our prediction rule had to be replaced by a proxy variable. Despite this restriction, and the fact that the validation study was performed in a different health care system and language, our score proved to be robust and performed equally in the derivation and validation cohorts.

### Conclusions

Our findings show that the use of a simple prediction rule based exclusively on symptoms and signs can help to rule out coronary artery disease in patients presenting with chest pain in a primary care setting. Our prediction rule proved to be robust when validated internally and externally.

The score is not meant to replace the clinical judgment of the primary care physician, especially when he or she encounters a patient with findings highly suggestive of acute coronary syndromes or any other reason requiring urgent hospital care. Rather, it is meant for the predominant number of remaining patients who present with chest pain and in whom coronary artery disease could still be the underlying reason. We expect that the prediction rule will help to prevent over-diagnosis and needless investigation.

## Footnotes

*   Previously published at [www.cmaj.ca](http://www.cmaj.ca)
    
    This article has been peer reviewed.
    
    **Competing interests:** Juergen Schaefer is a scientific advisor for MSD and Essex Pharma; he has received speaker fees from MSD, Essex Pharma, Braun, Genzyme and Solvay and holds a patent for percutaneous transluminal coronary angioplasty (PTCA) support devices. No competing interests declared by the other authors.
    
    **Contributors:** Norbert Donner-Banzhoff formulated the research question, designed the study and supervised its conduct together with Andreas Sönnichsen. Stefan Bösner, Jörg Haasenritter, Annette Becker, Konstantinos Karatolios, Paul Vaucher, Baris Gencer, Lilli Herzig, Monika Heinzel-Gutenbrunner, Juergen Schaefer, Maren Abu Hani, Heidi Keller, Erika Baum and Norbert Donner-Banzhoff were involved in the acquisition, analysis and interpretation of the data. Norbert Donner-Banzhoff is the guarantor. Stefan Bösner drafted the article, and the coauthors revised it for important intellectual content; all of the authors approved the final version of the manuscript submitted for publication.
    
    **Funding:** This study was funded by the German Federal Ministry of Education and Research (grant no. BMBF FKZ 01GK0401). The funding source had no involvement in the study.
    
    **Acknowledgement:** The authors thank all of the participating primary care physicians for their cooperation, Muazzez Ilhan for her help with data collection and documentation, and Justine Rochon for statistical advice.

## REFERENCES

1.  1. Ruigomez A, Rodriguez LA, Wallander MA, et al. Chest pain in general practice: incidence, comorbidity and mortality. Fam Pract 2006;23:167–74.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6ODoiMjMvMi8xNjciO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTgyLzEyLzEyOTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  2. Svavarsdottir AE, Jonasson MR, Gudmundsson GH, et al. Chest pain in family practice. Diagnosis and long-term outcome in a community setting. Can Fam Physician 1996;42:1122–8.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8704488&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1996UQ49400016&link_type=ISI) 

3.  3. Nilsson S, Scheike M, Engblom D, et al. Chest pain and ischaemic heart disease in primary care. Br J Gen Pract 2003;53:378–82.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTMvNDkwLzM3OCI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xODIvMTIvMTI5NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

4.  4. Verdon F, Burnand B, Herzig L, et al. Chest wall syndrome among primary care patients: a cohort study. BMC Fam Pract 2007;8:51.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1471-2296-8-51&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17850647&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 

5.  5. Verdon F, Herzig L, Burnand B, et al. Chest pain in daily practice: occurrence, causes and management. Swiss Med Wkly 2008;138:340–7.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18561039&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 

6.  6. Rutten FH, Kessels AG, Willems FF, et al. Electrocardiography in primary care; is it useful?Int J Cardiol 2000;74:199–205.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0167-5273(00)00284-9&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10962122&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000089568800014&link_type=ISI) 

7.  7. Ebell MH, White LL, Weismantel D. A systematic review of troponin T and I values as a prognostic tool for patients with chest pain. J Fam Pract 2000;49:746–53.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10947143&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 

8.  8. Mant J, McManus RJ, Oakes RA, et al. Systematic review and modelling of the investigation of acute and chronic chest pain presenting in primary care. *Health Technol Assess* 2004;8:iii, 1–158.
    
    

9.  9. Chun AA, McGee SR. Bedside diagnosis of coronary artery disease: a systematic review. Am J Med 2004;117:334–43.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.amjmed.2004.03.021&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15336583&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000223777500007&link_type=ISI) 

10. 10. Bruyninckx R, Aertgeerts B, Bruyninckx P, et al. Signs and symptoms in diagnosing acute myocardial infarction and acute coronary syndrome: a diagnostic meta-analysis. Br J Gen Pract 2008;58:105–11.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18307844&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000254185400011&link_type=ISI) 

11. 11. Lee TH, Cook EF, Weisberg M, et al. Acute chest pain in the emergency room. Identification and examination of low-risk patients. Arch Intern Med 1985;145:65–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/archinte.1985.00360010085013&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3970650&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1985ABV5100010&link_type=ISI) 

12. 12. Morise AP, Haddad WJ, Beckner D. Development and validation of a clinical score to estimate the probability of coronary artery disease in men and women presenting with suspected coronary disease. Am J Med 1997;102:350–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0002-9343(97)00086-7&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9217616&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997XA46300006&link_type=ISI) 

13. 13. Pryor DB, Shaw L, McCants CB, et al. Value of the history and physical in identifying patients at increased risk for coronary artery disease. Ann Intern Med 1993;118:81–90.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/0003-4819-118-2-199301150-00001&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8416322&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1993KF91800001&link_type=ISI) 

14. 14. Sox HC Jr, Hickam DH, Marton KI, et al. Using the patient’s history to estimate the probability of coronary artery disease: a comparison of primary care and referral practices. Am J Med 1990;89:7–14.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0002-9343(90)90090-Z&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2242131&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1990DN17900002&link_type=ISI) 

15. 15. Grijseels EW, Deckers JW, Hoes AW, et al. Pre-hospital triage of patients with suspected myocardial infarction. Evaluation of previously developed algorithms and new proposals. Eur Heart J 1995;16:325–32.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZWhqIjtzOjU6InJlc2lkIjtzOjg6IjE2LzMvMzI1IjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4Mi8xMi8xMjk1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

16. 16. Buntinx F, Knockaert D, Bruyninckx R, et al. Chest pain in general practice or in the hospital emergency department: is it the same?Fam Pract 2001;18:586–9.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6ODoiMTgvNi81ODYiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTgyLzEyLzEyOTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

17. 17. Gencer B, Vaucher P, Herzig L, et al. Ruling out coronary heart disease in primary care patients with chest pain: a clinical prediction score. BMC Med 2010;8:9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1741-7015-8-9&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20092615&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 

18. 18. Knottnerus JA, Muris JW. Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 2003;56:1118–28.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0895-4356(03)00206-3&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=14615003&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000186694000012&link_type=ISI) 

19. 19. Ollenschlager G, Lelgemann M, Kopp I. The German Program for Disease Management Guidelines: CHD Guideline 2006. Short review. [article in German]*Med Klin (Munich)* 2006;101:993–8.
    
    

20. 20. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488–94.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1997.03540300056034&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9020274&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WG05400037&link_type=ISI) 

21. 21. Martinez-Selles M, Bueno H, Sacristan A, et al. Chest pain in the emergency department: incidence, clinical characteristics and risk stratification. Rev Esp Cardiol 2008;61:953–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1157/13125517&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18775237&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 

22. 22. Ruiz-Ros V, Sanchis-Fores J, Bodi-Peris V, et al. Predictive value of chest pain score for the diagnosis of acute coronary syndromes. *Med Clin (Barc)* 2006;126:1–4.
    
    

23. 23. Sanchis J, Bodi V, Llacer A, et al. Risk stratification of patients with acute chest pain and normal troponin concentrations. Heart 2005;91:1013–8.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6OToiOTEvOC8xMDEzIjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4Mi8xMi8xMjk1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

24. 24. Westfall JM, Van Vorst RF, McGloin J, et al. Triage and diagnosis of chest pain in rural hospitals: implementation of the ACI-TIPI in the High Plains Research Network. Ann Fam Med 2006;4:153–8.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6NzoiNC8yLzE1MyI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xODIvMTIvMTI5NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

25. 25. Kohn MA, Kwan E, Gupta M, et al. Prevalence of acute myocardial infarction and other serious diagnoses in patients presenting to an urban emergency department with chest pain. J Emerg Med 2005;29:383–90.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jemermed.2005.04.010&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16243193&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000232959500003&link_type=ISI) 

26. 26. Hani MA, Keller H, Vandenesch J, et al. Different from what the textbooks say: how GPs diagnose coronary heart disease. Fam Pract 2007;24:622–7.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6ODoiMjQvNi82MjIiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTgyLzEyLzEyOTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. 27. Sackett DL, Haynes RB. The architecture of diagnostic research. BMJ 2002; 324:539–41.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjQvNzMzNi81MzkiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTgyLzEyLzEyOTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

28. 28. Hayden JA, Cote P, Steenstra IA, et al. Identifying phases of investigation helps planning, appraising, and applying the results of explanatory prognosis studies. J Clin Epidemiol 2008;61:552–60.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2007.08.005&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=18471659&link_type=MED&atom=%2Fcmaj%2F182%2F12%2F1295.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000256164700007&link_type=ISI)