Differences in operative mortality between high- and low-volume hospitals in Ontario for 5 major surgical procedures: estimating the number of lives potentially saved through regionalization
==============================================================================================================================================================================================

* David R. Urbach
* Chaim M. Bell
* Peter C. Austin

## Abstract

**Background:** Previous research has shown that persons undergoing certain high-risk surgical procedures at high-volume hospitals (HVHs) have a lower risk of postoperative death than those undergoing surgery at low-volume hospitals (LVHs). We estimated the absolute number of operative deaths that could potentially be avoided if 5 major surgical procedures in Ontario were restricted to HVHs.

**Methods:** We collected data on all persons who underwent esophagectomy (613), colon or rectal resection for colorectal cancer (18 898), pancreaticoduodenectomy (686), pulmonary lobectomy or pneumonectomy for lung cancer (5156) or repair of an unruptured abdominal aortic aneurysm (AAA) (6279) in Ontario from Apr. 1, 1994, to Mar. 31, 1999. We calculated the excess number of operative deaths (defined as deaths in the period from the day of the operation to 30 days thereafter), adjusted for age, sex and comorbidity, among the 75% of persons treated in LVHs, as compared with the 25% treated in the highest-volume quartile of hospitals. Bootstrap methods were used to estimate 95% confidence intervals (CIs).

**Results:** Of the 31 632 persons undergoing any of the 5 procedures, 1341 (4.24%) died within 30 days of surgery. If the 75% of persons treated at the LVHs had instead been treated at the HVHs, the annual number of lives potentially saved would have been 4 (95% CI, 0 to 9) for esophagectomy, 6 (95% CI, 1 to 11) for pancreaticoduodenectomy, 1 (95% CI, –10 to 13) for major lung resection and 14 (95% CI, 1 to 25) for repair of unruptured AAA. For resection of colon or rectum, the regionalization strategy would not have saved any lives, and 17 lives (95% CI, 36 to –3) would potentially have been lost.

**Interpretation:** A small number of operative deaths are potentially avoidable by performing 4 of 5 complex surgical procedures only at HVHs in Ontario. In determining health policy, the most compelling argument for regionalizing complex surgical procedures at HVHs may not be the prevention of a large number of such deaths.

Three decades of research has provided considerable support for the hypothesis that the outcomes of complex surgical procedures are better in hospitals where high volumes of similar operations are performed.1,2,3 Some observers have advocated that certain procedures be performed only in high-volume hospitals (HVHs), at least within large metropolitan areas, where travel to HVHs is practical.4,5 Controversy over the regionalization of complex surgical procedures in Canada is most notable in pediatric cardiac surgery,6 cancer surgery7,8 and adult coronary revascularization procedures.9 Recent studies have suggested a substantial potential reduction in postoperative mortality through regionalization of major surgery at HVHs. Estimates of the potential number of lives saved per year include 4266 for 10 major surgical procedures in the United States Medicare program,10 602 for the treatment of 11 conditions in California11 and 2581 for implementing the recommendation of the Leapfrog Coalition of employers in the United States12 (which would permit employees to enrol only in health insurance plans practising “evidence-based referral”).

The benefits of systematic attempts to regionalize major surgical procedures in Canada may not be as large as these studies suggest. Some studies of regionalization have included treatment of prevalent nonsurgical conditions, such as HIV infection.11 In Canada, as compared with the United States, reduced competition among providers and single-payer funding of health care have already led to a significant amount of regionalization. We studied the potential benefit, in terms of lives saved from operative death (defined as death in the period from the day of the operation to 30 days thereafter), of performing 5 major surgical procedures in Ontario only at HVHs.

## Methods

*Data sources:* We created cohorts of persons having 1 of 5 major surgical procedures in Ontario between Apr. 1, 1994, and Mar. 31, 1999, obtaining data from electronic databases maintained by the Canadian Institute for Health Information (CIHI) and the Ontario Registered Persons Database (RPDB). The CIHI database contains a record for each discharge from an acute-care hospital in Ontario. With the RPDB we determined the person's vital status 30 days after the surgical procedure. Data on the same individuals were linked between data sets by means of an anonymous unique identifier. This research was conducted with the approval of the Research Ethics Board of Sunnybrook and Women's College Health Sciences Centre.

*Surgical procedures:* We evaluated the potential impact of regionalization on the outcomes of 5 surgical procedures: esophagectomy, excision of a segment of colon or rectum for colorectal cancer, pancreaticoduodenectomy (the Whipple operation), lung lobectomy or pneumonectomy for lung cancer and repair of unruptured abdominal aortic aneurysm (AAA). We chose these procedures because they are complex, are associated with a nontrivial risk of operative death, are indicated when the patient's expectation of survival is longer than 1 or 2 months and are procedures for which regionalization based on hospital volume has been proposed.13 We deliberately selected procedures that are uncommon and associated with a relatively high operative mortality (esophagectomy and pancreaticoduodenectomy), as well as procedures that are common and associated with a relatively low operative mortality (colon resection, major lung resection and AAA repair). For the cohort of persons having colon or rectal surgery or major lung resection, we included only those with a diagnosis of primary cancer. The codes used to identify procedures14 and diagnoses15 are listed in Table 1. The reliability of coding surgical procedures in the Ontario health databases is good, with 88% to 96% agreement between databases for procedures such as cholecystectomy and hysterectomy.16

View this table:
[Table1](http://www.cmaj.ca/content/168/11/1409/T1)

Table 1. 

*Definition of hospital volume:* Because some of the codes identifying hospitals changed during the study period owing to corporate restructuring, we identified hospitals using the code in effect during the fiscal year 1999. For each of the 5 procedures, we ranked the hospitals in order of their average annual volume. We then created hospital-volume categories that most uniformly divided the patients into 4 equal groups.13 We defined HVHs as hospitals in which the highest quartile of subjects (with respect to average annual hospital volume) had their surgery; this category contained relatively few hospitals. This strategy provided a reasonable model for volume-based regionalization, in which the 75% of persons who ordinarily would not have had surgery at HVHs would be referred to 1 of these hospitals.

*Statistical analysis:* Trends in the crude risk of death across volume quartiles were evaluated with the Mantel–Haenszel chi-squared test.17 The odds ratio for death associated with being in each of the 3 low-volume quartiles was estimated by fitting logistic regression models for each procedure cohort, in which the response variable was operative death (defined as the period from the day of the operation to 30 days thereafter), and the independent variables were hospital-volume category, age, sex and comorbidity. Our data sources did not contain information on cancer stage or person-level information on socioeconomic status. Comorbidity was represented in the regression models by a modified Charlson comorbidity score.18,19 The score was calculated with the use of all diagnosis codes of the clinical modification of the *International Classification of Diseases*, 9th revision, for the index hospital admission in the CIHI data set, except for the code defined as the most responsible diagnosis. Age and comorbidity score were treated as continuous variables in multivariable models. To account for the prevalence of 30-day mortality, odds ratios were converted to relative risks.20

The number of excess deaths within each of the low-volume quartiles was estimated by multiplying the excess risk of death associated with being in this quartile by the number of subjects in the quartile and by the baseline risk of death in the high-volume category. The total annual number of potentially avoidable deaths was the sum of the number of excess deaths in each of the 3 low-volume quartiles, averaged over the 5-year study period and rounded up to the nearest integer (a more negative integer was used for negative numbers). Using bootstrap methods21 we estimated 95% confidence intervals (CIs) around estimates of the number of lives potentially saved. For each cohort, we generated 1000 sample data sets (of sample size equal to the number of persons in the cohort) by doing repeated random sampling of the entire cohort. We estimated the potential number of lives saved for each sample data set and used the 2.5th and 97.5th percentiles of the resulting distribution to represent, respectively, the lower and upper 95% confidence limits.

## Results

During the 5-year period of the study 31 632 persons in Ontario had 1 of the 5 procedures (or procedure-diagnosis combinations). Characteristics of the study subjects according to hospital-volume quartile are presented in Table 2. The mean age was lowest among those having pancreaticoduodenectomy and highest among those having repair of unruptured AAA. For all procedures, there were more males than females. The procedure with the largest male preponderance was AAA repair; in contrast, the proportion of male subjects having colon or rectal surgery was only slightly greater than half. Charlson comorbidity scores were highest for persons who had an esophagectomy and lowest for those who had an AAA repair.

View this table:
[Table2](http://www.cmaj.ca/content/168/11/1409/T2)

Table 2. 

The number of hospitals at which the procedures were done during the study period varied according to the procedure. Esophagectomy was performed at 47 hospitals, resection of the colon or rectum at 134, pancreaticoduodenectomy at 49, major lung resection at 54 and repair of unruptured AAA at 57. The number of hospitals in each volume stratum and the average annual volumes for each hospital are listed in Table 2. The highest average annual volumes ranged from 19.0 for esophagectomy to 149.8 for resection of the colon or rectum.

Of the subjects, 1341 (4.24%) died within 30 days of surgery. The 30-day mortality rate ranged from 3.8% for resection of the colon or rectum to 13.4% for esophagectomy. The crude and adjusted risks of death within 30 days according to quartile of average hospital volume are presented in Table 3. Point estimates of the relative risk at low-volume hospitals, adjusted for age, sex and comorbidity, were greater than 1.0 for each of the low-volume quartiles for all procedures except the 3 low-volume quartiles for resection of the colon or rectum and the third quartile for major lung resection.

View this table:
[Table3](http://www.cmaj.ca/content/168/11/1409/T3)

Table 3. 

The annual number of lives potentially saved by regionalization at HVHs was 4 (95% CI, 0 to 9) for esophagectomy, 6 (95% CI, 1 to 11) for pancreaticoduodenectomy, 1 (95% CI, –10 to 13) for major lung resection and 14 (95% CI, 1 to 25) for repair of unruptured AAA (Table 4). For resection of colon or rectum, however, there would be 17 (95% CI, 36 to –3) lives potentially lost because of regionalization at HVHs.

View this table:
[Table4](http://www.cmaj.ca/content/168/11/1409/T4)

Table 4. 

## Interpretation

A major impetus for the regionalization of complex surgical procedures at HVHs is the belief that many postoperative deaths would be prevented if more people had their surgery at HVHs. We found that for some complex surgical procedures a policy of restricting certain types of surgery to HVHs could indeed result in fewer deaths during or shortly after surgery. However, the number of potentially avoidable deaths each year is small: between 1 and 14 for 4 of the procedures that we evaluated. Our data suggest that there would be no such benefit to regionalizing resection of the colon or rectum for cancer at HVHs in Ontario.

Compared with other studies of the benefit of volume-based regionalization,10,11,12 our study did not identify potential for a large reduction in the number of operative deaths. Possible explanations are that we did not consider medical interventions aside from major surgical procedures, we evaluated only 5 procedures, and the population of Ontario is substantially smaller than that of the geographic areas of the other studies. Further, it is likely that surgical procedures are already relatively regionalized in Ontario as compared with areas of the United States. For example, coronary revascularization procedures (percutaneous transluminal coronary angioplasty and coronary-artery bypass grafting) were done in 25% of the hospitals caring for persons with myocardial infarction in the United States in 1991, as compared with only 3% of similar hospitals in Ontario.22 If complex surgical procedures have already been effectively regionalized, measures to promote further regionalization will have less incremental benefit.

The results of this study must be interpreted with caution, since we made several important assumptions in estimating the effect of volume-based regionalization. We assumed a cause-and-effect association between hospital volume and outcome, that persons sent to HVHs because of regionalization would have the same risk of death as other persons having surgery at HVHs and that regionalization at HVHs is feasible for all persons. To the extent that these assumptions are unrealistic, their bias would be towards an increased benefit of regionalization. We looked only at 30-day mortality and not in-hospital mortality or mortality over a different period, such as 60 days.23,24,25 Our data sources did not contain information on cancer stage or socioeconomic indicators, which are important determinants of prognosis. We did not evaluate outcomes other than short-term mortality, such as long-term survival after cancer surgery or limb ischemia after AAA repair. Therefore, we cannot exclude a more substantial benefit of regionalization for outcomes that may be more sensitive measures of the quality of surgical care.

What do our findings say about the potential value of regionalizing complex surgery at high-volume centres? First, our data suggest that the value of volume-based regionalization should be carefully studied before major policy initiatives are undertaken. The absolute health benefits of regionalization must be better quantified and should be weighed against potential drawbacks, such as patient preference for local care26 and the impact on the delivery of rural health care.4 Second, the benefit of regionalization is condition-specific and in general will be larger for procedures that are common, have a high mortality risk or have a strong association between volume and outcome. Third, further research should focus on the determinants of poor outcomes, such as short-term mortality, in lower-volume hospitals. If volume-based regionalization becomes impractical or impossible as a policy measure, then quality-improvement initiatives will necessarily be directed towards improving structures and processes of care at institutions with poorer outcomes. Finally, it is important to study outcomes other than short-term mortality in assessing the quality of surgical care. There may be good reasons for volume-based regionalization of certain complex surgical procedures; however, the perception that the main benefit of regionalization is a substantial reduction in postoperative mortality may be erroneous. Rather, the benefits of improved care might be better identified by using more sensitive and specific measures of the quality of care, such as long-term survival after cancer surgery, cancer-free survival or health-related quality of life, or procedure-specific outcomes such as renal dysfunction following AAA repair.

In conclusion, we found that under assumptions favouring the feasibility of regionalization, a small number of operative deaths are potentially avoidable by restricting 4 of the 5 complex surgical procedures we studied to HVHs. In determining health policy, the most compelling argument for regionalizing complex surgical procedures at these centres may not be the prevention of a substantial number of postoperative deaths.

## Footnotes

*   Portions of this paper were presented at the Canadian Surgical Forum, London, Ont., Sept. 20, 2002.
    
    This article has been peer reviewed.
    
    *Contributors:* Dr. Urbach contributed substantially to the study conception and design, acquisition of data, and analysis and interpretation of data, and he drafted the article. Dr. Bell contributed substantially to the study conception and to analysis and interpretation of the data; he also revised the manuscript critically for important intellectual content. Dr. Austin contributed substantially to the analysis and interpretation of the data and revised the manuscript critically for important intellectual content. All authors gave final approval to the version to be published.
    
    *Acknowledgements:* We are indebted to Minh Le Duong-Hua for linking and cleaning the data sets, to Deanna Rothwell for her assistance with programming the bootstraps and to Andreas Laupacis for insightful comments on an earlier version of this paper.
    
    This research was supported in part by an internal investigator-initiated grant from the Institute for Clinical Evaluative Sciences and by the physicians of Ontario through the Physicians' Services Incorporated Foundation. Dr. Urbach is a Career Scientist of the Ontario Ministry of Health and Long-Term Care, Health Research Personnel Development Program. Dr. Bell is supported by a Clinician Scientist Award from the Canadian Institutes of Health Research.
    
    *Competing interests:* None declared.

## References

1.  1. Luft HS, Bunker JP, Enthoven AC. Should operations be regionalized? The empirical relation between surgical volume and mortality. N Engl J Med 1979; 301: 1364-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM197912203012503&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=503167&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1979HX81100003&link_type=ISI) 

2.  2. Houghton A. Variation in outcome of surgical procedures. Br J Surg 1994; 81: 653-60.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8044538&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994NN63900005&link_type=ISI) 

3.  3. Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med 2002;137:511-20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/0003-4819-137-6-200209170-00012&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12230353&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000178069100006&link_type=ISI) 

4.  4. Birkmeyer JD. Should we regionalize major surgery? Potential benefits and policy considerations. J Am Coll Surg 2000;190:341-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S1072-7515(99)00270-7&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10703862&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000085495900012&link_type=ISI) 

5.  5. Birkmeyer JD. High-risk surgery — Follow the crowd. JAMA 2000;283: 1191-3.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.283.9.1191&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10703783&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000085424100030&link_type=ISI) 

6.  6. Sibbald B. Move to close low-volume surgical sites gaining speed. CMAJ 2002; 167:177.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNjcvMi8xNzciO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTY4LzExLzE0MDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

7.  7. Grilli R. Specialization and cancer: Words with too many meanings should be handled with care. CMAJ 2001;164:210-1.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNjQvMi8yMTAiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTY4LzExLzE0MDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

8.  8. Wexler MJ. More procedures, better quality of care? Is there a case for regionalization of pancreatic resection for neoplasm? CMAJ 1999;160:671-3.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJjbWFqIjtzOjU6InJlc2lkIjtzOjk6IjE2MC81LzY3MSI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xNjgvMTEvMTQwOS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

9.  9. Hemmelgarn BR, Ghali WA, Quan H. A case study of hospital closure and centralization of coronary revascularization procedures. CMAJ 2001;164: 1431-5.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTY0LzEwLzE0MzEiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTY4LzExLzE0MDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

10. 10. Birkmeyer JD, Lucas FL, Wennberg DE. Potential benefits of regionalizing major surgery in Medicare patients. Eff Clin Pract 1999;2:277-83.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10788026&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 

11. 11. Dudley RA, Johansen KL, Brand R, Rennie DJ, Milstein A. Selective referral to high-volume hospitals: estimating potentially avoidable deaths. JAMA 2000; 283: 1159-66.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.283.9.1159&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10703778&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000085424100025&link_type=ISI) 

12. 12. Birkmeyer JD, Finlayson EVA, Birkmeyer CM. Volume standards for high-risk surgical procedures: potential benefits of the Leapfrog initiative. Surgery 2001;130:415-22.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1067/msy.2001.117139&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11562662&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000171151000003&link_type=ISI) 

13. 13. Birkmeyer JD, Siewers AE, Finlayson EVA, Stukel TA, Lucas FL, Batista I, et al. Hospital volume and surgical mortality in the United States. N Engl J Med 2002;346:1128-37.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa012337&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11948273&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000174880600006&link_type=ISI) 

14. 14. Canadian Centre for Health Information Nosology Reference Centre. *Canadian classification of diagnostic, therapeutic, and surgical procedures*. Second printing. Ottawa: Statistics Canada; 1993. 
    
    

15. 15. *1999 Hospital & Payer ICD-9-CM: International classification of diseases, 9th rev, clinical modification.* 5th ed. Salt Lake City, Utah: Medicode; 1998.
    
    

16. 16. Iron K, Goel V, Williams JI. *Concordance of hospital discharge abstracts and physician claims for surgical procedures in Ontario* (ICES working paper series #42). North York (ON): Institute for Clinical Evaluative Sciences in Ontario; 1995.
    
    

17. 17. Mantel N. Chi-square tests with 1 degree of freedom: extensions of the Mantel–Haenszel procedure. J Am Stat Assoc 1963;58:690-700.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.2307/2282717&link_type=DOI) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1963P215900013&link_type=ISI) 

18. 18. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373-83.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0021-9681(87)90171-8&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3558716&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1987G855900002&link_type=ISI) 

19. 19. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0895-4356(92)90133-8&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1607900&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1992JA22600006&link_type=ISI) 

20. 20. Zhang J, Yu KF. What's the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes. JAMA 1998;280:1690-1.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.19.1690&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9832001&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076980700035&link_type=ISI) 

21. 21. Efron B, Tibshirani RJ. *An introduction to the bootstrap*. New York: Chapman and Hall; 1993.
    
    

22. 22. Tu JV, Pashos CL, Naylor D, Chen E, Normand SL, Newhouse JP, et al. Use of cardiac procedures and outcomes in elderly patients with myocardial infarction in the United States and Canada. N Engl J Med 1997;336:1500-5.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199705223362106&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9154770&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WZ76700006&link_type=ISI) 

23. 23. Begg CB, Cramer LD, Hoskins WJ, Brennan MF. Impact of hospital volume on operative mortality for major cancer surgery. JAMA 1998;280:1747-51.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.20.1747&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9842949&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000077081400029&link_type=ISI) 

24. 24. Schrag D, Cramer LD, Bach PB, Cohen AB, Warren JL, Begg CB. Influence of hospital procedure volume on outcomes following surgery for colon cancer. JAMA 2000;284:3028-35.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.284.23.3028&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11122590&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000165847200028&link_type=ISI) 

25. 25. Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, et al. The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491-507.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00000658-199810000-00006&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9790339&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076474300011&link_type=ISI) 

26. 26. Finlayson SR, Birkmeyer JD, Tosteson AN, Nease RF Jr. Patient preferences for location of care: implications for regionalization. Med Care 1999;37:204-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-199902000-00010&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10024124&link_type=MED&atom=%2Fcmaj%2F168%2F11%2F1409.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000078517500010&link_type=ISI)