Variations in rates of appropriate and inappropriate carotid endarterectomy for stroke prevention in 4 Canadian provinces
=========================================================================================================================

* James Kennedy
* Hude Quan
* William A. Ghali
* Thomas E. Feasby

## Abstract

**Background:** Carotid endarterectomy (CE), when performed on appropriate patients, reduces the incidence of stroke, yet there are marked variations in rates of this procedure. We sought to determine reasons for the variation in CE rates in 4 Canadian provinces.

**Methods:** We identified all CEs performed in 4 Canadian provinces between January 2000 and December 2001, inclusive. From chart review and expert assessment, we determined the proportion of these procedures that were appropriate, inappropriate or of uncertain appropriateness, using the RAND/UCLA Appropriateness Method. We sought to determine the variation in rates by province and whether the variation was due to differences in type of hospital, surgical specialty or surgical volume.

**Results:** Overall, 1656 (52.3%) of the 3167 CEs studied were performed for appropriate indications. The proportions of appropriate procedures were 78.2% (176/225) in Saskatchewan, 58.7% (481/819) in Alberta, 49.1% (350/713) in Manitoba and 46.0% (649/1410) in British Columbia (*p* < 0.001 across provinces). Rates of appropriate procedures per 100 000 population ranged from 44.3 in Manitoba to 16.2 in Saskatchewan (*p* < 0.001 across provinces). CEs were more likely to be appropriate when performed by a neurosurgeon compared with all other surgeons (74.4% v. 49.4% were appropriate; *p* < 0.001), when performed by surgeons doing fewer than 31 procedures over 2 years compared with surgeons doing more than 31 (70.1% v. 49.5% were appropriate; *p* < 0.001) and when performed in hospitals doing fewer than 135 procedures per year compared with hospitals doing more than 135 (63.4% v. 49.1% were appropriate; *p* < 0.001). Overall, 10.3% of procedures were done for inappropriate reasons.

**Interpretation:** Our findings suggest some overuse (for inappropriate or uncertain indications) but also some underuse (low population rates in some regions). High rates of CE are associated with lower rates of appropriateness for both surgeons and hospitals. That 1 in 10 CEs is done inappropriately suggests the need for preoperative assessment of appropriateness.

The efficacy of carotid endarterectomy (CE) to prevent stroke is well established.1,2,3,4Clinical trials have shown that CE reduces the 5-year risk of stroke by 16.0% when performed because of symptomatic lesions causing more than 70% stenosis.5 The risk reduction is more modest (4.6% and 5.9%, respectively) in cases of symptomatic moderate (50% to 69%) stenosis or asymptomatic stenosis (> 60%).4,5 However, concerns remain regarding the effectiveness of the procedure outside of clinical trials, when the potential benefit may be reduced.6,7 Although national societies have issued guidelines on indications for CE,8,9 in some cases CE is performed on patients who do not meet these guidelines.

The RAND/UCLA (University of California at Los Angeles) Appropriateness Method,10,11 developed in response to concerns about possible unnecessary use of procedures, is perhaps the most respected approach to defining appropriate care, combining best evidence and expert opinion.12 The first study of the appropriateness of CE, published in 1988, showed that only one-third of procedures were appropriate.13 A Canadian study in 1997 showed similar results.14The role of health system factors in choosing patients appropriately for CE is not well explored. Administrative databases allow only limited appreciation of the decision-making process that leads to the operating room.

Our objectives were to describe the variation in appropriateness of CE in 4 Canadian provinces, to document rates of appropriate CE in the provinces and to explore potential explanatory factors, such as hospital type, surgeon specialty and number of CEs performed each year per surgeon and per hospital.

## Methods

This study used the RAND/UCLA Appropriateness Method.10,11 We developed a comprehensive set of 203 clinical scenarios in which CE might be considered. The scenarios incorporated a host of relevant factors, including the type of symptoms (or lack thereof), the degree of carotid stenosis and the presence of other conditions. An example of a scenario is a recent history of a right-hemisphere transient ischemic attack (TIA) in a patient with more than 70% stenosis of the right carotid artery and atrial fibrillation.

We provided a Canadian expert panel (2 vascular surgeons, 2 neurosurgeons, 2 stroke neurologists, a neuroradiologist, a general internist and a family physician, all nominated by their specialty societies) with a summary of the relevant evidence for the efficacy of CE. The panellists independently rated each scenario on a 9-point scale on which a score of 1 to 3 indicated an inappropriate indication, 4 to 6 an uncertain indication and 7 to 9 an appropriate indication for CE. Inappropriate indications were defined as those for which the expected negative consequences for the patient (pain, disability, risk of death) exceed the expected benefit (in health status, quality of life or longevity) and may indicate overuse. Uncertain indications were those for which the net result of expected negative consequences and benefits is equivocal and highlight the need for further work to clarify the evidence. Appropriate indications were defined as those for which the expected benefit exceeds the expected negative consequences.

The panel was then convened and the participants provided with their initial rating and the median rating of the group for each scenario. Guided by a chair, the group discussed each scenario with the relevant evidence and individually rated each scenario again. The median of the final ratings for each scenario was used to determine the appropriateness of CE in that scenario.

Using administrative databases, we identified all CE procedures performed in Manitoba, Saskatchewan, Alberta and British Columbia (BC) between January 2000 and December 2001, inclusive. We then sought permission from each institutional review board to review the hospital charts for all cases. All hospitals performing CE in the 4 provinces agreed to take part in the study.

Two chart reviewers, trained in extracting data from a sample set of charts, performed chart reviews at each institution. Using 50 charts reviewed by each reviewer and the kappa statistic,15 we found substantial agreement between the reviewers (kappa = 0.66; 95% confidence interval 0.49–0.84) on the global rating of appropriateness.

Adults were defined as people 20 years of age or older. The numbers of adults in each province was determined from 2001 Canadian Census information.16 Teaching hospitals were defined as hospitals affiliated with a university medical school. Case volumes of hospitals and surgeons were classified as high or low according to the median number of operations performed in the study period. A descriptive analysis of the inappropriate operations was performed. The χ2 test was used to compare the proportions of appropriate, uncertain and inappropriate CEs across the 4 provinces, as well as across categories of other factors: hospital teaching status, hospital volume, surgical specialty and surgeon case volume.

## Results

There were 3360 CEs performed in Manitoba, Saskatchewan, Alberta and BC in the 2-year period 2000–2001; 3309 charts (98.5%) were available for review. We excluded 142 charts from analysis: 18 had missing demographic data, 78 had no description of the imaging of either carotid artery, 32 had no description of the contralateral carotid artery in cases of moderate symptomatic stenosis, and 14 could not be matched to 1 of the prespecified clinical scenarios. Thus, data for 3167 (94.2%) of the 3360 CEs were analyzed.

Charts were reviewed from 22 institutions, 13 of which were teaching hospitals. Of the 79 surgeons performing the operations, 19 were neurosurgeons, 47 vascular surgeons and 13 “other” (general and cardiothoracic) surgeons. Vascular surgeons performed most of the operations (2508 [79.2%]) compared with neurosurgeons (371 [11.7%]) and other surgeons (288 [9.1%]). Of the 3167 CEs, 1905 (60.2%) were performed because of symptomatic carotid artery disease. Table 1 shows the general characteristics of the patients undergoing CE across the 4 provinces.

View this table:
[Table1](http://www.cmaj.ca/content/171/5/455/T1)

Table 1. 

Overall, 1656 cases (52.3%) were determined to have an appropriate indication, 1185 (37.4%) to have an uncertain indication and 326 (10.3%) to have an inappropriate indication. The inappropriate indications for CE in 94 cases of symptomatic carotid disease (4.9% of the 1905 cases of symptomatic disease) included mild stenosis (< 50%), chronic occlusion of the internal carotid artery, a single TIA in the distant past plus moderate stenosis (50%–69%) and atrial fibrillation, and either dementia or recent myocardial infarction. The inappropriate indications for CE in 232 cases of asymptomatic carotid disease (18.4% of the 1262 cases of asymptomatic disease) included moderate stenosis with no evidence of brain infarction and mild stenosis.

The annual rate of CE per 100 000 adults (hereafter called the population rate of CE) was 44.3 in Manitoba (713 operations over 2 years), 16.2 in Saskatchewan (225 operations), 19.3 in Alberta (819 operations) and 24.1 in BC (1410 operations). Table 1 shows the proportions of appropriate, uncertain or inappropriate CE, which differed significantly across the 4 provinces, ranging from 46.0% in BC to 78.2% in Saskatchewan for appropriate, 16.0% in Saskatchewan to 43.9% in Manitoba for uncertain and 5.8% in Saskatchewan to 13.0% in BC for inappropriate.

Fig. 1 shows the provincial population rate of CE by appropriateness. The rate for an appropriate indication ranged from 11.1 in BC to 21.7 in Manitoba. Perhaps most notable is the contrast in results for Saskatchewan and Manitoba shown in Table 1 and Fig. 1. Saskatchewan had the highest percentage of appropriate CE (Table 1) but a low population rate of CE for appropriate indications (Fig. 1), a finding that hints at underuse. Meanwhile, Manitoba had a lower percentage of appropriate CE (Table 1), which suggests overuse, but in absolute terms had a higher population rate of CE for appropriate indications than the other provinces (Fig. 1).

![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/171/5/455/F1.medium.gif)

[Figure1](http://www.cmaj.ca/content/171/5/455/F1)

**Fig. 1: Annual rates of appropriate, uncertain and inappropriate carotid endarterectomy per 100 000 adults in 2000 and 2001, by province. Man = Manitoba, Sask = Saskatchewan, Alta = Alberta, BC = British Columbia.** 

The number of operations performed per hospital over the 2 years ranged from 8 to 429, with a median of 135. The proportion of operations by hospital with an appropriate indication ranged from 20.3% to 89.7%, with an uncertain indication 6.9% to 55.6% and with an inappropriate indication 0% to 25.7%. Teaching hospital status did not influence the rate at which appropriate CE was performed. Hospitals with a high volume were less likely to perform appropriate CE (*p* < 0.001) (Table 2).

View this table:
[Table2](http://www.cmaj.ca/content/171/5/455/T2)

Table 2. 

The number of operations performed per surgeon over the 2 years ranged from 1 to 337, with a median of 31. Neurosurgeons were more likely to perform appropriate operations (276/371 operations [74.4%]) than were vascular surgeons (1270/2508 [50.6%]) or other surgeons (110/288 [38.2%]) (*p* < 0.001) (Table 2). High-volume surgeons were less likely to perform appropriate operations (*p* < 0.001) (Table 2). The combination of high-volume surgeons in high-volume hospitals yielded the lowest rate of appropriateness (47.8%), and the combination of low-volume surgeons in low-volume hospitals yielded the highest rate (73.3%). High-volume surgeons in low-volume hospitals had an appropriateness rate of 65.4%. Low-volume surgeons in high-volume hospitals had an appropriateness rate of 57.8%.

The number of operations performed by each specialty of surgeon differed significantly across the provinces. Neurosurgeons performed 39.8% of the CEs (326 operations) in Alberta but only 0.9% (13), 1.1% (8) and 10.7% (24) of the CEs in BC, Manitoba and Saskatchewan, respectively.

## Interpretation

The appropriateness ratings developed for this study can be seen as an accurate reflection of the contemporary Canadian interpretation of the evidence for the use of CE. This is an important point because the Canadian medical community is more conservative than the US community in interpreting the evidence for the efficacy of CE for asymptomatic carotid disease,17,18as is well illustrated by the 37.4% rate of CEs performed for uncertain indications in this study (mostly asymptomatic cases), as compared with 4.5% in a recent US study.17

Our finding that 1 in 10 CEs was performed inappropriately indicates overuse. In fact, in one hospital more than 25% of CEs were inappropriate. Physicians are still performing CE contrary to well-established evidence. Two large randomized controlled trials have unequivocally shown that CE performed for symptomatic mild carotid stenosis is harmful. Inappropriate surgery could be curtailed by applying appropriateness ratings before surgery19 or by consistently educating surgeons with feedback of their own results.20 In addition to the patient factors used to determine appropriateness, surgical skill is of critical importance, and its assessment requires outcome measurements in all cases.20

This study allows a deeper understanding of the factors contributing to marked practice variation across western Canada than was previously described with the use of an administrative database.21 CE volume had a clear inverse relation with the rate of appropriate CE, both high-volume surgeons and high-volume hospitals having lower CE appropriateness rates. This finding was especially marked for high-volume surgeons in high-volume hospitals.

The relation between uncertainty, overuse and underuse in practice variation is complex. This study points to each of these phenomena as contributors to the differences demonstrated. Saskatchewan had the highest proportion of appropriate CEs (78.2%) and the lowest rate of overuse (only 0.9 inappropriate CEs per 100 000 population) but the lowest overall population rate of CE (16.2). BC had perhaps the least favourable combined profile, with both the lowest population rate of CE for appropriate cases (11.1) and a low proportion of appropriate CEs (46.0%). The BC profile simultaneously suggests overuse of CE overall and underuse for individuals who need the procedure.

Manitoba had the highest population rates of appropriate (21.7), uncertain (19.4), inappropriate (3.2) and total CE (44.3), indicating possible overuse. However, although Manitoba had the second-lowest proportion of appropriate CEs (49.1%), surgeons performed appropriate CE at almost twice the population rate of other provinces. This suggests underuse of CE in the other 3 provinces. These figures are most likely explained by Manitoba's having the lowest threshold for referral of all patients with carotid artery disease for surgical opinion.

Neurosurgeons had a higher appropriateness rate than their surgical colleagues, which may help explain why Alberta had the second-highest proportion of appropriate CEs. Either neurosurgeons have a better appreciation of the current evidence and therefore choose patients more carefully for CE or the patients in the different surgical groups have different pathways to operation, a symptomatic patient being more likely to see a neurologist and be referred to a neurosurgeon.

Procedures should be performed only on those who have the chance to benefit. The appropriate use of CE appears to be influenced by where a patient lives, the hospital attended and the physicians seen. Our findings point to a complex interplay between overuse for inappropriate indications and underuse in those with appropriate indications. More discussion is required to reduce the disconnection between clinical practice guidelines and clinical practice, thus ensuring the translation of evidence into practice.

𝛃 See related article page [473](http://www.cmaj.ca/lookup/volpage/171/473?iss=5)

## Footnotes

*   This article has been peer reviewed.
    
    *Contributors:* Co-investigators William Ghali and Thomas Feasby applied for funding, planned the study and analyzed the data. James Kennedy and Hude Quan helped develop the data collection tools and analyze the data. James Kennedy drafted the manuscript; all the coauthors revised the manuscript for important intellectual content, and all the authors approved the version to be published.
    
    *Acknowledgements:* Louis Girard developed the database programming. Marie McClelland and Caroline Stevenson performed the chart reviews. Tara Lye assisted with the literature review and the scenario preparation. Janet Bennett assisted with the logistics.
    
    The Expert Panel comprised the following: Dr. Anthony M. Kaufmann, neurosurgeon, Winnipeg, Man.; Dr. Daryl S. Kucey, vascular surgeon, Toronto, Ont.; Dr. Louise-Hélène Lebrun, neurologist, Montreal, Que.; Dr. Finlay A. McAllister, general internist, Edmonton, Alta.; Dr. Walter Montanero, neuroradiologist, Toronto, Ont.; Dr. Steven J. Phillips, neurologist, Halifax, NS; Dr. Anthony J.G. Salvian, vascular surgeon, Vancouver, BC; Dr. Rupinder K. Soin, family physician, Edmonton, Alta.; and Dr. M. Christopher Wallace, neurosurgeon, Toronto, Ont.
    
    This work was supported by research grants from the Heart and Stroke Foundation of Alberta, the Northwest Territories and Nunavut and the Canadian Institutes of Health Research (CIHR). Dr. Kennedy was supported by the Heart and Stroke Foundation, the Canadian Stroke Network, the Canadian Institutes of Health Research (CIHR), and the CIHR/Rx&D Research Program together with AstraZeneca Canada Inc. Dr. Quan is a Population Health Scholar of the Alberta Heritage Foundation for Medical Research. Dr. Ghali was supported by a Health Scholar Award from the Alberta Heritage Foundation for Medical Research and holds a Government of Canada Research Chair in Health Services Research.
    
    *Competing interests:* None declared.

## References

1.  1. North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade stenosis. N Engl J Med 1991;325:445-3.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199108153250701&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1852179&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1991GA76300001&link_type=ISI) 

2.  2. Barnett HJ, Taylor DW, Eliasziw M, Fox AJ, Ferguson GG, Haynes RB, et al. Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis. N Engl J Med 1998;339:1415-25.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199811123392002&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9811916&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000077051700002&link_type=ISI) 

3.  3. European Carotid Surgery Trialists' Collaborative Group. Randomized trial of endarterectomy for recently symptomatic carotid stenosis: final results of the MRC European Carotid Surgery Trial (ECST). Lancet 1998;351:1379-87.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0140-6736(97)09292-1&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9593407&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000073563400007&link_type=ISI) 

4.  4. Executive Committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis. JAMA 1995; 273:1421-8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1995.03520420037035&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7723155&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995QW27900029&link_type=ISI) 

5.  5. Rothwell PM, Eliasziw M, Gutnikov SA, Fox AJ, Taylor DW, Mayberg MR, et al. Analysis of pooled data from the randomised controlled trials of endarterectomy for symptomatic carotid stenosis. Lancet 2003;361:107-16.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0140-6736(03)12228-3&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12531577&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000180428000009&link_type=ISI) 

6.  6. Barnett HJM, Meldrum HE, Eliasziw M. The appropriate use of carotid endarterectomy. CMAJ 2002;166:1169-79.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTY2LzkvMTE2OSI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xNzEvNS80NTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

7.  7. Warlow CP. Carotid endarterectomy for asymptomatic carotid stenosis. Better data, but the case is still not convincing. BMJ 1998;317:1468.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMTcvNzE3MS8xNDY4IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE3MS81LzQ1NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

8.  8. Biller J, Feinberg WM, Castaldo JE, Whittemore AD, Harbaugh RE, Dempsey RJ, et al. Guidelines for carotid endarterectomy: a statement for healthcare professionals from a Special Writing Group of the Stroke Council, American Heart Association. Circulation 1998;97:501-9.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjg6Ijk3LzUvNTAxIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE3MS81LzQ1NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

9.  9. Findlay JM, Tucker WS, Ferguson GG, Holness RO, Wallace MC, Wong JH. Guidelines for the use of carotid endarterectomy: current recommendations from the Canadian Neurosurgical Society. CMAJ 1997;157:653-9.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNTcvNi82NTMiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTcxLzUvNDU1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. 10. Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care 1986;2(1):53-63.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1017/S0266462300002774&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10300718&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 

11. 11. Brook RH. The RAND/UCLA Appropriateness Method. In: McCormick KA, Moore SR, Siegel RA, editors. *Methodology perspectives*. AHCPR Pub no 95-0009. Rockville (MD): Public Health Service, US Department of Health and Human Services; 1994. p. 59-70.
    
    

12. 12. Naylor CD. What is appropriate care? N Engl J Med 1998;338:1918-20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199806253382612&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9637815&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000074352800012&link_type=ISI) 

13. 13. Winslow CM, Kosecoff JB, Chassin M, Kanouse DE, Brook RH. The appropriateness of carotid endarterectomy. N Engl J Med 1988;318:721-7.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3279313&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1988M602200001&link_type=ISI) 

14. 14. Wong JH, Findlay JM, Suarez-Almazor ME. Regional performance of carotid endarterectomy: appropriateness, outcomes, and risk factors for complications. Stroke 1997;28:891-8.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjg6IjI4LzUvODkxIjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE3MS81LzQ1NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

15. 15. Donner A, Eliasziw M. A goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance-testing and sample size estimation. Stat Med 1992;11:1511-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1002/sim.4780111109&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1410963&link_type=MED&atom=%2Fcmaj%2F171%2F5%2F455.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1992JN61400008&link_type=ISI) 

16. 16. Statistics Canada. Selected age distribution indexes, Canada, provinces and territories, 2001. In: *2001 Census Analysis Series*, supplement to *The Daily* (official release bulletin). Available: [http://www12.statcan.ca/english/census01/Products/Analytic/companion/age/ewt1.cfm](http://www12.statcan.ca/english/census01/Products/Analytic/companion/age/ewt1.cfm) (accessed 2004 June 7).
    
    

17. 17. Halm EA, Chassin MR, Tuhrim S, Hollier LH, Popp AJ, Ascher E, et al. Revisiting the appropriateness of carotid endarterectomy. Stroke 2003;34:1464-71.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjM0LzYvMTQ2NCI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xNzEvNS80NTUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

18. 18. Barnett HJM, Meldrum HE, Eliasziw M. The appropriate use of carotid endarterectomy. CMAJ 2002;166:1169-79.
    
    

19. 19. Wright CJ, Chambers GK, Robens-Paradise Y. Evaluation of indications for and outcomes of elective surgery. CMAJ 2002;167:461-6.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNjcvNS80NjEiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTcxLzUvNDU1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

20. 20. Findlay JM, Nykolyn L, Lubkey TB, Wong JH, Mouradian M, Senthilselvan A, et al. Auditing carotid endarterectomy: a regional experience. *Can J Neurol Sc*i 2002;29:326-32.
    
    

21. 21. Feasby TE, Quan H, Ghali WA. Geographic variation in the rate of carotid endarterectomy in Canada. Stroke 2001;32:2417-22.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjEwOiIzMi8xMC8yNDE3IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE3MS81LzQ1NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=)