Evaluation of indications for and outcomes of elective surgery
==============================================================

* Charles J. Wright
* G. Keith Chambers
* Yoel Robens-Paradise

## Abstract

**Background:** Wide small-area variations in the rates of elective surgical procedures and lack of systematic outcome measurement have raised questions about the appropriateness of such surgery. Our objective was to determine the feasibility of routine evaluation of indications for and outcomes of elective surgery.

**Methods:** Participants consisted of 138 surgeons and 5313 patients who underwent 1 or more of 6 specific surgical procedures (for a total of 6274 operations). Surgical indications were evaluated according to published guidelines. Patients' self-reported health-related quality of life (HRQOL) before and at appropriate intervals after surgery was measured with standard, validated generic and disease-specific instruments. Patient-specific results were routinely sent to the surgeons, from whom feedback was requested.

**Results:** Surgeons provided information on the indications for surgery for 44% to 95% of the 6 procedures, and the indications matched the guidelines in 73% to 99% of cases. Completed HRQOL questionnaires were returned by 58% of the patients. Postoperative HRQOL scores were markedly improved in most patients, but in 2% to 26% of the various procedures, there was either no change or a deterioration in HRQOL. In most of the procedure groups a small proportion of patients had relatively minor symptoms and disability preoperatively, but in the cataract surgery group this proportion was large. Opinion among the participating surgeons was divided as to the potential value of this method of evaluation. The cost of the outcome evaluation program was about $12/patient.

**Interpretation:** Evaluation of indications for and outcomes of elective surgery could be implemented systematically at reasonable cost and could be included in an accountability framework for health services. Most surgeons were not enthusiastic about this kind of evaluation.

The appropriateness of indications for elective surgery has stimulated much debate. The large variations in age- and sex-adjusted surgical rates that are reported whenever the issue is examined1,2,3 raise questions about the adequacy of access in some areas and the possibility of inappropriate surgery in others, especially for procedures known to have highly subjective indications.

It is remarkable that in reports of surgical results so little attention has been paid to measuring the outcome for which most elective surgical procedures were designed, namely health-related quality of life (HRQOL). There have been sporadic reports of HRQOL results4,5,6 in a research context, but there has been no previous attempt to introduce systematic HRQOL measurement as a routine component of the evaluation of elective surgery.

Because indications for and outcomes of surgery are becoming important measures of quality, we undertook this project to assess the feasibility and potential value to providers and decision-makers in the health care system of measuring each of them routinely.

## Methods

In Vancouver between November 1999 and September 2000, 5313 consecutive patients were booked for the following 6 high-volume elective surgical procedures: cataract replacement, cholecystectomy, hysterectomy, lumbar diskectomy, prostatectomy and total hip replacement. Patients with cancer and those who underwent arthroplastic revision were excluded. All acute care hospitals in the region participated, namely Vancouver Hospital and Health Sciences Centre, Providence Health Care (consisting of St. Paul's Hospital, St. Vincent's Hospital and Mount St. Joseph's Hospital) and Richmond Hospital. The specialty distribution of the 138 participating surgeons is displayed in Table 1. The project steering committee consisted of surgeons and administrators representing the BC Ministry of Health, the Vancouver/Richmond Health Board and the 6 surgical divisions within the 5 hospitals.

View this table:
[Table1](http://www.cmaj.ca/content/167/5/461/T1)

Table 1. 

For each patient booked for surgery, the surgeon or a designate was asked to complete a clinical indications form based on a set of proprietary guidelines developed by InterQual (a division of McKesson Corporation, San Francisco), known as Indications for Surgery and Procedures. The information gathered from the clinical indications form was reviewed by one of the project nurses to determine if the case profile matched the guidelines. A sample of 900 hospital charts was reviewed to determine whether the information necessary to review the surgical indications could have been obtained from the hospital record rather than directly from the surgeon.

For the outcomes part of the project, we used the SF-36 generic HRQOL questionnaire and an appropriate disease-specific HRQOL questionnaire for each procedure (see below for more details). The questionnaires were formatted to allow optical scanning of the responses. Questionnaires were sent to all patients before surgery (except those in the hysterectomy group [the Menorrhagia Outcomes Questionnaire was designed for single postoperative application only]) and at appropriate postoperative intervals ranging from 3 months for cataract to 1 year for lumbar diskectomy and total hip replacement. A custom-built computer program was used to track the approximately 13 000 patient surveys, flag the date for the postoperative mailing and scan the responses into a database. The per-patient cost of the outcomes measurement program was determined. The self-reported outcomes for each patient were sent to the relevant surgeon as soon as the postoperative surveys were received, and surgeons were asked to provide anonymous feedback about the value of this process.

The SF-367 was chosen as the generic HRQOL instrument for all procedures. The following disease-specific instruments were also used: the Visual Function Assessment (based on the VF-14 index8) for cataract extraction, the Gastrointestinal Quality of Life Index9 for cholecystectomy, the Menorrhagia Outcomes Questionnaire10 for hysterectomy (distributed after surgery only), the North American Spine Society lumbar spine outcome assessment instrument11 for lumbar disk surgery, the American Urological Association symptom index for benign prostatic hyperplasia12 for prostatectomy, and the Western Ontario and McMaster Universities Osteoarthritis Index13 (WOMAC) for total hip replacement. Because of the culturally diverse population in Vancouver, we ensured that staff were available to communicate with patients who spoke English, Mandarin, Cantonese or Punjabi. Chinese-language versions of the SF-36 and the Visual Function Assessment were also available.

The project raised controversy among the members of the steering committee about the distinction between clinical research and quality management, but the Behavioural Research Ethics Board of the University of British Columbia determined that patient consent was not required, because these activities were fundamentally related to the management and reporting of quality of care.

## Results

Only one department in one of the hospitals declined to participate in the study. The response rates to the requests for clinical information varied widely by surgeon, by procedure and by institution. The response rate for individual surgeons ranged from 0% to 100% of procedures. No obvious patterns of response in terms of type of procedure or volume of practice were evident. The overall collection rate of clinical indication forms varied from 44% to 94%, depending on the procedure (Table 1), with the lowest response rates for hysterectomy and total hip replacement and the highest response rate for lumbar diskectomy. For patients for whom a clinical indications form was submitted by the surgeon, the indications for surgery matched the guidelines for 73% to 99% of cases, depending on the procedure (Table 2).

View this table:
[Table2](http://www.cmaj.ca/content/167/5/461/T2)

Table 2. 

The response rate for the HRQOL survey packages mailed directly to patients (and reminder postcards and telephone calls where necessary) ranged from 52% to 77%, depending on the procedure. In the returned surveys, 95% of the questions were answered.

For each procedure, information was compiled on changes in all 8 domains of the SF-36 score and in the disease-specific instrument scores, as well as the distribution of preoperative (except for hysterectomy) and postoperative symptom and disability scores (Table 2). The complete data sets and graphs of results for all 6 procedures are available online.14

The most striking improvements in HRQOL were seen after lumbar disk surgery and total hip replacement; for these procedures, very large positive changes occurred in the disease-specific instruments and in almost all of the SF-36 domains (Table 2). The disease-specific instruments showed that most patients in all procedure groups experienced benefit from surgery, but for a certain proportion (ranging from 2% for total hip replacement to 26% for cataract replacement) HRQOL scores after the operation were worse than before. From the large data sets available for each procedure 2 illustrative examples were chosen for presentation here: cataract replacement and total hip replacement.

Of the patients booked for cataract surgery, 32% scored higher than 90 points on the 100-point Visual Function Assessment before surgery (Fig. 1). The mean scores before and after surgery for all patients were 79 (standard deviation [SD] 17) and 88 (SD 15) respectively. Seventy percent of the patients who underwent cataract surgery experienced improvement, as recorded by the Visual Function Assessment, but for 27% the score was the same or worse after the procedure (Fig. 2). As expected (because of the general nature of the questions), there was no change in any of the domains of the generic HRQOL instrument, the SF-36.

![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/167/5/461/F1.medium.gif)

[Figure1](http://www.cmaj.ca/content/167/5/461/F1)

**Fig. 2: Cataract surgery. Distribution of postoperative changes in Visual Function Assessment score. Data were missing for 3% of patients.** 


![Figure2](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/167/5/461/F2.medium.gif)

[Figure2](http://www.cmaj.ca/content/167/5/461/F2)

**Fig. 1: Cataract surgery. Distribution of preoperative Visual Function Assessment scores. A score of 100 indicates that the patient has no visual complaints whatever; a score of 0 indicates virtual blindness. Data were missing for 1% of patients.** 

For the patients who underwent total hip replacement, there was a large mean positive change in HRQOL as measured by both the SF-36 and the WOMAC scales (Table 2). For more than 90% of patients, the WOMAC scale indicated improvement after the procedure, whereas only 2% of patients reported deterioration (Fig. 3).

![Figure3](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/167/5/461/F3.medium.gif)

[Figure3](http://www.cmaj.ca/content/167/5/461/F3)

**Fig. 3: Total hip replacement. Distribution of postoperative changes in score on the Western Ontario and McMaster Universities Osteoarthritis Index. Data were missing for 4% of patients.** 

The staffing cost for producing, tracking and scanning the surveys for one patient was estimated as $2.69 on the basis of a workload estimation model. Supply costs including postage (mail-out and stamped self-addressed envelope) were $2.75 per survey, and the cost of producing the surgeon's report was approximately $0.50. Therefore, the total costs were approximately $6 per outcome survey or $12 for a preoperative and postoperative protocol. This estimate does not include the costs of nurses identifying patients, which could be eliminated by implementing an automated procedure in the booking offices, or the costs of surgeons' time, project management, data analysis and interpretation.

Just over half (21 or 53%) of the 40 surgeons who responded to our request for feedback had positive comments about the project and the information it produced, but the remainder felt it was of little value and indicated that they did not wish to receive such information on their patients in the future.

## Interpretation

These results suggest that it is feasible to carry out a guideline-based review of the indications for elective surgery and to measure HRQOL outcomes routinely at reasonable cost.

### Surgical indications

Given that the project involved 138 surgeons from 6 disciplines in 5 hospitals, it is unsurprising that the level of cooperation was variable, with data on indications reported for 0% to 100% of procedures by individual surgeon, and one division in one of the hospitals declining to participate at all. It is difficult to interpret the data on the rate of compliance with the guidelines (73% to 99%) for several reasons: not all of the surgeons agreed with the proprietary guidelines used, secondary review of individual cases by a physician was not done systematically, and information was incomplete or missing for more than one-third of patients enrolled, which suggests the possibility of selection bias by the surgeons. However, the data suggest that local practices often vary from the guidelines. For cholecystectomy, lumbar diskectomy and total hip replacement, the rates of criteria match were consistently high.

Our review of hospital records demonstrated that these records are not adequate sources of information on indications for surgery. For systematic implementation of a review of the indications for surgery, the necessary clinical information would have to be collected from the physicians' offices. Most of the project's personnel costs arose from time spent in hospital surgical booking offices to identify patients for the study and in attempts to obtain the necessary information from surgeons' offices. As with an outcomes management system, automation of the surgical booking process would be necessary for successful implementation on a large scale.

### Patient-reported outcomes

The preoperative HRQOL information indicated that patients booked for these 6 elective surgical procedures had a wide range of symptoms and disability. There was evidence that the more pain and disability experienced before operation, the more likely that there would be benefit from surgery. For most of the procedures, the proportion of patients reporting minimal symptoms and disability preoperatively was small, but of those scheduled for cataract surgery 32%, 15% and 4% had a preoperative visual function score of greater than 90, greater than 95 and 100 respectively (on a scale of 100). These findings suggest that the threshold indications for cataract surgery are now very low. The mean preoperative Visual Function Assessment score of 79 is comparable to the range of 64 to 77 reported in the literature.6,8,15

The disease-specific instruments showed substantial improvement in patients' self-reported HRQOL after surgery for all of the procedures, but in a certain proportion the postoperative score was unchanged or worse. The wide range in the proportion with poorer outcomes (2% for total hip replacement to 26% for cataract extraction) may reflect differences in surgical complication rates, severity of disease or appropriateness of the indications for surgery in the first place. For the generic HRQOL instrument, the changes were strongly positive after lumbar diskectomy and total hip replacement, moderately positive for cholecystectomy and hysterectomy, minimal for prostatectomy and absent for cataract surgery; these changes might be expected, given the likely overall health effects of the diseases studied.

We were disappointed by the lack of interest expressed by almost half of the surgeons in receiving patients' self-reported outcome information. This lack of interest suggests that many are not yet aware of the recent advances in the discipline of HRQOL measurement.8,9,10,11,16,17,18 Some surgeons may have difficulty accepting the evidence that patients' self-reported HRQOL is a more valid outcome measure than their own impressions of outcomes.19,20

## Implications for health services and policy

The appropriateness and outcomes of health care interventions are important quality-of-care issues that should be considered in the accountability framework for health services now under consideration by the provinces. The results of this study suggest that clinicians, managers and trustees must all be involved if indications and outcomes are to be evaluated. For systematic implementation of such evaluation, it would be effective and less costly to target specific areas in a structured sampling program, but strong incentives for participation by surgeons would be necessary.

The wide range of severity of symptoms and disability for which elective surgery was recommended raises questions about the appropriateness of some procedures. With systematic implementation of an outcomes assessment program and better understanding of the validity of patients' self-reported outcomes, surgeons and referring physicians may eventually use such information to investigate the reasons for poor outcomes in individual patients and thus refine the surgical selection process. Since all surgery carries risk, the potential benefits and harms of a procedure must be examined for patients with non-life-threatening conditions causing minimal symptoms. In elective surgery with high demand and limited resources, it seems reasonable to ask surgeons to document their reasons for operating and patients to assess their condition before and after surgery. Furthermore, making such information available to surgeons and referring physicians could improve the quality of care and clinical decision-making.

A detailed report of this project, including all the data, is available in PDF format at [www.resio.org](http://www.resio.org).

## Footnotes

*   *This article has been peer reviewed.* 
    
    *Contributors:* Charles J. Wright was the principal investigator and mainly responsible for the design of the protocol. He submitted the application for funding, chaired the steering committee and was involved at all stages of the project. G. Keith Chambers contributed to the design of the project, sat on the steering committee throughout the project's execution, and played a major role in the data analysis and interpretation. Yoel Robens-Paradise was the project director with the major responsibility for all aspects of the running of the project.
    
    *Acknowledgements:* We offer our warm appreciation to all of the members of the steering committee (Dr. Ron Calderisi, Associate Head of Surgery, Mount Saint Joseph Hospital and St. Vincent's Hospital, Providence Health Care, Vancouver; Dr. Keith Chambers, Assistant Director, Centre for Clinical Epidemiology, Vancouver Hospital and Health Sciences Centre, Vancouver; Mark Chase, Vice President, Planning and Evaluation, Vancouver/Richmond Health Board, Vancouver; Sue Emmons, Director, QUIST, Vancouver Hospital and Health Sciences Centre, Vancouver; Dr. Charles Fisher, Clinical Assistant Professor, Vancouver General Hospital, University of British Columbia, Vancouver; Dr. Ken Hughes, Head, Division of Orthopaedic Surgery, The Richmond Hospital, Richmond, BC; Dr. David Manson, Head, Department of Surgery, St. Paul's Hospital, University of British Columbia, Vancouver; Dr. David Rollins, Clinical Professor, Department of Ophthalmology, Medical Director, Eye Centre Operative Rooms, and Head, Division of Comprehensive Ophthalmology, Department of Ophthalmology, University of British Columbia, Vancouver Hospital and Health Sciences Centre, Vancouver; Dr. Barry Sanders, Clinical Professor, Vancouver Hospital and Health Sciences Centre and BC Women's Hospital, Vancouver; Dr. Allan Thompson, Executive Director, Information Support, BC Ministry of Health, Victoria; and Barbara Trerise, Leader, Evaluation and Utilization Management, Providence Health Care, Vancouver), the project staff (Gwynneth Aikens, Robert Chernuka, Tejinder J. Jagpal, Drew Mildon, Rena O'Neil, Pauline Sumel and Elsie Taylor) and the participating surgeons. This project was supported by a financial contribution from the Health Transition Fund, Health Canada, and was sponsored by the Vancouver/Richmond Health Board.
    
    *Competing interests:* None declared.

## References

1.  1. *The Dartmouth atlas of health care* [online]. Hanover (NH): Center for the Evaluative Clinical Sciences at Dartmouth Medical School; 2000. Available: [www.dartmouthatlas.org](http://www.dartmouthatlas.org) (accessed 2002 May 16).
    
    

2.  2. Institute for Clinical Evaluative Sciences. Atlas reports series. Toronto (ON): The Institute; [various dates]. Available: [www.ices.on.ca](http://www.ices.on.ca) [select “Publications”] (accessed 2002 May 16).
    
    

3.  3. Population utilization rates and referrals for easy comparative tables (PURRFECT) [electronic database on CD-ROM]. Version 5.0. Victoria: British Columbia Ministry of Health; 2000.
    
    

4.  4. Laupacis A, Bourne R, Rorabeck C, Feeny D, Wong C, Tugwell P, et al. The effect of elective total hip replacement on health-related quality of life. J Bone Joint Surg Am 1993;75:1619-26.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8245054&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 

5.  5. Barry MJ, O'Leary MP. Advances in benign prostatic hyperplasia. The developmental and clinical utility of symptom scores. Urol Clin North Am 1995; 22: 299-307.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7539176&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 

6.  6. Cassard SD, Patrick DL, Damiano AM, Legro MW, Tielsch JM, Diener-West M, et al. Reproducibility and responsiveness of the VF-14. An index of functional impairment in patients with cataracts. Arch Ophthalmol 1995; 113:1508-13.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/archopht.1995.01100120038005&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7487617&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995TJ06300005&link_type=ISI) 

7.  7. Ware JE Jr, Gandek B. Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) project. J Clin Epidemiol 1998; 51: 903-12.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0895-4356(98)00081-X&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9817107&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076855600003&link_type=ISI) 

8.  8. Steinberg EP, Tielsch JM, Schein OD, Javitt JC, Sharkey P, Cassard SD, et al. The VF-14. An index of functional impairment in patients with cataract. Arch Ophthalmol 1994;112:630-8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/archopht.1994.01090170074026&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8185520&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994NM02800020&link_type=ISI) 

9.  9. Eypasch E, Wood-Dauphinee S, Williams JI, Ure B, Neugebauer E, Troidl H. The Gastrointestinal Quality of Life Index. A clinical index for measuring patient status in gastroenterologic surgery [in German]. Chirurgie 1993;64: 264-74.
    
    

10. 10. Lamping DL, Rowe P, Clarke A, Black N, Lessof L. Development and validation of the Menorrhagia Outcomes Questionnaire. Br J Obstet Gynaecol 1998; 105: 766-79.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1471-0528.1998.tb10209.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9692419&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000075035100014&link_type=ISI) 

11. 11. Daltroy LH, Cats-Baril WL, Katz JN, Fossel AH, Liang MH. The North American Spine Society Lumbar Spine Outcome Assessment Instrument: reliability and validity tests. Spine 1996;21:741-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00007632-199603150-00017&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8882698&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1996UB09900017&link_type=ISI) 

12. 12. Barry MJ, Fowler FJ Jr, O'Leary MP, Bruskewitz RC, Holtgrewe HL, Mebust WK, et al. The American Urological Association symptom index for benign prostatic hyperplasia. J Urol 1992;148:1549-57.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1279218&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1992JW46000058&link_type=ISI) 

13. 13. Bellamy N, Buchanan WW, Goldsmith CH, Campbell J, Stitt LW. Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee. J Rheumatol 1988;15:1833-40.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3068365&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1988R854800020&link_type=ISI) 

14. 14. Wright CJ, Robens-Paradise Y. *Regional evaluation of surgical indications and outcomes* [report of Health Transition Fund project BC401]. Vancouver: RESIO; 2001 [updated 2001 Jun 21]. Available: [www.resio.org](http://www.resio.org) (accessed 2002 May 16).
    
    

15. 15. Espallargues M, Alonso J. Effectiveness of cataract surgery in Barcelona, Spain. Site results of an international study. Barcelona I-PORT investigators. International Patient Outcomes Research Team. J Clin Epidemiol 1998;51:843-52.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0895-4356(98)00067-5&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9762877&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076147400006&link_type=ISI) 

16. 16. Hadorn D, Baker D, Dracup K, Pitt B. Making judgments about treatment effectiveness based on health outcomes: theoretical and practical issues. J Qual Improvement 1994;20:547-54.
    
    

17. 17. Weinberger M, Oddone EZ, Samsa GP, Landsman PB. Are health-related quality-of-life measures affected by the mode of administration? J Clin Epidemiol 1996;49:135-40.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0895-4356(95)00556-0&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8606314&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1996UC79900002&link_type=ISI) 

18. 18. Davies GM, Watson DJ, Bellamy N. Comparison of the responsiveness and relative effect size of the Western Ontario and McMaster Universities Osteoarthritis Index and the short-form Medical Outcomes Study Survey in a randomized, clinical trial of osteoarthritis patients. Arthritis Care Res 1999;12:172-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1002/1529-0131(199906)12:3<172::AID-ART4>3.0.CO;2-Y&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10513507&link_type=MED&atom=%2Fcmaj%2F167%2F5%2F461.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000080808300004&link_type=ISI) 

19. 19. Lamping DL, Browne J, Smith SC. Patient-based measures to evaluate surgical outcomes: myths and realities. *Ann R Coll Surg Engl* *(Suppl)* 2001;83:78-81.
    
    

20. 20. Addington-Hall J, Kalra L. Who should measure quality of life? BMJ 2001;322:1417-20.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjIvNzI5OS8xNDE3IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE2Ny81LzQ2MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=)