Effect of surgical safety checklists on pediatric surgical complications in Ontario
===================================================================================

* James D. O’Leary
* Duminda N. Wijeysundera
* Mark W. Crawford

## Abstract

**Background:** In health care, most preventable adverse events occur in the operating room. Surgical safety checklists have become a standard of care for safe operating room practice, but there is conflicting evidence for the effectiveness of checklists to improve perioperative outcomes in some populations. Our objective was to determine whether surgical safety checklists are associated with a reduction in the proportion of children who had perioperative complications.

**Methods:** We conducted a retrospective cohort study using administrative health care databases housed at the Institute for Clinical Evaluative Sciences to compare the risk of perioperative complications in children undergoing common types of surgery before and after the mandated implementation of surgical safety checklists in 116 acute care hospitals in Ontario. The primary outcome was a composite outcome of 30-day all-cause mortality and perioperative complications.

**Results:** We identified 14 458 and 14 314 surgical procedures in pre- and postchecklist groups, respectively. The proportion of children who had perioperative complications was 4.08% (95% confidence interval [CI] 3.76%–4.40%) before the implementation of the checklist and 4.12% (95% CI 3.80%–4.45%) after implementation. After we adjusted for confounding factors, we found no significant difference in the odds of perioperative complications after the introduction of surgical safety checklists (adjusted odds ratio 1.01, 95% CI 0.90–1.14, *p* = 0.9).

**Interpretation:** The implementation of surgical safety checklists for pediatric surgery in Ontario was not associated with a reduction in the proportion of children who had perioperative complications.

**Trial registration:** ClinicalTrials.gov, no. [NCT02419053](http://www.cmaj.ca/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT02419053&atom=%2Fcmaj%2F188%2F9%2FE191.atom)

In health care, most preventable adverse events occur in the operating room.1 Checklists aim to minimize preventable errors by providing a standardized framework for undertaking complex procedures.2 The World Health Organization (WHO) developed a surgical safety checklist to improve perioperative safety,3 which has been shown to reduce rates of perioperative mortality and complications in a range of health care settings.4 As a result, surgical safety checklists have been established as a standard of care for safe operating room practice,5 but there is conflicting evidence for the effectiveness of checklists to improve perioperative outcomes in some populations.6

Pediatric surgery differs from adult surgery, particularly in terms of models of health care delivery, perioperative risks, and surgical procedure types and volume. The infrastructure of health care services differs for pediatric and adult surgery, with many types of pediatric surgery undertaken only in secondary or tertiary specialist centres. The risk of perioperative death in children is low, with the exception of congenital cardiac and newborn surgery.7 However, the risk of perioperative adverse events is relatively high.8 These risks are influenced by the specialty of the primary health care provider, the age of the child and delivery of health care at an academic centre.8 In addition, some pediatric surgical procedures and hospitals are considered to be low volume, which can also contribute to differences in perioperative adverse events between children and adults.9 Because of these factors, findings from previous investigations of the effect of surgical safety checklists on perioperative outcomes in adults may not be generalizable to children undergoing surgery.

We hypothesized that the proportion of children admitted to hospital for surgery who had perioperative complications would decrease after the mandated implementation of surgical safety checklists in Ontario. The primary aim of this study was to evaluate the effect of surgical safety checklists on perioperative complications in children who undergo common types of pediatric surgery. The secondary aim was to determine if the use of surgical safety checklists was associated with a reduction in measures of health care utilization (i.e., unplanned return to the operating room, length of hospital stay and visits to the emergency department).

## Methods

We conducted a retrospective cohort study of patients more than 28 days and less than 18 years of age who were admitted to hospital for surgery in Ontario before and after the mandated implementation of surgical safety checklists. The use of a 3-phase surgical safety checklist in all hospital operating rooms in Ontario was mandated by the Ministry of Health and Long-Term Care in September 2009, and public compliance reporting was implemented in July 2010. The surgical safety checklist proposed by the Ontario government was adapted by the Canadian Patient Safety Institute from the WHO surgical safety checklist but was subject to modification by individual hospitals.

We identified surgical admissions, patient demographic characteristics, perioperative complications and measures of health care utilization from Ontario health administrative and demographic databases housed at the Institute for Clinical Evaluative Sciences (Toronto): the Discharge Abstract Database of the Canadian Institute for Health Information (CIHI) and the Registered Persons Database.

We received approval for this study protocol from the Research Ethics Board of The Hospital for Sick Children (Toronto).

### Study population

We included children more than 28 days of age who were admitted to hospital in Ontario to undergo a surgical procedure (primary reason) in the study cohort; neonates were excluded because most surgical procedures in this population are typically low volume and undertaken in tertiary hospitals. We used the following age categories: infants (29–364 d), young children (1–7 yr) and older children (8–17 yr). The age limit (8 yr) used to differentiate between young and older children was chosen to reflect the potential for meaningful participation by older children in the preoperative component of the surgical safety checklist.

### Surgical admissions

We identified hospital admissions with a surgical procedure as the primary reason for admission using case mix group category codes from the Discharge Abstract Database. Cardiac surgery, solid organ transplant, admissions without a surgical procedure performed and low-volume (< 10 procedures undertaken in Ontario per yr) admission categories were excluded from the cohort. We used the Canadian Classification of Health Interventions (CCI) codes for therapeutic interventions to define all surgical procedures.

### Study periods

The study time frame was the same 12-month period before (October 2008 to September 2009) and after (October 2010 to September 2011) surgical safety checklist compliance reporting began in July 2010. We chose these periods to minimize seasonal and temporal effects on perioperative complications and to account for the variable implementation of checklists across hospitals.

### Outcomes

The primary outcome was the composite of all-cause mortality and a priori specified complications within 30 days of surgery. Perioperative mortality and complication rates were determined for each surgical procedure. We used the International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10) to code complications, which included acute renal failure, cardiac arrest, complications of implants or grafts, decubitus ulcer, deep vein thrombosis, disruption of wound, electrolyte or acid–base abnormality, hemorrhage or hematoma, pulmonary embolism, pulmonary collapse or pneumonia, surgical site infection, sepsis, shock, stroke and vascular graft failure. We used the Registered Persons Database to determine 30-day postoperative all-cause mortality and the Discharge Abstract Database to determine all other complications.

Secondary outcomes were measures of health care utilization: length of hospital stay, any unplanned return to the operating room or emergency department visits within 30 days of surgery. We calculated length of hospital stay as the difference between the relevant admission and discharge dates, and we identified unplanned return to the operating room and emergency department visits from their respective CIHI codes in the Discharge Abstract Database.

### Statistical analysis

We determined descriptive statistics for all participants by study group, which are presented as appropriate for the data distribution. We calculated proportions and 95% confidence intervals (CIs) where appropriate. We used multivariable logistic regression models to estimate the adjusted association of surgical safety checklists (independent variable) with primary (composite outcome of perioperative complications) and secondary outcomes (proportion of children with a visit to the emergency department or an unplanned return to the operating room). We used a negative binomial regression model to estimate the adjusted effect of surgical safety checklists on length of stay. Covariates used in the regression models (i.e., admission category, age category, average neighbourhood income quintile, hospital type, sex and rurality) were specified a priori. A backward, stepwise approach was used for model building. We assessed logistic model goodness-of-fit with the Hosmer–Lemeshow test. Odds ratio (OR) estimates and 95% CIs were used to summarize the results. We defined statistical significance as 2-tailed *p* < 0.05. All analyses were performed using SAS version 9.4 (SAS Institute).

## Results

A total of 127 categories of eligible pediatric surgical admissions were identified from 116 Ontario hospitals (Appendix 1, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.151333/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.151333/-/DC1)). The most frequent surgical admission categories were acute appendicitis (14.6%), oral cavity or pharynx surgery (12.6%), orthopedic surgery of upper body or limb (8.8%), complicated appendectomy (4.5%), and orthopedic surgery of the tibia, fibula or knee (1.9%). From these admissions, we identified 14 458 and 14 314 surgical procedures in the same 12-month periods before and after the introduction of the checklist, respectively. Characteristics of children who underwent surgery are summarized in Table 1. Most children were male (59.4%), were admitted to hospital urgently or emergently (54.2%) and underwent intervention in a teaching hospital (56.9%).

View this table:
[Table 1:](http://www.cmaj.ca/content/188/9/E191/T1)

Table 1: 
Demographic characteristics of patients who underwent pediatric surgery in Ontario (*n* = 28 772)

### Univariable analyses

The proportion of children with complications after surgery did not differ based on demographic or admission characteristics (admission category, age category, average neighbourhood income quintile, hospital type and sex), apart from rurality (Table 2). The proportion of children from an urban area who had at least one complication was 3.94% compared with 5.12% of patients from a rural area (unadjusted OR 0.76, 95% CI 0.65–0.89, *p* < 0.001).

View this table:
[Table 2:](http://www.cmaj.ca/content/188/9/E191/T2)

Table 2: 
Analysis of rate of complications in patients who were admitted to hospital for pediatric surgery in Ontario, by potential confounding factors

### Perioperative complications

The proportion of children who had complications that occurred within 30 days of surgery was 4.08% (95% CI 3.76%–4.40%) in the prechecklist group and 4.12% (95% CI 3.80%– 4.45%) in the postchecklist group. After adjusting for confounding factors, we found that there was no significant difference in the odds of perioperative complications after the introduction of surgical safety checklists (adjusted OR 1.01, 95% CI 0.90–1.14, *p* = 0.9).

### Health care utilization

Data for length of stay were not normally distributed. Length of stay differed significantly between pre-and postchecklist groups (*p* < 0.001); however, the difference was small, and both groups had the same median unadjusted length of stay (1 d, interquartile range [IQR] 1–3 d) and the same frequency distributions up to the 95th quantile (10 v. 9 d for pre- and postchecklist groups, respectively). After adjusting for confounding factors, the difference in length of stay between groups remained significant (*p* < 0.001).

The proportion of children who presented to the emergency department within 30 days of surgery was 3.35% and 3.53% in the prechecklist and postchecklist groups, respectively (*p* = 0.4). There was no difference in the odds of an emergency department visit after the introduction of surgical safety checklists (adjusted OR 1.06, 95% CI 0.93–1.21, *p* = 0.4).

The proportion of children who had an unplanned return to the operating room did not differ between the prechecklist (0.27%) and postchecklist (0.24%) groups (*p* = 0.6). The adjusted OR of an unplanned return to the operating room after the introduction of surgical safety checklists was 0.88 (95% CI 0.55–1.39; *p* = 0.6).

### Subgroup analysis

Unadjusted rates and risks of specific complications are summarized in Table 3. The proportion of children with individual complications did not differ between pre- and postchecklist groups, with the exception of electrolyte or acid–base abnormalities, which decreased from 0.12% to 0.03%. The unadjusted OR of an electrolyte or acid–base abnormality in the postchecklist group was 0.28 (95% CI 0.10–0.76; *p* = 0.01).

View this table:
[Table 3:](http://www.cmaj.ca/content/188/9/E191/T3)

Table 3: 
Unadjusted risk of complications in patients who underwent pediatric surgery in Ontario (*n* = 28 772)

## Interpretation

Our study of 28 772 pediatric surgical procedures in 116 hospitals in Ontario found no difference in the proportion of children who had perioperative complications before and after the mandated implementation of surgical safety checklists. There were no clinically important differences between groups in measures of health care utilization.

Properly implemented checklists have been shown to be associated with significantly improved perioperative outcomes and patient safety in most health care settings.10 However, several critical factors are necessary to ensure that checklist implementation is successful11 (e.g., overcoming contextual barriers to adoption that are dependent on cultural and organizational factors).12 Data from the Ontario Ministry of Health and Long-Term Care showed that the mandated public reporting of surgical safety checklists did increase their use in pediatric academic hospitals, and checklists were used for most (> 98%) of the surgical procedures in the postimplementation group of this cohort.13 However, these data do not evaluate other factors that could influence the effectiveness of surgical safety checklists, such as the quality of checklist completion or engagement of operating room staff in the checklist process. The lack of positive findings in this study might reflect the extent of organizational leadership and patient safety infrastructure that are necessary for effective implementation of surgical safety checklists. Introduction of surgical safety checklists without adequate staff education or local leadership can lead to disinterest and, in some instances, abandonment by operating room staff.14

This study may not have detected improvements in operating room safety after adoption of surgical safety checklists because of the types of outcomes evaluated in this relatively low-risk population. Many quantitative research studies evaluating surgical safety checklists have used major adverse events (e.g., adverse outcomes defined by the American College of Surgeons National Surgical Quality Improvement Program) to evaluate the effect of checklists on patient safety. However, these outcomes can be uncommon in some surgical populations (i.e., ambulatory surgery and patients without significant comorbidities) and may not be suitable for detecting improvements in patient safety or quality of health care in populations with already low rates of perioperative complications and death. In this study, the reduced rate of electrolyte or acid–base abnormalities, an outcome not typically measured by studies evaluating the effect of surgical safety checklists, may be a spurious finding. However, it could also indicate that other trigger outcomes for adverse events are useful for detecting improvements in patient safety for low-risk surgeries or patients.8

### Comparison with other studies

When Haynes and colleagues first reported that the WHO surgical safety checklist was an effective tool for reducing perioperative mortality,4 some experts argued that the size of the observed improvements was misleading, given the predominance of health care settings with relatively high rates of perioperative death and complications in the cohort, and that similar results were unlikely to be achieved in populations with lower rates of adverse outcomes.15 Despite these concerns, the overall effect of surgical safety checklists on perioperative patient safety has been striking,16 and the body of evidence from quantitative and qualitative research supporting the use of surgical safety checklists continues to grow. A recent randomized controlled trial (RCT) involving adults undergoing surgery in Norway reported that use of the WHO surgical safety checklist was associated with a substantial reduction in perioperative complications (19.9% v. 11.5% before and after implementation, respectively) and mortality (1.6% v. 1.0% before and after implementation, respectively).17 This study was notable for both the study design (i.e., RCT) and setting (i.e., a developed health care service with low rates of perioperative death).

Another study by Urbach and colleagues using Ontario health administrative databases found no difference in perioperative mortality or complications after the introduction of surgical safety checklists in a predominantly adult population in Ontario.6 In this study, the adjusted risks of perioperative death and complications were low (0.71% and 3.86%, respectively) before the introduction of surgical safety checklists. 6 This is similar to our study in which the proportion of children who had perioperative complications before the introduction of surgical safety checklists was low (4.08%). Although our study used a short time interval between mandatory public reporting of surgical safety checklists in Ontario and measuring the checklist effect to minimize secular changes that may also occur, this time frame was greater than in other clinical studies that detected improved perioperative outcomes with the implementation of surgical safety checklists.4 It is possible that surgical safety checklists cannot effect further reductions in major adverse events in populations with already low rates of complications. A ceiling effect, if it does exist, may influence how we approach both the evaluation and implementation of surgical safety checklists in populations considered to be at low risk for major complications (e.g., ambulatory surgery).

### Strengths and limitations

Strengths of this study include the generalizability of the population-based cohort and the use of provincial health administrative and demographic databases. Exclusion of high-risk types of surgery undertaken in tertiary specialist pediatric hospitals (e.g., cardiac and neonatal surgery) and types of pediatric surgery that are infrequently undertaken from this cohort increased the external validity of our study. By using databases housed at the Institute for Clinical Evaluative Sciences, we were able to adjust for potential confounding by socioeconomic and geographical factors that were identified in Ontario demographic databases. In addition, the use of a retrospective study design could also have been advantageous by reducing the risk of a Hawthorne effect associated with some prospective study designs.18

Some caution must be used when interpreting the results of this study. The mandated implementation of surgical safety checklists by the provincial government in Ontario prevented the use of a study design with greater strength of evidence (i.e., a prospective cohort study or RCT) in this population. In addition, ambulatory surgery is underrepresented in this cohort; ambulatory care in Ontario is most often reported in the National Ambulatory Care Reporting System, which was not included in the current study owing to the use of a different case mix methodology. As a result, our findings may be best applied to children undergoing inpatient surgery in secondary or tertiary care settings. Furthermore, although CIHI health administrative databases undergo rigorous data quality controls to ensure accuracy, reliability and comparability over time,19 there is potential for inaccuracy in the calculated rates of perioperative complications because of coding and abstracting errors. However, there is no basis for such coding discrepancies to differ between the study periods.

### Unanswered questions and future research

The lack of positive findings in this study prompts several questions, including the following: What contextual factors influence the success of surgical safety checklists? What quantitative outcomes can be used to evaluate the effect of surgical safety checklists in populations with already low rates of adverse events and death? Future research will also be influenced by the increasing time interval since the introduction of surgical safety checklists and the absence of suitable control groups owing to the mandated use of surgical safety checklists in most populations. These factors could necessitate the use of other study designs for quantitative research in this field.20

### Conclusion

There is increasing evidence that surgical safety checklists can improve perioperative outcomes in many health care settings, but the mandated implementation of surgical safety checklists in Ontario was not associated with a reduction in the proportion of children who had perioperative complications.

## Acknowledgement

This study used deidentified data from the Institute for Clinical Evaluative Sciences (ICES) Data Repository, which is managed by the ICES with support from its funders and partners: Canada’s Strategy for Patient-Oriented Research (SPOR), the Ontario SPOR Support Unit, the Canadian Institutes of Health Research and the Government of Ontario.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** All of the authors conceived and designed the study. James O’Leary performed the statistical analysis and drafted the manuscript. All of the authors critically revised the manuscript for important intellectual content, approved the final version to be published and agreed to be guarantors of the work.

*   **Funding:** This study was supported by a Perioperative Services Innovation Award from The Hospital for Sick Children, Toronto. Duminda Wijeysundera is supported in part by a New Investigator Award from the Canadian Institutes of Health Research and a Merit Award from the Department of Anesthesia at the University of Toronto. The sponsors had no role in the design or conduct of the study; collection, management, analysis or interpretation of the data; preparation, review or approval of the manuscript; or the decision to submit the manuscript for publication.

*   Accepted January 22, 2016.

## References

1.  de Vries EN, Ramrattan MA, Smorenburg SM, et al. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 2008;17:216–23.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE3LzMvMjE2IjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqLzE4OC85L0UxOTEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  Winters BD, Gurses AP, Lehmann H, et al. Clinical review: checklists — translating evidence into practice. Crit Care 2009;13:210.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/cc7792&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20064195&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

3.  WHO guidelines for safe surgery 2009: safe surgery saves lives. Geneva: World Health Organization (WHO); 2009. Available: [http://apps.who.int/iris/bitstream/10665/44185](http://apps.who.int/iris/bitstream/10665/44185) (accessed 2016 Feb. 24).
    
    

4.  Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009;360:491–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa0810119&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19144931&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000262812400008&link_type=ISI) 

5.  Birkmeyer JD. Strategies for improving surgical quality – checklists and beyond. N Engl J Med 2010;363:1963–5.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMe1009542&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21067390&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000284047600015&link_type=ISI) 

6.  Urbach DR, Govindarajan A, Saskin R, et al. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med 2014; 370:1029–38.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa1308261&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24620866&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000332689100010&link_type=ISI) 

7.  van der Griend BF, Lister NA, McKenzie IM, et al. Postoperative mortality in children after 101,885 anesthetics at a tertiary pediatric hospital. Anesth Analg 2011;112:1440–7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1213/ANE.0b013e318213be52&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21543787&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

8.  Matlow AG, Baker GR, Flintoft V, et al. Adverse events among children in Canadian hospitals: the Canadian Paediatric Adverse Events Study. CMAJ 2012;184:E709–18.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTg0LzEzL0U3MDkiO3M6NDoiYXRvbSI7czoyMToiL2NtYWovMTg4LzkvRTE5MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

9.  Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med 2002; 346:1128–37.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMsa012337&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11948273&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000174880600006&link_type=ISI) 

10. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: a systematic review of impacts and implementation. BMJ Qual Saf 2014;23:299–318.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIzLzQvMjk5IjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqLzE4OC85L0UxOTEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

11. Russ SJ, Sevdalis N, Moorthy K, et al. A qualitative evaluation of the barriers and facilitators toward implementation of the WHO surgical safety checklist across hospitals in England: lessons from the “Surgical Checklist Implementation Project”. Ann Surg 2015;261:81–91.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/SLA.0000000000000793&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25072435&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

12. Fourcade A, Blache JL, Grenier C, et al. Barriers to staff adoption of a surgical safety checklist. BMJ Qual Saf 2012;21:191–7.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjIxLzMvMTkxIjtzOjQ6ImF0b20iO3M6MjE6Ii9jbWFqLzE4OC85L0UxOTEuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

13. Public reporting: patient safety. Toronto: Health Quality Ontario. Available: [www.hqontario.ca/public-reporting/patient-safety](http://www.hqontario.ca/public-reporting/patient-safety) (accessed 2015 July 27).
    
    

14. Conley DM, Singer SJ, Edmondson L, et al. Effective surgical safety checklist implementation. J Am Coll Surg 2011;212:873–9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jamcollsurg.2011.01.052&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21398154&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

15. Martin IC, Mason M, Findlay G. A surgical safety checklist. N Engl J Med 2009;360:2372–3; author reply 2374–5.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMc090417&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19474440&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

16. Borchard A, Schwappach DL, Barbir A, et al. A systematic review of the effectiveness, compliance, and critical factors for implementation of safety checklists in surgery. Ann Surg 2012; 256:925–33.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/SLA.0b013e3182682f27&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=22968074&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

17. Haugen AS, Softeland E, Almeland SK, et al. Effect of the World Health Organization checklist on patient outcomes: a stepped wedge cluster randomized controlled trial. Ann Surg 2015;261:821–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/SLA.0000000000000716&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24824415&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

18. McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol 2014; 67: 267–77.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.08.015&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24275499&link_type=MED&atom=%2Fcmaj%2F188%2F9%2FE191.atom) 

19. The CIHI Data Quality Framework. Ottawa: Canadian Institute for Health Information (CIHI); 2009. Available: [https://www.cihi.ca/en/data\_quality\_framework\_2009\_en.pdf](https://www.cihi.ca/en/data\_quality_framework_2009_en.pdf) (accessed 2015 July 27).
    
    

20. Ho PM, Peterson PN, Masoudi FA. Evaluating the evidence: Is there a rigid hierarchy? Circulation 2008;118:1675–84.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjExOiIxMTgvMTYvMTY3NSI7czo0OiJhdG9tIjtzOjIxOiIvY21hai8xODgvOS9FMTkxLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==)