Validation of quality indicators for end-of-life communication: results of a multicentre survey
===============================================================================================

* Daren K. Heyland
* Peter Dodek
* John J. You
* Tasnim Sinuff
* Tim Hiebert
* Carolyn Tayler
* Xuran Jiang
* Jessica Simon
* James Downar

## Abstract

**BACKGROUND:** The lack of validated quality indicators is a major barrier to improving end-of-life communication and decision-making. We sought to show the feasibility of and provide initial validation for a set of quality indicators related to end-of-life communication and decision-making.

**METHODS:** We administered a questionnaire to patients and their family members in 12 hospitals and asked them about advance care planning and goals-of-care discussions. Responses were used to calculate a quality indicator score. To validate this score, we determined its correlation with the concordance between the patients’ expressed wishes and the medical order for life-sustaining treatments recorded in the hospital chart. We compared the correlation with concordance for the advance care planning component score with that for the goal-of-care discussion scores.

**RESULTS:** We enrolled 297 patients and 209 family members. At all sites, both overall quality indicators and individual domain scores were low and there was wide variability around the point estimates. The highest-ranking institution had an overall quality indicator score (95% confidence interval) of 40% (36%–44%) and the lowest had a score of 18% (11%–25%). There was a strong correlation between the overall quality indicator score and the concordance measure (*r* = 0.72, *p* = 0.008); the estimated correlation between the advance care planning score and the concordance measure (*r* = 0.35) was weaker than that between the goal-of-care discussion scores and the concordance measure (*r* = 0.53).

**INTERPRETATION:** Quality of end-of-life communication and decision-making appears low overall, with considerable variability across hospitals. The proposed quality indicator measure shows feasibility and partial validity. **Study registration:** ClinicalTrials.gov, no. [NCT01362855](http://www.cmaj.ca/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT01362855&atom=%2Fcmaj%2F189%2F30%2FE980.atom)

Despite numerous calls to improve end-of-life care, and communication and decision-making in particular,1–4 over 2 decades there has been little progress in this arena.5,6 One of the major reasons for this limited progress may be the lack of quality indicators that are meaningful at a health system level to guide decision-makers as they introduce policies and practices to improve end-of-life care.7,8 We recently published a study that used a modified Delphi consensus process to develop a set of quality indicators related to advance care planning, goals-of-care discussions and documentation of these plans and goals (Table 1).9 Quality indicators should be evaluated based on their reliability, validity, responsiveness to change and ease of implementation.10 Because of the robust methodology used in developing these quality indicators,9 we believe they are clinically sensible and have face and content validity. Here we describe implementation of these quality indicators in a multicentre study to evaluate their feasibility and construct validity. A secondary objective was to establish a method for ranking institutions with respect to their performance in end-of-life–related communication and decision-making.

View this table:
[Table 1:](http://www.cmaj.ca/content/189/30/E980/T1)

Table 1: 
List of quality indicators

## Methods

This work was guided by a conceptual model of communication and decision-making in seriously ill and older patients who had been admitted to hospital, and the impact of these processes on patient-centred outcomes (Figure 1).9 To validate the aforementioned quality indicators, we conducted a multicentre, prospective audit of communication and decision-making related to the use or nonuse of life-sustaining treatments during serious illness, from the perspectives of patients and their family members; this was called the Audit of Communication, Care Planning, and Documentation (ACCEPT) Study.12 The association between our conceptual model and the ACCEPT audit is illustrated by the box in Figure 1 (downstream medical care actually received and satisfaction with hospital care were not evaluated).

![Figure 1:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/189/30/E980/F1.medium.gif)

[Figure 1:](http://www.cmaj.ca/content/189/30/E980/F1)

Figure 1: 
Conceptual model of improving end-of-life communication and decision-making, showing Audit of Communication, Care Planning, and Documentation (ACCEPT) Study quality indicators.9 In this model, we propose that advance care planning should ideally occur before hospital admission and inform goals-of-care discussions that occur within hospital. Ultimately, these plans and goals are translated into written documents or medical orders for the use or nonuse of life-sustaining treatments. For medical care at end of life to be consistent with patient values and preferences (the primary outcome), medical orders documented within the hospital records (surrogate outcome) must be concordant with these expressed preferences. Ultimately, the processes of communication and decision-making affect patient and family-centred outcomes, such as knowledge, stress, anxiety, other measures of psychological well-being and overall satisfaction. Given that advance care planning should occur before serious illness or admission to hospital, admission is an opportunity to assess satisfaction with previous advance care planning conversations. Similarly, discharge from hospital is an opportunity to measure patient satisfaction with goal-of-care discussion and decision-making.

The method for the ACCEPT audit has been described previously.1 In brief, inpatients from 12 teaching hospitals in Ontario, Manitoba, Alberta and British Columbia, Canada, were screened for eligibility (Appendix 1, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.160515/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.160515/-/DC1)). Patients were enrolled if they had advanced medical diseases (Appendix 2, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.160515/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.160515/-/DC1)) or if they were 80 years of age or older. If none of the above criteria were met, we also included any patient whose death within the next 6 months would not surprise any member of their care team.13

We approached consecutive, eligible patients and their family members for their written consent. Patients who were unable to communicate owing to language barriers (English or French only) or cognitive reasons were excluded, but if their family member was available and able to communicate, they were approached independently. Enrolled patients were asked to identify an adult family member who knew them best (inclusive of partners, significant others or close friends) and who had visited the patient in hospital at least once. We approached the patient 2 to 5 days after admission to allow for abatement of symptoms present at the time of admission and to allow for participation of both patient and family members.

Upon enrolment, the research assistant conducted separate face-to-face interviews with patients and family members using the previously validated ACCEPT questionnaire.14 In brief, we assessed respondents’ engagement in advance care planning before they were admitted to hospital and then asked questions about whether key steps related to goal-of-care discussion had occurred during the current hospital admission. These questions related to advance care planning and goal-of-care discussion were based on our previously developed quality indicators.9 To elicit preferences about use of life-sustaining treatments, we used the following taxonomy: respondents were asked to select the one that best represented their preferences: 1) Use machines and all possible measures including cardiopulmonary resuscitation (CPR) with a focus on keeping me (or my family member) alive at all costs; 2) Use machines and all possible measures with a focus on keeping me (or my family member) alive, but if my heart stops, no CPR; 3) Use machines only in the short term to see if I (or my family member) will get better, but if my illness is prolonged, change focus to comfort measures only. If the heart stops, no CPR; 4) Use full medical care to prolong my life (or life of my family member), but if my heart or breathing stops, no CPR or breathing machines; 5) Use comfort measures only, with a focus on improving quality of life and comfort. Allow natural death and no artificial prolongation of life and no CPR; or 6) Unsure.

We also assessed satisfaction with end-of-life care, including communication and decision-making, in the 4 weeks before hospital admission through the use of the validated Canadian Health Care Evaluation Project (CANHELP) Questionnaire.15 The CANHELP instrument is a questionnaire that quantifies patients’ and family caregivers’ satisfaction on a 5-point Likert scale.

After completion of the ACCEPT questionnaire, the research nurse reviewed the hospital record to document the medical orders for the use of life-sustaining treatments and items from the documentation domain of the quality indicator framework.9

### Statistical analysis

Demographic characteristics of patients and their family members are described as means, standard deviations and ranges for continuous variables, and as counts and percentages for categorical variables. Each quality indicator belonged to one of the advance care planning, goal-of-care discussion or documentation domains. Item scores are the proportion of patients (or family members) who affirmed that each quality indicator listed in Table 1 had occurred, multiplied by a weighting. Weights for each item were derived from the importance ratings from our previous work on the development of these quality indicators (Table 1).9 The overall quality indicator score, known as the ACCEPT quality indicator score, is the raw sum of all quality indicator items divided by the raw sum of highest possible values of these items (on a scale of 0% to 100%). We combined both patients’ and family members’ responses to develop average site overall and domain scores. We evaluated the internal consistency of the overall and domain scores using the Cronbach’s α.

To provide a measure of validity for this quality indicator scoring approach, a priori, we hypothesized that the overall ACCEPT quality indicator score would correlate with concordance between stated preferences and documented goals of care; concordance was calculated only for participants who had both of these items available. Patients or family members who had missing data or who were “unsure” of their preferences were excluded from these analyses. If the patient or family member preferred CPR and if there were no documented goals of care on the chart, the case was considered concordant because the default in all participating institutions was resuscitation. If patients or family members preferred not to be resuscitated and there was no documentation, such cases were considered to be discordant and consistent with a medical error. This concordance measure is one of the quality indicators in the documentation domain but was excluded from this domain, to stand alone as an outcome (Figure 1) for this validation exercise.

Because the goal-of-care discussion items were more likely to influence the concordance statistic in hospital, we postulated that the advance care planning domain score (a measure of prehospital activity) would correlate less strongly with concordance than the goal-of-care discussion domain scores. Moreover, because the CANHELP satisfaction measure was administered early in the hospital course, we postulated that the pertinent domains and items from the CANHELP questionnaire relevant to advance care planning would correlate with the advance care planning domain of the quality indicators. Associations between the whole or individual domains of the quality indicator score, the concordance measure, and the items and domains of the CANHELP scores were assessed by Pearson correlation coefficients.

### Ethics approval

The research ethics boards of all participating institutions approved the study.

## Results

We approached 580 eligible patients and 318 eligible family members; 297 patients and 209 family members consented, yielding an enrolment rate of 51% and 66%, respectively (Figure 2). The average age of patients was 80 years and 43% lived alone (Table 2). The average age of family members was 61 years; most were children of the patient (55%; Table 3). Most patients and family members were white and spoke either English or French. Hospital site characteristics are provided in Table 4.

![Figure 2:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/189/30/E980/F2.medium.gif)

[Figure 2:](http://www.cmaj.ca/content/189/30/E980/F2)

Figure 2: 
Patient flow diagram. Patients or family members who are excluded may fall into multiple categories of exclusion. ACCEPT = Audit of Communication, Care Planning, and Documentation study.

View this table:
[Table 2:](http://www.cmaj.ca/content/189/30/E980/T2)

Table 2: 
Patient characteristics

View this table:
[Table 3:](http://www.cmaj.ca/content/189/30/E980/T3)

Table 3: 
Family member characteristics

View this table:
[Table 4:](http://www.cmaj.ca/content/189/30/E980/T4)

Table 4: 
Site characteristics

Overall, all quality indicator scores were low, with wide confidence limits around these scores, and variability in performance among institutions (Table 5). Nevertheless, quality indicator scores for the highest-ranked hospital did not overlap with those from the lowest-ranked hospital; the highest-ranked hospital had an overall score (95% confidence interval [CI]) of 40% (36%–44%) and the lowest-ranked hospital had a score of 18% (11%–25%).

View this table:
[Table 5:](http://www.cmaj.ca/content/189/30/E980/T5)

Table 5: 
Scores and ranking of quality indicators by hospital*

Concordance at a hospital level ranged from 6% to 65%. Overall, there was a strong and statistically significant correlation between the overall ACCEPT quality indicator score and concordance (Table 6; *r* = 0.72, *p* = 0.008). As expected, the correlations between the advance care planning domain, and concordance and overall quality indicator score were weaker than the correlations between the goal-of-care discussion domain and concordance (Table 6). There were no statistically significant correlations between the advance care planning quality indicator domain score and different aspects of the CANHELP satisfaction questionnaire (overall score, domain scores or selected relevant item scores; Table 7). The Cronbach’s α for the overall score was 0.82, and for the advance care planning, goal-of-care discussion and documentation domains, scores were 0.64, 0.82 and 0.37, respectively. The low scores in the documentation domain were driven largely by the third and fifth items; when these items were excluded, the Cronbach’s α was 0.77 and the correlation with the overall quality indicator score remained relatively unchanged (*r* = 0.71, *p* = 0.01).

View this table:
[Table 6:](http://www.cmaj.ca/content/189/30/E980/T6)

Table 6: 
Associations between site ACCEPT quality indicator score and concordance measures (*n* = 12)*

View this table:
[Table 7:](http://www.cmaj.ca/content/189/30/E980/T7)

Table 7: 
Associations between site advance care planning domain score and site CANHELP scores (*n* = 12 sites)*

## Interpretation

In this multicentre audit of end-of-life communication and decision-making practices, we implemented a quality indicator measurement framework to show the feasibility of multisite measurements, evaluate its internal consistency, add further construct validation to the quality indicator scoring system, and discriminate among sites based on the quality of their end-of-life communication and decision-making. The purpose of these quality indicators and performance rankings is to improve quality of care by emphasizing best practices, ensuring accountability of providers and stimulating healthy competition.16

Importantly, our analysis was based on an a priori conceptual framework. In this framework, we positioned concordance between a patient’s expressed wishes and documented orders for life-sustaining treatments as the reference measure. We observed that our overall quality indicator score correlated strongly with this measure and, consistent with our hypothesis, the correlations between the goal-of-care discussion domain and the concordance measure were greater than the correlation between the advance care planning domain and the concordance measure. Although we observed significant variability in hospital performance, overall and domain quality indicator scores were low. Notwithstanding this fact, we were able to discriminate the highest-ranked hospitals from the lowest-ranked hospitals.

We and others have previously documented substantial quality concerns related to end-of-life communication and decision-making in patients in hospital.1–6 Measuring quality indicators related to these important processes may be a stimulus for improvement. However, few health care organizations routinely measure the quality of end-of-life care in general, and even fewer routinely evaluate the quality of communication and decision-making.8 Recently, some quality indicators were developed in the broad field of palliative/end-of-life care.17,19,20 Some of these include aspects of communication and decision-making, but these domains are not the focus of these measures. Accordingly, we developed9 and have now validated a novel set of quality indicators that are specifically related to end-of-life communication and decision-making.

Validation of quality indicators is done only infrequently.21 In our initial attempt to validate our quality indicators, we postulated that prehospital communication and decision-making (advance care planning) would have less impact on overall quality of communication and decision-making than in-hospital activities (goal-of-care discussion and documentation), as judged by the correlation of these domains with concordance between stated preferences and documented goals of care. Indeed, this is exactly what we found. However, we could not show any association between prehospital advance care planning activities and a measure of satisfaction (CANHELP questionnaire) used at admission to hospital (and meant to reflect the satisfaction with prehospital activities over the last 4 weeks). This weakens our attempt to provide some validation to the quality indicator measures. However, it may be that advance care planning activities conducted in the community do not influence satisfaction (as this is not the primary objective of advance care planning), or that satisfaction with advance care planning communication that had occurred before the 4 weeks prior to hospital admission was not captured by the time-limited framing of the CANHELP questionnaire.

Our findings do not mean that community-based advance care planning activities are not relevant for in-hospital goal-of-care discussion or documentation; they contribute by preparing patients for in-hospital communication and decision-making.12,18 But these complementary prehospital advance care planning activities are clearly not a substitute for goal-of-care discussion and documentation that should occur at the time of an admission to hospital. Furthermore, the lack of a strong association between advance care planning and our overall quality indicator score and concordance measure may, in part, be due to the disconnect between what happens in the community and what happens in the hospital. In health care systems where primary care is more directly related and integrated into acute care21 (unlike the current situation in Canada), there may be a stronger correlation between advance care planning and the overall quality indicator score and the concordance measure. To simplify the measurement process, some hospitals could omit the advance care planning domain from the measurement framework and focus their time and resources on the domains that they are better positioned to address (goal-of-care discussion). We also acknowledge that although our weights were empirically derived, they may not be reflective of the constituents served by other hospitals. In considering the adoption of this measurement framework to local hospitals, these weights could be either re-evaluated or omitted all together. Finally, we suggest that users omit the third and fifth items from the documentation domain to achieve results with higher internal consistency.

### Limitations

One limitation of this study is the small sample size. The limited number of participants from each site contributes to imprecision with respect to the estimates of site scores. In addition, all of the participating hospitals were teaching hospitals; therefore, our findings may not be generalizable to nonteaching hospitals. We also acknowledge that our initial validation work is limited to hospital-based outcomes. We did not evaluate the association between our quality indicators and long-term outcomes. Finally, we did not assess the reliability of these quality indicators; further studies need to include this type of psychometric evaluation.

### Conclusions

In conclusion, we have developed and validated quality indicators that can be used to measure the quality of end-of-life communication and decision-making. The results of this analysis suggest that it is feasible to implement these quality indicators in a variety of hospitals, that they are internally consistent and have construct validity, and that they can discriminate between highest-and lowest-ranked hospitals. Widespread adoption of this quality indicator framework could assist in process improvement for individual hospitals and improve the experience of patients who have serious illnesses, and that of their family members.

## Footnotes

*   [See related article at www.cmaj.ca/lookup/doi/10.1503/cmaj.170280](http://www.cmaj.ca/lookup/volpage/189/E978)

*   **Competing interests:** Jessica Simon reports grants from Alberta Innovates Health Solutions and personal fees from Alberta Health Services, during the conduct of the study. James Downar reports receiving fees for statistical analysis from Toronto General/Toronto Western Hospital Foundation and personal fees from Associated Medical Services Inc., during the conduct of the study. No other competing interests were declared.

*   This article has been peer reviewed.

*   **Contributors:** Xuran Jiang conducted the analysis. All authors participated in the acquisition of data, the interpretation of results and the drafting of the manuscript, and approved the final version to be published.

*   **Funding:** This study was supported by funding from the Canadian Institutes of Health Research, The Michael Smith Health Services Research Foundation in British Columbia, Alberta Innovates, and the Alternative Funding Plan Innovation Fund in Ontario. The funders had no role in the design, interpretation of results or decision to publish.

*   Accepted March 3, 2017.

## References

1.  Heyland DK, Barwich D, Pichora D, et al.ACCEPT (Advance Care Planning Evaluation in Elderly Patients) study team; Canadian Researchers at the End of Life Network (CARENET). Failure to engage hospitalized elderly patients and their families in advance care planning. JAMA Intern Med 2013;173:778–87.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jamainternmed.2013.180&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=23545563&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

2.  1.  Field MJ, 
    2.  Cassel CK
    
    , editors. Approaching death: improving care at the end of life. Washington: National Academy Press; 1997.
    
    

3.  Carstairs S, Beaudoin GA. Quality end of life care: the right of every Canadian. Parliament of Canada; 2000.
    
    

4.  Carstairs S. Still not there. Quality end-of-life care: a progress report. Ottawa: Canadian Hospice Palliative Care Association; 2005.
    
    

5.  The 2015 Quality of Death Index: ranking palliative care across the world. London/NewYork/Hong Kong: The Economist Intelligence Unit; 2015. Available: [www.eiuperspectives.economist.com/healthcare/2015-quality-death-index](http://www.eiuperspectives.economist.com/healthcare/2015-quality-death-index) (accessed 2016 Apr. 19).
    
    

6.  Teno JM, Freedman VA, Kasper JD, et al. Is care for the dying improving in the United States? J Palliat Med 2015;18:662–6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1089/jpm.2015.0039&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25922970&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

7.  Curtis JR, Engelberg RA. Measuring success of interventions to improve the quality of end-of-life care in the intensive care unit. Crit Care Med 2006;34(Suppl):S341–7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/01.CCM.0000237048.30032.29&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17057596&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000241737400008&link_type=ISI) 

8.  Weiss A, Downar J. Ontario hospitals are not using palliative care performance indicators in their balanced scorecards. J Pain Symptom Manage 2013;46:e1–5.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2013.06.014&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=23932177&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

9.  Sinuff T, Dodek P, You JJ, et al. Improving end-of-life communication and decision making: the development of a conceptual framework and quality indicators. J Pain Symptom Manage 2015;49:1070–80.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2014.12.007&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25623923&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

10. Stelfox HT, Straus SE. Measuring quality of care: considering conceptual approaches to quality indicator development and evaluation. J Clin Epidemiol 2013;66:1328–37.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

11. Howard M, Bernard C, Tan A, et al. Advance care planning: let’s start sooner. Can Fam Physician 2015;61:663–5.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjg6IjYxLzgvNjYzIjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4OS8zMC9FOTgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Canadian Researchers at the End of Life Network [homepage]. Kingston (ON). Available: [www.thecarenet.ca/784-researchers](http://www.thecarenet.ca/784-researchers) (accessed 2017 July 11).
    
    

13. Pattison M, Romer AL. Improving care through the end of life: launching a primary care clinic-based program. J Palliat Med 2001;4:249–54.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1089/109662101750290335&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11441634&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

14. Heyland DK, Pichora D, Dodek P, et al. The development and validation of a questionnaire to audit advance care planning. J Palliative Care Med 2012;2:119.
    
    

15. Heyland DK, Cook DJ, Rocker GM, et al.Canadian Researchers at the End of Life Network. The development and validation of a novel questionnaire to measure patient and family satisfaction with end-of-life care: the Canadian Health Care Evaluation Project (CANHELP) questionnaire. Palliat Med 2010;24:682–95.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1177/0269216310373168&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20605850&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

16. Adab P, Rouse AM, Mohammed MA, et al. Performance league tables: the NHS deserves better. BMJ 2002;324:95–8.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjExOiIzMjQvNzMyOS85NSI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xODkvMzAvRTk4MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

17. De Roo ML, Leemans K, Claessen SJ, et al.EURO IMPACT. Quality indicators for palliative care: update of a systematic review. J Pain Symptom Manage 2013; 46:556–72.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2012.09.013&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=23809769&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000325463300014&link_type=ISI) 

18. Sudore RL, Fried TR. Redefining the “planning” in advance care planning: preparing for end-of-life decision making. Ann Intern Med 2010;153:256–61.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/0003-4819-153-4-201008170-00008&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20713793&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000280973000006&link_type=ISI) 

19. Pasman HR, Brandt HE, Deliens L, et al. Quality indicators for palliative care: a systematic review. J Pain Symptom Manage 2009;38:145–56.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2008.07.008&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19615636&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000268780900020&link_type=ISI) 

20. Dy SM, Kiley KB, Ast K, et al. Measuring what matters: top-ranked quality indicators for hospice and palliative care from the American Academy of Hospice and Palliative Medicine and Hospice and Palliative Nurses Association. J Pain Symptom Manage 2015;49:773–81.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.jpainsymman.2015.01.012&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25697097&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom) 

21. Hammes BJ, Rooney BL, Gundrum JD. A comparative, retrospective, observational study of the prevalence, availability, and specificity of advance care plans in a county that implemented an advance care planning microsystem. J Am Geriatr Soc 2010;58:1249–55.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1532-5415.2010.02956.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20649688&link_type=MED&atom=%2Fcmaj%2F189%2F30%2FE980.atom)