Toward evidence-based policy
============================

* Tara Kiran

See also page [E375](http://www.cmaj.ca/lookup/volpage/188/E375) and [www.cmaj.ca/lookup/doi/10.1503/cmaj.150858](http://www.cmaj.ca/lookup/doi/10.1503/cmaj.150858) and page [1063](http://www.cmaj.ca/lookup/volpage/188/1063) and [www.cmaj.ca/lookup/doi/10.1503/cmaj.161084](http://www.cmaj.ca/lookup/doi/10.1503/cmaj.161084)

Many people have become enamoured with the idea that targeted payments to physicians can improve quality of care. Real-life experiences, however, have been almost universally disappointing. At first glance, a linked study by Lavergne and colleagues1 seems to be yet another example of a failed attempt at “pay for performance” in health care. But, on closer examination, the study’s findings are less an indictment of financial incentives and more an example of governments’ failing to take an evidence-based approach to policy-making.

Lavergne and colleagues undertook a rigorous post-hoc evaluation of the annual complex care management fee, one of several new primary care fees introduced in British Columbia over the last decade. British Columbia has maintained a traditional fee-for-service payment system for primary care physicians, and many of the new fee codes were intended to encourage different ways of working to meet the needs of patients with chronic conditions. The complex care management fee was set at $315 per patient per year and ultimately accounted for 3% of total billings for participating primary care physicians. Physicians could bill the fee if they completed a complex care plan for a patient with two or more eligible conditions, such as congestive heart failure and diabetes. The study’s authors found that introduction of the fee had no significant impact on the number of primary care contacts or continuity with the designated primary care physician, that there was no associated decrease in all-cause hospital admissions and that cost per patient rose.

Post-hoc evaluations of policy, however, are always tricky because the objectives of policy-makers are often unclear, the most relevant clinical data are usually unavailable and it is impossible to cleanly disentangle the effect of the policy from other factors influencing outcomes. Perhaps it is unsurprising then that this study leaves us with more questions than answers.

First, we do not know whether the new fee code changed physicians’ approach to management of complex chronic disease or whether some physicians just got paid more for what they were already doing. Lavergne and colleagues found that about two-thirds of eligible patients had the fee code billed during the study period. But, these patients were more likely to have higher continuity, fewer hospital admissions and lower costs than patients who did not have a physician bill the incentive, and that was true even *before* billing of the incentive. These patients may have been fortunate to have primary care physicians who engaged in complex care planning before introduction of the incentive. Observational evaluations from other jurisdictions have found that pay for performance rewards physicians with higher baseline performance2,3 and that payment reforms can favour physicians who treat healthier patients.4

Second, we do not know how physicians who billed the fee code implemented the recommendations on care plans. There are parallels with efforts around medication reconciliation. Despite imperfect evidence, medication reconciliation has been included in accreditation standards in the United States and Canada. Critics hypothesize that whereas some organizations invested in the resources required for successful implementation, others have simply improved *documentation* of medication reconciliation.5 Similarly, it is unclear whether physicians who billed the fee code embedded care plans into day-to-day management or whether the focus was on improved documentation, possibly for activities they felt they were already doing.

Third, it is uncertain whether even ideal implementation of care plans would reduce hospital visits. Comprehensive care plans seem like common sense and have been encouraged by government agencies in other jurisdictions.6,7 But despite the propagation of care plans, there is little evidence linking them to better outcomes. There is evidence supporting care coordination for patients with more than one chronic condition, but successful interventions involve clear roles for other health professionals, leverage health information technology and have largely been implemented in high-performing primary care practices.8,9

Finally, it is unclear what the BC government hoped to achieve with introduction of the fee code. The authors of the linked study hypothesized that the incentives were primarily intended to reduce health care use and costs. They are likely right. But perhaps the government was aiming for an improved care experience for patients and families, or greater patient self-confidence in disease management. Maybe they were trying to get physicians to accept more patients with complex illness, or address relative pay differences between family physicians and specialists. If the latter was the goal, then the policy was likely a success.

Governments should be explicit about their goals for reform and incorporate evidence into policy-making. This type of approach can be challenging. Evidence to guide change is not always available, rigorous evaluation can be difficult and time-consuming, and clearly defined goals increase the risk of overt failure.

At the time BC introduced the new fee code, policy experts touted the potential benefits of pay for performance, and the United Kingdom had just introduced one of the most ambitious primary care pay-for-performance schemes in the world. But, there was little evidence supporting the approach. Even now, the most recent systematic review on the topic found there was little evidence to support or not support financial incentives to improve quality in primary care.10 Notably, the systematic review found only seven high-quality studies on the topic — despite widespread attempts at physician-payment reform worldwide.

Lack of evidence makes it all the more important for governments to clarify goals and prospectively evaluate payment reform. Physicians and the public would need to know that the intervention might not result in the desired outcome. Ideally there would be a shared understanding at the outset to halt an intervention that did not meet prespecified objectives. A retrospective analysis from the UK found that withdrawing incentives did not reduce quality of care.11

British Columbia spent $240 million dollars on the Complex Care Initiative over six years. Could this money have been spent on evidence-based interventions that are currently underprovided, such as psychotherapy for patients with mental illness? Other Canadian provinces have also spent large sums on physician-payment reform and have come under criticism for not evaluating the impact of changes.12 Payment reform can be challenging because it is negotiated between governments and medical associations. But, in an era of increasing accountability and fiscal restraint, both parties should be transparent about the health system goals underpinning reforms, how reforms will be evaluated and what steps would be taken if goals are not achieved.

Key points
*   Post-hoc evaluations of policy are tricky because the objectives are often unclear, the most relevant clinical data are usually unavailable and it is impossible to disentangle the effects of the policy from other factors influencing outcomes.

*   There is little high-quality evidence to support or not support financial incentives to improve quality in primary care, despite widespread attempts at physician-payment reform worldwide.

*   Governments should prospectively evaluate payment reforms and commit to change if objectives are not met.

## Footnotes

*   **Competing interests:** None declared.

*   This article was solicited and has not been peer reviewed.

## References

1.  Lavergne R, Law M, Peterson S, et al. A population-based analysis of incentive payments to primary care physicians for the care of patients with complex conditions. CMAJ 2016;188:E375–83.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTg4LzE1L0UzNzUiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg4LzE1LzEwNjUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  Kiran T, Victor JC, Kopp A, et al. The relationship between financial incentives and quality of diabetes care in Ontario, Canada. Diabetes Care 2012;35:1038–46.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZGlhY2FyZSI7czo1OiJyZXNpZCI7czo5OiIzNS81LzEwMzgiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg4LzE1LzEwNjUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Rosenthal MB, Frank RG, Li Z, et al. Early experience with pay-for-performance: from concept to practice. JAMA 2005; 294: 1788–93.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.294.14.1788&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16219882&link_type=MED&atom=%2Fcmaj%2F188%2F15%2F1065.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000232472000018&link_type=ISI) 

4.  Glazier RH, Klein-Geltink J, Kopp A, et al. Capitation and enhanced fee-for-service models for primary care reform: a population-based evaluation. CMAJ 2009;180:E72.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTgwLzExL0U3MiI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xODgvMTUvMTA2NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

5.  Pevnick JM, Shane R, Schnipper JL. The problem with medication reconciliation. BMJ Qual Saf 2016;25:726–30.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI1LzkvNzI2IjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4OC8xNS8xMDY1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

6.  Edwards ST, Landon BE. Medicare’s chronic care management payment — payment reform for primary care. N Engl J Med 2014;371:2049–51.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMp1410790&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25427110&link_type=MED&atom=%2Fcmaj%2F188%2F15%2F1065.atom) 

7.  Transforming Ontario’s health care system: Community Health Links provide coordinated, efficient and effective care to patients with complex needs. Toronto: Ontario Ministry of Health and Long-Term Care; modified 2016. Available: [www.health.gov.on.ca/en/pro/programs/transformation/community.aspx](http://www.health.gov.on.ca/en/pro/programs/transformation/community.aspx) (accessed 2016 June 14).
    
    

8.  Bodenheimer T, Ghorob A, Willard-Grace R, et al. The 10 building blocks of high-performing primary care. Ann Fam Med 2014;12:166–71.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTIvMi8xNjYiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTg4LzE1LzEwNjUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

9.  Katon WJ, Lin EH, Von Korff M, et al. Collaborative care for patients with depression and chronic illnesses. N Engl J Med 2010;363:2611–20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJMoa1003955&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21190455&link_type=MED&atom=%2Fcmaj%2F188%2F15%2F1065.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000285763700006&link_type=ISI) 

10. Scott A, Sivey P, Ait Ouakrim D, et al. The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev 2011;(9): CD008451.
    
    

11. Kontopantelis E, Springate D, Reeves D, et al. Withdrawing performance indicators: retrospective analysis of general practice performance under UK Quality and Outcomes Framework. BMJ 2014;348:g330.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzNDgvamFuMjdfNS9nMzMwIjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE4OC8xNS8xMDY1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Chapter 3.06: Funding alternatives for family physicians. In: Annual report of the Office of the Auditor General of Ontario. Toronto: Ministry of Health and Long-Term Care; 2011:150–70.