Effect of an interactive voice response system on oral anticoagulant management =============================================================================== * Natalie Oake * Carl van Walraven * Marc A. Rodger * Alan J. Forster ## Abstract **Background:** Monitoring oral anticoagulants is logistically challenging for both patients and medical staff. We evaluated the effect of adding an interactive voice response system to computerized decision support for oral anticoagulant management. **Methods:** We developed an interactive voice response system to communicate to patients the results of international normalized ratio testing and their dosage schedules for anticoagulation therapy. The system also reminded patients of upcoming and missed appointments for blood tests. We recruited patients whose anticoagulation control was stable after at least 3 months of warfarin therapy. We prospectively examined clinical data and outcomes for these patients for an intervention period of at least 3 months. We also collected retrospective data for each patient for the 3 months before study enrolment. **Results:** We recruited 226 patients between Nov. 23, 2006, and Aug. 1, 2007. The mean duration of the intervention period (prospective data collection) was 4.2 months. Anticoagulation control was similar for the periods during and preceding the intervention (mean time within the therapeutic range 80.3%, 95% confidence interval [CI] 77.5% to 83.1% v. 79.9%, 95% CI 77.3% to 82.6%). The interactive voice response system delivered 1211 (77.8%) of 1557 scheduled dosage messages, with no further input required from clinic staff. The most common reason for clinic staff having to deliver the remaining messages (accounting for 143 [9.2%] of all messages) was an international normalized ratio that was excessively high or low, (i.e., 0.5 or more outside the therapeutic range). When given the option, 76.6% of patients (164/214) chose to continue with the interactive voice response system for management of their anticoagulation after the study was completed. The system reduced staff workload for monitoring anticoagulation therapy by 48 min/wk, a 33% reduction from the baseline of 2.4 hours. **Interpretation:** Interactive voice response systems have a potential role in improving the monitoring of patients taking oral anticoagulants. Further work is required to determine the generalizability and cost-effectiveness of these results. Health care institutions are increasingly using interactive voice response systems. Applications of this technology include screening for disease (e.g., depression), 1 monitoring disease symptoms, 2 monitoring behaviour (e.g., substance abuse), 3 conducting behavioural counselling, 4 assessing adherence to medication regimens 5 and increasing attendance at appointments. In a recent systematic review, our group comprehensively described the populations, interventions and outcomes of clinical trials involving interactive voice response systems. 6 Most of these trials examined low-risk interventions and low-risk patient populations. None of the studies examined an intervention in which patients received instructions about their medication therapy without human intervention. Previous authors have reported a combined thromboembolic and hemorrhagic event rate of 15% per year and a fatal hemorrhagic event rate of 1% per year among patients taking oral anticoagulants. 7–9 Clinical event rates are typically minimal when patients’ international normalized ratio are within the therapeutic range, 10 but more than half of adverse events related to oral anticoagulation are directly attributable to international normalized ratio values outside the therapeutic range. 11 More than half of the adverse drug events that affect ambulatory patients occur because of inadequate monitoring, 12,13 making this aspect of care an attractive target for safety strategies. Several interventions, including the use of automated telephone messages 14 and academic detailing, 15 appear to improve physician and patient compliance with laboratory monitoring. Strategies to enhance monitoring could be improved by targeting patients who take high-risk medications, such as oral anticoagulants. 13,16 Many anticoagulation clinics already use computerized decision support systems, because these systems are effective in improving anticoagulation control. 17–20 However, even with the help of a computerized decision support system, monitoring patients is labour-intensive, because of the staff time required to call patients to communicate medication instructions. Additional staff time is needed to follow up with patients who forget to go for scheduled blood tests. Therefore, we developed an information technology-based solution to facilitate the monitoring of patients receiving oral anticoagulants. We were uncertain whether our system could relay complex instructions to patients in a manner that would be acceptable to the patients. In addition, the potential risk associated with using the system was high, since patients would be receiving dosage instructions without human intervention for a medication with a narrow therapeutic window. Therefore, we used a “proof-of-concept” approach to evaluate the feasibility of adding an interactive voice response system to a computerized decision support system under optimal conditions. ## Methods ### Study design and setting We conducted an observational study using a 1-group, pretest–posttest design. 21 Data collection was retrospective for the pre-intervention period and prospective for the intervention period. The pre-intervention period (retrospective) was the 3-month period before a patient’s enrolment in the study. The intervention period (prospective) was the 3-month period after study enrolment. We extended the prospective observation period for any patient who chose to continue with the intervention beyond 3 months. We conducted this study within the Oral Anticoagulation Management Service of The Ottawa Hospital Thrombosis Program in Ottawa, Canada. At the time of the study, this service was monitoring about 1200 patients from eastern Ontario who were receiving oral anticoagulation therapy. Program staff used a computerized decision support system (DawnAC, 4S Information Systems Ltd., Milnthorpe, England) to assist in monitoring patients. The study was approved by the Ottawa Hospital Research Ethics Board. ### Eligibility criteria Patients were potentially eligible for the study if they had completed 3 months of oral anticoagulation therapy with warfarin and their anticoagulation control was stable. We defined “stable control” as 2 consecutive international normalized ratio values within the therapeutic range during the month before recruitment. We used stability of control as an eligibility criterion because we wanted to evaluate the effectiveness of the interactive voice response system under optimal conditions. We did not exclude patients who had experienced previous hemorrhagic or thromboembolic events. We did exclude patients whose anticoagulation control was unstable and those who did not speak English, were receiving an oral anticoagulant other than warfarin, had hearing problems, received calls at a telephone number with an extension, self-managed their warfarin dosage or had plans to stop being monitored by the clinic. We recruited consecutive patients from the clinic who met the inclusion criteria during the recruitment period. ### Intervention We linked an interactive voice response system (Call-AssureCDM, Vocantas Inc., Ottawa, Canada) to a computerized decision support system and a telephone network to facilitate monitoring of oral anticoagulation therapy. Interactive voice response systems allow a computer database to communicate with people via the telephone, 22 for example, by automatically calling to deliver and retrieve information. In our case, health care professionals used the computerized decision support system to determine the new oral anticoagulant dosage and timing of the next international normalized ratio test. The interactive voice response system then communicated this information to patients automatically. The interactive voice response system communicated with patients using 3 types of messages: dosage, reminder and missed. The “dosage” message reported the patient’s latest international normalized ratio, the weekly dosage schedule and the date of the next appointment for testing of international normalized ratio. This message also asked the patient if he or she wanted to speak with someone from the clinic and if he or she had started any new medications. If the patient responded “yes” to either question, the interactive voice response system notified a health care professional via email to follow-up with the patient. The “reminder” message notified patients of upcoming appointments for international normalized ratio testing. We programmed the interactive voice response system to deliver this message 2 days before the patient’s appointment. The “missed” message notified patients who had missed an appointment for testing of international normalized ratio and asked them to go for testing the next day. The interactive voice response system documented the details (e.g., date and time) of all calls made in a report that could be accessed via a web-based interface. A health care professional reviewed this report daily and contacted any patients for whom delivery of a dosage message had been unsuccessful. We did not attempt to contact patients for whom delivery of a reminder or missed message was unsuccessful. ### Protocol for individual patients We recruited eligible patients by telephone using a standardized script. The intervention period for a particular patient started with his or her first international normalized ratio test after provision of consent. We followed each patient for a minimum of 3 months, during which time the patient had regular appointments for testing of international normalized ratio (Appendix 1, available at [www.cmaj.ca/cgi/content/full/180/9/927/DC2](http://www.cmaj.ca/cgi/content/full/180/9/927/DC2)). The laboratories forwarded international normalized ratio results to the clinic, as usual. The pharmacist manually entered each test result into the patient’s profile in the computerized decision support system. The support system then recommended a new oral dosage of anticoagulant and the date of the next international normalized ratio test. The pharmacist reviewed and approved or changed these recommendations. The interactive voice response system then called the patient to deliver a dosage message. If the system reached the patient, a caregiver or an answering machine, the message was delivered. Otherwise, the system disconnected and attempted to contact the patient later. We programmed the interactive voice response system to make up to 3 attempts to contact each patient. At the end of the study, we contacted patients and used a semistructured interview to elicit feedback about the interactive voice response system. We gave patients the option of continuing with the interactive voice response system or returning to the clinic’s standard monitoring system. We extended the study observation period for those patients who chose to continue with the interactive voice response system. ### Study outcomes The primary outcome of the study was anticoagulation control. 23 We collected patients’ international normalized ratio data from the clinic’s computerized decision support system for both the pre-intervention and intervention periods. We also evaluated the interactive voice response system using a health technology assessment framework that incorporated outcome, process and structure indicators. 24 Within this framework, we measured 2 outcome indicators: the rate of hemorrhagic and thromboembolic events during the 2 study periods, and patients’ satisfaction with the interactive voice response system, defined as the proportion of eligible patients who continued with the interactive voice response system after the study. We used the clinic’s computerized decision support system to identify hemorrhagic and thromboembolic events that occurred during the pre-intervention and intervention periods. We used semistructured interviews to assess patient satisfaction. Also within the health technology assessment framework, we measured 2 process indicators: the utility of the interactive voice response system, defined as the proportion of scheduled dosage messages that were successfully delivered by the interactive voice response system and that did not require further input from clinic staff, and the change in workload of the clinic staff. We calculated the utility of the interactive voice response system using data stored in the Web-accessible report generated by the system. For the second process indicator, we manually recorded and compared the time required to monitor the interactive voice response system and the time required to communicate with patients using the standard method. To generate the structure indicator, we described the setting and resources required to implement the interactive voice response system. ### Data analyses and sample size calculation We used 3 steps to determine the extent of anticoagulation control. First, we used linear interpolation 23 to calculate international normalized ratio values for the days between actual measurements. Second, for each patient, we calculated the proportion of days, for his or her total observation period, on which the international normalized ratio was within the therapeutic range. Third, we calculated the overall mean (i.e., the mean of individual patient proportions) and 95% confidence intervals (CIs). We applied this 3-step process to data for both the pre-intervention and intervention periods. We then performed a noninferiority test 25,26 to compare anticoagulation control for the 2 study periods. We selected a margin of non-inferiority of 5%, expressed as an absolute difference, based on the smallest minimal important difference 25 reported by published randomized controlled trials 18,27–30 in which anticoagulation control was the primary outcome. We also conducted subgroup analyses to investigate if anticoagulation control during the intervention period differed according to patients’ indication for oral anticoagulant use, sex, age, duration of oral anticoagulant use and satisfaction with the interactive voice response system. We included 2 regression models in our analyses. We used logistic regression 31 to investigate if patient factors were associated with whether a patient continued with the interactive voice response system. In addition, we used Poisson regression 32 to investigate whether the usefulness of the interactive voice response system was related to age. To estimate the time required to monitor patients using the interactive voice response system, we measured the time required to perform each monitoring task over a 1-week period. The monitoring tasks included identifying patients whose dosage messages were unsuccessful, contacting and receiving calls from patients, and relaying information about patients to the pharmacist. To estimate the time it would take staff to deliver the scheduled dosage messages, we determined the average time required to deliver 1 message, as follows. For 4 different periods in 1 week, the clerk at the clinic recorded the exact time required to successfully deliver information to 20 patients. Because we had a single study group, we calculated the sample size using the desired final CI, rather than a predetermined effect size. 33 We selected a desired 95% CI around the intervention estimate of anticoagulation control of 8%. We also used data from a population-based study 34 to select an expected standard deviation estimate of anticoagulation control in a population (30.1%). We determined that 226 patients would be required for our study. ## Results ### Patient characteristics and follow-up We recruited patients between Nov. 23, 2006, and Aug. 1, 2007. We approached a total of 355 patients (Figure 1). We excluded 41 patients because they did not meet the eligibility criteria at the start of the study. Of the 314 eligible patients, 88 patients declined to participate, 15 (17%) of these because they preferred not to participate in research. Therefore, we enrolled a total of 226 eligible patients in the study during the 9-month recruitment period. About half of the study participants were female (Table 1), and the median age was 58 (interquartile range 48–68, range 21–88) years. By far, the most common indication for warfarin was venous thromboembolism (179 patients [79.2%]). In total, 181 patients (80.0%) had been taking warfarin for longer than 1 year. View this table: [Table1](http://www.cmaj.ca/content/180/9/927/T1) **Table 1:** Characteristics of the study population ![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/180/9/927/F1.medium.gif) [Figure1](http://www.cmaj.ca/content/180/9/927/F1) **Figure 1:** Flow diagram of patient recruitment, follow-up and analysis. Note: IVRS = interactive voice response system. The overall intervention period began on May 7, 2007, and ended on Dec. 14, 2007. We followed patients prospectively for a total of 942.2 months (78.5 years), with a mean follow-up period of 4.2 months (standard deviation 1.8 months, range 1 day to 7.2 months) (Table 1). We collected 3 months’ worth of retrospective data for each patient, for a total of 56.5 years. A total of 193 patients (85.4%) completed the 3-month intervention period (Figure 1). Twenty-one patients (9.3%) withdrew from the study because they found the automated instructions confusing or too fast (*n* = 19) or they missed the personal contact with clinic staff (*n* = 2). We excluded an additional 12 patients (5.3%) because they discontinued warfarin (*n* = 8) or stopped being monitored by the clinic (*n* = 4). ### Outcome indicators The primary outcome indicator was anticoagulation control, expressed as proportion of time within the therapeutic range. Anticoagulation control during the intervention period was similar to that during the pre-intervention period. With the interactive voice response system, international normalized ratio values were within the therapeutic range a mean of 80.3% of the time (95% CI 77.5% to 83.1%). In the pre-intervention period, values were within the therapeutic range a mean of 79.9% of the time (95% CI 77.3% to 82.6%). The mean difference in anticoagulation control between the 2 periods was 0.36% (95% CI –2.95% to 3.67%). This difference was non-inferior because the 95% CI of the mean difference included zero and excluded the 5% margin of noninferiority. According to our subgroup analyses, anticoagulation control during the intervention period did not differ significantly by indication for oral anticoagulant use, sex, age, duration of oral anticoagulant use or patients’ satisfaction with the interactive voice response system. No hemorrhagic or thromboembolic events occurred during the study period. Most of the patients were satisfied with the interactive voice response system. A total of 164 patients continued with the interactive voice response system after the study (Table 2), representing 76.6% of the 214 patients who were eligible to do so. The most common reason for continuing with the interactive voice response system, cited by 86 patients, was its clear and timely delivery of information. Twenty-nine patients (13.6%) did not continue with the system after the study. The most common reasons for not continuing were missing the personal contact with staff (*n* = 12) and finding the automated instructions confusing or too fast (*n* = 12). Age was significantly associated with the decision to continue using the system. The likelihood of continuing use decreased with greater age (odds ratio 0.96, 95% CI 0.93 to 0.99). View this table: [Table2](http://www.cmaj.ca/content/180/9/927/T2) **Table 2:** Patients’ satisfaction with the interactive voice response system ### Process indicators The interactive voice response system was useful for communicating information to patients, as indicated by the outcomes for dosage messages (Table 3). During the intervention period, we needed to deliver a total of 1557 dosage messages. The interactive voice response system successfully delivered 1211 (77.8%) of these without further input from clinic staff. The remaining 346 messages (22.2%) required input from clinic staff, most often because the patient’s international normalized ratio was 0.5 or more outside the therapeutic range (*n* = 143). The pharmacist contacted these patients to identify possible explanations for the out-of-range values. Overall, the system was unable to deliver 155 (10.0%) of the dosage messages (Table 3). Our Poisson regression model revealed a trend toward decreasing effectiveness of the interactive voice response system with increasing age of the patients. View this table: [Table3](http://www.cmaj.ca/content/180/9/927/T3) **Table 3:** Overall utility of the interactive voice response system* Clinic staff spent a total of 1.6 hours per week monitoring the interactive voice response system. We found that it took 2 minutes and 50 seconds to deliver a single message in the absence of the interactive voice response system and estimated that it would have taken staff about 2.4 hours per week to deliver the scheduled messages. Therefore, we estimated that the system reduced overall staff workload by about 48 minutes per week (33%). ### Structure indicators We implemented the interactive voice response system in an anticoagulation clinic. Physicians, nurses and pharmacists work in the clinic and had experience using the computerized decision support system before our study began. The bulk of the work after implementation involved monitoring the system’s functioning. This required clerical training and basic computing skills. We periodically required technical support from the manufacturer to trouble-shoot problems. However, none of the problems necessitated turning off the interactive voice response system. ## Interpretation In this proof-of-concept study, we evaluated the addition of an interactive voice response system to a computerized decision support system to facilitate the management of oral anticoagulation therapy in patients whose anticoagulation control was already stable. Anticoagulation control was similar with and without the interactive voice response system. Most of the patients were satisfied with the system. The system was effective in communicating complex information, as indicated by the high rate of successful delivery of messages (77.8%) without input from staff. Importantly, the interactive voice response system reduced the workload of clinic staff by 33%. However, these results may have limited generalizability to unselected patients. Anticoagulation control in our patient population was exceptionally good at baseline. In a systematic review 35 of studies evaluating anticoagulation control, we found that the average time within the therapeutic range was 56.7% in community settings and 65.6% in anticoagulation clinics. In contrast, the patients in this study spent 79.9% of the time within the therapeutic range before the intervention, which made it very unlikely that the intervention would result in a significant improvement. Nonetheless, it was reassuring that anticoagulation control did not decrease during the intervention. In addition, the interactive voice response system required less work from staff members. The absence of any hemorrhagic or thromboembolic events in our cohort might be considered notable. However, we feel that this result was unsurprising, for 3 reasons. First, we followed a relatively small number of patients for a short period of time (a total of 135.0 patient-years of observation). Given the overall risk of adverse events of 15% per year, it is possible that very few, if any, events would have occurred in our cohort simply by chance. Second, patients were within the therapeutic range close to 80% of the time during the study. Because the risk of adverse events is minimized when the international normalized ratio is within therapeutic range, the expected risk of events was low. Third, the patients in our cohort were younger and healthier than patients evaluated in previous studies. As age and comorbidity are strong predictors of the risk of bleeding, our study might have been biased toward a low rate of adverse events. A previous evaluation of an interactive voice response system used in managing oral anticoagulation therapy 36 was limited because it did not measure anticoagulation control or structural indicators. Also, the system in the earlier study was not integrated with a computerized decision support system and was therefore more cumbersome to use. The interactive voice response system in our study was easy to use and could be implemented by large health management organizations. This decision support tool would be even more efficient if the computerized decision support system were integrated with a laboratory information system, so as to make manual data entry unnecessary. ### Strengths and limitations Our study had important strengths. First, we had perfect follow-up of our study population. All patients completed the post-study interview. In addition, once the study was complete, most chose to continue using the interactive voice response system rather than reverting to the standard monitoring system. Furthermore, the follow-up period for each patient was adequate to accurately assess anticoagulation control. Although the total observation time for the intervention period was 22 years longer than that for the pre-intervention period, we believe that the 2 periods were sufficiently long to justify our comparisons. Second, we selected our study population using clinically relevant, transparent eligibility criteria. These criteria yielded a clear inception cohort. In addition, a high proportion of the patients whom we approached agreed to participate, which indicates that the study population was highly representative of the clinic’s population. Third, our range of relevant, objective indicators provided a comprehensive assessment of the interactive voice response system. Our study also had limitations. First, the study population consisted of a highly selected group of users of oral anticoagulants. Most of the patients monitored by the anticoagulation clinic had stable anticoagulation control and were therefore not representative of most community-based patients. 35 Second, we were unable to determine the true utility of the interactive voice response system because our study lacked a concurrent control. The results of this pretest–posttest study suggested that care with the interactive voice response system may be noninferior. However, the limited statistical power of the study and the lack of a randomized design prevented us from conclusively demonstrating noninferiority. A randomized controlled trial would be required to obtain more robust evidence. ### Conclusions We have demonstrated that interactive voice response systems have a potential role in improving the monitoring of patients who are taking oral anticoagulants. Future randomized studies will be required to conclusively demonstrate the effectiveness of this technology. Such studies should also be based on a more representative sample of users of oral anticoagulants. Given our experiences, we also recommend that this type of decision support tool be considered for management of other high-risk medication therapies for which laboratory monitoring is required. See related commentary by Gurwitz, page [909](http://www.cmaj.ca/lookup/volpage/180/909?iss=9) ## Footnotes * This article has been peer reviewed. **Competing interests:** There are no conflicts of interest between any of the authors and the company that manufactured the interactive voice response system, Vocantas Inc. The data were collected and analyzed independent of Vocantas Inc. Natalie Oake had full access to all of the study data and takes responsibility for the integrity of the data and the accuracy of the analyses. Marc Rodger has received honoraria for speaking engagements and consulting from makers of oral anticoagulants, including Bayer, Pfizer and Boehringer Ingelheim. These funds were placed in the research trust funds of the Ottawa Health Research Institute. None declared for Natalie Oake, Carl van Walraven and Alan Forster. **Contributors:** All of the authors were involved in implementing the interactive voice response system at the study clinic. Natalie Oake recruited the study participants and collected the data. All of the authors were involved in the analyses and interpretation of the data, the writing and revision of the manuscript and gave approval of the final version for publication. **Acknowledgements:** We thank the staff of the Oral Anticoagulation Management Service, The Ottawa Hospital Thrombosis Program, especially Shemina Kherani, Lesley Yeung and Geoff Lewis. **Funding:** Marc Rodger is the recipient of a Career Investigator Award from the Heart and Stroke Foundation of Canada. Alan Forster holds a Career Scientist Award from the Ontario Ministry of Health and Long-Term Care and an Early Research Award from the Ontario Ministry of Research and Innovation. ## REFERENCES 1. 1. Kim H, Bracha Y, Tipnis A. Automated depression screening in disadvantaged pregnant women in an urban obstetric clinic. Arch Womens Ment Health 2007;10:163–9. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/s00737-007-0189-5&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17593320&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000249430900005&link_type=ISI) 2. 2. Shaw WS, Verma SK. Data equivalency of an interactive voice response system for home assessment of back pain and function. Pain Res Manag 2007;12:23–30. [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17372631&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) 3. 3. Gruenewald PJ, Searles J, Helzer J, et al. Exploring drinking dynamics using interactive voice response technology. J Stud Alcohol 2005;66:571–6. [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16240566&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) 4. 4. Friedman RH. Automated telephone conversations to assess health behavior and deliver behavioral interventions. J Med Syst 1998;22:95–102. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1023/A:1022695119046&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9571516&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) 5. 5. Stuart GW, Laraia MT, Ornstein SM, et al. An interactive voice response system to enhance antidepressant medication compliance. Top Health Inf Manage 2003;24: 15–20. [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12674391&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) 6. 6. Oake N, Jennings A, van Walraven C, et al. The use and effectiveness of interactive voice response systems for improving the delivery of ambulatory care: a systematic review and meta-analysis. *Am J Manag Care*. In press. 7. 7. Ansell J, Jacobson A, Levy J, et al. Guidelines for implementation of patient self-testing and patient self-management of oral anticoagulation. Int J Cardiol 2005;99:37–45. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.ijcard.2003.11.008&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15721497&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000227540600006&link_type=ISI) 8. 8. Gitter MJ, Jaeger TM, Petterson TM, et al. Bleeding and thromboembolism during anticoagulant therapy: a population-based study in Rochester, Minnesota. Mayo Clin Proc 1995;70:725–33. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.4065/70.8.725&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7630209&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995RN55100002&link_type=ISI) 9. 9. Steffensen FH, Kristensen K, Ejlersen E, et al. Major haemorrhagic complications during oral anticoagulant therapy in a Danish population-based cohort. J Intern Med 1997;242:497–503. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1365-2796.1997.tb00023.x&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9437411&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000071210800008&link_type=ISI) 10. 10. Oake N, Jennings A, Forster AJ, et al. Anticoagulation intensity and outcomes among patients prescribed oral anticoagulant therapy: a systematic review and meta-analysis. CMAJ 2008;179:235–44. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNzkvMy8yMzUiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTgwLzkvOTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 11. 11. Oake N, Fergusson DA, Forster AJ, et al. Frequency of adverse events in patients with poor anticoagulation: a meta-analysis. CMAJ 2007;176:1589–94. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTc2LzExLzE1ODkiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTgwLzkvOTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 12. 12. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA 2003;289:1107–16. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.289.9.1107&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12622580&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000181289400026&link_type=ISI) 13. 13. Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities. Am J Med 2005;118:251–8. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.amjmed.2004.09.018&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15745723&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000227653900011&link_type=ISI) 14. 14. Mundt JC. Interactive voice response systems in clinical research and treatment. Psychiatr Serv 1997;48:611–2. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1176/ps.48.5.611&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9144814&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WY16800004&link_type=ISI) 15. 15. Lafata JE, Gunter MJ, Hsu J, et al. Academic detailing to improve laboratory testing among outpatient medication users. Med Care 2007;45:966–72. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/MLR.0b013e3180546856&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17890994&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) 16. 16. Forster AJ, Murff HJ, Peterson JF, et al. Adverse drug events occurring following hospital discharge. J Gen Intern Med 2005;20:317–23. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1525-1497.2005.30390.x&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15857487&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000228836300001&link_type=ISI) 17. 17. Fitzmaurice DA, Hobbs FD, Murray ET, et al. Evaluation of computerized decision support for oral anticoagulation management based in primary care. Br J Gen Pract 1996;46:533–5. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNDYvNDEwLzUzMyI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xODAvOS85MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 18. 18. Fitzmaurice DA, Hobbs FD, Murray ET, et al. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med 2000;160:2343–8. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/archinte.160.15.2343&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10927732&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000088705800014&link_type=ISI) 19. 19. Poller L, Shiach CR, MacCallum PK, et al. Multicentre randomised study of com-puterised anticoagulant dosage. European Concerted Action on Anticoagulation. Lancet 1998;352:1505–9. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0140-6736(98)04147-6&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9820298&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076862300008&link_type=ISI) 20. 20. Manotti C, Moia M, Palareti G, et al. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multi-center trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment). Haematologica 2001;86:1060–70. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGFlbWF0b2wiO3M6NToicmVzaWQiO3M6MTA6Ijg2LzEwLzEwNjAiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTgwLzkvOTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 21. 21. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 2006;13:16–23. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiamFtaW5mbyI7czo1OiJyZXNpZCI7czo3OiIxMy8xLzE2IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE4MC85LzkyNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 22. 22. Lee H, Friedman ME, Cukor P, et al. Interactive voice response system (IVRS) in health care services. Nurs Outlook 2003;51:277–83. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0029-6554(03)00161-1&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=14688763&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000187582400007&link_type=ISI) 23. 23. Rosendaal FR, Cannegieter SC, van der Meer FJ, et al. A method to determine the optimal intensity of oral anticoagulant therapy. Thromb Haemost 1993;69:236–9. [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8470047&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1993KQ44300006&link_type=ISI) 24. 24. Steuten L, Vrijhoef B, Severens H, et al. Are we measuring what matters in health technology assessment of disease management? Systematic literature review. Int J Technol Assess Health Care 2006;22:47–57. [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16673680&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000235102700007&link_type=ISI) 25. 25. Jones B, Jarvis P, Lewis JA, et al. Trials to assess equivalence: the importance of rigorous methods. BMJ 1996;313:36–9. [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjExOiIzMTMvNzA0OC8zNiI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xODAvOS85MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 26. 26. Piaggio G, Elbourne DR, Altman DG, et al. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 2006;295:1152–60. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.295.10.1152&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16522836&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000235822500024&link_type=ISI) 27. 27. Fitzmaurice DA, Murray ET, McCahon D, et al. Self management of oral anticoagulation: randomised trial. BMJ 2005;331:1057. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMzEvNzUyNC8xMDU3IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE4MC85LzkyNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 28. 28. Chan FWH, Wong RSM, Lau WH, et al. Management of Chinese patients on warfarin therapy in two models of anticoagulation service — a prospective randomized trial. Br J Clin Pharmacol 2006;62:601–9. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1365-2125.2006.02693.x&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=17061966&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000241241000012&link_type=ISI) 29. 29. Menendez-Jandula B, Souto JC. Comparing self-management of oral anticoagulant therapy with clinic management: a randomized trial. Ann Intern Med 2005;142:1–10. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/0003-4819-142-1-200501040-00006&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=15630104&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000228366200001&link_type=ISI) 30. 30. Wilson SJ, Wells PS, Kovacs MJ, et al. Comparing the quality of oral anticoagulant management by anticoagulation clinics and by family physicians: a randomized controlled trial [published erratum appears in *CMAJ* 2004;170:451]. CMAJ 2003;169:293–8. [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNjkvNC8yOTMiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTgwLzkvOTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 31. 31. Kleinbaum DG, Gail M, editors. *Modeling strategy guidelines: logistic regression — a self learning text*. New York (NY): Springer-Verlag; 1994. 32. 32. Stokes ME, Davis CS, Koch GG. *Categorical data analysis using the SAS system*. 2nd ed. Cary (NC): SAS Institute Inc.; 2000. 33. 33. Lewis RJ. Power analysis and sample size determination: concepts and software tools. *Proceedings of the 2000 Annual Meeting of the Society for Academic Emergency Medicine*; 2000 May 22–25; San Francisco (CA). Available: [www.saem.org/download/lewis4.pdf](http://www.saem.org/download/lewis4.pdf) (accessed 2009 Mar. 30). 34. 34. van Walraven C, Austin PC, Oake N, et al. The effect of hospitalization on oral anticoagulation control: a population-based study. Thromb Res 2007;119:705–14. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.thromres.2006.05.017&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16844204&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000246226600007&link_type=ISI) 35. 35. van Walraven C, Jennings A, Oake N, et al. Effect of study setting on anticoagulation control: a systematic review and metaregression. Chest 2006;129:1155–66. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1378/chest.129.5.1155&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16685005&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000237618400010&link_type=ISI) 36. 36. Cervi PL, Everitt AS. Automatic voice mail for delivering computer-generated anticoagulant dose advice to patients. J Telemed Telecare 2002;8:259–63. [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1258/135763302760314216&link_type=DOI) [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12396853&link_type=MED&atom=%2Fcmaj%2F180%2F9%2F927.atom) [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000178538900003&link_type=ISI)