Predicting performance on the Royal College of Physicians and Surgeons of Canada internal medicine written examination
======================================================================================================================

* Patrick Brill-Edwards
* Louis Couture
* Gerald Evans
* Peter Hamilton
* Irene Hramiak
* David Megran
* Mary Lou Schmuck
* Gary Cole
* Nadia Mikhael
* Geoff Norman

## Abstract

**Background:** Although the written component of the Royal College of Physicians and Surgeons of Canada (RCPSC) internal medicine examination is important for obtaining licensure and certification as a specialist, no methods exist to predict a candidate's performance on the examination.

**Method:** We obtained data from 5 Canadian universities from 1988 to 1998 in order to compare raw scores from the American Internal Medicine In-Training Examination (AIMI-TE) with raw scores and outcomes (pass or fail) of the written component of the RCPSC internal medicine examination.

**Results:** Mean scores on the AIMI-TE correlated well with scores on the RCPSC internal medicine written examination for all postgraduate years (*r* = 0.62, *r* = 0.55 and *r* = 0.65 for postgraduate years 1, 2 and 3 respectively). Scores above the 50th percentile on the AIMI-TE were predictive of a low failure rate (< 1.5%) on the RCPSC internal medicine written examination, whereas scores at or below the 10th percentile were associated with a high failure rate (about 24%).

**Interpretation:** Candidates who are eligible to take the written component of the RCPSC certification examination in internal medicine can use the AIMI-TE to predict their performance on the Canadian examination. The AIMI-TE is a useful test for residents in all levels of training, because the examination scores have a strong relation to expected performance on the Canadian examination for each year of postgraduate training.

The Royal College of Physicians and Surgeons of Canada (RCPSC) certification examinations have been referred to as “high stakes examinations,” because success or failure in these examinations has a major impact on a candidate's career. Since 1994, with very few exceptions, licensure in all provinces has been granted only after a candidate has obtained certification in either family medicine or a specialty.1 No method exists to predict the results of the RCPSC internal medicine written examination.

The American College of Physicians and the American Society of Internal Medicine offer a practice written examination, the In-Training Examination (AIMI-TE), that correlates with scores on the American Board of Internal Medicine (ABIM).2,3 Although 18 other US specialty societies and boards sponsor their own in-training examinations, it has been shown that only the results of tests in surgery and in family medicine can be used to predict future performance on a certifying examination.4,5 The AIMI-TE contains more multiple-choice questions than the RCPSC internal medicine written examination (360 v. 200), and these questions are drawn from a much larger pool than those in the Canadian examination. The AIMI-TE was designed to assess the knowledge base of residents in internal medicine (especially those in the second postgraduate year) preparing for the ABIM certification examination. In addition to demonstrating the correlation between AIMI-TE scores and the outcome of the ABIM examination, 2 recent studies of the AIMI-TE have demonstrated that there is a greater risk of failure on the ABIM certifying examination for candidates who have scores in the lower percentiles on the AIMI-TE and a greater likelihood of passing the examination for candidates who have scores in the higher percentiles.2,3

If results on the AIMI-TE predict outcome on the RCPSC internal medicine written examination reliably, then candidates will have a method for assessing their preparation strategies. To determine whether a relation exists, we compared raw scores from the AIMI-TE with raw scores and outcome (pass or fail) on the RCPSC internal medicine written examination.

## Methods

We approached members of the Canadian Association of Internal Medicine Program Directors for help with this study in 1998. Five universities (Université Laval, Quebec City, Que.; Queens University, Kingston, Ont.; the University of Alberta, Edmonton, Alta.; the University of Western Ontario, London, Ont.; and the University of Calgary, Calgary, Alta.) volunteered AIMI-TE data (raw scores, means and standard deviations) for examination results from 1988 to 1998. As some programs offer the AIMI-TE to all internal medicine trainees in all postgraduate years, results for each attempt at the AIMI-TE during all 3 postgraduate years (PGYs) were obtained. To maintain confidentiality, the results were collected by one of the authors who is not an Internal Medicine program director (G.N.). These results were forwarded to the RCPSC and compared with the raw scores of first attempts at the RCPSC internal medicine written examination. Final pairings of AIMI-TE scores and RCPSC internal medicine written examination scores were performed, and then information that could identify candidates was removed. The McMaster University/Hamilton Health Sciences Research Ethics Board concluded that this study was a quality improvement project and not a research study and, therefore, their approval was not required.

To examine the relation between the AIMI-TE scores and RCPSC internal medicine written examination scores, it was necessary to compute a *z* score to correct for varying examination difficulty in different years. This was accomplished using the means and standard deviations of scores on the AIMI-TE for each calendar year and each postgraduate year.

Fifty-two percent of the candidates attempted the AIMI-TE in more than one postgraduate year. In a separate analysis, the results from this subset of candidates were examined to determine whether an individual candidate's correlation between AIMI-TE and RCPSC scores changed significantly with successive attempts at the AIMI-TE.

## Results

The mean scores for the AIMI-TE rose consistently from PGY-1 to PGY-3 (63.3%, 69.3% and 74.6% for PGY-1, PGY-2 and PGY-3 respectively, *p* = < 0.001). There was a strong relation between the results of the AIMI-TE and the RCPSC internal medicine written examination. The Pearson correlations of the *z* scores for each postgraduate year (*p* = 0.001) are shown in Table 1.

View this table:
[Table1](http://www.cmaj.ca/content/165/10/1305/T1)

Table 1. 

Although it might be expected that the correlations would increase as the time interval between examinations became shorter, this did not occur in a consistent fashion. The correlation between the AIMI-TE and the RCPSC internal medicine written examination for all candidates in PGY-1 was similar to that for all candidates in PGY-3 (Table 1). This finding, which demonstrates that RCPSC performance can be predicted early in postgraduate training, suggests that there may be a role for a change in learning strategies for candidates who perform poorly in PGY-1.

Analysis of the results of the subset of individual candidates who attempted the AIMI-TE in more than one year of training demonstrates that individuals' scores remained similar in subsequent attempts (Pearson *r* = 0.74, *r* = 0.84 and *r* = 0.86 for PGY-1, PGY-2 and PGY-3 respectively, *p* < 0.0001). Therefore, assuming that there is no change in learning strategy, it is highly likely that an individual will achieve a score that is similar to his or her first result when making successive attempts at the AIMI-TE.

Finally, the probability of passing the RCPSC internal medicine written examination was directly related to the percentile rank on the AIMI-TE. These are shown for each postgraduate year in Table 2.

View this table:
[Table2](http://www.cmaj.ca/content/165/10/1305/T2)

Table 2. 

The findings were consistent across all years. Candidates in any postgraduate year whose AIMI-TE scores were at or below the 10th percentile had a high likelihood of failure (23%–25%) on the RCPSC internal medicine written examination. Conversely, candidates whose AIMI-TE scores were above the 10th percentile, regardless of postgraduate year, had a low likelihood of failure (2.2%–3.1%) on the RCPSC internal medicine written examination. This difference in failure rates according to AIMI-TE percentile scores corresponds to relative risks of failure of 9.2 (95% CI 2.43–34.1), 8.0 (95% CI 4.14–15.48) and 11.4 (95% CI 2.58–29.4) for candidates in the first, second and third postgraduate years whose scores are at or below the 10th percentile compared with candidates whose scores are above the 10th percentile.

## Interpretation

This study demonstrates that a strong relation exists between AIMI-TE scores and RCPSC internal medicine written examination scores. It is advantageous that the AIMI-TE is a predictive test, because it is affordable and available to all candidates in Canadian internal medicine programs. The cost of developing a predictive examination that is specific to the RCPSC internal medicine written examination is likely to be prohibitive.6 We agree with Grossman and colleagues3 that the AIMI-TE should be used as a tool to identify those candidates who are most likely to benefit from a change in learning strategies. We do not endorse the notion of using the AIMI-TE results to prevent candidates from attempting the RCPSC internal medicine written examination or for any other punitive purpose. As Waxman and colleagues point out,2 the organizations that sponsor the AIMI-TE have clearly stipulated that performance on the AIMI-TE should not be considered when making decisions about resident retention, promotion or subspecialty selection.

Until there is evidence that specific individual or program strategies can improve the results of candidates who appear to be likely to fail the RCPSC internal medicine written examination, it seems reasonable, first, to make all candidates aware of their performance and the implications thereof, second, to have candidates take the examination as early as during the first postgraduate year to allow for changes in learning strategy and, third, to mandate that performance on the AIMI-TE will not be used to assess suitability for the internal medicine program or eligibility for the RCPSC internal medicine written examination.

In future, the RCPSC written examination results will be combined with the results of the oral component of the examination to reach an overall pass or fail decision. However, it is certain that the results from the RCPSC written examination will continue to play a significant role in determining candidates' overall results. Thus, the results of the AIMI-TE will continue to be a useful form of feedback for RCPSC candidates during their training years.

## Acknowledgments

We thank the Canadian Association of Internal Medicine Program Directors for their helpful suggestions; Dr. Lisa Elden for writing assistance; and Dr. Herb Waxman for data relating to the AIMI-TE.

## Footnotes

*   *This article has been peer reviewed.* 
    
    *Competing interests:* None declared.

## References

1.  1. Ryten E, Thurber D, Buske L. The class of 1989 and post-MD training. CMAJ 1998;158(6):731-7. Available: [www.cma.ca/cmaj/vol-158/issue-6/0731.htm](http://www.cma.ca/cmaj/vol-158/issue-6/0731.htm)
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNTgvNi83MzEiO3M6NDoiYXRvbSI7czoyMjoiL2NtYWovMTY1LzEwLzEzMDUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

2.  2. Waxman H, Braunstein G, Dantzker D, Goldberg S, Lefrak S, Lichstein E, et al. Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med 1994;9(12):692-4.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/BF02599012&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7876953&link_type=MED&atom=%2Fcmaj%2F165%2F10%2F1305.atom) 

3.  3. Grossman RS, Fincher RM, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med 1992; 7 (1) :63-7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/BF02599105&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1548550&link_type=MED&atom=%2Fcmaj%2F165%2F10%2F1305.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1992HA09500012&link_type=ISI) 

4.  4. Shetler PL. Observations on the American Board of Surgery in-training examination, board results, and conference attendance. Am J Surg 1982; 144: 292-4.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0002-9610(82)90002-2&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7114363&link_type=MED&atom=%2Fcmaj%2F165%2F10%2F1305.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1982PG56800002&link_type=ISI) 

5.  5. Leigh TM, Johnson TP, Pisacano NJ. Predictive validity of the American Board of Family Practice In-Training Examination. Acad Med 1990;65:454-7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00001888-199007000-00009&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2242200&link_type=MED&atom=%2Fcmaj%2F165%2F10%2F1305.atom) 

6.  6. Cusimano MD, Cohen R, Tucker W, Murnaghan J, Kodama R, Reznick R. A comparative analysis of the costs of administration of an OSCE (objective structured clinical examination). Acad Med 1994;69(7):571-6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00001888-199407000-00014&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8018269&link_type=MED&atom=%2Fcmaj%2F165%2F10%2F1305.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994NX17800021&link_type=ISI)