Scientists from several areas of health research say Ottawa’s billion-dollar health research funding agency is hurting the integrity of Canadian science with ill-conceived reforms to its scientific peer reviews.
Among those voicing these charges are 10 leading scientists recruited by the Canadian Institutes of Health Research (CIHR) to serve as its Reforms Advisory Working Group. These included Dr. Jim Woodgett, director of research at Toronto’s Lunenfeld-Tanenbaum Research Institute at Mount Sinai Hospital, one of the world’s leading biomedical research facilities. “We could not have been clearer in warning CIHR’s science council not to do what it did,” says Woodgett, referring to a June 17, 2013 letter to CIHR’s Science Council that the group recently released to CMAJ. “CIHR almost entirely ignored our advice.”
Among a litany of flaws now described by Woodgett and other leading scientists, the most startling is that a set of 443 peer reviews launched in June 2014 using CIHR’s refashioned approach was riddled with potential conflicts of interest.
This problem had become evident, they say, at the conclusion of the initial peer-review process for its Foundation Scheme, in which 1366 scientists applied for long-term support from a $500-million CIHR funding stream.
In a Dec. 14, 2014 letter, seven researchers in Toronto, Vancouver and Montréal told CIHR that there was definitely a conflict-of-interest problem among its peer reviewers.
According to the letter, “at least two” signatories had submitted applications for Phase 1 of CIHR’s Foundation Scheme and then found themselves reviewing applications from the very same pool they were in. This is problematic, the seven scientists wrote, because “as reviewers, we could technically manipulate the ranking of the applicants we feel are most in competition with us, decreasing their overall chance to make it to the next phase.”
Rod Bremner, a researcher at the Lunenfeld-Tanenbaum Research Institute and one of the signatories on the December letter, was both a reviewer and an applicant to the fund. He summarizes the situation bluntly: “I submitted an application to the competition and I judged my competitors. It was a major flaw in the process.”
Bremner says he made a full disclosure of all potential conflicts of interest to CIHR, and that CIHR was made aware of the conflict problem almost immediately after the reviews were completed. The researchers also warned CIHR that by serving both as reviewers and applicants within the same pool, they could gain an unfair advantage in the next phase of competition because they have seen their competitors’ applications. “On most grant review panels, applicants are excluded from the review process to avoid potential conflicts of interest such as the ones we have described,” they wrote.
Despite all these concerns, CIHR has not yet taken any steps to exclude reviewers with potential conflicts of interest, acknowledged Jennifer O’Donoughue, CIHR’s executive director for Reforms Implementation in a Jan. 23 interview with CMAJ. “We rely on the reviewers to declare their conflicts. If they declared a conflict arising from a specific application they did not receive that application to review.”
If reviewers feel they may have had a conflict of interest, or if applicants are concerned their applications may have been reviewed by reviewers with a conflict of interest, O’Donoughue said, “they should call us and let us know.”
The underlying issue that led CIHR to ask grant applicants to review the work of other applicants with whom they were in competition can be traced back to the peer review reform strategy CIHR began implementing last June, says Woodgett.
Woodgett notes that in June 2013, he and the nine other senior scientists on the CIHR Reforms Advisory Working Group cautioned the agency that its peer review reform plans, including recruiting several thousand qualified reviewers was unrealistic and high-risk.
A chart released by CIHR on Dec. 10, 2014, depicts the results of the new Phase 1 Foundation Scheme review process and, according to Woodgett, reveals a highly unusual degree of reviewer variability that is firm evidence of a serious problem beyond the problem with potential conflicts of interest.
“It is telling that there was larger than expected variance in reviewer scores — as much as two and a half times higher variance than expected,” Woodgett explains. “This indicates that reviewers had a very hard time in parsing or ordering their reviews in a consistent manner. I don’t blame the reviewers; it was the process they had to follow that was a problem”
Philip Hieter, a geneticist at University of British Columbia in Vancouver who served with Woodgett on the working group, concurs: “I find this variability to be concerning.” CIHR’s data on the new Phase 1 review outcomes offer “no evidence the rank order generated is of equal quality to the old system.”
At the CIHR, O’Donoughue says CIHR stands by the integrity of its Foundation Scheme reviews, but is committed to investigating all reported conflicts of interest.