Abstract
Background Industry involvement has been associated with more favourable cost-effectiveness ratios in cost-effectiveness analyses, but the mechanisms for this association are unclear. We evaluated whether the assumed accuracy of the Papanicolaou (Pap) test was correlated with the features of cost-effectiveness analysis studies.
Methods We searched PubMed (last updated April 2010) for cost-effectiveness analysis studies in which at least one strategy involved the Pap test for cervical cancer. We assessed the baseline assumed diagnostic sensitivity and specificity of the Pap test in each study and the association of these values with three levels of manufacturer involvement in the study.
Results Among 88 analyzed cost-effectiveness analysis studies, the assumed sensitivity of the Pap test was lower in studies with manufacturer-affiliated authors, manufacturer funding or manufacturer-related competing interests versus studies without (mean sensitivity 60% v. 70%, p < 0.001). The assumed specificity of the Pap test was lower in cost-effectiveness analyses involving new screening tests (mean 93% v. 96%, p = 0.016). The assumed specificity did not differ between trials with manufacturer involvement versus those without (mean 95% v. 95%, p = 0.755).
Interpretation The results of cost-effectiveness analyses may be affected by a downgrading of the assumed diagnostic accuracy of the standard Pap test against which newer tests or interventions are compared. New technology then seems to have more favourable results against a straw-man comparator.
See related commentary by Bell and colleagues at www.cmaj.ca/cgi/doi/10.1503/cmaj.110384.
Given the substantial impact of cost-effectiveness analyses on public health policies, it is important to safeguard their robustness. However, most published cost-effectiveness analyses report favourable incremental cost-effectiveness ratios, and industry-funded analyses are more likely to report desirable ratios.1 Some journals discourage cost-effectiveness analysis studies that have been conducted or funded by sponsors of the examined interventions.2 Examples exist in which cost-effectiveness analysis studies on the same topic have reached different conclusions depending on who funded them.3 Yet the mechanism for such discrepancies in seemingly rigorous quantitative analyses is unclear.
The process of cost-effectiveness analysis involves making assumptions about the main parameters of interest that enter calculations. Baseline values and plausible ranges are specified. When baseline assumptions vary across different cost-effectiveness analyses, results may diverge considerably. We evaluated the baseline assumptions of the diagnostic accuracy of a widely used and studied screening test,4 the Papanicolaou (Pap) test (i.e., conventional cervical cytology) in cost-effectiveness analysis studies. The diagnostic accuracy of the Pap test has been studied extensively,5–7 and assumptions about its accuracy are pivotal for any cost-effectiveness analysis related to screening for and prevention of cervical cancer. Moreover, alternative approaches, such as DNA testing for human papilloma virus (HPV) or HPV vaccines, are quite expensive. New tests may get favourable cost-effectiveness ratios against the Pap test if the performance of the Pap test is assumed to be suboptimal.
We therefore examined whether cost-effectiveness analysis studies involving screening for and prevention of cervical cancer assumed different diagnostic performance estimates for the Pap test and whether differences reflected sponsor-related biases.
Methods
Search strategy
We searched for cost-effectiveness analysis studies that involved screening for and prevention of cervical cancer. Two independent investigators (N.P. and A.V.) searched MEDLINE through April 1, 2010, without restriction on year or language, using the following search string: (cervical OR cervix) AND (cancer OR carcinoma OR neoplasia OR neoplasm) AND (cost-effective OR cost-effectiveness OR cost-benefit OR cost analysis OR cost utility OR costs OR economic evaluation). In addition, we searched previous systematic reviews of related cost-effectiveness analyses and the references of eligible studies.
Study selection
Studies eligible for inclusion were cost-effectiveness analyses of strategies for screening and prevention of cervical cancer in which the Pap test was involved in one or more compared strategies. We included cost-effectiveness analysis studies regardless of whether they involved the comparison of Pap test strategies against other, newer screening strategies or vaccines or comparison of different Pap test strategies only (e.g., different frequency or timing of administration), given that the latter were relevant for providing estimates of the assumed diagnostic accuracy of the Pap test. We excluded reviews, commentaries, editorials, letters, studies of cost without formal cost-effectiveness analysis, and cost-effectiveness analyses involving treatments for cervical cancer.
Data extraction
Data were extracted by two independent investigators (N.P. and A.V.); discrepancies were discussed and consensus was reached by involvement of the two other investigators (D.M. and J.I.).
From each eligible study, we recorded the author, journal, year and country, the screening or vaccine strategies that were compared and which strategy was deemed preferable by authors. We considered novel strategies to be those involving any screening or vaccine technology other than conventional cytology.
We recorded the assumptions of Pap test sensitivity and specificity (baseline and range). We also recorded whether the article mentioned the thresholds of cytologic abnormality of the Pap test (i.e., atypical squamous cells of unknown significance [ASCUS], low-grade squamous intraepithelial lesions [LSIL] or high-grade squamous intraepithelial lesions [HSIL]). When different estimates of sensitivity and specificity were given for different types of lesions, we recorded them separately. When sensitivity and specificity estimates were provided for various types of lesions, we used the following order of preference: all lesions, HSIL, other. When only diagnostic measures other than sensitivity and specificity were given (e.g., predictive values or likelihood ratios), we calculated the corresponding sensitivity and specificity when feasible. When studies involved multiple countries with different specificity and sensitivity estimates per country, we calculated average values.
We also recorded whether the authors provided references to support their choice of these measures for Pap test performance. We captured the number of references and whether they included any meta-analyses. Finally, we documented whether any of the authors were affiliated with a test or intervention whose cost-effectiveness was assessed in the paper, whether the study was directly funded by a relevant manufacturer, and whether authors reported manufacturer-related competing interests. We also recorded funding from sources other than manufacturers. Data on funding status were abstracted after all sensitivity and specificity estimates had been abstracted.
Statistical analysis
We addressed the extent of differences in the assumed sensitivity and specificity of the Pap test across cost-effectiveness analysis studies according to various features of the studies. Our main analysis focused on the baseline assumptions. We performed additional analyses for the higher and lower values of the sensitivity analysis range. Analyses were performed for all types and grades of lesions, as well as separately for LSIL and HSIL.
We tested whether the values of diagnostic performance of conventional cytology that were adopted by the authors of each cost-effectiveness analysis were associated with involvement of the manufacturer of a compared technology (as defined by author affiliations, reported competing interests and funding sources) and involvement of other funding sources.
We used the t test or the Mann–Whitney U test for analysis of two groups, and analysis of variance or Kruskall–Wallis for analysis of more than two groups. The p values were two-tailed. Unweighted averages and standard errors of the assumed sensitivity and specificity were calculated for each group of cost-effectiveness analyses.
Results
Of 1704 cost-effectiveness analysis studies retrieved through PubMed, 159 were considered eligible for full-text evaluation. Of those, 53 were excluded (17 because they were review articles, 18 because they were cost studies without a formal cost-effectiveness analysis, and 18 because they were not relevant to screening or prevention of cervical cancer), leaving 106 eligible studies. An additional eight eligible studies were retrieved through reference searches. Of the total 114 cost-effectiveness analysis studies (listed in Appendix 1: Tables A–C, available at www.cmaj.ca/cgi/content/full/cmaj.101506/DC1), five could not be retrieved in full text. Of the remaining 109 studies, 88 provided assumptions for diagnostic performance of the Pap test that could be analyzed. The characteristics of these 109 studies, as well as those of the other 21, appear in Table 1 and Appendix 1: Table D (available at www.cmaj.ca/cgi/content/full/cmaj.101506/DC1).
Twenty-two (25%) of the cost-effectiveness analysis studies we analyzed were published in general medical journals. Fifty-two trials (59%) were published in the last five years, and most (60%) were conducted in the United States or Europe. The large majority (90%) included comparisons between novel strategies and conventional cytology, and 57 trials (65%) compared more than three strategies. Strategies other than the conventional Pap test included various vaccines (42 studies), HPV DNA test (39 studies), liquid-based cytology (21 studies), automated Pap rescreening (3 studies), visual inspection with acetic acid (7 studies) and speculoscopy (1 study). Among the 79 studies that involved comparisons of novel methods with conventional cytology, five (6%) found conventional cytology to be the most cost-effective strategy.
Assumptions of Pap test accuracy
Twenty-nine (33%) of the cost-effectiveness analysis studies we analyzed provided information about the assumed cytologic threshold of the Pap test; with one exception, this threshold was always ASCUS or higher. The assumed histologic grade was HSIL in 40% of studies and higher than LSIL in another 20%. Thirty-five studies provided estimates for sensitivity or specificity of the Pap test for several grades. Over 85% of the adopted sensitivity and specificity estimates were accompanied by a specific reference, but less than half cited a meta-analysis (Appendix 1: Table D, available at www.cmaj.ca/cgi/content/full/cmaj.101506/DC1).
The average assumed sensitivity of the conventional Pap test across the 88 cost-effectiveness analysis studies was 63%, and the average assumed specificity was 94%. The average lower–upper range of sensitivity estimates was 48%–83%, and the average lower–upper range of specificity estimates was 84%–97%.
Manufacturer involvement and funding
Manufacturer involvement as reflected by at least one manufacturer-affiliated author was evident in 17% of cost-effectiveness analysis studies, by provision of funding in 23% of studies, and by manufacturer-related competing interests declared by authors in 31% of studies. At least one of these three types of involvement was evident in 41 studies (Table 1). Funding from sources other than the manufacturer was reported in 50 of the studies, with predominantly public funding reported in 34 of the 50.
None of the studies that included at least one novel strategy and in which the manufacturer was involved (as defined by any of the three types of involvement) found the Pap test to be the preferred strategy.
Association between study features and assumed accuracy of the Pap test
The average assumed specificity of the Pap test was significantly lower in cost-effectiveness analysis studies that included only screening strategies (Pap test, liquid-based cytology, automated Pap rescreening, visual inspection with acetic acid or speculoscopy) than in those that also included HPV vaccines (93% v. 96%, p = 0.016). This difference remained the same when we excluded studies comparing only different Pap test strategies.
The average assumed sensitivity of the Pap test was significantly lower (by 9%) in studies that received funding from the manufacturer (p < 0.001) (Table 2). Studies with declared competing interests had a significantly lower (by 6%) assumption of Pap test sensitivity than studies in which no competing interests were declared, and studies lacking a section for the declaration of competing interests had the lowest average assumed sensitivity (54%).
Overall, studies in which there was manufacturer involvement at any level (i.e., through either authorship, funding or declared competing interest) had a 10% lower assumption of Pap test sensitivity than studies in which there was no manufacturer involvement (mean sensitivity 60% v. 70%, p < 0.001). This difference became more apparent when baseline assumptions about Pap test sensitivity and specificity were plotted as a receiver operating characteristic (ROC) curve (Figure 1). The studies with manufacturer involvement clustered at much lower sensitivity values, while high specificity was assumed in most studies regardless of manufacturer involvement. Finally, studies with other (nonmanufacturer) funding had significantly (7%) higher assumptions of Pap test sensitivity compared with studies that had no other reported funding (p = 0.009). These significant differences pertained to the baseline assumptions only, not the lower–upper range (Table 2).
The association with manufacturer involvement was apparent also in subgroups of cost-effectiveness analysis studies that involved vaccines versus studies involving only screening strategies (Appendix 1: Table E, available at www.cmaj.ca/cgi/content/full/cmaj.101506/DC1). The observed differences in assumptions were similar or greater when we examined baseline estimates according to the histologic grade of lesions at which Pap test accuracy referred to (LSIL and HSIL) (Appendix 1: Table F, available at www.cmaj.ca/cgi/content/full/cmaj.101506/DC1).
Interpretation
In our evaluation of 88 cost-effectiveness analysis studies involving strategies for screening or prevention of cervical cancer, we observed that the baseline assumed specificity of the Pap test was lower in studies involving comparison of the Pap test against other novel screening strategies. More prominently, manufacturer involvement was associated with lower baseline assumptions of the sensitivity of the Pap test for both vaccine-related and screening-related cost-effectiveness analysis studies.
The relation between industry sponsorship and research outcome has been evaluated primarily for drug trials. Sponsored trials are four times more likely to report results favouring the industry sponsor.8 Industry-sponsored studies are not of inferior quality versus nonsponsored studies.8 Their more favourable results are obtained mostly by comparing the industry-sponsored medication against inferior comparators (e.g., placebo or a “straw-man” medication with little or no effectiveness).8 Similarly, several studies suggest that company-sponsored cost-effectiveness analyses are more likely to report favourable qualitative conclusions9–13 and one systematic review concluded that such studies reported favourable incremental cost-effectiveness ratios1 compared with nonsponsored studies. For example, in an empirical evaluation of 495 cost-effectiveness analyses, industry sponsorship increased threefold the odds of an incremental cost-effectiveness ratio below $50 000 per quality-adjusted life year.14
Our study shows a mechanism through which sponsors may obtain more favourable results. Industry-sponsored cost-effectiveness analyses might assume the comparator standard test to have inferior performance. Therefore, the industry product would prevail against a straw man. In contrast to sponsored randomized trials, in cost-effectiveness analyses it is more difficult to select a comparator strategy that is inferior. For example, here it would be inappropriate for a cost-effectiveness analysis to compare a new cervical cancer screening test or vaccine against no screening at all. The cost-effectiveness analysis does consider the Pap test.
Unrealistic assumptions in modelling studies may have a substantial influence on results.15 Such assumptions may affect not only the performance of the standard strategy to which new tests and interventions are compared. One may also inflate the performance of the new tests and interventions or manipulate the cost estimates or other parameters in the modelling.16 These aspects are more difficult to tackle, because there is justifiably greater uncertainty about the performance and effectiveness of new tests and interventions, given that there are often too many tests and interventions, each evaluated in a handful of cost-effectiveness analyses. Cost is difficult to compare for cost-effectiveness analyses performed in very diverse countries, settings and years. Thus, the systematic evaluation of the impact of manufacturer involvement on each of these parameters is more difficult. Another empirical evaluation has shown that, for the same products, manufacturer-derived cost-effectiveness analyses assumed lower effectiveness for the comparators and different costs than independent, university-derived assessments.17–19
Limitations
Some limitations should be discussed. First, we focused on only one disease (cervical cancer) and only one test. However, the Pap test is a widely adopted test, and there is a large body of literature on its diagnostic performance. If anything, the Pap test should be harder to misrepresent than tests and interventions that have more ambiguous evidence. Even for the Pap test, meta-analyses do not agree completely on its diagnostic accuracy.5–7
Second, for some studies there was no mention at all of competing interests. These studies actually had the lowest assumptions of the sensitivity of the Pap test. Most had some involvement by the manufacturers (e.g., funding from or an author affiliated with the manufacturer); thus, nonreporting of competing interests may even be a marker of bias. Almost all journals (89%) currently have conflict-of-interest policies.20 However, given that the accuracy of disclosures may be questionable, especially when involving indirect payments,21 we suspect that manufacturer involvement in these cost-effectiveness analysis studies is underestimated based on the reported information. For example, we noticed that Diaz and colleagues22 reported no manufacturer involvement; however, an author of this paper is a paid speaker for HPV vaccines and has a company grant to study the vaccine, as stated in another article.23
It is very difficult to detect occult conflicts and even then, it is not certain that they would be relevant and pertinent at the time the study of interest is performed. If anything, nonreporting of manufacturer involvement and competing interests may result in underestimation of the impact that such involvement might have on assumptions of diagnostic performance. This lack of full transparency of information reinforces the need for meticulous reporting of all potential competing interests in the literature.24
Third, differences in diagnostic accuracy of the Pap test may have variable impact on the results of cost-effectiveness analysis studies. Few such studies reported numerical data on the exact impact of sensitivity analyses, and even these typically focused on the accuracy or performance of the new technology rather than the standard comparator (Pap test). One study showed that across the whole range of Pap test sensitivity, the estimated incremental cost-effectiveness ratio varied threefold.25 The impact may vary depending on other assumptions and model-building, and such information should be more clearly reported. Finally, some studies may have been unpublished, but these are unlikely to have had a major, direct impact on the scientific literature.
Conclusion
The baseline assumptions of cost-effectiveness analyses need to be scrutinized, but doing so is difficult for the average reader. Even experts don’t have time to review in detail all of the evidence supporting each assumption and all possible flaws.26 Not surprisingly, industry involvement in economic analyses may affect the derived conclusions27 without being stalled by the peer-review process.
Adherence to good standards for the conduct of cost-effectiveness analyses for medication at least appears to have improved over time for both industry-sponsored and nonsponsored studies.28 However, opportunities for bias are still numerous, given the discretionary nature of model-building and data selection in cost-effectiveness analysis studies. Such studies should ideally receive funding from public sources or private foundations29 without direct or hidden30–32 competing interests in the assessed products.
Footnotes
-
Competing interests: None declared.
-
This article has been peer reviewed.
-
Contributors: John Ioannidis conceived the study. Nikolaos Polyzos, Antonis Valachis and John Ioannidis performed the analyses. Nikolaos Polyzos, Antonis Valachis and Davide Mauri performed the literature searches. All of the authors were involved in the extraction of the data. Nikolaos Polyzos, Antonis Valachis and John Ioannidis wrote the manuscript. All of the authors critically revised the manuscript for important intellectual content and approved the final version submitted for publication.