Abstract
Background: Many clinical trials examine a composite outcome of admission to hospital and death, or infer a relationship between hospital admission and survival benefit. This assumes concordance of the outcomes “hospital admission” and “death.” However, whether the effects of a treatment on hospital admissions and readmissions correlate to its effect on serious outcomes such as death is unknown. We aimed to assess the correlation and concordance of effects of medical interventions on admission rates and mortality.
Methods: We searched the Cochrane Database of Systematic Reviews from its inception to January 2012 (issue 1, 2012) for systematic reviews of treatment comparisons that included meta-analyses for both admission and mortality outcomes. For each meta-analysis, we synthesized treatment effects on admissions and death, from respective randomized trials reporting those outcomes, using random-effects models. We then measured the concordance of directions of effect sizes and the correlation of summary estimates for the 2 outcomes.
Results: We identified 61 meta-analyses including 398 trials reporting mortality and 182 trials reporting admission rates; 125 trials reported both outcomes. In 27.9% of comparisons, the point estimates of treatment effects for the 2 outcomes were in opposite directions; in 8.2% of trials, the 95% confidence intervals did not overlap. We found no significant correlation between effect sizes for admission and death (Pearson r = 0.07, p = 0.6). Our results were similar when we limited our analysis to trials reporting both outcomes.
Interpretation: In this metaepidemiological study, admission and mortality outcomes did not correlate, and discordances occurred in about one-third of the treatment comparisons included in our analyses. Both outcomes convey useful information and should be reported separately, but extrapolating the benefits of admission to survival is unreliable and should be avoided.
Health care decisions often rely on effects of interventions described using rates of admission or readmission to hospital.1,2 These outcomes are typically regarded as indicators of insufficient quality of care and inefficient spending of health care resources;1,2 however, whether they can predict other serious clinical outcomes, such as death, is unknown.
Although effects on admission or readmission rates are often analyzed using large sets of routinely collected data, such as from administrative databases and electronic health records, many randomized controlled trials (RCTs) also collect data on admission rates, and some RCTs collect mortality data. Moreover, some trials combine death and admission to hospital as the primary composite outcome3 to increase the study’s power to detect significant differences and reduce the required study size.4 However, the interpretation of such a combination is difficult when the treatment effects on the 2 components are not concordant,5 for example, when more patients survive but rates of admission increase. In such cases, composite outcomes may dilute or obscure clinically significant treatment effects on important individual components,4,6 and incomplete disclosure of individual effects may mislead the interpretation of the results.4
We investigated systematic reviews of treatment comparisons that included meta-analyses of RCTs assessing effects on both rates of admission and mortality. We used the reported trial data to assess whether effects on admission rates were concordant with effects on mortality or whether it was possible to identify interventions and diseases in which these 2 outcomes would provide differing pictures of the merits of the tested interventions.
Methods
Data identification and eligibility
We searched the Cochrane Database of Systematic Reviews from its inception to January 2012 (issue 1, 2012) for systematic reviews of treatment comparisons that included meta-analyses of RCTs assessing rates of admission to hospital and meta-analyses of RCTs assessing mortality. We considered any comparison of interventions with drugs, biologics, vaccines or dietary supplements against other interventions, placebo or no treatment. Comparisons of different dosing schemes, routes of administration or timings of application were eligible for inclusion.
We searched the database for the following terms: “hospitalization,” “hospital stay,” “admission,” “readmission” and “mortality.” We performed our last search on Jan. 22, 2012 (Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.130430/-/DC1). Titles and abstracts of retrieved references were screened, and potentially eligible articles were reviewed in full text. We considered Cochrane reviews that included at least 1 meta-analysis on an admission outcome for further analysis. Eligible admission outcomes were stays in hospital for which participants were not admitted at randomization. In patients admitted to hospital, the pertinent outcome was readmission. Length of hospital stay and outcomes indirectly associated with admission, such as retransplantation, were not eligible for inclusion. For topics pertaining to maternal–fetal conditions, we analyzed the outcomes for the mother and fetus separately. For newborns, the pertinent outcome was admission to a neonatal intensive care unit (NICU), because newborns who require hospital care are admitted directly to the NICU. However, for older children and adults, admission specifically to intensive care units was considered beyond the scope of our project and therefore not an eligible outcome.
For each review, all pertinent treatment comparisons reporting an admission outcome were considered separately. For each treatment comparison with an eligible meta-analysis on admission outcomes, we evaluated whether an analysis of mortality was also reported for the same comparison. When both all-cause and cause-specific mortality data were available, we preferred using the former in our analyses.
For both outcomes, we excluded composite outcomes, meta-analyses with fewer than 5 events and analyses without a quantitative data synthesis (when there were 2 or more trials with events). When several subgroup analyses were reported and there was no overall quantitative data synthesis, we selected the subgroup analysis with the most events (or the most trials when this information was missing).
Finally, we excluded comparisons where the experimental intervention could not be clearly defined (based on the approval dates of the interventions [we defined older treatments as controls]).
Data extraction
Screening and data extraction were performed independently by 2 investigators, and discrepancies were solved by a third.
We extracted the following data for each eligible comparison from the Cochrane review: condition or disease category; interventions compared; category of admission outcome (admission, readmission); type of admission outcome (all cause, cause specific); mortality outcome (all cause, cause specific); comparator (active treatment v. placebo or no treatment); and population (adults v. children). From the individual RCTs included in the meta-analyses, we extracted the numbers of events and the numbers of patients in the intervention and control groups (2 × 2 table). Finally, we captured whether trials reporting on admission rates were double-blind, because the decision to admit to hospital may be subjective and might therefore be biased in unmasked trials. All of the data used for our analyses were included in the Cochrane reviews; we did not use publications of primary studies.
Statistical analysis
For each included treatment comparison, we used the reported RCT data (i.e., 2 × 2 tables) to calculate the summary treatment effect estimate (odds ratios [ORs]) on admission rates and mortality with a random-effects model.7 If there were no events in 1 or more study arms, we added 0.5 to all cells of the 2 × 2 table to allow for a meaningful calculation of the OR. We omitted comparisons for which only a single trial was available and for which all participants in both groups were admitted to hospital. In cases in which the reported trial-level data did not allow us to calculate the OR, we used the relative risks reported by the review authors with the assumption that they were approximations of the OR. We inverted our results so that an OR of less than 1 corresponded to a decreased risk with the experimental intervention compared with the control.
We examined whether the distribution of estimates differed for the 2 outcomes. We estimated the average intervention effect for an admission outcome and for a mortality outcome across all eligible comparisons using random-effects models. We described the between-comparison heterogeneity using the I2 metric and the between-comparison variance using τ2 and their respective 95% confidence intervals (CIs). These calculations combine the summary effects of comparison-specific meta-analyses on the same type of outcome into over-arching meta-analyses and provide an estimate of the average and spread of effect sizes typically seen for this outcome across diverse comparisons.
We recorded the frequency with which we saw the following: the point estimates of the admission and mortality outcomes were concordant or discordant; the effects for each outcome were nominally significant; an intervention significantly decreased the risk of admission while increasing the risk of death (and vice versa); and 95% CIs of effects did not overlap.
We calculated the correlation between the ORs of effects on rates of admission and mortality using Pearson’s correlation coefficient (r). In addition, we considered the uncertainty of the effect estimates by weighting using the inverse of the sum of variances of the effect estimates.
Our sensitivity and subgroup analyses included an analysis with only the results of trials in which rates of admission and mortality were both reported, a comparison of effects for different conditions or diseases, an analysis to investigate all-cause or cause-specific admission and death, an analysis of hospital admissions or readmissions, an analysis to compare effects for different study populations (e.g., adults, children, pregnant women and newborns), an analysis to investigate the use of active comparators, an analysis of admissions from double-blind trials only and an analysis using fixed-effect models.7
We performed all analyses using Stata 12.1. We report 2-tailed p values and 95% CIs.
Results
We identified 909 potentially eligible reviews (Figure 1), 55 of which met our inclusion criteria (Table 1). These reviews involved 61 treatment comparisons from 398 trials with mortality data and 182 trials with admissions data; 125 RCTs reported both outcomes. Of the 182 trials with admissions data, 115 were double-blind.
Across the comparisons with complete trial data for both outcomes (50/61 comparisons), meta-analyses included a median of 1651 patients (interquartile range [IQR] 764–5114) for mortality outcomes and a median of 971 patients (IQR 379–4387) for admission outcomes (Mann–Whitney U test p = 0.2). More single studies (median 4, IQR 3–9) and a lower median number of events (53, IQR 11–182) were included per meta-analysis on mortality compared with meta-analyses on admission, which included a median of 3 studies (IQR 1–4; p < 0.01) and 150 events (IQR 63–333; p < 0.01). The event rate was 6-fold higher for admission (median 17.62 per 100 patients, IQR 7.38–37.64) than for death (2.94, IQR 1.08–11.67) (p < 0.01). The median year of publication of the trials was 1999 (IQR 1992–2005).
Effect sizes for admissions and death
In the 61 comparisons, most of the mortality point estimates were between 0.4 and 2.5 (Figure 2); even for the 5 outliers, the 95% CIs reached well into this region.16,37–39,44 There was a somewhat wider spread in the point estimates for admission, with 8 comparisons (from 7 reviews) lying outside the 0.4 to 2.5 range;20,22,26,32,33,34,37 for 3 comparisons (from 2 reviews), the 95% CIs did not reach into this region.32,34 Overall, nominally significant effects were seen in 20 of 61 comparisons for admission and 12 of 61 comparisons for death (McNemar test p = 0.1). The average OR for admission was 0.82 (95% CI 0.74–0.91), with considerable between-comparison heterogeneity (I2 = 81% [95% CI 76%–84%], τ2 = 0.098 [95% CI 0.074–0.125]). The average OR for death was 0.87 (95% CI 0.82–0.93), with less heterogeneity (I2 = 42% [95% CI 18%–57%], τ2 = 0.014 (95% CI 0.004–0.025]).
Admission versus mortality effects
Concordant improvement of both outcomes was seen for 35 experimental treatments, and concordant harmful effects on both outcomes were seen for 9 experimental treatments. Concordant effects were more frequent than discordant effects (44 v. 17 comparisons; binomial test p < 0.001) (Table 2). Among the 17 comparisons with discordant results, admission rates were decreased and mortality was increased in 9 comparisons; the opposite occurred in 8 comparisons.
Three comparisons had concordant effects that were significant for both outcomes. One comparison had significant effects for both outcomes that were discordant: thiazide with spironolactone increased the risk for readmission to hospital for respiratory deterioration in infants born preterm with established or developing chronic lung disease (OR 4.38, 95% CI 1.21–15.81), but decreased in-hospital mortality (OR 0.16, 95% CI 0.03–0.81).37 There were 5 comparisons for which the 95% CIs of the effects on both outcomes did not overlap (Table 2).
We found similar results in analyses using a fixed-effects model (data not shown) and when we limited our analysis to trials for which both outcomes were reported within the same trial (Table 2).
Correlation of effect sizes
We found no significant correlation between outcomes across the 61 comparisons (Figure 3, Table 3, Pearson r = 0.07, p = 0.6). Using admission effects from double-blinded trials only or from trials reporting on both outcomes showed similar results. In 2 subgroup analyses of 15 comparisons in the context of childbirth and pregnancy and 44 comparisons investigating all-cause admission to hospital, we found a moderate correlation of borderline significance, which was not stable in our sensitivity analyses.
Sensitivity and subgroup analyses
None of the sensitivity and subgroup analyses showed significant correlations. However, some subgroups had very few comparisons, so inferences should be made cautiously (Table 3).
Interpretation
Our empirical evaluation suggests that admission and mortality outcomes in about one-third of cases yield different impressions about the merits of the experimental treatment.
Differences in mortality were typically small and rarely reached nominal significance. Conversely, differences in admission rates tended to have a more dynamic range, with wider possible values and considerable heterogeneity across different comparisons and diseases, and some large effects might also be seen.
Although the point estimates were more likely to be concordant than not for both outcomes, the effect sizes showed almost no correlation between the 2 outcomes. In 27.9% of cases, the point estimates were discordant (38.2% when only trials reporting both outcomes are included). Even when the statistical uncertainty was considered and the overlap of 95% CIs was analyzed, we found that effects on both outcomes significantly disagreed in about 10% of cases.
Our findings suggest that trial data on admission rates cannot be used to predict mortality, thus questioning the suitability of admission to hospital as a surrogate outcome for death. Moreover, this lack of good concordance in our analysis emphasizes that before mortality and admission rates are combined in composite outcomes, the assumption that the effects are concordant needs to be supported by sufficient evidence to have confidence in such an outcome.3,5
Situations may occur in which a positive correlation between admission rates and mortality cannot reasonably be expected. Sometimes, patients admitted to hospital are not at risk of dying, requiring hospital care to avoid complications that are not life-threatening. Moreover, some interventions may show their true benefits not by lowering admission rates, but by increasing them — for example, an intervention that allows the diagnosis of early symptoms or stabilizes a patient’s condition so that he or she may reach a hospital alive. In these cases, admissions would be less an indicator (or proxy) of life-threatening health problems than an indicator of good medical care. Thus, in such an example, decreasing admission rates could not be used as a measure of increasing quality of care.
In some diseases, death may be an uncommon event, so other outcomes (including admission to hospital) may attract primary attention. However, no matter how uncommon, deaths cannot be discarded. Owing to the wide perspective of our approach, we considered comparisons that may not be intuitively associated with clear-cut clinical effects on mortality; for example, vitamin supplementation for preventing miscarriage,55 mucolytics for chronic obstructive pulmonary disorder8 or treatments for schizophrenia.42 Even in such cases, death was not uncommon, with corresponding event rates in the control groups of included trials of 8.9%,55 3.4%8 and 2.4%.42 Comparisons with very low event rates (fewer than 5 events in an entire meta-analysis) were excluded, because treatment effect estimates from such analyses would be quite unstable and imprecise.
Admission rates can be useful in assessing medical interventions. In some cases, admissions may be a major aspect of the cost of managing a disease and may even be the most patient-relevant outcome. However, it is also probable that decisions to admit patients are subjective,63–67 and the threshold to admit may vary across populations, clinical centres, settings and health care systems.68–70 Conversely, mortality is unlikely to be equally affected by these factors. This is in line with the dynamic range of admission effect estimates we saw and their accompanying heterogeneity. The between-comparison variance for effects on admission was higher than that for effects on death, and much of the observed heterogeneity could be due to true differences in the underlying interventions, reflecting the clinical diversity of the included comparisons. Conversely, mortality effects were clustered more closely together and showed only moderate heterogeneity, even across diverse comparisons.
Comparison with other studies
An analysis of parallel group RCTs that had a composite primary outcome found that, in half of the trials, death was combined with hospital admission (or admission-requiring procedures), and data on the individual components of the composite outcome were often missing.3 Few of the authors of the included trials provided a clear rationale supporting the use of the composite outcome (3% of trials). The results of our analysis clearly emphasize the necessity of providing such rationale before combining these outcomes. Our results agree with those of a systematic review of 167 trials that had composite primary outcomes with mortality components.4 In 36% of those trials, results were significant for the composite outcome, but not for the mortality component (the opposite occurred in 4% of the trials). Moreover, in 13% of the trials, reporting for individual components of the composite outcome was inadequate.
Limitations
Even though we screened the entire Cochrane database, the accumulated data are still limited; the estimates of the effect sizes are uncertain for many meta-analyses. In addition, our analysis does not directly address the question of surrogacy — that is, whether admission to hospital can be a surrogate outcome for death. Such an analysis would require individual-level data and, ideally, could be done in trials that have already shown large effect sizes for the main outcome of interest (death).71,72 However, because admission to hospital is more common than death for most of the diseases studied in the trials we included in our analysis, admission to hospital would probably mislabel many patients’ illnesses as terminal if it were used as a strict surrogate of death.
We accepted that data from some primary studies were included in meta-analyses on different topics. In addition, of the 182 trials with admissions data, only 115 were double-blind, and the double-blinding may not have been totally successful in masking allocation. However, there was no strong evidence to suggest that concordance was better in these trials for admission outcomes than for mortality outcomes.
Conclusion
Effect sizes for admission rates and mortality do not appear to be correlated. Both outcomes may convey useful information, which should be reported separately. A clear rationale should be provided when these outcomes are combined as a composite outcome in clinical trials. When death is clearly the outcome of interest, randomized evidence should be properly powered to address differences in mortality rather than differences in admission rates alone. Extrapolating the benefits of admission to hospital to a reduced risk of death would be unreliable and should be avoided.
Footnotes
Competing interests: Lars Hemkens received support from The Commonwealth Fund (Harkness Fellowship in Health Care Policy and Practice) for the submitted work. No other competing interests were declared.
This article has been peer reviewed.
Contributors: All of the authors conceived the study, analyzed the data, interpreted the results, drafted the manuscript and approved the final version submitted for publication.
Funding: Supported by The Commonwealth Fund, a private independent foundation based in New York, New York. The views presented here are those of the authors and not necessarily those of The Commonwealth Fund, its directors, officers or staff. The funders had no role in the design or conduct of the study; the collection, management, analysis or interpretation of the data; or the preparation, review or approval of the manuscript.