Abstract
BACKGROUND: In the prehospital setting, differentiating patients who have sepsis from those who have infection but no organ dysfunction is important to initiate sepsis treatments appropriately. We aimed to identify which published screening strategies for paramedics to use in identifying patients with sepsis provide the most certainty for prehospital diagnosis.
METHODS: We identified published strategies for screening by paramedics through a literature search. We then conducted a validation study in Alberta, Canada, from April 2015 to March 2016. For adult patients (≥ 18 yr) who were transferred by ambulance, we linked records to an administrative database and then restricted the search to patients with infection diagnosed in the emergency department. For each patient, the classification from each strategy was determined and compared with the diagnosis recorded in the emergency department. For all strategies that generated numeric scores, we constructed diagnostic prediction models to estimate the probability of sepsis being diagnosed in the emergency department.
RESULTS: We identified 21 unique prehospital screening strategies, 14 of which had numeric scores. We linked a total of 131 745 eligible patients to hospital databases. No single strategy had both high sensitivity (overall range 0.02–0.85) and high specificity (overall range 0.38–0.99) for classifying sepsis. However, the Critical Illness Prediction (CIP) score, the National Early Warning Score (NEWS) and the Quick Sepsis-Related Organ Failure Assessment (qSOFA) score predicted a low to high probability of a sepsis diagnosis at different scores. The qSOFA identified patients with a 7% (lowest score) to 87% (highest score) probability of sepsis diagnosis.
INTERPRETATION: The CIP, NEWS and qSOFA scores are tools with good predictive ability for sepsis diagnosis in the prehospital setting. The qSOFA score is simple to calculate and may be useful to paramedics in screening patients with possible sepsis.
Sepsis is a syndrome characterized by life-threatening organ dysfunction due to a dysregulated host response to infection.1 Early identification and prompt intervention are critical to improving outcomes for patients with sepsis.2,3 Paramedics are the first to evaluate and manage most patients with sepsis,4 often for an extended period before arrival in the emergency department. However, in the prehospital setting, without access to laboratory results, it can be challenging to differentiate patients who have sepsis from those who have infection without organ dysfunction.
The potential for paramedics to contribute to the early identification of sepsis using clinical signs and symptoms has been discussed but seldom rigorously studied.5–8 Studies that propose screening strategies for identification of sepsis by paramedics are frequently limited by incomplete prehospital measurements, small sample size or the use of convenience samples comprising only patients with a diagnosis of sepsis made in the hospital.5 Furthermore, these studies have often relied solely on measures of test accuracy that depend on the known diagnosis of sepsis (sensitivity and specificity), which are sensitive to spectrum bias due to underlying disease severity. 9 They also require that a threshold be established to define a positive versus negative test result, which may conceal the diagnostic information in individual test results that is more relevant to clinical decision-making for individual patients in different settings.10
To determine which approach to screening for sepsis is optimal in the prehospital setting, we completed a validation of the accuracy and predictive ability of published approaches for identification of patients with sepsis within a large cohort of patients with suspected infection who were transported by emergency medical services.
Methods
Identification of screening strategies
We re-ran the search strategy from our previously published systematic review5 to find additional screening strategies for identification of infection or sepsis in the prehospital setting (search dates Oct. 1, 2015, to July 22, 2019). The search strategy and methods are described in Appendix 1, Part A1-1 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190966/-/DC1).
Study design
In this 1-year validation study, we compared the classification and predictive ability of published prehospital sepsis screening strategies applied to a cohort of patients with a single reference standard for diagnosis of sepsis. We used the STROBE11 and RECORD12 statements to guide reporting.
Study population and setting
Records for all adult patients (age ≥ 18 yr) transported between Apr. 1, 2015, and Mar. 31, 2016, by a large provincial emergency medical service in Alberta were deterministically linked to a population-based emergency administrative database (National Ambulatory Care Reporting System) and an inpatient database (Discharge Abstract Database) by analysts in the emergency medical service using each patient’s unique health number, birth date, time and initial destination for patient transport.
Sepsis should be considered when clinicians have a strong suspicion of infection;1 therefore, we assembled a cohort consisting only of patients who had an infection diagnosed in the emergency department. We identified these patients using previously validated diagnosis codes for use in the emergency department consistent with a bacterial or fungal infection13 (as listed in Appendix 2, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190966/-/DC1). We also assembled a subcohort of patients for whom paramedics documented a suspected infection (i.e., real-world application). Paramedic-suspected infection was determined by examining several fields, including those for the chief complaint (patient perspective) and the provider impression (paramedic perspective), for terms consistent with infection (e.g., cellulitis, urinary tract infection, pneumonia), sepsis, cough or flu-like symptoms. We also included patients for whom paramedics had selected the sepsis treatment protocol.
For patients with multiple transports or emergency admissions in a single day, we retained only the initial emergency medical service record and the final emergency department record, because these were the most complete records. Patients who were admitted to the initial destination hospital were also linked to an inpatient administrative database (the Discharge Abstract Database of the Canadian Institute for Health Information) to determine in-hospital disposition (admission to the intensive care unit, mechanical ventilation, length of stay and death).
Variables
We extracted from the emergency medical service records all measured patient characteristics (age, weight, vital signs), the documented physical examination findings (including Glasgow Coma Scale score14) and operational characteristics (date and time stamps). For each characteristic, the first available measure was used for evaluation of all screening strategies, as we hypothesized that initial measurements were least likely to be influenced by medical intervention and most likely to inform subsequent care by paramedics. These measures are entered directly into the patient care record during the patient encounter or are imported from a monitoring device and verified by the paramedics before they leave the hospital.
Application of screening strategies
We applied the prehospital screening strategies identified in our updated search to our cohort (all patients with confirmed infection) using the recommended measures, which resulted in a “positive” or “negative” screening result for each strategy for each patient. For strategies with a numeric score, we determined both the numeric score and the screening result based on the recommended threshold. We conducted sensitivity analyses to evaluate the ability of each approach to identify sepsis in the entire population of transported patients and also in the subcohort of patients for whom paramedics documented a suspected infection.
Outcome measure
The primary goal of this study was to compare the ability of the screening strategies to identify the outcome of sepsis. We identified cases of sepsis using a strategy based on the Canadian version of the International Statistical Classification of Diseases and Related Health Problems, 10th revision, validated for the 2012 Surviving Sepsis definition,15 modified to align with the Sepsis-3 definition. This strategy identified patients as having sepsis if they were diagnosed with infection in the emergency department and were found to have organ dysfunction characteristic of sepsis. Organ dysfunction was identified from diagnostic codes or altered vital signs consistent with organ dysfunction (identified on the basis of abnormalities in documented pulse oximetry, mean arterial pressure or Glasgow Coma Scale score that would be consistent with a sequential organ failure assessment score of 2 or greater1). We excluded patients who were discharged from the emergency department. This approach was found to be reliable, and it had good criterion and construct validity for identifying patients with sepsis in the emergency department.13
Statistical analysis
Within the cohort of patients with infection diagnosed in the emergency department, we assessed diagnostic accuracy by calculating sensitivity, specificity and the corresponding positive and negative predictive values according to the result from each screening strategy. We assessed the predictive ability of strategies with a numeric score using diagnostic prediction models, with diagnosis of sepsis in the emergency department as the binary outcome.16 Each patient’s score was calculated and included in the model, along with their age and sex (if not already a component of the strategy) to adjust for nonrandom differences in these variables within our population. We addressed the possibility of dependence between observations due to clustering by destination hospital using the Huber-White robust covariance matrix estimates.17 We created a visual comparison of the predicted probabilities from each score, representing a patient’s probability of having sepsis from the minimum to maximum level of each score. We assessed discrimination with the C statistic, and we assessed calibration visually with calibration plots (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190966/-/DC1).18,19 Probability estimates for sepsis diagnosis with 95% confidence intervals are reported here. We used normal-value imputation of missing values for both analyses. We also completed a sensitivity analysis that excluded patients for whom any measure was missing.
All statistical analyses were completed in R statistical and computing software. We used the “tableone” package for descriptive statistics, the Quan method (using the “icd” package) for calculating Charlson scores20 and the “rms” package for constructing prediction models.17,21,22
Ethics approval
This study was reviewed and approved under a waiver of informed consent by the University of Calgary Conjoint Health Research Ethics Board and the University of Toronto Health Science Research Ethics Boards.
Results
Screening strategies to identify sepsis
We identified 32 studies,23–54 1 abstract55 and 3 ongoing registered studies56–58 that described 21 unique screening strategies for the identification of sepsis in the prehospital setting (Table 1). Among these strategies, 14 had a numeric score. All of the strategies included more than 1 patient measure, whereas 3 of the strategies (Critical Illness Prediction [CIP], Screening to Enhance Prehospital Identification of Sepsis [SEPSIS] and Prehospital Severe Sepsis [PRESS]) also included patient characteristics other than measured patient values. A detailed description of the components of each screening strategy is available in Appendix 1, Table A1-1.
Patients
Of 146 626 adult patients transported during the study period, 131 745 (90%) were successfully linked to hospital databases. The most common reasons for linkage failure were lack of a unique health number documented on the emergency medical service report (36%) and inability to match emergency medical service and hospital records (38%). Among the patients with successful linkage, 12 740 had infection and therefore constituted our primary cohort; for 2740 (22%) of these, sepsis was diagnosed in the emergency department (Figure 1). The proportion of patients with missing values was low (range 0%–4%) for most vital signs, but was high for blood glucose level (24%) and end-tidal carbon dioxide (86%). Characteristics of the patients are presented in Table 2.
Accuracy of classification
We observed considerable variation in classification accuracy among the screening strategies. None of the screening strategies was both highly sensitive and highly specific for the diagnosis of sepsis. Positive predictive values ranged from 0.23 to 0.68, with only 6 strategies having positive predictive values above 0.5 (Table 3). When we excluded patients with missing measures, strategies for which measures were missing for a large proportion of patients (e.g., end-tidal carbon dioxide) had increased sensitivity and decreased specificity (Appendix 4, Table A4-1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190966/-/DC1). Classification accuracy for the entire cohort of patients for whom linkage was successful (n = 131 745) is presented in Appendix 4, Table A4-2.
The findings of our sensitivity analysis to evaluate the ability of each approach to identify sepsis in the entire population of transported patients is presented in Appendix 4, Table A4-2 and Table A4-3. In the sensitivity analysis conducted in the subcohort of 4138 patients for whom paramedics documented a suspected infection, 420 (10%) had sepsis diagnosed in the emergency department. Relative to the primary analysis, the positive predictive values for all screening strategies decreased (Appendix 4, Table A4-4).
Predictive ability
For the 14 screening strategies that generated a numeric score (Table 1), assessing predictive ability provides knowledge about the probability of sepsis diagnosis across each level of the numeric scores. We observed considerable variation in discrimination (C statistic range 0.61–0.79) and considerable change in probabilities for increasing scores among different strategies (Figure 2). Strategies using more measures and with a greater range of possible points generally identified patients with the highest probability of sepsis (e.g., CIP, National Early Warning Score [NEWS]; Appendix 3, Table A3-1). Strategies that included measures of the Glasgow Coma Scale and systolic blood pressure (e.g., CIP, NEWS, Quick Sepsis-Related Organ Failure Assessment [qSOFA]) generally identified patients with a higher probability of sepsis than strategies incorporating a similar number of predictors without either of these 2 measures (e.g., Prehospital Sepsis Assessment Tool, Systemic Inflammatory Response Syndrome [SIRS], Prehospital Sepsis Project [PSP]; Table 1). Two strategies with only 3 points on the scoring system, namely the qSOFA and the BAS (Blood Pressure Andningsfrekvens [“respiratory rate” in Swedish] Saturation), identified patients with a 20% to 30% increase in probability for each additional point (qSOFA range 0.07–0.87; BAS range 0.13–0.82); that is, qSOFA identified patients with a 7% (lowest score) to 87% (highest score) probability of sepsis diagnosis. However, another simple strategy (SIRS) had little change in probability of sepsis across the entire range of the score, identifying patients with only a 19% difference in probability between the minimum and maximum scores (Appendix 3, Table A3-1). Among patients with paramedic-suspected infection, the overall discrimination improved for all strategies (C statistic range 0.71–0.84), but the probabilities of sepsis diagnosis decreased for all strategies (Appendix 3, Table A3-2 and Figure A3-1). Calibration curves showed that the BAS and PSP strategies overestimated the probability of sepsis at high scores, whereas the PRESS score underestimated sepsis probability at high scores. Calibration for the Prehospital Early Sepsis Detection score was poor, whereas the remaining scores had consistent overlap of observed and predicted probabilities, which indicated good calibration (Appendix 3, Figures A3-2 to A3-15).
Interpretation
The accuracy of prehospital screening strategies for identification of sepsis by paramedics varied considerably, with no strategy having both high sensitivity and high specificity. However, in validating the predictive ability of strategies that used a numeric score, we found 3 strategies (CIP, NEWS and qSOFA) that had good discrimination and good calibration. With these strategies, low scores identify patients with low probability of sepsis, and high scores identify those with high probability of sepsis in the prehospital setting.
Sepsis is a syndrome rather than a disease.1 Thus, a spectrum of severity of illness among patients is expected, and a gold standard test to accurately diagnose patients with sepsis is not available. 61 To effectively navigate this uncertainty, clinicians need to know what information a screening strategy provides about an individual patient’s risk of having sepsis. Accuracy of classification provides limited information about uncertainty, because it relies on a single result or threshold to identify the patient as having the disease or not.9,62–64 Our approach of evaluating the predictive ability of sepsis diagnosis across the entire range of scores helps to address this uncertainty by highlighting the change in risk at different scores. Screening strategies that can identify patients with low and high probability of sepsis may help clinicians determine which patients with suspected infection have low risk, and which patients are at high risk of having sepsis. Conversely, screening strategies with little change in probability from their lowest to their highest scores do not convey useful information to clinicians about an individual patient’s risk of sepsis.
Previous studies have compared the accuracy of classification of screening strategies for prehospital identification of sepsis in the same population30,53,55 or in systematic reviews.5,6,65 However, these comparisons were limited because they assessed only a few of the available screening strategies, they compared studies using different case definitions for sepsis, or they used diagnostic accuracy metrics. In contrast, in our study, we compared all published strategies within the same population using the same case definition for sepsis and using diagnostic prediction models. No screening strategy will be perfectly accurate for the diagnosis of sepsis, but our estimates of the diagnostic predictive value or the probability of diagnosis with each strategy provide clinicians with knowledge about how certain a diagnosis may be, given an individual patient’s presentation, thus allowing them to determine who might benefit from earlier intervention.
The CIP, NEWS and qSOFA scores had good predictive ability and the greatest range in the probability of sepsis diagnosis from their minimum to their maximum scores. Prehospital systems may consider integration of these screening strategies into paramedic treatment protocols, using a higher probability of sepsis (i.e., a higher score) to inform a stepwise approach to more aggressive intervention by paramedics during transport of these patients. For example, paramedics might consider notifying the emergency department in advance if screening reveals that a patient has moderate probability of sepsis (e.g., qSOFA score of 1 or 2), whereas they might initiate prehospital interventions and emergency transport and provide advance notification for a patient with a higher probability of sepsis (e.g., qSOFA score of 3). Screening strategies requiring only 3 measures, such as qSOFA, are simple and more likely to be used by paramedics. Future studies could test the clinical benefit and feasibility of adopting these approaches to guide paramedic interventions prospectively, or they could investigate the predictive ability of these same screening strategies for other important patient outcomes, as has been tested for predicting mortality with the qSOFA score.44,66
Limitations
This study had some limitations. The update to our previous search, which we used to identify existing paramedic screening strategies, was not systematic and was limited by the use of only 1 reviewer and restriction of the search to English articles published in peer-reviewed journals.
In the absence of a gold standard for sepsis diagnosis, we adopted a validated approach that aligns with the most recent consensus definition.13 Previous studies have found consistent undercoding of sepsis when administrative databases are used;15 in our study, such undercoding of sepsis would result in missed cases and more conservative estimates of predictive ability overall.
Our use of patient measures to identify patients with organ dysfunction due to sepsis might have introduced some incorporation bias into the assessment of screening strategies that used these same measures to diagnose sepsis. However, our use of the strategy results rather than the individual measures would have reduced this bias; there was no direct incorporation of the measures into the model as predictors. We considered that this approach was superior to the alternative of excluding patients with clinically important organ dysfunction from the sepsis outcome classification due to the consistent undercoding discussed above.
A delay occurs between paramedic assessments and determination of sepsis in the emergency department. Sepsis is a syndrome that may progress during this period, and screening strategies that paramedics apply at initial assessment may not correctly identify patients whose condition will continue to worsen and in whom sepsis is subsequently diagnosed by emergency physicians. This limitation could decrease the apparent accuracy of the prehospital screening strategies for use by paramedics.
We did not have access to information about the location from which patients were transferred; this information might have helped us to identify populations at higher risk (e.g., nursing home residents).
In our sensitivity analysis, the rate of identification of infections by paramedics was low. The application of the screening strategies for detecting sepsis depends on recognition of infections; therefore, paramedics should be trained to improve infection recognition in the prehospital setting.
Conclusion
Validation of the predictive ability of available screening strategies for possible sepsis in the prehospital setting showed that certain scores identified patients with both low and high probability for sepsis diagnosis, despite poor sensitivity or specificity at recommended thresholds. The CIP, NEWS and qSOFA scores each identified patients with low probability of sepsis at low scores and high probability of sepsis at high scores. The qSOFA score is the simplest of these to calculate and should be considered for use by paramedics.
Footnotes
Competing interests: Sheldon Cheskes has received investigator-initiated grant funding from Zoll Medical for several research programs (AED on the Fly, Community Responder Program for Out-of-Hospital Cardiac Arrest and Measuring Ventilation During Out-of-Hospital Cardiac Arrest). He also sits on the advisory board of Drone Delivery Canada. Damon Scales holds operating grants from the Canadian Institutes of Health Research. No other competing interests were declared.
This article has been peer reviewed.
Contributors: Daniel Lane and Damon Scales conceived of the study. All of the authors contributed to design of the study and interpretation of the data. Daniel Lane performed the primary analysis, supervised by Refik Saskin and Damon Scales. Daniel Lane and Damon Scales wrote the primary draft of the manuscript. All of the authors revised the manuscript for important intellectual content, approved the final version for publication and agreed to be accountable for the work.
Funding: No specific funding was received for this work. Laurie Morrison is the Robert and Dorothy Pitts Chair in Acute Care and Emergency Medicine at St. Michael’s Hospital and the University of Toronto.
Data sharing: The data for this study were obtained under a research agreement with Alberta Health Services, Emergency Medical Services. Access to these data may be requested from Alberta Health Services through the process outlined on this site: www.albertahealthservices.ca/ems/Page13364.aspx
- Accepted January 15, 2020.