Computer-aided National Early Warning Score to predict the risk of sepsis following emergency medical admission to hospital: a model development and external validation study
==============================================================================================================================================================================

* Muhammad Faisal
* Donald Richardson
* Andrew J. Scally
* Robin Howes
* Kevin Beatson
* Kevin Speed
* Mohammed A. Mohammed

## Abstract

**BACKGROUND:** In hospitals in England, patients’ vital signs are monitored and summarized into the National Early Warning Score (NEWS); this score is more accurate than the Quick Sepsis-related Organ Failure Assessment (qSOFA) score at identifying patients with sepsis. We investigated the extent to which the accuracy of the NEWS is enhanced by developing and comparing 3 computer-aided NEWS (cNEWS) models (M0 = NEWS alone, M1 = M0 + age + sex, M2 = M1 + subcomponents of NEWS + diastolic blood pressure) to predict the risk of sepsis.

**METHODS:** We included all emergency medical admissions of patients 16 years of age and older discharged over 24 months from 2 acute care hospital centres (York Hospital [YH] for model development and a combined data set from 2 hospitals [Diana, Princess of Wales Hospital and Scunthorpe General Hospital] in the Northern Lincolnshire and Goole National Health Service Foundation Trust [NH] for external model validation). We used a validated Canadian method for defining sepsis from administrative hospital data.

**RESULTS:** The prevalence of sepsis was lower in YH (4.5%, 1596/35 807) than in NH (8.5%, 2983/35 161). The C statistic increased across models (YH: M0 0.705, M1 0.763, M2 0.777; NH: M0 0.708, M1 0.777, M2 0.791). For NEWS of 5 or higher, sensitivity increased (YH: 47.24% v. 50.56% v. 52.69%; NH: 37.91% v. 43.35% v. 48.07%), the positive likelihood ratio increased (YH: 2.77 v. 2.99 v. 3.06; NH: 3.18 v. 3.32 v. 3.45) and the positive predictive value increased (YH: 11.44% v. 12.24% v. 12.49%; NH: 22.75% v. 23.55% v. 24.21%).

**INTERPRETATION:** From the 3 cNEWS models, model M2 is the most accurate. Given that it places no additional burden of data collection on clinicians and can be automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

Sepsis is a major cause of mortality in hospitals. Survival is dependent on early recognition and treatment. Although each hour of delay is associated with a 7% reduction in survival,1,2 studies have found that treatment delays are not uncommon in hospitals.3 Several risk scores have been devised to aid the early detection of sepsis.4 Two widely used scores are the Systemic Inflammatory Response Syndrome (SIRS)5 and the Quick Sepsis-related Organ Failure Assessment (qSOFA) scores.4 Although studies have found SIRS to be more accurate than qSOFA for diagnosis of sepsis,6 a recent study found that the National Early Warning Score (NEWS) compares favourably with qSOFA.7,8

The NEWS was introduced in 2012 by the Royal College of Physicians of London to identify acutely ill patients, including those with sepsis.9 The NEWS is used to identify those at risk of death and increased morbidity in all patient diagnostic groups (with some noted exceptions, e.g., head injury). This score has been widely adopted in National Health Service (NHS) hospitals in England and other countries.9

The NEWS is derived from 7 physiologic variables or vital signs — respiration rate, oxygen saturations, any supplemental oxygen, temperature, systolic blood pressure, heart rate and level of consciousness as measured by the AVPU (alert, voice, pain, unresponsive) scale — that are routinely collected by nursing staff as an integral part of the process of care, usually within 30 minutes for most patients and subsequently repeated at a frequency dependent on local hospital protocols. NEWS points are allocated according to basic clinical observations, and the higher the NEWS the more likely it is that the patient is developing a critical illness (see Appendix 1, available at [www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181418/-/DC1](http://www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.181418/-/DC1), for further details on the NEWS).

The clinical rationale for the NEWS is that early recognition of deterioration in the vital signs of a patient can provide opportunities for earlier, more effective intervention. Furthermore, studies have shown that electronically collected NEWS10 are highly reliable and accurate when compared with paper-based methods,11–13 and about two-thirds of NHS hospitals now report the use of electronic NEWS (eNEWS).14

A NEWS of 5 or higher has been recommended as a trigger to screen for sepsis by the National Institute for Health and Care Excellence in England15,16 and has been widely adopted in NHS hospitals. However, given the widespread use of the eNEWS and the potential of this system to support real-time computer-aided screening for sepsis, we investigated the extent to which the accuracy of the NEWS for predicting sepsis could be enhanced by developing computer-aided NEWS (cNEWS) models. An important feature of our cNEWS models is that they are not designed for paper-based systems and do not place any additional burden of data collection or calculation on clinicians, because cNEWS relies on data that are routinely collected as part of the process of care, are already stored in the patient’s electronic health record and are accessible in real time, thus offering the prospects of real-time risk predictions without hindering clinical workflows.

We sought to examine the accuracy of 3 cNEWS models in predicting sepsis that include age, sex and the subcomponents of NEWS compared with a reference model that uses NEWS only, using a validated method for defining sepsis developed by Jolley and colleagues in Canada.17

## Methods

### Setting and data

Our cohorts comprised emergency medical admissions from 3 acute care hospitals located about 100 km apart in the Yorkshire & Humberside region of England — the Diana, Princess of Wales Hospital (about 400 beds) and Scunthorpe General Hospital (about 400 beds) managed by the Northern Lincolnshire and Goole NHS Foundation Trust (NLAG), and York Hospital (YH) (about 700 beds), managed by York Teaching Hospitals NHS Foundation Trust. For the purposes of this study, the 2 acute hospitals in NLAG are combined into a single data set and collectively referred to as NLAG Hospitals (NH). Both NH and YH use electronic NEWS scoring exclusively as part of their in-house electronic patient record systems. We selected these hospitals because these data have been collected as part of patients’ process of care since at least 2013, and the hospitals were agreeable to the study.

We considered all patients 16 years of age and older with emergency medical admissions, discharged during a 24-month period (Jan. 1, 2014, to Dec. 31, 2015), with eNEWS data. For each emergency admission, we obtained a pseudoanonymized patient identifier, patient’s age (years), sex (male/female), discharge status (alive/dead), admission and discharge date and time, and eNEWS (including its subcomponents: respiratory rate, temperature, systolic blood pressure, pulse rate, oxygen saturation, oxygen supplementation and alertness). The NEWS does not include diastolic blood pressure, but we incorporated it in our statistical models because this data item is routinely collected.

The NEWS scale ranges from 0 (indicating the lowest severity of illness) to a maximum of 20 (see Appendix 1 for further details). The index eNEWS was defined as the first score electronically recorded within ± 24 hours of the admission time. We excluded records in which the index eNEWS was not within ± 24 hours of recorded admission or for which eNEWS was not available (Appendix 1, Table S1 and S2).

We define sepsis (with at least 1 organ failure or septic shock)4 based on 84 selected *International Classification of Diseases, 10th Revision* (ICD-10) codes identified by an optimized validated method reported by Jolley and colleagues in Canada17 (which we adapted to our study by excluding 6 Canadian-specific ICD-10 codes and 3 procedure codes). We used this optimized approach for identifying sepsis using ICD-10 codes because other methods are known to underestimate sepsis from administrative data.17,18

### Statistical analysis

We reported the statistical differences in characteristics of our 2 hospitals using 2 independent sample *t* test (for continuous data) and χ2 proportion test (for categorical data).

We first performed exploratory analyses including scatter plots and box plots that showed the relation between covariates and risk of sepsis in our hospitals. We developed 3 logistic regression models for the risk of sepsis. The models (M0, M1 and M2) use the index or first-recorded eNEWS within ± 24 hours of admission. Model M0 uses eNEWS alone; model M1 extends M0 with age and sex, and model M2 extends M1 with all the subcomponents of NEWS plus diastolic blood pressure. We used likelihood ratio tests to determine the extent to which progressing from models M0 to M2 improved the goodness of fit.

We used the *qladder* function (Stata19), which displays the quantiles of transformed variable against the quantiles of a normal distribution according to the ladder powers (*x*3, *x*2, *x*1, *x*, ✓*x*, log(*x*), *x*−1, *x*−2, *x*−3) for each continuous covariate and chose the following transformations: loge (respiratory rate), loge (pulse rate), loge (systolic blood pressure) and loge (diastolic blood pressure).

All models were developed to predict the risk of sepsis following emergency medical admission using data from only YH (development data set). We then externally validated these models using data from the NH data set (external validation). We report discrimination and calibration statistics as performance measures for these models.20

Discrimination relates to how well a model can separate (or discriminate) between cases with and without sepsis and is given by the area under the receiver operating characteristic (ROC) curve (AUC) or C statistic after adjusting for differences in the baseline21 risk of sepsis in our 2 hospitals. The 95% confidence interval for the C statistic was derived using DeLong’s method as implemented in the pROC library22 in R.23

Calibration is the relation between the observed and predicted risk of sepsis24 and can be readily seen on a scatter plot (y-axis observed risk, x-axis predicted risk). Perfect predictions should be on the 45° line. The intercept (a) and slope (b) of this line gives an assessment of “calibration-in-the-large.” At model development, a = 0 and b = 1, but at external validation, calibration-in-the-large problems are indicated if “a” is not 0 and if “b” is more or less than 1, as this reflects problems of underprediction or overprediction.

The cut-off point of a NEWS of 5 or higher is the recommended threshold for screening sepsis.15,16 We determined the sensitivity, specificity, positive and negative predictive values, and likelihood ratios for these models (M0, M1 and M2) at eNEWS thresholds of 4 or higher, 5 or higher, and 6 or higher.25 For the best performing model (M2), we further analyzed its performance across a range of risks of sepsis (5%–15%) to highlight the performance characteristics of this model, which may inform choice of thresholds in routine clinical practice.

Analyses were carried out using the statistical language R,23 the ROCR25 library and Stata.19

### Ethics approval

We obtained ethics approval for this study from Yorkshire & The Humber — Leeds West Research Ethics Committee (reference number 15/YH/0348).

## Results

The number (YH: *n* = 36 751; NH: *n* = 37 100) of emergency medical admissions over a 24-month period was similar in YH and NH. We excluded 2.6% (944/36 751) of admissions in YH and 5.2% (1939/37 100) in NH because the index eNEWS was not recorded within 24 hours of the admission or there was no eNEWS recorded at all (Appendix 1, Tables S1 and S2).

The characteristics of the admissions included in our study are shown in Table 1. Patients with an emergency admission at YH were older than those at NH (67.8 yr v. 66.4 yr), less likely to be male (47.3% v. 49.8%), had higher index eNEWS (2.5 v. 2.1), and a much lower prevalence of sepsis (4.5% v. 8.5%) but similar in-hospital mortality (5.8% v. 5.4%). The prevalence of oxygen supplementation was lower at the YH than at the NH (11.3% v. 19.2%). See accompanying scatter and boxplots in Appendix 1, Figures S1 to S4. Figure 1 shows the relation between the index eNEWS with sepsis in each hospital. As the index eNEWS increases, so too does the risk of sepsis.

View this table:
[Table 1:](http://www.cmaj.ca/content/191/14/E382/T1)

Table 1: 
Characteristics of emergency medical admissions in the development and validation data sets

![Figure 1:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/191/14/E382/F1.medium.gif)

[Figure 1:](http://www.cmaj.ca/content/191/14/E382/F1)

Figure 1: 
Observed sepsis risk versus index eNEWS in YH and NH hospitals. Vertical bars are exact binomial 95% confidence intervals. For visualization purposes, we capped NEWS at 12. Note: eNEWS = electronic National Early Warning Score, NH = NLAG Hospitals, NLAG = Northern Lincolnshire and Goole NHS Foundation Trust, YH = York Hospital.

### Model comparison

We compared the 3 cNEWS models described previously to predict the risk of sepsis, using data from YH for model development and data from NH for validation.

The likelihood ratio test showed significant improvement in model goodness of fits (M0 v. M1: χ2 = 416.8 [df = 2], *p* < 0.001; M1 v. M2: χ2 = 161.8 [df = 10], *p* < 0.001). The ROC plots for each model are shown in Figure 2, and the accompanying discrimination and calibration statistics are shown in Table 2. Model M0 had the lowest C statistic in the development (0.705) and validation data sets (0.708). Models M1 and M2 had higher C statistics (Table 2).

![Figure 2:](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/191/14/E382/F2.medium.gif)

[Figure 2:](http://www.cmaj.ca/content/191/14/E382/F2)

Figure 2: 
Receiver operating characteristic curve for 3 models (M0, M1 and M2) in predicting the risk of sepsis in the YH and NH hospitals. Note: NH = NLAG Hospitals, NLAG = Northern Lincolnshire and Goole NHS Foundation Trust, YH = York Hospital.

View this table:
[Table 2:](http://www.cmaj.ca/content/191/14/E382/T2)

Table 2: 
Performance of 3 computer-aided National Early Warning Score models in predicting the risk of sepsis in the development and validation data sets

The external validation slope reduced from 1.18 (M0) to 1.15 (M2). The internal and external validation plots are shown in Appendix 1, Figure S5.

Table 3 shows the sensitivity, specificity, positive predictive value and negative predictive value at eNEWS of 4 or higher, 5 or higher, and 6 or higher for models M0, M1 and M2 for predicting the risk of sepsis.

View this table:
[Table 3:](http://www.cmaj.ca/content/191/14/E382/T3)

Table 3: 
Sensitivity, specificity and related analyses of M0, M1 and M2 to predict the risk of sepsis in the development and external validation data sets at selected eNEWS thresholds

At the current recommended threshold of a NEWS of 5 or higher for screening for sepsis, sensitivity increased across models M0, M1 and M2 in the development data set (47.24% v. 50.56% v. 52.69%) and the external validation data set (37.91% v. 43.35% v. 48.07%, respectively). Specificity changed little in the development data set (82.94% v. 83.09% v. 82.77%) and external validation data set (88.07 v. 86.96 v. 86.05). The positive likelihood ratio increased across models in the development data set (2.77 v. 2.99 v. 3.06) and the external validation data set (3.18 v. 3.32 v. 3.45). The negative likelihood ratio was reduced across models in the development data set (0.64 v. 0.59 v. 0.57) and the external validation data set (0.70 v. 0.65 v. 0.60). The positive predictive value increased in the development data set (11.44 v. 12.24 v. 12.49) and the external validation data set (22.75 v. 23.55 v. 24.21). The negative predictive value increased slightly in the development data set from (97.12 v. 97.30 v. 97.40) and the external validation data set (93.87 v. 94.30 v. 94.70).

We further explored the behaviour of the best performing cNEWS model (M2) across a range of cut-off probabilities (5%–15%) (Table 4). For YH, a cut-off point of 0.06 appears to offer reasonable performance; likewise, 0.12 appears to be a reasonable cut-off point for NH.

View this table:
[Table 4:](http://www.cmaj.ca/content/191/14/E382/T4)

Table 4: 
Sensitivity, specificity and related analyses of M2 to predict the risk of sepsis in the development and external validation data sets at range of predicted probabilities cut-off points

## Interpretation

We developed 3 computer-aided versions of eNEWS models, which incorporated progressively more information, and found that M2 (M1 + subcomponents of NEWS + diastolic blood pressure) was the best model to predict sepsis. This is unsurprising because it incorporates additional information about the patient’s age and other vital signs.

The main advantages of these cNEWS models is that they are designed to incorporate data that are already available in the patient’s electronic health record and so place no additional data collection or computational burden on clinicians, and are readily automated. Nonetheless, we note that computer-aided risk scores are not designed to replace clinical judgment. They are intended and designed to support, not undermine, clinical decision-making and can be overridden by clinical concern.9,26 However, cNEWS scores may enhance situational awareness of sepsis by processing information already available without impeding the workflow of clinical staff.

Although our previously published Computer-aided Risk of Sepsis Score,27 which is based on physiologic variables and blood results, offers more accuracy in predicting sepsis than cNEWS, it has 2 main disadvantages. First, up to one-quarter of emergency medical admissions do not have a blood test undertaken within 24 hours of admission; second, there is usually a delay between drawing blood and reporting of results. These disadvantages would delay automated assessment of sepsis risk. The advantage of cNEWS is that it can trigger for sepsis screening as soon as the first set of physiologic observations has been electronically recorded — usually within 30 minutes of admission for most patients.

### Limitations

There are several important limitations of our study. We identified sepsis based on a validated optimized algorithm using ICD-10 codes.17 Nonetheless, the extent to which differences between this approach to identifying sepsis and more recent consensus clinical definitions of sepsis4 uphold or undermine the evaluation of cNEWS merits further study.7,28 Although our cNEWS models performed well in external validation, the 95% confidence interval of the external calibration slope, despite adjustment for baseline differences in prevalence of sepsis between our 2 hospitals, did not include the ideal value of 1; this indicates some differences between observed and predicted risk of sepsis in the external data set. Further work is required to understand why this is the case (e.g., it may be attributable to different ways of recording sepsis between the 2 hospitals).

We used the index eNEWS data in our cNEWS models, which reflect the “on-admission” risk of sepsis of the patient. Nonetheless, eNEWS is repeatedly updated for each patient according to local hospital protocols, and the extent to which changes in eNEWS over time reflect changes in sepsis risk that need to be incorporated in our cNEWS models needs further study. Since cNEWS is based on NEWS and escalation protocols in hospitals are based on NEWS, work is required to determine how to successfully blend the risk estimates into existing escalation policies. For example, we could start by using a cut-off point that is similar to the current threshold of a NEWS of 5 or higher,15,16 but this will increase the number of patients that will trigger for screening.

An updated version of NEWS, known as NEWS2, has now been released,29 which includes a second oxygen scale for patients with proven type 2 respiratory failure and the inclusion of confusion (C) in the AVPU scale for conscious level (“alert, voice, pain, unresponsive” scale becomes ACVPU). The extent to which these changes to NEWS enhance our cNEWS models requires further investigation. A crucial next phase of this work is to field test cNEWS by carefully engineering it to build on the current use of NEWS in routine clinical practice30,31 to see if it does support the earlier detection and treatment of sepsis in emergency medical patients without unintended adverse consequences.

### Conclusion

From the 3 cNEWS models evaluated, we found model M2 to be the most accurate. Given that using the model places no additional data collection burden on clinicians and can be automated, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

## Footnotes

*   **Competing interests:** None declared.

*   This article has been peer reviewed.

*   **Contributors:** Donald Richardson and Mohammed A. Mohammed had the original idea for this work. Robin Howes and Kevin Beatson extracted the necessary data frames. Donald Richardson gave a clinical perspective. Muhammad Faisal, Andrew Scally and Mohammed A. Mohammed undertook the statistical analyses. Kevin Speed helped with the design of the study and acquisition of the data, and provided a clinical perspective. Muhammad Faisal and Donald Richardson wrote the first draft of this paper, and all authors subsequently assisted in redrafting and have approved the final version to be published. All of the authors agreed to be accountable for all aspects of the work.

*   **Funding:** This research was supported by The Health Foundation. The Health Foundation is an independent charity working to improve the quality of health care in the United Kingdom. This research was supported by the National Institute for Health Research (NIHR) Yorkshire and Humberside Patient Safety Translational Research Centre. The views expressed in this article are those of the authors and not necessarily those of the National Health Service, the NIHR or the Department of Health and Social Care.

*   Accepted March 20, 2019.

## References

1.  Ferrer R, Martin-Loeches I, Phillips G, et al. Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour. Crit Care Med 2014;42:1749–55. Available: [www.ncbi.nlm.nih.gov/pubmed/24717459](http://www.ncbi.nlm.nih.gov/pubmed/24717459) (accessed 2018 Oct. 24).
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/CCM.0000000000000330&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=24717459&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000339330700023&link_type=ISI) 

2.  Kumar A, Roberts D, Wood KE, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock*. Crit Care Med 2006;34:1589–96. Available: [www.ncbi.nlm.nih.gov/pubmed/16625125](http://www.ncbi.nlm.nih.gov/pubmed/16625125) (accessed 2018 Oct. 24).
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/01.CCM.0000217961.75225.E9&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16625125&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000237884300001&link_type=ISI) 

3.  Marik PE. Don’t miss the diagnosis of sepsis! Crit Care 2014;18:529. Available: [www.ncbi.nlm.nih.gov/pubmed/25675360](http://www.ncbi.nlm.nih.gov/pubmed/25675360) (accessed 2019 Feb. 24).
    
    

4.  Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016; 315:801–10.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.2016.0287&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=26903338&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

5.  Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med 2003;29:530–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/s00134-003-1662-x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=12664219&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

6.  Serafim R, Gomes JA, Salluh J, et al. A comparison of the Quick-SOFA and Systemic Inflammatory Response Syndrome criteria for the diagnosis of sepsis and prediction of mortality. Chest 2018;153:646–55. Available: [www.ncbi.nlm.nih.gov/pubmed/29289687](http://www.ncbi.nlm.nih.gov/pubmed/29289687) (accessed 2019 Feb. 24).
    
    

7.  Churpek MM, Snyder A, Han X, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195:906–11.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1164/rccm.201604-0854OC&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=27649072&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

8.  Goulden R, Hoyle M-C, Monis J, et al. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J 2018;35:345–9.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZW1lcm1lZCI7czo1OiJyZXNpZCI7czo4OiIzNS82LzM0NSI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xOTEvMTQvRTM4Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

9.  National Early Warning Score (NEWS): Standardising the assessment of acute illness severity in the NHS — report of a working party. London: Royal College of Physicians; 2012.
    
    

10. Smith GB, Prytherch DR, Schmidt P, et al. Hospital-wide physiological surveillance — a new approach to the early identification and management of the sick patient. Resuscitation 2006;71:19–28.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2006.03.008&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16945465&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000241177500004&link_type=ISI) 

11. Edwards M, McKay H, Van Leuvan C, et al. Modified early warning scores: Inaccurate summation or inaccurate assignment of score? Crit Care 2010; 14(Suppl 1):257.
    
    

12. Prytherch DR, Smith GB, Schmidt P, et al. Calculating early warning scores — a classroom comparison of pen and paper and hand-held computer methods. Resuscitation 2006;70:173–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.resuscitation.2005.12.002&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16806641&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000240488800003&link_type=ISI) 

13. Mohammed M, Hayton R, Clements G, et al. Improving accuracy and efficiency of early warning scores in acute care. Br J Nurs 2009;18:18–24.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.12968/bjon.2009.18.1.32072&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=19127227&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

14. NHS Improvement. Patient Safety Measurement Unit. United Kingdom: National Health Service; 2019. Available: [http://psmu.improvement.nhs.uk/workstreams/deterioration/news-baseline-survey](http://psmu.improvement.nhs.uk/workstreams/deterioration/news-baseline-survey) (accessed 2017 January).
    
    

15. Sepsis: quality standard. London (UK) National Institute for Health and Care Excellence; 2017. Available: [www.nice.org.uk/guidance/qs161/resources/sepsis-pdf-75545595402181](http://www.nice.org.uk/guidance/qs161/resources/sepsis-pdf-75545595402181) (accessed 2017 January).
    
    

16. Sepsis guidance implementation advice for adults. England: National Health Service; 2017. Available: [www.england.nhs.uk/wp-content/uploads/2017/09/sepsis-guidance-implementation-advice-for-adults.pdf](http://www.england.nhs.uk/wp-content/uploads/2017/09/sepsis-guidance-implementation-advice-for-adults.pdf) (accessed 2017 January).
    
    

17. Jolley RJ, Quan H, Jetté N, et al. Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open 2015;5:e009487.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiNS8xMi9lMDA5NDg3IjtzOjQ6ImF0b20iO3M6MjI6Ii9jbWFqLzE5MS8xNC9FMzgyLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

18. Jolley RJ, Sawka KJ, Yergens DW, et al. Validity of administrative data in recording sepsis: a systematic review. Crit Care 2015;19:139.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/s13054-015-0847-3&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=25887596&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

19. Stata: Release 14. Statistical Software. College Station (TX): StataCorp LP; 2016.
    
    

20. Steyerberg EW. Clinical prediction models: a practical approach to development, validation and updating. New York: Springer-Verlag New York; 2008.
    
    

21. Faisal M, Howes R, Steyerberg EW, et al. Using routine blood test results to predict the risk of death for emergency medical admissions to hospital: an external model validation study. QJM 2017;110:27–31.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1093/qjmed/hcw110&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=27486263&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

22. Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011;12:77.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1186/1471-2105-12-77&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=21414208&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

23. R: a language and environment for statistical computing. Vienna (Austria): R Project for Statistical Computing; 2015. Available: [www.r-project.org](http://www.r-project.org) (accessed 2017 January).
    
    

24. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–38.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/EDE.0b013e3181c30fb2&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=20010215&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000272872900023&link_type=ISI) 

25. Sing T, Sander O, Beerenwinkel N, et al. ROCR: visualizing classifier performance in R. Bioinformatics 2005;21:3940–1.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1093/bioinformatics/bti623&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=16096348&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000232596300023&link_type=ISI) 

26. Balamuth F, Alpern ER, Abbadessa MK, et al. Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign-based electronic alert and bedside clinician identification. Ann Emerg Med 2017; 70:759–68.e2.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/j.annemergmed.2017.03.019&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fcmaj%2F191%2F14%2FE382.atom) 

27. Faisal M, Scally A, Richardson D, et al. Development and external validation of an automated computer-aided risk score for predicting sepsis in emergency medical admissions using the patient’s first electronically recorded vital signs and blood test results. Crit Care Med 2018;46:612–8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/CCM.0000000000002967&link_type=DOI) 

28. Corfield AR, Lees F, Zealley I, et al. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg Med J 2014;31: 482–7.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZW1lcm1lZCI7czo1OiJyZXNpZCI7czo4OiIzMS82LzQ4MiI7czo0OiJhdG9tIjtzOjIyOiIvY21hai8xOTEvMTQvRTM4Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

29. National Early Warning Score (NEWS) 2: Standardising the assessment of acute illness severity in the NHS. Updated report of a working party. London: Royal College of Physicians; 2017
    
    

30. Escobar GJ, Dellinger RP. Early detection, prevention, and mitigation of critical illness outside intensive care settings. J Hosp Med 2016;11:S5–10.
    
    

31. Escobar GJ, Turk BJ, Ragins A, et al. Piloting electronic medical record-based early detection of inpatient deterioration in community hospitals. J Hosp Med 2016;11:S18–24.