This paper is in the following e-collection/theme issue:

Health Services Research (482) JMIR Theme Issue: COVID-19 Special Issue (2452) Clinical Information and Decision Making (1407)

Published on in Vol 11, No 2 (2022): Jul-Dec

This is a member publication of University College London (Jisc)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/41520, first published .
Frailty, Comorbidity, and Associations With In-Hospital Mortality in Older COVID-19 Patients: Exploratory Study of Administrative Data

Frailty, Comorbidity, and Associations With In-Hospital Mortality in Older COVID-19 Patients: Exploratory Study of Administrative Data

Frailty, Comorbidity, and Associations With In-Hospital Mortality in Older COVID-19 Patients: Exploratory Study of Administrative Data

Authors of this article:

Johannes Heyl1, 2 Author Orcid Image ;   Flavien Hardy2 Author Orcid Image ;   Katie Tucker3 Author Orcid Image ;   Adrian Hopper2, 4 Author Orcid Image ;   Maria J M Marchã5 Author Orcid Image ;   Annakan V Navaratnam6 Author Orcid Image ;   Tim W R Briggs2, 7 Author Orcid Image ;   Jeremy Yates8 Author Orcid Image ;   Jamie Day2 Author Orcid Image ;   Andrew Wheeler2 Author Orcid Image ;   Sue Eve-Jones2 Author Orcid Image ;   William K Gray2 Author Orcid Image
Article Authors Cited by (7) Tweetations (5) Metrics

    Original Paper

    1Department of Physics and Astronomy, University College London, London, United Kingdom

    2Getting It Right First Time programme, National Health Service England and National Health Service Improvement, London, United Kingdom

    3Innovation and Intelligent Automation Unit, Royal Free London National Health Service Foundation Trust, London, United Kingdom

    4Guy’s and St Thomas’ National Health Service Foundation Trust, London, United Kingdom

    5Science and Technology Facilities Council Distributed Research Utilising Advanced Computing High Performance Computing Facility, University College London, London, United Kingdom

    6University College London Hospitals National Health Service Foundation Trust, London, United Kingdom

    7Royal National Orthopaedic Hospital National Health Service Trust, London, United Kingdom

    8Department of Computer Science, University College London, London, United Kingdom

    Corresponding Author:

    Johannes Heyl, MSci

    Department of Physics and Astronomy

    University College London

    Gower St

    London, WC1E 6BT

    United Kingdom

    Phone: 44 020 7679 7155

    Email: johannes.heyl.19@ucl.ac.uk


    Background: Older adults have worse outcomes following hospitalization with COVID-19, but within this group there is substantial variation. Although frailty and comorbidity are key determinants of mortality, it is less clear which specific manifestations of frailty and comorbidity are associated with the worst outcomes.

    Objective: We aimed to identify the key comorbidities and domains of frailty that were associated with in-hospital mortality in older patients with COVID-19 using models developed for machine learning algorithms.

    Methods: This was a retrospective study that used the Hospital Episode Statistics administrative data set from March 1, 2020, to February 28, 2021, for hospitalized patients in England aged 65 years or older. The data set was split into separate training (70%), test (15%), and validation (15%) data sets during model development. Global frailty was assessed using the Hospital Frailty Risk Score (HFRS) and specific domains of frailty were identified using the Global Frailty Scale (GFS). Comorbidity was assessed using the Charlson Comorbidity Index (CCI). Additional features employed in the random forest algorithms included age, sex, deprivation, ethnicity, discharge month and year, geographical region, hospital trust, disease severity, and International Statistical Classification of Disease, 10th Edition codes recorded during the admission. Features were selected, preprocessed, and input into a series of random forest classification algorithms developed to identify factors strongly associated with in-hospital mortality. Two models were developed; the first model included the demographic, hospital-related, and disease-related items described above, as well as individual GFS domains and CCI items. The second model was similar to the first but replaced the GFS domains and CCI items with the HFRS as a global measure of frailty. Model performance was assessed using the area under the receiver operating characteristic (AUROC) curve and measures of model accuracy.

    Results: In total, 215,831 patients were included. The model using the individual GFS domains and CCI items had an AUROC curve for in-hospital mortality of 90% and a predictive accuracy of 83%. The model using the HFRS had similar performance (AUROC curve 90%, predictive accuracy 82%). The most important frailty items in the GFS were dementia/delirium, falls/fractures, and pressure ulcers/weight loss. The most important comorbidity items in the CCI were cancer, heart failure, and renal disease.

    Conclusions: The physical manifestations of frailty and comorbidity, particularly a history of cognitive impairment and falls, may be useful in identification of patients who need additional support during hospitalization with COVID-19.

    Interact J Med Res 2022;11(2):e41520

    doi:10.2196/41520

    Keywords

    COVID-19; coronavirus; SARS-CoV-2; frailty; comorbidity; mortality; death; hospitalization; hospital admission; hospitalisation; patient; age; sex; ethnicity; disease; hospital; cancer; heart; heart failure; weight loss; weight; renal disease; support; geriatric; older adult; elder; descriptive statistics; machine learning; model 


    Various studies have been conducted to look at the factors that contribute the most to poorer outcomes for people with COVID-19. In both community-based and hospital-based studies, age has consistently been found to be the strongest predictor of mortality in people with COVID-19 [1]. However, distinguishing between the effects of chronological age and the effects of age-related changes in health status linked to frailty and comorbidities could improve patient-centered care and health care resource allocation [2-5].

    Many previous studies of frailty in COVID-19 have used the Clinical Frailty Scale (CFS) to assess frailty status [6-11]. CFS-assessed frailty has been found to be consistently associated with mortality risk in COVID-19 patients [12]. However, as a clinical tool, the CFS score is usually not recorded in large databases, and these studies tend to be of relatively small cohorts. A recent systematic review of studies using the CFS identified a strong link between frailty and mortality but noted that most studies were at high risk of bias and suggested that further studies were warranted [13]. Larger studies have been conducted, but have often focused on specific cohorts of patients, such as those in critical care [14,15].

    A number of tools have been developed to identify frailty and comorbidity from large administrative databases, including some developed using artificial intelligence algorithms [16,17]. A recent review [12] identified 5 such tools, including the electronic Frailty Index [18] (for use in primary care), the Hospital Frailty Risk Score (HFRS) [19], the Global Frailty Scale (GFS) [20], and the Charlson Comorbidity Index (CCI) [21]. Such tools rely on coded diagnostic data and may help provide insights beyond those that can be obtained from smaller clinical studies of COVID-19 patients.

    The aim of this study was to assess the potential of an administrative database of patients aged 65 years or older to explore the relationship between frailty and comorbidities (defined using coded diagnostic data) and COVID-19 in-hospital mortality. We used machine learning algorithms to analyze the data. Machine learning offers a flexible approach to exploratory analysis, as it makes no a priori assumptions about the hierarchy of variables or their relationships. This allowed us to assess the relative importance of the various frailty and comorbidity features in relation to in-hospital mortality. It is particularly important to be able to identify the relative importance of these frailty and comorbidity features, which are typically long-term in nature, relative to admission-specific items.


    Ethical Considerations

    Ethical approval was not sought for the present study because it did not directly involve human participants. Consent from individuals involved in this study was not required for this analysis of the Hospital Episodes Statistics (HES) administrative data set. Guidance from National Health Service (NHS) Digital for the use and reporting of HES data for research purposes was followed, with anonymization to the level required by the ISB1523 Anonymisation Standard for Publishing Health and Social Care Data [22]. This study was completed in accordance with the Helsinki Declaration as revised in 2013.

    Study Design and Data Collection

    This was a retrospective, exploratory analysis of HES data. HES data are collected by NHS Digital for all NHS-funded patients admitted to hospitals in England. Data are entered by trained clinical coders in each hospital trust; data collection and reporting are mandatory. The data collected include demographics, the nature and timing of admission and discharge, diagnoses, and procedures undertaken.

    Timing, Case Ascertainment, and Inclusion and Exclusion Criteria

    We reviewed HES data for all completed episodes of hospital care in England with a discharge date from March 1, 2020, to February 28, 2021, that involved a diagnosis of COVID-19. We only considered completed episodes of care in which the patient had been discharged and their outcome (died or survived) was known. Patients aged <65 years were excluded. Cases of COVID-19 were identified using the International Statistical Classification of Disease, 10th Edition (ICD-10) codes (2019 version) U07.1 (ie, presence of COVID-19 has been confirmed by laboratory testing) and U07.2 (ie, clinical or epidemiological diagnosis of COVID-19 where laboratory confirmation is inconclusive or not available). The diagnoses were made either on admission or during the stay and could be primary or secondary. These 2 codes were created by the World Health Organization to code COVID-19 data [23].

    Where a patient had multiple admissions during the study period, only the chronologically last admission was retained. This ensured that all admissions were independent of one another at a patient level and avoided biasing the data by including cases where the outcome was predefined by virtue of a subsequent admission.

    Outcomes

    The outcome of interest was in-hospital mortality, as recorded by the Office for National Statistics. All data were available to us though NHS Digital and linked at a patient level using a pseudonymized patient identifier. An in-hospital death was recorded if the date of death was the same as or within 1 day of the hospital discharge date. Data on length of stay were also extracted and used to compare the relationship between these 2 patient outcomes.

    Features

    Frailty/Comorbidity Features

    The HFRS was categorized as none, mild, moderate, or severe for the descriptive analysis and as a continuous score in the machine learning algorithm [19]. The HFRS is calculated from 109 ICD-10–coded diagnoses during the index admission of any admission in the previous 2 years to give a weighted score. The HFRS gives a global assessment of frailty status and cannot be broken down into individual domains. It has been validated for use in a number of settings. [19,24-26]

    The GFS defines 7 domains of frailty (dementia and delirium; mobility problems; falls and fractures; pressure ulcers and weight loss; incontinence; and anxiety and depression) based on ICD-10 codes for hospital admissions during the previous year [20]. The GFS is closely aligned with the key clinical subdomains of frailty and considers the impact of manifestations of frailty on functional ability. It was developed by considering the relationship between the frailty domains and long hospital stays, 30-day nonelective readmission, and in-hospital mortality. It has not been validated outside of the original development study. The domain of dependency/care was not used, as an exploratory analysis suggested that the 2 ICD-10 codes used to define it (Z74 and Z75) were used in HES to identify patients who had survived to discharge but could not be discharged due to an unmet social care need.

    The CCI identifies 14 specific medical conditions identified as secondary diagnoses in the index admission and primary or secondary diagnoses in any admission during the previous year. The conditions are peripheral vascular disease, congestive heart failure, acute myocardial infarction, cerebrovascular disease, dementia, chronic pulmonary disease, connective tissue disease/rheumatic disease, peptic ulcer, liver disease (mild and moderate/severe), diabetes (with and without chronic complications), paraplegia/hemiplegia, renal disease, cancer (primary and metastatic), and HIV/AIDS [21]. It has been extensively validated [27].

    An index admission diagnosis of obesity was based on ICD-10 code E66.

    Non–Frailty/Comorbidity Features

    Age was categorized as bands (65-69 years, 70-79 years, and 80 years or older) for descriptive analysis and as a continuous variable when input into the machine learning algorithm.

    Sex was categorized as female or male.

    Ethnicity was categorized as White, Black or Black British, South Asian or South Asian British, other Asian or other Asian British, mixed, or other. For a number of patients, an ethnicity category was not recorded. In these cases, the HES database was searched for the most recent prior hospital admission for the same patient where ethnicity had been recorded and this value was used.

    The index of multiple deprivation (IMD) score (2019 version) was used to categorize relative deprivation. It is measured in England by assigning each of England’s 32,844 lower layer super output areas (LSOAs) a deprivation score calculated from a weighted average of 7 deprivation-related domains: income (22.5%), employment (22.5%), health deprivation and disability (13.5%), education or skills training (13.5%), crime (9.3%), barriers to housing and services (9.3%), and living environment (9.3%) [28-30]. The IMD score is reported as deciles in the descriptive analysis and used as a continuous variable in the machine learning algorithm.

    Hospital trusts typically run between 1 and 4 NHS hospitals covering a geographically defined catchment.

    NHS regions include London; the southeast, southwest, and east of England; the Midlands; the northeast and Yorkshire, and the northwest.

    The individual ICD-10 codes recorded in the diagnostic record during the hospital stay were included as binary features.

    Data Analysis and Model Building

    Data were analyzed using the Python programming language (version 3.9, Python Software Foundation). Descriptive statistics techniques were used to summarize the data in the covariate categories described above.

    All machine learning models were developed using the scikit-learn library. Random forest classifiers were used to identify key covariates associated with in-hospital mortality. Random forest classifiers are ensemble classifiers that fit decision trees to portions of the data and average over all decision trees. This is of particular importance if a machine learning model is to provide useful information about the relationship between the features and the outcome variable to clinicians. Machine learning has an advantage over traditional statistical models because it does not make any assumptions about the nature of the model. Machine learning has shown benefits in analyzing health care data [31-33].

    To identify the most important features for each model, we used the SHAP (Shapley additive explanation) feature importance method [34]. Feature importance values were calculated using TreeSHAP, an efficient estimation approach for tree-based models. The SHAP feature importance method allows for the identification of the nature of the relationship between the individual features and the output variable [35]. In a plot of SHAP values, each dot in the plot represents a patient. The dots are colored red or blue. The color of the dot represents the size of the feature relative to the range of values that feature can take, with red representing large feature values and blue low feature values. A positive SHAP value can be interpreted as meaning the feature is associated with in-hospital mortality. A negative SHAP value can be interpreted as meaning the feature is associated with the patient surviving to discharge. The features are ranked by the mean of the absolute value of the SHAP values.

    Two different random forest models were constructed to classify patients according to mortality status, and their predictive accuracy was compared. The models differed in their choice of features. Model 1 included age, sex, deprivation, ethnicity, region, NHS trust, ICD-10 codes, the 14 CCI items, and the GFS domains. Model 2 included the same items as model 1, except the HFRS bands were added as a feature and the CCI items and the GFS domains were removed. The 2 models allowed a comparison of the performance of a model that included individual frailty domains and comorbidities (model 1) and one that included a single global measure of frailty. All listed variables were included in the final model, although only the most important features are described.

    To avoid collinearity, features with a high degree of correlation (ie, a bivariate correlation coefficient >0.5) were excluded. The dementia item from the CCI and the dementia/delirium item from the GFS had a correlation coefficient of 0.6. As the GFS item had the broader definition, this was used as a covariate and the CCI item was excluded. No other items were excluded due to high correlation.

    For data preprocessing, the data set was randomly split at a ratio of 70:15:15 into a training set, a testing set, and a validation set, respectively. All 3 data sets contained patients who had died and patients who had survived. The machine learning algorithm was trained on the training set and its performance was evaluated based on how well it could predict mortality in the test set. To ensure that the model did not simply classify according to the majority outcome (ie, survival), the training set was reduced further by randomly removing patients who had survived to ensure that there were an equal number of patients who had died and who had survived in the training set. This eliminated the effect of the class imbalance on the model performance and ensured that the model had sufficient exposure to patients who died. However, the test set on which the trained model was evaluated was not balanced, increasing the model’s external validity. The validation set was used to tune the hyperparameters of the random forest. There are several hyperparameters specific to the random forest classifier that can be tuned. The combination of hyperparameters with the highest area under the receiver operating characteristic (AUROC) curve was selected. The optimal hyperparameters were found by using the Bayesian optimization library. The hyperparameter ranges used are listed in Table S1 in Multimedia Appendix 1. These hyperparameters included the number of trees (n=112), the minimum samples per split (n=8) and the minimum samples per leaf (n=1). The AUROC curve was plotted as sensitivity versus 1–specificity [36].

    Categorical variables were one-hot encoded. This involved creating a binary column for each value that the variable could take. For example, for NHS region, a patient treated in the Midlands would have a value of 1 in the Midlands column, but a value of 0 in the other regional categories. The algorithm for model 1 was used to construct a model of the relationship between length of stay and in-hospital mortality.

    In the sensitivity analysis, the performance of the random forest classifier was compared to extreme gradient boosting (XGBoost) and multivariable logistic regression models.

    Other than for ethnicity (see “Features”), missing data were relatively rare, and no attempt was made to impute missing values. Patients with missing data were omitted from the analysis. The number of missing values for each variable is given in Table 1.

    Table 1. Demographic characteristics and in-hospital deaths of patients.
    CharacteristicsNumber of patients (N=215,831)In-hospital deaths (n=77,738)Chi-square (df)P value
    Age band (years), n (%)4213.2 (2)<.001

    		65-6927,401 (12.7)6431 (23.5)


    		70-7973,568 (34)23,277 (31.6)


    		≥80114,862 (53.2)48,030 (41.8)

    Sexa, n (%)1646.9 (1)<.001

    		Female101,989 (47.3)32,351 (31.7)


    		Male113,826 (52.7)45,382 (40)

    Deprivation decileb, n (%)16.2 (9).06

    		1 (most deprived)25,053 (11.6)8862 (35.4)


    		224,937 (11.3)8679 (35.6)


    		323,320 (10.8)8441 (36.2)


    		421,756 (10.1)7884 (36.2)


    		521,044 (9.8)7701 (36.6)


    		621,004 (9.7)7732 (36.8)


    		720,149 (9.3)7273 (36.1)


    		819,787 (9.2)7212 (36.4)


    		918,764 (8.7)6724 (35.8)


    		10 (least deprived)17,012 (7.9)6123 (36)

    Region in Englandc, n (%)246.1 (6)<.001

    		East22,934 (10.6)9096 (39.7)


    		London35,912 (16.6)12,617 (35.1)


    		Midlands44,590 (20.7)16,072 (36)


    		Northeast and Yorkshire34,850 (16.1)12,187 (35)


    		Northwest35,281 (16.3)12,971 (36.8)


    		Southeast29,562 (13.7)10,554 (35.7)


    		Southwest12,028 (5.6)4085 (34)

    Ethnicityd, n (%)46.1 (5)<.001

    		White181,453 (84.1)65,440 (36.1)


    		Black or Black British5794 (2.7)2108 (36.4)


    		South Asian or South Asian British10,216 (4.7)3910 (38.3)


    		Other Asian or other Asian British2659 (1.2)953 (35.8)


    		Mixed963 (0.4)342 (35.5)


    		Other4484 (2.1)1488 (33.2)

    Disease severity, n (%)


    		Pneumonia144,206 (65.9)66,323 (46.6)18,757.8 (1)<.001

    		Renal disease55,155 (25.6)29,353 (53.2)9512.3 (1)<.001

    		Blood clotting6836 (3.2)3017 (44.1)201.8 (1)<.001

    		Cardiology/circulation4967 (2.3)2529 (50.9)489.7 (1)<.001

    		Neurology6986 (3.2)3022 (43.3)164.2 (1)<.001

    		Digestive system235 (0.1)134 (57)45 (1)<.001

    		Sepsis16,327 (7.5)9534 (58.4)3837.6 (1)<.001

    aThere were 16 missing values.

    bThere were 3545 missing values.

    cThere were 674 non–National Health Service providers.

    dNot stated in 10,262 values.


    The data extraction process resulted in a data set of 215,831 patients (Figure S1 in Multimedia Appendix 1). The crude mortality rate was 36% (77,738/215,831). The breakdown of patient numbers and the associated mortality rate is presented by age, sex, deprivation decile, region, ethnicity, and disease severity marker in Table 1 and by GFS domain and CCI item in Table 2. Higher in-hospital crude mortality rates were seen in older age groups, men, and in almost all comorbidity and frailty groups, except those with mild liver disease and anxiety or depression. There was no obvious relationship between in-hospital mortality and deprivation and a relatively modest difference between the different ethnic groups, with South Asian patients having the highest in-hospital mortality rate. The median length of hospital stay was 10 (IQR 5-20) days in patients who survived to discharge and 9 (IQR 4-17) days in those who died in hospital.

    Table 2. Mortality rates by comorbidity/frailty measure. Significance was tested using the chi-square test with significance set at 5%. For each comorbidity and frailty item, the number of patients with the condition is given together with the number of in-hospital deaths.
    Comorbidity/frailty itemsPatients (N=215,831), n (%)In-hospital deaths (n=77,738), n (%)Chi-square (df)P value
    Charlson Comorbidity Index

    		Peripheral vascular disease15,519 (7.2)6663 (42.9)358.9 (1)<.001

    		Congestive heart failure42,370 (19.6)20,433 (48.2)3412.8 (1)<.001

    		Acute myocardial infarction26,670 (12.4)11,416 (42.8)611.2 (1)<.001

    		Cerebrovascular disease28,773 (13.3)11,241 (39.1)137.9 (1)<.001

    		Dementia44,036 (20.4)18,749 (42.6)1098.1 (1)<.001

    		Chronic pulmonary disease63,244 (29.3)24,298 (38.4)227.3 (1)<.001

    		Connective tissue/rheumatic disease7867 (3.6)2964 (37.7)8.2 (1).004

    		Peptic ulcer1979 (0.9)764 (38.6)7.0 (1).008

    		Mild liver disease7402 (3.4)2664 (36)0 (1).92

    		Moderate or severe liver disease1706 (0.8)975 (57.2)344.9 (1)<.001

    		Diabetes without chronic complications59,815 (27.7)22,704 (38)133.5 (1)<.001

    		Diabetes with chronic complications7190 (3.3)2864 (39.8)43.2 (1)<.001

    		Paraplegia and hemiplegia5667 (2.6)2253 (40)38.2 (1)<.001

    		Renal disease55,652 (25.8)24,947 (44.8)2533.5 (1)<.001

    		Primary cancer21,822 (10.1)9764 (44.7)864.1 (1)<.001

    		Metastatic carcinoma8095 (3.8)3675 (45.4)378.3 (1)<.001

    		HIV/AIDS72 (0.03)19 (26.4)7.4 (1).006

    		Obesity14,766 (6.8)5222 (35.5)5.4 (1).02
    Global Frailty Scale

    		Dementia and delirium76,669 (35.5)32,011 (41.8)1696.6 (1)<.001

    		Mobility problems29,191 (13.5)11,207 (38.4)82.6 (1)<.001

    		Falls and fractures81,805 (37.9)31,957 (39.1)530.7 (1)<.001

    		Pressure ulcers and weight loss23,249 (10.8)10,814 (46.5)1245.5 (1)<.001

    		Incontinence15,359 (7.1)6095 (39.9)96.4 (1)<.001

    		Anxiety and depression25,268 (11.7)8123 (32.1)186.0 (1)<.001

    The training data set included 151,081 patients, the test data set included 32,374 patients, and the validation data set included 32,376 patients. Table 3 shows the performance of the random forest classifier on the test set in the 2 models developed. The best performing model was model 1, which included the GFS domains and CCI items and had an accuracy of 83%, an AUROC curve of 90%, and a true positive rate of 81%. Model 2 had slightly poorer performance, with an accuracy of 82%, an AUROC curve of 90%, and a true positive rate of 80%. The AUROC curve for model 1 is shown in Figure S2 in Multimedia Appendix 1.

    Figure 1 shows the SHAP value dot plots for the 30 most important features for model 1. The most important disease severity items that the random forest identified as predictive of mortality were pneumonia, renal failure, and sepsis. The most important frailty items were dementia and delirium, falls and fractures, and pressure ulcers and weight loss. The most important comorbidities were renal disease, heart failure, and primary cancer. Figure 2 shows the probability of in-hospital mortality as calculated by the random forest algorithm as a function of length of stay. In-hospital mortality risk was low for those with length of stay less than 3 days, was relatively stable between 3 and 20 days, and declined with increasing length of stay thereafter. Figure 3 shows the SHAP value dot plots for the 30 most important features for model 2. The HFRS band ranks as one of the most important features.

    Table 3. Performance of the random forest classifier on the various models. For each model, the random forest classifier accuracy, the area under the receiver operating characteristic curve, and the true positive rate are listed. The true positive rate is the fraction of patients who died who were predicted to have died by the model.
    ModelRandom forest accuracy, %Area under the receiver operating characteristic curve, %True positive rate, %
    1839081
    2829080
    ‎
    Figure 1. Shapley value dot summary plot for model 1. Each dot in the plot represents a patient. The x-axis indicates whether there is a positive or negative correlation between the value of the feature and its contribution to the model prediction of a patient dying. The color of the dot represents the size of the feature relative to the range of values that feature can take, with red representing large feature values and blue low feature values. The horizontal axis represents the association of the feature value with the outcome. A positive SHAP value means the feature is associated with mortality. A negative SHAP value means the feature contributes to the patient surviving to discharge. The features are ranked by the mean of the absolute value of the SHAP values. CCI: Charlson Comorbidity Index; GFS: Global Frailty Scale; ICD-10: International Statistical Classification of Disease, 10th Edition; IMD: index of multiple deprivation; SHAP: Shapley additive explanation.
    View this figure
    ‎
    Figure 2. Plot of the predicted probability of death as a function of the length of stay.
    View this figure
    ‎
    Figure 3. SHAP value dot summary plot for model 2. Each dot in the plot represents a patient. The x-axis indicates whether there is a positive or negative correlation between the value of the feature and its contribution to the model prediction of a patient dying. The color of the dot represents the size of the feature relative to the range of values that feature can take, with red representing large feature values and blue low feature values. The horizontal axis represents the association of the feature value with the outcome. A positive SHAP value means the feature is associated with mortality. A negative SHAP value means the feature contributes to the patient surviving to discharge. The features are ranked by the mean of the absolute value of the SHAP values. HFRS: Hospital Frailty Risk Score; ICD-10: International Statistical Classification of Disease, 10th Edition; IMD: index of multiple deprivation; SHAP: Shapley additive explanation.
    View this figure

    Figure S3 in Multimedia Appendix 1 shows the critical care admissions by age band, with the decline in critical care use for older patients reflecting decisions regarding ceilings of care. Figure S4 in Multimedia Appendix 1 shows the time series of the number of hospital admissions and deaths over the course of the study period; higher patient numbers and lower in-hospital mortality rate in the second wave during winter 2020-2021 are apparent. Figures S5 to S7 in Multimedia Appendix 1 are plots of the random forest classifier’s prediction of the probability of mortality as a function of age for patients with and without dementia and delirium, pressure ulcers and weight loss, and falls and fractures. The presence of each domain of frailty was associated with a higher mortality rate for all domains. Figures S8 to S10 in Multimedia Appendix 1 are plots of the random forest classifier’s prediction of the probability of mortality as a function of age for patients with and without cancer, heart failure, and renal disease. Patients with any of these comorbidities had a noticeably higher risk of mortality. Figure S11 in Multimedia Appendix 1 shows the prediction of mortality as a function of age for the 4 HFRS bands and shows the association between greater frailty and in-hospital mortality risk across all age bands.

    From the sensitivity analysis, Table S2 in Multimedia Appendix 1 details the AUROC curve for the XGBoost and multivariable logistic regression models. Both models had an AUROC curve of 89%.


    Our study is one of very few to use machine learning techniques to explore the role of frailty and comorbidities in COVID-19 outcomes in hospitalized older adults, and by far the largest to date [37]. Measures such as the CFS and HFRS give a global measure of frailty but give little detail on the role of specific aspects of frailty and comorbidity in determining outcomes [38]. As such, their use in guiding decision-making has been questioned [39]. Our study provides a different perspective and explores specific domains of frailty and comorbidities associated with COVID-19 mortality using an administrative data set.

    In our study, preexisting dementia, falls and fractures, pressure ulcers and weight loss, renal disease, heart failure, and cancer were all important features in the model.

    Dementia/delirium was found to be the most important feature of all the frailty and comorbidity items investigated, with a consistent relationship between dementia/delirium across all ages. Studies from Italy and Brazil have found a higher COVID-19 mortality rate in those with delirium than those without [40,41]. An Italian study of 332 patients found that neurological comorbidities, which included dementia, were associated with a 2-fold increase in mortality, though dementia was not considered in isolation [42].

    Various studies have found that patients who have suffered from fractures are at increased risk of dying from COVID-19 [43,44], with one study noting that even though the volume of fracture patients admitted to hospital had decreased during the pandemic, the mortality rate had increased [45]. Respiratory diseases and cardiovascular diseases have been identified as associated with increased COVID-19 mortality risk in other studies [46]. In our study, we identified a substantial increase in the probability of death among patients with falls and fractures compared to those without.

    A previous study by members of our team using HES data for all hospitalized adults in England found that all comorbidities in the CCI, except mild liver disease and peptic ulcer, were strong predictors of in-hospital mortality [47]. This is broadly supported by other studies of large administrative databases [48-53].

    Age and male sex were important features in all models, which is consistent with previous reports [9,54-56]. The deprivation score was one of the most important features in both our models. Previous studies are inconsistent on the relative importance of deprivation in COVID-19 mortality [57,58]. However, there is a strong relationship between deprivation, ethnicity, age, and other covariates, and it is likely that different modeling approaches address the relationship in different ways.

    We found that length of stay had a strong relationship with in-hospital mortality. The risk of death increased between 0 and 3 days before decreasing again after 20 days.

    This study has numerous strengths. The use of the HES data set ensures that all hospital activity in England over the first year of the pandemic was captured, minimizing collider bias. We have demonstrated that a random forest classification algorithm is able to predict mortality with reasonable accuracy from an administrative data set. The accuracy of this work can be demonstrated by comparing the true positive rate of model 1 (81%) to the QCOVID risk algorithm, which had a sensitivity of 75.7% for identifying deaths within 97 days in the top 5% of at-risk patients [59]. An external validation of the QCOVID prediction algorithm found the sensitivity in predicting mortality to be 65.94% for men and 71.67% for women in the top 5% of most at-risk patients [60]. Model 1 is clearly comparable to these, despite being trained on an administrative data set lacking clinical details regarding presentation. The risk model for QCOVID used clinical markers for disease severity. It was not our aim to develop a risk prediction algorithm, and we would caution against using our findings to do so, given concerns over data poverty and model accuracy in underrepresented groups (eg, non-White ethnicities). However, provided these concerns can be addressed (eg, through the use of transfer learning in model development [61]), there is clear potential to use large administrative data sets to develop highly accurate models.

    There are also limitations to our study, mainly related to the nature of the HES data set. Comorbidities may only be coded if they are deemed relevant to the patient’s condition. As such, the reported prevalence of various domains of frailty and comorbidities is likely to underestimate their true prevalence. For example, it is possible that only the most severe cases of dementia/delirium were recorded in the HES database, which could explain the strong association in our study. Coding of COVID-19 will have been less consistent at the start of the pandemic, particularly with limited testing capacity. For this reason, we included patients diagnosed on clinical grounds, as well as those with a positive test.

    We also acknowledge that some secondary diagnoses may have been recorded in the HES database more commonly than others. Issues arise when different trusts’ coding teams code to a different depth of information and when some long-term conditions (eg, diabetes or dementia) are mandatory [62]. We also recognize that in cases of patient transfer to a different trust for treatment, the first admission would have been recorded in our data set as an earlier admission and removed. Thus, the admission period would appear shorter than it actually was. Issues around coding consistency across countries were identified during the GFS development study [20]. This could have impacted the reported relative importance of each frailty/comorbidity feature in the model.

    In summary, machine learning has proven useful in understanding the impacts of frailty and comorbidity on mortality. Our findings should help clinicians to identify which COVID-19 patients are most at risk of poor outcomes and help guide treatment strategies during future case surges. Artificial intelligence systems have already found use in guiding treatment strategies for palliative care. [63] A similar approach could be used to triage patients with COVID-19, building on insights from our work.

    Acknowledgments

    We acknowledge National Health Service Digital for permission to use their data in this report. The Getting It Right First Time program provides a framework for examining contemporary clinical practice in unprecedented detail and breadth. We also thank all staff within the individual National Health Service trusts who collected and entered the data used in this study, and we thank the Getting It Right First Time clinical leads for their advice: Michael Jones, Philip Dyer, Chris Moulton, Anna Batchelor, Michael Swart, Christopher Snowden, Martin Allen, Partha Kar, and Gerry Rayman. JH and FH received fellowships from Distributed Research Utilising Advanced Computing, which paid their salaries. This research received no other specific funding from any funding agency in the public, commercial, or not-for-profit sectors.

    Data Availability

    This report does not contain patient identifiable data. Consent from individuals involved in this study was not required. Requests for any underlying data cannot be granted by the authors because the data were acquired from data under license and a data sharing agreement from National Health Service Digital, for which conditions of use and further use apply. Individuals and organizations wishing to access Hospital Episodes Statistics data can make a request directly to National Health Service Digital.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Supplementary tables and figures.

    DOCX File , 817 KB
    1. Booth A, Reed AB, Ponzo S, Yassaee A, Aral M, Plans D, et al. Population risk factors for severe disease and mortality in COVID-19: A global systematic review and meta-analysis. PLoS One 2021;16(3):e0247461 [FREE Full text] [CrossRef] [Medline]
    2. Lewis EG, Breckons M, Lee RP, Dotchin C, Walker R. Rationing care by frailty during the COVID-19 pandemic. Age Ageing 2021 Jan 08;50(1):7-10 [FREE Full text] [CrossRef] [Medline]
    3. Aliberti MJR, Szlejf C, Avelino-Silva VI, Suemoto CK, Apolinario D, Dias MB, COVID HCFMUSP Study Group. COVID-19 is not over and age is not enough: Using frailty for prognostication in hospitalized patients. J Am Geriatr Soc 2021 May;69(5):1116-1127 [FREE Full text] [CrossRef] [Medline]
    4. D'ascanio M, Innammorato M, Pasquariello L, Pizzirusso D, Guerrieri G, Castelli S, et al. Age is not the only risk factor in COVID-19: the role of comorbidities and of long staying in residential care homes. BMC Geriatr 2021 Jan 15;21(1):63 [FREE Full text] [CrossRef] [Medline]
    5. Abdallah L, Stolee P, Lopez KJ, Whate A, Boger J, Tong C. The impact of COVID-19 on older adults' perceptions of virtual care: qualitative study. JMIR Aging 2022 Oct 20;5(4):e38546 [FREE Full text] [CrossRef] [Medline]
    6. Dumitrascu F, Branje KE, Hladkowicz ES, Lalu M, McIsaac DI. Association of frailty with outcomes in individuals with COVID-19: A living review and meta-analysis. J Am Geriatr Soc 2021 Sep;69(9):2419-2429 [FREE Full text] [CrossRef] [Medline]
    7. Zhang X, Jiao J, Cao J, Huo X, Zhu C, Wu X, et al. Frailty as a predictor of mortality among patients with COVID-19: a systematic review and meta-analysis. BMC Geriatr 2021 Mar 17;21(1):186 [FREE Full text] [CrossRef] [Medline]
    8. Hewitt J, Carter B, Vilches-Moraga A, Quinn TJ, Braude P, Verduri A, COPE Study Collaborators. The effect of frailty on survival in patients with COVID-19 (COPE): a multicentre, European, observational cohort study. Lancet Public Health 2020 Aug;5(8):e444-e451 [FREE Full text] [CrossRef] [Medline]
    9. Geriatric Medicine Research Collaborative, Covid Collaborative, Welch C. Age and frailty are independently associated with increased COVID-19 mortality and increased care needs in survivors: results of an international multi-centre study. Age Ageing 2021 May 05;50(3):617-630 [FREE Full text] [CrossRef] [Medline]
    10. Sablerolles RSG, Lafeber M, van Kempen JAL, van de Loo BPA, Boersma E, Rietdijk WJR, COMET research team. Association between Clinical Frailty Scale score and hospital mortality in adult patients with COVID-19 (COMET): an international, multicentre, retrospective, observational cohort study. Lancet Healthy Longev 2021 Mar;2(3):e163-e170 [FREE Full text] [CrossRef] [Medline]
    11. Blomaard LC, van der Linden CMJ, van der Bol JM, Jansen SWM, Polinder-Bos HA, Willems HC, et al. Frailty is associated with in-hospital mortality in older hospitalised COVID-19 patients in the Netherlands: the COVID-OLD study. Age Ageing 2021 May 05;50(3):631-640 [FREE Full text] [CrossRef] [Medline]
    12. Nghiem S, Sajeewani D, Henderson K, Afoakwah C, Byrnes J, Moyle W, et al. Development of frailty measurement tools using administrative health data: A systematic review. Arch Gerontol Geriatr 2020;89:104102. [CrossRef] [Medline]
    13. Rottler M, Ocskay K, Sipos Z, Görbe A, Virág M, Hegyi P, et al. Clinical Frailty Scale (CFS) indicated frailty is associated with increased in-hospital and 30-day mortality in COVID-19 patients: a systematic review and meta-analysis. Ann Intensive Care 2022 Feb 20;12(1):17 [FREE Full text] [CrossRef] [Medline]
    14. Jung C, Flaatten H, Fjølner J, Bruno RR, Wernly B, Artigas A, COVIP study group. The impact of frailty on survival in elderly intensive care patients with COVID-19: the COVIP study. Crit Care 2021 Apr 19;25(1):149 [FREE Full text] [CrossRef] [Medline]
    15. Jung C, Fjølner J, Bruno RR, Wernly B, Artigas A, Bollen Pinto B, ‚ COVIP Study Group. Differences in mortality in critically ill elderly patients during the second COVID-19 surge in Europe. Crit Care 2021 Sep 23;25(1):344 [FREE Full text] [CrossRef] [Medline]
    16. Oates J, Shafiabady N, Ambagtsheer R, Beilby J, Seiboth C, Dent E. Evolving hybrid partial genetic algorithm classification model for cost-effective frailty screening: investigative study. JMIR Aging 2022 Oct 07;5(4):e38464 [FREE Full text] [CrossRef] [Medline]
    17. Tarekegn A, Ricceri F, Costa G, Ferracin E, Giacobini M. Predictive modeling for frailty conditions in elderly people: machine learning approaches. JMIR Med Inform 2020 Jun 04;8(6):e16678 [FREE Full text] [CrossRef] [Medline]
    18. Clegg A, Bates C, Young J, Ryan R, Nichols L, Ann Teale E, et al. Development and validation of an electronic frailty index using routine primary care electronic health record data. Age Ageing 2016 May;45(3):353-360 [FREE Full text] [CrossRef] [Medline]
    19. Gilbert T, Neuburger J, Kraindler J, Keeble E, Smith P, Ariti C, et al. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 2018 May 05;391(10132):1775-1782 [FREE Full text] [CrossRef] [Medline]
    20. Soong JTY, Kaubryte J, Liew D, Peden CJ, Bottle A, Bell D, et al. Dr Foster global frailty score: an international retrospective observational study developing and validating a risk prediction model for hospitalised older persons from administrative data sets. BMJ Open 2019 Jun 22;9(6):e026759 [FREE Full text] [CrossRef] [Medline]
    21. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40(5):373-383 [FREE Full text] [CrossRef] [Medline]
    22. Anonymisation Standard for Publishing Health and and Social Care Data Specification (Process Standard). NHS Digital.   URL: https://tinyurl.com/ychvwe4u [accessed 2022-11-30]
    23. Living guidance for clinical management of COVID-19: Reporting and coding during the COVID-19 pandemic (mortality and morbidity). World Health Organization.   URL: https://apps.who.int/iris/bitstream/handle/10665/349321/WHO-2019-nCoV-clinical-2021.2-eng.pdf [accessed 2022-11-24]
    24. McAlister F, van Walraven C. External validation of the Hospital Frailty Risk Score and comparison with the Hospital-patient One-year Mortality Risk Score to predict outcomes in elderly hospitalised patients: a retrospective cohort study. BMJ Qual Saf 2019 Apr;28(4):284-288. [CrossRef] [Medline]
    25. Shebeshi DS, Dolja-Gore X, Byles J. Validation of hospital frailty risk score to predict hospital use in older people: Evidence from the Australian Longitudinal Study on Women's Health. Arch Gerontol Geriatr 2021;92:104282. [CrossRef] [Medline]
    26. Eckart A, Hauser SI, Haubitz S, Struja T, Kutz A, Koch D, et al. Validation of the hospital frailty risk score in a tertiary care hospital in Switzerland: results of a prospective, observational study. BMJ Open 2019 Jan 15;9(1):e026923 [FREE Full text] [CrossRef] [Medline]
    27. Frenkel WJ, Jongerius EJ, Mandjes-van Uitert MJ, van Munster BC, de Rooij SE. Validation of the Charlson Comorbidity Index in acutely hospitalized elderly adults: a prospective cohort study. J Am Geriatr Soc 2014 Feb;62(2):342-346. [CrossRef] [Medline]
    28. Office for National Statistics. Lower layer Super Output Area population estimates.   URL: https://tinyurl.com/3y4jsxw2 [accessed 2022-11-30]
    29. The English Indices of Deprivation 2019. Ministry of Housing, Communities & Local Government.   URL: https://www.gov.uk/government/statistics/english-indices-of-deprivation-2019 [accessed 2022-11-24]
    30. Lower Layer Super Output Area. NHS Data Model and Dictionary.   URL: https://tinyurl.com/mr36hfmd [accessed 2022-11-30]
    31. Subudhi S, Verma A, Patel AB, Hardin CC, Khandekar MJ, Lee H, et al. Comparing machine learning algorithms for predicting ICU admission and mortality in COVID-19. NPJ Digit Med 2021 May 21;4(1):87 [FREE Full text] [CrossRef] [Medline]
    32. Baqui P, Marra V, Alaa AM, Bica I, Ercole A, van der Schaar M. Comparing COVID-19 risk factors in Brazil using machine learning: the importance of socioeconomic, demographic and structural factors. Sci Rep 2021 Aug 02;11(1):15591 [FREE Full text] [CrossRef] [Medline]
    33. Estiri H, Strasser ZH, Klann JG, Naseri P, Wagholikar KB, Murphy SN. Predicting COVID-19 mortality with electronic medical records. NPJ Digit Med 2021 Feb 04;4(1):15 [FREE Full text] [CrossRef] [Medline]
    34. Lundberg S, Lee SI. A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017 Presented at: 31st International Conference on Neural Information Processing Systems; Dec 4, 2017; Long Beach, CA p. 4768-4777.
    35. Lundberg S, Erion G, Lee SI. Consistent individualized feature attribution for tree ensembles. ArXiv Preprint posted online on February 12, 2018. [CrossRef]
    36. Janssens ACJW, Martens FK. Reflection on modern methods: Revisiting the area under the ROC Curve. Int J Epidemiol 2020 Aug 01;49(4):1397-1403. [CrossRef] [Medline]
    37. Jung C, Mamandipoor B, Fjølner J, Bruno RR, Wernly B, Artigas A, et al. Disease-course adapting machine learning prognostication models in elderly patients critically ill with COVID-19: multicenter cohort study with external validation. JMIR Med Inform 2022 Mar 31;10(3):e32949 [FREE Full text] [CrossRef] [Medline]
    38. Maynou L, Owen R, Konstant-Hambling R, Imam T, Arkill S, Bertfield D, et al. The association between frailty risk and COVID-19-associated all-mortality in hospitalised older people: a national cohort study. Eur Geriatr Med 2022 Oct;13(5):1149-1157 [FREE Full text] [CrossRef] [Medline]
    39. Hogan DB, Maxwell CJ, Afilalo J, Arora RC, Bagshaw SM, Basran J, et al. A scoping review of frailty and acute care in middle-aged and older individuals with recommendations for future research. Can Geriatr J 2017 Mar;20(1):22-37 [FREE Full text] [CrossRef] [Medline]
    40. Garcez FB, Aliberti MJR, Poco PCE, Hiratsuka M, Takahashi SDF, Coelho VA, et al. Delirium and adverse outcomes in hospitalized patients with COVID-19. J Am Geriatr Soc 2020 Nov;68(11):2440-2446 [FREE Full text] [CrossRef] [Medline]
    41. Rebora P, Rozzini R, Bianchetti A, Blangiardo P, Marchegiani A, Piazzoli A, CoViD-19 Lombardia Team. Delirium in patients with SARS-CoV-2 Infection: a multicenter study. J Am Geriatr Soc 2021 Feb;69(2):293-299 [FREE Full text] [CrossRef] [Medline]
    42. Romagnolo A, Imbalzano G, Artusi CA, Balestrino R, Ledda C, De Rosa FG, et al. Neurological comorbidities and COVID-19-related case fatality: A cohort study. J Neurol Sci 2021 Sep 15;428:117610 [FREE Full text] [CrossRef] [Medline]
    43. Muñoz Vives JM, Jornet-Gibert M, Cámara-Cabrera J, Esteban PL, Brunet L, Delgado-Flores L, Spanish HIP-COVID Investigation Group. Mortality rates of patients with proximal femoral fracture in a worldwide pandemic: preliminary results of the Spanish HIP-COVID Observational Study. J Bone Joint Surg Am 2020 Jul 01;102(13):e69 [FREE Full text] [CrossRef] [Medline]
    44. Dupley L, Oputa TJ, Bourne JT, North West COVID NOF Study Group. 30-day mortality for fractured neck of femur patients with concurrent COVID-19 infection. Eur J Orthop Surg Traumatol 2021 Feb;31(2):341-347 [FREE Full text] [CrossRef] [Medline]
    45. Kumar Jain V, Lal H, Kumar Patralekh M, Vaishya R. Fracture management during COVID-19 pandemic: A systematic review. J Clin Orthop Trauma 2020 Jul;11(Suppl 4):S431-S441 [FREE Full text] [CrossRef] [Medline]
    46. Yang J, Zheng Y, Gou X, Pu K, Chen Z, Guo Q, et al. Prevalence of comorbidities and its effects in patients infected with SARS-CoV-2: a systematic review and meta-analysis. Int J Infect Dis 2020 May;94:91-95 [FREE Full text] [CrossRef] [Medline]
    47. Navaratnam AV, Gray WK, Day J, Wendon J, Briggs TWR. Patient factors and temporal trends associated with COVID-19 in-hospital mortality in England: an observational study using administrative data. Lancet Respir Med 2021 Apr;9(4):397-406 [FREE Full text] [CrossRef] [Medline]
    48. Docherty AB, Harrison EM, Green CA, Hardwick HE, Pius R, Norman L, ISARIC4C investigators. Features of 20 133 UK patients in hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: prospective observational cohort study. BMJ 2020 May 22;369:m1985 [FREE Full text] [CrossRef] [Medline]
    49. Gray WK, Navaratnam AV, Day J, Wendon J, Briggs TWR. Changes in COVID-19 in-hospital mortality in hospitalised adults in England over the first seven months of the pandemic: An observational study using administrative data. Lancet Reg Health Eur 2021 Jun;5:100104 [FREE Full text] [CrossRef] [Medline]
    50. Grasselli G, Greco M, Zanella A, Albano G, Antonelli M, Bellani G, COVID-19 Lombardy ICU Network. Risk factors associated with mortality among patients with COVID-19 in intensive care units in Lombardy, Italy. JAMA Intern Med 2020 Oct 01;180(10):1345-1355 [FREE Full text] [CrossRef] [Medline]
    51. Zhou Y, Yang Q, Chi J, Dong B, Lv W, Shen L, et al. Comorbidities and the risk of severe or fatal outcomes associated with coronavirus disease 2019: A systematic review and meta-analysis. Int J Infect Dis 2020 Oct;99:47-56 [FREE Full text] [CrossRef] [Medline]
    52. Deng G, Yin M, Chen X, Zeng F. Clinical determinants for fatality of 44,672 patients with COVID-19. Crit Care 2020 Apr 28;24(1):179 [FREE Full text] [CrossRef] [Medline]
    53. Cummings MJ, Baldwin MR, Abrams D, Jacobson SD, Meyer BJ, Balough EM, et al. Epidemiology, clinical course, and outcomes of critically ill adults with COVID-19 in New York City: a prospective cohort study. Lancet 2020 Jun 06;395(10239):1763-1770 [FREE Full text] [CrossRef] [Medline]
    54. Hägg S, Jylhävä J, Wang Y, Xu H, Metzner C, Annetorp M, et al. Age, frailty, and comorbidity as prognostic factors for short-term outcomes in patients with coronavirus disease 2019 in geriatric care. J Am Med Dir Assoc 2020 Nov;21(11):1555-1559.e2 [FREE Full text] [CrossRef] [Medline]
    55. Onder G, Rezza G, Brusaferro S. Case-fatality rate and characteristics of patients dying in relation to COVID-19 in Italy. JAMA 2020 May 12;323(18):1775-1776. [CrossRef] [Medline]
    56. Ramos-Rincon J, Moreno-Perez O, Pinargote-Celorio H, Leon-Ramirez J, Andres M, Reus S, COVID-19 ALC Research Group. Clinical Frailty Score vs Hospital Frailty Risk Score for predicting mortality and other adverse outcome in hospitalised patients with COVID-19: Spanish case series. Int J Clin Pract 2021 Oct;75(10):e14599 [FREE Full text] [CrossRef] [Medline]
    57. Apea VJ, Wan YI, Dhairyawan R, Puthucheary ZA, Pearse RM, Orkin CM, et al. Ethnicity and outcomes in patients hospitalised with COVID-19 infection in East London: an observational cohort study. BMJ Open 2021 Jan 17;11(1):e042140 [FREE Full text] [CrossRef] [Medline]
    58. Williamson EJ, Walker AJ, Bhaskaran K, Bacon S, Bates C, Morton CE, et al. Factors associated with COVID-19-related death using OpenSAFELY. Nature 2020 Aug;584(7821):430-436 [FREE Full text] [CrossRef] [Medline]
    59. Clift AK, Coupland CAC, Keogh RH, Diaz-Ordaz K, Williamson E, Harrison EM, et al. Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ 2020 Oct 20;371:m3731 [FREE Full text] [CrossRef] [Medline]
    60. Nafilyan V, Humberstone B, Mehta N, Diamond I, Coupland C, Lorenzi L, et al. An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: a national validation cohort study in England. Lancet Digit Health 2021 Jul;3(7):e425-e433 [FREE Full text] [CrossRef] [Medline]
    61. Gao Y, Cui Y. Deep transfer learning for reducing health care disparities arising from biomedical data inequality. Nat Commun 2020 Oct 12;11(1):5131 [FREE Full text] [CrossRef] [Medline]
    62. Hospital Episode Statistics (HES) Analysis Guide. NHS Digital.   URL: https:/​/digital.​nhs.uk/​data-and-information/​data-tools-and-services/​data-services/​hospital-episode-statistics/​users-uses-and-access-to-hospital-episode-statistics [accessed 2022-11-30]
    63. Major VJ, Aphinyanaphongs Y. Development, implementation, and prospective validation of a model to predict 60-day end-of-life in hospitalized adults upon admission at three sites. BMC Med Inform Decis Mak 2020 Sep 07;20(1):214 [FREE Full text] [CrossRef] [Medline]

    ‎
    AUROC: area under the receiver operating characteristic
    CCI: Charlson Comorbidity Index
    CFS: Clinical Frailty Scale
    GFS: Global Frailty Scale
    HES: Hospital Episodes Statistics
    HFRS: Hospital Frailty Risk Score
    ICD-10: International Statistical Classification of Disease, 10th Edition
    IMD: index of multiple deprivation
    LSOA: lower layer super output area
    NHS: National Health Service
    SHAP: Shapley additive explanation
    XGBoost: extreme gradient boosting

    Edited by A Mavragani; submitted 29.07.22; peer-reviewed by DK Yon, C Ji; comments to author 02.11.22; revised version received 22.11.22; accepted 24.11.22; published 12.12.22

    Copyright

    ©Johannes Heyl, Flavien Hardy, Katie Tucker, Adrian Hopper, Maria J M Marchã, Annakan V Navaratnam, Tim W R Briggs, Jeremy Yates, Jamie Day, Andrew Wheeler, Sue Eve-Jones, William K Gray. Originally published in the Interactive Journal of Medical Research (https://www.i-jmr.org/), 12.12.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.i-jmr.org/, as well as this copyright and license information must be included.