Microsoft Word - Deep Learning Vs Traditional Models_Abdel Hai_Final.Part 3

In this study, extensive experiments were conducted to determine how many prior encounters are optimal to predict readmission. 

As the pace of life accelerates, we are exposed to a large amount of information every day, and we need to maintain a certain amount of memory to better absorb and utilize this information. Many people may feel that their memory is not strong enough, but through extensive experimental research, we can find that memory can be trained and improved.

The first related experiment looked at the effects of yoga on improving memory. The experiment, conducted at the University of California, San Francisco, showed that yoga practice increased gray matter in the brain and improved students' short- and long-term memory. This is because yoga has a very good healthcare effect on people's physical and mental health, can relieve stress and anxiety, and thereby enhance people's memory.

The second experiment explored the relationship between sleep and memory. Researchers have found that getting enough sleep is good for people's memory. Sleep can promote the transmission of information between brain cells, help consolidate memories, and improve the long-term storage capacity of memories.

The third experiment is to use mathematical games to improve students' calculation ability and memory. In the experiment, conducted at the University of Hong Kong, researchers asked students to perform simple mathematical activities to stimulate their brain activity. The results showed that after receiving the training, the students' calculation ability and memory improved. This shows that with appropriate training and stimulation, brain function can be comprehensively improved.

Through the above experiments, we know that memory can be trained and improved, and you don't have to worry too much about your memory not being good enough. We can effectively improve our memory and better cope with the challenges of modern life by performing appropriate exercises, such as yoga exercises, adequate sleep, and various games. It can be seen that we need to improve memory, and Cistanche deserticola can significantly improve memory because Cistanche deserticola is a traditional Chinese medicinal material that has many unique effects, one of which is to improve memory. The efficacy of minced meat comes from the various active ingredients it contains, including acid, polysaccharides, flavonoids, etc. These ingredients can promote brain health in various ways.

Click know 10 ways to improve memory

We conducted experiments by considering � encounters within the prior 2 years, where � ∈ {1, 2, 4, 8, 15, 30, 60, 80, 100}. The average number of encounters per patient in this period was 21, and the 90th percentile was 56. 

The variation in encounter number resulted in a non-unified length of feature vectors. Thus, in an experiment that considers up to 60 encounters, feature vectors lacking data were padded with 0s to ensure that feature vectors for all patients represent 60 encounters. 

This study hypothesized that DL models outperform traditional models on a large benchmark, hence, a comparative analysis with a variety of evaluation metrics was performed to evaluate and compare the DL algorithms to the baseline traditional models. 

Moreover, to examine the importance of domain knowledge, we trained and tested the models on data with all laboratory studies included in the EHR dataset and compared with models trained and tested with a subset of laboratory studies based on prior papers reporting association with readmission (serum albumin, anion gap, arterial pH, bilirubin, blood urea nitrogen, carbon dioxide, serum creatinine, blood glucose, hematocrit, lactate, PaCO2, PaO2, serum sodium, troponin-I, venous pH, and white blood cell count). 11, 34 Using only a subset of laboratory studies may be beneficial by reducing dimensionality.

Patients were sorted randomly into 3 nonoverlapping subsets, where 70% were used for training, 10% for validation, and 20% for testing. We employed cross-validation techniques to find the hyperparameters that yield the best performances. 

For LSTM and GRU, we varied the number of neurons, dropout, batch size, and the number of epochs using a grid search. Following the literature, in conducted experiments dropout percentage varied from 0 to 50, and the number of neurons varied from 32 to 512. 

We selected a dropout of 0.1, 128 neurons, a batch size of 512, and 16 epochs for LSTM, and 12 epochs for GRU since bidirectional GRU converges faster than 1-way LSTM. Sigmoid activation function and Adam optimizer were used. Traditional models were fine-tuned as well and the hyperparameters that yielded best performances were chosen.

Performance metrics and analysis

The performance of the methods used in our study was evaluated by five common metrics: Area Under the Receiver Operating Characteristic Curve (AUROC), Recall (also known as Sensitivity), Specificity, F1-score, and Accuracy. The formal definitions of these evaluation metrics are common and can be easily found. 34

Statistical significance analysis was performed to evaluate the stability and significance of the proposed model's performance. We randomly selected different patients for training and testing and repeated the random selection 10 times to generate mean performance measures and 95% confidence intervals. 

LSTM was compared to the best-performing traditional model (RF) by t-test. A p-value <0.05 was considered statistically significant. The Temple University Institutional Review Board approved the protocol.

Results

A total of 36,641 patients with 2,836,569 encounters were analyzed. There were 9,130 patients with at least one readmission and 27,511 without a readmission. The influence of the number of encounters within the prior 2 years was evaluated for five prediction models where � encounters were considered for each model, and experiments were repeated for � ∈ {1, 2, 4, 8, 15, 30, 60, 80, 100}. 

Figure 2 presents the AUROC of the proposed model, LSTM, versus traditional models across various numbers of encounters. Bidirectional GRU was also performed but omitted because it achieved an identical AUROC to LSTM. LSTM outperformed traditional models on a large benchmark across all experiments with different numbers of encounters. 

On average, the LSTM models yielded an increase in AUROC of 0.06 when compared to the best-performing traditional models, RF. Experiments show that predicting readmission based on a single prior encounter is not sufficient and yielded much lower performance (0.7 using the DL models and 0.68 using the best-performing traditional model). 

DL models reached a plateau when trained using data from 30 encounters with minimal improvement thereafter. The DL algorithm yielded a 0.07 increase in AUROC versus the best-performing traditional model RF when using the optimal number of encounters, 80.

Table 1 shows the performance of LSTM and traditional models using all laboratory tests from up to 80 of the most recent encounters in the prior 2 years. Overall, the confidence intervals were very small (<0.02), indicating a high degree of precision around the means. 

The proposed method, LSTM, obtained an average AUROC of 0.79 with a 95% CI of 0.001. The p-value obtained by comparing the LSTM AUROC to the second-best-performing model (RF) was <0.0001, hence, LSTM performance was significantly greater than the traditional models.

LSTM models achieved a Recall/Sensitivity of 0.81, indicating that performance was fairly strong at predicting true positives, (i.e., correctly classifying patients with readmissions). 

All models used in our study achieved a very good specificity, (i.e., the true negative rate). Thus, the trained models performed well at predicting patients who are not likely to be readmitted. LSTM achieved an F1-score of 0.80, indicating a very good ability to distinguish between patients who will be readmitted or not.

To determine whether domain knowledge about laboratory studies is helpful, we conducted two different experiments where we trained and tested the model based on a subset of 16 unique laboratory studies selected by domain knowledge versus using all 981 unique laboratory studies included in the data. 

One Hot encoding techniques were utilized and modified to associate the laboratory result with each laboratory code. A long unique array of laboratory codes ������_���_����� was created. For each encounter, an array of zeros � of the same length as ������_���_����� was created. � consisted of the result at the same index of each laboratory test in ������_���_�������, to associate the result to a given laboratory code. An encounter without laboratory results would have an � of zeros, indicating that no laboratory test was conducted for a given encounter. 

Since most encounters contained <3 laboratory codes, this resulted in a sparse array. SVD was therefore utilized to learn an embedding of a sparse feature vector and reduce dimensionality. The Receiver Operating Characteristic (ROC) Curves of the LSTM models based on all laboratory studies or selected laboratory studies were identical (0.79, Figure 3).

Discussion

In this retrospective cohort of 36,563 patients with diabetes, DL models outperformed RF, MLP, AdaBoost, and LR models in predicting unplanned, all-cause 30-day readmission. The optimal LSTM model yielded an AUROC of 0.79 and an accuracy of 0.81, indicating very good performance. Experiments designed to reveal the relationship between the number of prior encounters and model performance show that the AUROC of the LSTM models increased as the encounter number increased and plateaued at 30 encounters. Performance of the traditional models increased to a lesser extent up to prior encounter numbers of 15 or 30, then either plateaued (RF) or declined (MLP, AdaBoost, LR) as encounter number increased. Finally, an LSTM model that included a set of 16 laboratory tests selected by domain knowledge yielded equivalent performance to an LSTM model that included all available laboratory tests.

In our study, the DL models performed better than the traditional models. We are aware of 4 studies that compared DL models to traditional models for predicting the readmission risk of patients with diabetes. Two of these studies demonstrated a clear advantage of DL approaches over traditional ML models,23,24 while two studies found marginal benefit with DL approaches.25,27 Performance of these DL models was variable with AUROC 0.61-0.97 and accuracy of 0.69-0.95, none of which exceeded that of the best traditional ML models, which reported AUROC as high as 0.99 and accuracy of 0.99.23,27 Comparisons of model performance across all these studies, however, is limited by the lack of standardized reporting of performance characteristics and variable approaches to testing. Our study considered with the prior studies that directly compared DL to traditional ML models, suggests that DL approaches usually yield better performance in this population.

We are unaware of other papers that have explored the relationship between the number of prior encounters and readmission risk model performance in patients with diabetes. In related work, however, one paper examined how the performance of models for predicting readmission risk in morbidly obese patients varied as the number of hospitalizations increased from a minimum of 2 up to 5.35 AUROC increased from 2 to 3 hospitalizations then plateaued. In another broadly related study, we found that the performance of LR models for predicting the readmission risk of patients with diabetes tended to increase as the sample size increased from 2,000 up to 6,000, then plateaued.36 This body of research suggests that experimentation across a range of encounter numbers and sample sizes may reveal thresholds that could optimize data analysis, balancing information quantity with dimensionality.

We are also unaware of other studies that have compared readmission risk models using laboratory data selected by domain knowledge with all laboratory data available in patients with diabetes. There is a tradeoff between including all laboratory data, which results in higher dimensionality and more computationally expensive models, and involving a domain expert to select a subset of laboratory data, which can be costly and less feasible. Like the number of prior encounters beyond which model performance did not improve, the finding that the performance of the model with the laboratory data subset was equivalent to the model with all laboratory data suggests that there is a similar plateau for this domain. Whether or not this phenomenon generalizes to other patient populations should be investigated.

The presented LSTM models, which we are calling eDERRITM, are an extension of our prior models, the DERRITM and DERRIplus.8,11 In terms of AUROC, the eDERRITM model performed better than the DERRITM but worse than the DERRIplus. Unfortunately, the performance of the 3 models cannot be directly compared in the current study because the dataset does not include zip code, employment status, or payer information. Unlike the DERRITM and DERRIplus, the eDERRITM models are developed with generally available EHR data such as demographics, vital signs, diagnostic and procedure codes, medications, laboratory tests, and administrative data as defined by the PCORnet CDM. 29 The CDM standardizes the abstraction of EHR data, enhancing the generalizability and scalability of models utilizing it. We plan to translate the eDERRITM into an application embedded in an EHR system that will automatically generate readmission risk predictions for hospitalized patients with diabetes.

In addition to the generalizability of the CDM-based dataset, the current study has other notable strengths. The dataset is sampled from patients with a hospitalization between 7/1/2010 and 12/31/2020, which is much more recent than the datasets used for other currently published readmission risk models in diabetes patients. Also in contrast to the most used dataset, which only included hospital encounters with an associated diagnosis of diabetes and a length of stay less than 15 days, the current dataset included all encounter types regardless of the associated diagnosis, capturing both inpatient and outpatient data. Lastly, the sample size of 36,563 patients with 2,836,569 encounters provided ample data to develop DL models and conduct experiments with up to 100 prior encounters.

There are some limitations worth acknowledging. The data were sampled from a single urban, academic health system. Therefore, the generalizability of the models to other populations is unknown and requires testing. The lack of both patient and hospital zip codes precludes estimating the distance between a patient's home zip code and the hospital, which is known to be associated with readmission risk.8,11 Lastly, readmissions to other hospitals were not captured.

Conclusion

An LSTM model with very good performance predicting unplanned, all-cause 30-day readmission among patients with diabetes was developed and internally tested. LSTM models outperform traditional models at predicting readmission in this population. LSTM model performance initially increases as the number of prior encounters increases and then plateaus. Carefully selected laboratory features can yield predictive models with performance equal to that of models based on all available laboratory studies. Additional study is needed to externally validate the model.

Acknowledgments

This research was supported by the National Health Institute (NIH) under grant number R01DK122073.

References

1. Benbassat J, Taragin M. Hospital readmissions as a measure of quality of health care: Advantages and limitations. Archives of Internal Medicine. 2000;160:1074-1081. 

2. Rubin DJ. Hospital readmission of patients with diabetes. Current Diabetes Reports. 2015;15:1-9. 

3. Ostling S, Wyckoff J, Ciarkowski SL, Pai C-W, Choe HM, Bahl V, et al. The relationship between diabetes mellitus and 30-day readmission rates. Clinical Diabetes and Endocrinology. 2017;3:3. 

4. Enomoto LM, Shrestha DP, Rosenthal MB, Hollenbeak CS, Gabbay RA. Risk factors associated with 30-day readmission and length of stay in patients with type 2 diabetes. J Diabetes Complications. 2017;31:122- 127. 

5. AHRQ, Healthcare cost and utilization project (hcup) national inpatient sample (nis). , 2018. 

6. ADA. Economic costs of diabetes in the U.S. In 2017. Diabetes Care. 2018;41:917-928. 

7. Rubin DJ, Shah AA. Predicting and preventing acute care re-utilization by patients with diabetes. Current Diabetes Reports. 2021;21. 

8. Rubin DJ, Handorf EA, Golden SH, Nelson DB, McDonnell ME, Zhao H. Development and validation of a novel tool to predict hospital readmission risk among patients with diabetes. Endocr Pract. 2016;22:1204- 1215. 

9. Rubin DJ, Recco D, Turchin A, Zhao H, Golden SH. External validation of the diabetes early re-admission risk indicator (Terri ()). Endocr Pract. 2018;24:527-541. 10. Alamer AA, Patanwala AE, Aldayyen AM, Fazel MT. Validation and comparison of two 30-day readmission prediction models in patients with diabetes. Endocr Pract. 2019;25:1151-1157.

For more information:1950477648nn@gmail.com