The Role And Challenge Of Artificial Intelligence in New Coronavirus Pneumonia CT Diagnosis

For more information:ali.ma@wecistanche.com

Highlights

This paper novelly considers the scientificity and normativity based on methodology requirements of a clinical trial, by regarding the artificial intelligence (AI) software developing process as computerized tomography (CT) image diagnostic research clinical trials. The paper also raises 4 ways to promote AI diagnostic software helping in actual clinical problems and bringing patients real clinical benefits.

Click to cistanche side effects and Cistanche for immunity 

Abstract

In early 2020, the new coronavirus pneumonia (COVID-19) broke out in China. Many medical-related products have rapidly appeared in the Artificial Intelligence (AI) field, which has played an important role in fighting against the pandemic. This article summarizes the current research and application status of AI in radiology and pandemic control and analyzes the common problems of AI technology in the research of COVID-19 diagnosis. It mainly includes the thoughts on clinical study design, difficulties in research implementation, and challenges in the reliability verification of AI models. In response to the above difficulties, suggestions are proposed for optimizing the scientificity and quality of AI diagnostic research.

Keywords: COVID-19 pandemic, Artificial intelligence, Computerized Tomography, Clinical research

The important role of diversified Artificial intelligence products in anti-epidemic

In early 2020, the outbreak of the new coronavirus pneumonia(Coronavirus, COVID-19) put the implementation of disease prevention and control into a great challenge. For example, how to quickly measure everyone's body temperature in a traffic hub with a large traffic flow; how to rapidly screen out potential effective drugs among massive possibilities; how to screen suspected cases in a large population; how to deal with the shortage of medical professionals and cross-infection during the treatment of diagnosed patients. Artificial Intelligence(Al, as one of the most popular fields in recent years, resolves some of the issues through the application of new AI products, which improves the efficiency of disease prevention, control, and diagnosis.

AI helps monitor and simulate the epidemic development trend by analyzing the trail of people's activity, to give early warning to the potential spreading areas. AI can also analyze the propagation routes, locate the close contacts of the diagnostic patients, and quickly take quarantine and treatment. The infrared thermal imager with AI image recognition technology performs fever detection in public places to locate persons with abnormal body temperature[1]. In the development of new drugs, AI helps screen the most potent antiviral and anti-inflammatory drugs from hundreds of drugs for further development [2]. In the diagnosis and treatment, Deep Neural Network (DNN)model was adopted to recognize computed tomography(CT)graphic data ("CT+ AI)to help doctors quickly make diagnoses.

Challenges faced by "CT+AI" in assisting diagnosis of COVID-19

The application of Artificial Intelligence in radiology has attracted special attention. DNN has been widely used in various medical scanning technologies, for instance, diagnosis of pneumonia on chest digital radiography(DR)[3-5], detection of cancerous pulmonary nodules [6] and tuberculosis [7], detection of fractures, and prediction of bone age through X-ray [8-10].examination and evaluation of breast x-rays [11,12]; detection and diagnosis of pulmonary nodules [13,14], pneumonia [15], liver masses [16], pancreatic cancer[17] and vertebral compression fractures [18]on CT images; outlines the ventricle in the heart magnetic resonance imaging [19]. An ultrasound exam inspection, the AI model can perform the diagnosis and quantitative analysis of cardiac imaging [20.21]. as well as the detection of ultrasound thyroid nodules and the diagnosis of benign and malignant [22,23](Table 1).

In the diagnosis of new coronavirus pneumonia, CT, DR, and ultrasound are commonly used imaging examination techniques, in respective processes and scenarios according to their own characteristics. CT is the priority in the early stage of lesion detection among them. Since the "COVID-19 Pneumonia Diagnosis and Treatment Program(Trial Version 5)" is released, CT imaging diagnosis is also included in the clinical diagnosis standards of the new COVID-19 [24]. However, the diagnosis of a single patient requires manual observation of more than one hundred CT images, With a large number of clinical needs, doctors suffer a large workload with low efficiency. The software-enabled AI technology can solve these clinical bottlenecks. CT inspection has become the preferred imaging solution for the current pneumonia diagnostic AI software due to its high accuracy, unified standards, and deep accumulation of industry data and technology [27. In practical applications, diagnostic models can identify the pneumonia image through a specific algorithm to predict whether it is sick [27-29]. The lung quantitative analysis model can detect the location of the lesion, count the number of lesions, outline the scope of the lesion, calculate the infection rate of the lung lesion area, and cooperate with the follow-up software to manage disease progression and assess prognosis [28]. Aided by software, doctors' workload is reduced, and the speed and accuracy of diagnosis and treatment are improved, however, it is important to emphasize some common problems in the research of Al aided new coronavirus pneumonia CT diagnosis, as follows:

Study design

In the new COVID-19 pneumonia CT diagnosis, the model design should be considered diversely to adapt different diagnosis and treatment phases at the most initial stage of the "CT+AI" diagnostic model development. For example, the control group used for training early screening model should be differentiated from the group chosen in antidiastole: 1)Decisions in the early screening problem prefer high sensitivity to separate suspected cases from the healthy population, therefore, the control case samples trained by the AI model should be CT images of healthy lungs against images of unhealthy lungs. The unhealthy lungs cases are composed of multiple suspicious pneumonia types to ensure the model performance of a low missed diagnosis rate. 2)In the antidiastole scenario, the capability of high specificity is essential to distinguish the new coronavirus pneumonia from those pneumonia caused by other infections. More concretely, given the real clinical circumstances, the ideal control design should compare two groups with similar clinical symptoms or epidemiological history such as fever, cough, CT lung image abnormalities but negative and positive results in nucleic acid test (or other diagnostic gold standards). Such design was intended to consolidate more distinctive radiomic features of new coronavirus pneumonia. In addition, the evaluation metric chosen for assessing the model performance needs to be carefully considered. For example, when evaluating a "CT+AI" diagnostic model, the accuracy ratio is not a proper indicator that can fully assess the model. The imbalance between positive and negative samples of test data would lead to an issue of performance overestimation (e.g. a test set consisting of 96 positive and 4 negative cases could lure a naive model giving a high accuracy up to 96%, even if the model only tell the positive).AI diagnostic software is applied in specific clinical problem for better capability and efficiency. Before implementation, the study objectives need to be clarified and cleared for the specific clinical problems. Then a methodology design in a scientific way should be conducted. A complete research plan should also be formulated. Objects of study, rule of inclusive-exclusive, and metrics of endpoint evaluation should be fully considered. Leading with scientific design, the risk of bias can be minimized, high-quality research evidence can be obtained, and reliable guidance can be provided for clinical application.

Research implementation

The training process of the artificial intelligence DNN model is purely data-driven. It relies on a large number of accurately labeled image samples during the training phase. The larger the amount of data, the better the discriminative performance of the model. The diabetic retinopathy diagnosis system, as an AI medical device, developed by the American technology giant Google has passed the second phase of clinical trials. The training process of the system uses 130 million pictures of 10,000 cases, which is close to the level of medical experts [30]. In contrast, although the designated medical units accumulated new coronavirus CT image data in cooperation with technology manufacturers for software training and development in the epidemic, the total amount is relatively small and the distribution is relatively scattered, plus the scarce resource of experts on CT image labeling in the initial stage, resulting in only a small data set available for model training and hard to guarantee labeling quality. Insufficient training data may cause the DNN model to "remember" information in CT images that is not relevant to the actual diagnosis due to its structural features with strong expressive power, causing overfitting and reducing the ability to judge future data. To achieve a sufficient training effect under the condition of less data volume, it is often necessary to design a more complex internal model structure and model training skills, which also increases the difficulty of implementation.

Consideration on the exceptionalism new of COVID-19

Based on clinical experience, the chest CT image of new coronavirus pneumonia has the characteristics of a "similar image but different disease". It is difficult to distinguish by human eyes which increases the difficulty in antidiastole. The difficulty is more prominent if the training sample size is small. In addition, as one of the diagnosis criteria of the new COVID-19, the results of the nucleic acid kit show high specificity but low sensitivity. If it is used for labeling training samples as the gold standard, the actual positive samples would tend to be marked as negative wrongly. Training with the mistakenly labeled data will directly reduce the model performance.

Verification of the reliability of AI diagnosis

It is not reliable to evaluate the diagnosis effect only by the internal data test after the implementation of the AI diagnosis model. The confounder underneath the data caused by different CT equipment models, geographic regions, actual clinical environment, virus variation, and other factors, could affect the diagnostic result of the model. The model can barely have the capability to deal with these confounding factors with limited internal data tests. When the AI diagnostic software is put into clinical use without full validation, it can only be iterated through offline training and optimization since it cannot improve the diagnosis and treatment level by continuing to learn new cases like the human doctors. Compared with the misdiagnosis of a human doctor, the potential iatrogenic risks caused by the limitations of computer software will be more prominent. According to the new version of the "Medical Device Classification Catalog"(CFDA 2017 No.143), AI diagnostic software, as a medical device, should proceed with a systematic audit and extensive simulation, even prospective clinical trials, to fully verify the accuracy and reliability of the clinical diagnosis. The Artificial Intelligence Group of the National Institute of Food and Drug Control (NIFDC)established a security system of AI medical devices and stipulated the test method based on standard data set verification [30]. At present, there is no standard database for the new COVID-19 disease entity, so it is difficult to verify the high repeatability accuracy of the existing AI software in most medical scan diagnoses in the real world.

Improve the quality and scientific standardization of the "CT+AI" diagnostic model

The application of Artificial Intelligence in the medical industry is still at an early stage in China, while it has received strong attention and rapid development in recent years, which has even been included in the national strategic blueprint [30]. Many artificial intelligence products have shown promising prospects in the medical field. In the next stage, for commercialization, the focus of the industry would be on design research scientifically and standardize implementation and verification process; develop artificial intelligence in the medical industry healthily, standardize the management of Al-assisted diagnostic products according to New Medical Apparatus specification. Although many AI products that emerged during the epidemic have brought convenience to doctors in disease diagnosis and treatment, the overall scientific rigor and quality reliability still need further improvement and perfection.

Improve the study design of AI technology in clinical problems

Presently, most research on the new COVID-19 CT-assisted diagnosis use retrospective case-control methods. One of the advantages is to quickly obtain various information from limited research subjects, while there is the risk of bias. Samples in case-control studies should be taken from total diagnostic cases or randomly selected from them, however, due to limited sources, the samples cannot represent all patients. Special control sampling can be used to reduce the bias and enhance the level of evidence according to the actual situation, for example, selecting cases from the same medical institution, using two or more control groups, and matching variables. In the assessment of diagnostic model performance, the true positive rate and false positive rate chart(receiver operating characteristic curve (ROC curve)) can be used to compare AI algorithm interpretation with the physician evaluation, or the area under the curve (AUC) as the reference of model performance. Even so, it is still difficult to fully explain the clinical benefits relying solely on comparison or indicators, other clinical decision-making factors, in reality, need to be considered as well for comprehensive evaluation.

Improving the accuracy of the label, expanding the number and dimension of training samples during the model implementation process

In the primary stage, the models trained by a small count of CT image samples have already shown a good trend. Over time, more cases would improve better basic support in model training. The more images learned, the higher potential conquering the difficulty of "same images, different cases" in the COVID-19 diagnosis. In response to this situation, based on CT image data, the model can also expand the domain of information by combining the patient's clinical signals, epidemiology, laboratory examination, and other data to comprehensively assess the disease and improve the diagnosis accuracy. The improvement of sample labeling accuracy can be achieved by optimization of the gold standard strategy. Apply multiple intervals of nucleic acid test to compensate for the possibility of a single false-negative result, or cross-verify the result referring to other diagnostic kits, such as IgM/lgG antibody detection.

Establish a new coronavirus pneumonia CT standard testing database

After the implementation and internal verification of AI modeling, external reliability verification is required through a standard database test system. Concerning the established standard database system of fundus diseases and pulmonary nodules, the disease entities come from different medical institutions across the country, including underdeveloped areas. The data contains a variety of specifications and is compatible with devices from different models and parameters. Doctors participating in the test data labeling have AI medical research experience and have been well trained. A special research team will be formed by doctors with high accuracy, stability, plus enriched clinical experience. The standard test data set erases company and machine traces and strictly controls data bias to ensure fair and objective performance assessment in a closed environment [30]. To summarize, it is difficult to establish a new COVID-19 CT standard database by relying on individual institutions solely. The nation should provide corresponding support during the special period of the epidemic. For example, quickly establish a new COVID-19 special artificial intelligence group to guide and coordinate the cooperation and resource sharing of all parties across the country, and jointly formulate a special disease test database and other verification standards.

Standardize the data management of artificial intelligence medical research

On July 3, 2019, the Centre for Medical Device Evaluation released "Key Points of Deep Learning Assisted Decision-Making Medical Device Software Review"(key points). It provides the technical guidance of AI medical products registered as third-type medical devices and eliminates the policy bottleneck before the product launch. However, there are no legal restrictions on ethics and data security. Artificial intelligence on medical research must conform to ethics and protect the security and privacy of individual data. If national or regional laws and regulations on patient privacy protection can be formulated, while a standardized data management platform for research can be established, a research project then can be effectively reviewed to discover potential risks in the design and implementation process promptly. Guidance of data security will be implemented to avoid the risk of hindering and destroy of human health.

References:

1 zc [Internet]. How to use AI to detect and fight against new coronavirus [cited 2020 March 20].

2. Huang Q[Internet]. Academician Li Lanjuan's team released: these two drugs can effectively inhibit the new coronavirus [cited 2020 March 20].

3. Wang X S, Peng Y F, Lu L, et al. ChestX-Ray8: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 3462-3471.

4. Li Z, Wang C, Han M, et al. Thoracic Disease Identification and Localization with Limited Supervision. arXiv e-prints, 2017. arXiv:1711.06373.

5. Singh R, Kalra MK, Nitiwarangkul C, et al. Deep learning in chest radiography: Detection of findings and presence of change. PLoS One. 2018;13(10):e0204155. Published 2018 Oct 4.

6. Nam J G, Park S, Hwang E J, et al. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology, 2019. 

7. Lakhani P, Sundaram B. Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks. Radiology. 2017;284(2):574–582. 

8. Gale W, Oakden-Rayner L, Carneiro G, et al. Detecting hip fractures with radiologist-level performance using deep neural networks. arXiv e-prints, 2017. arXiv:1711.06504. 

9. Rajpurkar P, Irvin J, Bagul A, et al. MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs. arXiv e-prints, 2017. arXiv:1712.06957. 

10. Ridley EL[Internet]. Deep learning shows promise for bone age assessment [cited 2017 November 15]. 

11. Yee KM[Internet]. AI algorithm matches radiologists in breast screening exams [cited 2017 December 13]. 

12. Lehman CD, Yala A, Schuster T, et al. Mammographic Breast Density Assessment Using Deep Learning: Clinical Implementation. Radiology, 2019. 290(1): p. 52-58. 

13. Ridley EL. The deep-learning algorithm can stratify lung nodule risk [cited 2017 November 26]. 

14. Ali I, Hart GR, Gunabushanam G, et al. Lung Nodule Detection via Deep Reinforcement Learning. Front Oncol. 2018;8:108. Published 2018 Apr 16. doi:10.3389/fonc.2018.00108. 

15. Walsh SLF, Calandriello L, Silva M, et al. Deep learning for classifying fibrotic lung disease on high-resolution computed tomography: a case-cohort study. Lancet Respir Med. 2018;6(11):837–845. 

16. Yasaka K, Akai H, Abe O, et al. Deep Learning with Convolutional Neural Network for Differentiation of Liver Masses at Dynamic Contrast-enhanced CT: A Preliminary Study. Radiology. 2018;286(3):887–896. 

17. Liu F Z, Xie L X, Xia YD, et al. Joint Shape Representation and Classification for Detecting PDAC. arXiv e-prints, 2018. arXiv:1804.10684. 

18. Bar A, Wolf L, Amitai OB, et al. Compression fractures detection on CT. arXiv e-prints, 2017. arXiv:1706.01671. 

19. Lieman-Sifry J, Le M, Lau F, et al. FastVentricle: Cardiac Segmentation with ENet. arXiv e-prints, 2017. arXiv:1704.04296. 

20. Madani A, Arnaout R, Mofrad M, et al. Fast and accurate view classification of echocardiograms using deep learning. npj Digital Med 1, 6 (2018). 

21. Zhang J, Gajjala S, Agrawal P, et al. Fully Automated Echocardiogram Interpretation in Clinical Practice. Circulation, 2018. 138(16): p. 1623-1635. 

22. Li X C, Zhang S, Zhang Q, et al. Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study. Lancet Oncol. 2019;20(2):193–201. doi:10.1016/S1470-2045(18)30762-9. 

23. Li HL, Weng J, Shi Y J, et al. An improved deep learning approach for detection of thyroid papillary cancer in ultrasound images. Sci Rep. 2018;8(1):6600. Published 2018 Apr 26. 

24. Office of National Health Commission [Internet]. Diagnosis and treatment of novel coronavirus pneumonia (trial version fifth) [cited 2020 February 04].

25. Alibaba DAMO Academy [Internet]. What do you think of the new coronavirus pneumonia CT imaging AI diagnosis technology developed by the Ali Damo Academy, with an average recognition rate of 96% in less than 20 seconds? [ 2020-02-19]. 

26. United imaging. Conquering the "epidemic" breakthrough - uAI new coronavirus pneumonia intelligent assistant analysis system officially launched [cited 2020 February 10]

27. Gozes O, Frid-Adar M, Greenspan H, et al. Rapid AI Development Cycle for the Coronavirus (COVID-19) Pandemic: Initial Results for Automated Detection & Patient Monitoring using Deep Learning CT Image Analysis. ArXiv abs/2003.05037 (2020): n. pag. 

28. Huang L, Han R, Ai T, et al. Serial Quantitative Chest CT Assessment of COVID-19: Deep-Learning Approach. Radiology: Cardiothoracic Imaging 2020 2:2. 

29. Li L, Qin L X, Xu Z G, et al. Artificial Intelligence Distinguishes COVID-19 from Community-Acquired Pneumonia on Chest CT. Radiology. 2020;200905. 

30. Jin Y H, Qiu M J. China Artificial Intelligence Medical White Paper. Shanghai: Shanghai Jiao Tong University Artificial Intelligence Institute, 2019. (Chinese)