A Novel Fuzzy DBNet For Medical Image Segmentation Part 1

Abstract: When doctors are fatigued, they often make diagnostic errors. Similarly, pharmacists may also make mistakes in dispensing medication. Therefore, object segmentation plays a vital role in many healthcare-related areas, such as symptom analysis in biomedical imaging and drug classification. However, many traditional deep-learning algorithms use a single view of an image for segmentation or classification. When the image is blurry or incomplete, these algorithms fail to segment the pathological area or the shape of the drugs accurately, which can then affect subsequent treatment plans. Consequently, we propose the Fuzzy DBNet, which combines the dual butterfly network and the fuzzy ASPP in a deep-learning network and processes images from both sides of an object simultaneously. Our experiments used multi-category pill and lung X-ray datasets for training. The average Dice coefficient of our proposed model reached 95.05% in multi-pill segmentation and 97.05% in lung segmentation. The results showed that our proposed model outperformed other state-of-the-art networks in both applications, demonstrating that our model can use multiple views of an image to obtain image segmentation or identification.

Cistanche can act as an anti-fatigue and stamina enhancer, and experimental studies have shown that the decoction of Cistanche tubulosa could effectively protect the liver hepatocytes and endothelial cells damaged in weight-bearing swimming mice, upregulate the expression of NOS3, and promote hepatic glycogen synthesis, thus exerting anti-fatigue efficacy. Phenylethanoid glycoside-rich Cistanche tubulosa extract could significantly reduce the serum creatine kinase, lactate dehydrogenase, and lactate levels, and increase the hemoglobin (HB) and glucose levels in ICR mice, and this could play an anti-fatigue role by decreasing the muscle damage and delaying the lactic acid enrichment for energy storage in mice. Compound Cistanche Tubulosa Tablets significantly prolonged the weight-bearing swimming time, increased the hepatic glycogen reserve, and decreased the serum urea level after exercise in mice, showing its anti-fatigue effect. The decoction of Cistanchis can improve endurance and accelerate the elimination of fatigue in exercising mice, and can also reduce the elevation of serum creatine kinase after load exercise and keep the ultrastructure of skeletal muscle of mice normal after exercise, which indicates that it has the effects of enhancing physical strength and anti-fatigue. Cistanchis also significantly prolonged the survival time of nitrite-poisoned mice and enhanced the tolerance against hypoxia and fatigue.

Click on extreme fatigue

【For more info:george.deng@wecistanche.com / WhatsApp:8613632399501】

1. Introduction

There have been numerous cases where image segmentation in medicine has been implemented primarily because interpreting medical images from CT and MRI scans requires substantial medical knowledge and time. In addition, statistics reveal that medical errors contribute to the deaths of 7000–9000 people annually in the United States [1]. Hence, to address these issues, several technological solutions have been proposed. For instance, the U-Net developed by Ronneberger et al. [2] is used in the segmentation of biomedical images. The DoubleU-Net employed by Debesh Jha et al. [3] is used in procedures such as colonoscopy. It is primarily built upon the U-Net and VGG-19, composed of two encoders and decoders. Similarly, Chin et al. [4] use the Mask R-CNN deep learning algorithm to segment the vocal cords and glottis regions from larynx videos, aiding doctors in diagnosis and treatment.

Among lung image recognition, Jakub et al. pointed out that the results of X-ray images of disease lesions are easily occluded by blood vessels. Though machine learning methods for lung image recognition can help reduce the burden on medical personnel, their accuracy is only 91% [5]. In addition, the deep-learning model for lung disease identification has proven to be superior to traditional machine-learning methods [6,7]. Therefore, we used deep learning for image recognition. In the case of pneumothorax thoracostomy, the doctor needs to look at the chest X-ray image to find the translucent pleural line that overlaps the ribs. Because this image is prone to blurring from tissue overlap, traditional image recognition is not effective [8], especially because air accumulates on the front side of the body rather than at the top, making it difficult for doctors to interpret pneumothorax areas. Because the initial symptoms of many lung lesions are not obvious and lungs have complex pathological characteristics [9], many scholars have proposed deep-learning methods to improve diagnosis accuracy and efficiency [10]. Laura et al. [11] proposed a complex network for lung image recognition, and their experiment showed a high accuracy for texture recognition. Accordingly, a complex network method can extract important features. Alhassan et al. [12] used the ensemble learning method to detect pneumonia in chest X-ray images with an improved accuracy rate of 93%. Based on the ensemble learning characteristics, the method is used to improve model-identification accuracy. Mohammad et al. [13] proposed a deep-learning model to aid early detection of COVID-19, which helped medical staff reduce their workload. For lung image segmentation, Feidao et al. [14] proposed a three-terminal attention mechanism to highlight the target area automatically and improve lung segmentation performance. As a result, an attention gate module is used to improve the training effect of the model. However, these models did not use chest X-ray images fully. There are two modes for taking lung X-ray images: anteroposterior and posteroanterior. Anteroposterior lung X-rays are taken from the chest cavity; posteroanterior lung X-rays are taken from the back. Existing deep-learning models can only input one image at a time for recognition [15]. Therefore, the model may not be able to identify the location of the lesion since only one side of the X-ray image is input [16]. Consequently, we used deep-learning methods for lung image identification.

Among pill image recognition, the majority of existing classification methods rely only on single-sided information, but in some cases, many types of pills cannot be classified. For instance, the same type of pill may have different shapes when viewed from different angles; some pills may have lettering on only one side; and those with similar shapes and colors can also pose a challenge. In such cases, the model may not be able to classify a group of pills accurately. Because of the pandemic, large numbers of people flock to hospitals every day, leading to a substantial increase in pill usage and medical errors [17,18]. To address these problems, Ou et al. [19] proposed a two-stage deep-learning architecture for detecting and subsequently classifying 1000 types of pills. In addition, improving medication knowledge and providing patients with adequate information have become important issues for avoiding medication waste and harmful side effects [20]. However, appearance-based pill identification remains a daunting task for patients. Wang et al. [21]  used the GoogLeNet Inception Network to train deep-learning architecture and image enhancement techniques by focusing on color, shape, and markings, but it can only identify a single type of pill, which is its big flaw. The simultaneous identification of multiple types of pills can meet the needs of the public better. In the field of pill image segmentation, Kwon et al. [22] used Mask R-CNN, and the training datasets they used contained only 27 types of pills, each of which had different shapes and colors. In reality, though, many more types of pills have a similar appearance. To suit real-world needs, 93 types of pills were used to train our model. They had similar shapes and colors and varied mainly in different imprints, making our model more applicable to clinical needs. In pill object detection, Lu et al. [23] proposed many object detection methods. Although these models detected the position of pills, the map is only 87%. Additionally, nprevious scholars have not effectively addressed issues in pill identification, including their random placement and the presence of multiple pills in an image. The rotation angle of the pill is also difficult to determine and standardize for each pill class.

For computer vision, other cases use unmanned aerial vehicles (UAVs) combined with deep learning, Keiller et al. [24] studied plant classification from spatial and spectral perspectives using RGB and spectral UAV images based on 2D-CNN technology.

Current methods for image segmentation mostly use a single image as input. When a single low-quality image is input into the model, issues with imprecise object segmentation and classification arise. In the medical field, due to the complex characteristics of images, Akinobu et al. [25] proposed BtrflyNet to identify bone metastases, which can accept two input images simultaneously. The experimental results indicated that it can enhance the success rate of model training. However, this model is only applicable to bone metastasis images. Based on this paper, we improved the BtrflyNet to propose Fuzzy DBNet, which has achieved excellent results in pill image and lung image recognition.

2. Materials and Methods 

2.1. Datasets 

In this paper, chest X-ray and pill datasets were used. Chest X-ray data were obtained from the NIH chest X-ray dataset [26] containing 112,120 images, from which we extracted 72,324 of both the anteroposterior (AP) and posteroanterior (PA) views of the same patients. We then paired the AP images with their corresponding PA images, resulting in a total of 267,105 AP–PA pairs after augmentation.

The pill dataset comprises 93 categories having a total of 1238 images: round, oval, rectangular, triangular, and of different colors. To capture both sides of each pill, images were taken after the pills were scattered on a transparent board and photographed from directly above and below. The dataset was then divided into 80% for training, 10% for testing, and 10% for validation. To improve the model's accuracy, we applied the AutoAugment scheme [27] to discretize each operation magnitude (M) from [0, 10] for data augmentation, resulting in an increased dataset of 2476 images.

The dataset was the most significant limitation of our model. First, our input images had to be dual-sided, not only general images but also images that penetrated objects, such as X-rays. Second, fixed correspondences of object positions in the images were required. To address this, our team made efforts to align the positions of the pills between their dual-sided images during the collection of the dataset.

It is important to note that the lung X-ray dataset used in this study was provided by the NIH, while the pill dataset was captured by our team's photography.

2.2. Data Pre-Processing

To assist pharmacists in pill classification, 93 types of pills were annotated. We used the VGG image Annotator [28] to label each one with its name in 93 different categories. We labeled the edge of the pill and converted the labeled data into a JSON file as the ground truth of the training data. The pre-processing algorithm can be found in Algorithm A1.

2.3. Fuzzy DBNet

This paper proposes a new deep-learning architecture, Fuzzy Double-Butterfly Network (DBNet), in which two images with complementary properties can be input to perform image segmentation. It is mainly composed of three parts: double-butterfly encoder–decoder architecture, Fuzzy Atrous Spatial Pyramid Pooling (ASPP) block, and an attention gate, as shown in Figure 1.

A trained VGG 19 was used to encode the first butterfly architecture to extract image features for saving training time and preventing overfitting. Between the two butterfly-shaped encoders and decoders, there were two concatenate blocks that connected two sets of Fuzzy ASPP blocks, thereby exchanging the different characteristics of the images. Then, the initial result output by the first butterfly-shaped architecture was multiplied with the original image as shown in the multiply block in Figure 1. This enhanced the specific gravity of the feature to achieve a more precise segmentation. The Fuzzy DBNet algorithm can be found in Algorithm A2.

Fuzzy ASPP combined ASPP with fuzzy theory and was placed between the encoder and decoder of the two butterfly-type networks. Figure 2 shows the structure of Fuzzy ASPP.

In the Fuzzy Pooling part, a bell-shaped membership function was used to complete the operation, and the results of each pooling patch were dynamically adjusted. Algorithm A3 shows the dynamic adjustment algorithm. The main purpose was to reduce the proportion of the feature influenced by uncertainty factors. The Fuzzy ASPP algorithm can be found in Algorithm A4.

In the skip connections, an attention mechanism was used to eliminate noisy and irrelevant responses by using features extracted from coarser feature maps. It effectively reduced noise and unnecessary features in the model and improved its performance and accuracy. Figure 3 shows the structure of the attention gate.

3. Result 

3.1. Experiment Setting

The training validation and testing of the proposed model was done on a computer with an 8-core CPU (Intel Xeon W-3223), 64 GB memory, a GPU (RTX 3090) with 24 GB graphics memory, and 10,496 CUDA cores. Implementation was done using the PyTorch framework. Table 1 shows the hyperparameters used for all experiments.

The total loss function (LDC) combines the averaged Dice Loss (LavgDice) with the Categorical Cross-Entropy Loss (LCCE) and is calculated as follows:

where N is the number of samples and C is the number of classes. Since the output of the network was multi-class images, we calculated the LCCE using Equation (3). Then, we calculated the LDice for each class using Equation (2). It was possible to zero out all pixels in the Ppred that were not active in the Ptrue. For activated pixels, low-confidence predictions were mostly penalized, while higher prediction values obtained higher Dice coefficients. Therefore, the model learned objects of different classes and sizes through LDice and LCCE.

3.2. Performance Evaluation Index

In this study, we used three metrics to evaluate model performance: pixel-wise accuracy, averaged Dice coefficient (Dice), and mean Intersection over Union (me). These indicators are shown in the following formula.

where Xi denotes the ground truth values, and Yi denotes the predicted values. The TP, FP, TN, and FN depict the case numbers of true positives, false positives, true negatives, and false negatives, respectively. Pixel-wise accuracy measured the percentage of correctly identified pixels in the image; the Dice score measured the overlap between the predicted segmentation and ground truth; and mIoU measured the predicted segmentation with the ground truth. These metrics were chosen because they provided a comprehensive view of the model's performance and allowed for meaningful comparisons with other models in the field. Higher values for these metrics indicated better model performance.

3.3. Segmentation of Lung X-ray Images

We conducted experiments to train the proposed method and validated the model by using a validation set. As shown in Figure 4, the training loss of Fuzzy DBNet on the lung X-ray dataset reached convergence at around the 100th epoch and achieved complete convergence at approximately the 300th.

On the lung X-ray dataset, we selected a set of images as examples for model testing. It comprised six images: anterior–posterior and posteroanterior raw images, the corresponding ground truth images, and the segmented results. These images are displayed in Figure 5.

We compared the outputs of Fuzzy DBNet and its ground truth on the testing data using average Dice coefficient, mIoU, and pixel-wise accuracy to measure the performance of the model. The results are shown in Table 2.

We selected two sets of lung X-rays from the segmentation results of our test set as examples. In Figure 6, the segmentation completeness of our model was much better than that for BtrflyNet. In Figure 7, when the original lung images were blurry, the segmentation performance of DoubleU-Net was poor, while our model accurately segmented the lungs.

【For more info:george.deng@wecistanche.com / WhatsApp:8613632399501】