Research On Audio Recognition Based On The Deep Neural Network in Music Teaching

Solfeggio is an important basic course for music majors, and audio recognition training is one of the important links. With the improvement of computer performance, audio recognition has been widely used in smart wearable devices. In recent years, the development of deep learning has accelerated the research process of audio recognition. However, there is a lot of sound interference in a music teaching environment, which leads to the performance of the audio class…er, that cannot meet the actual demand. To solve this problem, an improved audio recognition system based on YOLO-v4 is proposed, which mainly improves the network structure. 

Sight-singing and ear training are inseparable from memory. In the process of learning music, sight-singing and ear training is a very important skill. The purpose of sight-singing ear training is to allow students to practice their musical skills and memory by listening and singing specified notes.

The main content of sight-singing ear training includes pitch, rhythm, voice, melody, harmony, etc. Through sight-singing ear training, we can continuously exercise our ears, allowing us to more accurately identify various notes and rhythms, and quickly convert the notes we hear into symbols in the music map, thereby improving Our ability to remember.

Sight-singing and ear training can also help us better understand and master the laws of music. When we use our ears to feel and understand music directly, we can understand the rhythm and melody of the music more deeply, thereby improving our memory ability faster.

In addition, sight-singing and ear training can also help us develop a good sense of music and strong musical emotions. Through training, our understanding and feelings of music will become more and more profound, thereby improving our love and appreciation of music.

Memory is a very important factor in the process of learning music. Good memory skills are one of the necessary conditions for learning any instrument and composing music. Through sight-singing and ear training, we can improve our memory and better master musical techniques and skills. When faced with complex repertoire, we can remember notes and rhythms faster, helping us to perform musical works better.

In short, sight-singing ear training and memory are inseparable, and the two complement each other. Through continuous sight-singing and ear-training training, we can improve our musical skills and memory abilities to better master music. It can be seen that we need to improve memory, and Cistanche deserticola can significantly improve memory because Cistanche deserticola is a traditional Chinese medicinal material that has many unique effects, one of which is to improve memory. The efficacy of minced meat comes from the various active ingredients it contains, including acid, polysaccharides, flavonoids, etc. These ingredients can promote brain health in various ways.

Click know ways to improve brain function

First, the Mel frequency cepstrum number is used to process the original audio and extract the corresponding features. en, try to apply the YOLO-v4 model in the …eld of deep learning to the …eld of audio recognition and improve it by combining it with the spatial pyramid pool module to strengthen the generalization ability of data in different audio formats. Second, the stacking method in ensemble learning is used to fuse the independent submodels of two different channels. Experimental results show that compared with other deep learning technologies, the improved YOLO-v4 model can improve the performance of audio recognition, and it has better performance in processing data of different audio formats, which shows better generalization ability.

1. Introduction

Music is an abstract art form with sound as its means of expression. In the process of music teaching, solfeggio can strengthen students’ musical memory ability, enable students to accurately identify music works, and thus obtain better “musical perception.” As an important link in solfeggio, audio recognition training is very difficult for junior students. is because students need to master all kinds of clefs, distinguish the length and duration represented by different notes, and the pitch difference between different notes.

Audio signal analysis based on embedded intelligent devices has attracted more and more researchers’ attention [1–7]. Smart wearable devices with audio recognition functions can help students solve the above problems and realize music teaching assistance. e task of audio recognition needs to preprocess the collected audio signals …rest, extract valuable features for distinguishing music scores from them, and …nally classify them according to these features. Classi…cation is a very important method of data mining [8–10]. Classi…cation refers to generating a classic- …cation function according to certain rules based on training set data. is function can map the data of the test set to one of the given categories, thus realizing the category prediction of unknown data. At present, common classic…ers include decision trees, logistic regression, support vector machine (SVM), Naive Bayes, k-nearest neighbor algorithm (KNN), BP neural network, and deep learning [11–13].

e previous machine learning methods often need to manually extract the features that can represent the original data as the input of the class…er. However, deep learning can automatically extract the high-dimensional features of samples (without manual feature extraction), as long as the input data covers the original data's information as much as possible, which is suitable for large-scale data. *e deep learning method can realize specific audio recognition tasks with the help of a large amount of audio data collected by intelligent devices. *e convolutional neural network (CNN), as a kind of deep learning architecture, is widely used in image classification, speech recognition, natural language processing, and other fields because of its superior performance in local feature learning [14]. Different from other neural network models (such as Boltzmann machine and recurrent neural network), the CNN is characterized in that core operation is convolution operation. *e YOLO network draws lessons from the CNN classification network structure and shows good advantages in the field of image recognition, which has attracted the attention of many researchers.

*Therefore, this study tries to apply the YOLO-v4 model to the field of audio recognition and improves its network structure. In addition, the stacking method in ensemble learning is used to fuse two independent submodels of different channels, and the classification performance of the fused system is further improved compared with the single submodel.

2. Related Works

Nowadays, with the emergence of a large number of smart devices, excellent computer performance and the development of deep learning technology have jointly promoted the research process in the audio field. Combined with the main research contents of this study, the current research status will be introduced from two aspects: convolutional neural network and audio recognition.

*e convolutional neural network structure originated from a study by Yann LeCun in 1998 called the Le Net-5 artificial neural network. *e convolutional neural network, like other neural networks, can be trained by the backpropagation algorithm [15]. In 2012, Alex Krizhevsky and others adopted CNN technology for the first time in complex computer vision tasks. By using 3 fully connected layers, 5 convolution layers, and a Softmax classifier, a convolutional neural network with 8 layers is constructed, which is named AlexNet. AlexNet uses the ReLU activation function, and at the same time, it also uses regularization (dropout) to prevent overfitting. In 2014, the Google computer vision team put forward the GoogLeNet network [16], with a network depth of 22 layers, which contains a new structure, incident. It integrates the features of different depths and the same scale, and the detection accuracy is improved. Based on the GoogLeNet network, YOLO and SSD algorithms appeared. Both methods are based on a single end-to-end network, which can complete the input from the original image to the output of the object position and category.

In the aspect of audio recognition, Yang and Zhao [17] proposed an acoustic scene classification method based on the support vector machine (SVM), which enhanced the sound texture to improve the classification accuracy. Greco et al. [18] proposed a voice recognition system based on the heuristic deep learning method. Demir et al. [19] proposed a new pyramid cascade CNN method for environmental sound classification. Zhu et al. [20] proposed an improved YOLO-v4 algorithm for sound imaging instruments, which effectively improved the accuracy of acoustic phase cloud image detection. *The above methods all show excellent performance in dealing with audio recognition tasks in a single acoustic scene, but there are many sound disturbances in the music teaching environment, and it is necessary to deal with a variety of different audio format data.

*Therefore, this study proposes an audio recognition system based on the improved YOLO-v4 network model. *e main innovations and contributions include the following: (1) try to apply YOLO-v4 network architecture, which is excellent in the field of deep learning, to the field of audio recognition, and improve it by combining the spatial pyramid pool module. *e improved YOLO-v4 network architecture effectively utilizes the spatial information in audio files, thus strengthening the generalization ability of data in different audio formats. (2) *e stacking method in ensemble learning is used to fuse two independent submodels of different channels, and the classification performance of the fused system is improved.

3. Extraction and Processing of Audio Features

Extracting the best parameter representation of the audio signal is one of the important tasks to produce better recognition performance. *e Feature extraction in this stage is very important for the classifier classification in the next stage because it will directly affect the classification efficiency.

In the classification task, especially the audio classification task, the Mel frequency cepstrum coefficient (MFCC) which describes the spectral shape has a long history. Although the MFCC extraction process will cause lossy compression of data, its classification and recognition effect are quite available even when the data rate is very low. In addition, compared with other classification features, MFCC is widely used because it is more in line with the auditory frequency response curve of human ears.

*The reason why human beings can judge different environments in complex sound environments lies in the credit of the cochlea. *e cochlea can be seen as a filter bank to help people filter 20-20 kHz audio. *The problem is that the sensitivity of the cochlea to frequencies in the auditory range is not linear, but there is a mapping relationship. MFCC can simulate the frequency response of the human ear. MFCC feature extraction consists of seven steps, and the whole process is shown in Figure 1.

Common audio signals have the phenomenon that the low-frequency energy is large, but the high-frequency energy is small. If it is transmitted directly, it will lead to a high signal-to-noise ratio at low frequencies and an insufficient signal-to-noise ratio at high frequencies. To make up for this loss of audio signal during transmission, preemphasis is introduced to compensate for the input signal so that the high-frequency characteristics of the audio signal can be highlighted. Preemphasis is usually achieved using a high-pass filter [21–23].

Framing divides audio samples obtained from analog-to-digital conversion (ADC) into small frames with a length in the range of 20–40 milliseconds. After preemphasis and framing are completed, it is necessary to add a Hamming window to each frame. Windowing is to control the amount of data processing, and only the data in the window are processed at a time. *e frequency range in the fast Fourier transform spectrum is very wide, which leads to the speech signal not following the linear scale [24–26]. *therefore, it is necessary to pass the Mel scale filter bank as shown in Figure 2.

Figure 2 shows a set of triangular filters, which are used to calculate the weighted sum of the spectral components of the filters so that the processed output approximates the Mel scale. *e amplitude-frequency response of each filter is triangular. *e Mel spectrum of a given frequency f is calculated as follows:

13 first-order differential features represent the changes between frames of cepstrum in MFCC features, while 39 second-order differential features represent the changes between frames in first-order differential features. *e first-order difference is calculated as follows:

4. SPP-YOLO-v4 Network Structure

4.1. Spatial Pyramid Pool (SPP) Module. SPP can avoid information distortion caused by scaling, stretching, clipping, and other operations and provide output that is not affected by the input size, which cannot be achieved by sliding window pooling technology [27]. Second, SPP can pool with multiple scales, while sliding window pooling only uses one window scale. *The basic structure of the SPP module is shown in Figure 3. It can be seen that because the input size is flexible, SPP can combine the features of data in different audio formats. *e dimension of the transformed feature vector is the same as that of the fully connected layer while alleviating the generalization problem.

4.2. SPP-YOLO-v4. YOLO-v4 is a high-precision real-time single-stage detection algorithm integrating YOLO-v1, YOLO-v2, and YOLO-v3. YOLO-v4 constructs the CSP cross-stage partial network (CSPNet) in the residual module, in which the feature layer is the input and the feature information of the higher layer is the output. *shows that the learning objectives of YOLO-v4 in the ResNet module are different between output and input. *Therefore, residual learning is realized, and the model parameters are reduced, so the feature learning ability is enhanced. Considering the application environment of music teaching, some changes are made based on the original network, and the final network structure is shown in Figure 4.

First, the feature layer is convolved three times, and then, the input feature layer is maximally pooled by using the maximum pooled cores of different sizes. After convolution and upsampling, different feature layers are connected in series to realize feature fusion. *en, perform downsampling, compress height and width, and finally stack with the previous feature layer to realize more feature fusion (5 times). *The classification module uses the features extracted from the network to make classification judgments. Take the 13 ×13 grid as an example, which is equal to dividing the input Mel spectrogram into 13 ×13 squares; then, each square will be preset with three prior frames. *e classification results of the network will adjust the positions of these three prior boxes and finally filter by the nonmaximum suppression (NMS) algorithm [28], to get the final classification results.

5. Audio Recognition System Based on SPPYOLO-v4

5.1. System Architecture. As shown in Figure 5, after audio input, the proposed audio recognition system first divides the audio sequence data into two parts. *The first part comes from the stereo channel, while the second part is compressed into mono. *e audio signals of the two channels are extracted by MFCC spectrogram and input into the SPP-YOLO-v4 model as features. *en, two groups of SPP-YOLO-v4 models are integrated, and the stacking method is adopted in the integration. After the integrated learning of the two models, the audio classification results are finally output. *e details of the SPP-YOLO-v4 model are shown in Figure 4.

5.2. Stacking Integrated Learning. As shown in Figure 5, the system uses ensemble learning technology to get the final classification result. *The basic idea of ensemble learning is to form a strong classifier through the combination of several weak classifiers. Even if some weak classifiers make wrong predictions, they can be corrected by other weak classifiers with correct predictions, thus achieving the effect of improving the system's performance.

Assuming that x is input, mi (i � 1, 2, . . . , k) is a group of classifiers and the output of the classifiers is the probability distribution mi (x, cj) of each class cj (i � 1, 2, . . . , k), the final output y(x) of the integrated classifier can be expressed as

classification target. At present, popular ensemble learning algorithms include stacking, bagging, boosting, ensemble selection, and so on. *The ensemble learning algorithm selected in this study is the stacking method.

Stacking is a process of second-order learning with the output of the first-order learning process as input, also known as “meta-learning.” *The stacking method has become a popular ensemble learning method, not only because its implementation is quite simple but also because it can significantly improve the generalization ability of the system, which is very consistent with the purpose of this study. *The basic principle of the stacking method is shown in Figure 6.

6. Experiment and Result Analysis

6.1. Experimental Environment and Dataset. *e hardware platform of this study is Intel Core i3-M350 CPU@ Dualcore 2.20 GHz, 8 GB of DDR2 memory, Nvidia RTX2080Ti GPU, and 11 GB of video memory. *e PyCharm integrated development tool is developed in Python 3.5.0 language. *e YOLO annotation framework written in Python is used to convert the numerical format so that it can be read by YOLO. *e comparison methods are the Gaussian mixture model (GMM), CNN, and R-CNN.

*e experimental dataset is recorded audio files in the real teaching environment. *e dataset consists of audio types of four different labels (D1, D2, D3, and D4). All audio files are cut into 30-second clips. *There are 12 audio file formats including MPEG, MP3, and WMA. Each recording is performed at a different location, and the average recording duration is 3–5 minutes. *e recording equipment includes a two-channel Soundman OKM II Classic/studio A3 in-ear microphone and Roland Edirol R09 waveform recorder with 44.1 kHz sampling rate and 24-bit resolution.

*The used dataset contains 1404 audio files, and the number of audio files of each type is 351. About 70% of the data is used for training the audio recognition model, and the remaining 30% is used for testing. *e system settings are given in Table 1.

6.3. Verification of SPP-YOLO-v4 Performance. To verify the promotion effect of the proposed improved YOLO-v4 (SPP-YOLO-v4) on generalization ability, it is compared with the traditional YOLO-v4 model. In the experiment, 3 of 12 audio file formats were selected: MPEG, MP3, and WMA. *The generalization ability of SPP-YOLOv4 is given in Table 2.

From Table 2, it can be found that the overall accuracy of SPP-YOLO-v4 is higher than that of traditional YOLO-v4, which verifies its generalization ability for data in different audio formats. *is because compared with the original method, SPP of SPP-YOLO-v4 contains more layers, but it also increases the processing time.

6.4. Comparison of Test Results. Table 3 provides the results of training loss, mAP, and so on for all categories after 8000 rounds of training. It can be seen that the training model of the proposed method can effectively identify audio types. It has certain advantages in accuracy, recall rate, and F1 score, and its loss value is also the lowest of all methods, only 0.0122. *Therefore, the stability and accuracy of the proposed method are better. *is mainly due to the high resolution and receptive field (RF) of SPP-YOLO-v4, and the addition of the SPP module in the connection layer retains the advantages brought by SPP. In terms of training time, SPP-YOLO-v4 is only slightly more than GMM. *e CNN needs to train a lot of convolution operations, so its training time is longer.

Finally, the experiment uses data from 12 different audio formats to test and compare the four methods. Table 4 provides the values of test accuracy and test time. It can be seen that the average accuracy of the method proposed in this study is 99.0%, and the average detection time is 0.449ss. *Therefore, the proposed method achieves better performance among the four methods compared. It can be concluded that the upsampling and maximum pooling of SPP-YOLO-v4 brought significant benefits. Maximum pooling selects the maximum value from adjacent areas to slightly delete some maximum frequency noise in the audio sequence. *Therefore, convolution subsampling can be better operated in the subsequent sampling layer. *rough these advantages, SPP can improve the performance of the backbone network.

7. Conclusions

*is study presents an audio recognition system suitable for a music teaching environment. Use SPP to improve YOLO-v4 network architecture, that is to say, use SPP to select local areas on different scales of the same convolution layer to learn the characteristics of the multiscale system. In addition, the stacking method in ensemble learning is used to fuse independent submodels of two different channels. *e experimental results show that the proposed method can improve the recognition accuracy of audio types and has better performance for different audio file formats. Due to the limitation of audio recording conditions, there are few audio types in the experimental dataset and the classification performance of audio files recorded by different devices has yet to be verified. More tests will be conducted on these two issues in the future.

Data Availability

*e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

*The authors declare that they have no conflicts of interest.

References

[1] S. Wu, D. Zhang, Z. Zhang, N. Yang, M. Li, and M. Zhou, “Dependency-to-Dependency neural machine translation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 11, pp. 2132–2141, 2018. 

[2] J. Zou, W. Li, C. Chen, and Q. Du, “Scene classification using local and global features with collaborative representation fusion,” Information Sciences, vol. 348, no. 2, pp. 209–226, 2016. 

[3] H. Phan, L. Hertel, M. Maass, P. Koch, R. Mazur, and A. Mertins, “Improved audio scene classification based on label-tree embeddings and convolutional neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 6, pp. 1278–1290, 2017. 

[4] S. Bayatli, “Unsupervised weighting of transfer rules in rule-based machine translation using maximum-entropy approach,” Journal of Information Science and Engineering, vol. 36, no. 2, pp. 309–322, 2020.

[5] A. Rakotomamonjy, “Supervised representation learning for audio scene classification,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 6, pp. 1253–1265, 2017. 

[6] W. Yang and S. Krishnan, “Combining temporal features by local binary pattern for acoustic scene classification,” IEEE/ ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 6, pp. 1315–1321, 2017. 

[7] A. S. Dhanjal and W. Singh, “An automatic machine translation system for multi-lingual speech to Indian sign language,” Multimedia Tools and Applications, vol. 81, no. 3, pp. 4283–4321, 2021. 

[8] M. A. . Alamir, “A novel acoustic scene classification model using the late fusion of convolutional neural networks and different ensemble classifiers,” Applied Acoustics, vol. 172, no. 3, pp. 112–122, 2020. 

[9] J. G. Makin, D. A. Moses, and E. F. Chang, “Machine translation of cortical activity to text with an encoder–decoder framework,” Nature Neuroscience, vol. 23, no. 4, pp. 575–582, 2020. 

[10] S. Waldekar and G. Saha, “Two-level fusion-based acoustic scene classification,” Applied Acoustics, vol. 170, no. 5, Article ID 107502, 2020.

For more information:1950477648nn@gmail.com