An Ensemble of Convolutional Neural Networks for Audio Classification

Abstract

Research in sound classification and recognition is rapidly advancing in the field of pattern recognition. One important area in this field is environmental sound recognition, whether it concerns the identification of endangered species in different habitats or the type of interfering noise in urban environments. Since environmental audio datasets are often limited in size, a robust model able to perform well across different datasets is of strong research interest. In this paper, ensembles of classifiers are combined that exploit six data augmentation techniques and four signal representations for retraining five pre-trained convolutional neural networks (CNNs); these ensembles are tested on three freely available environmental audio benchmark datasets: (i) bird calls, (ii) cat sounds, and (iii) the Environmental Sound Classification (ESC-50) database for identifying sources of noise in environments. To the best of our knowledge, this is the most extensive study investigating ensembles of CNNs for audio classification. The best-performing ensembles are compared and shown to either outperform or perform comparatively to the best methods reported in the literature on these datasets, including on the challenging ESC-50 dataset. We obtained a 97% accuracy on the bird dataset, 90.51% on the cat dataset, and 88.65% on ESC-50 using different approaches. In addition, the same ensemble model trained on the three datasets managed to reach the same results on the bird and cat datasets while losing only 0.1% on ESC-50. Thus, we have managed to create an off-the-shelf ensemble that can be trained on different datasets and reach performances competitive with the state of the art.

1. Introduction

Sound classification and recognition have long been included in the field of pattern recognition. Some of the more popular application domains include speech recognition [1], music classification [2], biometric identification [3], and environmental sound recognition [4], the topic of interest in this work. Following the three classical pattern recognition steps of (i) preprocessing, (ii) feature/descriptor extraction, and (iii) classification, most early work in sound classification began by extracting features from audio recordings such as the Statistical Spectrum Descriptor or Rhythm Histogram [5]. Once it was recognized, however, that visual representations of audio, such as spectrograms [6] and Mel-frequency Cepstral Coefficients spectrograms (Mel) [7], contain valuable information, powerful texture extraction techniques like local binary patterns (LBP) [8] and its many variants [9] began to be explored for audio classification [2,10]. In [2], for example, ensembles of classifiers designed to fuse sets of the most robust texture descriptors with acoustic features extracted from the audio traces on multiple datasets were exhaustively investigated; this study showed that the accuracy of systems based solely on acoustic or visual features could be enhanced by combining texture features.

Recently, deep learners have proven even more robust in image recognition and classification than have texture analysis techniques. In audio biodiversity assessment, for example, a task that intends to monitor animal species at risk, convolution neural networks (CNNs) have greatly enhanced the performance of animal [11] and bird identification [12,13,14]. Deep learners have also been adapted to identify the sounds of marine animals [11] and fish [15]. In both these works, the authors combined CNN with visual features; the fusion of CNNs with traditional techniques was shown to outperform both the stand-alone conventional and single deep learning approaches.

Another environmental audio recognition problem that is growing in relevancy has to do with identifying sources of noise in environments. This audio classification problem is of particular concern for cell phone developers since noise interferes with conversation. Consequently, datasets of extraneous sounds have been released to develop systems for handling different kinds of noise. The Environmental Sound Classification (ESC-50) dataset, for instance, contains 2000 labeled samples divided into fifty classes of environmental sounds that range from dogs barking to the sound of sea waves and chainsaws. In [16], a deep CNN achieved results superior to human classification on this dataset. Other work of interest in this area includes [17,18,19,20,21].

For all its power, deep learning also has significant drawbacks when it comes to environmental sound classification. For one, deep learning approaches require massive training data [22]. For audio classification, this means collecting large numbers of labeled audio signals and transforming them into visual representations, a task that is prohibitively expensive and labor-intensive. Fortunately, there are methods for increasing the number of images in small datasets. One such method is to apply data augmentation techniques. Audio signals can be augmented in both the time and frequency domains, and these augmentation techniques can be directly applied either on the raw signals themselves or on the images obtained after they have been converted into spectrograms. In [23], for example, several augmentation techniques were applied to the training set in the BirdCLEF 2018 dataset. The augmentation pipeline involved taking the original bird audio signals, chunking them, and then augmenting them in the time domain (e.g., by adding background/atmospheric noise) and in the frequency domain (e.g., by applying pitch shifts and frequency stretches). This augmentation process not only enlarged the dataset but also produced nearly a 10% improvement in identification performance. Similarly, some standard audio augmentation techniques such a time and pitch shifts for bird audio classification were applied in [24]. Samples were also generated in [24] by summing separate samples belonging to the same class. This summing technique was used for domestic sound classification in [25,26]. In [27], new data was generated by computing the weighted sum of two samples belonging to different classes and teaching the network to predict the weights of the sum. Audio signal augmentation on a domestic cat sound dataset was produced in [28] by random time stretching, pitch shifting, compressing the dynamic range, and inserting noise. Data augmentation techniques that are standard in speech recognition have also proven beneficial for animal sound identification, as in [29,30].

The goal of this work is to perform an extensive study of CNN classification using different architectures combined with an investigation of multiple sets of different data augmentation approaches and methods for representing audio signals as images. Building such ensembles is motivated by two observations: (1) it is well known that ensembles of neural networks generally perform better than stand-alone models due to the instability of the training process [31], and (2) it has been shown in other classification tasks that an ensemble of multiple networks trained with different augmentation protocols performs much better than do stand-alone networks [32]. The classification scores of the CNNs are combined by sum rule and compared and tested across three different audio classification datasets: domestic cat sound classification [28,33], bird call classification [34], and environmental noise classification [4].

The main contribution of this study is the exhaustive tests performed on ensembles made by fusing (see Figure 1) five CNNs trained with four audio representations of raw sound signals combined with six different data augmentation methods (totaling thirty-five subtypes). The performances of the best performing ensembles are compared across the three datasets and with the state of the art. In the fair comparisons, our method outperforms all others. Another contribution of this work is the free availability of the MATLAB source code used in this study, available at https://github.com/LorisNanni.

2. Audio Image Representations

Since the input to a CNN is in the form of a matrix, the following four methods were used to map the audio signals into spectrograms:

• The Discrete Gabor Transform (DGT): this is a short-time Fourier transform (STFT) with a Gaussian kernel as the window function. The continuous version of DGT can be defined as the convolution between the product of the signal with a complex exponential and a Gaussian, as:

  $$G\left(\tau ,\omega \right)=\frac{1}{{\sigma }^2}{\int }_{-\infty }^{+\infty }\mathit{x}\left(t\right)e^{i\omega t}{e}^{-\pi {\sigma }^2{\left(t-\tau \right)}^2}dt,$$

  (1)

  where s$\left(t\right)$ is the signal, $\omega$ is a frequency, and $i$ is the imaginary unit. The width of the Gaussian window is defined by ${\sigma }^2$. The discrete version of DGT applies the discrete convolution rather than the continuous convolution. The output $G\left(\tau ,\omega \right)$ is a matrix, where the columns represent the frequencies of the signal at a fixed time. The DGT implementation used in this study (see, [35]) is available at http://ltfat.github.io/doc/gabor/sgram.html (accessed on 6 January 2021).
• Mel spectrograms (MEL) [36]: these spectrograms are computed by extracting the coefficients relative to the compositional frequencies with STFT. Extraction is accomplished by passing each frame of the frequency-domain representation through a Mel filter bank (the idea is to mimic the non-linear human ear perception of sound, which discriminates lower frequencies better than higher frequencies). Conversion between Hertz ($f$) and Mel ($m$) is defined as:

  $$m=2595{\log }_{10}\left(1+700f\right).$$

  (2)

The filters in the filter bank are all triangular, which means that each has a response of 1 at the center frequency, which decreases linearly towards 0 until it reaches the center frequencies of the two adjacent filters, where the response is 0.

3.

Gammatone (GA) band-pass filters: this is a bank of GA filters whose bandwidth increases with the increasing central frequency. The functional form of Gammatone is inspired by the response of the cochlea membrane in the inner ear of the human auditory system [37]. The impulse response of a Gammatone filter is the product of a statistical distribution (Gamma) and a sinusoidal carrier tone. This response can be defined as:

$$h_i\left(t\right)=\left\{\begin{array}{ll}a·t^{n-1}{e}^{-2\pi B_{i}t}\cos \left(2\pi {\omega }_{i}t+\varphi \right)& ,t\ge 0\\ 0& ,t<0\end{array}\right.$$

(3)

where ${\omega }_i$ is the central frequency of the filter and $\varphi$ its phase. Gain is controlled by the constant $a$, and $n$ is the order of the filter. $B_i$ is a decay parameter that determines the bandwidth of the band-pass filter.

4.

Cochleagram (CO): this mapping models the frequency selectivity property of the human cochlea [38]. To extract a cochleagram, it is first necessary to filter the original signal with a gammatone filter bank (see, Equation (3) above). The filtered signal must then be divided into overlapping windows. For each window and every frequency, the energy of the signal is calculated.

Each of the four spectrograms is then mapped to a gray-scale image using a linear transformation that maps the minimum value to 0 and the maximum value to 255, with the value of each pixel rounded to the closest smaller integer.

3. Convolutional Neural Networks (CNNs)

Aside from the input and output layers, CNNs are composed of one or more of the following specialized hidden layers: convolutional (CONV), activation (ACT), pooling (POOL), and fully-connected (FC), or classification layer. The CONV layers pull out features from the input volume and work by convolving a local region of the input volume (the receptive field) to filters of the same size. Once the convolution is computed, these filters slide into the next receptive field, where once again, the convolution between the new receptive field and the same filter is computed. This process is iterated over the entire input image, whereupon it produces the input for the next layer, a non-linear ACT layer, which improves the learning capabilities and classification performance of the network. Typical activation functions include (i) the non-saturating Rectified Linear Activation Function (ReLU) function $f\left(x\right)=\tan \mathrm{h}\left(x\right)$, (ii) the saturating hyperbolic tangent $f\left(x\right)=\tan \mathrm{h}\left(x\right)$, $f\left(x\right)=\left|\tan \mathrm{h}\left(x\right)\right|$, and (iii) the sigmoid function $f\left(x\right)={\left(1+e^{-x}\right)}^{-1}$. Pool layers are often interspersed between CONV layers and perform non-linear downsampling operations (max or average pool) that serve to reduce the spatial size of the representation, which in turn has the benefit of reducing the number of parameters, the possibility of overfitting, and the computational complexity of the CNN. FC layers typically make up the last hidden layers and have FC neurons to all the activations in the previous layer. SoftMax is generally used as the activation function for the output CLASS layer, which performs the final classification (also typically using the SoftMax function).

In this study, five CNNs pre-trained on ImageNet [39] or Places365 [40] are adapted to the problem of environmental sound classification as defined in the datasets used in this work. The architecture of the following pre-trained CNNs remains unaltered except for the last three layers, which are replaced by an FC layer, an ACT layer using SoftMax, and a CLASS layer also using SoftMax:

• AlexNet [41] was the first neural network to win (and by a large margin) the ILSVRC 2012 competition. AlexNet has a structure composed of five CONV blocks followed by three FC layers. The dimension of the hidden layers in the network is gradually reduced with max-pooling layers. The architecture of AlexNet is simple since every hidden layer has only one input layer and one output layer.
• GoogleNet [42] is the winner of ILSVRC 2014 challenge. The architecture of GoogleNet involves twenty-two layers and five POOL layers. GoogleNet was unique in its introduction of a novel Inception module, which is a subnetwork made up of parallel convolutional filters. Because the output of these filters is concatenated, the number of learnable parameters is significantly reduced. This study uses two pre-trained GoogleNets: the first is trained on the ImageNet database [39], and the second is trained on the Places365 [40] datasets.
• VGGNet [43] is a CNN that took second place in ILSVRC 2014. Because VGGNet includes 16 (VGG-16) or 19 (VGG-19) CONV/FC layers, it is considered extremely deep. All the CONV layers are homogeneous. Unlike AlexNet [41], which applies a POOL layer after every CONV layer, VGGNet is composed of relatively tiny 3 × 3 convolutional filters with a POOL layer applied every two to three CONV layers. Both VGG-16 and VGG-19 are used in this study, and both are pre-trained on the ImageNet database [39].
• ResNet [44] was the winner of ILSVRC 2015 and is much deeper than VGGNet. ResNet is distinguished by introducing a novel network-in-network architecture composed of residual (RES) layers. ResNet is also unique in applying global average pooling layers at the end of the network rather than the more typical set of FC layers. These architectural advances produce a model that is eight times deeper than VGGNet yet significantly smaller in size. Both ResNet50 (a 50-layer residual network) and ResNet101 (the deeper variant of ResNet50) are investigated in this study. Both CNNs have an input size of 224 × 224 pixels.
• InceptionV3 [45] advances GoogleNet by making the auxiliary classifiers perform as regulators rather than as classifiers. This is accomplished by factorizing 7 × 7 convolutions into two or three consecutive layers of 3 × 3 convolutions and applying the RMSProp Optimizer. InceptionV3 accepts images of size 299 × 299 pixels.

In

Table 1. Specifications of the different network architectures.

Networks	Layers	Parameters	Size	Input Size
AlexNet	8	61 million	227 Mb	227 × 227
GoogleNet	22	7 million	27 Mb	224 × 224
VGG16	16	138 million	515 Mb	224 × 224
VGG19	19	147 million	535 Mb	224 × 224
ResNet50	50	25.6 million	96 Mb	224 × 224
ResNet101	101	44.6 million	167 Mb	224 × 224
Inception	48	23.9 million	89 Mb	224 × 224

, we report the specifications of the different network architectures that we used in our paper.

4. Data Augmentation Methods

Below is a description of the augmentation protocols that were combined and tested in this study. For each data augmentation method used to train a CNN, both the original and the artificially generated patterns were included in the training set.

4.1. Standard Signal Augmentation (SGN)

Standard signal augmentation (SGN) is the application of the built-in data augmentation methods for audio signals available in MATLAB (see samples in Figure 2). For each training signal, ten new ones are generated by applying the following labeled transformations with 50% probability:

• SpeedupFactorRange scales the speed of the signal by a random number in the range of [0.8, 1.2];
• SemitoneShiftRange shifts the pitch of the signal by a random number in the range of [−2, 2] semitones;
• VolumeGainRange increases or decreases the gain of the signal by a random number in the range of [−3, 3] dB;
• SNR injects random noise into the signal in the range of [0, 10] dB;
• TimeShiftRange shifts the time of the signal in the range of [−0.005, 0.005] seconds.

4.2. Short Signal Augmentation (SSA)

Short signal augmentation (SSA) works directly on the raw audio signals (see Figure 2 for some samples). For every original image, the following ten augmentations are applied to produce ten new images:

• applyWowResampling implements wow resampling, a variant of pitch shift that changes the intensity in time. The formula for the wow transformation is:

  $$F\left(s\right)=s+a_m\frac{\sin \left(2\pi f_{m}s\right)}{2\pi f_m}$$

  (4)

  where $s$ is the input signal. In this study, $a_m=3$ and $f_m=2;$
• applyNoise is the insertion of white noise so that the ratio between the signal and the noise $\mathrm{X}$ dB (here, $\mathrm{X}=10$);
• applyClipping normalizes the audio signal by leaving 10% of the samples out of [−1, 1], with the out-of-range samples ($x$) clipped to sign ($x$)
• applySpeedUp not only increases but also decreases the speed of the audio signal; in this study, the speed was augmented by 15%;
• applyHarmonicDistortion is the repeated application of quadratic distortion to the signal; in this study, the following distortion was applied five consecutive times:

  $$s_{out}=\sin \left(2\pi s_{in}\right);$$

  (5)
• applyGain increases the gain by a specific number of decibels (set here to 10 dB);
• applyRandTimeShift randomly divides each audio signal into two signals and swaps them by mounting them back into a randomly shifted signal. If we call $s_{in}\left(t\right)$ the value of the input audio signal at time $t$, $T$ is the length of the signal and $t^{*}$ is a random time between 0 and $T$:

  $$s_{out}\left(t\right)=s_{in}\left(\mathrm{mod}(t^{*}+t,T\right))$$

  (6)
• applyDynamicRangeCompressor applies dynamic range compression (DRC) [46] to a sample audio signal. DRC boosts the lower intensities of an audio signal and attenuates the higher intensities by applying an increasing piecewise linear function. DRC, in other words, compresses an audio signal’s dynamic range;
• applyPitchShift shifts the pitch of an audio signal by a specific number of semitones. We chose to increase it by two semitones;
• We use applyPitchShift again to decrease the pitch of the audio signal by two semitones.

4.3. Super Signal Augmentation (SSiA)

With this protocol, twenty-nine new images are generated from every original image. The following five augmentations are applied to every sample, with the parameters of the augmentations randomized to generate the new images:

• applyWowResampling, as in SSA;
• applySpeedUp, as in SSA; but, in this case, the speed is either increased or decreased by a random number of percentage points in the range $\left[-5,5\right]$;
• applyGain, as in SSA, but the gain factor is sampled randomly in the range of $\left[-0.5,0.5\right]$;
• applyRandTimeShift, as in SSA;
• applyPitchShift, as in SSA, but the pitch is shifted in the range of $\left[-0.5,0.5\right]$.

Small parameters are selected because applying multiple transformations to the input introduces changes that are too large. The difference between the protocols in SSiA and SSA is that the SSiA protocols create a large number of images through multiple small transformations (See Figure 2). Conversely, the images created by SSA protocols are generated with only one large transformation.

4.4. Time-Scale Modification (TSM)

This protocol applies the five algorithms contained in the time-scale modification (TSM) toolbox [47]. TSM methods (samples in Figure 2) are commonly used in music production software to change the speed of signals without changing their pitch. Since two different constant stretching factors (0.5 and 1.8) were used for each TSM method, this augmentation approach produced ten new images. For a detailed description of the TSM toolbox, see [48]. A brief description of the five TSM algorithms follows:

• OverLap Add (OLA): this algorithm is the simplest TSM method. It covers the input signals with overlapping windows of size $H_a$ and maps them into overlapping windows of size $H_s$. The number $H_{\alpha }$ depends on the implementation of the algorithm, while the ratio $\alpha =\raisebox{1ex}{$H_s$}\!\left/ \!\raisebox{-1ex}{$H_a$}\right.$ is the speed-up factor, which can be optionally set. The settings investigated in this study were 0.8 and 1.5. These same values were used for each TSM method;
• Waveform Similarity OverLap Add (WSOLA): this is a modification of OLA where the overlap of the windows is not fixed but has some tolerance to better represent the output signal in cases where there is a difference of phase;
• Phase Vocoder addresses the same phase problem as WSOLA. However, it exploits the dual approach by matching the windows in the frequency domain: first, the Fourier transforms of the signal are calculated; second, the frequencies are matched, and the signal is pulled back into the time domain;
• Phase Vocoder with identity phase locking: this TSM method is a modification of Phase Vocoder where the frequencies are matched as if they were not independent of each other. This modification was introduced by Laroche and Dolson [49];
• Harmonic-Percussive Source Separation (HPSS): this augmentation technique decomposes an audio signal into its harmonic sound components, which form structures in the time direction, and its percussive sounds, which yield structures in the frequency direction. After decomposing the signal in this way, the phase vocoder is applied with the identity phase locking to the harmonic component, and OLA is applied to the percussive component. Finally, these two components are merged to form a new signal.

4.5. Short Spectrogram Augmentation (SSpA)

Short spectrogram augmentation (see samples in Figure 3): works directly on spectrograms and generates five transformed versions of each original

• applySpectrogramRandomShifts randomly applies pitch shift and time shift;
• VocalTractLengthNormalization (apply VTLN) creates a new image by applying VTLP [29], which divides a given spectrogram into 10 unique temporal slices. Once so divided, each slice passes through the following transformation:

  $$G\left(f\right)=\left\{\begin{array}{ll}\alpha f& ,0\le f<f_0\\ \frac{f_{max}-\alpha f_0}{f_{max}-f_0}\left(f-f_0\right)+\alpha f_0& ,f_0\le f\le f_{max},\end{array}\right.$$

  (7)

  where $f_0$, $f_{max}$ are the basic and maximum frequency, and $\alpha \in \left[a,b\right]$ is randomly chosen. In this study, a and b are set to 0.9 and 1.1, respectively;
• applyRandTimeShift does what its name indicates by randomly picking the shift value $\mathrm{T}$ in $\left[1,\mathrm{M}\right]$, where $\mathrm{M}$ is the horizontal size of the input spectrogram. A given spectrogram is cut into two different images: ${\mathrm{S}}_1$ and ${\mathrm{S}}_2$, the first taken before and second after time $\mathrm{T}$. The new image is generated by inverting the order of ${\mathrm{S}}_1$ and ${\mathrm{S}}_2$;
• applyRandomImageWarp creates a new image by applying thin-spline image warping (TPS-Warp) [50] to a given spectrogram. TPS-Warp is a perturbation method applied to the original image by randomly changing the position of a subset $\mathrm{S}$ of the input pixels. It adapts pixels that do not belong to $\mathrm{S}$ via linear interpolation. In this study, the spectrogram is changed on the horizontal axis only. Also, a frequency-time mask is applied by setting to zero the values of two rows and one column of the spectrogram. In this study, the width of the rows is set to 5 pixels and the width of the column to 15 pixels;
• applyNoiseS applies pointwise random noise to spectrograms. The value of a pixel is multiplied by a uniform random variable of average one and variance one, with a probability $0.3$

4.6. Super Spectro Augmentation (SuSA)

In this protocol, 29 new images are generated from each original sample. The following five augmentation methods are applied to each signal, with parameters randomized to produce different samples (see samples in Figure 3):

• applySpectrogramRandomShifts as in SSpA, but with the time shift equal to zero and random pitch shift in the range $\left[-1,1\right]$;
• applyVTLN as in SSpA;
• applyRandTimeShift as in SSpA;
• applyFrequencyMasking sets to zero at most two random columns (which represent times) and at most two random rows (which represent frequencies);
• applyNoiseS applies pointwise random noise to spectrograms. The value of a pixel is multiplied by a uniform random variable in $\left[0.3,1.7\right]$ with a probability of $0.1$.

5. Experimental Results

5.1. Datasets

The approach presented here was tested on three sound datasets (see Figure 4 for some examples):

• BIRDZ [34]: a control audio dataset, where recordings in the wilds were downloaded from the Xeno-canto Archive (http://www.xeno-canto.org/ accessed on 6 January 2021). BIRDZ contains 2762 bird acoustic events with 339 detected “unknown” events that are either noise or other vocalizations aside from the eleven labeled North American bird species. Many spectrogram types (constant frequency, broadband with varying frequency components, broadband pulses, frequency modulated whistles, and strong harmonics) are included;
• CAT [28,33]: a balanced dataset of 300 samples of 10 classes of cat vocalizations collected from Kaggle, YouTube, and Flickr. The average duration of each sound is ~ 4 s;
• ESC-50 [4]: an environmental sound classification dataset with 2000 samples evenly divided into 50 classes and five folds; each fold contains eight samples. The five folds with class names in parentheses are Animals (Dog, Rooster, Pig, Cow, Frog, Cat, Hen, Flying Insects, Sheep, and Crow), Natural soundscapes and water sounds (Rain, Sea waves, Crackling fire, Crickets, Chirping birds, Water drops, Wind, Pouring water, Toilet flush, and Thunderstorm), Human non-speech (Crying baby, Sneezing, Clapping, Breathing, Coughing, Footsteps, Laughing, Brushing teeth, Snoring, and Drinking/sipping), Interior and domestic sounds (Door knock, Mouse click, Keyboard typing, Door, wood creaks, Can opening, Washing machine, Vacuum cleaner, Clock alarm, Clock tick, and Glass breaking), and Exterior and urban noises (Helicopter, Chainsaw, Siren, Car horn, Engine, Train, Church bells, Airplane, Fireworks, and Hand saw).

The data augmentation techniques explored in this study are assessed on each dataset using the same testing protocols described in the original papers. The recognition rate (the average accuracy across all folds) is used as the performance indicator. Also, it should be noted that many papers [16,27,46,48,51,52,53] report classification results on the datasets listed above that are superior to human performance.

5.2. Results

In

Table 2. Performance on the CAT (mean accuracy over ten-fold cross-validation). Values of interest are in bold.

CAT	NoAUG	SGN	SSA	SSpA	SSiA	SuSA	TSM
AlexNet	83.73	85.76	86.10	83.39	87.12	86.78	87.12
GoogleNet	82.98	86.10	87.80	83.39	86.78	85.08	87.80
VGG16	84.07	87.12	88.47	85.76	87.80	87.80	88.47
VGG19	83.05	85.42	87.80	84.75	86.10	86.10	89.15
ResNet50	79.32	81.36	85.42	76.95	85.08	82.03	87.12
ResNet101	80.34	84.75	85.42	75.59	82.03	73.56	86.78
Inception	79.66	82.71	---	66.44	---	84.07	86.10
GoogleNetPlaces365	85.15	86.44	85.76	83.73	86.10	86.10	88.47
VGG16BatchSize	---	86.10	88.14	86.78	89.49	86.10	89.15
FusionLocal	88.14	88.47	89.83	86.78	89.83	89.83	90.51
FusionShort	88.47
FusionShortSuper	89.83
FusionSuper	90.17
FusionALL	89.83
FusionSuperVGG16	89.83

,

Table 3. Performance on the BIRDZ (mean accuracy over ten-split training/test set).

BIRDZ	NoAUG	SGN	SSA	SSpA	SSiA	SuSA	TSM
AlexNet	94.48	94.96	95.40	94.02	95.05	95.76	88.51
GoogleNet	92.41	94.66	94.84	91.48	93.85	95.85	82.91
VGG16	95.30	95.59	95.60	94.69	95.44	96.18	94.63
VGG19	95.19	95.77	87.15 *	94.50	95.44	96.04	94.88
ResNet50	90.02	94.02	93.22	90.48	92.95	94.16	91.75
ResNet101	89.64	94.00	92.76	88.36	92.84	94.20	90.62
Inception	87.23	93.84	92.48	83.81	92.30	94.01	90.52
GoogleNetPlaces365	92.94	94.81	95.10	92.43	94.76	95.80	86.91
VGG16BatchSize	---	95.84	----	94.91	95.81	96.31	94.78
FusionLocal	95.81	96.32	96.24	95.76	96.39	96.89	95.27
FusionShort	96.47
FusionShortSuper	96.79
FusionSuper	96.90
FusionALL	96.89
FusionSuperVGG16	96.78

The * indicates a fold that failed to converge so producing a random performance in that fold. Values of interest are in bold.

,

Table 4. Performance on the ESC-50 (mean accuracy over the five-fold cross-validation). Values of interest are in bold.

ESC-50	NoAUG	SGN	SSA	SSpA	SSiA	SuSA	TSM
AlexNet	60.80	72.75	73.85	65.75	73.30	64.65	70.95
GoogleNet	60.00	72.30	73.70	67.85	73.20	71.70	73.55
VGG16	71.60	79.40	80.90	75.95	79.35	77.85	79.05
VGG19	71.30	78.95	78.80	74.10	78.00	76.40	77.45
ResNet50	62.90	76.65	75.95	70.65	77.20	73.95	77.40
ResNet101	59.10	75.25	75.65	70.05	77.50	72.30	74.85
Inception	51.10	71.60	74.70	63.45	75.55	71.10	70.65
GoogleNetPlaces365	63.60	75.15	76.10	71.35	74.00	71.60	73.55
VGG16BatchSize	---	79.40	80.50	73.45	79.35	77.85	80.00
FusionLocal	75.95	84.75	85.30	80.25	85.25	82.25	85.30
FusionShort	86.45
FusionShortSuper	87.15
FusionSuper	87.55
FusionALL	87.30
FusionSuperVGG16	85.75

and

Table 5. Performance using different methods for representing the signal as an image.

	CAT			BIRD			ESC-50
	GA	MEL	CO	GA	MEL	CO	GA	MEL	CO
AlexNet	82.03	83.73	79.32	91.85	91.43	87.54	73.95	73.50	65.50
GoogleNet	74.07	84.07	77.97	90.71	88.96	86.95	73.75	73.25	66.15
VGG16	83.39	86.10	80.00	92.65	93.17	88.82	77.60	79.20	66.75
VGG19	85.76	83.73	77.97	92.93	93.22	89.07	76.40	77.55	65.85
ResNet50	82.03	83.05	75.93	90.87	90.74	86.98	75.80	76.05	67.75
ResNet101	82.71	82.37	79.32	91.15	91.00	87.28	75.00	74.80	64.90
Inception	79.66	84.75	77.63	89.53	89.86	87.35	73.95	72.55	67.50
GoogleNetPlaces365	83.05	82.71	77.63	90.88	88.31	86.75	73.60	75.50	68.70
VGG16BatchSize	85.42	87.80	81.02	93.09	93.22	89.43	77.80	78.95	67.50
FusionLocal	87.46	88.47	82.37	93.76	93.97	90.57	81.90	83.80	73.25

, the accuracy obtained by some of the data augmentation protocols is reported and compared with the baseline that skips the augmentation step (NoAUG). The CNNs were trained for 30 epochs with a learning rate of 0.0001, except for the last fully connected layer that has a learning rate of 0.001 and a batch size of 60. The one exception is the CNN labeled ‘VGG16BatchSize’, the standard VGG16 with a fixed batch size of 30. For NoAUG, the batch size was set to 30. Also, seven fusions are reported in Table 2, Table 3, Table 4 and Table 5. We combined the results of the CNNs in an ensemble using the sum rule. The sum rule consists of averaging all the output probability vectors of the stand-alone CNNs in the ensemble to create a new probability vector that is used for classification. The rationale behind fusion, as Hansen [31] notes, is that “the collective decision produced by the ensemble is less likely to be in error than the decision made by any of the individual networks.” The labels used in the tables and a brief description of the seven ensembles follow:

• GoogleGoogle365: sum rule of GoogleNet and GoogleNetPlaces365 trained with each of the data augmentation protocols;
• FusionLocal: sum rule of CNNs where each one is trained with a different data augmentation method;
• FusionShort: sum rule of all CNNs trained with SGN, SSA, and SSpA;
• FusionShortSuper: sum rule of all CNNs trained with SGN, SSA, SSpA, SSiA, and SuSA;
• FusionSuper: sum rule of all CNNs trained with SGN, SSiA, SuSA, and TSM;
• FusionSuperVGG16: sum rule of VGG16 trained with SGN, SSiA, SuSA, and TSM;
• FusionALL: sum rule of all CNNs trained with SGN, SSA, SSpA, SSiA, SuSA, and TSM.

VGG16 can fail to converge; when this happens, VGG16 undergoes a second training. VGG16 can also produce a numeric problem by assigning the same scores to all patterns. In this case, all scores are considered zeros. Other numeric problems in the fusions by sum rule can occur. To avoid such issues, all scores that do not produce not-a-number value are treated as zero.

In Table 2, Table 3 and Table 4, the DGT spectrogram is used for representing the signal as an image. Any cell with ‘---’ means that the given CNN was not trained successfully (mainly due to memory problems with the graphical processing units (GPUs). We also tested three additional methods GA, MEL, and CO to represent a signal as an image, coupled with SGN only, as reported in Table 5.

In

Table 6. Comparison of our best sound classification ensemble with state of the art.

Descriptor	BIRDZ	CAT	ESC-50
[11]	96.45	89.15	85.85
FusionGlobal	96.82	90.51	88.65
FusionGlobal-CO	97.00	90.51	88.55
[2]	95.1	---	---
[34]	93.6	---	---
[33]	---	87.7	---
[28]	---	91.1	---
[28]—CNN	---	90.8	---
[51]	96.7	---	---
[20]	---	---	94.10
[54]	---	---	89.50
[21]	---	---	87.10
[19]	---	---	88.50
[27]	---	---	84.90
[16]	---	---	86.50
[17]	---	---	83.50
[52]	---	---	83.50
[18]	---	---	81.95
Human Accuracy [4]	---	---	81.30

, our best ensembles FusionGlobal and FusionGlobal-CO are compared with the state of the art. FusionGlobal is built with the CNNs belonging to Fusion Super and those reported in Table 5. FusionGlobal-CO is constructed similarly to FusionGlobal but without considering the CNNs trained using CO as a signal representation approach. The performance reported in [11] in Table 5 is different from that reported in the original paper since, for a fair comparison with this work, we ran the method without considering the supervised data augmentation approaches.

The following conclusions can be drawn from the reported results:

• There is no single data augmentation protocol that outperforms all the others across all the tests. TSM performs best on CAT and ESC-50 but works poorly on BIRDZ. Data augmentation at the spectrogram level works poorly on ESC-50 as well as on two other datasets. SGN and data augmentation at the signal level work well across all the datasets. On average, the best data augmentation approach is SSA. Although it produces a performance that is close to SSiA, the training time for SSA is shorter;
• The best stand-alone CNNs are VGG16 and VGG19;
• DGT works better than the other signal representations;
• Combining different CNNs enhances performance across all the tested datasets;
• For the ensemble FusionLocal, data augmentation is marginally beneficial on CAT and BIRDZ but produces excellent results on ESC-50. Compared to the stand-alone CNNs, data augmentation improves results on all three datasets. Of note, an ensemble of VGG16 (FusionSuperVGG16) outperforms the stand-alone VGG16;
• The performance of the ensemble of CNNs trained with different augmentation policies (FusionALL) can be further improved by adding to the ensemble those networks trained using different signal representations (FusionGlobal). However, this performance improvement required considerable computation time, mainly during the training step;
• The approach in [20] manages to outperform our results, but the authors pretrained their networks on AudioSet; hence the comparison with our approach is not fair.

The methods reported in Table 6 were based solely on deep learning approaches. As mentioned in the introduction, several papers have proposed sound classification methods based on texture features. It is also possible to construct ensembles that combine deep learning with texture methods. To examine the potential of combining ensembles trained on texture descriptors with deep learning approaches, the following two fusion rules were examined:

• Sum rule between FusionGlobal and the ensemble of texture features proposed in [55] (extracted from DGT images) obtains an accuracy of 98.51% (higher than that obtained by FusionGlobal). In BIRDZ, the ensemble of texture features obtains an accuracy of 96.87%, which is close to that obtained by our deep learning approach. The ensemble with texture descriptors works poorly on ESC-50, producing an accuracy of only 70.6%. As a result, it is not advantageous to combine the texture approach with FusionGlobal;
• The sum rule1 between FusionGlobal and [34] obtains an excellent accuracy of 98.96% compared to [34], which achieves an accuracy of 93.6%. This means that the features extracted in [34] and by the deep learning approach access different information.

In terms of computation time, the most expensive activity is the conversion of audio signals into spectrograms since the conversion runs on a CPU and not on a GPU. In

Table 7. Computation time (in seconds) between CNNs and signal representations.

Signal Representation	Computation Time	CNN	Computation Time
DGT	1.29	AlexNet	0.01
GA	0.02	GoogleNet	0.03
MEL	0.01	VGG16	0.01
CO	0.08	VGG19	0.01
		ResNet50	0.02
		ResNet101	0.03
		Inception	0.03

, computation time is reported for different CNNs and signal representations on a machine equipped with an i7-7700HQ 2.80 GHz processor, 16 GB RAM, and a GTX 1080 GPU. This test was run on an audio file of length 1.27 s with a sample rate of 32 kHz. It is interesting to note that FusionGlobal takes less than three seconds using a laptop. However, a speed-up is possible: audio files can be classified simultaneously with DGT since it can be parallelized.

In

Table 8. Comparisons among the different approaches of audio classification.

Paper	Method	Advantages
Ours	Ensemble of networks with different data augmentation and different spectrograms.	Great performances over a variety of datasets and easy to implement but requires large computational power.
[20]	Multi-stage sequential learning with knowledge transfer from a large audio dataset.	State of the art on ESC-50. However, it requires large computational power, and the model is pretrained on a large audio dataset.
[54]	Pretrained model with multi-channel features.	Simple model with performances similar to ours on ESC-50.
[21]	CNN model with potential compression.	Performance close to the state of the art on ESC-50 and suitable for mobile devices.
[16]	CNN with filter banks learned using convolutional Restricted Boltzmann Machines + fusion with gammatone spectral coefficient and Mel energies.	Performance close to ours on ESC-50 but unsuitable for mobile devices.
[56]	CNN with mix-up and data augmentation.	Very light network for mobile devices that performs very well, but not as much as the state-of-the art models.

, we compare some of the advantages and disadvantages of our system with audio approaches proposed in the literature whose backbone is a neural network trained on image-based audio features.

6. Conclusions

In this paper, we presented the largest study conducted so far that investigates ensembles of CNNs using different data augmentation techniques for audio classification. Several data augmentation approaches designed for audio signals were tested and compared with each other and with a baseline approach that did not include data augmentation. Data augmentation methods were applied to the raw audio signals and their visual representations using different spectrograms. CNNs were trained on different sets of data augmentation approaches and fused via the sum rule.

Experimental results clearly demonstrate that ensembles composed of fine-tuned CNNs with different architectures maximized performance on the tested three audio classification problems, with some of the ensembles obtaining results comparable with the systems, including on the ESC-50 dataset. To the best of our knowledge, this is the largest, most exhaustive study of CNN ensembles applied to the task of audio classification. Our best ensemble, FusionGlobal is composed of 63 networks with nine different architectures and seven different trainings for every network. Four of the networks were obtained using different data augmentation strategies and three different methods to create a spectrogram. It is worth noting that the same ensemble is competitive with the state of the art on the three datasets. To be more precise, we reach 96.82% on the BIRDZ dataset, 90.51% on the CAT dataset, and 88.65% on ESC-50.

This work can be expanded further by investigating which augmentation methods (spectrogram augmentation vs. signal augmentation) work best for classifying different kinds of sounds. We also plan to apply transfer learning using spectrograms instead of natural images. A systematic selection of augmentation approaches, e.g., by iteratively evaluating an increasing subset of augmentation techniques (as is typical when evaluating different features), would require an enormous amount of time and computation power. An expert-based approach that utilizes the knowledge of environmental scientists would be the best way of handling this challenge.

This study could also be expanded by including more datasets, which would provide a more comprehensive validation of the proposed fusions. Furthermore, there is a need to investigate the impact on performance when different CNN topologies and parameter settings in the retuning step are combined with different types of data augmentation.

The technique proposed here is not exclusively applicable to environmental sound classification but can be applied to many related audio tasks. For example, this technique should work well for real-time audio event detection and speech recognition. An important application area related to the ESC-50 dataset for noise identification would be to apply our approach to the more refined problem of fault diagnosis of drilling machines and other machinery [57,58]. Most studies in this area have focused on extracting handcrafted texture features from visual representations of sound [58]. But recently, there has been some interesting work that has successfully used deep learning to solve this problem [57]. Investigating the performance of our proposed method to tasks such as these would involve combining a different set of network architectures, but the main idea of ensembling different models obtained from many audio augmentation protocols should increase classification performance on these novel tasks and would be worth investigating in the future. Finally, we plan on exploring some possibilities presented in [59].