Using Traffic Sensors in Smart Cities to Enhance a Spatio-Temporal Deep Learning Model for COVID-19 Forecasting

Abstract

Respiratory viruses, such as COVID-19, are spread over time and space based on human-to-human interactions. Human mobility plays a key role in the propagation of the virus. Different types of sensors in smart cities are able to continuously monitor traffic-related human mobility, showing the impact of COVID-19 on traffic volumes and patterns. In a similar way, traffic volumes measured by smart traffic sensors provide a proxy variable to capture human mobility, which is expected to have an impact on new COVID-19 infections. Adding traffic data from smart city sensors to machine learning models designed to estimate upcoming COVID-19 incidence values should provide optimized results compared to models based on COVID-19 data alone. This paper proposes a novel model to extract spatio-temporal patterns in the spread of the COVID-19 virus for short-term predictions by organizing COVID-19 incidence and traffic data as interrelated temporal sequences of spatial images. The model is trained and validated with real data from the city of Madrid in Spain for 84 weeks, combining information from 4372 traffic measuring points and 143 COVID-19 PCR test centers. The results are compared with a baseline model designed for the extraction of spatio-temporal patterns from COVID-19-only sequences of images, showing that using traffic information enhances the results when forecasting a new wave of infections (MSE values are reduced by a 70% factor). The information that traffic data has on the spread of the COVID-19 virus is also analyzed, showing that traffic data alone is not sufficient for accurate COVID-19 forecasting.

1. Introduction

The COVID-19 pandemic has generated an unprecedented effort worldwide in understanding virus infection mechanisms and propagation patterns. Although the worst part of the pandemic is over in many parts of the world thanks to vaccination campaigns, it is estimated that COVID-19 still causes around 400 deaths per day in the US [1]. COVID-19 has significantly influenced each aspect of our society: health, economy, employment, and mobility [2], but by using the data gathered during the pandemic, new models can be designed to fight the current and future spread of similar respiratory viruses.

The propagation of the COVID-19 virus is influenced by people-to-people interactions over time and space. At the same time, the spread of the virus has caused changes in human mobility patterns. Human mobility volumes and patterns as well as the means of transportation used have been influenced by the COVID-19 pandemic. Urban traffic sensors, together with data analytics, are able to provide real-time insights into the impacts of COVID-19 on urban governance [3]. Data-driven models can be used to characterize the impact of the COVID-19 pandemic on public and private mobility [2]. Different regression models have been shown to provide accurate results for traffic data estimation during lockdown and mobility-restricted periods [2]. Internet of Things sensors and smart city data have played a crucial role in the COVID-19 crisis, shaping decision-making and synchronized reactions as regards policy intervention [4].

Predictive methods for forecasting the dynamics of the virus play a major role in saving lives and are being used by policymakers in different parts of the world to mitigate the effects of the spreading of the virus, implement optimal policies, and optimize the use of healthcare resources [5]. Human mobility significantly contributes to the spread of the virus, and human mobility data have been shown to improve the accuracy of predictive methods to estimate upcoming values for COVID-19 incidence [6].

Different types of methods and models have been proposed in previous research studies in order to provide estimations for the spread of the COVID-19 virus, such as epidemic [7], simulation-based [8], statistical [9], machine-learning-based [10], and hybrid models [11]. The availability of open datasets that have been gathered during the pandemic period provides an opportunity to design and optimize new data-driven models that will help in mitigating not only the spread of the current COVID-19 virus but also to be better prepared for other respiratory viruses. This paper proposes, implements, and validates a new data-driven, machine-learning model to forecast COVID-19 infections, which makes use of a convolutional neural network (CNN) to extract the spatial component of the pandemic and a recurrent neural network (RNN) using long short-term memory (LSTM) cells to find the temporal patterns of the spread of the virus. The major novelty of the model is that it is able to extract patterns from sequences of images that combine a geolocated time series of COVID-19 images (provided by the different healthcare centers measuring virus incidence in each area) and geolocated temporal sequences of traffic images (from intelligent traffic sensors in a smart city covering the same areas) as a proxy data source to estimate the movements of the population to enhance the temporal predictions for the spread of the virus. Real data for a period of 20 months are used to validate the proposed model. Open data from the city of Madrid (Spain) are used. The major contributions of this paper are as follows:

• Using a new deep learning model to combine both space and time information for short-term COVID-19 forecasting;
• Using geolocated data on the number of new COVID-19 infections to generate incidence maps;
• Using geolocated traffic data to generate traffic maps;
• Validation of the proposed model with real data in order to assess if spatio-temporal patterns can be learned and used to improve the accuracy in the forecast of the evolution of the COVID-19 pandemic for each location in a region.

This paper is divided into seven sections. Section 1 introduces and motivates the research and objectives of the work conducted. Section 2 summarizes some previous related work, showing the need for more studies extracting combined patterns from the combination of spatial and temporal information (interleaving COVID-19 incidence and traffic data together). Then, Section 3 presents the methods used in this paper to generate sequences of images from traffic sensor data and from COVID-19 PCR reported data by primary healthcare centers. Section 4 captures the details of the models proposed in this paper. Section 4 also presents the baseline model that will be used to compare results with previous research found in the literature. Section 5 describes the datasets that will be used to train and validate the models presented in this paper, and Section 6 presents the validation of the results. Finally, some major conclusions are presented in Section 7.

2. Related Work

COVID-19 incidence values and human mobility data, as measured by proxy variables such as traffic sensors in smart cities, have been previously combined in previous research studies. Machine learning models have been proposed to either forecast traffic values or COVID-19 incidence data. Regression models have been shown to accurately anticipate traffic volumes influenced by COVID-19 data [6]. Regression Trees [12] and Gaussian Process Regressors (GPR) [13] have shown the best performance as compared to other regression models for traffic volume forecasting [6]. Machine learning models based on time-series analysis are able to achieve better COVID-19 short-term predictions when using traffic data measured by smart city sensors [14]. Several machine learning models have been proposed to provide answers to different COVID-19-related questions, such as improving the diagnosis of positive cases based on reported symptoms [15], X-ray images [16], or laboratory data [17] or estimating the probability that positive cases will develop complications that will require hospitalization [18]. The current paper will focus on forecasting the evolution of the COVID-19 pandemic based on the detection of similar patterns in historic infection and traffic-based estimated mobility data. This section introduces previous machine learning models that have been used for forecasting the evolution of COVID-19 infection cases in general and based on the use of traffic data in particular.

Many of the existing data-driven models able to forecast the evolution of COVID-19 positive cases proposed in previous research studies are based on the analyses of mobility-agnostic incidence-based time series. The authors in [19] made use of 3 different datasets, gathered during the COVID-19 pandemic, in order to analyze several machine learning models designed to process time series of COVID-19 incidence data to forecast upcoming values. The authors studied 4 different models (the autoregressive integrated moving average model, the support vector machine model, the least-squares support vector machine, and the autoregressive integrated moving average-support vector machine model) and proposed a combined model to optimize results. The accuracy of different machine learning models for short-term COVID-19 forecasting was compared using data from 32 European countries in [20]. The authors concluded that ensemble models could be used to optimize the results.

Different deep-learning-based models for time series data analysis have been used to forecast COVID-19 incidence data. The authors in [21] used data from 12 different countries to train several Recurrent Neural Network (RNN) architectures such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) in order to forecast new COVID-19 cases. The authors found that LSTM-based models were able to provide the best results among the tested models. Shahid et al. [22] used similar models to estimate deaths and recoveries in 10 major countries showing that the bidirectional LSTM (Bi-LSTM) model was able to generate more accurate predictions than other non-time series-oriented machine learning models such as Support Vector Regression (SVR). Haviluddin et al. [23] analyzed different variants of Convolutional Neural Networks (CNN) used in order to extract patterns in temporal COVID-19 data showing their performance when using several loss functions. The authors in [24] proposed to combine LSTM and CNN models over time series data for COVID-19 forecasting, showing that such combined models were able to outperform either LSTM-based or CNN-based models. The current paper will use this result in order to propose a new way to organize the input data to better optimize the CNN part of the combined model to further optimize results (the proposed model will be enhanced by adding mobility data using a similar spatio-temporal format based on sequences of images to capture the information that mobility has on the spread of the COVID-19 virus).

Although mobility-agnostic time-series-based forecasting models have shown promising results, the spread of the COVID-19 virus has both a spatial and a temporal component. Models considering the spatial distribution of the virus to enhance time-series predictions have also been developed. Huang et al. [25] used incidence data for the evolution of the COVID-19 virus in 3 different countries (Spain, Germany, and Italy) to develop a model based on a combined CNN and BiGRU which considered the location information to improve predictions for the total number of cases. However, the contribution of the spatial component was very limited since only three non-adjacent locations were used as the input to the model. Graph Neural Networks (GNN) have also been used to extract spatial patterns in mobility scenarios. The authors in [26] proposed a GNN to model the interactions among individuals in order to spread the virus. A GNN is used together with a Recurrent Neural Network to combine space and time interactions [27]. Liu et al. [28] proposed the use of a Geographically Weighted Regression (GWR) model to include geographic information. A different approach to using spatial data is proposed in [11]. The authors generated COVID-19 incidence maps for Germany dividing the country into a grid of 412 equally sized areas and added up the COVID-19 cases reported each week for each area in order to generate an incidence image. The spatial images were used to extract information to complement a temporal Susceptible-Undiagnosed-Infected-Removed (SUIR) model, defining a hybrid model. The model applied a first machine learning method and a subsequent epidemic model. The epidemic model tried to improve its prediction results by inferring basic mobility data based on the number of COVID-19 cases in the adjacent areas and using a configurable transition probability among areas. The model assumed a uniform transition probability epsilon, that can be configured in the epidemic model to capture mobility restriction policies. A related publication [29] has proposed and validated the use of a combined CNN and LSTM deep machine learning model to extract both temporal and spatial patterns from COVID-19 incidence images, similar to those used in [11] but applied to data from the Madrid area in Spain. The proposed model was compared with machine learning models found in previous studies and applied to the same dataset. By adding the pattern extraction from COVID-19 incidence images to the model, the accuracy when forecasting new and cumulated cases one week ahead was improved.

Human mobility has highly contributed to the spread of the COVID-19 virus. Lau et al. [30] studied the correlation of domestic and international passenger volumes with the temporal and spatial spread of the COVID-19 virus. The authors found a strong linear correlation between passenger volumes in domestic flights and COVID-19 new cases within China. A similar result is presented in [31]. Air traffic information however had less influence when partial lockdowns were imposed on regions in which mobility was restricted to such regions. A different source for estimating user mobility is used in [32]. The authors estimated people’s mobility based on cellular network traffic data and defined a model to forecast the number of COVID-19 infections in the future. The model estimated transition probabilities to define a Markov chain by analyzing user-level mobility events between antennas from the cellular network connectivity logs. The authors showed that mobility-aware models were able to improve the accuracy of a baseline linear regression model that did not use mobility data as an input feature. The spread of the COVID-19 virus and its effects on traffic data have been analyzed in recent studies such as [33] where using an Artificial Neural Network (ANN) traffic volumes were estimated in response to COVID-19 imposed measures and [34] which showed that micro-mobility was profoundly shaped by the COVID-19 pandemic. Traffic data, therefore, incorporate information about the current state of the pandemic and can have an impact on its future evolution.

This paper expands the model validated in [29] to incorporate traffic images in order to improve the accuracy when forecasting new COVID-19 infections one week ahead. Time sequences of COVID-19 incidence images are combined with similar sequences of traffic images in order to feed a CNN that extracts spatial patterns. The output of the CNN is fed into an LSTM-based RNN in order to extract temporal patterns and perform one-week ahead predictions for each point in the map. The model is validated with open data from the Madrid city and compared with the model presented in [29] as the baseline in order to assess the benefits of adding traffic data for COVID-19 forecasting. Related ideas have been implemented for other chaotic time series predictions (non-linear systems that produce non-periodic output sensitive to initial conditions [35]). The authors in [35] proposed a combination of a former CNN model to extract deep features from chaotic time series and a subsequent RNN to extract time patterns related to the models proposed in this paper.

A summary of the related studies categorized by the type of ML model used and the variable that is predicted is captured in

Table 1. Summary of the related studies by model type and predicted variable.

References	Model Types	Predicted Variable
[6,12,13]	Regression Trees, Gaussian processes	Traffic volumes
[15,16,17,18]	Shallow ML models	COVID-19 diagnoses
[19,20]	Shallow ML models	COVID-19 incidence
[21,22,23]	Deep ML models	COVID-19 incidence
[11,25,26,27]	Space-time models	Mobility-enhanced COVID-19 estimations
[30,31,32,33,34]	Shallow ML models	Mobility estimations caused by COVID-19

.

3. Generating Images from Sensor Data

Smart cities capture data from different types of sensors that provide information about the state of the city. This information can be used to optimize internal processes, estimate the upcoming need for resources in the near future, or facilitate the decision-making process by local authorities. This paper makes use of two COVID-19-related sensor types in order to feed a new deep learning model to estimate the number of upcoming COVID-19 infections. The first sensor type is related to the Polymerase Chain Reaction (PCR) COVID-19 tests which are carried out daily in each primary care health center. Each center is able to sense the amount of positive cases in a particular region. The second sensor type is associated with each traffic measuring point able to sense the traffic intensity in the particular location where it is installed. Traffic sensors provide valuable data for Intelligent Transportation Systems (ITS), providing a real-time data source to assess the mobility in a smart city. Both sensor types provide temporal series of space-distributed data. In order to feed a deep learning model able to extract spatio-temporal patterns, the sensed data have to be processed into a sequence of images.

A similar approach as the one presented in [11] will be used to generate COVID-19 incidence images. The region under study will be divided into a grid of equal-sized squares (areas) covering it. Each primary health center performing PCR COVID-19 testing will contribute with the measured positive cases to the square (area) in the image according to its coordinates. An image like the one captured in Figure 1 is obtained by adding the cases measured for the same period of time for all the primary health centers. The number of vertical and horizontal pixels in the generated image should be adjusted to the number of COVID-19 measuring points so that there are not many measuring points per square, or many squares in the image without any associated measuring point. In order to optimize the training of the machine learning models, a normalization process could be applied. Figure 1 shows a COVID incidence image after applying a linear scaling so that the values are normalized between 0 and 1 following the expression in (1).

$$i_n=\frac{i-i_{min}}{i_{max}-i_{min}}$$

(1)

where i represents the number of positive cases in a particular square of the image, i_min the minimum value in all the incidence images, and i_max the maximum value in all the incidence images. The same normalization values are therefore applied to all the images in order to preserve the relative weekly changes in COVID-19 incidence values.

Traffic images are computed from traffic intensity data provided by traffic flow sensors. The same region in space will be divided using the same grid of squares as in the COVID-19 incidence image. The value for each square in the image will be calculated as the average value for the traffic intensity measures provided by all traffic sensors located in that square. A similar normalization process could be applied to the generated traffic images in order to speed up the training of the machine learning model. Figure 2 shows an example of a generated traffic image.

4. Models

The new model proposed in this paper is captured in this section. The model uses a combined traffic mobility and COVID-19 incidence data input representation from which to extract the patterns governing the spread of the virus. The results will be compared with a baseline model that will only use the COVID-19 incidence data in order to generate COVID-19 incidence short-term predictions. The baseline model will be used to assess the model gain when adding mobility information. The proposed model will be divided into 2 different versions which are described in Section 4.1 and Section 4.2. Section 4.1 describes a new model in which both COVID-19 incidence and traffic images are processed together using a single Convolutional Neural Network (CNN) in order to extract common spatial patterns. Section 4.2 proposes a small modification of the model in Section 4.1 in which a different CNN is used to extract spatial patterns for each type of image and then the outputs of both CNNs are combined in order to feed the final long short-term memory (LSTM)-based Recurrent Neural Network (RNN). The baseline model is presented in Section 4.3. In the baseline model, only a single type of image is fed into the CNN. The baseline model was trained with COVID-19 data from the Community of Madrid (Spain) region in [29] and was compared with previous methods found in related literature showing better results for the same dataset. In the current paper, we will use the baseline model with COVID-19 images from the city of Madrid (located in the center of the Community of Madrid in Spain). We will also use the baseline model when trained with traffic-only images to assess how much COVID-19 information can be extracted by analyzing traffic patterns to complement previous studies such as [32].

4.1. Combined Traffic and COVID-19 Incidence Images Model

In this first version of the proposed model, the COVID-19 incidence and the traffic images for each instant of time are combined into a “n by n by 2” image. A sequence of m consecutive images will be used to feed the model. Figure 3 captures a graphical representation of the model while the implementation details using the Keras [36] library are captured in Figure 4. A convolutional layer is first applied to each of the combined images. The size and the number of filters in the convolutional layer can be optimized to achieve optimal results. After the convolutional layer, a max pooling layer is used to reduce the size of the output. The size of the max pooling layer is another parameter that can be optimized. A final dense layer (or fully connected layer) will provide a summary of the information in the combined image to be used as the input for the time pattern analysis which will be performed using an LSTM-based RNN. The number of memory units in each LSTM cell is another parameter of the model. After the final time step in the RNN, a dense layer will be used to generate a one-week-ahead estimation for the new expected cases for each square in the incidence image.

The model in Figure 3 performs a spatial pattern extraction using filters that combine the traffic volumes and infected cases together. The idea is to use the information about both traffic volumes and COVID-19 incidence for each location when processing the spatial patterns.

The input layer in Figure 4 is divided into five input boxes, each one receives the combined input image for one of the five previous weeks. Each image contains two colors, one for traffic data and the other for COVID-19 data. A different CNN model is applied to each image in Figure 4.

4.2. Independent Traffic and COVID-19 Incidence Images

The second version of the proposed model will extract the spatial patterns for COVID-19 incidence images and traffic images independently using different CNNs. The idea is that each CNN could be fine-tuned to capture the particularities of each type of sensor. Figure 5 captures a graphical representation of this version of the proposed model while Figure 6 captures its implementation in Keras [36]. The summary of the spatial patterns extracted from each image (after the CNN model) at a particular instant of time are fed as inputs to one LSTM cell. LSTM cells are connected to an RNN model. A final dense layer is used as in the previous version of the model to provide a one-week-ahead estimation for the COVID-19 expected new cases.

The input layer in Figure 6 contains 10 input boxes; the first 5 receive the COVID-19 input images for each of the five previous weeks, and the last 5 receive the equivalent traffic images. A different CNN model is applied to each image in Figure 6.

4.3. COVID-19 and Traffic Only Models

The model validated in [29] for COVID-19 images was adapted to the dataset in this paper and used as a baseline model in order to assess the results of the models in Section 4.1 and Section 4.2. This baseline model was also used for learning from traffic-only images in order to assess how much COVID-19 information can be extracted from traffic images. Figure 7 captures the graphical representation of the model while Figure 8 shows an implementation of the model in Keras [36]. The results from the baseline model for COVID-19-only data were compared in [29] with previous research studies proposing different machine learning models for COVID-19 short-term forecasting, showing promising results. The results in the current paper can be compared with previous similar studies using the results in [29].

The input layer in Figure 8 contains five input boxes; each one receives a COVID-19 monochromatic input image for one of the five previous weeks. No traffic information is used in this case. A different CNN model is applied to each image in Figure 8.

5. Datasets

Two major sources of data were used to validate the model proposed in this paper: traffic data from traffic sensors in the city of Madrid and COVID-19 incidence data for the same region as measured by the public health service.

Data gathered by the traffic sensors are obtained from [37]. Data for 4372 sensors since 2013 using a sampling frequency of 15 min are available. The data are provided as separate files per month. Traffic sensors provide information for the intensity of the traffic, the occupation of the road, and the average speed as measured on a particular location of the city. Some of the sensors are located in urban streets of the city of Madrid while others measure traffic data on high-speed segments.

The dataset describing the coordinates where each traffic sensor is located can be downloaded from [38]. Each sensor is identified using a unique identifier that links the data in [38] with the data in [37]. The dataset in [38] also provides descriptive information about each traffic sensor such as the city district in which the sensor is placed and some information about the street that is sensed.

Figure 9 shows the location of the traffic sensors in [37,38]. Different colors are used for urban and highway sensors.

COVID-19 incidence data in the Madrid region are also publicly available as provided by the regional government in [38]. New weekly COVID-19 infection data together with cumulative values are captured. COVID-19 infections are diagnosed and recorded at primary care health centers (each one covering a basic health area or zone). In this way, new infections can be assigned to a particular area in the city. The dataset also contains information about the geographic shape of each health zone. Figure 10 captures the locations for the 143 health zones in Madrid.

Three different consecutive periods of data have been recorded in [39]. Different data-gathering protocols were used for each period. In the first period, which covers the first months of the COVID-19 pandemic until July 2000, the newly detected COVID-19 infections were reported on a daily base. The second period spans from 2 July 2020 to 29 March 2022. In the second period, COVID-19 data was reported on a weekly basis. Finally, the third period started in April 2022 when the majority of COVID-19-related restrictions in Spain were lifted. During the first period, there were not enough PCR tests available, and not all the cases were therefore recorded. During the third period, many cases were not detected or diagnosed and only cases concerning people over 60 years old were PCR tested. For those reasons, only the data in the second period were used in this paper.

6. Results

This section captures the results when applying the models presented in Section 4 to the traffic and COVID-19 incidence images computed from the datasets described in Section 5 using the mechanisms presented in Section 3. The models in Section 4 contain a set of configurable parameters that can be optimized to optimally learn spatial and temporal patterns from the datasets described in Section 5. The first subsection is dedicated to performing a parameter optimization. The best parameters are then used in Section 6.2 to validate the results.

6.1. Parameter Optimization

The models in Figure 3, Figure 5 and Figure 7 have five major parameters that can be optimized:

• Num_filters: the number of filters used by the CNN layer. The higher the number of filters the more spatial patterns can be extracted from the input images but more input data could be needed to prevent overfitting;
• Filter_size: the convolution operation will be applied to a square region in the input image. The filter_size parameter controls the size of the area used in the convolution operation. The models in Section 4 use square filters of filter_size by filter_size that go through the input images to extract the different spatial features. The bigger the size of the filters the higher the impact of faraway locations in the input image but the lower the spatial granularity;
• Pool_size: the max pooling operation performs a summary of the output of the convolution operation. The size of the summary is controlled by the pool_size parameter.
• Num_neurons_input_LSTM: represents the number of neurons at the output of the dense layer after the max pooling layer. The output neurons provide a final summary of the spatial information for each input image at each instant of time and should be able to capture the information required by the LSTM-based RNN at the final part of the models to perform optimal estimations;
• LSTM_units: represents the number of memory units in the LSTM cells. The mission of the memory units is to store the temporal information to be able to perform temporal forecasting. The number of memory units should therefore be enough to extract temporal patterns but not bigger than required in order to minimize overfitting training effects.

The results of the optimization process for the three models using the datasets described in Section 5 are captured in

Table 2. Optimal values for the parameters in the models.

Parameter/Model	Figure 3	Figure 5	Figure 7
Num_filters	4	16	32
Filter_size	3	3	4
Pool_size	2	2	2
Num_neurons_input_LSTM	8	8	16
LSTM_units	16	16	32

. An Adam optimizer was used for the training of the models using 300 epochs and a learning rate of 0.001. Those parameters were tuned to achieve stable convergence of the training process in all cases. The increase in the validation loss after 10 epochs was used to stop the training in order to avoid overfitting. Models in Figure 3 (combined traffic and COVID-19 incidence images) and Figure 5 (independent traffic and COVID-19 incidence images) show similar results for the parameters, except for the number of filters used by the CNN layer. The major difference between models in Figure 3 and Figure 5 is the way in which the CNN layer is applied to the input images and the num_filters parameter is able to capture that difference. The graphical representations for the different Mean Squared Errors (MSE) obtained for different values for the input parameters when applied to optimize the model in Figure 3 are presented in Figure 11 and Figure 12. Figure 11 shows the combined MSE errors for the filter_size and num_neurons_input_LSTM averaged for the rest of the parameters. Figure 12 captures the average results for num_filters parameter for the model in Figure 3. MSE errors varied from 0.004 to 0.0055 for input images that were normalized to the range from 0 to 1 as described in Section 3.

The results in Table 2 show that the baseline model in Figure 7 tends to require a higher number of neurons inside the different layers of the model as compared to the model presented in this paper. The baseline model is a simpler model that only focuses on a single type of image and compensates for the simplicity of the model with higher values for the parameters and operations.

6.2. Validation Results

The results for applying the models described in Section 4 to the dataset described in Section 5 are shown using three different validation strategies: 5-fold cross-validation, 10-fold cross-validation, and removing an entire wave in the spread of the virus when training and using it for validation. The data used for validating the models comprise data from July 2020 to March 2022 as presented in Section 5. This period of time captures data from the second to the sixth wave of COVID-19 cases in the Madrid area (and in Spain in general). The sixth wave shows a slightly different time sequence since it was dominated by the new Omicron variant of the COVID-19 virus and therefore we used the information in the fifth wave to validate the model. Both lockdown policies and vaccination campaigns have had an impact on the spread of the virus [31,32,33,34]. Major lockdowns in Spain were lifted on June 2020 and vaccination campaigns reached the majority of the population by the end of 2021. In this paper, a period of time has been selected after major lockdowns were lifted. In order to avoid the effect of vaccinations on the model performance, the sixth wave has not been used for validation purposes. In future work, lockdown measures or vaccination rates will be added to the proposed models to improve their prediction performance.

The model in Figure 7 (baseline model) is applied independently to COVID-19 incidence images and to traffic intensity images. Using the baseline model for COVID-19 incidence images replicates the results presented in [29] for the particular case of the city of Madrid in order to compare the results obtained by the models proposed in this paper (models in Figure 3 and Figure 5), which are also applied to the datasets described in Section 5 for the city of Madrid. The baseline model was applied to data for the entire community of Madrid in [29] and was compared to previous models found in previous research studies using the same dataset, showing that the spatial information provided by COVID-19 images was able to improve the accuracy of one-week ahead forecasting. The models proposed in this paper add traffic information to the input of the models. This section compares the validation results for the proposed models as compared to the baseline model using the three different validation approaches described in the previous paragraph. The baseline model is also trained and validated using traffic-only information in order to complement the previous study in [29] and provide a baseline about how much COVID-19-related information can be extracted from traffic images.

Figure 13 shows the validation results for the models described in Section 4 applying the 3 different validation strategies. The model in Figure 5 (“independent images”) shows the best results for both 5-fold and 10-fold cross-validation while the model in Figure 3 (“combined images”) is able to outperform the model in Figure 5 when using it to predict a new wave (the 5th wave in the dataset). Both 5-fold and 10-fold cross-validation strategies use random information in the entire dataset for training and validation. The training set therefore will contain information for parts of all the different waves (as well as the validation set which will contain the remaining parts of all the COVID-19 waves). Removing an entire wave from the dataset for training and using that information for validation is a more challenging scenario in which the trained model has not seen the details for an entire segment of the dataset. The MSE errors for both models in Figure 3 and Figure 5 are higher when used to predict the 5th wave but the model in Figure 3 is able to better generalize to this validation scenario.

Figure 13 also captures the validation results for the baseline model (Figure 7) when applied to COVID-19-only and traffic-only images. The validation results show that the COVID-19-only images provide enough information to train the baseline model to get similar results as models in Figure 3 (“combined images”) and Figure 5 (“independent images”) when using 5-fold and 10-fold cross-validation schemes and the MSE errors for the validation set in both cases are similar. However, the validation results when using COVID-19-only images worsened significantly as compared to models in Figure 3 and Figure 5 when predicting the entire fifth wave of the spread of the virus. By extracting traffic spatio-temporal patterns and using them to complement COVID-19 images, models are able to provide better estimates for a new wave of COVID-19 cases (models in Figure 3 and Figure 5). Finally, applying the baseline model to traffic only images provided worse results for all the validation schemes. Traffic information can therefore be used to enhance COVID-19 data but do not provide good enough predictions when used alone.

Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 show the results for the different models when using a 10-fold cross-validation approach for estimating both new cases one week ahead for a particular location (latitude = 40.412585 and longitude = −3.6928138). The results in Figure 14 (for the model in Figure 3), Figure 15 (for the model in Figure 5), and Figure 16 (for the model in Figure 7 when applied to COVID-19-only images) are visually similar in accordance with the results presented in Figure 13 (similar MSE errors are achieved in these scenarios). The results for the same location and using a similar 10-fold cross-validation when applying the baseline model in Figure 7 to traffic only images are captured in Figure 17. The estimations are visually worse than in previous cases. This result shows that traffic-only information does not have enough information in order to provide accurate predictions for new COVID-19 infections.

The results for the models applied to the data when the fifth wave is removed from the training set are presented in Figure 18, Figure 19, Figure 20 and Figure 21. The fifth wave took place between July and August 2021. The data from 22 June to 7 September 2021 are used in this case for validation. Figure 18 captures the use of the combined model for estimating the fifth wave. The results in Figure 18 show that the model can estimate the instances of time when the new COVID cases are going to increase or decrease but the peak levels are not properly predicted. Figure 19 shows similar results for the case of using the model in Figure 5 for estimating the fifth wave for the same location. Results are very similar as expected from MSE error values in Figure 13. Figure 20 captures the results when using the baseline model in Figure 7 with COVID-19-only images. The results in this case worsen as compared with the models using both COVID-19 and traffic data. Traffic data can provide information about the movement of the population in the region under study which helps with the propagation of the COVID-19 virus and is able to improve the estimation of results one week ahead for models in Figure 3 and Figure 5 compared to the baseline model in Figure 7 (using COVID-19 only images). Finally, Figure 21 captures the use of traffic-only images to try to estimate COVID-19 upcoming cases when removing the 5th wave for training. The model is not able to follow the COVID-19 incidence curve in this case. The fact that the fifth wave took place during the summer holiday period significantly affected the traffic patterns and the population movements in and out the Madrid region for the holiday period.

6.3. Model Explanation

The estimation of the importance of the input features in order to explain the achieved prediction results is a key part of explainable AI models. Different methods have been proposed for estimating the importance of the input features in explaining the outputs of AI models such as [40,41]. The best-performing machine learning model presented in this paper, as evaluated in Section 6.2, is the one combining COVID-19 and traffic information into two-colored sequences of images (as presented in Figure 3). The Integrated Gradient method [40] is used in this section in order to assess the importance of the input features since the method scales better for sequences of input images than Shapley values [41].

Integrated gradients (IG) should be calculated for each input feature. In our case, each point in the sequences of COVID-19 and traffic images was an input feature to the forecasting model. Since far-away in-space points are expected to play a less significant contribution to the generated results the IG has been computed for a six-by-six image containing the information in the surrounding area of the geographical location for which COVID-19 incidence will be estimated. An example of the results achieved in the fifth wave for the geographical location used in Section 6.2 is captured in Figure 22 (for the COVID-19 images) and Figure 23 (for the traffic images). The information to be estimated by the model is the COVID-19 incidence for a time t in the fifth wave (July 2021) for the spatial location with indexes (3,3) in a time sequence of input images. The model uses the previous five weeks (from t-1 to t-5) to provide an estimation for the COVID-19 incidence at time t and location with indexes (3,3).

Figure 22 shows that the last week COVID-19 data is the most relevant information for the model in order to estimate the incidence values one week later. Figure 22 also captures the surrounding areas which have a higher impact on the estimation of the result. Figure 23 performs a similar analysis for the traffic data. Again, traffic data one week before show a higher impact on predictions. The traffic in the predicted location shows a significant impact while geographic areas farther in space and time show a smaller contribution to the generated one-week ahead COVID-19 estimations.

6.4. Ablation Study

In order to investigate the performance of the proposed model in Figure 3 when assessing the importance of each internal block in the model in the contribution to the final result, an ablation study has been performed. The combined images (COVID-19 and traffic) model in Figure 3 is based on 5 major building blocks that extract spatial features independently from an input image in the temporal sequence using a CNN which are then fed into an LSTM layer. An ablation study is presented by removing the CNN processing each sequential image at a time and comparing the MSE values with the overall model.

Table 3. Ablation study removing the CNN processing each instant of time in the sequence of combined images.

Block Removed	MSE Values
None	0.003242
CNN processing image at t-5	0.003245
CNN processing image at t-4	0.004514
CNN processing image at t-3	0.005416
CNN processing image at t-2	0.005744
CNN processing image at t-1	0.006095

captures the results for the MSE values when removing the different CNN blocks in Figure 4. The optimal value for the MSE is when the entire model is preserved. When removing the processing of an input instant of time in the image sequence processing, the most important blocks are those processing images which are closer in time to the image being forecasted. These results are aligned with the results in Figure 22 and Figure 23 in which information in closer images in time is able to better explain the results of the model.

7. Conclusions

This paper proposed and validated a new machine learning model able to combine spatio-temporal traffic information with COVID-19 data in order to forecast new infections one week in advance. The model is based on sequences of traffic and COVID-19 incidence images that are generated from geolocated sensor data. The results are compared with a baseline model based on a single type of images (either traffic or COVID-19 incidence images). The baseline model has already been validated in previous research when based on COVID-19 images and showed better results than non-spatial temporal machine learning models for COVID-19 new cases estimation. Both n-fold cross-validation and leave-one-wave-out validation approaches have been used for all cases (using combined traffic and COVID-19 incidence images, independent images, or traffic or incidence images alone). Leaving an entire wave of infections out from the training data and using the trained model to estimate that wave of information provides worse estimation results in all the cases (showing that the time series information is not fully time independent). Using traffic data does not have a significant impact when using a 5-fold or 10-fold cross-validation strategy. However, traffic information is able to increase the accuracy of one week ahead of new COVID-19 case estimations when using the models to predict an entire wave of the pandemic spread. Traffic data is therefore able to add the required information in the input to the machine learning model to compensate for the time dependencies in the COVID-19 alone data.

The results of the paper also show that traffic-only images do not provide enough information for accurate estimations in the evolution of the pandemic. Traffic images will be combined with other mobility modalities in future work to better estimate human mobility and its contribution to the spread of the virus. As a future work, models able to consider air traffic and long traffic movements in and out of the Madrid area will be developed. The proposed model is likely to apply to the forecasting of similar respiratory viruses. As a future study, the model will be applied to influenza and other examples.

The major limitation of the proposed method is the availability of homogeneous data over time and space. Several methods have been used for COVID-19 data collection as described in the paper and traffic sensors are continually been added and removed to monitor the traffic in Madrid. As a future study, a method to homogenize data will be developed, which will make the model more generalizable to other zones.