A Machine Learning Approach for Air Quality Prediction: Model Regularization and Optimization

Abstract

In this paper, we tackle air quality forecasting by using machine learning approaches to predict the hourly concentration of air pollutants (e.g., ozone, particle matter (${\mathrm{PM}}_{2.5}$) and sulfur dioxide). Machine learning, as one of the most popular techniques, is able to efficiently train a model on big data by using large-scale optimization algorithms. Although there exist some works applying machine learning to air quality prediction, most of the prior studies are restricted to several-year data and simply train standard regression models (linear or nonlinear) to predict the hourly air pollution concentration. In this work, we propose refined models to predict the hourly air pollution concentration on the basis of meteorological data of previous days by formulating the prediction over 24 h as a multi-task learning (MTL) problem. This enables us to select a good model with different regularization techniques. We propose a useful regularization by enforcing the prediction models of consecutive hours to be close to each other and compare it with several typical regularizations for MTL, including standard Frobenius norm regularization, nuclear norm regularization, and ${\ell }_{2,1}$-norm regularization. Our experiments have showed that the proposed parameter-reducing formulations and consecutive-hour-related regularizations achieve better performance than existing standard regression models and existing regularizations.

1. Introduction

Adverse health impacts from exposure to outdoor air pollutants are complicated functions of pollutant compositions and concentrations [1]. Major outdoor air pollutants in cities include ozone (${\mathrm{O}}_3$), particle matter (PM), sulfur dioxide (${\mathrm{SO}}_2$), carbon monoxide (CO), nitrogen oxides (${\mathrm{NO}}_{\mathrm{x}}$), volatile organic compounds (VOCs), pesticides, and metals, among others [2,3]. Increased mortality and morbidity rates have been found in association with increased air pollutants (such as ${\mathrm{O}}_3$, PM and ${\mathrm{SO}}_2$) concentrations [3,4,5]. According to the report from the American Lung Association [6], a 10 parts per billion (ppb) increase in the ${\mathrm{O}}_3$ mixing ratio might cause over 3700 premature deaths annually in the United States (U.S.). Chicago, as for many other megacities in U.S., has struggled with air pollution as a result of industrialization and urbanization. Although ${\mathrm{O}}_3$ precursor (such as VOCs, ${\mathrm{NO}}_{\mathrm{x}}$, and CO) emissions have significantly decreased since the late 1970s, ${\mathrm{O}}_3$ levels in Chicago have not been in compliance with standards set by the Environmental Protection Agency (EPA) to protect public health [7]. Particle size is critical in determining the particle deposition location in the human respiratory system [8]. ${\mathrm{PM}}_{2.5}$, referring to particles with a diameter less than or equal to 2.5 $\mathsf{\mu }$m, has been an increasing concern, as these particles can be deposited into the lung gas-exchange region, the alveoli [9]. The U.S. EPA revised the annual standard of ${\mathrm{PM}}_{2.5}$ by lowering the concentration to 12 $\mathsf{\mu }$g/m${}^3$ to provide improved protection against health effects associated with long- and short-term exposure [10]. ${\mathrm{SO}}_2$, as an important precursor of new particle formation and particle growth, has also been found to be associated with respiratory diseases in many countries [11,12,13,14,15]. Therefore, we selected ${\mathrm{O}}_3$, ${\mathrm{PM}}_{2.5}$ and ${\mathrm{SO}}_2$ for testing in this study.

Meteorological conditions, including regional and synoptic meteorology, are critical in determining the air pollutant concentrations [16,17,18,19,20,21]. According to the study by Holloway et al. [22], the ${\mathrm{O}}_3$ concentration over Chicago was found to be most sensitive to air temperature, wind speed and direction, relative humidity, incoming solar radiation, and cloud cover. For example, a lower ambient temperature and incoming solar radiation slow down photochemical reactions and lead to less secondary air pollutants, such as ${\mathrm{O}}_3$ [23]. Increasing wind speed could either increase or decrease the air pollutant concentrations. For instance, when the wind speed was low (weak dispersion/ventilation), the pollutants associated with traffic were found at the highest concentrations [24,25]. However, strong wind speeds might form dust storms by blowing up the particles on the ground [26]. High humidity is usually associated with high concentrations of certain air pollutants (such as PM, CO and ${\mathrm{SO}}_2$) but with low concentrations of other air pollutants (such as ${\mathrm{NO}}_2$ and ${\mathrm{O}}_3$) because of various formation and removal mechanisms [25]. In addition, high humidity can be an indicator of precipitation events, which result in strong wet deposition leading to low concentrations of air pollutants [27]. Because various particle compositions and their interactions with light were found to be the most important factors in attenuating visibility [28,29], low visibility could be an indicator of high PM concentrations. Cloud can scatter and absorb solar radiation, which is significant for the formation of some air pollutants (e.g., ${\mathrm{O}}_3$) [23,30]. Therefore, these important meteorological variables were selected to predict air pollutant concentrations in this study.

Statistical models have been applied for air pollution prediction on the basis of meteorological data [31,32,33,34,35]. However, existing studies on statistical modeling have mostly been restricted to simply utilizing standard classification or regression models, which have neglected the nature of the problem itself or ignored the correlation between sub-models in different time slots. On the other hand, machine learning approaches have been developing for over 60 years and have achieved tremendous success in a variety of areas [36,37,38,39,40,41]. There exist various new tools and techniques invented by the machine learning community, which allow for more refined modeling of a specific problem. In particular, model regularization is a fundamental technique for improving the generalization performance of a predictive model. Accordingly, many efficient optimization algorithms have been developed for solving various machine learning formulations with different regularizations.

In this study, we focus on refined modeling for predicting hourly air pollutant concentrations on the basis of historical metrological data and air pollution data. A striking difference between this work and the previous works is that we emphasize how to regularize the model in order to improve its generalization performance and how to learn a complex regularized model from big data with advanced optimization algorithms. We collected 10 years worth of meteorological and air pollution data from the Chicago area. The air pollutant data was from the EPA [42,43], and the meteorological data was from MesoWest [44]. From their databases, we fetched consecutive hourly measurements of various meteorological variables and pollutants reported by two air quality monitoring stations and two air pollutant monitoring sites in the Chicago area. Each record of hourly measurements included meteorological variables such as solar radiation, wind direction and speed, temperature, and atmospheric pressure; as well as air pollutants, including ${\mathrm{PM}}_{2.5}$, ${\mathrm{O}}_3$, and ${\mathrm{SO}}_2$. We used two methods for model regularization: (i) explicitly controlling the number of parameters in the model; (ii) explicitly enforcing a certain structure in the model parameters. For controlling the number of parameters in the model, we compared three different model formulations, which can be considered in a unified multi-task learning (MTL) framework with a diagonal- or full-matrix model. For enforcing the model matrix into a certain structure, we have considered the relationship between prediction models of different hours and compared three different regularizations with standard Frobenius norm regularization. The experimental results show that the model with the intermediate size and the proposed regularization, which enforces the prediction models of two consecutive hours to be close, achieved the best results and was far better than standard regression models. We have also developed efficient optimization algorithms for solving different formulations and demonstrated their effectiveness through experiments.

The rest of the paper is organized as follows. In Section 2, we discuss related work. In Section 3, we describe the data collection and preprocessing. In Section 4, we describe the proposed solutions, including formulations, regularizations and optimizations. In Section 5, we present the experimental studies and the results. In Section 6, we give conclusions and indicate future work.

2. Related Work

Many previous works have been proposed to apply machine learning algorithms to air quality predictions. Some researchers have aimed to predict targets into discretized levels. Kalapanidas et al. [32] elaborated effects on air pollution only from meteorological features such as temperature, wind, precipitation, solar radiation, and humidity and classified air pollution into different levels (low, med, high, and alarm) by using a lazy learning approach, the case-based reasoning (CBR) system. Athanasiadis et al. [45] employed the $\sigma$-fuzzy lattice neurocomputing classifier to predict and categorize O${}_3$ concentrations into three levels (low, mid, and high) on the basis of meteorological features and other pollutants such as ${\mathrm{SO}}_2$, NO, ${\mathrm{NO}}_2$, and so on. Kurt and Oktay [33] modeled geographic connections into a neural network model and predicted daily concentration levels of SO${}_2$, CO, and PM${}_{10}$ 3 days in advance. However, the process of converting regression tasks to classification tasks is problematic, as it ignores the magnitude of the numeric data and consequently is inaccurate.

Other researchers have worked on predicting concentrations of pollutants. Corani [46] worked on training neural network models to predict hourly ${\mathrm{O}}_3$ and PM${}_{10}$ concentrations on the basis of data from the previous day. Mainly compared were the performances of feed-forward neural networks (FFNNs) and pruned neural networks (PNNs). Further efforts have been made on FFNNs: Fu et al. [47] applied a rolling mechanism and gray model to improve traditional FFNN models. Jiang et al. [48] explored multiple models (physical and chemical model, regression model, and multiple layer perceptron) on the air pollutant prediction task, and their results show that statistical models are competitive with the classical physical and chemical models. Ni, X. Y. et al. [49] compared multiple statistical models on the basis of ${\mathrm{PM}}_{2.5}$ data around Beijing, and their results implied that linear regression models can in some cases be better than the other models.

MTL focuses on learning multiple tasks that have commonalities [50] that can improve the efficiency and accuracy of the models. It has achieved tremendous successes in many fields, such as natural language processing [37], image recognition [38], bioinformatics [39,40], marketing prediction [41], and so on. A variety of regularizations can be utilized to enhance the commonalities of the related tasks, including the ${\ell }_{2,1}$-norm [51], nuclear norm [52], spectral norm [53], Frobenius norm [54], and so on. However, most of the former machine learning works on air pollutant prediction did not consider the similarities between the models and only focused on improving the model performance for a single task, that is, improving prediction performance for each hour either separately or identically.

Therefore, we decided to use meteorological and pollutant data to perform predictions of hourly concentrations on the basis of linear models. In this work, we focused on three different prediction model formulations and used the MTL framework with different regularizations. To the best of our knowledge, this is the first work that has utilized MTL for the air pollutant prediction task. We exploited analytical approaches and optimization techniques to obtain the optimal solutions. The model’s evaluation metric was the root-mean-squared error (RMSE).

3. Data Collection and Preprocessing

3.1. Data Collection

We collected air pollutant data from two air quality monitoring sites and meteorological data from two weather stations from 2006 to 2015 (summarized in

Table 1. Summary of measurement sites and observed variables.

Measurement Sites	Variables
Alsip Village (AV)	Ozone concentration and PM${}_{2.5}$ concentration
Lemont Village (LV)	Ozone concentration and sulfur dioxide concentration
Lansing Municipal Airport (LMA)	Temperature, relative humidity, wind speed and direction, wind gust, precipitation accumulation, visibility, dew point, wind cardinal direction, pressure, and weather conditions
Lewis University (LU)	The same as for LMA site

). The air pollutant data in this study included the concentrations of ${\mathrm{O}}_3$, ${\mathrm{PM}}_{2.5}$ and ${\mathrm{SO}}_2$. We downloaded the air pollutant data from the U.S. EPA’s Air Quality System (AQS) database (https://www.epa.gov/outdoor-air-quality-data), which has been widely used for model evaluation [42,43]. We selected the meteorological variables that would affect the air pollutant concentrations, including air temperature, relative humidity, wind speed and direction, wind gust, precipitation accumulation, visibility, dew point, wind cardinal direction, pressure, and weather conditions. We downloaded the meteorological data from MesoWest (http://mesowest.utah.edu/), a project within the Department of Meteorology at the University of Utah, which has been aggregating meteorological data since 2002 [44].

The locations of the two air quality monitoring sites and two weather stations are shown in Figure 1. The Alsip Village (AV) air quality monitoring site is also located in a suburban residential area, which is in southern Cook County, Illinois (AQS ID: 17-031-0001; latitude/longitude: 41.670992/−87.732457. The Lemont Village (LV) air quality monitoring site is located in a suburban residential area, which is in southwestern Cook County, Illinois (AQS ID: 17-031-1601; latitude/longitude: 41.66812/−87.99057. The weather station situated in Lansing Municipal Airport (LMA) is the closest meteorological site (MesoWest ID: KIGQ; latitude/longitude: 41.54125/−87.52822) to the AV air quality monitoring site. The weather station positioned at Lewis University (LU) is the closest meteorological site (MesoWest ID: KLOT; latitude/longitude: 41.60307/−88.10164) to the LV air quality monitoring site.

3.2. Preprocessing

We paired the collected meteorological data and air pollutant data on the basis of time to obtain the required data format for applying the machine learning methods. In particular, for each variable, we formed one value for each hour. However, the original data may have contained multiple records or missing values at some hours. To preprocess the data, we calculated the hourly mean value of each numeric variable if there were multiple observed records within an hour and chose the category with the highest frequency per hour for each categorical variable if there were multiple values. Missing values existed for some variables, which was not tolerable for applying the machine learning methods used in this study. Therefore, we imputed the missing values by using the closest-neighbor values for four continuous variables and one categorical variable: wind gust, pressure, altimeter reading, precipitation, and weather conditions. We deleted the days that still had missing values after imputing. We applied dummy coding for two categorical variables, the cardinal wind direction (16 values, e.g., N, S, E, W, etc.) and weather conditions (31 values, e.g., sunny, rainy, windy, etc.). Then, we added the weekday and weekend as two boolean features. Finally, we obtained 60 features in total (9 numerical meteorological features, 16 dummy codings for wind direction, 31 dummy codings for weather conditions, 2 boolean features for weekday/weekend, 1 numerical feature for pollutants, and 1 bias term). We applied normalization for all the features and pollutant targets to make their values fall in the range $[0,1]$.

4. Machine Learning Approaches for Air Pollution Prediction

In this section, we describe the proposed approaches for predicting the ambient concentration of air pollutants.

4.1. A General Formulation

Our goal is to predict the concentration of air pollutants of the next day on the basis of the historical meteorological and air pollutant data. In this work, we have focused on using the former day’s data to predict the next day’s hourly pollutants. In particular, we let $({\mathbf{x}}_i;y_i)$ denote the ith training data, where $y_i\in {\mathbb{R}}^{24\times 1}$ denotes the concentration of a certain air pollutant on a day, and ${\mathbf{x}}_i=({\mathbf{u}}_i;{\mathbf{v}}_i)$ denotes the observed data on the previous day that include two components, where a semicolon “;” represents the column layout. The first component ${\mathbf{u}}_i=({\mathbf{u}}_{i,1};\dots ;{\mathbf{u}}_{i,D})\in {\mathbb{R}}^{24·D\times 1}$ includes all meteorological data over 24 h for the previous day, where ${\mathbf{u}}_{i,j}\in {\mathbb{R}}^{24\times 1}$ denotes the jth meteorological feature of the 24 h and D is the number of meteorological features; the second component ${\mathbf{v}}_i\in {\mathbb{R}}^{24\times 1}$ includes the hourly concentration of the same air pollutant on the previous day. The general formulation can be expressed as

$$\begin{array}{c}\hfill \underset{W}{min}\frac{1}{n}\sum _{i=1}^n{\parallel f(W,{\mathbf{x}}_i)-y_i\parallel }_2^2+\phi \left(W\right)\end{array}$$

(1)

where W denotes the parameters of the model, $f(W,{\mathbf{x}}_i)$ denotes the prediction of the air pollutant concentration, and $\phi (·)$ denotes a regularization function of the model parameters W.

Next, we introduce two levels of model regularization. The first level is to explicitly control the number of model parameters. The second level is to explicitly impose a certain regularization on the model parameter. For the first level, we consider three models that are described below:

• Baseline Model. The first model is a baseline model that has been considered in existing studies and has the fewest number of parameters. In particular, the prediction of the air pollutant concentration is given by

  $$\begin{array}{c}\hfill f_k(W,{\mathbf{x}}_i)=\sum _{j=1}^D{\mathbf{e}}_k^{\top }{\mathbf{u}}_{i,j}·w_j+{\mathbf{e}}_k^{\top }{\mathbf{v}}_i·w_{D+1}+w_0,k=1,\dots ,24\end{array}$$

  where ${\mathbf{e}}_k\in {\mathbb{R}}^{24\times 1}$ is a basis vector with 1 at only the kth position and 0 at other positions; $w_0,w_1,\dots ,w_D,w_{D+1}\in \mathbb{R}$ are the model parameters, where $w_0$ is the bias term. We denote this model by $W={(w_0,w_1,\dots ,w_{D+1})}^{\top }$. It is notable that this model predicts the hourly concentration on the basis of the same hourly historical data of the previous day and that it has $D+2$ parameters. This simple model assumes that all 24 h share the same model parameter.
• Heavy Model. The second model takes all the data of the previous day into account when predicting the concentration of every hour of the second day. In particular, for the kth hour, the prediction is given by

  $$\begin{array}{c}\hfill f_k(W,{\mathbf{x}}_i)=\sum _{j=1}^D{{\mathbf{u}}_{i,j}}^{\top }{\mathbf{w}}_{k,j}+{\mathbf{v}}_i^{\top }{\mathbf{w}}_{k,D+1}+w_{k,0},k=1,\dots ,24\end{array}$$

  where ${\mathbf{w}}_{k,j}\in {\mathbb{R}}^{24\times 1},j=1,\dots ,D+1$ and $w_{k,0}\in \mathbb{R}$. This model is defined by

  $$\begin{array}{c}\hfill W=\left[\begin{array}{cccc}{w}_{1,0}& w_{2,0}& \dots & w_{24,0}\\ {\mathbf{w}}_{1,1}& {\mathbf{w}}_{2,1}& \dots & {\mathbf{w}}_{24,1}\\ \dots & \dots & \dots & \dots \\ {\mathbf{w}}_{1,D+1}& {\mathbf{w}}_{2,D+1}& \dots & {\mathbf{w}}_{24,D+1}\end{array}\right]\end{array}$$
  We note that each column of W corresponds to the prediction model for each hour. There are a total of $24\times$ (24 $\times (D+1)+1)$ parameters. It is notable that the baseline model is a special case by enforcing all columns of W to be the same and because each ${\mathbf{w}}_{k,j}$ has only one non-zero element at the kth position.
• Light Model. The third model is between the baseline model and the heavy model. It considers the 24 h pattern of the air pollutants in the previous day and the same hourly meteorological data of the previous day to predict the concentration at a particular hour. The prediction is given by

  $$\begin{array}{c}\hfill f_k(W,{\mathbf{x}}_i)=\sum _{j=1}^D{\mathbf{e}}_k^{\top }{\mathbf{u}}_{i,j}·w_{k,j}+{\mathbf{v}}_i^{\top }{\mathbf{w}}_{k,D+1}+w_{k,0},k=1,\dots ,24\end{array}$$

  where $w_{k,j}\in \mathbb{R},j=1,\dots ,D$ and ${\mathbf{w}}_{k,D+1}\in {\mathbb{R}}^{24\times 1}$. This model is defined by

  $$\begin{array}{c}\hfill W=\left[\begin{array}{cccc}{w}_{1,0}& w_{2,0}& \dots & w_{24,0}\\ w_{1,1}& w_{2,1}& \dots & w_{24,1}\\ \dots & \dots & \dots & \dots \\ {\mathbf{w}}_{1,D+1}& {\mathbf{w}}_{2,D+1}& \dots & {\mathbf{w}}_{24,D+1}\end{array}\right]\end{array}$$
  It is also notable that each column corresponds to the predictive model for one hour and that W has a total of $24\times (D+1)+24\times 24\times 1$ parameters.

4.2. Regularization of Model Parameters

In this section, we describe different regularizations for the model parameter matrices W in the heavy and light models. We consider the problem using MTL, in which predicting the concentration of air pollutants over one hour is one task. In the literature, a number of regularizations have been proposed by considering the relationship between different tasks. We first describe three baseline regularizations in the literature and then present the proposed regularization that takes the dimension of time into consideration for modeling the relationship between models at different times.

• Frobenius norm regularization. Frobenius norm regularization is a generalization of standard Euclidean norm regularization to the matrix case, for which

  $$\phi \left(W\right)={\lambda \left|\right|W\left|\right|}_F^2$$

  where $\lambda >0$ is a regularization parameter.
• ${\ell }_{\mathbf{2},\mathbf{1}}$-norm regularization. ${\ell }_{2,1}$-norm regularization has been used for feature selection in MTL. The norm is formed by first computing the ${\ell }_2$-norm of each row of the W matrix (across different tasks) and then computing the ${\ell }_1$-norm of the resulting vector. In particular, for $W\in {\mathbb{R}}^{d\times K}$,

  $$\begin{array}{c}\hfill {\parallel W\parallel }_{2,1}=\sum _{j=1}^d{\parallel W_{j,\ast }\parallel }_2\end{array}$$

  where $W_{j,\ast }$ denotes the jth row of W. We consider a ${\ell }_{2,1}$-norm regularizer $\phi \left(W\right)={\lambda \parallel W\parallel }_{2,1}$.
• Nuclear norm regularization. The nuclear norm is defined as the sum of singular values of a matrix, which is a standard regularization for enforcing a matrix to have a low rank. The motivation for using a low-rank matrix is that models for consecutive hours are highly correlated, which could render the matrix W to be low rank. We denote by ${\parallel W\parallel }_{\ast }$ the nuclear norm of a matrix W; the regularization is $\phi \left(W\right)={\lambda \parallel W\parallel }_{\ast }$.
• Consecutive close (CC) regularization. Finally, we propose a useful regularization for the considered problem that explicitly enforces the predictive models for two consecutive hours to be close to each other. The intuition is that usually the concentrations of air pollutants for two consecutive hours are close to each other. We denote the model by $W=({\mathbf{w}}_1,\dots ,{\mathbf{w}}_K)$ and by $Cons\left(W\right)=[({\mathbf{w}}_1-{\mathbf{w}}_2),({\mathbf{w}}_2-{\mathbf{w}}_3),\dots ,({\mathbf{w}}_{K-1}-{\mathbf{w}}_K)]$. The CC regularization is given by

  $$\begin{array}{c}\hfill \phi \left(W\right)=\lambda \sum _{j=1}^{K-1}{\parallel {\mathbf{w}}_j-{\mathbf{w}}_{j+1}\parallel }_p^p\end{array}$$

  (2)

  where $p=1$ or $p=2$.

4.3. Stochastic Optimization Algorithms for Different Formulations

With the exception that the Frobenius norm regularized model (with ${\ell }_2$-norm CC regularization or not) has a closed-form solution, we solved the other models via advanced stochastic optimization techniques. We denote the following: $F(W,{\mathbf{x}}_i)=[f_1(W,{\mathbf{x}}_i),\dots ,f_{24}(W,{\mathbf{x}}_i)]$ and $Y_i=[y_{i,1},\dots ,y_{i,24}]$; the total number of features is D. Although the standard stochastic (sub)gradient method [55] could be utilized to solve all the formulations considered in this work, it does not necessary yield the fastest convergence. To address this issue, we considered advanced stochastic optimization techniques tailored for solving each formulation.

4.3.1. Optimizing ${\ell }_{2,1}$-Norm Regularized Model

We utilized the accelerated stochastic subgradient (ASSG) method [56] with proximal mapping to optimize this model. The algorithm runs in mutliple stages, and each stage calls the standard stochastic gradient method with a constant step size. To handle the non-smooth ${\ell }_{2,1}$-norm, we used proximal mapping [57]. The stochastic gradient descent part is

$$W_t^{\prime }=W_{t-1}-2{\eta }_s\frac{\partial F(W_{t-1},{\mathbf{x}}_i)}{\partial W_{t-1}}{\mathbf{e}}^{\top }(F(W_{t-1},{\mathbf{x}}_i)-Y_i)$$

(3)

where ${\eta }_s$ is the stage-wise step size, i is a sampled index, and $\mathbf{e}$ is a vector with 1 for all its elements. Then a proximal mapping is as follows (denoted by $\tilde{\lambda }=2{\eta }_s\lambda$):

$$W_t=arg\underset{W}{min}\parallel W-{W^{\prime }}_t{\parallel }_F^2+\tilde{\lambda }{\parallel W\parallel }_{2,1}$$

(4)

The above problem has analytical solutions. We denote ${\mathbf{w}}_i$ as a column vector for $W^{\top }$ and ${\mathbf{w}}_i^{\prime }$ as a column vector for ${W^{\prime }}_t^{\top }$. Then the solution to Equation (4) can be computed by the following [51]:

$${\mathbf{w}}_i=\left\{\begin{array}{cc}\hfill (1-\frac{\tilde{\lambda }}{\parallel {\mathbf{w}}_i^{\prime }{\parallel }_2}){\mathbf{w}}_i^{\prime },& \tilde{\lambda }>0,{\parallel {\mathbf{w}}_i^{\prime }\parallel }_2>\tilde{\lambda }\hfill \\ \hfill \mathbf{0},& \tilde{\lambda }>0,{\parallel {\mathbf{w}}_i^{\prime }\parallel }_2\le \tilde{\lambda }\hfill \\ \hfill {\mathbf{w}}_i^{\prime },& \tilde{\lambda }=0\hfill \end{array}\right.$$

(5)

The pseudocode of the algorithm is as follows:

Algorithm 1: ASSG method with proximal mapping solving ${\ell }_{2,1}$-norm regularized model.

Input: X, Y, $W_0$, ${\eta }_0$, S, and T

4.3.2. Optimizing Nuclear Norm Regularized Model

The challenge in solving the nuclear norm reguralized problem of most optimization algorithms lies with computing the full singular value decomposition (SVD) of the involved matrix W, which is an expensive operation. To avoid full SVD, the SVD-free convex–concave algorithm extension to a stochastic setting (SECONE-S) [58] was employed to solve the problem. The algorithm solves the following minimum–maximum problem:

$$\underset{W\in {\mathbb{R}}^{D\times K}}{min}\underset{U\in {\mathbb{R}}^{D\times K}}{max}\frac{1}{n}\sum _{i=1}^n\parallel F(W,{\mathbf{x}}_i)-Y_i{\parallel }_2^2+\lambda tr\left(U^{\top }W\right)-{\rho [\parallel U\parallel }_2{-1]}_{+}$$

Then stochastic gradient descent and ascent are used to update W and U at each iteration:

$$\begin{array}{cc}\hfill W_t& =W_{t-1}-{\eta }_{t-1}(2\frac{\partial F(W_{t-1},{\mathbf{x}}_i)}{\partial W_{t-1}}{\mathbf{e}}^{\top }(F(W_{t-1},{\mathbf{x}}_i)-Y_i)+\lambda U_{t-1})\hfill \\ \hfill U_t& =U_{t-1}+{\tau }_{t-1}(\lambda W_{t-1}-\rho \partial [\parallel U_{t-1}{\parallel }_2-1{]}_{+})\hfill \end{array}$$

(6)

where $\rho \ge {\parallel Y\parallel }_F^2$ and $\partial [\parallel U_t{{\parallel }_2-1]}_{+}$ can be computed by ${\mathbf{u}}_1{\mathbf{v}}_1^{\top }\mathbf{1}[{\sigma }_1>1]$, with $({\mathbf{u}}_1,{\mathbf{v}}_1)$ being the top-left and -right singular vectors of $U_t$ and ${\sigma }_1$ being the top singular value. The pseudocode for the algorithm is as follows:

Algorithm 2: SECONE-S solving nuclear norm regularized model.

Input: X, Y, T, ${\eta }_0$, and ${\tau }_0$

4.3.3. Optimizing Consecutive Close Regularized Model

The challenge of tackling the proposed CC regularization lies in that the standard proximal mapping cannot be computed efficiently. We addressed this challenge by using the alternating-direction method of multipliers. We utilized a recently proposed locally adaptive stochastic alternating-direction method of multipliers (LA-SADMM) [59] to solve the CC regularized model. Below, we discuss the updates for the choice of $p=1$ (i.e., using the ${\ell }_1$-norm) in Equation (2). The updates for the choice of $p=2$ can be derived similarly.

The objective function can be written as

$$\underset{W\in {\mathbb{R}}^{D\times K}}{min}\frac{1}{n}\sum _{i=1}^n\parallel F(W,{\mathbf{x}}_i)-Y_i{\parallel }_2^2+\lambda {\parallel WE\parallel }_{1,1}$$

Here, $E=({\widehat{\mathbf{e}}}_1,\dots ,{\widehat{\mathbf{e}}}_{k-1})$, where ${\widehat{\mathbf{e}}}_i={(0,\dots ,1,-1,\dots ,0)}^T,i=1,\dots ,k-1$, the ith element is 1 and the ($i+1$)th element is $-1$. Therefore, $Cons\left(W\right)=WE$. A dummy variable $U=WE$ was introduced to decouple the last term from the first term, and a Lagrangian function was formed as follows:

$$\begin{array}{c}\hfill L(W,U,\Lambda )=\frac{1}{n}\sum _{i=1}^n\parallel F(W,{\mathbf{x}}_i)-Y_i{\parallel }_2^2+{\lambda \parallel U\parallel }_{1,1}-\mathrm{tr}\left({\Lambda }^{\top }(WE-U)\right)+\frac{\beta }{2}{\parallel WE-U\parallel }_F^2\end{array}$$

(7)

where $\Lambda$ is the Lagrangian multiplier and $\beta$ is the penalty parameter.

This could then be solved by optimizing each variable alternatively. The update rules for SADMM are as follows:

$$\begin{array}{c}{W}_{\tau }=arg\underset{W\in {\mathbb{R}}^{D\times K}}{min}L(W,U_{\tau -1},{\Lambda }_{\tau -1})=arg\underset{W\in {\mathbb{R}}^{D\times K}}{min}\tilde{F}(W_{\tau -1},{\mathbf{x}}_i)+\mathrm{tr}r\{{\frac{\partial \tilde{F}(W_{\tau -1},{\mathbf{x}}_i)}{\partial W}}^{\top }(W-W_{\tau -1})\}\hfill \\ +\frac{\beta }{2}{\parallel WE-U_{\tau -1}-\frac{1}{\beta }{\Lambda }_{\tau -1}^T\parallel }_F^2+\frac{\parallel W-W_{\tau -1}{\parallel }_F^2}{{\eta }_{\tau -1}}\hfill \\ U_{\tau }=arg\underset{U\in {\mathbb{R}}^{D\times K}}{min}L(W_{\tau },U,{\Lambda }_{\tau -1})=arg\underset{U\in {\mathbb{R}}^{D\times K}}{min}{\gamma \parallel U\parallel }_{1,1}+\frac{\beta }{2}{\parallel W_{\tau }E-U-\frac{1}{\beta }{\Lambda }_{\tau -1}^T\parallel }_F^2\hfill \\ {\Lambda }_{\tau }={\Lambda }_{\tau -1}-\beta {(W_{\tau }E-U_{\tau })}^T\hfill \end{array}$$

(8)

where $\tilde{F}(W_{\tau -1},{\mathbf{x}}_i)={\parallel F(W_{\tau -1},{\mathbf{x}}_i)-Y_i\parallel }_2^2$.

LA-SADMM solves the problem more efficiently by doing stage-wise penalty increasing. The pseudocode for the algorithm is as follows:

Algorithm 3: LA-SADMM solving consecutive close (CC) regularized problem with ${\ell }_1$-norm.

Input: X, Y, $W_0$, $U_0$, ${\Lambda }_0$, ${\beta }_1$, ${\eta }_1$, S, and T

4.4. Extensive Discussion

It is noteworthy that the main contribution of this work is the incorporation of model parameter reduction and MTL with regularization into air pollutant prediction. As the previous content has illustrated, for the parameter reduction part, our light formulation reduces model parameters by removing heavy meteorological parameters of the other hours for one hour’s submodel. For the MTL part, we considered that there could be some similarities for consecutive hours’ models; therefore, we could add appropriate regularizers for this purpose.

The high-level idea of MTL lies in transfer learning, which generally aims to transfer knowledge from a related source task to a target task and consequently improve the performance for the target task. There are multiple variants for transfer learning, such as inductive transfer learning, transductive transfer learning and unsupervised transfer learning, and the approaches for transfer learning mainly include instance transfer, feature-representation transfer, parameter transfer and relational-knowledge transfer [60]. One of the most common examples is feature-representation transfer for deep neural networks. After either supervised or unsupervised learning from other related datasets, the pretrained model can be appropriately reused for learning the target task with a better performance. The MTL technique in this work is an example of parameter transfer in an inductive-transfer-learning setting.

A similar idea can be applied to other kinds of work. First, if the submodels are not built for each hour but for each day (or even for each location from a spatial perspective), we can still apply the parameter reduction idea that only keeps more important information and removes the information with low priority. Second, for the MTL part, we can still add regularizations for the similarities of the submodels. Furthermore, in this work, the submodel ${\mathbf{w}}_i$ was a linear regression model; it is also practical to replace it with support vector regression (SVR), nonlinear regression, neural networks, and so on. Finally, the techniques used in this work can be further combined with many other transfer learning techniques, such as feature-representation transfer for deep neural networks.

5. Experiments

We used the names of the paired air quality monitoring sites and two weather stations to denote the two datasets, that is, LU–LV and LMA–AV. LU–LV contained the data to predict the concentration of the two air pollutants ${\mathrm{O}}_3$ and ${\mathrm{SO}}_2$. LMA–AV contained the data to predict the concentration of the two air pollutants ${\mathrm{O}}_3$ and ${\mathrm{PM}}_{2.5}$.

We compared 11 different models that were learned with different combinations of model formulations and regularizations. The 11 models were the following:

• Baseline: the baseline model with standard Frobenius norm regularization.
• Heavy–F: the heavy model with standard Frobenius norm regularization.
• Light–F: the heavy model with standard Frobenius norm regularization.
• Heavy–${\ell }_{2,1}$: the heavy model with ${\ell }_{2,1}$-norm regularization.
• Heavy–nuclear: the heavy model with nuclear-norm regularization.
• Heavy–CCL2: the heavy model with CC regularization using the ${\ell }_2$-norm.
• Heavy–CCL1: the heavy model with CC regularization using the ${\ell }_1$-norm.
• Light–${\ell }_{2,1}$: the light model with ${\ell }_{2,1}$-norm regularization.
• Light–nuclear: the light model with nuclear-norm regularization.
• Light–CCL2: the light model with CC regularization using the ${\ell }_2$-norm.
• Light–CCL1: the light model with CC regularization using the ${\ell }_1$-norm.

It is noteworthy that we also added the standard Frobenius norm regularizer for the heavy/light–nuclear, –CCL2, and –CCL1 models, because their regularizers were mainly considered for controlling the similarities of submodels and may not have been enough for preventing overfitting. We divided each dataset into two parts: training data and testing data. Each model was trained on the training data with proper regularization parameters and the learning rate selected on the basis of 5-fold cross-validation. Each trained model was evaluated on the testing data. The splitting of the data was done by dividing all days into a number of chunks of 11 consecutive days, for which the first 8 days were used for training and the next 3 days were used for testing. We have used the RMSE as the evaluation metric.

We first report the improvement of each method over the baseline method. The improvement was measured by a positive or negative percentage over the performance of the baseline method, that is, (RMSE of compared method - RMSE of the baseline method)×100/RMSE of the baseline method. The results are shown in Figure 2 and Figure 3. To facilitate the comparison between different methods, for each air pollutant of each dataset, we report two figures, with one grouping the results by regularizations and the other grouping the results by the model formulations. From the results, we can see that (i) the light model formulation had a clear advantage over the heavy model formulation and the baseline model formulation, which implied that controlling the number of parameters is important for improving generalization performance; and (ii) the proposed CC regularization yielded a better performance than other regularizations, which verified that considering the similarities between models of consecutive hours is helpful. We also report the exact RMSE of each method in

Table 2. Root-mean-squared error (RMSE) for all approaches and datasets. The best approaches are marked as bold.

Approaches	LMA-AV: ${\mathbf{O}}_3$	LMA-AV: ${\mathbf{PM}}_{2.5}$	LU-LV: ${\mathbf{O}}_3$	LU-LV: ${\mathbf{SO}}_2$
Baseline	0.1324	0.0399	0.0971	0.0334
Heavy–F	0.1193	0.0394	0.0882	0.0333
Heavy–${\ell }_{2,1}$	0.12569	0.041	0.0883	0.033591
Heavy–nuclear	0.1197	0.0398	0.0893	0.0333
Heavy–CCL2	0.11896	0.0391	0.0882	0.033148
Heavy–CCL1	0.11897	0.039134	0.0882	0.033261
Light–F	0.1158	0.0372	0.0848	0.0331
Light–${\ell }_{2,1}$	0.11591	0.037	0.085376	0.033411
Light–nuclear	0.1161	0.0368	0.0849	0.0326
Light–CCL2	0.116	0.0369	0.0845	0.03253
Light–CCL1	0.11535	0.03684	0.085	0.03248

.

Finally, we compared the convergence speed of the employed optimization algorithms with their standard counterparts. In particular, we compared the ASSG and SSG methods for optimizing the ${\ell }_{2,1}$-norm regularized problem, and SSG for solving the nuclear norm regularized problem, and and SADMM for solving the CC regularized problem. The results are plotted in Figure 4 and demonstrate that the employed advanced optimization techniques converged much faster than the classical techniques.

6. Conclusions

In this paper, we have developed efficient machine learning methods for air pollutant prediction. We have formulated the problem as regularized MTL and employed advanced optimization algorithms for solving different formulations. We have focused on alleviating model complexity by reducing the number of model parameters and on improving the performance by using a structured regularizer. Our results show that the proposed light formulation achieves much better performance than the other two model formulations and that the regularization by enforcing prediction models for two consecutive hours to be close can also boost the performance of predictions. We have also shown that advanced optimization techniques are important for improving the convergence of optimization and that they speed up the training process for big data. For future work, we will further consider the commonalities between nearby meteorology stations and combine them in a MTL framework, which may provide a further boosting for the prediction.