1 Introduction

Air pollution poses a threat not only to environmental quality but also has profound effects on human health. According to the World Health Organization, more than 7 million people die each year from diseases attributed to air pollution. So, air pollution has emerged as an urgent issue concerning both the environment and public health. Numerous countries and regions have implemented a range of policies to address this challenge. For instance, the European Union has established stringent air quality standards aimed at reducing the presence of harmful substances in the air [1]. In Asia, nations such as China and India are actively advancing clean air action plans to enhance air quality and safeguard public health [2]. The study by Danek et al. offers an in-depth analysis of air pollution policies in Krakow, which are designed to decrease pollutants by curbing industrial emissions and promoting green transportation [3].

Air pollution poses a threat not only to environmental quality but also has profound effects on human health. According to the World Health Organization, more than 7 million people die each year from diseases attributed to air pollution [4-6]. Moreover, air pollution causes damage to ecosystems, affects the growth of crops, and exacerbates climate change [7]. Therefore, developing effective algorithms for predicting the concentration of air pollutants is crucial for formulating scientific pollution control strategies and safeguarding public ecological and environmental rights.

At first, scholars simulated the transformation process of pollutants in the atmosphere based on atmospheric dynamics to predict the concentration, such as Carruthers et al. [8] used the ADMS model to predict the concentration of air pollutants in the UK. The predicted values for SO2 and NOX pollutants were consistent with the actual values, but the prediction accuracy for other pollutants was not high. Afzali et al. [9] combined the WRF weather forecast model and AERMOD air quality model to simulate the spatial changes of different air pollutants in multiple industrial sites in Malaysia. After evaluation, it was found that the results were consistent with the actual values. Holnicki et al. [10] evaluated the performance of urban scale air quality prediction using the CALPUFF model and demonstrated that the model performs much better in long-term prediction than in short-term prediction. Shahbazi et al. [11] used the WRF-CAMx model to predict the concentration of air pollutants in Tehran, while using regression to interpolate missing values, which has advantages on datasets with a large proportion of missing values. Nabavi et al. [12] proposed a WRF Chem model based on the WASF source function to simulate dust concentration, but there is still uncertainty. Karaca et al. [13] used USEPA's IAQX to simulate a regional concept of concentrations related to several household cooking activities, accurately capturing the trend of PM2.5 concentration changes. The above air pollutant concentration prediction model based on atmospheric dynamics simulation needs to combine multiple input characteristics such as terrain and pollution sources, which makes it difficult to collect complete data and has regional limitations. It requires reanalysis and simulation for different regions, making research and development difficult.

With the continuous improvement of monitoring technology, it is becoming easier and easier to obtain data of air pollutant concentration. More and more scholars tend to use statistical models to predict air pollutant concentration, such as ARIMA model [14-17], SVM model [18-21], Decision Tree [22], RF model [23], BPNN model [24-26]. However, these models are shallow structures with limited ability of nonlinear representation of data, and it's hard to fully mine time correlation of historical information, so prediction accuracy still cannot reach the expected effect [27, 28]. Deep Neural Networks (DNNs) have a strong learning ability for rich data, and a variety of deep learning-based air pollution prediction solutions have been proposed, achieving better prediction accuracy than shallow statistical methods in many cases. Such as, Lu et al. [29] used 3D flattened variational methods to obtain air quality assimilation data, and merged with Long-Short Term Memory Network (LSTM) models to mine features, improving the predictive performance of PM2.5. Huang et al. [30] trains a gated recurrent unit neural network (GRU) using a stationary subsequence of PM2.5 concentration from Empirical Mode Decomposition (EMD), which improves the prediction accuracy by about 40% compared to a single GRU model. Ma et al. [31] proposed an LSTM network with delay layers (Lag-LSTM) for multivariate prediction of air quality, and in order to improve model performance, Bayesian methods were used to optimize the parameters of the network. Zhang et al. [32] used an encoder decoder structure prediction model to predict pollutant concentrations and demonstrated the superiority of this structure. Tu et al. [33] designed an autoencoder network with attention mechanism to mine the long-term trend of air pollution, and added a time decay factor to the attention module, which improved the adaptability of the model. Ma et al. [34] combines transfer learning theory with bidirectional LSTM (TLS-BLSMT) for air quality prediction research with missing data. This method effectively improves the air quality prediction of new sites by transferring the trained site model to a new site. Samal et al. [35] designed a Multi directional Time Convolutional Artificial Neural Network (MATCN) that can perform both feature learning and sequence modeling simultaneously, reducing a significant amount of computation time. Smith et al. [36] proposed a heuristic-based multiscale depth-wise separable adaptive TCN for real-time air quality prediction, achieving state-of-the-art performance on multiple datasets. Similarly, Wang et al. [37] developed a hybrid model that combines feature selection with TCNs to predict air pollutant concentrations, highlighting the model's ability to capture complex temporal patterns. Wu et al. [38] analyzes the periodicity of air pollutants through autocorrelation and uses LSTM training to learn their intrinsic temporal correlation, which has high fitting performance for different pollutants. Luo et al. [39] combines ARIMA with LSTM to explore the nonlinear information of air pollutant concentration, and optimizes LSTM using whale algorithm to obtain the most unique hyperparameters of LSTM. Dalal et al. [40] introduced a hybrid model that integrates Particle Swarm Optimization (PSO) with Long Short-Term Memory (LSTM) networks, aimed at enhancing air quality forecasting. This approach has been demonstrated to be effective in boosting the model's predictive accuracy and computational efficiency. Wang et al. [40] decomposed the atmospheric pollutant sequence twice using empirical mode decomposition and variational mode decomposition, and then processed the decomposition sequence using multi-layer perceptrons and gated loop units, verifying that the decomposed model performs better. Ding et al. [37] combines weighted random forest and LSTM(RF-LSTM) algorithm to predict PM2.5 concentration, which has an advantage in generalization ability compared to simple LSTM models. Ma et al. [41] used a two-stage attention model to mine the historical temporal correlation of air pollutant concentrations at source domain sites, and introduced the learned information into the air pollutant concentration prediction task at new sites through transfer learning, effectively improving the accuracy of prediction. Liu et al. [42] designed an LSTM model with ensemble empirical mode decomposition (EEMD-LSTM) attention to achieve PM2.5 concentration prediction, which has the advantage of effectively reducing the nonlinear complexity of historical data through EEMD and significantly improving the stability of the model. Muley et al. [43] integrated 2 two-way LSTM models for air quality prediction through XGBoost, which solved the problem of easy over fitting of a single model. Dalal et al. [44] integrated the Curiosity-based Motivation method with LSTM for air pollution prediction, achieving good predictive performance. Smith et al. [40] developed a heuristic-based multiscale depth-wise separable adaptive temporal convolutional network for predicting ambient air quality in real-time, offering innovative insights into the field of air pollution prediction.

To enhance the comprehension of the characteristics of various models and their limitations in predicting air pollutant concentrations, this paper presents a comparison of the strengths and weaknesses of different air pollution forecasting models through Table 1.

Although the above air pollutant concentration prediction algorithms have achieved certain results, there are still two challenges. One is how to explore the impact of information brought by time features on air pollutant concentration; another is how to synchronously utilize the local and global semantic information of historical features to improve the accuracy of prediction models. To address the aforementioned challenges, this study aims to engineer a deep aggregated seq2seq network that incorporates temporal features to enhance the precision of air pollutant concentration forecasting. More specifically, this research endeavors to bridge the gaps in existing models in the following respects: first, by delving into the impact of temporal features on forecast outcomes; and second, by introducing a novel approach for feature integration that leverages both local and global semantic information from historical data, thereby boosting the predictive performance of the model, which is novel in that:

First, as the time feature is integrated into the historical pollution concentration data through the form of cross-attention, the influence of time feature on the predicted value can be mined more effectively.

Second, the deep aggregation seq2seq network structure is designed to explore the temporal correlation of fusion features. Each node in the structure can effectively mine local time correlation through gated linear unit, and then obtain global time correlation through recursion and deep aggregation, which improves the predictive performance of the model.

Finally, the method is compared on the actual data sets of two areas in Beijing. The simulation results show that the prediction accuracy of this model is higher than that of the existing advanced models.

In summary, we have introduced a deep aggregated seq2seq network that integrates temporal feature fusion for the prediction of air pollutant concentrations in smart cities. Compared to existing research, our approach not only incorporates time features but also leverages local and global temporal correlations within historical data in tandem. Moreover, our method represents a significant advancement in handling nonlinearities and time-series data, an area that has not been thoroughly explored in prior studies.

The structure of this paper is organized as follows: The Introduction section provides a background on air pollution and its impact on public health and environmental quality, emphasizing the importance of accurate air pollutant concentration prediction. The Methodology section details the proposed deep aggregation seq2seq network model, including the feature fusion and deep feature extraction processes. The Experimental Analysis section presents the performance of our model on real datasets from Beijing's Changping and Shunyi districts, comparing its accuracy with state-of-the-art models. Finally, the Conclusion summarizes the key findings and discusses the implications of our model for smart city air quality management. This paper aims to provide a comprehensive guide for readers interested in the advancement of air pollutant concentration prediction models.

2 Methods

The prediction problem of air pollutant concentration defined is described: Learn a complex mapping relationship $$$ F $$$ based on historical data and time feature, and then calculate the air pollutant concentration in the next period through the mapping relationship, as follows:

In this paper, a deep aggregate seq2seq network with time feature fusion prediction model is designed as a mapping relationship for air pollutant concentration prediction, as shown in Figure 1, which mainly consists of two parts: feature fusion and feature extraction. Feature fusion includes multi-head cross-attention and layer normalization modules, and feature extraction includes patching and deep aggregation seq2seq network modules. This paper selected the deep aggregated seq2seq network based on criteria such as the model's predictive accuracy, computational efficiency, and its capacity to handle time-series data. The network excels in processing sequential data, particularly in capturing long-term dependencies. Furthermore, by incorporating temporal feature fusion and deep aggregation mechanisms, our model is capable of more precisely forecasting future pollutant concentrations.

2.1 Feature Fusion

Temporal feature encoding is selected for its ability to capture the seasonal and cyclical patterns inherent in air pollutant data. Time attributes such as month, day of the month, and hour are critical in understanding the dynamics of air pollutant concentrations, which vary significantly across different timescales.

This article sets the time feature embedding of each point as a 1 × 3 sequence: $$$ E=\left[{ME}_i,{DE}_j,{HE}_k\right] $$$, where $$$ {ME}_i $$$ is the month embedding value of the $$$ i $$$-th month of a year, $$$ {DE}_j $$$ is the day encoder value of the $$$ j $$$-th day of a month and $$$ {HE}_k $$$ is the day encoder value of the $$$ k $$$-th day of a day. Their specific calculation formula is as follows:

$$$$ \left\{\begin{array}{l}{ME}_i=\frac{i}{11}-0.5,i\in \left[0,1,\cdots, 11\right]\\ {}{DE}_j=\frac{j}{30}-0.5,j\in \left[0,1,\cdots, 30\right]\\ {}{HE}_k=\frac{k}{23}-0.5,k\in \left[0,1,\cdots, 23\right]\end{array}\right. $$$$ (2)

After the time feature of each time point is obtained, it is integrated into the historical pollutant concentration data by means of multi-head cross-attention, as shown in Figure 2. Assume that $$$ X\in {R}^{1\times {\tau}_2} $$$ is the characteristic of historical pollutant concentration and $$$ E\in {R}^{3\times {\tau}_2} $$$ is the characteristic of historical time embedding. First, the query feature $$$ Q $$$ of pollutant concentration data is generated through a linear operation and the key feature $$$ K $$$ and value feature $$$ V $$$ of time feature embedding are obtained through two linear layers:

$$$$ \left\{\begin{array}{l}Q={W}_qX\\ {}K={W}_kE\\ {}V={W}_vE\end{array}\right. $$$$ (3)

where $$$ {W}_q $$$, $$$ {W}_k $$$, and $$$ {W}_v $$$ are the linear layer weights for generating the query, key and value feature.

Then, the attention feature of time feature embedding is generated by cross-attention:

$$$$ A=\frac{S\mathrm{oftmax}\left({QK}^T\right)}{\sqrt{d_k}}V $$$$ (4)

where $$$ {d}_k $$$ is the first dimension of the key matrix.

Finally, the query matrix of pollutant concentration data and the attention feature of time feature embedding are added to obtain the fusion feature of each head, and the features of each head are fused in an average operation:

$$$$ H=\frac{1}{N}\sum \limits_{i=1}^N\left({A}_i+{Q}_i\right) $$$$ (5)

where $$$ H $$$ is the fusion feature of multi-head cross-attention, $$$ N $$$ is the number of heads.

2.2 Deep Feature Extraction

Deep aggregation was chosen for its effectiveness in handling sequential data and capturing long-term dependencies. This method allows the model to aggregate information across different time scales, which is essential for understanding the complex interactions in pollutant dispersion.

Before feature extraction, the fused embedded features are first divided into multiple patches along the time dimension. This treats each patch as a local time period features, and feature extraction on it can mine local temporal correlations.

Assuming that the time length contained in each patch is $$$ {\tau}_p $$$ and the step length between two adjacent patches is $$$ {s}_p $$$, then the number of extracted patches $$$ p $$$:

$$$$ p=\frac{\tau_2-{\tau}_p}{s_p}-1 $$$$ (6)

After the output features of multi-patches are obtained, the temporal correlation is mined through deep aggregation seq2seq network, which consists of encoder and decoder. The initial deep aggregation seq2seq network is mainly composed of four types of nodes, as shown in Figure 3, where the input nodes of encoder are the input feature of patches, Assuming $$$ \left\{{x}_i|i=1,2,\cdots, p\operatorname{}\right\} $$$ are the input features of $$$ p $$$ patches, the number of input nodes is also $$$ p $$$. The current recurrent nodes of encoder are obtained by calculating the input node and the previous recurrent node through a gated convolutional network:

$$$$ {r}_i=\Phi \left({x}_i+{r}_{i-1}\right) $$$$ (7)

where $$$ {r}_i $$$ is the $$$ i $$$-th recurrent node and $$$ \Phi \left(\cdot \right) $$$ is the mapping relationship of the gated convolutional network. The core idea of gated convolutional networks is to introduce one or more gated layers after the traditional convolutional layer. These gating layers can calculate a gating feature of the same size as the output of the convolutional layer, and then multiply the gating feature with the output of the convolutional layer element by element, so as to realize the dynamic adjustment of the convolutional layer output:

$$$$ Z={Conv}_1(X)\otimes \sigma \left[{Conv}_2(X)\right] $$$$ (8)

where $$$ Z $$$ and $$$ X $$$ are the output and input features of gated convolutional networks, $$$ {Conv}_1\left(\cdot \right) $$$ and $$$ {Conv}_2\left(\cdot \right) $$$ are convolutional networks with two convolutional kernels of the same size, $$$ \sigma $$$ is the Sigmoid activation function.

Aggregation nodes of encoder are obtained from the nodes of the upper layer through the binary tree structure:

$$$$ {a}_i^l=\left\{\begin{array}{l}\Phi \left({r}_{2i}+{r}_{2i-1}\right),l=1\\ {}\Phi \left({a}_{2i}^{l-1}+{a}_{2i-1}^{l-1}\right),l>1\end{array}\right. $$$$ (9)

where $$$ {a}_i^l $$$ is the $$$ i $$$-th aggregation node in layer $$$ l $$$.

The solution method for the current nodes in the decoder is different from that in the encoder. It is obtained by calculating the aggregation nodes of the last layer generated by the last patch of the encoder and the previous current node through gated convolutional network:

$$$$ {r}_i=\Phi \left({r}_p^L+{r}_{i-1}\right) $$$$ (10)

where $$$ {r}_p^L $$$ is the aggregation nodes of the last layer generated by the last patch of the encoder.

The solution method for the aggregation nodes in the decoder is the same as that in the encoder. Then, the initial deep aggregation seq2seq network, is improved through two steps. The first step as shown in Figure 4, the connections between the cyclic nodes at the time position where the aggregation nodes exist are removed and the aggregation nodes at the last layer are connected with the cyclic nodes at the next time position. The second step, as shown in Figure 1, merges the aggregation nodes at a time location with multiple aggregation nodes, keeping only the aggregation node of the last layer at that time location and preserving connections to all nodes at the previous time location.

Finally, the final output node is obtained through the node at the last layer of each time step in decoder:

$$$$ {y}_i=\left\{\begin{array}{l}\varPhi \left({a}_i^L\right), if\ Aggregation\ node\ exists\\ {}\varPhi \left({r}_i\right), otherwise\end{array}\right. $$$$ (11)

Following the establishment of the model as detailed, it is imperative to train it with an extensive dataset before its application. During the training, the mini-batch Adam algorithm is leveraged as the optimizer. The loss function for the output layer is designated as Mean Squared Error (MSE). To avert overfitting, we have embedded L2 regularization within our model. This regularization technique imposes a penalty on the magnitude of the coefficients in the loss function, which is instrumental in diminishing the model's complexity and deterring it from becoming excessively tailored to the training dataset:

$$$$ {L}_{MSE+L2}=\frac{1}{2}\sum \limits_{i=1}^N{\left({y}_i-{\hat{y}}_i\right)}^2+\lambda \sum \limits_{i=1}^M{w}_i^2 $$$$ (12)

3 Results

The entire simulation experiment was completed on a computer with NVIDIA GeForce RTX 3090, and the model was construct using the Pytorch2.0.1 framework. In this study, we utilized highly sensitive sensors for PM2.5 and SO2, with a total of 50 sensors deployed across major traffic arteries and residential areas in Changping and Shunyi districts of Beijing. These sensors boast an accuracy of ±1 microgram per cubic meter, ensuring the precision of the collected data. The data collection spanned from March 1, 2013, to February 28, 2017, encompassing the variations across all four seasons, which is crucial for understanding the seasonal fluctuations of pollutants. The specific pollutants under investigation include PM2.5 and SO2, selected due to their significant impact on public health and environmental quality.

3.2 Performance Superiority Analysis

First, the prediction accuracy of each prediction model is compared. Mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation metrics:

$$$$ MAE=\frac{1}{T_{te}}\sum \limits_{i=1}^{T_{te}}\left|{\hat{y}}_i-{y}_i\right| $$$$ (13)

$$$$ RMSE=\sqrt{\frac{1}{T_{te}}\sum \limits_{i=1}^{T_{te}}{\left({\hat{y}}_i-{y}_i\right)}^2} $$$$ (14)

Table 2 offers a comparative analysis of the MAE and RMSE for several forecasting models across two datasets. The one with the smallest error is shown in bold, and the one with the second smallest error is shown by underline. ARIMA model is a linear model, which is difficult to mine the nonlinear information of the historical air pollutant series, so the error of ARIMA model is the largest for the prediction task on each data set. Although the error of SVM and BPNN models is smaller than that of ARIMA model, they are shallow statistical structures with weak nonlinear characterization of historical air pollutant concentration data, so their prediction performance is still not very good. Lag-LSTM, EMD-GRU, TLS-BLSTM and MTCAN are a deep learning models based on deep neural network, and their nonlinear representation ability is stronger, so achieve better prediction performance. The deep aggregation seq2seq network we designed not only integrates temporal features, but also synchronously utilizes local and global temporal correlations of historical data. Compared with the baseline models, the MAE of PM2.5 in the Changping dataset is reduced by 9.34% and RMSE is reduced by 10.51%. The MAE of SO2 is reduced by 6.59% and RMSE is reduced by 7.46%; Compared to the baseline models, the MAE of PM2.5 in the Shunyi dataset decreased by 10.96% and RMSE decreased by 4.62%, while the MAE of SO2 decreased by 3.73% and RMSE decreased by 0.51%.

In this paper, it is verified that deep aggregation seq2seq network with time feature also has good prediction accuracy in other prediction horizons, and its performance is compared with SVM, Lag-LSTM, and MTCAN model when the prediction horizon is 1–11 h. Figure 5 shows MAE of the four models in different prediction horizons. It can be seen that the MAE of deep aggregation seq2seq network with time feature in different prediction horizons is smaller than that of other baseline models, indicating that the model still maintains good prediction performance with the increase of prediction horizons.

3.5 Performance Analysis of Key Modules

In order to verify the influence of two key modules in deep aggregation seq2seq network, feature fusion and feature extraction, on the prediction accuracy, three ablation models were used to verify the influence of these two modules. In ablation model 1, feature fusion is removed from the original model. In ablation model 2, Aggregation nodes are removed from the original model. In ablation model 3, Recurrent nodes are removed from the original model. The specific results are shown in Tables 5 and 6.

TABLE 5. Comparison of ablation models on ChangPing dataset.

Pollutant	Feature fusion	Aggregation nodes	Recurrent nodes	MAE	RMSE
PM2.5	×	√	√	13.0363	22.3092
	√	×	√	14.3056	24.3791
	√	√	×	16.3604	27.0990
	√	√	√	11.2443	19.4102
SO2	×	√	√	3.0302	5.5343
	√	×	√	3.7349	6.5606
	√	√	×	4.5958	7.5724
	√	√	√	2.5382	5.0532

TABLE 6. Comparison of ablation models on ShunYi dataset.

Pollutant	Feature fusion	Aggregation nodes	Recurrent nodes	MAE	RMSE
PM2.5	×	√	√	13.3617	23.7389
	√	×	√	14.6664	24.9653
	√	√	×	16.5326	26.8769
	√	√	√	11.3795	20.9197
SO2	×	√	√	2.8486	5.9776
	√	×	√	2.9494	6.2906
	√	√	×	3.7361	7.2816
	√	√	√	2.4642	5.8243

As can be seen from Tables 2 and 3, Feature fusion, both aggregation and recurrent units contribute to enhancing the model's predictive precision, and recurrent nodes have the greatest impact on the prediction accuracy of the model. Recurrent nodes are designed to tap into the time-related information from prior records, which is the most important for the impact of future data (Figure 6).

In order to more intuitively show the impact of each key module on model performance, the real values of high concentration, medium concentration and low concentration time periods and the predicted values of different four prediction models were randomly selected from the PM2.5 data set in Changping for visualization, as shown in Figure 4. It can be seen that for the test samples of high concentration and medium concentration time periods, the predicted value of the original model is closer to the true value than that of the three ablation models, but for the test samples of low concentration time periods, the predicted value of the original model and the predicted value of the three ablation models is almost the same as the true value. It also shows that these key modules are more likely to affect the predicted value of the higher concentration period, and are more valuable for practical applications.

To demonstrate the computational cost of each module, we have visualized the average computation time of the original model and each ablation model across the two datasets, as shown in Figure 7. It is observable that the Feature Fusion module incurs the least computational time, while the Recurrent Nodes demand the highest. This is attributed to the fact that Recurrent Nodes diminish the model's capacity for parallel processing, thereby incurring the most significant increase in computational time.

4 Access Policy Analysis

In this section, we delve into the access policies related to air quality data and their implications for the practical applicability and effectiveness of air pollution prediction models. Access policies play a crucial role in determining how data is collected, shared, and utilized, which directly impacts the performance and generalizability of predictive models.

5 Discussion

In this paper, a deep aggregation seq2seq network with time feature fusion is proposed to solve the problem of air pollutant concentration prediction, which integrates time feature embedding into historical air pollutant concentration series through a cross-attention module, and then designs a deep aggregation seq2seq network to mine local and global temporal correlation. The experimental results based on real datasets from the Changping and Shunyi districts of Beijing demonstrate that our proposed model achieves an approximate enhancement of 5% to 10% in predictive accuracy over the latest deep learning models, as measured by the Mean Absolute Error (MAE). Furthermore, through the ablation study of key model components, we confirmed the positive impact of the feature fusion and feature extraction modules on the model's predictive accuracy. Specifically, the feature fusion module effectively integrates temporal features with pollution concentration data through a multi-head cross-attention mechanism, while the feature extraction module uncovers temporal correlations using a deep aggregation sequential network. The synergistic effect of these two modules significantly boosts the model's performance.

Nonetheless, the model is not without its limitations, such as reliance on high-quality historical data, high computational complexity, and the risk of overfitting. Future research could investigate ways to enhance the model's generalizability, alleviate computational load, and refine it for real-time data streaming. These enhancements would aid in the model's application across various settings and provide more effective tools for air quality management in smart cities.

Author Contributions

Yunzhu Liu: conceptualization, investigation, writing – original draft, methodology, validation, writing – review and editing, visualization, software, formal analysis, project administration, data curation, resources, supervision.

Conflicts of Interest

The author declares no conflicts of interest.