2.1. HTTPSmell Framework

When using HTTPSmell to identify HTTP traffic, the input needs to be preprocessed as text, split into word lists by our proposed segmentation method, and converted into vector statements using Word2vec. Then, the UDA-based semisupervised training method is used for label refactoring. Finally, the detection task is performed by utilizing the deep learning model.

2.2. Data Augmentation

Poor data augmentation algorithms that process unlabeled samples will weaken the performance of UDA models. When the sample diversity and semantic information after data enhancement processing are not guaranteed, the model will be prone to produce overfitting, especially the deep learning model with many parameters and strong learning ability. Traditional data augmentation algorithms such as back translation, RandAugment, and TF-IDF word replacement used in literature [12] will destroy the semantics of traffic samples because the replacement of keywords will weaken malicious traffic characteristics into benign traffic. Applying an appropriate noise-adding method to training samples can enhance the diversity of training samples and help the machine learning models counteract the impact of noise, thus lowering the risk of overfitting.

Therefore, this paper adopts the idea of feature reservation. On the one hand, the critical feature text is retained. On the other hand, the nonkey feature text is randomly replaced to achieve data augmentation while preserving the critical information of web traffic.

2.3. Deep Learning Models

One of the primary purposes of HTTPSmell is to verify the effectiveness of the data augmentation method in the DL-based detection algorithms. For the demonstration, two DL algorithms are chosen. One is TextCNN, based on convolutional neural networks [13] using supervised learning for classification, and the other is the UDA mechanism [12] leveraging semisupervised learning for label reconstruction.

2.3.2. UDA Model

Another model employed in HTTPSmell is UDA, an unsupervised data enhancement technique and a semisupervised learning approach introduced by the Google team in 2020. By applying a data augmentation algorithm to a small amount of labeled data and a large amount of unlabeled data, it trains a deep learning model that outperforms the ones trained with abundant labeled data. HTTPSmell combines UDA with the TextCNN model to detect malicious HTTP attacks.

Without enhanced training data and unsupervised consistency training in the semisupervised training, the procedure will result in poor label refactoring, and the model would be prone to the risk of overfitting or underfitting [12]. Therefore, equations (2) and (3) are combined in the model training phase. Based on the improved method of consistency training applied in the UDA, minimizing the KL divergence of the original unlabeled data and augmented data is used to enhance the smoothness and improve the generalization ability of the model (as shown in equation (2)), where q ($$ | x) is a data augmentation transformation and $$ is a fixed copy of the current parameters θ indicating that the gradient is not propagated through $$. Equation (2) aims to achieve predictive consistency between each unlabeled sample and its augmented sample by minimizing the KL divergence between them. With the help of data enhancement methods, the distribution of unlabeled samples is closer to the realistic data distribution and increases the number and diversity of unlabeled samples. As a result, the training data are expanded and the accuracy and credibility of the model prediction are improved.

During the semisupervised training process, the final loss consists of the supervised cross-entropy loss and the unsupervised consistency loss weighted by λ in equation (3) By minimizing the supervised loss, the model learns an initial classification boundary on labeled samples. Minimizing the unsupervised loss enables the model to adapt to unlabeled samples, extending the classification boundary of labeled samples to unlabeled samples, which produces reliable pseudolabels for unlabeled samples. By minimizing the loss in equation (3) through continuous iterative training of the model, the UDA method propagates label information from labeled data to unlabeled data.

$$ (2)

$$ (3)

Figure 2 shows using the UDA-based semisupervised training procedure for label refactoring, considering consistency between labeled data and unlabeled data. Meanwhile, TextCNN is conducted in the final classification.

First, we need to extract potential word order and semantic associations of keywords in malicious traffic (e.g., malicious statements, a second “script” will appear after the first “script” to generate closed tag pairs). Then, Word2vec is used to generate a word vector file from the before-augmented traffic training set and noised traffic training set. The marked traffic samples and unlabeled training set are used as input when training the model, and the word vector is inputted in the embedding layer to construct a semantic association. Finally, the detection model is obtained.

3.1. Dataset Introduction

We collect a set of open-source WAF (web application firewall) request datasets from Fsecurify based on HTTP protocol as training set and test set. 45,000 malicious traffic with malicious behavior characteristics and 45,000 normal traffic were selected randomly and then divided into a training set and test set according to a ratio of 7 : 2 under the supervised learning model (only TextCNN) as the control group. The training set under the semisupervised learning model is composed of 700 labeled data and 70,000 unlabeled data, and test set is the same as the supervised training. The ratio of positive and negative samples among the 700 labeled data is 1 : 1, and the ratio of path traversal, sql, and xss data in positive samples is 3 : 4 : 3.

3.2. Experiment Setup

3.3. Effectiveness of Data Augmentation

In this experiment, we use the same convolutional neural network structure of TextCNN (see Figure 2) to evaluate the detection performance of the full supervised learning model that requires a large number of labeled samples to train and the semisupervised learning model (UDA combined with TextCNN) that only needs a small portion of labeled samples in the training period. In order to demonstrate the effectiveness of the KLA-based data augmentation algorithm, the experiments of different noise-added ratios in the data augmentation method for unlabeled samples in the UDA model are conducted as well.

From the experimental results in Table 6, it can be seen that the supervised model generated based on a large number of labeled samples has good performance on the test set with 96.96% accuracy and 96.35% precision. However, the semisupervised model can narrow the gap after training on the noise-adding dataset and achieve comparable performances as the supervised model with 96.85% accuracy and 96.80% precision. Table 7 demonstrates that the semisupervised model can adapt well to the other open-source traffic datasets and maintain relatively better performance compared with a fully supervised model without any noises added, which indicates the noise-adding method can effectively strengthen the generalization ability of the model. We can conclude that with only 700 labeled samples, HTTPSmell outperforms the supervised training model with 70,000 labeled samples in comprehensive performance, which reflects the superiority of our KLA-based data augmentation algorithm. Although 70,000 unlabeled samples are used in the training phase of HTTPSmell, it is easier to obtain unlabeled samples and, most importantly, reduce the workload of manually labeling samples.

The addition of 100% noise (replacing all nonkeywords in each sample) to the training dataset enhances the detection accuracy, generalization ability, and overall performance of the model in distinguishing normal and malicious traffic. In order to verify the model performs best when adding 100% noise data, a more detailed test is conducted on the ratio near 10% (that is, adding 90% noise for testing). The final results show that when the noise ratio of the training set is 90%, the performance of the model is similar to that of adding 80% noise, which exhibits poor detection performance on the test dataset.

As shown in Tables 6 and 7, the model’s detection capability reaches the peak after the ratio of the noise increases to 100%. This implies that augmenting the samples by replacing all words that are not in the keyword library can improve the diversity of the dataset and simultaneously boost the robustness of the semisupervised model against noise. Otherwise, the model will tend to be underfitting (80%∼90% noise ratio). A possible cause of the underfitting is that the noise proportion is out of a certain threshold range, which will reduce the weight of keywords order and result in misjudgment of model detection. For example, the string in malicious traffic such as “echo” and “jsp2” will be misjudged as benign traffic by the model.

To sum up, the KLA-based data augmentation algorithm improves the generalization ability of the model, and the model achieves its optimal detection performance when the noise-adding ratio is set to 100% during the training phase.

3.4. Preprocessing Influence

In the data preprocessing step, we use our preprocessing method, segmentation based on customized special tokens and automatically extracted keywords, and the methods of [3, 6, 17, 18] to obtain the corresponding training models. The performance of models trained by different data preprocessing methods is compared in six aspects (see Table 2). The same training set and test set are used in different data preprocessing methods, and the same neural network is used to train the model. In this experiment, we use the previously trained model HTTPSmell (100% noise-adding) to compare with other models to illustrate the superiority of our data preprocessing method. The comparison results are shown in Tables 8 and 9.

Based on the data in Tables 8 and 9, it can be seen that the method proposed in this paper has higher accuracy and lower false alarm rate, and the overall performance is optimal compared with other existing methods when using small training traffic samples. The model obtained by Park et al. [6] based on common characters segmentation adopts the way of splitting sentences with 68 common characters to get character-level representation. This model has high generalization ability, but it ignores the significance of keyword features, which prevents it from differentiating malicious samples from normal traffics in complex cyberattack scenarios.

However, the method based on special characters segmentation [17] splits sentences with several customized special characters to get word-level representation. Training the model with a large dataset processed in this way results in underfitting in malicious traffic detection due to the lack of consideration of malicious features. It is likely to identify a large number of malicious traffic as normal traffic, which declines the overall accuracy and weakens the ability to distinguish malicious traffic. The user-defined keyword segmentation method [3] manually summarized special keywords from 95 URL-valid characters, but it does not take into account the keyword distribution of benign traffic samples, which leads to a high false alarm rate. Although the model can detect and identify conventional malicious traffic well, it is difficult to manually construct a keywords library that encompasses all malicious traffic keywords and potential attack payloads due to the diversity of HTTP attack statements. This ultimately leads to the weak generalization ability of the model.

In this paper, we split the samples according to keywords and special characters. The keywords library is automatically established through statistics to maximize the retention of keywords in malicious traffic, while the high-frequency words that appear in benign traffic samples are removed. Then, the training dataset is fed into the model after sample augmentation processing. Minimizing the loss between original samples and augmented samples makes the detection model more easier to extract the keywords and related word order features. Moreover, the extensive training dataset also helps the detection model avoid the risk of overfitting during the training phase.

After examining the malicious traffic dataset from a recently deployed honeypot and the attack traffic collected from an ISP company, the analysis shows that the wild attack vectors tend to be more obfuscated and advanced. For example, the attack data is confused by multiple encoding and grammatical substitution to bypass WAF’s keyword detection. Because the method in this paper can automatically build a keywords library based on a large amount of malicious traffics, it is difficult to avoid all keywords even if some keywords are reduced in the attack payload. Accordingly, the data preprocessing method in this paper still has a good detection effect. However, in the face of the above kind of attack traffic based on a user-defined keyword segmentation method, the preset malicious traffic keywords are easy to bypass through the confusion and syntax substitution. For example, the change of “select” into “se/ ^∗∗/lect” makes the keyword select unable to be captured, which reduces the ability of model detection. As the segmentation method based on common characters produces large dimensions and ignores the semantic relationship between keywords, the ability of model detection will decrease when the attacker highly obfuscates the attack payload. The method based on particular symbol segmentation can easily ignore essential information when dealing with nonletter confusion attack data with a large number of special symbols, for example, “(∼%8F%97%8F%96%91%99%90) (); ” means “phpinfo(); ” After confounding malicious payload, the capability of the model detection is reduced.

In binary classification problems, the ROC (receiver operating characteristic) curve and AUC (area under the curve) values are often used to evaluate a binary classifier. ROC is a comprehensive indicator reflecting the continuous variables of sensitivity and specificity. Precisely, each point on the ROC curve reflects the sensitivity to the same signal stimulus. The value of AUC is the area under the ROC curve. The calculation method of AUC considers the classifier’s classification ability for both positive and negative samples and can still make a reasonable evaluation of the classifier in the case of unbalanced samples. From Figure 3, it can be seen that whether it is the test set or the cross-data set, the ROC curve of the model obtained by the data preprocessing method of HTTPSmell is farther away from the diagonal than the other data preprocessing methods, the curve is smoother, and its AUC value is also the highest.

3.5. Overhead Analysis

This experiment explores the overhead of the HTTPSmell (optimal parameters) under the computational latency and memory usage. Table 10 and Figure 4 show the experimental results.

To demonstrate the lightweight features and practical application value of HTTPSmell, we also perform some comparative experiments of performance overhead with WebSmell [11] and its Albert [19] implementation. It should be noted that the original implementation of the WebSmell model is based on Bert [5], a multilayer bidirectional transformer encoder model that performs self-supervised learning on a large-scale corpus in the pretraining stage. It conducts two types of tasks, masked language model (MLM) and next sentence prediction (NSP), to learn text representations with rich semantic information. Bert can achieve excellent performance on different downstream tasks through fine-tuning. Still, it has a vast amount of parameters, which means it will consume too many computational resources during the inference period. To this end, we additionally replace the Bert model in WebSmell with Albert, which reduces the model parameters through factorized embedding parameterization and cross-layer parameter-sharing techniques. Compared with the 110 M parameters of BERTbase, the Albert-base model has only 12 M parameters, 10 times less, and the model training speed is 5 times faster. To solve the problem of model performance degradation after parameter reduction, Albert replaced the next sentence prediction (NSP) task with sentence order prediction (SOP) in the pretraining stage, which even makes Albert have higher average scores than Bert in multiple NLP tasks. We use the Python packages “timeit” and “memory-profiler” as performance analysis tools to conduct an experimental analysis of these three models regarding computing latency and memory overhead. Finally, we will make a comprehensive comparison of their detection performance.

To compare the performance overhead of these three models, we use two types of computing units: GPU (RTX 4090) and CPU (Intel (R) Core (TM) i5-11400) as the devices for model training. The computing efficiency and throughput (number of samples processed per second) of WebSmell running on a GPU are nearly 66 times higher than that of running on a CPU device. At the same time, the training time of the model speedup almost 2 times. This is because the GPU device has more parallel computing units that can handle a large number of matrix operations simultaneously, while the CPU can only execute instructions sequentially. The Bert model involves a large number of matrix operations during the inference process. Therefore, WebSmell runs more efficiently on GPU devices with higher parallelism. When WebSmell (Albert) runs on the GPU, its model training time is about 8 minutes, nearly 2 times faster than WebSmell (Bert), and its computing throughput is also nearly twice as large, reaching up to 4158 per second. These indicate that techniques such as parameter reduction and sentence-order prediction can reduce the computational resource overhead. In contrast to the other two WebSmell models, HTTPSmell achieves significantly better performance on a CPU device in terms of loading time, training time, and throughput. It can process up to 27089 samples per second and complete the training in only 90 seconds. The throughput of HTTPSmell on a CPU is nearly an order of magnitude higher than that of WebSmell on a GPU. The end-to-end detection efficiency is one of the most critical considerations of deploying the model to the production environment, where CPU resources are often more cost-effective and scalable. This can substantially reduce the overall task latency, accelerate data delivery, and provide more decision time for downstream tasks.

We also analyze the memory overhead required for loading the model parameters and external dependencies into the main memory. It can be seen from Figure 4 that during the process of loading the best parameters saved in the training stage into the main memory, the memory usage increases over time. Due to the numerous parameters of the model itself and the additional external packages for function implementation, WebSmell needs to occupy a significant amount of memory, averaging 3 GB once fully loaded. In contrast, HTTPSmell is more lightweight and only requires around 60 MB of memory.

Besides the low-overhead characteristics, it is essential to ensure the excellent detection capabilities of the model as well. To this end, we further analyze the detection performance of these three models. Figure 5 shows the model’s performance on the test set, which contains 10,000 benign and 10,000 malicious samples, constituting a quarter of our data set, and the remaining three quarters are used for model training. We compare and evaluate the detection performance of the three models on the six metrics in Table 2 under different noise ratios. The y-axis in Figure 5 represents the corresponding detection rate of each metric under investigation, and the x-axis represents the noise ratio added during the model training stage. To distinguish the performance difference among the comparison models, we set the starting position of the y-axis to start from 0.85 in the four subgraphs (a, d, e, and f). The results show that the Albert implementation of WebSmell (100% noise ratio) has the worst performance among all metrics, with the lowest values of true-positive rate (88.64%), F1-score (92.08%), and accuracy (92.38%) and also the highest values of false-positive rate (3.88%) and false-negative rate (11.36%), which indicates that even if the model added noised samples during the training stage, it cannot learn high-level and low-level features that are important for distinguishing malicious traffic from benign traffic in the word vector representation. WebSmell’s Bert implementation is superior to Albert’s in all indicators, owing to its powerful learning ability with 110 M parameters. It also achieves the highest performance among all models in terms of precision (98.32%). However, it is still inferior to HTTPSmell in several other indicators. We can see from Figure 5 that HTTPSmell (100% noise ratio) outperforms WebSmell almost in all six indicators, with the highest accuracy rate (96.85%) and the lowest false detection rates (3.2% and 3.11%). Meanwhile, it keeps 10 times the throughput of WebSmell and reduces the memory usage by up to 50 times (∼3000 MB) while preserving outstanding detection performance. The detection efficiency does not suffer a significant loss, with only a minor drop in precision rate (∼1%), which is acceptable considering the significant reduction in performance overhead.

Through the comparative analysis of the abovementioned experiments, we can conclude that complex models represented by Bert and Albert have few advantages in classifying small samples, unbalanced categories, and short texts. On the contrary, simple structures and fast training models represented by TextCNN are a better solution. The experimental results also confirm that TextCNN has more advantages in training and inference performance, and it is a better choice to use in low-end devices and low-latency scenarios. The HTTPSmell combines TextCNN with the UDA model and leverages a semisupervised learning method to solve the problem of labeled sample scarcity. It simultaneously achieves outstanding detection performance, reduces memory usage and runtime latency, and significantly improves throughput.

3.6. Detection Performance in Enterprise Environment

To explore the detection performance and generalization ability of the HTTPSmell (optimal parameters) under real-world enterprise cyberattacks, we further conducted experiments on the attack traffics that realistic enterprise encountered. Table 9 presents the experimental results.

This dataset encompasses various attack techniques, ranging from SQL injection, XSS, and path traversal to other sophisticated attacks such as code injection, CVE exploitation, and covert channels. The diversity and complexity of the test sample set can verify the ability of the model to detect unknown cyberattacks. The experimental result in Table 9 demonstrates that the model is robust enough to detect advanced attack traffic in the enterprise environment and achieves an accuracy rate as high as 96.77% under the validation set with unbalanced samples obtained from the enterprise. This also implies that HTTPSmell can learn the characteristics of these unknown attacks even if they are not present in the training dataset. This is sufficient to attest the strong learning ability of our model, which can generalize well to unknown attack vectors. If we can use higher quality datasets in the training phase, such as attack samples in the enterprise environment, the detection ability of the HTTPSmell will be stimulated to a greater extent.

4.1. Malicious HTTP Traffic Detection

Most malicious traffic is encoded based on some ordinary malicious traffic, and many variants are added. There is a large amount of reused data between them, which has significant characteristics of malicious traffic. Based on the normal behavior in the network, any network traffic inconsistent with the expected normal behavior is considered abnormal. Most of them use machine learning, data mining, or statistical methods to infer the normal traffic characteristics and detect malicious traffic.

In terms of extracting effective data from web traffic, Xu et al. [7] detected evaded network attacks by using the inconsistency of content-type field in HTTP, which can find unknown malware from traffic effectively. Arzhakov et al. [8] used honeypot technology to collect statistical information on attackers’ behavior and used statistical classification methods to distinguish malicious traffic. Torrano-Gimenez et al. [9] proposed an intrusion detection method of network traffic based on exceptions, using XML files to classify incoming requests into normal and abnormal requests. The N-gram model was used to extract the corresponding features from the three fields of HTTP log: web source, query attributes, and user agents. Then, some algorithms in machine learning are used to process the data. Vartouni et al. [20] analyze the HTTP traffic and construct features from HTTP data based on the n-gram character-based model. To solve the dimensionality problem, a stacked auto-encoder is leveraged to extract relevant features from data. In the final, isolation forest has been used as a one-class learner to detect anomalies. Torrano-Gimenez et al. [21] proposed the web attack detection method of applying generic feature selection to reduce redundant and irrelevant features and combine expert knowledge with the n-gram feature construction method for reliability and efficiency. Aiming at the root causes of WAF’s inefficiency, including target computation complexity and unnecessary recomputation, Chen et al. [22] proposed to build an efficient rule caching engine under two scenarios. The rule-based processing of WAF is very time-consuming, and at the same time, the order of the rule set and the maintenance of the white list require rich expert experience and knowledge support. The HTTPSmell traffic detection scheme based on the natural language processing (NLP) model proposed in this paper can achieve high throughput and accuracy (see Section 3.5) without the support of rich expert experience.

In terms of malicious traffic detection, many studies have used different models and data preprocessing methods to extract web traffic characteristics. Park et al. [6] proposed a method for anomaly detection of HTTP messages based on a character-level binary image transformation, named CAE (convolutional automatic encoder), which is superior to the traditional heuristic machine learning method for selecting input features, using a data preprocessing method based on character-level segmentation. Zolotukhin et al. [23] proposed an anomaly detection method for web attacks by analyzing HTTP logs. Yang et al. [3] designed a convolutional gated-recurrent-unit (CGRU) neural network for malicious URLs detection based on characters as text classification features. Yu et al. [17] proposed a Bi-LSTM (bidirectional long short-term memory) and the deep neural network model of attention mechanism, which models HTTP traffic as a natural language sequence to detect malicious traffic. Malekghaini et al. [24] investigate the impact of data drift on two state-of-the-art models for classifying encrypted traffic. Research shows that traffic shape-based models are more robust to data drift. For network traffics, data drift can lead to model performance degradation, especially when the model is trained on historical data and applied to new data in a production environment. As discussed in Section 2.2, the KLA-based algorithm can effectively enhance the model’s generalization ability for various attack traffic scenarios. The results of Experiment III-F further confirm the advantages of this system design. Lin et al. [25] designed a multilevel feature fusion model that combines timing, byte, and statistical features for traffic anomaly detection in an IoT environment. Zheng et al. [26] conducted a research work on encrypted malicious traffic detection. They proposed an efficient encrypted malicious traffic detection method based on graph convolutional network (GCN), named GCN-ETA. Hou et al. [27] introduce the related research work on intrusion detection system. Various deep learning techniques and models are used, such as LSTM, CNN, and GRU, as well as methods such as word embeddings and statistical features, to build intrusion detection systems.

In terms of the hybrid model, Corona et al. [28] designed a multiclassifier system to detect web attacks by modeling normal requests. The system uses a set of predefined models, which is based on different fields in the HTTP request and adopts two basic models: statistical distribution model and hidden Markov model. Choras and Kozik [29] proposed a machine learning method to simulate the normal behavior of web applications and detect network attacks at the same time. This model obtains information from HTTP requests, using graph-based segmentation technology and dynamic programming to generate models. Al-Obeidat and El-Alfy [30] proposed a new supervised hybrid machine learning method based on fuzzy decision tree and attribute selection for traffic analysis. Erfani et al. [31] presented a hybrid model where an unsupervised deep belief network (DBN) is trained to extract generic underlying features, and a one-class support vector machine (SVM) is trained from the features learned by the DBN. LSTM is generally used for anomaly detection and diagnosis in system logs [32]. Zhang et al. [33] proposed to use multiple models to check HTTP request messages to identify attacks. Three learning models, probability distribution model, hidden Markov model, and one-class of support vector machine model, were used to detect different fields of HTTP request messages. If one model reported anomalies, the overall detection result was abnormal. Dong et al. [34] proposed a network abnormal traffic detection model based on semisupervised deep reinforcement learning. The proposed model uses a double deep Q-network (DDQN), a representative of a deep reinforcement learning algorithm, and includes unsupervised techniques like autoencoder and K-means clustering algorithms to reduce manual labeling and detect unknown attacks. They tested the model using NSL-KDD and AWID datasets, achieving good results in various evaluation metrics.

Moreover, in detecting traffic packet content, malicious traffic was identified through the analysis and statistics of the content of the packets. Tekerek [35] proposed a web attack detection architecture based on CNN deep learning algorithm. The proposed method uses the bag-of-words model to preprocess data, generates an HTTP request image based on the URL and payload, and feeds the image as a CNN model input to obtain the classification result. Some malicious traffic may inject itself into legitimate flows, causing the header entropy of illegal traffic to be higher than that of legitimate traffic. Yadav et al. [36] presented an anomalous traffic detection method based on comparing the header field entropy of a plurality of flows with a predetermined amount. Shi et al. [37] used the LSTM for traffic anomaly detection. The input of the model is a history of recent traffic preprocessed by the big data processing framework of Kafka and Spark, and the output is the label of the recent data of a flow.

The above work uses a variety of models and extracts different features to model and classify the traffic, but most of the work in the data preprocessing part uses simple N-grams word segmentation or data statistics. Using simple data statistics to preprocess the traffic will lead to the problem that the traffic data characteristics cannot be well preserved while using the N-grams method will cause a dimensional explosion. Moreover, in most of the web traffic, there are no semantic association characters and encoding characters, and then, using N-grams to extract keywords of different lengths will generate many coded characters and invalid strings, while traditional malicious keywords are often not reserved.

This paper proposes to segment web traffic, retain traditional malicious keywords, merge the keywords set of different types of malicious web traffic, remove single characters, and maximize the reserved keywords representing the malicious web traffic. The reserved malicious web traffic keywords also play a crucial role in data augmentation methods.

4.2. Data Augmentation for Machine Learning

Most of the current data augmentation methods are suitable for image or audio data, for example, by shifting the image, transforming the angle of view, transforming the size, and applying Gaussian noise, which makes the added noise data have a lower level of information distortion. However, these methods are not suitable for the augmentation task of text data. For text data, existing researchers have proposed synonym substitution [23], EDA [38] (e.g., synonym replacement, random insertion, random swap, and random deletion), and the method of constructing weighted undirected graphs [8]. However, this method is suitable for texts such as articles and Q&A, mainly studying semantic association characteristics. Most of the data in HTTP traffic belongs to computer code, such as “script,” “for,” “insert,” so all kinds of characters and words are unable to adopt the way of synonym substitution for data augmentation. In addition, malicious HTTP traffic often has a specific word-order structure and semantic associations, such as SQL injection, and there will be a combination of “select from,” and “<script></script>” in XSS attacks. Furthermore, this kind of word-order structure generally does not exist in normal web traffic. It is not easy to extract features through simple regular matching using the word-order structure of web traffic.

The key to detecting malicious traffic through HTTP traffic is using text data classification methods to classify HTTP traffic. However, web attack statements are highly time-sensitive and may be encrypted, confused, and replaced in various ways. For obvious reasons, the amount of publicly available datasets of real-world user browsing behavior is scarce. Therefore, it is necessary to augment the training data appropriately so that the neural network can better extract features of malicious traffic.

In the field of natural language processing, since natural language itself is a discrete abstract symbol, a small change may lead to a massive deviation in meaning, and the conventional data augmentation methods can be carried out through multiple translation conversions [39], such as translation from Chinese to English and then to Chinese, translation from English to French and Chinese, then to English. In the data preprocessing stage, the existing methods [3, 6] augment the data by simple statement segmentation and character normalization, thereby enhancing the feature extraction of the model. Data augmentation aims at creating novel and realistic-looking training data by applying a transformation to an example without changing its label [40].

Zhang et al. [41] proposed to replace words or phrases with synonyms, and they used an English thesaurus to expand the text data. Existing studies have also proposed methods for constructing weighted undirected graphs. Chen et al. [42] proposed the HDA method to augment the original training data by randomly rotating, masking individual samples, and mixing sample pairs in arbitrary proportions. In audio processing, the background sound is different due to the different environments of audio recording. Although various denoising models can be applied, there may be differences in accent and intonation of audio. Therefore, data augmentation is often carried out by changing the tone, introducing multiple accents, multiaudio fusion, etc.

In the text data augmentation, it is unreasonable to use image or speech recognition signal conversion to increase data because the sequence of characters will affect the syntax and semantics [43]. Moreover, most of the characters in web traffic data are independent or meaningless encoding and nonsemantic words. The use of synonym replacement will affect the original feature extraction and destroy the malicious keywords in malicious traffic. This paper proposes a malicious HTTP traffic detection framework via data augmentation, which augments the web traffic data based on retaining most keywords in the malicious traffic.

A preliminary version of the work is reported in WebSmell [11]. Compared with the work WebSmell, our system has the following advantages. (1) We propose an optimized framework to augment data with a high scenario coverage rate, which can improve the diversity and quality of the training data and reduce the data bias. (2) We propose a particular UDA-based method to effectively deal with the problem of poor generalization ability in complex real-world scenarios, which can enhance the model’s adaptability to unseen attacks. (3) We combine two learning models: one is for label refactoring and data augmentation and another is for data training, which significantly improved the accuracy and robustness of the model. (4) We conducted extensive performance tests, which not only demonstrated the effectiveness of our keyword extraction and data augmentation method but also verified the lightweight feature and efficient detection capability of HTTPSmell in enterprise environments, both superior to the WebSmell.