3.1. Oversampling of Multiclass Samples Using the SMOTEWB

Although SMOTEWB is an effective technique for binary oversampling, it is not ideally suited for addressing imbalanced datasets with multiple classes, such as those frequently encountered in avionics equipment fault diagnosis. Specifically, datasets in this field often exhibit a significant disparity between the number of samples in the normal operating class and those in various fault classes. This variation arises due to the differing frequencies and detectability of faults across the diverse avionics modules. To tackle this challenge, our study proposes a successive one-versus-many balancing strategy that utilizes SMOTEWB to oversample the minority classes, excluding the class with the highest number of samples. This strategy aims to equalize the sample sizes across all classes, thereby enhancing the accuracy and reliability of the multiclass imbalanced dataset. The implementation process is illustrated in Figure 1.

The successive one-versus-many strategy is designed to diminish the disparity in sample numbers across classes by progressively increasing the samples in each minority class. This approach considers the oversampling outcomes of previous classes, which better preserves the characteristics of the data distribution. Thanks to the robust noise handling capabilities of SMOTEWB, when combined with this strategy, it can generate higher-quality synthetic samples for minority classes, thereby achieving a more balanced distribution for multiclass imbalanced data.

By using the successive one-versus-many strategy to enhance SMOTEWB’s multiclass oversampling, we can achieve a dataset where each class has equal numbers of samples. This not only increases the quantity of training samples but also ensures a balance across different types of data, providing higher-quality training samples for subsequent LGBM pattern recognition.

3.2. LGBM Method for Fault Pattern Recognition

LGBM is a distributed system developed by Ke et al. [11] from Microsoft Research Asia in 2017. It is an enhancement of Gradient Boosting Decision Tree (GBDT) [12], known for its excellent classification capabilities [13]. After oversampling multiclass samples using the SMOTEWB model, the balanced data is fed into the LGBM model for training, enabling precise fault pattern recognition.

Here, f(x; γ_m) represents the weak model, β_m represents the weight coefficient, and γ_m represents the parameters of the weak model.

Here, c is a constant value that can minimize the loss function.

Here, I(x_i ∈ R_mj) is an indicator function. If x_i is in the R_mj region, it results in one; otherwise, it is zero.

During the fitting process of decision trees, LGBM converts continuous floating-point eigenvalues into q integers, constructs a histogram of width q, and identifies the optimal segmentation point, substantially reducing storage requirements. The leaf-wise growth strategy, utilized in node splitting, minimizes search and splitting time while increasing accuracy by focusing on the leaves with the highest splitting gain. Furthermore, techniques such as gradient-based one-sided sampling and exclusive feature bundling in LGBM maximize data utilization and accelerate training efficiency.

3.3. Stratified K-Fold Cross-Validation

K-fold cross-validation divides the dataset into K subsets of equal size, selecting one subset to assess the model’s performance, while employing the remaining K-1 subsets for training. Given the imbalanced nature of the dataset, the stratified K-fold method is utilized to ensure that the proportion of each class in each subset mirrors that in the original dataset [14]. This approach effectively utilizes the data and mitigates the risk of overfitting. In this study, K is set to five, meaning that five-fold cross-validation is implemented. The training and validation sets are randomly divided into five parts, where each of the five subsets is oversampled using SMOTEWB to create five oversampled subsets.

The schematic diagram of five-fold cross-validation is shown in Figure 2, and to validate the effectiveness of this method, each 50% unsampled subset is used in turn as a validation set, with the corresponding four oversampled subsets serving as the training set. After completing five validation rounds, the average result of these validations represents the outcome of the cross-validation.

3.4. TPE Hyperparameter Optimization

In 2011, Bergstra et al. [15] from Harvard University introduced the TPE, a method that models the relationship between hyperparameters using a tree structure and effectively addresses multidimensional optimization challenges. With only a few iterations, TPE can achieve superior hyperparameter selection. Consequently, TPE is an optimal method for tuning the hyperparameters of the SMOTEWB-LGBM model, requiring minimal computing resources while ensuring efficient operation. This method is adaptable to different objective functions, further enhancing the quality of hyperparameter selection.

Here, x represents the observation point, which is the hyperparameter vector of the model to be optimized; y is the observation value, the outcome of the objective function (loss or evaluation function) for the given parameter x; and y^∗ is a threshold, representing a specific quantile of the TPE algorithm used to divide the observations into two density functions, l(x) and g(x), with the sum ranging from zero to one. The algorithm sets y^∗ based on existing observations, segregating them into two distinct density functions: l(x) is formed by those observations {x_(i)} whose loss is f(x_(i)) < y^∗, and g(x) consists of the remaining observations. TPE employs Bayesian optimization principles to minimize ineffective search space.

Let γ = p(y < y^∗), and to simplify the above formula, construct the denominator asp(x) = ∫p(x|y)p(y)dy = γl(x) + (1 − γ)g(x).

From this formula, maximizing EI involves ensuring that the probability of l(x) is high and that of g(x) is low. In each iteration, the algorithm returns the candidate hyperparameter x^∗ with the highest EI value, $$. During the maximization process, the hyperparameter x that achieves the highest l(x) and the lowest g(x) probabilities obtains the highest EI value. The TPE algorithm uses l(x) and g(x) to generate a collection of hyperparameter samples and evaluates x by the ratio of l(x)/g(x). Each iteration returns the point with the highest EI value. Using this approach, we identify the optimal hyperparameters for the SMOTEWB-LGBM model. In optimizing these parameters, the new x is employed to construct the fault diagnosis model for the SMOTEWB-LGBM algorithm, and the observation y is obtained through training convergence. The new observation point is then compared with the original, updating the probabilistic surrogate model to select the hyperparameter x corresponding to the best result.

In this study, we utilize the microaverage F1 (Micro-F1) score from the five-fold cross-validation as the objective function for optimizing parameters through the TPE method. This method helps identify the optimal hyperparameters by maximizing the aforementioned score.

We apply the TPE algorithm to optimize hyperparameters for both SMOTEWB and LGBM simultaneously. Once optimized, the LGBM hyperparameters are then used in the LGBM model for classification diagnosis testing. Although the SMOTEWB hyperparameters do not participate directly in the testing phase, they play a crucial role during the training phase by altering the distribution of the training data through the generation of synthetic samples. This alteration aids in training a more robust LGBM classification model. The framework of this SMOTEWB-LGBM multiclass fault diagnosis is illustrated in Figure 3.

4.1. Evaluation Indicators for Class Imbalance Problem

In traditional classification problems, classification accuracy is commonly used as a key metric to evaluate classifier performance. However, when dealing with imbalanced datasets, relying solely on classification accuracy may not accurately reflect the overall classifier performance. To overcome this limitation, this study incorporates the weighted precision (Weighted-P) as suggested in the literature [16], along with the macroaverage F1 (Macro-F1) score and the Micro-F1 score from the literature [17] as metrics to evaluate classification performance. The confusion matrix for binary classification is presented in Table 1.

To extend the F1 score to multiclass classification, two types of averaging are usually used, namely, Macro-F1 score and Micro-F1 score.

These three indicators offer different perspectives to evaluate model performance. The weighted average precision and Micro-F1 score are more sensitive to large classes, while the Macro-F1 score treats all classes as equally important. The three indicators are employed to facilitate a comprehensive analysis of algorithm performance.

4.2. Fault Diagnosis Process

The fault diagnosis process of SMOTEWB-LGBM is as follows.

Step 1. Data preprocessing including Min–Max normalization.

Step 2. Application of the SMOTEWB method with a one-to-many balancing strategy to oversample multiclass imbalanced data, equalizing the sample count across all classes.

Step 3. Pattern recognition using the LGBM ensemble learning method on the balanced data.

Step 4. Optimization of SMOTEWB and LGBM hyperparameters using the TPE algorithm and five-fold cross-validation, followed by model output.

Step 5. Determination of the mode category for new samples and identification of the fault mode.

5.1. Experiments on UCI Datasets

The UCI datasets, which are publicly available and commonly used to benchmark algorithm performance in machine learning, were selected to test the effectiveness of the proposed method. Six datasets with varying imbalances were used: Balance-scale, Diabetes, Dermatology, Cell-cycle, User-knowledge, and Glass. The details of these datasets are provided in Table 2.

Eighty percent of the data from each dataset were randomly selected as training samples, with the remaining 20% serving as the test set. To minimize the impact of randomness, 10 experiments were conducted on each dataset, and the results were averaged. The findings from these experiments are displayed in Table 3, with the optimal values for each indicator highlighted in bold. Corresponding graphical representations are shown in Figure 4.

As depicted in Table 3 and Figure 4, the SMOTEWB-LGBM method achieved the highest scores across all evaluation metrics for three datasets—namely, Diabetes, Dermatology, and User-knowledge, and registered the best results in two metrics across three datasets, specifically Balance-scale, Cell-cycle, and Glass. Notably, in the Dermatology dataset, it significantly outperformed other methods. This superior performance underscores the effectiveness of combining the SMOTEWB oversampling method, which adeptly minimizes noise introduction, with the robust classification capabilities of the LGBM ensemble learning method, thus significantly enhancing the classification outcomes for multiclass imbalanced data. The success of this method across multiple datasets also demonstrates its broad applicability.

5.2. Front-End Receiver Fault Diagnosis Experiment

To assess the practical effectiveness of the SMOTEWB-LGBM diagnostic model in avionics equipment diagnosis, we applied it to the fault diagnosis of the front-end receiver in a specific type of aircraft electronic countermeasure system. Figure 5 illustrates the basic working principle of the front-end receiver. The primary function of the front-end receiver is to amplify and filter signals within its operational frequency range, convert them into video signals containing threat information, and forward these signals to both the signal processor for further processing and to the central receiver along with the low-power radio frequency unit. These radio frequency signals carry essential frequency band information pertaining to threat radars.

This study conducted module-level fault diagnosis on the front-end receiver. The diagnosis is distinguished between a normal mode and three fault modes based on the receiver’s functionalities. The modes are denoted as follows: normal mode (F0) and three fault modes—amplification unit module failure (F1), microwave unit 1 module failure (F2), and microwave unit 2 module failure (F3). The diagnostic parameters included 12 test measurements, representing 12-dimensional sample features: sensitivity at five frequency points, dynamic range at five frequency points, and gain at two radio frequencies.

Data collection utilized the Automatic Test System (ATS) for this type of aviation equipment, employing standard signal sources, power meters, and spectrum analyzers as specified in the factory maintenance manual. A total of 450 datasets were compiled, with 306 sets of normal samples (F0) and 63, 31, and 50 sets of fault samples for F1, F2, and F3 modes, respectively. This dataset exhibited a pronounced class imbalance.

The data were randomly divided into 80% training samples and 20% test samples. Using the Optuna hyperparameter tuning framework, the TPE algorithm, and stratified five-fold cross-validation, we simultaneously tuned the parameters of the SMOTEWB oversampling algorithm and the LGBM pattern recognition method. The four-fold training set comprised the sample post-SMOTEWB oversampling, while the validation set utilized the original data samples. SMOTEWB required tuning for two parameters, and LGBM required tuning for six parameters. Table 4 displays the optimal hyperparameters obtained through this optimization.

Figure 6 shows the distribution of class samples before and after SMOTEWB oversampling. Initially, there was a marked imbalance among the classes, with the F0 class having the most samples and the F2 class the fewest—over three times fewer than F0. This imbalance hindered optimal pattern recognition. However, after employing SMOTEWB’s successive one-versus-many strategy for oversampling, the number of samples in each class matched that of the largest class before oversampling, significantly increasing the size of the training dataset and improving the LGBM’s classification performance.

Figure 7 displays a radar chart comparing the mean values of different features for the original samples and the new samples generated by SMOTEWB oversampling. It is evident that no new samples were generated for the F0 class, which was the largest class. In contrast, new samples were created for the F1, F2, and F3 classes, with the mean values of the features for these new samples closely mirroring those of the original samples. This similarity indicates that the new samples preserved the characteristics of the original data and maintained a similar distribution, thereby more accurately representing the original minority samples.

Figure 8 illustrates the trend of the multiclass logarithmic loss value from the first to the 100th iteration of the SMOTEWB-LGBM model under the optimal hyperparameter combination. The chart shows a rapid decrease in the training set loss, stabilizing at approximately 0.08284 after about 60 iterations. The test set loss converged to 0.12148. These results indicate that the default 100 iterations were adequate. The slightly higher loss on the test set compared to the training set, while still low, demonstrates that the model performed robustly on unseen data without overfitting. The minor difference between the loss values of the training and test sets confirms that the model was well-fitted, avoiding overfitting, and was effective in classifying and identifying faults.

Figure 9 shows the time spent on each iteration of the model training process. The time per iteration follows a nonlinear increasing trend, suggesting that as the number of iterations grows, so does the complexity and computational load of the training process. This nonlinear increase may stem from the dynamic allocation of computational resources. Despite this rise, the overall time per iteration remains within an acceptable range, from approximately 0.01–0.041 s per iteration. The brief testing time of the model indicates its suitability for fault diagnosis in avionics equipment, fulfilling engineering requirements.

To better assess the fault diagnosis performance of the classification model under the imbalanced dataset, Figure 10 presents the model’s classification confusion matrix. The y-axis in the figure represents the actual fault labels of the data, while the x-axis shows the predicted labels.

The model achieved 100% prediction accuracy for the F2 and F3 classes, and 94% and 92% classification accuracy for the F0 and F1 classes, respectively. The confusion matrix further confirms that the proposed method can effectively predict multiple classes of data under an imbalanced dataset setting, meeting the high-precision requirements for fault diagnosis of avionics equipment.

The proposed method was also experimentally compared with SMOTEBoost, RUSBoost, Adacost, ImECOC, and other methods, including LGBM. Each method was run 10 times to calculate the average value. Table 5 shows the experimental results, and Figure 11 graphically illustrates these outcomes. Additionally, the time cost of the models was compared, using default hyperparameters for each. Table 5 records the corresponding average training and testing times.

The experimental results presented in Table 5 and Figure 11 demonstrate that in the front-end receiver fault diagnosis experiment, the SMOTEWB-LGBM model outperformed other models, achieving a weighted-average precision of 96.05%, a Macro-F1 score of 95.10%, and a Micro-F1 score of 95.92%. These results indicate that the proposed method effectively meets the fault diagnosis requirement of avionics equipment.

Table 6 details the experimental outcomes of the front-end receiver under various data split ratios. It is evident that most algorithms perform optimally with an 8 : 2 data split ratio. This preference is likely due to the advantage of having a larger training set in the context of small-sample data, which provides more information for the model to learn and understand the intrinsic patterns and features of the data. This comprehensive learning typically results in better model generalization during training, leading to enhanced performance on the test set. Furthermore, the SMOTEWB-LGBM method demonstrates consistently high performance across different data split ratios, underscoring its sophistication and robustness.