3.1.1. Hybrid Feature Fusion Approach

To improve the proposed ensemble HER, we propose a hybrid feature fusion that combines traditional feature extraction with deep learning–driven representation. FER has long relied on traditional hand-crafted features such as PCA, HOGs, and local binary models (LBP). However, illumination changes, occlusion, and feature constancy are not uncommon. However, deep learning models such as EfficientNet, ViTs, RegNet, and SE-ResNeXt use learned hierarchical feature representations to improve their generality and accuracy. However, these algorithms are very computationally intensive and require large datasets.

This hybrid approach to feature fusion combines the generality of deep learning models with the robustness of the system to changes in illumination, face position, and occlusion.

3.4.1. TL in Pretrained CNNs

3.4.1.1. EfficientNet Architecture

Reusing previously trained models to solve new problems or TL content has many important benefits, including shortening training time, improving neural network performance, and reducing data consumption. Improved pretrained models such as EfficientNet with fully linked layers such as global average pooling layers, exclusion, and dense layers improve ER performance. The work of Mahendran [37] emphasizes how TL works in deep neural networks (DNNs) such as CNNs. Visualization approaches show that later layers of the CNN recognize more complex elements, such as texture and shape, while early layers of the CNN capture the basic features of the input image. Transition learning is particularly useful for ER because it does not require layer-specific feature extraction, which is difficult when training a DCNN from scratch. Transition learning is successfully performed on ER tasks by tuning the model and using pretrained weights.

The EfficientNet model is particularly suitable for ER. The model was trained with large user-defined datasets consisting of seven categories: sadness, happinens, neutral, fear, disgust, surprise, and anger. Here, the architecture of transition learning generally consists of adding a fully connected classifier in additional layers to replace the classification step while maintaining the convolutional basis of the pretrained DCNN. This modular technique consists of an XGBoost classifier, a fully connected fine-tuning layer, and a convolutional basis for feature extraction. The EfficientNet family of models, including EfficientNet-B0–B7, utilizes composite scaling to improve accuracy, reduce floating point dimensionality and computational cost, and improve model performance. The three relevant hyperparameters (depth [dt], width [wt], resolution [rt]) given in equation (4) are adjusted during composite scaling as

$$ ()

The resolution of the network is determined by the specified constants A, B, and Γ. The first step is to create a basic unification configuration called EfficientNet-B0. This is achieved by setting the compound coefficient ϕ to 1. Using the same structure, a grid search is performed to maximize the coefficients A,  B, and Γ and one other relevant variable to achieve the desired result.

$$ ()

where A ≥ 1,  B ≥ 1,  Γ ≥ 1

The ideal values of A,  B and Γ are 1.2,  1.1 and 1.15, respectively, and equation (5) shows the ideal values of these variables. By changing the value of ϕ, we can extend the version of EfficientNet with equation (4) from B1 to B7. The root, block, and header modules of the EfficientNet B0 base architecture must be used for feature extraction.

The Stem module contains convolutional layers (kernel size 3 × 3), batch normalization layers, and swish activation functions integrated into the first processing stage. The block module contains several moving inverse bottleneck convolutions (MBConvs), namely, MBConv1 and MBConv6, which differ depending on the expansion rate. These MBConv layers are crucial for capturing complex functions in different phases of the network.

To summarize, the efficient expansion and modular architecture of EfficientNet, especially thanks to the optimal configuration of its components, allow the model to extract features more easily and quickly while maintaining high accuracy and low computational costs. The adaptability and performance of the architecture make it a reliable choice for a wide range of HCI applications as shown in the following:

$$ ()

$$ ()

where $$, BN − batch normalization, SE − squeeze excitation, $$

The block module of the EfficientNet architecture for HRI consists of 16 layers, each with a different configuration of the MBConv. Specifically, these layers are MBConv1 with twice-repeated 3 × 3 cores, MBConv6 with twice-repeated 3 × 3 cores, MBConv6 with twice-repeated 5 × 5 cores, MBConv6 with three times-repeated 3 × 3 cores, and MBConv6 with three times-repeated 3 × 3 cores. MBConv6 with 5 × 5 cores was repeated three times, MBConv6 with 5 × 5 cores was repeated four times, and MBConv6 with 3 × 3 cores was repeated.

Convolutional layers, batch normalization, swish activation, pooling, dropout, and fully concatenated layers are all integrated into EfficientNet’s core module to improve performance. This layered structure optimizes feature extraction and processing and ensures efficient and accurate HRI.

The complex design of these layers, especially the various MBConv layers, allows the model to efficiently capture and process complex features in equation (8), making EfficientNet a powerful tool in HRI.

$$ ()

In [30], the authors show the detailed architecture of the EfficientNet CNN model. The difference between the MBConv6 level with 5 × 5 cores and the MBConv6 level with 3 × 3 cores is remarkable. Compared to MBConv6 with a 3 × 3 kernel, MBConv6 with a 5 × 5 kernel is applied to a larger kernel size, allowing the model to capture different levels of spatial features. This change in kernel size is important to improve the model’s ability to find and process complex patterns in the data. As a result, EfficientNet architecture provides better performance on tasks involving humans and robots. Since the scaling of the model does not affect the underlying operator levels $$ in the underlying network, it is important to have a strong network in the HRI domain. We will use an existing ConvNet to test our scaling technique. The mobile core network, EfficientNet, is now being developed to demonstrate the strategy’s effectiveness. After reading [31], we developed our base network and investigated the multiobjective neural architecture to achieve the best possible results regarding precision and floating-point operations per second (FLOPS). The problem space should be the same as in [31], and the optimization objective should be defined as ACC (m) × [FLOPS(m) = T]wt. Here, ACC (m) and FLOPS(m) Stand for the accuracy and FLOPS of model m, respectively, and for FLOPS and speed.

In model m, T represents the target FLOPS, as w equals −0. Hyperparameter 07 can be used to specify the trade-off between FLOPS and accuracy. In contrast to the previous research by the authors in [31], this study does not target a specific hardware device. Therefore, this study focuses on maximizing FLOPS rather than latency. EfficientNet-B0 is the name of the efficient network to be designed. Since it uses the same search space as [31], EfficientNet-B0 is designed to be equivalent to MnasNet. However, due to the higher FLOPS target, it is significantly larger, reaching 400 M. The structure of EfficientNet-B0 is shown in [37]. According to [31, 32], its basic component is the bottleneck-shifted convolutional bottleneck (MBConv). In addition, the incentive optimization and compression proposed by the authors in [40] are combined to improve the performance.

The architecture shown in Figure 9 of EfficientNet consists of three basic elements labeled (a-c). Each MBConv block accepts an input defined by height (ht), width (wt), and channel (c), with the output channel labeled C. This modular structure is essential to the network’s ability to process and analyze complex data patterns efficiently. Starting with the EfficientNet-B0 benchmark, a composite scaling method will improve the model’s performance. The method consists of two main steps: first, the network’s depth, width, and resolution are balanced. Second, these parameters are optimized to maintain a consistent trade-off between accuracy and computational efficiency. This approach ensures that scaled versions of EfficientNet maintain high performance while adapting to different HRI application requirements. Composite scaling methods can extract more features and achieve better results by scaling up the network incrementally without excessive computational costs. This balance makes EfficientNet a powerful tool for tasks that require rich data interpretation and high performance. For applying the compound scaling method, the steps are as follows.

• 1.

  Assuming, fix ∅ = 1, that twice as many resources are available, we solve the problem and perform a small grid search ∝,  β,  γ using equations (9) and (10). We find that the optimal value α = 1.2,  β = 1.1 and γ = 1.15 under the constraint of α.β².γ² ≈ 2 of EfficientNet-B0 is bounded.

• 2.

  Then, we use them as constants α,  β,  γ, and extend the base mesh with different ∅ values using equation (10).

  $$ ()

• •

  where a factor scales the width $$ of the mesh, and the base mesh has default settings for depth and resolution, $$.

Interestingly, searching ∝β,  γ for large patterns provides better performance, but searching for large patterns becomes prohibitively expensive. The method solves this problem by performing a single search on a small base mesh (Step 1) and then using the same scaling factor for all other models (Step 2).

We propose a new composite scaling method, as shown in Figure 10, which uses a composite coefficient φ to scale the width, depth, and resolution $$ of the mesh uniformly in a principled way as

$$ ()

In the context of HRI, the constants ∝β,  γ are determined via a small grid search, which provides an intuitive representation of the allocation of additional resources for updating the model scale. User-defined ϕ coefficients determine the availability of these resources. Specifically, ∝it influences the mesh width, β the mesh depth, and γ the resolution. The FLOPS of a standard convolution operation is proportional to dt,  Wt²,  rt². Therefore, doubling the mesh depth doubles the FLOPS, while doubling the width or resolution of the mesh quadruples the FLOPS. Since convolutional operations usually dominate the computational cost in ConvNet, we can scale ConvNet using equation (9).

$$ ()

This results in the total number of FLOPS in equation (11) increasing by approximately 2^ϕ when constrained by α. β². γ² ≈ 2. This scaling method balances computational efficiency and model performance, which is crucial for effective HRI applications.

Figure 10 illustrates the different methods for scaling CNNs in the context of the EfficientNet architecture, as described in the “TL in Pretrained CNNs” section. A series of layers, each performing specific operations such as convolution, pooling, and activation, processes input images at a baseline network resolution (A). Traditional scaling methods, including width scaling (B), depth scaling (C), and resolution scaling (D), focus on adding one dimension at a time. Width scaling improves the network by adding more channels at each layer, allowing it to capture finer features. Deep scaling involves adding additional layers, allowing the network to learn more abstract representations by applying additional transformations. Resolution scaling increases the resolution of the input image, allowing the network to capture more detailed information. The EfficientNet architecture introduces composite scaling (E) of the network for fixed ratio, which consistently scales the resolution, width (wt), and depth (dt) of the network. This balanced approach ensures efficient use of resources, resulting in improved performance and greater accuracy with fewer parameters than traditional sizing methods that add only a single dimension.

Algorithm 1: EfficientNet scaling.
•  

  Require:

•  

  B_base : Baseline network

•  

  W_f : Width factor

•  

  D_f : Depthfactor

•  

  R_f : Resolutionfactor

•  

  X : Input image

•  

  Steps:

•  

  1.  Width scaling are

•  

  B_scaled⟵ B_base

•  

  for each layer l ∈ B_scaled

•  

  l.channels  ← l.channels × W_f

•  

  2.  Depth scaling

•  

  for i = 1 to D_f

•  

  Create a new layer l_new

•  

  Addl_newto B_scaled

•  

  3.  Resolution scaling

•  

  H_new⟵ X.height × R_f

•  

  W_new⟵ X.width × R_f

•  

  $$

•  

  4.  Compound scaling

•  

  Apply width scaling to B_basewith W_f

•  

  Apply depth scaling to B_basewith D_f

•  

  Apply resolution scaling to X with R_f

•  

  Output ⟵ B_scaled(X_resized)

•  

  5.  Return

•  

  output

3.4.2. Fully Connected Layer

This module consists of three levels: high-density level, churn level, and global average pool. Global average pooling: The fully connected layer of traditional CNN is replaced by a global average pooling layer. The goal of the last level is to create a feature set for each classification level. Instead, we use the average of each feature map. We create a fully connected layer on top of the feature map. Global average aggregation and ConvNet topologies share many of the same fundamental benefits. Improving the connectivity between feature maps and the classes they correlate with is important. Another advantage is that there is no overfitting since there are no parameters to optimize when aggregating the global average. Global average pooling works in a special way. We use a medium pool. The system uses average clustering across spatial dimensions until each dimension is considered separate and exits the system. Additional size: Based on the image size ,  N₃, the global average pooling layer transforms the (N₁,  N₂,  N₃) feature set and feature map N₁,  N₂. This indicates that N₃ filters are used.

Dense layers are the next level of neural networks. Thick layers communicate with the next layer, penetrating deep into the neural network. Each neuron in the bottom layer is connected to every other neuron. Those matrix and vector multiplications are represented by neurons in this dense layer. Neurons in the model’s dense layer block receive outputs from all neurons in the layer above. The matrix–vector product is performed on the nodes of the dense layer. The row vector of the product is the same as the column vector of the dense layer output from the previous layer. The main hyperparameters that need to be optimized at this level are the units and activation functions. In dense layers, the unit of measurement becomes the main parameter. This dimension is always greater than one and defines the system in thick layers. An activation function modifies the neuron’s input values. Essentially, this introduces nonlinearity into the network, making exploring correlations between input and output values easier.

These steps provide a concise and structured approach to developing and evaluating an EfficientNet-based ER system for HRI.

4.1.1. FER-2013 Dataset

In HRI research, several benchmark datasets are used to evaluate the performance of the ER approach accurately. First, the FER-2013 dataset was presented at the ICML-2013 conference as a primary source for ER research [19]. The dataset contains 35,887 images, each with a 48 × 48 pixels resolution. The FER-2013 images mainly represent real-life scenarios and provide a solid basis for analyzing and evaluating the proposed method. We divide the dataset into a training set with 28,709 images and a test set with 3589 images. We capture these images automatically via Google Image Search using Google’s application programming interface (API). This automated capture ensures a variety of expressions and conditions, increasing the dataset’s completeness. An important aspect of ER is accurately identifying the six primary emotions, including neutral expressions. Despite low contrast and occasional face occlusions, the FER-2013 dataset is a widely used reference for face recognition tasks. Figure 3 from the study shows examples of this dataset and illustrates the variability and challenges involved. Using FER-2013 and other datasets, we perform an in-depth evaluation of the model and compare its performance with existing state-of-the-art methods. This approach validates our results and highlights the advances in ER in HRI applications.

4.1.2. CK+ Dataset

The CK+ AU-coded facial expression dataset is widely used in facial expression research and provides a comprehensive testing environment for automatic facial image analysis. This publicly available dataset consists of approximately 500 image sequences from 100 subjects annotated with FACS action units and specific emotional expressions [20, 21]. The CK+ dataset captures posed and nonposed expressions and enables detailed analysis and validation of FER systems. Subjects in the CK+ dataset were between 18 and 30 and were demographically composed of 65% female, 15% African–American, and 3% Asian or Latino. Data collection occurred in an observation room with chairs for the subjects and two Panasonic WV3230 cameras. One of the cameras was positioned directly in front of the subject, and the other was placed at a 30-degree angle to the right. Then, we connected it to a Panasonic S-VHS AG-7500 recorder with a synchronized Horita timecode generator. Currently, only the image data from the front camera is available. This extensive dataset supports recognizing facial emotion expressions as a unit of activity and contributes to developing and testing ER systems. Figure 4 from the study shows sample images from the CK+ dataset that illustrate the variety and complexity of the captured expressions. The extensive annotations and large topic library of the CK+ dataset make it a valuable resource for HRI research, especially for improving robotic systems’ emotional responsiveness and empathy.

4.1.3. Development of a Novel HERD

While developing HERD, we created unique datasets and used them for in-depth analysis and evaluation. This dataset integrates the proposed and existing datasets to improve HER, as shown in Figure 11.

Deep learning techniques analyze each image in the dataset to extract specific features. The subjects’ ERs were recorded for up to 15 min during data collection. The male subjects were between 25 and 40 years old and had variations such as beard growth, shaving, and wearing hats to ensure the diversity of the dataset. In addition, the videos are recorded and analyzed in real time, considering dynamic obstacles and changing lighting conditions, which is crucial for a robust HER system. Deep learning models are trained on the HERD and benchmark datasets for analyzing and evaluating ER. This comprehensive approach ensures that systems can accurately recognize and interpret human emotions in various realistic scenarios, significantly advancing the field of HRI.

4.2.1. Experiments on the CK+ Dataset

This study presents results and discusses our ER model tested on the CK+ dataset using the EfficientNet model. The approach involves a two-step method for deriving random assignments. In the first step, use the original CK+ dataset, which includes 444 images for training (70% of the total) and 192 images for validation (30%), and in the second step, we augment the dataset by training 14,652 images and validating 6336 images. The accuracy and loss for the training and validation phases on the dataset and the accuracy and loss graph for the original CK+ dataset are shown in Figure 12, and they show a training accuracy of 85.88%.

In comparison, the augmented dataset achieves a higher accuracy of 89.23%, as shown in Figure 13. A confusion matrix for each dataset using the EfficientNET CNN model is shown in Figures 14(a) and 14(b). Details of the test accuracy of each category of datasets are shown in Tables 2 and 3. In addition, the Cohen’s Kappa statistic (calculated as 0.020949409404947716) shows the reliability of our model for the CK+ dataset. Figures 15(a) and 15(b) show the ROC curves for the original CK+ dataset and the augmented CK+ dataset, respectively, and provide a comprehensive evaluation of our model’s performance. These results highlight the effectiveness of our ER method in the context of HRI and demonstrate the model’s ability to recognize and classify ER under different conditions and datasets accurately.

The ROC curves for the original and augmented CK+ datasets shown in Figure 15 use the EfficientNet model to test the performance of HER systems. These graphs are crucial for demonstrating the effectiveness of the proposed system. The ROC curves for the original and augmented CK+ datasets are shown in Figures 14(a) and 14(b). They show the true-positive rate (TPR) compared to the false-positive rate (FPR) for the seven basic emotions mentioned above in the tables. In the original dataset, the area under the curve (AUC) values for these emotions ranged from 0.50 to 0.60, with anger having the highest AUC value of 0.60.

In comparison, the AUC values for the augmented dataset ranged from 0.49 to 0.51, indicating slightly lower performance than the original dataset. This difference suggests that while augmentation can increase the diversity of a dataset, it also introduces additional complexity that affects the classifier’s performance. The proposed approach utilizes deep ensemble classification techniques to improve the accuracy and reliability of ER systems in HRI. Analyzing the ROC curve is an important evaluation metric that highlights areas for improvement. Although the AUC value is not very high, using advanced models such as EfficientNet and ensemble methods provides a solid framework for solving ER problems. This approach is crucial for developing intuitive and responsive systems that understand and react to HE, significantly improving HRI.

4.2.2. Experiments on the FER-2013 Dataset

The study follows a randomized, split approach conducted in two phases using the FER-2013 dataset. In the first stage, we used the original ARR-2013 dataset, which includes 777 images for training (70% of the total) and 321 images for validation (30% of the total). The augmented FER-2013 dataset is used in the second stage, which has been significantly expanded to 23,569 training images and 10,658 validation images. The validation and training loss using the EfficientNet CNN model for the FER-2013 original dataset is shown in Figure 16. The overall accuracy achieved with the original ARR 2013 dataset is 89.93%. The model performance is illustrated in the confusion matrix using different categories of original and augmented FER-2013 datasets, as shown in Figure 17, while detailed accuracy is shown in Table 4 for the original dataset. The FER-2013 dataset yielded a Cohen’s Kappa statistic of −0.0004065744654040415, indicating poor agreement between the predicted and actual classifications. These results highlight the challenges and complexity of recognizing ER, especially when using augmented datasets. Despite these challenges, the EfficientNet model shows strong performance and provides a reliable basis for improving ER in HRI systems.

This study on HRI achieved an accuracy of 94.36% with the augmented FER-2013 dataset using the EfficientNet model. The accuracy loss and validation plots for the augmented FER-2013 dataset are shown in Figure 18, indicating that the model performs well. Table 5 shows the detailed test accuracy by category for the augmented FER-2013 dataset. Figure 19 shows the ROC curves for the original and augmented FER-2013 dataset. These give a complete picture of the model’s effectiveness in different categories. The EfficientNet CNN model greatly improves accuracy when applied to an augmented dataset. This suggests that it can be used to improve ER systems in HRI. These results emphasize the model’s ability to process complex datasets and improve the reliability and responsiveness of robotic systems in interpreting human emotions.

This study’s graphs demonstrate the proposed system’s effectiveness in evaluating HER performance. The original and augmented FER-2013 datasets’ ROC curves are shown in Figures 19(a) and 19(b). We compare the TPR and FPR for emotions such as neutral, happy, angry, fearful, disgusted, sad, surprised, and glad. The AUC values for these emotions ranged from 0.45 to 0.59, indicating moderate performance, with the highest AUC obtained for disgust (0.59). The AUC values ranged from 0.45 to 0.54, indicating that the classification performance of some emotions is slightly improved compared to the original dataset, such as surprise (AUC: 0.54), while other emotions slightly worsened.

4.2.3. Novel HERD

The study will be conducted using a random assignment method. In the first stage, we split the dataset into 1925 images for training (70% of the total) and 825 images for validation (30% of the total). In the second stage, the augmented dataset comprises 5789 training images and 2481 validation images. The EfficientNet model using the original and augmented HERDs achieves an accuracy of 95.21% and 97.01%, as shown in Figures 20 and 21. Figure 22 shows the confusion matrices for the original and augmented HERDs, respectively. Table 6 shows the original HERDs’ detailed test accuracy by category. The Cohen’s Kappa coefficient we calculated for the HERD is −0.0058877095758373965, indicating a slight similarity. The ANOVA test yielded an F-value of 0.22150059161964505 and a p value of 0.8027571040296516, indicating that the observed difference was not statistically significant. These results emphasize the effectiveness of the EfficientNet model in improving ER, which is crucial for advancing HRI systems. The robust performance in the original and augmented datasets emphasizes the potential of this model for real-world applications to improve the responsiveness and emotional understanding of robotic systems.

The EfficientNet CNN model achieves an impressive accuracy of 97.01% on the augmented HERD, as shown in Figure 21, demonstrating the robust performance of the model, while Table 7 shows the detailed test accuracy by category. In addition, Figure 23 shows the ROC curves for the original and augmented HERDs, allowing a comprehensive assessment of the model’s performance for different emotion categories. These results emphasize the effectiveness of the EfficientNet CNN model in identifying and classifying HE, which is crucial for improving HRI. The high accuracy on large datasets emphasizes the model’s ability to handle complex real-world scenarios and improves the responsiveness and empathy of robotic systems.

4.2.4. Analysis of Datasets’ Imbalance

We conducted a comprehensive analysis of the impact of class imbalance in the dataset of human emotions on the effectiveness of the proposed innovative HER ensemble classification model. Figures 24(a), 24(b), and 24(c) illustrate the category-specific performance metrics for the results of the datasets such as CK+, FER-2013, and the HERD. This shows the lower effectiveness of underrepresented categories such as “fear” and “surprise” compared to dominant categories such as “happy” and “neutral.”

The classwise distribution of the dataset used in this study is shown in Figures 24(a), 24(b), and 24(c), including CK+, FER-2013, and HERD. The distribution represents the number of samples for each emotion category for each dataset, which highlights the differences in sample size between basic human emotion categories. The dataset shows significant differences in sample size between categories, with categories such as “fear,”, “disgust,” and “sadness” often underrepresented. These differences are the subject of this study. We investigate data augmentation methods to improve the generalization and classification accuracy of HER in HRI. Results show that due to a lack of training data, underrepresented categories such as “fear” and “surprise” have lower F1-scores, leading to higher misclassification rates. Oversampling techniques are used to rebalance the dataset during training to overcome this problem. Data augmentation methods such as random rotation, flipping, and brightness changes were then used to increase the sample size of these underrepresented categories artificially.

4.2.5. Impact of Augmentation on the Performance of ER

We created a new deep ensemble classification model for HER and tested it on the selected datasets, such as CK+, FER-2013, and HERD (with and without augmentation), to see how it performs. The results show that underrepresented categories significantly improve their performance after augmentation (Figures 25, 26, and 27). Upsampling, shuffle, rotation, and brightness variation are some of the data augmentation techniques used to equalize the dataset and reduce the effects of imbalance artificially. The data showed significant improvements in classification accuracy for underrepresented categories. An example of the effectiveness of these methods in reducing the imbalance of the dataset is the 13% increase in the F1-score for “disgust” and the 11% increase in the F1-score for “fear” after augmentation.

Figure 25 shows the comparison of the accuracy scores of the ER category classification model before and after data augmentation. The accuracy for a given class (disgust, sad, anger, neutral, fear, surprise, and happy) is the ratio between the number of expected cases and the number of accurately predicted events. The results demonstrate that the data augmentation of the dataset successfully removes the imbalance in the datasets and provides significant improvements in underrepresented categories such as “fear” and “disgust” of a proposed novel deep ensemble classification model.

We analyze each HER category before and after data augmentation, as shown in Figure 26, the performance comparison, and the recall scores for different emotion categories. Recall evaluates the ability of a proposed novel deep ensemble classification model to recognize all instances of a particular class. This improvement significantly improves the recall of underrepresented categories such as “fear” and “sadness,” suggesting that the model’s ability to detect and recognize these emotions of humans is improved by correcting the class imbalance in all the datasets used.

The F1-score of each HER category is assessed before and after applying the data augmentation strategy, as illustrated in Figure 27, and the harmonic means of recall and precision. The F1-score shows a significant increase after data augmentation, especially for some categories such as “disgust” and “fear,” demonstrating the overall effectiveness and balance of the proposed novel deep ensemble classification model.

The confusion matrix provides a comprehensive examination of the performance of the recently proposed deep ensemble classification model for classifying different emotion categories with HER in the context of HRI, as shown in Figure 28. The matrix shows that the model is not only relatively accurate for general categories such as “happiness” and “neutrality” but also misclassifies underrepresented categories such as “disgust” at a very high rate. We implement data augmentation techniques to mitigate these limitations, as shown in Figures 25, 26, and 27. The results show that F1-scores for the underrepresented categories (“fear” and “surprise”) increased by more than 10% after augmentation. This demonstrates the effectiveness of these techniques in eliminating class imbalance and strengthening the overall model. The results show that model performance is significantly affected by imbalanced datasets, especially those with underrepresented classes. These imbalances are corrected by oversampling and augmentation, which significantly improves the generalization and accuracy of the predictions. These results show how important it is to consider class imbalance in the datasets used for HER in HRI.

4.2.6. Scalability and Real-Time Effectiveness

Some sample images taken from the Intelligent Manufacturing Technology Institute lab shown in Figures 29(a) and 29(b) illustrate the importance of evaluating the scalability and real-time effectiveness of the proposed deep ensemble classification model for potential applications in the HRI domain. Further simulations were then performed to evaluate the effectiveness of the model in various real-world scenarios. These include performing a latency-based scaling analysis (see Figure 30) and evaluating light fluctuations (see Figures 31, 32, and 33).

Figures 31, 32, and 33 show the results of the annotated HER for the image under three different lighting conditions as shown above, that is, low lighting, normal lighting, and overexposed lighting. Each image contains a box showing the silhouette of a recognized human face and a title with the associated mood and confidence level. These above results illustrate the effectiveness of a revolutionary deep ensemble classification model for HER for a variety of lighting conditions using HRI. The first image simulates low light conditions by reducing the brightness, while the second image serves as a reference point and illustrates normal lighting conditions. The last image is an illustration of overexposed light, characterized by a significant increase in brightness. A frame surrounds each recognizable human face emotion, and key emotions and confidence levels are clearly shown. The results demonstrate the reliability of the proposed model in challenging lighting scenarios, which is crucial for real-world HRI applications.

Figure 34 shows waveforms of emotions that show how the intensity of the emotions in the image taken in the IIMT lab fluctuates over time. Individual subplots show the development of the intensity of different emotions of human emotions (such as happiness, sadness, indifference, anger, surprise, fear, or disgust) over the course of the story. This is intended to illustrate how feelings change over time, and distinct colors and line patterns are used to make the waveforms more visible. This representation is ideal for real-time feedback systems and interactive systems that detect human emotions using HRI.

4.2.6.2. Impact of Lighting Conditions

To conduct experiments on real lighting conditions, as ER algorithms are highly dependent on the right lighting, which plays a critical role in the practical implementation, during image processing, certain features (e.g., low light) were implemented to represent poorer lighting conditions (e.g., at night or in dimly lit environments). Ambient light is the light inside and outside most buildings, while direct sunlight is an example of dazzling light that overexposed people try to imitate (Figure 35).

The confusion matrix evaluates the performance of the proposed HER ensemble classification model in categorizing seven basic different types of human emotion, such as anger, disgust, fear, happiness, sadness, surprise, and neutrality, from the spectrum of HE. Correctly classified samples are represented by diagonal elements, while misclassified samples represent misclassifications. We learn more about the strengths and weaknesses of the model from this image. For example, if certain emotions (such as fear and sadness) are frequently misclassified, this may indicate that the model needs further training or that the dataset needs to be improved. The ability of the model to accurately recognize HE is crucial for the practical implementation of HCI using the proposed ensemble model, as the overall distribution shows. The proposed ensemble classification model shows consistent performance and virtually no classification errors, even under challenging conditions such as low lighting, normal lighting, and overexposure lighting. The robustness of the model is graphically demonstrated under different lighting scenarios. In realistic implementations of HRI, the ability of a model to adapt to different lighting conditions is crucial, which is essential for real-world HRI applications.

4.2.7. Evaluating Ensemble Models, Traditional versus Deep Learning Approaches

We conduct experiments using FER-2013, CK+, and our custom-developed dataset (HERD) and evaluate the models based on accuracy, F1-score, and computational efficiency as shown in Table 10.

Handcrafted feature–based models (LBP + HOG + PCA + SVM) result in significantly poorer performance and are poorly adapted to real-world variations, especially on complex datasets such as FER-2013. The better performance of deep models is attributed to their superior performance in hierarchical feature learning, supported by the performance of EfficientNet, ViT, RegNet, and SE-ResNeXt. The proposed ensemble model outperforms both methods in terms of robustness and accuracy, especially on HERD, a dataset specifically developed for this purpose. Figure 36 shows that the proposed ensemble model performs better on the new datasets (HERD), CK+, and FER-2013 than handcrafted feature–based models and standalone deep learning models.

4.2.8. The Effectiveness of Computation and Its Application in the Real World

The accuracy, efficiency, and practical performance of the proposed ensemble classification model are evaluated. The advantages of EfficientNet, ViT, RegNet, and SE-ResNeXt are shown in Table 11, while their synergistic properties to achieve the highest accuracy in HER can be seen in Table 12. A comparison of the effectiveness of the proposed model with the number of parameters, inference time, and FLOPs of some state-of-the-art methods such as Densnet, Resnet50, and InceptionV3 is shown in Table 12. The method demonstrates the trade-off between computational requirements and accuracy, and the measurement results show the ease of computation and the highest accuracy of the method. When these latter properties are combined with EfficientNet, ViT, RegNet, and SE-ResNeXt, the accuracy, robustness, and generalization of the proposed model are significantly improved. These features are particularly important for HRI applications that need to recognize emotions with high accuracy in real time. Figure 37 shows the integration that takes advantage of the complementary nature of all extractors to achieve a higher accuracy of up to 99.90%. Therefore, this set shows the strength of feature fusion, as shown by the highest accuracy in Figure 38.

The integration of EfficientNet into the proposed model can improve the efficiency of parameters and computations. Table 12 shows that among the three networks, EfficientNet, ResNet50, and Densenet, the first one develops better feature extraction, with ResNet50 having 25.56 million parameters and Densenet having 7.98 million parameters. Real-time HCI systems usually have limited computational resources therefore optimized parameters mean higher quality features, which form the basis for EfficientNet to become an integral part of real-time HRI systems.

A comparative analysis of this work shows that the proposed model is significantly better compared to other alternatives such as DenseNet121, InceptionV3, and ResNet50. The arguments can be grouped around the following basic elements.

4.2.8.1. Accuracy Across Multiple Datasets

The proposed model achieves a 99.90% accuracy, which is higher than DenseNet121 (94.36%), InceptionV3 (89.23%), and ResNet50 (97.01%) shown in Figure 39. This improvement stems from the ensemble approach combining advanced feature extractors such as EfficientNet and ViT with complementary learning mechanisms (CNN, GRU, and MLP). The higher accuracy demonstrates the model’s ability to generalize effectively across datasets, including CK+, FER-2013, and HERD. The proposed model (99.90%) outperforms the results of DenseNet121%-94.36%, InceptionV3-89.23%, and ResNet50-97.01%, as shown in Figure 39. This is due to the nature of the ensemble approach, which combines the state-of-the-art features of extractors, ViT, and EfficientNet, with collaborative learning methods, CNN, GRU, and MLP. More precisely derived results show the generalization capability of the model on alternative datasets such as HERD, CK+, and FER-2013.

4.2.8.3. Analysis of Computational Complexity

A more in-depth analysis of computational complexity (quantified in FLOPs and other additional features) can be found in Figures 40 and 41. Compared to the higher computational cost (23.28 GMac, 91.68 million parameters), the large improvement in claimed accuracy is justified, as high accuracy is crucial in practice, such as in HRI, as such a trade-off is not only reasonable but often essential.

4.2.8.4. Real-Time Suitability

The inference time of the proposed model is 1.137 s, which is higher than the other models shown in Figure 42, but still within the practical range, and this high overall performance justifies the expected trade-off of a longer inference time due to complexity.

4.4.1. Strengths of the Proposed Model

The proposed classification architecture using ensembles involves the use of multiple deep models, including EfficientNet, ViTs, RegNet, and SE-ResNeXt, to achieve robust feature extraction and classification. The key benefits are as follows.

4.4.2. Limitations and Challenges

Despite the advantages of the proposed ensemble-based HRI model, some limitations still need to be addressed to make it more applicable and fairer in real-life scenarios. One of the main problems is the generalization of the dataset and the issue of bias. The benchmark dataset used here does not differentiate by ethnicity, gender, or age groups and this can lead to bias in real-world applications. In addition, the dataset may not include all population people, which could affect the fairness of the model. To address these limitations, more diverse datasets on ethnicity, environmental variables, and age will be used in future studies to increase generalizability. Another challenge is the complexity and feasibility of implementation. The high computing power required for integration-based models makes real-time implementation on in-car devices impractical. The latency time for inference (1.137 s) is also much higher than for the standalone model, which would make real-time execution in automotive applications impossible. By optimizing through information distillation, pruning, and quantization, the latency can be reduced while maintaining high accuracy, making the model suitable for practical use in HRI tracking.

Dealing with occlusions and extreme changes in facial expression is also a problem. The model relies heavily on preliminary images which can lead to lower accuracy when side views or partially occluded faces occur. In addition, the classification probability is lower for subtle and nonexaggerated facial expressions, which can reduce the classification performance. Adding a multiview face recognition process to make the model more robust and analyzing using 3D faces can improve the model’s ability to recognize emotions in different scenarios. Finally, ethical and privacy issues pose a major challenge when using facial recognition in human – robot surveillance systems. Continuous data collection and analysis raises concerns about data security, privacy, and user consent. In addition, algorithmic biases in the data during training can lead to inconsistent or discriminatory classification in the real world. To address this issue, on-device analytics and privacy-friendly models such as federated learning can increase security by minimizing the risks associated with data misuse and unauthorized access. Taking these limitations into account can improve a given model with greater accuracy, efficiency, fairness, and real-world applicability, making HER systems not only practical but also ethical.

4.4.3. Comparative Analysis of Existing Methods

Table 14 is a comprehensive comparison between the proposed ensemble classification model and existing deep learning models. SE-ResNeXt, RegNet, ViTs, and EfficientNet are the models recommended for integration for feature extraction. For the final classification, the ensemble uses CNN, GRU, and MLP classifiers. To summarize, the resulting model outperforms the traditional design and has the highest accuracy of 97.01%.

4.4.4. Future Directions and Improvements

Some important improvements can be made to further increase the usability of the proposed model. The MobileNet architecture is one of the areas of research for lightweight applications as it has the potential to achieve real-time performance on low processing capacity devices. It will increase the model’s adaptability to device processing in human monitoring systems and reduce its dependence on computationally intensive hardware. Another important goal of avoiding bias is to increase the fairness of the model across demographic groups. Using debiasing techniques to train systems to remove gender, age, and race biases can improve the fairness of the system and its generalizability in a variety of real-world applications. In addition, the functionality of the system can be improved by using multimodal ER. By combining body measurements such as pulse rate and skin conductance with facial expressions and speech, the model can gain a deep understanding of human emotions. When facial expressions are not sufficient to convey emotions in multisensory approaches, the recognition performance increases significantly, and by debugging and optimizing the model for practical use, we can maximize the model’s versatility, similarity, and effectiveness in a variety of applications. Ensemble-based HER models can significantly improve the accuracy of all-encompassing feature extraction, dataset generalization, and sentiment analysis. However, to achieve full implementation, issues such as computational cost, dataset diversity, and practical feasibility need to be addressed. This article focuses on areas in need of optimization and the most important factors to create a comprehensive solution for future research directions and implementation strategies.