1 Introduction

Glaucoma damages the optic nerve, leading to vision loss [1]. It is the second leading cause of permanent blindness worldwide [2]. Glaucoma is a progressive optic neuropathy that is associated with impairments in the visual field. According to estimations, the number of individuals between the ages of 40 and 80 with glaucoma will rise from 64.3 million in 2013 to 76 million in 2020, and is projected to reach 111.8 million by 2040 [3].

Previous studies have identified several risk factors associated with developing glaucoma, including age, race, gender, intraocular pressure, diabetes, and a family history of the disease [4, 5]. Research conducted on populations aged 40 and above reports a prevalence of glaucoma ranging from 1.44% to 4%. Notably, over 50% of glaucoma patients were unaware of their condition. Additionally, the first phase of the Shahroud Eye Cohort Study (SHECS) revealed that older age and higher intraocular pressure were associated with a greater likelihood of developing glaucoma [6].

Retinal image analysis is a diagnostic tool used in the identification of glaucomatous optic neuropathy where it assesses indicators such as a high cup disc ratio (CDR), optic disc hemorrhage, and retinal nerve fiber layer (RNFL) defects [7]. Recently, various advanced tools have been developed to aid in glaucoma diagnosis, including optical coherence tomography (OCT). OCT is a non-invasive and non-contact imaging technology that allows the quantitative assessment of retinal structures, providing objective measurements of the cornea and the optic nerve head (ONH) [8]. Additionally, the detection of RNFL thinning and the loss of ganglion cells in the early stages using OCT can facilitate the early diagnosis of glaucoma [9]. RNFL thinning occurs even in the early stages of glaucoma and is therefore considered a sensitive indicator of glaucomatous damage. Macular thickness decreases in eyes affected by glaucoma, whereas in normal eyes, the macula includes the area around the fovea with the highest density of retinal ganglion cells (RGCs) [10]. OCT can diagnose glaucoma with high sensitivity and specificity by examining the optic disc and macular region [11]. Diagnosing glaucoma requires measuring various OCT scan parameters, including disc topography, peripapillary RNFL thickness, RGC [12] layer thickness (including the inner plexiform layer) [13], and ganglion cell complex (GCC) thickness [14].

Accurate and early diagnosis of glaucoma remains a critical challenge in ophthalmology, particularly due to data scarcity and imbalanced datasets. One of the significant challenges in existing methods for glaucoma diagnosis is the imbalanced datasets, characterized by a scarcity of cases in the glaucoma class and a limited availability of data pertaining to glaucoma. These issues are recognized as challenges in classifying imbalanced data, and various methods have been proposed to address them. However, another important challenge is the low accuracy and sensitivity in the early stages of the disease. Many traditional algorithms encounter difficulties in detecting subtle changes in retinal structures and often depend on manual feature extraction, a process that is both time-consuming and susceptible to errors. These problems make timely and accurate diagnosis of glaucoma a major challenge. Previous studies have attempted to address these issues through various deep learning techniques, but they often struggle with generalization due to the limited availability of glaucoma data and the complexity of retinal structural changes.

Nowadays, the advantages of deep learning models in medical image processing have become increasingly prevalent [15-17]. But deep learning requires a large amount of data to minimize over-fitting and improve performance [18-20]. However, obtaining large datasets for medical images of rare diseases is challenging. Therefore, using deep learning for diagnosing rare diseases in medical images can prove challenging. Consequently, a more effective classification strategy is essential for handling small datasets. For instance, in [21] the glaucoma dataset comprises only 2.34% of the total, highlighting a significant challenge in handling highly imbalanced datasets.

The main focus of this article is to address the challenges of highly imbalanced data and limited availability of glaucoma images through a novel, multi-faceted approach. The dataset consists of a normal-to-glaucoma image ratio of over 30:1. To address this, the glaucoma data were expanded sixfold using data augmentation, while the normal data were divided into five categories. Glaucoma data were then added to each category to reduce the imbalance, enabling the training of five separate models. In addition to this data strategy, the proposed method introduces three key innovations. First, multi-scale feature extraction was implemented to capture information at varying resolutions, enhancing the model's ability to detect both localized and global patterns. Second, the cross-attention mechanism was integrated to learn relationships between features across different scales, refining contextual understanding. Third, the CSAM was employed to improve feature selection, focusing on critical channels and regions relevant to glaucoma. These innovations, combined with a weighted ensemble learning technique for integrating individual models, significantly improve diagnostic accuracy. This approach not only enhances the timely and accurate diagnosis of glaucoma but also provides a robust framework for addressing similar challenges in the diagnosis of rare diseases with small and imbalanced datasets.

2 Proposed Method

In this study, we addressed the challenges of limited glaucoma image availability and significant class imbalance between normal and glaucoma (abnormal) images. Data augmentation techniques, such as flips and scaling, were applied to increase the number of glaucoma images. Despite this, the imbalance persisted. To mitigate this, normal data were partitioned into five folds, and all augmented glaucoma data were included in each fold. This enabled the training of five independent models using a convolutional neural network (CNN), and a weighted voting mechanism was employed for final classification, forming an ensemble learning framework.

To further enhance the model's performance and feature extraction capabilities, we integrated key components into the architecture: multi-scale feature extraction, a cross-attention mechanism, and the CSAM. These components were designed to capture features at multiple resolutions, refine contextual relationships, and focus on critical image regions, significantly improving the model's accuracy and interpretability [22]. The proposed method consists of two main stages: training and testing. Figure 1 illustrates the major steps in our approach, which are detailed in the following sections.

2.3 Data Pre-Processing

After data preparation, pre-processing was performed to enhance image quality and improve model performance. The pre-processing pipeline consists of two main steps: Unsharp masking for image enhancement and data augmentation to address class imbalance. These steps were applied to all images in the dataset, as illustrated in Figure 2.

• Data Augmentation

  Data augmentation is a technique used in image processing that can be used to increase OCT data. In OCT imaging, data augmentation artificially expands the size of the training dataset by applying various modifications to the original images. This approach enhances the performance and generalization of deep learning models that are trained on OCT images. Some common data augmentation techniques used in OCT image analysis include:

• Rotation: Rotating the image at a specific angle to simulate different scanning orientations in OCT images.
• Flipping: Flipping the image horizontally or vertically creates mirrored images.
• Scaling: Resizing the image to simulate different magnification levels.
• Translation: Shifting the image horizontally or vertically to simulate different scan positions in OCT images.
• Adding noise: Adding random noise to the image to enhance the model's robustness against noise in real-world conditions.

By applying data augmentation techniques, the model can effectively detect patterns and features in OCT images, leading to improved performance in tasks such as image classification, segmentation, and disease detection. Figure 3 illustrates examples of these augmentation techniques applied to both normal and glaucoma OCT images, demonstrating how these transformations enhance the model's ability to learn diverse patterns. Notably, data augmentation was specifically employed to increase the number of glaucoma samples, resulting in a sixfold expansion of the original glaucoma dataset, which helps mitigate class imbalance and improve glaucoma detection accuracy.

• Unsharp masking for image enhancement

  Unsharp masking is a widely used image enhancement technique that increases contrast and sharpness, making subtle retinal structures more distinguishable. This technique enhances OCT images by emphasizing fine details, which are crucial for detecting early-stage glaucoma.

As shown in Figure 4, Unsharp masking significantly enhances the clarity of OCT images. The figure presents examples of this technique applied to both normal and glaucoma images, highlighting how contrast and fine structures are improved. This enhancement aids the deep learning model in extracting more relevant features for classification [24].

In the testing phase, after data pre-processing, the proposed model was used to classify the images into two classes: normal and glaucoma.

2.4 Proposed Model Architecture

The proposed model, as depicted in Figure 5, integrates a pre-trained ResNet backbone, multi-scale feature extraction, cross-attention mechanisms, and a CSAM to effectively address the challenges posed by imbalanced and complex OCT datasets. Each component of the architecture is meticulously designed to enhance feature representation, focus on relevant regions, and improve classification performance, specifically for glaucoma detection.

• Pre-trained ResNet backbone

  The architecture employs a pre-trained ResNet model as the primary feature extractor. ResNet's success in handling vanishing gradients through its deep residual connections makes it an ideal choice for medical imaging tasks, where fine-grained structural details are critical. By leveraging a pre-trained network, the model benefits from transfer learning, effectively using features learned from large-scale datasets.

  In our design, the ResNet backbone is segmented into four hierarchical blocks, each progressively refining feature representations. Feature maps extracted from these blocks undergo a transformation via $$$ 1\times 1 $$$ convolutions to reduce dimensionality while preserving spatial information. This dimensionality reduction is essential for computational efficiency and facilitates integration with subsequent multi-scale processing layers. The pre-trained ResNet enables the model to extract rich semantic features, capturing the subtle morphological changes in retinal layers indicative of glaucoma.

• Multi-scale feature extraction

  Given the varied spatial resolutions and structural features in OCT images, the architecture incorporates a multi-scale feature extraction strategy. This step ensures that the model captures details from both local (fine) and global (coarse) perspectives. Feature maps from different stages of the ResNet backbone are pooled and aligned to a uniform dimensionality using $$$ 1\times 1 $$$ convolutional layers. This operation ensures compatibility among feature maps of varying resolutions while retaining essential spatial and contextual information.

  The multi-scale design enables the model to effectively analyze small, localized RNFL thinning while simultaneously considering global structural changes, such as variations in ONH morphology. By combining information from different scales, the model builds a comprehensive understanding of glaucomatous patterns.

• Cross-attention mechanism

  To further enhance the feature representation, the architecture integrates a cross-attention mechanism. This module operates on the multi-scale feature maps and learns dependencies across different resolutions. By incorporating cross-attention, the model can focus on complementary information derived from varying receptive fields.

  Specifically, the cross-attention module employs query (q), key (k), and value (v) matrices to compute attention weights. These weights guide the model in identifying critical regions and relationships between feature maps, enabling the network to prioritize features that contribute most to distinguishing glaucomatous patterns. Unlike self-attention mechanisms, cross-attention promotes interaction between feature maps of different scales, enriching the spatial and contextual understanding of the input image. The outputs of the cross-attention module are subsequently concatenated, creating a unified and discriminative feature representation.

• Channel and Spatial Attention Module (CSAM)

  The CSAM is a pivotal component of the architecture, designed to enhance feature selectivity by focusing on informative channels and spatial regions. The CSAM module combines two complementary attention mechanisms:

  Channel attention: The channel attention mechanism selectively emphasizes important channels within feature maps, highlighting features most relevant to the diagnosis of glaucoma. This process begins with global average pooling to generate a channel descriptor vector, summarizing the global context for each channel. This descriptor is passed through two fully connected layers with non-linear activation functions (ReLU and Sigmoid), resulting in a set of attention weights. These weights are applied to the feature maps through element-wise multiplication, amplifying significant channels while suppressing irrelevant ones.

  Spatial attention: The spatial attention mechanism identifies the most critical regions within the OCT images by generating a spatial descriptor map. This map is computed through a convolutional operation on the input feature maps, followed by activation layers (ReLU and Sigmoid). The spatial attention weights are then multiplied with the input feature maps, enabling the model to focus on regions exhibiting glaucomatous damage, such as localized RNFL thinning or ONH changes.

  The CSAM module combines these two attention mechanisms to create a refined feature representation, effectively addressing the inherent complexity and variability in OCT images. By enhancing both channel-level and spatial-level feature discriminability, the CSAM block significantly improves the model's ability to detect subtle pathological changes.

• Integration and classification

  The outputs from the CSAM modules are concatenated and passed through fully connected layers to reduce the dimensionality of the feature representation. These layers are configured with dropout and batch normalization to mitigate overfitting and stabilize learning. The final classification layer employs a softmax activation function, outputting probabilities for the two classes (normal and glaucomatous). This classification stage ensures that the model effectively translates the learned feature representations into accurate predictions.

  In the final stages of the network, before the dense layers, we utilized the Global Average Pooling (GAP) operator to reduce the feature map to 256 neurons. GAP operates by computing the average of each feature map, effectively transforming the spatial dimensions of the feature map into a single value per feature. This helps to significantly reduce the number of parameters and computational complexity, while also mitigating the risk of overfitting. By using GAP, the model is able to focus on the most important global features across the entire image, ensuring that the spatial information is summarized efficiently before being passed to the fully connected layers for final classification or prediction.

  The integration of the pre-trained ResNet backbone, multi-scale processing, cross-attention mechanisms, and CSAM modules enables the proposed architecture to excel in the analysis of OCT images. The model not only achieves robust feature extraction but also enhances interpretability, allowing clinicians to concentrate on critical regions highlighted by the attention mechanisms. This revised architecture exhibits superior performance in glaucoma diagnosis, effectively addressing significant challenges such as feature variability, data imbalance, and the subtlety of pathological changes within retinal structures.

3 Results

Due to the limited size of the glaucoma image dataset and the unsatisfactory results without data augmentation, which led to model overfitting, we opted not to present the results of models without data augmentation. This paper's two main innovations are the introduction of the CSAM module and the application of weighted ensemble learning. Consequently, the following subsections present the results obtained through the model with the CSAM module, the model without the CSAM module, and the integration of the CSAM module with weighted ensemble learning into the model. Additionally, the final model is compared with previous works.

The experiments were conducted separately in two stages. In the first stage, all OCT images in the database were categorized into two classes: diseased and healthy, and the proposed model was trained accordingly. In this instance, 10% of the data was allocated for the test set, and 90% for the training set. Subsequently, 10-fold cross validation was employed for training, yielding 10 runs for each category where the best model on the validation data was obtained.

In the modeling stage, 10% of the data (91 OCT images) were randomly selected for the test set, consisting of 11 glaucoma images and 80 healthy images. The results of all models on this dataset are reported. The remaining images comprised 85 glaucoma images and 3690 healthy images. Subsequently, the remaining glaucoma images were augmented sevenfold using data augmentation algorithms, including rotation, flipping, cropping, translation, and noise addition.

After data augmentation, the number of normal datasets increased to approximately six times that of the glaucoma datasets. Moreover, since the number of models for ensemble learning must be odd, the normal OCT image dataset was partitioned into five groups. Subsequently, all glaucoma datasets were included in each group. Each group then underwent 10-fold cross-validation during training, resulting in the creation of 10 models per group. The top-performing model from these 10 models was selected for evaluation. The experimental results for the best model from each group on the validation data are presented in Table 2.

Furthermore, to provide a comprehensive inspection of the models' performance, Table 3 presents the average results from experiments on validation data, accompanied by the standard deviation index for each set of 10 models per fold.

Based on the findings from Table 3, we ranked the models and assigned corresponding weights. The model based on fold1, achieving the highest accuracy, was assigned a weight of 5, while models 3 and 5 were given weights of 3 each, and models 2 and 4 received weights of 1 each. This resulted in a total weight sum of 13. Employing the weighted ensemble learning technique, we applied a decision threshold of 7 or higher for classifying the output class.

In Bagging Ensemble Learning, the evaluation model usually incorporates a voting mechanism to combine predictions from the base models. Majority voting is the most common voting mechanism for classification models. Each base model makes a prediction, and the class with the most votes is selected as the final prediction. Therefore, the label of the class that receives the highest number of votes from the base models is considered the predicted class label for the ensemble. In this study, we employed “weighted ensemble learning” to determine the final result from the output of the final layer.

The performance of each of the five models, as well as the final model on the test dataset, is detailed in Tables 4 and 5.

We used the same evaluation dataset for both the proposed model and the pre-trained networks to assess and compare their performance. These pre-trained networks were trained by the researchers involved in this study. In doing so, all layers of the models except the final layer were frozen, and the weights of the final layer were updated using the training data. The evaluation results on the test dataset are detailed in Table 6.

Upon comparing the results of the proposed model with those of pre-trained networks, it was found that the proposed model achieved an accuracy of 98.90%, surpassing pre-trained models by 2.2%. Ensemble learning methods integrate predictions from multiple individual models to enhance overall accuracy and mitigate errors. In contrast, pre-trained models are neural networks trained on extensive datasets for specific tasks. While pre-trained models can excel in specific scenarios, they may not consistently outperform ensemble learning methods in all contexts. Ensemble learning methods excel in capturing diverse aspects of data and learning from multiple perspectives, thereby enhancing overall model performance. Their capability to harness the strengths of multiple individual models and integrate predictions effectively often leads to superior performance compared to pre-trained models, as they mitigate errors and exploit collective insights more effectively.

In this study, we calculated mean, SD, F-value, and p value to determine whether there are any significant differences between the means of five groups and indicated no statistically significant differences among the groups under examination.

3.2 Robustness Analysis Against Noise and Low-Contrast Images

To comprehensively evaluate the robustness of the proposed model, we conducted experiments by introducing Gaussian noise and Salt and Pepper noise into the dataset [27].

To construct a robust classification framework, we categorized the normal data into five folds while ensuring that all glaucoma data were included in each fold. This resulted in five distinct folds of data. For each fold, we trained an independent CNN-based model, leading to the creation of five separate models.

To simulate real-world noise conditions, we introduced:

• Gaussian noise with variance levels (σ = 0.1, 0.3, 0.6)
• Salt and Pepper noise with noise densities (d = 0.1, 0.3, 0.6)

Each model was independently trained and tested using the noisy images. To enhance the overall classification performance and mitigate the impact of noise, we employed an ensemble learning technique based on a weighted voting mechanism. This approach aggregates predictions from all five models to determine the final class label for each image. Figure 6 shows an example of a noisy image, illustrating how adding Gaussian and salt and pepper noise significantly degrades OCT image clarity and simulates the real-world noise conditions under which the model was tested.

As observed in Tables 8 and 9, the ensemble-based strategy effectively stabilizes classification accuracy despite increasing noise levels, confirming the robustness of the proposed approach.

TABLE 8. The results of the second stage modeling experiment with the ensemble technique with Gaussian noise.

	Accuracy %	Precision %	Recall %	F1_score %	TNR %	MCC %	Balanced accuracy %
σ = 0.1	96.70	90.00	81.81	98.13	81.81	83.98	81.81
σ = 0.3	95.60	91.81	81.81	97.50	81.81	79.32	81.81
σ = 0.6	94.50	71.42	90.90	96.81	90.90	77.62	90.90

TABLE 9. The results of the second stage modeling experiment with ensemble technique withsalt-and-pepper noise density.

Noise density	Accuracy %	Precision %	Recall %	F1_score %	TNR %	MCC %	Balanced accuracy %
0.1	95.60	81.81	81.81	97.50	81.81	79.32	81.81
0.3	94.50	75.00	81.81	96.85	81.81	75.22	81.81
0.6	9340	72.72	72.72	96.25	72.72	68.98	72.72

4 Discussion

This study proposed an ensemble learning technique for the effective classification of imbalanced data in a private OCT image database used for diagnosing glaucoma. The Ensemble Learning Bagging method was employed to manage the imbalanced dataset. Data analysis and modeling were conducted using convolutional deep neural networks.

According to Table 6, the experimental results indicated that the proposed model, when used in the weighted ensemble learning technique, enhances the diagnosis of glaucoma with an accuracy of 98.90%, a precision of 91.67%, a recall of 100%, an F1-score of 95.65%, a TNR of 98.75%, and a balanced accuracy of 99.37%. Additionally, the proposed model's accuracy in diagnosing glaucoma is 2.2% higher than that of the pre-trained models used in this study. This significant improvement is primarily attributed to the inclusion of key components, such as the multi-scale feature extraction phase, the cross-attention mechanism, and the CSAM module, as evidenced by the results of the ablation study (Table 8). Each component demonstrated a measurable contribution to the overall performance, emphasizing their critical role in achieving state-of-the-art diagnostic accuracy.

Furthermore, the proposed model has fewer parameters compared to the pre-trained models, making it more efficient. The multi-scale feature extraction module allows the model to effectively capture patterns at varying resolutions, while the cross-attention mechanism improves the contextual integration of features from different scales. Additionally, the CSAM module enhances both channel- and spatial-level focus, allowing the network to concentrate on the most relevant features for glaucoma detection. These advancements collectively refine the model's feature representation capabilities, leading to the observed performance gains.

The utilization of OCT images in diagnosing glaucoma presents several advantages, such as high resolution, quantitative measurements, early detection capabilities, non-invasiveness, objective analysis, subgroup differentiation, treatment monitoring, and compatibility with artificial intelligence technologies. These attributes render OCT a valuable asset in the clinical assessment and management of glaucoma. Numerous studies have explored diverse methods for diagnosing glaucoma, with Table 7 outlining specifics from prior research in this area. Unlike prior studies, this work uniquely integrates advanced attention mechanisms, such as the cross-attention and CSAM modules, with multi-scale feature extraction to enhance the diagnostic process. This combination results in significant improvements over existing methods.

Moreover, the proposed model demonstrates a capacity for generalization across imbalanced datasets, which is critical in medical imaging scenarios where pathological cases are often underrepresented. The ensemble learning strategy further stabilizes predictions by mitigating the effects of class imbalance, and the improved feature representations from the novel components enable better handling of subtle variations in OCT images. These aspects ensure the model's robustness and clinical applicability.

5 Conclusion

This study introduced a weighted ensemble learning technique to address imbalanced data in deep learning tasks, using Phase III OCT images from the Shahroud Eye Cohort. The proposed model achieved superior diagnostic accuracy (98.90%) compared to traditional transfer learning methods, largely due to the integration of multi-scale feature extraction, cross-attention mechanisms, and the CSAM module. These components enhanced feature refinement and focus, enabling more precise glaucoma detection.

While the model effectively distinguishes between individuals with glaucoma and healthy controls, it does not yet address glaucoma subtypes. Future research should focus on developing models for subtype diagnosis and integrating multimodal data, such as demographic, genetic, and visual field information, to enhance diagnostic performance. Additionally, validating the model on diverse datasets from different populations and imaging devices is crucial for improving its generalizability and real-world applicability.

Beyond these aspects, several research directions remain open for further exploration. Developing explainable AI (XAI) frameworks could enhance model transparency, fostering greater trust among clinicians. The integration of OCT imaging with complementary modalities such as fundus photography and perimetry data may further improve diagnostic accuracy. Moreover, optimizing the model for real-time applications in portable imaging systems could facilitate early detection in resource-limited settings. Another promising direction is extending the model to monitor glaucoma progression over time, enabling personalized treatment strategies. Addressing variability in image quality, resolution, and noise from different imaging systems is also essential for ensuring robust performance across diverse clinical environments.

This study highlights the potential of advanced attention mechanisms and OCT imaging in creating robust diagnostic tools for ophthalmology, paving the way for improved clinical outcomes. By addressing the outlined challenges and research gaps, future studies can contribute to the development of more interpretable, scalable, and clinically viable AI-driven solutions for glaucoma diagnosis and management.

Author Contributions

Hamid Reza Khajeha: software, writing – original draft, data curation, conceptualization. Mansoor Fateh: writing – review and editing, supervision, conceptualization. Vahid Abolghasemi: resources, writing – review and editing. Amir Reza Fateh: validation, visualization, writing – review and editing. Mohammad Hassan Emamian: validation, formal analysis, resources. Hassan Hashemi: resources, validation. Akbar Fotouhi: resources, validation.

Conflicts of Interest

The authors declare no conflicts of interest.