Cassava is a rich source of carbohydrates, and it is vulnerable to virus diseases. Literature survey shows that the image recognition and integrated deep learning approach is successfully employed for Cassava leaf disease classification. Mostly, transfer learning based on a convolutional neural network (CNN) models were successfully applied for Cassava leaf disease classification. However, existing approaches are not effective in identifying the tiny portion of the disease in the overall leaf area. Identifying and focussing on regions affected by the disease is vital to achieving a good classification accuracy. An attention-based approach is integrated into pretrained CNN-based EfficientNet models to locate and identify the tiny infected regions in Cassava leaf. Penultimate layer features of attention-based EfficientNet models such as A_EfficientNetB4, A_EfficientNetB5, and A_EfficientNetB6 were extracted. Next, the dimensionality of the extracted features was reduced using kernel principal component analysis. The reduced features were fused and passed into a stacked ensemble meta-classifier for Cassava leaf disease classification. A stacked ensemble meta-classifier is a two-stage approach in which the first stage employs random forest and support vector machine (SVM) for prediction followed by logistic regression for classification. Detailed investigation and analysis of the proposed method, attention, and non-attention-based approaches with CNN pretrained models were tested using a publicly available benchmark dataset of Cassava leaf disease images. The proposed method achieved better performances in all experiments than several existing methods as well as various attention and non-attention-based CNN pretrained models. The proposed approach can be used as a deployable tool for Cassava leaf disease classification in agricultural field.