Breast cancer is detected by identifying malignancy on breast tissue. Emerging technologies in medical image processing are used to interpret histopathology images. For analyzing medical imaging and pathological data, modified deep neural networks are being used. Automatic detection of malignancy is usually achieved in deep learning by capturing features from a convolutional neural network (CNN) and then categorizing them using a fully connected network. A framework to automatically diagnose malignancy using an ensemble approach, including CNN and extraction of image texture features, is implemented in this research. In the CNN phase, the nine-layer modified CNN is used to classify images. Texture features are derived and their dimension is minimized using maximum variance unfolding to enhance the efficiency of classification in the extraction-based phase. The results of each phase were then merged to obtain the final decision. The testing specificity and accuracy of our ensemble method on MIAS repository are 98.9% and 99%, respectively and for the DDSM repository are 98.3% and 98.1%. The ensemble approach increases the measurement metrics compared to each phase separately, as per the experimental findings.