An infectious lung illness known as “tuberculosis” (TB) causes more than 1.5 million casualties each year. TB has been a major problem in South Asia's low-income countries that have limited access to specialized radiologists for diagnosis. Despite the strong performance of the existing deep learning-based diagnosis, most of them are based on extremely small datasets, such as the Montgomery County and Shenzhen (MCS) dataset, which contains fewer than 1,000 images. The differentiation of sick X-ray images from chest X-rays, along with the recognition of normal and TB images, has also become necessary in the COVID-19 era in order to help the doctors prepare for the diagnosis of other diseases. Our effort intends to create an ensemble model that can classify chest X-rays as sick, health, or TB images. For this, we proposed two ensemble-based models by merging three fine-tuned state-of-art classifiers like SqueezeNet, ChexNet, and EfficientNet-B0. The MCS dataset and the largest TB dataset, TBX11K, are used to fine-tune these classifiers. We have experimented with methods of lung segmentation, image enhancements, and multichannel input to boost the performance of the classifiers. For the purpose of segmenting the lung, we created a U-Net model. Three filtering techniques comprising contrast-limited adaptive histogram equalization (CLAHE), high-emphasis filter (HEF), and bilateral filter (BF) have been used to enhance the hidden features. The three-channel input image produced by combining these filtered images was used to fine-tune our classifiers. The two types of ensemble models, the sum of probabilities (SOP) and stacked generalization (SG), were then developed using these refined models. On the TBX11K dataset, the top model derived from the experiment has accuracy, precision, recall, and F1-score values of 97.94%, 98.06%, 97.81%, and 97.93%, correspondingly. Our model demonstrated competitive performance on the MCS datasets, demonstrating the credibility and robustness of the approach.