Dentistry diseases are worldwide concern with the fact that 5% of medical budget is spent on it. Dental diagnosis requires frequent visits to clinic and multiple personal check-ups by an expert. This delays the process of diagnosis as well as introduces the danger of oral infection spread. Automation is boon to speed up the process of diagnosis, reduce expert's involvement and help in handling volume of patients. In situations like pandemic as seen in this decade, computer based automatic systems in health have proven their importance and necessity. For dental diagnosis, images are useful tool for better anatomical views and accurate decision of treatment. However, manual dental analysis increases load on dentists for initial check-up that can be easily performed with automated systems or self-kits. Oral cavity is a dental illness, if not identified and treated on time may lead to other serious ailments. This work presents a framework that performs tooth image classification for cavity and non-cavity with new bag of features (NBoF) method. NBoF method is an attempt to improve the performance of bag of features (BoF) using the proposed reinforcement Aquila optimization (RAO) and weighted Bayesian Gaussian mixture modelling (WBGMM). An analysis of the performance of the NBoF using the proposed RAO and WBGMM is conducted using standard metrics. The comparative study of results proves that the proposed NBoF method outperforms the existing state-of-the-art algorithms.