The identification of surface damage and structural components is critical for structural health monitoring (SHM) in order to evaluate building safety. Recently, deep neural networks (DNNs)–based approaches have emerged rapidly. However, the existing approaches often encounter catastrophic forgetting when the trained model is used to learn new classes of interest. Conventionally, joint training of the network on both the previous and new data is employed, which is time-consuming and demanding for computation and memory storage. To address this issue, we propose a new approach that integrates two continual learning (CL) algorithms, i.e., elastic weight consolidation (EWC) and learning without forgetting (LwF), denoted as EWCLwF. We also investigate two scenarios for a comprehensive discussion, incrementally learning the classes with similar versus dissimilar data characteristics. Results have demonstrated that EWCLwF requires significantly less training time and data storage compared to joint training, and the average accuracy is enhanced by 0.7%–4.5% compared against other baseline references in both scenarios. Furthermore, our findings reveal that all CL-based approaches benefit from similar data characteristics, while joint training not only fails to benefit but performs even worse, which indicates a scenario that can emphasize the advantage of our proposed approach. The outcome of this study will enhance the long-term monitoring of progressively increasing learning classes in SHM, leading to more efficient usage and management of computing resources.