To ensure the integrity and reliability of structures, it is of great importance to understand the fatigue damage mechanism of materials under different amplitude loading conditions. For this purpose, a nonlinear fatigue damage accumulation model based on material configurational forces is developed in this paper. Specially, since the material configurational force is defined on the material space point and directly associated with the evolution of material configuration, it is introduced as an internal variable to describe fatigue damage accumulation. Additionally, the corresponding relation between configurational force and stress state is presented. Meanwhile, the damage increasing among every loading cycle, that is, the damage rate, is dependent on the stress state and damage state variable. Therefore, the formulation of damage rate is proposed by a function of material configurational force and damage state variable. Further, numerical implementation of this proposed nonlinear fatigue damage accumulation model has been carried out through two notched structures to describe the fatigue damage evolution and predict the fatigue life of materials. The results indicate that the numerical results are consistent with previous experimental data.