In modern communication systems, digital predistortion (DPD) improve the linearity of PA and reduce the influence of nonlinear behavior on signal transmission. When identifying the inverse model of PA with traditional variable step size algorithm, the rate of convergence is slower, easy to be effected by noise and convergence instability exists at the optimal solution, which affects the optimization degree of the whole predistortion system. In this article, a digital predistortion technique based on improved NLMS algorithm is proposed to improve the accuracy of inverse model identification, convergence rate and anti-noise performance of the predistorter by changing the iteration function and introducing the autocorrelation matrix of the error of adjacent moments. The instability of the system at the optimal solution is reduced by introducing a new weight coefficient to update the correlation term. The simulation results show that the anti-noise performance, convergence rate and stability of the predistortion system based on New variable step size NLMS algorithm(NVNLMS) are obviously better than NLMS, and the out-of-band suppression of predistortion is optimized by 10 dB compared with the original system EVM is improved by 0.0031, ACPR is improved 9.1 dB.