Modern offshore wind turbines (OWTs) are constructed with increasingly long blades and slender towers to capture wind resources more effectively. Consequently, OWTs have become vulnerable to wind and sea wave excitations. Mitigations of unfavorable OWT vibrations have been extensively investigated, with the majority focusing on passive vibration control strategies with control performance sensitive to structural frequency changes. Nonlinear energy sinks (NESs) are regarded as effective vibration control methods because their broadband fashion is robust against variations in structural frequencies. A novel NES with an improved track profile that combines both second- and fourth-order polynomials (Track II NES) is proposed in the present study to improve the vibration mitigation effectiveness of traditional Track I NES with a track profile of a fourth-order polynomial only. Governing equations of a single-degree-of-freedom system with Track II NES are first established, and an equivalent linearization method is adopted to optimize the track profile and damping of the Track II NES. Moreover, a detailed 3D finite element model of a representative 5-MW OWT is developed. Control effectiveness of the Track II NES is examined under different structural stiffnesses and mean wind speeds and then compared with that of conventional tuned mass damper (TMD) and Track I NES. Numerical results showed that the Track II NES can effectively suppress displacement and acceleration responses of OWTs and outperform its counterpart Track I NES. Moreover, the Track II NES can obtain reduction ratios close to those of the TMD but with better robustness against the detuning effect.