Cables are among the most important structural components of cable-stayed bridges; however, their vibrations due to the wind and vehicles significantly affect the serviceability of bridges. To overcome the expensive and time- and effort-intensive installation of sensors in measuring cable vibrations, computer-vision-based displacement methods have been introduced. Even with the improved accuracy and convenience of vision-based displacement methods, there are two main issues in their application in measuring cable vibrations: the manual selection of the region of interest (ROI) and loss of feature points using traditional tracking algorithms. To address these, this paper proposes a new specialized method to measure the structural displacement of cables. The proposed method automatically selects the ROI using a convolutional neural network and tracks the feature points in the cable more robustly using a modified Kanade–Lucas–Tomasi (KLT) algorithm. The automatic cable ROI selection network (Cable-ROI Net) predicted the cable with an intersection over union of 89.59%. Meanwhile, modified KLT (Uni-KLT) successfully measured the cable displacement with a root mean square error of 1.077 pixels, the conventional KLT failed to track the feature points. The on-site validation test at a cable-stayed bridge in Korea demonstrated the ability of the proposed method to automatically select the ROI and successfully measure the cable vibration for the entire duration of 30 min, whereas the traditional method required manual ROI selection and lost all feature points four times throughout the test. Therefore, the proposed method is expected to improve the current methods for measuring cable vibrations.