RESEARCH ARTICLE
Image processing–based real-time displacement monitoring methods using smart devices
Ashish Shrestha,
Ji Dang,
Keisuke Nakajima,
Xin Wang,
Corresponding Author
Ashish Shrestha
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Correspondence
Ashish Shrestha, Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan.
Email: [email protected]
Search for more papers by this authorJi Dang
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Search for more papers by this authorKeisuke Nakajima
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Search for more papers by this authorXin Wang
Department of Architecture, Faculty of Science and Technology, Tokyo University of Science, Chiba, Japan
Search for more papers by this authorAshish Shrestha,
Ji Dang,
Keisuke Nakajima,
Xin Wang,
Corresponding Author
Ashish Shrestha
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Correspondence
Ashish Shrestha, Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan.
Email: [email protected]
Search for more papers by this authorJi Dang
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Search for more papers by this authorKeisuke Nakajima
Department of Civil and Environmental Engineering, Saitama University, Saitama, Japan
Search for more papers by this authorXin Wang
Department of Architecture, Faculty of Science and Technology, Tokyo University of Science, Chiba, Japan
Search for more papers by this authorSummary
Displacement measurement is one of the most important methods for structural health monitoring. However, because of high cost and heavy on-site installation, conducting displacement measurement using conventional sensors is not easy in practice. In this case, with improvements in image sensors in smart devices, it seems possible to measure displacement using image processing methods. In this study, smart device–based bridge displacement monitoring system was developed. Images captured by the image sensor are processed in real time to obtain the feature-responsive displacement. First, shaking table tests based on sine wave loading were conducted so that the reliable domain for frequency and amplitude measurement for different smart devices were identified by comparing with the reference-accurate displacement sensor. Then, on-site application and stability of the proposed system were demonstrated through 1-day field measurement on a real in-service bridge. Comparison of the displacement due to traffic and temperature using smart devices with accurate displacement sensor shows significant potential of the proposed approach.
REFERENCES
- 1Takahashi, Y. (2012). “ Damage of rubber bearings and dampers of bridges in 2011 Great East Japan Earthquake.” Proceedings of the International Symposium on Engineering Lessons Learned From the 2011 Great East Japan Earthquake, pp. 1333-1342
- 2Matsuda, T., Kakimoto, R., Suzuki, M., Yamao, T., Kitazono, Y. and Takahasi, Y. (2016). “ Quick report on the 2016 Kumamoto Earthquake. Japan Society of Civil Engineers.” International Activities Center, IAC News, Special Issue no. 10
- 3Ikeda, T., Konagai, K. and Kiyota, T. (2016). “ Preliminary report of geotechnical and structural damage along the surface rupture in Nishihara Village caused by the April 16th 2016 Kumamoto Earthquake.” JSCE Journal of Disaster FactSheets, FS2016-E-0004
- 4Hayashi, K., Adachi, Y., Sakamoto, N., Igarashi, A. and Dang, J. (2015). “ Experimental evaluation of seismic residual performance for deteriorated rubber bearings in highway bridges.” 11th International Workshop on Advanced Smart Materials and Smart Structures Technology, University of Illinois, Urbana-Champaign, United States
- 5 JSCE. Infrastructure health report (2016). Road Sector Trial Version.
- 6 White Paper on Land, Infrastructure, Transport and Tourism in Japan, 2015. Ministry of Land, Infrastructure, Transport and Tourism
- 7Fujino, Y. and Siringoringo, D. M.(2011). “Bridge monitoring in Japan: The needs and strategies.” Struct Infrastruct Eng, vol. 7, no. 7, pp. 597-611, First published on: 14 July 2010 (iFirst)
- 8Ou J, Li H. Structural health monitoring in Mainland China: Review and future trends. Struct Health Monitor. 2010; 9(3): 219-223.
- 9Farrar CR, Worden K. An introduction to structural health monitoring. Phil Trans Royal Soc A: Math, Phys Eng Sci. 2007; 365(1851): 303-315.
- 10Sohn, H., Farrar, C. R., Hemez, F. M. et al. (2004). “ A review of structural health monitoring literature: 1996–2001”, Los Alamos National Laboratory Report LA-13976-MS.
- 11Gomez F, Park J-W, Spencer BF. Reference-free structural dynamic displacement estimation method. Struct Control Health Monit. 2018; 25(8): 1-14, e2209. https://doi.org/10.1002/stc.2209
- 12Cho, S., Park, J.W., Palanisamy, R.P., and Sim, S.H. (2016). “Reference-free displacement estimation of bridges using Kalman filter–based multimetric data Fusion.” J Sensors. Volume 2016, Article ID 3791856, 1-9. https://doi.org/10.1155/2016/3791856
- 13Park, J.W., Moon, D.S., Yoon, H., Gomez, F., Spencer, B.F., and Kim, J.R. (2017). “Visual–inertial displacement sensing using data fusion of vision-based displacement with acceleration.” Struct Control Health Monit 2018; 25(3): 1-14, e2122. https://doi.org/10.1002/stc.2122
- 14Liu B, Ozdagli AI, Moreu F. Direct reference-free measurement of displacements for railroad bridge management. Struct Control Health Monit. 2018; 25(10): 1-16, e2241. https://doi.org/10.1002/stc.2241
- 15Smyth A, Wu ML. Multi-rate Kalman filtering for the data fusion of displacement and acceleration response measurements in dynamic system monitoring. Mech Syst Signal Proc Vol. 2007; 21: 706-723.
- 16Celebi M. GPS in dynamic monitoring of long-period structures. Soil Dyn Earthq Eng. 2000; 20(5–8): 477-483.
- 17Nakamura SI. GPS measurement of wind-induced suspension bridge girder displacements. J Struct Eng. 2000; 126(12): 1413-1419.
- 18Knecht, A. and Manetti, L. (2001). “Using GPS in structural health monitoring.” Smart Structures and Materials 2001: Sensory Phenomena and Measurement Instrumentation for Smart Structures and Materials., vol. 4328, pp. 122-129, Newport Beach, Calif, USA
10.1117/12.435515 Google Scholar
- 19Nassif HH, Gindy M, Davis J. Comparison of laser Doppler vibrometer with contact sensors for monitoring bridge deflection and vibration. NDT E Int. 2005; 38(3): 213-218.
- 20Lee J, Lee K, Lee S, Lee Y, Sim S. Long-term displacement measurement of bridges using a LiDAR system. Struct Control Health Monit. 2019; 26(10): 1-15, e2428. https://doi.org/10.1002/stc.2428
- 21Wahbeh AM, Caffrey JP, Masri SF. A vision-based approach for the direct measurement of displacements in vibrating systems. Smart Mater Struct. 2003; 12(5): 785-794.
- 22Lee JJ, Shinozuka M. A vision-based system for remote sensing of bridge displacement. NDT E Int. 2007; 39(5): 425-431.
- 23Lee JJ, Fukuda Y, Shinozuka M, Cho S, Yun CB. Development and application of a vision-based displacement measurement system for structural health monitoring of civil structures. Smart Struct Syst. 2007; 3(3): 373-384.
- 24Fukuda Y, Feng MQ, Shinozuka M. Cost-effective vision-based system for monitoring dynamic response of civil engineering structures. Struct Control Health Monit. 2010; 17(8): 918-936.
- 25Choi HS, Cheung JH, Kim SH, Ahn JH. Structural dynamic displacement vision system using digital image processing. NDT E Int. 2011; 44(7): 597-608.
- 26Feng D, Feng M, Ozer E, Fukuda Y. A vision-based sensor for noncontact structural displacement measurement. Sensors. 2015; 15(7): 16557-16575. https://doi.org/10.3390/s150716557
- 27Feng, D. and Feng, M.Q. (2015). “Vision-based multipoint displacement measurement for structural health monitoring.” Struct Control Health Monit 23(5): 876-890. https://doi.org/10.1002/stc.1819
- 28Ho, H., Lee, J., Park, Y., and Lee, J. (2012). “A synchronized multipoint vision-based system for displacement measurement of civil infrastructures.” Scientific World Journal, Volume 2012, Article ID 519146, 1-9 https://doi.org/10.1100/2012/519146
- 29Kim, S., and Kim, N. (2011). “ Multi-point displacement response measurement of civil infrastructures using digital image processing.” The Twelfth East Asia-Pacific Conference on Structural Engineering and Construction, vol. 14, pp. 195-203
- 30Yoon H, Elanwar H, Choi H, Fard M, Spencer B. Target-free approach for vision-based structural system identification using consumer grade cameras. Struct Control Health Monit. 2016; 23(12): 1405-1416. https://doi.org/10.1002/stc.1850
- 31Wu LJ, Casciati F, Casciati S. Dynamic testing of a laboratory model via vision-based sensing. Eng Struct. 2014; 60: 113-125. https://doi.org/10.1016/j.engstruct.2013.12.002
- 32Acikgoz S, DeJong MJ, Soga K. Sensing dynamic displacements in masonry rail bridges using 2D digital image correlation. Struct Control Health Monit. 2018; 25(8): 1-24, e2187. https://doi.org/10.1002/stc.2187
- 33Xu Y, Brownjohn J, Kong D. A non-contact vision-based system for multipoint displacement monitoring in a cable-stayed footbridge. Struct Control Health Monit. 2018; 25(5): 1-23, e2155. https://doi.org/10.1002/stc.2155
- 34Alavi AH, Buttlar WG. An overview of smartphone technology for citizen-centered, real-time and scalable civil infrastructure monitoring. Future Gen Comp Syst. 2019; 93: 651-672. https://doi.org/10.1016/j.future.2018.10.059
- 35Sony S, Laventure S, Sadhu A. A literature review of next-generation smart sensing technology in structural health monitoring. Struct Control Health Monit. 2019; 26(3): 1-22, e2321. https://doi.org/10.1002/stc.2321
- 36Indrawan P, Budiyatno S, Ridho NM, Sari RF. Face recognition for social media with mobile cloud computing. Int J Cloud Computing: Serv Arch (IJCCSA). 2013; 3(1): 23-35.
- 37Grossi M. A sensor-centric survey on the development of smartphone measurement and sensing systems. Measurement. 2018; 135: 572-592. https://doi.org/10.1016/j.measurement
- 38Zhao, X., Ri, K., Han, R., Yu, Y., Li, M., and Ou, J. (2016). Experimental research on quick structural health monitoring technique for bridges using smartphone. Adv Mater Sci Eng. 2016(5): 1-14, Article ID 1871230, https://doi.org/10.1155/2016/1871230
- 39Mei Q, Gul M. A crowdsourcing-based methodology using smartphones for bridge health monitoring. Struct Control Health Monit. 2018; 18(5-6): 1602-1619. https://doi.org/10.1177/1475921718815457
- 40Min J, Nikolas JG, Hongki J. Non-contact and real-time dynamic displacement monitoring using smartphone technologies. Life Cycle Reliabil Safety Eng. 2015; 4(2): 40-51.
- 41 Dropbox, Dropbox Developer, Sync API (2014). <http://www.dropbox.com/developers/sync>
- 42Larson, B. (2014). “ An open source iOS framework for GPU-based image and video processing.” GitHub, Web. 04 Nov 2014<http://github.com/BradLarson/GPUImage>
- 43 AVFoundation, Apple Developer. <https://developer.apple.com/documentation/avfoundation?language=objc>
- 44Ren, Y., Chua, C.S., and Ho, T.K. (2003). “Motion detection with nonstationary background,” Machine Vision Appl, vol. 13, n 5-6, pp. 332-343.
- 45Guo, J., Rajan, D., and Chng, E.S. (2005). “ Motion detection with adaptive background and dynamic thresholds,” 2005 Fifth International Conference on Information, Communications and Signal Processing, pp. 41-45.
- 46Elnagar A, Basu A. Motion detection using background constraints. Pattern Recog. 1995; 28(10): 1537-1554.
- 47Vazquez, J.F., Mazo, M., Lazaro, J.L., Luna, C.A., Urefla, J., Garcia, J.J., and Guillan, E. (2005). “ Adaptive threshold for motion detection in outdoor environment using computer vision,” Proceedings of the IEEE International Symposium on Industrial Electronics, ISIE 2005, vol. 3, pp. 1233-1237.
- 48Harris, C. and Stephens, M. (1988). “ A combined corner and edge detector.” Proceedings of the 4th Alvey Vision Conference, vol. 15, no. 50, pp. 10-5244
- 49Alfthan, J. (2008). “ Robust detection of two-dimensional barcodes in blurry images,” M.S. thesis, KTH Stockholm, Sweden.
- 50Herout, A., Dubsk'a, M., and Havel, J. (2013). “Real-time precise detection of regular grids and matrix codes,” J Real-Time Image Proc, 11(1): 193-200 1–8.
- 51Shrestha A, Dang J, Wang X. Development of a smart-device–based vibration measurement system: Effectiveness examination and application cases to existing structure. Struct Control Health Monit. 2018; 25(3): 1-18, e2120. https://doi.org/10.1002/stc.2120
- 52 APS-113 ELECTRO-SEIS® APS Dynamics, Inc. (2014). <http://www.apsdynamics.com/images/stories/Prospekte/APS_Shaker/APS_113__APS_113-HF/APS_113_HF_Data_Sheet_en.pdf%3e>
- 53Keyence IL Series CMOS Multi-Function Analog Sensor, <http://www.movetec.fi/images/pdf/Keyence-lasermittaus1.pdf>
- 54Rodgers JL, Nicewander WA. Thirteen ways to look at the correlation coefficient. Am Statistic. 1988; 42(1): 59-66.
- 55 Core Image, Apple Developer. <https://developer.apple.com/documentation/coreimage>
- 56 Keyence NR-600 Series. <http://www.keyence.co.jp/keisokuki/>
