Urban functional zone mapping by coupling domain knowledge graphs and high-resolution satellite images
Yixiang Chen
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorXu Dang
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorDaoyou Zhu
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorYi Huang
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorCorresponding Author
Kun Qin
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, China
Correspondence
Kun Qin, School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430070, China.
Email: [email protected]
Search for more papers by this authorYixiang Chen
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorXu Dang
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorDaoyou Zhu
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorYi Huang
School of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China
Search for more papers by this authorCorresponding Author
Kun Qin
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, China
Correspondence
Kun Qin, School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430070, China.
Email: [email protected]
Search for more papers by this authorAbstract
Timely and accurate mapping of urban functional zones (UFZs) is crucial to urban planning and management. Although existing methods for identifying urban functions have made remarkable progress, they still suffer from limitations, such as high sample dependency, insufficient semantic relationship representation, and poor interpretability due to being data-driven models. To bridge this gap between these methods and the way humans identify functional areas, a new framework that couples domain knowledge and remote sensing images was proposed for the mapping of UFZs. First, to model the concepts, attributes, and spatial and semantic relationships of urban functional objects, a UFZ knowledge graph (UFZ-KG) was constructed to assist in the mapping of UFZs. Then, the contrastive language-image pretraining model was adopted to encode jointly the semantic features of UFZ-KG and the visual features of UFZ images. In this model, a nonlinear embedding module was designed to achieve semantic alignment of these two different modal features in shared space. The effectiveness of the proposed method was verified in three test areas in Shenzhen, China. Results demonstrate that the proposed method of coupling UFZ-KG with satellite images significantly enhances the UFZ classification accuracy compared to the method that solely relies on image features. Furthermore, it exhibits good generalization performance.
CONFLICT OF INTEREST STATEMENT
The authors declare that there are no competing interests.
Open Research
DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request.
REFERENCES
- Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450. http://arxiv.org/abs/1607.06450
- Cao, R., Tu, W., Yang, C., Li, Q., Liu, J., Zhu, J., Zhang, Q., Li, Q., & Qiu, G. (2020). Deep learning-based remote and social sensing data fusion for urban region function recognition. ISPRS Journal of Photogrammetry and Remote Sensing, 163, 82–97. https://doi.org/10.1016/j.isprsjprs.2020.02.014
- Chang, S., Wang, Z., Mao, D., Liu, F., Lai, L., & Yu, H. (2021). Identifying urban functional areas in China's Changchun City from Sentinel-2 images and social sensing data. Remote Sensing, 13(22), 4512. https://doi.org/10.3390/rs13224512
- Chen, B., Tu, Y., Song, Y., Theobald, D. M., Zhang, T., Ren, Z., Li, X., Yang, J., Wang, J., Wang, X., Gong, P., Bai, Y., & Xu, B. (2021). Mapping essential urban land use categories with open big data: Results for five metropolitan areas in the United States of America. ISPRS Journal of Photogrammetry and Remote Sensing, 178, 203–218. https://doi.org/10.1016/j.isprsjprs.2021.06.010
- Chen, C., Yan, J., Wang, L., Liang, D., & Zhang, W. (2020). Classification of urban functional areas from remote sensing images and time-series user behavior data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 1207–1221. https://doi.org/10.1109/JSTARS.2020.3044250
- Chen, T., Lin, L., Chen, R., Wu, Y., & Luo, X. (2018). Knowledge-embedded representation learning for fine-grained image recognition. arXiv preprint arXiv:1807.00505. http://arxiv.org/abs/1807.00505
- Chen, Y., Liu, X., Li, X., Liu, X., Yao, Y., Hu, G., Xu, X., & Pei, F. (2017). Delineating urban functional areas with building-level social media data: A dynamic time warping (DTW) distance based k-medoids method. Landscape and Urban Planning, 160, 48–60. https://doi.org/10.1016/j.landurbplan.2016.12.001
- Chen, Y., Shi, W., Dang, X., Wu, C., & Li, S. (2022). Classification of urban functional zones by integrating spatial features of VHR satellite images and semantic features of POI data. Paper presented at the 29th International Conference on Geoinformatics. https://doi.org/10.1109/Geoinformatics57846.2022.9963812
10.1109/Geoinformatics57846.2022.9963812 Google Scholar
- Chen, Y., Yao, S., Hu, Z., Huang, B., Miao, L., & Zhang, J. (2023). Built-up area extraction combing densely connected dual-attention network and multiscale context. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 5128–5143. https://doi.org/10.1109/JSTARS.2023.3281363
- Chen, Z., Tang, L., Guo, X., & Zheng, G. (2024). A self-supervised detection method for mixed urban functions based on trajectory temporal image. Computers, Environment and Urban Systems, 110, 102113. https://doi.org/10.1016/j.compenvurbsys.2024.102113
- Cui, P., Liu, S., & Zhu, W. (2017). General knowledge embedded image representation learning. IEEE Transactions on Multimedia, 20(1), 198–207. https://doi.org/10.1109/TMM.2017.2724843
- Dong, X., Xu, Y., Huang, L., Liu, Z., Xu, Y., Zhang, K., Hu, Z., & Wu, G. (2020). Exploring impact of spatial unit on urban land use mapping with multisource data. Remote Sensing, 12(21), 3597. https://doi.org/10.3390/rs12213597
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., & Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. https://arxiv.org/abs/2010.11929
- Du, J., Wang, S., Ye, X., Sinton, D. S., & Kemp, K. (2022). GIS-KG: Building a large-scale hierarchical knowledge graph for geographic information science. International Journal of Geographical Information Science, 36(5), 873–897. https://doi.org/10.1080/13658816.2021.2005795
- Du, S., Du, S., Liu, B., & Zhang, X. (2019). Context-enabled extraction of large-scale urban functional zones from very-high-resolution images: A multiscale segmentation approach. Remote Sensing, 11(16), 1902. https://doi.org/10.3390/rs11161902
- Du, S., Du, S., Liu, B., & Zhang, X. (2021). Mapping large-scale and fine-grained urban functional zones from VHR images using a multi-scale semantic segmentation network and object based approach. Remote Sensing of Environment, 261, 112480. https://doi.org/10.1016/j.rse.2021.112480
- Du, S., Du, S., Liu, B., Zhang, X., & Zheng, Z. (2020). Large-scale urban functional zone mapping by integrating remote sensing images and open social data. GIScience & Remote Sensing, 57(3), 411–430. https://doi.org/10.1080/15481603.2020.1724707
- Du, Z., Zhang, X., Li, W., Zhang, F., & Liu, R. (2020). A multi-modal transportation data-driven approach to identify urban functional zones: An exploration based on Hangzhou City, China. Transactions in GIS, 24(1), 123–141. https://doi.org/10.1111/tgis.12591
- Fan, R., Feng, R., Han, W., & Wang, L. (2021). Urban functional zone mapping with a bibranch neural network via fusing remote sensing and social sensing data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14, 11737–11749. https://doi.org/10.1109/JSTARS.2021.3127246
- Grippa, T., Georganos, S., Zarougui, S., Bognounou, P., Diboulo, E., Forget, Y., Lennert, M., Vanhuysse, S., Mboga, N., & Wolff, E. (2018). Mapping urban land use at street block level using openstreetmap, remote sensing data, and spatial metrics. ISPRS International Journal of Geo-Information, 7(7), 246. https://doi.org/10.3390/ijgi7070246
- Hao, X., Ji, Z., Li, X., Yin, L., Liu, L., Sun, M., Liu, Q., & Yang, R. (2021). Construction and application of a knowledge graph. Remote Sensing, 13(13), 2511. https://doi.org/10.3390/rs13132511
- He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Paper presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.90
10.1109/CVPR.2016.90 Google Scholar
- Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (GELUS). arXiv preprint arXiv:1606.08415.https://doi.org/10.48550/arXiv.1606.08415
10.48550/arXiv.1606.08415 Google Scholar
- Hoffmann, E. J., Wang, Y., Werner, M., Kang, J., & Zhu, X. X. (2019). Model fusion for building type classification from aerial and street view images. Remote Sensing, 11(11), 1259. https://doi.org/10.3390/rs11111259
- Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. Paper presented at the International Conference on Machine Learning. https://doi.org/10.48550/arXiv.1902.00751
10.48550/arXiv.1902.00751 Google Scholar
- Hu, Q., Bai, G., Wang, S., & Ai, M. (2019). Extraction and monitoring approach of dynamic urban commercial area using check-in data from Weibo. Sustainable Cities and Society, 45, 508–521. https://doi.org/10.1016/j.scs.2018.11.039
- Hu, S. (2023). Research on the modeling and application of urban functional regions (PhD Dissertation). China University of Geosciences. https://doi.org/10.27492/d.cnki.gzdzu.2023.000159
- Hu, S., Gao, S., Wu, L., Xu, Y., Zhang, Z., Cui, H., & Gong, X. (2021). Urban function classification at road segment level using taxi trajectory data: A graph convolutional neural network approach. Computers, Environment and Urban Systems, 87, 101619. https://doi.org/10.1016/j.compenvurbsys.2021.101619
- Hu, S., He, Z., Wu, L., Yin, L., Xu, Y., & Cui, H. (2020). A framework for extracting urban functional regions based on multiprototype word embeddings using points-of-interest data. Computers, Environment and Urban Systems, 80, 101442. https://doi.org/10.1016/j.compenvurbsys.2019.101442
- Hu, T., Yang, J., Li, X., & Gong, P. (2016). Mapping urban land use by using landsat images and open social data. Remote Sensing, 8(2), 151. https://doi.org/10.3390/rs8020151
- Huang, W., Cui, L., Chen, M., Zhang, D., & Yao, Y. (2022). Estimating urban functional distributions with semantics preserved POI embedding. International Journal of Geographical Information Science, 36, 1905–1930. https://doi.org/10.1080/13658816.2022.2040510
- Huang, W., Zhang, D., Mai, G., Guo, X., & Cui, L. (2023). Learning urban region representations with POIs and hierarchical graph infomax. ISPRS Journal of Photogrammetry and Remote Sensing, 196, 134–145. https://doi.org/10.1016/j.isprsjprs.2022.11.021
- Huang, Y., Yuan, M., Sheng, Y., Min, X., & Cao, Y. (2019). Using geographic ontologies and geo-characterization to represent geographic scenarios. ISPRS International Journal of Geo-Information, 8(12), 566. https://doi.org/10.3390/ijgi8120566
- Ji, S., Pan, S., Cambria, E., Marttinen, P., & Philip, S. Y. (2021). A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2), 494–514. https://doi.org/10.1109/TNNLS.2021.3070843
- Jing, L., & Tian, Y. (2020). Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11), 4037–4058. https://doi.org/10.1109/TPAMI.2020.2992393
- Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980
10.48550/arXiv.1412.6980 Google Scholar
- Li, Y., Kong, D., Zhang, Y., Tan, Y., & Chen, L. (2021). Robust deep alignment network with remote sensing knowledge graph for zero-shot and generalized zero-shot remote sensing image scene classification. ISPRS Journal of Photogrammetry and Remote Sensing, 179, 145–158. https://doi.org/10.1016/j.isprsjprs.2021.08.001
- Li, Y., Liu, C., & Li, Y. (2022). Identification of urban functional areas and their mixing degree using point of interest analyses. Land, 11(7), 996. https://doi.org/10.3390/land11070996
- Li, Y., Ouyang, S., & Zhang, Y. (2022). Combining deep learning and ontology reasoning for remote sensing image semantic segmentation. Knowledge-Based Systems, 243, 108469. https://doi.org/10.1016/j.knosys.2022.108469
- Liu, X., He, J., Yao, Y., Zhang, J., Liang, H., Wang, H., & Hong, Y. (2017). Classifying urban land use by integrating remote sensing and social media data. International Journal of Geographical Information Science, 31(8), 1675–1696. https://doi.org/10.1080/13658816.2017.1324976
- Liu, Y., Liu, X., Gao, S., Gong, L., Kang, C., Zhi, Y., Chi, G., & Shi, L. (2015). Social sensing: A new approach to understanding our socioeconomic environments. Annals of the Association of American Geographers, 105(3), 512–530. https://doi.org/10.1080/00045608.2015.1018773
- Lu, W., Tao, C., Li, H., Qi, J., & Li, Y. (2022). A unified deep learning framework for urban functional zone extraction based on multi-source heterogeneous data. Remote Sensing of Environment, 270, 112830. https://doi.org/10.1016/j.rse.2021.112830
- Luo, S., Liu, Y., Du, M., Gao, S., Wang, P., & Liu, X. (2021). The influence of spatial grid division on the layout analysis of urban functional areas. ISPRS International Journal of Geo-Information, 10(3), 189. https://doi.org/10.3390/ijgi10030189
- Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024). Unifying large language models and knowledge graphs: A roadmap. ArXiv, abs/2306.08302. https://doi.org/10.1109/TKDE.2024.3352100
10.1109/TKDE.2024.3352100 Google Scholar
- Qian, Z., Liu, X., Tao, F., & Zhou, T. (2020). Identification of urban functional areas by coupling satellite images and taxi GPS trajectories. Remote Sensing, 12(15), 2449. https://doi.org/10.3390/rs12152449
- Qin, Q., Xu, S., Du, M., & Li, S. (2022). Identifying urban functional zones by capturing multi-spatial distribution patterns of points of interest. International Journal of Digital Earth, 15(1), 2468–2494. https://doi.org/10.1080/17538947.2022.2160841
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., & Clark, J. (2021). Learning transferable visual models from natural language supervision. Paper presented at the International Conference on Machine Learning. https://doi.org/10.48550/arXiv.2103.00020
10.48550/arXiv.2103.00020 Google Scholar
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.
- Tu, W., Cao, J., Yue, Y., Shaw, S.-L., Zhou, M., Wang, Z., Chang, X., Xu, Y., & Li, Q. (2017). Coupling mobile phone and social media data: A new approach to understanding urban functions and diurnal patterns. International Journal of Geographical Information Science, 31(12), 2331–2358. https://doi.org/10.1080/13658816.2017.1356464
- Tu, W., Hu, Z., Li, L., Cao, J., Jiang, J., Li, Q., & Li, Q. (2018). Portraying urban functional zones by coupling remote sensing imagery and human sensing data. Remote Sensing, 10(1), 141. https://doi.org/10.3390/rs10010141
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. http://arxiv.org/abs/1706.03762
- Wang, Q., Mao, Z., Wang, B., & Guo, L. (2017). Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12), 2724–2743. https://doi.org/10.1109/TKDE.2017.2754499
- Wang, S., Zhang, X., Ye, P., Du, M., Lu, Y., & Xue, H. (2019). Geographic knowledge graph (GeoKG): A formalized geographic knowledge representation. ISPRS International Journal of Geo-Information, 8(4), 184. https://doi.org/10.3390/ijgi8040184
- Wu, H., Luo, W., Lin, A., Hao, F., Olteanu-Raimond, A.-M., Liu, L., & Li, Y. (2023). SALT: A multifeature ensemble learning framework for mapping urban functional zones from VGI data and VHR images. Computers, Environment and Urban Systems, 100, 101921. https://doi.org/10.1016/j.compenvurbsys.2022.101921
- Wu, L., Cheng, X., Kang, C., Zhu, D., Huang, Z., & Liu, Y. (2020). A framework for mixed-use decomposition based on temporal activity signatures extracted from big geo-data. International Journal of Digital Earth, 13(6), 708–726. https://doi.org/10.1080/17538947.2018.1556353
- Xia, G. S., Hu, J., Hu, F., Shi, B., Bai, X., Zhong, Y., Zhang, L., & Lu, X. (2017). AID: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7), 3965–3981. https://doi.org/10.1109/TGRS.2017.2685945
- Xing, H., & Meng, Y. (2018). Integrating landscape metrics and socioeconomic features for urban functional region classification. Computers, Environment and Urban Systems, 72, 134–145. https://doi.org/10.1016/j.compenvurbsys.2018.06.005
- Xu, N., Luo, J., Wu, T., Dong, W., Liu, W., & Zhou, N. (2021). Identification and portrait of urban functional zones based on multisource heterogeneous data and ensemble learning. Remote Sensing, 13(3), 373. https://doi.org/10.3390/rs13030373
- Xue, F., Hong, R., He, X., Wang, J., Qian, S., & Xu, C. (2019). Knowledge-based topic model for multi-modal social event analysis. IEEE Transactions on Multimedia, 22(8), 2098–2110. https://doi.org/10.1109/TMM.2019.2951194
- Yan, Y., Wang, Y., Du, Z., Zhang, F., Liu, R., & Ye, X. (2020). Where urban youth work and live: A data-driven approach to identify urban functional areas at a fine scale. ISPRS International Journal of Geo-Information, 9(1), 42. https://doi.org/10.3390/ijgi9010042
- Yang, M., Kong, B., Dang, R., & Yan, X. (2022). Classifying urban functional regions by integrating buildings and points-of-interest using a stacking ensemble method. International Journal of Applied Earth Observation and Geoinformation, 108, 102753. https://doi.org/10.1016/j.jag.2022.102753
- Yao, Y., Li, X., Liu, X., Liu, P., Liang, Z., Zhang, J., & Mai, K. (2017). Sensing spatial distribution of urban land use by integrating points-of-interest and Google Word2Vec model. International Journal of Geographical Information Science, 31(4), 825–848. https://doi.org/10.1080/13658816.2016.1244608
- Yuan, N. J., Zheng, Y., Xie, X., Wang, Y., Zheng, K., & Xiong, H. (2014). Discovering urban functional zones using latent activity trajectories. IEEE Transactions on Knowledge and Data Engineering, 27(3), 712–725. https://doi.org/10.1109/TKDE.2014.2345405
- Zhai, W., Bai, X., Shi, Y., Han, Y., Peng, Z.-R., & Gu, C. (2019). Beyond Word2vec: An approach for urban functional region extraction and identification by combining Place2vec and POIs. Computers, Environment and Urban Systems, 74, 1–12. https://doi.org/10.1016/j.compenvurbsys.2018.11.008
- Zhang, B., Zhong, C., Gao, Q., Shabrina, Z., & Tu, W. (2022). Delineating urban functional zones using mobile phone data: A case study of cross-boundary integration in Shenzhen-Dongguan-Huizhou area. Computers, Environment and Urban Systems, 98, 101872. https://doi.org/10.1016/j.compenvurbsys.2022.101872
- Zhang, K., Ming, D., Du, S., Xu, L., Ling, X., Zeng, B., & Lv, X. (2022). Distance weight-graph attention model-based high-resolution remote sensing urban functional zone identification. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–18. https://doi.org/10.1109/TGRS.2021.3115972
- Zhang, X., Du, S., & Wang, Q. (2017). Hierarchical semantic cognition for urban functional zones with VHR satellite images and POI data. ISPRS Journal of Photogrammetry and Remote Sensing, 132, 170–184. https://doi.org/10.1016/j.isprsjprs.2017.09.007
- Zhang, X., Du, S., & Zheng, Z. (2020). Heuristic sample learning for complex urban scenes: Application to urban functional-zone mapping with VHR images and POI data. ISPRS Journal of Photogrammetry and Remote Sensing, 161, 1–12. https://doi.org/10.1016/j.isprsjprs.2020.01.005
- Zhang, X., Huang, Y., Zhang, C., & Ye, P. (2022). Geoscience Knowledge Graph (GeoKG): Development, construction and challenges. Transactions in GIS, 26(6), 2480–2494. https://doi.org/10.1111/tgis.12985
- Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2022). Contrastive learning of medical visual representations from paired images and text. Paper presented at the Machine Learning for Healthcare Conference. https://doi.org/10.48550/arXiv.2010.00747
10.48550/arXiv.2010.00747 Google Scholar
- Zhang, Y., Liu, P., & Biljecki, F. (2023). Knowledge and topology: A two layer spatially dependent graph neural networks to identify urban functions with time-series street view image. ISPRS Journal of Photogrammetry and Remote Sensing, 198, 153–168. https://doi.org/10.1016/j.isprsjprs.2023.03.008
- Zheng, K., Xie, M. H., Zhang, J. B., Xie, J., & Xia, S. H. (2022). A knowledge representation model based on the geographic spatiotemporal process. International Journal of Geographical Information Science, 36, 674–691. https://doi.org/10.1080/13658816.2021.1962527
- Zhou, W., Ming, D., Lv, X., Zhou, K., Bao, H., & Hong, Z. (2020). SO–CNN based urban functional zone fine division with VHR remote sensing image. Remote Sensing of Environment, 236, 111458. https://doi.org/10.1016/j.rse.2019.111458
- Zhu, J., Dang, P., Cao, Y., Lai, J., Guo, Y., Wang, P., & Li, W. (2024). A flood knowledge-constrained large language model interactable with GIS: Enhancing public risk perception of floods. International Journal of Geographical Information Science, 38(4), 603–625. https://doi.org/10.1080/13658816.2024.2306167
Citing Literature
September 2024
Pages 1510-1535