Terrain Super-resolution through Aerial Imagery and Fully Convolutional Networks
O. Argudo
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorA. Chica
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorC. Andujar
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorO. Argudo
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorA. Chica
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorC. Andujar
VirVIG, Computer Science Department, Universitat Politecnica de Catalunya, Jordi Girona 1–3, Barcelona, Spain
Search for more papers by this authorAbstract
Despite recent advances in surveying techniques, publicly available Digital Elevation Models (DEMs) of terrains are low-resolution except for selected places on Earth. In this paper we present a new method to turn low-resolution DEMs into plausible and faithful high-resolution terrains. Unlike other approaches for terrain synthesis/amplification (fractal noise, hydraulic and thermal erosion, multi-resolution dictionaries), we benefit from high-resolution aerial images to produce highly-detailed DEMs mimicking the features of the real terrain. We explore different architectures for Fully Convolutional Neural Networks to learn upsampling patterns for DEMs from detailed training sets (high-resolution DEMs and orthophotos), yielding up to one order of magnitude more resolution. Our comparative results show that our method outperforms competing data amplification approaches in terms of elevation accuracy and terrain plausibility.
Supporting Information
Filename | Description |
---|---|
cgf13345-sup-0001-S1.jpg5.8 MB | Supplement Material |
cgf13345-sup-0002-S2.jpg5.9 MB | Supplement Material |
cgf13345-sup-0003-S3.jpg5.9 MB | Supplement Material |
cgf13345-sup-0004-S4.jpg4.6 MB | Supplement Material |
cgf13345-sup-0005-S5.jpg5.8 MB | Supplement Material |
cgf13345-sup-0006-S6.jpg4.6 MB | Supplement Material |
cgf13345-sup-0006-S6.jpg4.6 MB | Supplement Material |
cgf13345-sup-0007-S7.txt243 B | Supplement Material |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Argudo O., Andújar C., Chica A., Guérin E., Digne J., Peytavie A., Galin E.: Coherent multi-layer landscape synthesis. The Visual Computer 33, 6 (2017), 1005–1015. 2, 7, 8
- Andújar C., Chica A., Vico M. A., Moya S., Brunet P.: Inexpensive reconstruction and rendering of realistic roadside landscapes. Computer Graphics Forum 33, 6 (2014), 101–117. 2
- Benes B., Forsbach R.: Layered data representation for visual simulation of terrain erosion. In Proceedings of the 17th Spring Conference on Computer Graphics (2001), SCCG ‘01, pp. 80–. 2
- Bors A. G., Hancock E. R., Wilson R. C.: Terrain analysis using radar shape-from-shading. IEEE Trans. Pattern Anal. Mach. Intell. 25, 8 (2003), 974–992. 3
- Beneš B., Těšínsky V., Hornyš J., Bhatia S. K.: Hydraulic erosion. Computer Animation and Virtual Worlds 17,2 (2006), 99–108. 2
- Cordonnier G., Braun J., Cani M.-P., Benes B., Galin E., Peytavie A., Guérin E.: Large scale terrain generation from tectonic uplift and fluvial erosion. In Proc. of Eurographics (2016), pp. 165–175. 2
- Cordonnier G., Galin E., Gain J., Benes B., Guérin E., Peytavie A., Cani M.-P.: Authoring landscapes by combining ecosystem and terrain erosion simulation. ACM Trans. Graph. 36, 4 (2017), 134: 1–134:12. 2
- Chen Z., Sun T., Qin Q., Zhang H.: DEM densification using SFS with single multi-spectral satellite image. In Remote Sensing for Agriculture, Ecosystems, and Hydrology XIII (Nov. 2011), vol. 8174 of Proceedings of the SPIE. 3
- Dachsbacher C., Bolch T., Stamminger M.: Procedural Reproduction of Terrain Textures with Geographic Data. In Vision, Modeling and Visualization (VMV 2006) (2006), pp. 105–112. 2
- Drori I., Cohen-Or D., Yeshurun H.: Fragment-based image completion. ACM Trans. Graph. 22, 3 (2003), 303–312. 2
- Deussen O., Hanrahan P., Lintermann B., Měch R., Pharr M., Prusinkiewicz P.: Realistic modeling and rendering of plant ecosystems. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (1998), SIGGRAPH ‘98, pp. 275–286. 2
- Dong C., Loy C. C., He K., Tang X.: Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 2 (2016), 295–307. 3
- Dachsbacher C., Meyer M., Stamminger M.: Height-field synthesis by non-parametric sampling. Vision, Modeling and Visualization 2005 (2005), 297–302. 2
- Dachsbacher C., Stamminger M.: Rendering procedural terrain by geometry image warping. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques (Aire-la-Ville, Switzerland, 2004), EGSR'04, Eurographics Association, pp. 103–110. 2
- Efros A. A., Freeman W. T.: Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), ACM, pp. 341–346. 2
- Ebert D. S., Musgrave F. K., Peachey D., Perlin K., Worley S.: Preface. In Texturing and Modeling (3), The Morgan Kaufmann Series in Computer Graphics. Morgan Kaufmann, 2003, pp. xx–xxiii. 2
- Emilien A., Vimont U., Cani M.-P., Poulin P., Benes B.: Worldbrush: Interactive example-based synthesis of procedural virtual worlds. ACM Trans. Graph. 34, 4 (2015), 106: 1–106:11. 2
- Freeman W. T., Jones T. R., Pasztor E. C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22, 2 (2002), 56–65. 2
- Glasner D., Bagon S., Irani M.: Super-resolution from a single image. In 2009 IEEE 12th International Conference on Computer Vision (2009), pp. 349–356. 2
- Guérin E., Digne J., Galin E., Peytavie A., Wolf C., Benes B., Martinez B.: Interactive Example-based Terrain Authoring with Conditional Generative Adversarial Networks. ACM Trans. Graph. 36, 6 (2017), 228: 1–228:13. 3
- Guérin E., Digne J., Galin E., Peytavie A.: Sparse representation of terrains for procedural modeling. Computer Graphics Forum (proc. of Eurographics 2016) 35, 2 (2016), 177–187. 2, 7, 8
- Gatys L., Ecker A. S., Bethge M.: Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems (2015), pp. 262–270. 2
- Gain J., Merry B., Marais P.: Parallel, realistic and controllable terrain synthesis. Comput. Graph. Forum 34, 2 (2015), 105–116. 2
- Gain J., Marais P., Strasser W.: Terrain sketching. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (2009), I3D ‘09, pp. 31–38. 2
- Hays J., Efros A. A.: Scene completion using millions of photographs. ACM Trans. Graph. 26, 3 (July 2007). 2
- Institut Cartogràfic i Geologic de Catalunya. http://www.icc.cat/vissir3. Online; accessed 01 October 2017. 3
- Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., Darrell T.: Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093 (2014). 4, 7
- Kingma D., Ba J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). 4
- Krištof P., Benes B., Křivánek J., Štáva O.: Hydraulic erosion using smoothed particle hydrodynamics. In Computer Graphics Forum (2009), vol. 28(2), pp. 219–228. 2
- Kim K. I., Kwon Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2010), 1127–1133. 2
- Kim J., Lee J. K., Lee K. M.: Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27–30, 2016 (2016), pp. 1646–1654. 3
- Kruschke J.: Doing Bayesian Data Analysis, 2nd ed. Academic Press, 2014. 8
- Krizhevsky A., Sutskever I., Hinton G. E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84–90. 2
- LeCun Y., Boser B., Denker J. S., Henderson D., Howard R. E., Hubbard W., Jackel L. D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 4 (1989), 541–551. 2
- Lane B., Prusinkiewicz P.: Generating spatial distributions for multilevel models of plant communities. In Proceedings of the Graphics Interface 2002 Conference, May 27–29, 2002, Calgary, Alberta, Canada (2002), pp. 69–80. 2
- Long J., Shelhamer E., Darrell T.: Fully convolutional networks for semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2015). 3, 4
- Lim B., Son S., Kim H., Nah S., Lee K. M.: Enhanced deep residual networks for single image super-resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017), vol.1, p. 3. 3
- Ledig C., Theis L., Huszár F., Caballero J., Cunningham A., Acosta A., Aitken A., Tejani A., Totz J., Wang Z., ET AL.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv:1609.04802 (2016). 3
- Mac Aodha O., Campbell N. D. F., Nair A., Brostow G. J.: Patch Based Synthesis for Single Depth Image Super-Resolution. Springer Berlin Heidelberg, 2012, pp. 71–84. 2
- Musgrave F. K., Kolb C. E., Mace R. S.: The synthesis and rendering of eroded fractal terrains. SIGGRAPH ‘89 23, 3 (1989), 41–50. 2
10.1145/74334.74337 Google Scholar
- Měch R., Prusinkiewicz P.: Visual models of plants interacting with their environment. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (1996), SIGGRAPH ‘96, pp. 397–410. 2
- Nair V., Hinton G. E.: Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (2010), ICML'10, pp. 807–814. 2
- Natali M., Lidal E. M., Parulek J., Viola I., Patel D.: Modeling terrains and subsurface geology. In EuroGraphics 2013 State of the Art Reports (STARs) (2013), pp. 155–173. 2
- OpenDEM. http://opendem.info/opendemsearcher.html. Online; accessed 01 October 2017. 1
- Perlin K.: An image synthesizer. SIGGRAPH ‘85 19, 3 (1985), 287–296. 2
10.1145/325165.325247 Google Scholar
- Rajabi M. A., Blais J. A. R.: Densification of digital terrain elevations using shape from shading with single satellite imagery. In Proceedings of the International Conference on Computational Science-Part II (2001), ICCS ‘01, pp. 3–12. 3
- Ren H., El-Khamy M., Lee J.: Image Super Resolution Based on Fusing Multiple Convolution Neural Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), pp. 1050–1057. 3
- Saupe D.: Point Evaluation of Multi-Variable Random Fractals. Springer Berlin Heidelberg, 1989, pp. 114–126. 2
- Südtiroler Bürgernetz GeoKatalog. http://geokatalog.buergernetz.bz.it/geokatalog. Online; accessed 01 October 2017. 3
- Schneider J., Boldte T., Westermann R.: Real-time editing, synthesis, and rendering of infinite landscapes on GPUs. In Vision, Modeling and Visualization 2006 (2006).
- Smelik R. M., Tutenel T., Bidarra R., Benes B.: A survey on procedural modelling for virtual worlds. Computer Graphics Forum 33, 6 (2014), 31–50. 2
- Timofte R., Agustsson E., Gool L. V., Yang M. H., Zhang L.: NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017), pp. 1110–1121. 2
- Timofte R., De V., Gool L. V.: Anchored neighborhood regression for fast example-based super-resolution. In 2013 IEEE International Conference on Computer Vision (2013), pp. 1920–1927. 2
- USGS 3D Elevation Program (3DEP). https://www.sciencebase.gov/catalog/item/4f70a58ce4b058caae3f8ddb. Online; accessed 01 October 2017. 1
- Wei L.-Y., Levoy M.: Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (2000), pp. 479–488. 2
- Wei L.-Y., Levoy M.: Order-independent texture synthesis. arXiv preprint arXiv:1406.7338 (2014). 2
- Yang J., Lin Z., Cohen S.: Fast image super-resolution based on in-place example regression. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (2013), CVPR ‘13, pp. 1059–1066. 2
- Yang C.-Y., Ma C., Yang M.-H.: Single-Image Super-Resolution: A Benchmark. Springer International Publishing, 2014, pp. 372–386. 2
- Yang Q., Yang R., Davis J., Nister D.: Spatial-Depth Super Resolution for Range Images. In 2007 IEEE Conference on Computer Vision and Pattern Recognition (2007), pp. 1–8. 2, 7, 8
- Zhou H., Sun J., Turk G., Rehg J. M.: Terrain synthesis from digital elevation models. IEEE Transactions on Visualization and Computer Graphics 13, 4 (2007), 834–848. 2
- Zhang R., Tsai P.-S., Cryer J. E., Shah M.: Shape from shading: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 21, 8 (1999), 690–706. 3