Hallucinating Stereoscopy from a Single Image
Abstract
We introduce a novel method for enabling stereoscopic viewing of a scene from a single pre-segmented image. Rather than attempting full 3D reconstruction or accurate depth map recovery, we hallucinate a rough approximation of the scene's 3D model using a number of simple depth and occlusion cues and shape priors. We begin by depth-sorting the segments, each of which is assumed to represent a separate object in the scene, resulting in a collection of depth layers. The shapes and textures of the partially occluded segments are then completed using symmetry and convexity priors. Next, each completed segment is converted to a union of generalized cylinders yielding a rough 3D model for each object. Finally, the object depths are refined using an iterative ground fitting process. The hallucinated 3D model of the scene may then be used to generate a stereoscopic image pair, or to produce images from novel viewpoints within a small neighborhood of the original view. Despite the simplicity of our approach, we show that it compares favorably with state-of-the-art depth ordering methods. A user study was conducted showing that our method produces more convincing stereoscopic images than existing semi-interactive and automatic single image depth recovery methods.
Supporting Information
Filename | Description |
---|---|
cgf12536-sup-0001-S1.mov97.4 MB | Supporting Information |
cgf12536-sup-0002-S1.zip40.8 MB | Supporting Information |
cgf12536-sup-0003-S1.zip245.3 MB | Supporting Information |
cgf12536-sup-0004-S1.zip575.8 MB | Supporting Information |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Asafi S., Goren A., Cohen-Or D.: Weak convex decomposition by lines-of-sight. Computer Graphics Forum 32, 5 (2013), 23–31. 5
- Arbeláez P., Maire M., Fowlkes C., Malik J.: Contour detection and hierarchical image segmentation. IEEE Trans. PAMI 33, 5 (Apr. 2011), 898–916. 10
- Amer M.R., Raich R., Todorovic S.: Monocular Extraction of 2.1D Sketch. In Proc. ICIP. 2010, pp. 3437–3440. 3
- Assa J., Wolf L.: Diorama construction from a single image. Computer Graphics Forum 26, 3 (2007), 599–608. 3
- Barnes C., Shechtman E., Finkelstein A., Goldman D.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009), 24. 5
- Boykov Y., Veksler O., Zabih R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (2001), 1222–1239. 4
- Chen S.E., Williams L.: View interpolation for image synthesis. In SIGGRAPH'93 (1993), ACM, pp. 279–288. 2
10.1145/166117.166153 Google Scholar
- Chen T., Zhu Z., Shamir A., Hu S.-M., Cohen-Or D.: 3-Sweep: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6 (Nov. 2013), 195:1–195:10. 2
- Debevec P.E., Taylor C.J., Malik J.: Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Proc. SIGGRAPH ‘96 (1996), ACM, pp. 11–20. 2
10.1145/237170.237191 Google Scholar
- Gao J., Liao M., Yang R., Gong M.: Video stereolization: Combining motion analysis with user interaction. IEEE Trans. Vis. Comp. Graphics 18, 7 (2012), 1079–1088. 3
- Guttmann M., Wolf L., Cohen-Or D.: Semiautomatic stereo extraction from video footage. In Proc. ICCV (2009), pp. 136–142. 3
- Horry Y., Aniyo K.-I., Arai K.: Tour into the picture: Using a spidery mesh interface to make animation from a single image. In SIGGRAPH'97 (1997), ACM, pp. 225–232. 2
10.1145/258734.258854 Google Scholar
- Hoiem D., Efros A.A., Hebert M.: Automatic photo pop-up. ACM Trans. Graph. 24, 3 (July 2005), 577–584. 3, 10
- Hoiem D., Stein A.N., Efros A.A., Hebert M.: Recovering occlusion boundaries from a single image. In Proc. ICCV (2007), pp. 1–8. 2, 3, 10
- IJsselsteijn W.A., de Ridder H., Hamberg R.: Perceptual factors in stereoscopic displays: the effect of stereoscopic filming parameters on perceived quality and reported eye-strain. In Proceedings of SPIE: Human vision and electronic imaging III (1998), B.E. Rogowitz, T.N. Pappas, (Eds.), vol. 3299, SPIE, pp. 282–291. 1
10.1117/12.320119 Google Scholar
- Igarashi T., Matsuoka S., Tanaka H.: Teddy: A sketching interface for 3d freeform design. In Proc. SIGGRAPH (1999), pp. 409–416. 2
- Jia Z., Gallagher A., Chang Y.-J., Chen T.: A learning-based framework for depth ordering. In Proc. CVPR (2012), pp. 294–301. 3, 5, 8
- Kang S.B.: Depth painting for image-based rendering applications. Tech. rep., Compaq Cambridge Research Lab, 1998. 2
- Karsch K., Liu C., Kang S.: DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling. IEEE Trans. PAMI, to appear (2014). 2, 3, 9
- Khan E.A., Reinhard E., Fleming R.W., Bülthoff H. H.: Image-based material editing. ACM Trans. Graph. 25, 3 (July 2006), 654–663. 3
- Liu B., Gould S., Koller D.: Single image depth estimation from predicted semantic labels. In Proc. CVPR (June 2010), pp. 1253–1260. 3
- Levoy M., Hanrahan P.: Light field rendering. In Proc. SIGGRAPH’ 96 (1996), ACM, pp. 31–42. 2
- Liu X., Mao X., Yang X., Zhang L., Wong T.-T.: Stereoscopizing cel animations. ACM Trans. Graph. 32, 6 (Nov. 2013), 223:1–223:10. 3, 5
- Liu C., Yuen J., Torralba A.: SIFT Flow: Dense correspondence across scenes and its applications. IEEE Trans. PAMI 33, 5 (2011), 978–994. 3
- McMillan L., Bishop G.: Plenoptic modeling: An image-based rendering system. In Proc. SIGGRAPH ‘95 (1995), ACM, pp. 39–46. 2
10.1145/218380.218398 Google Scholar
- Oh B.M., Chen M., Dorsey J., Durand F.: Image-based modeling and photo editing. In Proc. SIGGRAPH ‘01 (2001), ACM, pp. 433–442. 2
10.1145/383259.383310 Google Scholar
- Oliveira M.M.: Image-based modeling and rendering techniques: A survey. RITA - Revista de Informática Teórica e Aplicada IX, 2 (October 2002), 37–66. 2
- Palou G., Salembier P.: Monocular depth ordering using T-junctions and convexity occlusion cues. IEEE Trans. Image Proc. 22, 5 (2013), 1926–1939. 3, 8
- Rother C., Kolmogorov V., Blake A.: “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23, 3 (Aug. 2004), 309–314. 7
- Russell B.C., Torralba A.: Building a database of 3D scenes from user annotations. In Proc. CVPR. June 2009, pp. 2711–2718. 2, 3, 9
- Schneider P.J., Eberly D.H.: Geometric Tools for Computer Graphics. Elsevier Science (USA), 2003. 6
- Silberman N., Hoiem D., Kohli P., Fergus R.: Indoor segmentation and support inference from rgbd images. In Proc. ECCV (2012). 9
- Saxena A., Sun M., Ng A.Y.: Make3D: Learning 3D scene structure from a single still image. IEEE Trans. PAMI 31, 5 (2009), 824–840. 2, 3, 9, 10
- Taubin G.: A signal processing approach to fair surface design. In Proc. SIGGRAPH ‘95 (1995), ACM, pp. 351–358. 6
10.1145/218380.218473 Google Scholar
- Töppe E., Oswald M.R., Cremers D., Rother C.: Image-based 3d modeling via Cheeger sets. In Computer Vision - ACCV 2010, vol. 6492 of Lecture Notes in Computer Science. Springer, 2011, pp. 53–64. 2
- Ward B., Kang S.B., Bennett E.: Depth director: A system for adding depth to movies. IEEE Computer Graphics and Applications 31, 1 (2011), 36–48. 3
- Wang O., Lang M., Frei M., Hornung A., Smolic A., Gross M.: StereoBrush: interactive 2D to 3D conversion using discontinuous warps. In SBIM. 2011. 3
- Yu F., Liu J., Ren Y., Sun J., Gao Y., Liu W.: Depth generation method for 2D to 3D conversion. In Proc. 3DTV-CON (May 2011), pp. 1–4. 3
- Zhang L., Dugas-Phocion G., Samson J.-S., Seitz S.M.: Single view modeling of free-form scenes. In Proc. CVPR (2001), IEEE, pp. 990–997. 2