Augmented and/or Mixed Reality

The aim of augmented reality (AR) is to increase user perception by adding information, such as sound, textual notations or virtual objects to a perceived scene. By its very nature, AR is interactive and three-dimensional (3D), meaning that at any time, added elements must be correctly placed in relation to the real world as seen by the user. This chapter discusses the state of the art of two key points in AR: (i) real-time pose computation including pose computation requirements and image/model mapping, and (ii) the acquisition of a scene model including generation of 3D models such as offline modeling and online modeling, in the common framework of monocular vision.

Bibliography

[AGA 09] AGARWAL S., SNAVELY N., SIMON I., et al., “Building rome in a day”, Proceedings of the 9th International Conference on Computer Vision, Kyoto, Japon, October 2009.
Google Scholar
[ANS 03] ANSAR A., DANIILIDIS K., “Linear pose estimation from points or lines”, IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 25, pp. 578–589, 2003.
10.1109/TPAMI.2003.1195992
Web of Science® Google Scholar
[ARO 06] ARON M., SIMON G., BERGER M.-O., “Use of inertial sensors to support video tracking”, Computer Animation and Virtual Worlds, vol. 18, pp. 57–68, 2006.
10.1002/cav.161
Web of Science® Google Scholar
[AZU 01] AZUMA R.T., BAILLOT Y., BEHRINGER R., et al., “Recent advances in augmented reality”, Computer Grahics Applications, vol. 21, pp. 34–47, December 2001.
10.1109/38.963459
Web of Science® Google Scholar
[BUN 08] BUNNUN P., MAYOL-CUEVAS W.W., “OutlinAR: an assisted interactive model building system with reduced computational effort”, IEEE/ACM International Symposium on Mixed and Augmented Reality, IEEE Computer Society, Los Alamitos, CA, pp. 61–64, 2008.
Web of Science® Google Scholar
[CAL 12] CALONDER M., LEPETIT V., OZUYSAL M., et al., “BRIEF: computing a local binary descriptor very fast”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, pp. 1281–1298, 2012.
10.1109/TPAMI.2011.222
PubMed Web of Science® Google Scholar
[CHA 02] CHANG P., HEBERT M., “Robust tracking and structure from motion through sampling based uncertainty representation”, Proceedings of ICRA '02, Washington, May 2002.
Google Scholar
[DAV 02] DAVISON A.J., MURRAY D.W., “Simultaneous localization and map-building using active vision”, IEEE Transactions on PAMI, vol. 24, pp. 865–880, 2002.
10.1109/TPAMI.2002.1017615
Web of Science® Google Scholar
[DEM 95] DEMENTHON D., DAVIS L., “Model based object pose in 25 lines of code”, International Journal of Computer Vision, vol. 15, pp. 123–141, 1995.
10.1007/BF01450852
Web of Science® Google Scholar
[DRU 99] DRUMMOND T., CIPOLLA R., “Real-time tracking of complex structures with on-line camera calibration”, Proceedings of the British Machine Vision Conference, BMVC 99, Nottingham, 1999.
Google Scholar
[EAD 06] EADE E., DRUMMOND T., “Scalable monocular SLAM”, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, Washington, DC, pp. 469–476, 2006.
Google Scholar
[FIS 81] FISCHLER M.A., BOLLES R.C., “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Commununications of the ACM, vol. 24, no. 6, pp. 381–395, June 1981.
10.1145/358669.358692
Web of Science® Google Scholar
[FRE 08] FREEDMAN B., SHPUNT A., MACHLINE M., et al., “Depth mapping using projected patterns”, Patent Application, 10 2008. WO 2008/120217 A2, 2008.
Google Scholar
[FRU 01] FRUH C., ZAKHOR A., “3D model generation for cities using aerial photographs and ground level laser scans”, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, CVPR 2001, vol. 2, pp. II-31–II-38, 2001.
Google Scholar
[HAR 00] HARTLEY R.I., ZISSERMAN A., Multiple View Geometry in Computer Vision, Cambridge University Press, 2000.
Web of Science® Google Scholar
[HES 11] HESCH J.A., ROUMELIOTIS S.I., “A direct least-squares (DLS) method for PnP”, International Conference on Computer Vision, Barcelona, pp. 383–390, 2011.
Google Scholar
[HSI 10] HSIAO E., COLLET ROMEA A., HEBERT M., “Making specific features less discriminative to improve point-based 3D object recognition”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, June 2010.
Google Scholar
[HUB 81] HUBER P.J., Robust Statistics, Wiley, New York, 1981.
10.1002/0471725250
PubMed Google Scholar
[ISA 96] ISARD M., BLAKE A., “Contour tracking by stochastic propagation of conditional density”, Proceedings of the 4th European Conference on Computer Vision, Cambridge, UK, vol. 1064, pp. 343–356, 1996.
Google Scholar
[IZA 11] IZADI S., KIM D., HILLIGES O., et al., “Kinect-fusion: real-time 3D reconstruction and interaction using a moving depth camera”, ACM Symposium on User Interface Software and Technology, Santa Barbara, USA, pp. 559–568, 2011.
Google Scholar
[KAT 99] KATO H., BILLINGHURST M., “Marker tracking and HMD calibration for a video-based augmented reality conferencing system”, Proceedings of the 2nd International Workshop on Augmented Reality, San Francisco, CA, 1999.
Google Scholar
[KE 04] KE Y., SUKTHANKAR R., “PCA-SIFT: A more distinctive representation for local image descriptors”, 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, vol. 2, pp. 506–513, 2004.
Web of Science® Google Scholar
[KLE 07] KLEIN G., MURRAY D., “Parallel tracking and mapping for small AR workspaces”, Proceeding of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'07), Nara, Japan, November 2007.
Google Scholar
[KOL 92] KOLLER D., DANIILIDIS K., NAGEL H.H., “Model-based object tracking in traffic scenes”, Proceedings of the 2nd European Conference on Computer Vision, Santa Margherita Ligure, Italy, vol. 588, pp. 437–452, 7 October, 1992.
Google Scholar
[KOS 05] KOSECKA J., ZHANG W., “Extraction, matching and pose recovery based on dominant rectangular structures”, Computer Vision and Image Understanding, vol. 100, no. 3, pp. 274–293, 2005.
10.1016/j.cviu.2005.04.005
Web of Science® Google Scholar
[LEP 04] LEPETIT V., PILET J., FUA P., “Point matching as a classification problem for fast and robust object pose estimation”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, pp. 244–250, 2004.
Google Scholar
[LEP 05] LEPETIT V., FUA P., “Monocular model-based 3D tracking of rigid objects: a survey”, Foundations and Trends in Computer Graphics and Vision, vol. 1, no. 1, pp. 1–89, 2005.
10.1561/0600000001
Google Scholar
[LEP 09] LEPETIT V., MORENO-NOGUER F., FUA P., “EPnP: an accurate O(n) solution to the PnP problem”, International Journal of Computer Vision, vol. 81, no. 2, pp. 155–166, February 2009.
10.1007/s11263-008-0152-6
Web of Science® Google Scholar
[LOW 87] LOWE D.G., “Three-dimensional object recognition from single two-dimensional images”, Artificial Intelligence, vol. 31, pp. 355–395, 1987.
10.1016/0004-3702(87)90070-1
Web of Science® Google Scholar
[OH 05] OH J., STUERZLINGER W., DANAHY J., “Comparing SESAME and sketching on paper for conceptual 3D Design”, EUROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling, Konstanz, Germany, 2005.
Google Scholar
[OZU 06] OZUYSAL M., LEPETIT V., FLEURET F., et al., “Feature harvesting for tracking-by-detection”, Proceedings of European Conference on Computer Vision, Graz, Austria, 2006.
Google Scholar
[PAN 09] PAN Q., REITMAYR G., DRUMMOND T., “ProFORMA: probabilistic feature-based on-line rapid model acquisition”, Proceedings of the 20th British Machine Vision Conference (BMVC), London, September 2009.
Google Scholar
[PUP 05] PUPILLI M., CALWAY A., “Real-time camera tracking using a particle filter”, Proceedings of British Machine Vision Conference, British Machine Vision Association 2009, pp. 519–528, 2005.
Google Scholar
[RUB 11] RUBLEE E., RABAUD V., KONOLIGE K., et al., “ORB: an efficient alternative to SIFT or SURF”, International Conference on Computer Vision (ICCV), Barcelona, Spain, pp. 2564–2571, 2011.
Google Scholar
[SIM 98] SIMON G., BERGER M.-O., “A two-stage robust statistical method for temporal registration from features of various type”, Proceedings of the 6th International Conference on Computer Vision, Bombay, India, pp. 261–266, 1998.
Google Scholar
[SIM 00] SIMON G., FITZGIBBON A., ZISSERMAN A., “Markerless tracking using planar structures in the scene”, Proceedings of International Symposium on Augmented Reality, Darmstadt, Germany, pp. 120–128, October 2000.
Google Scholar
[SIM 11] SIMON G., BERGER M.-O., “Interactive building and augmentation of piecewise planar environments using the intersection lines”, The Visual Computer, vol. 27, no. 9, pp. 827–841, February 2011.
10.1007/s00371-011-0556-0
Web of Science® Google Scholar
[STE 99] STEWART C.V., “Robust parameter estimation in computer vision”, SIAM Reviews, vol. 41, pp. 513–537, 1999.
10.1137/S0036144598345802
Web of Science® Google Scholar
[TRI 00] TRIGGS B., MCLAUCHLAN P., HARTLEY R., et al., “Bundle adjustment – a modern synthesis”, Vision Algorithms: Theory and Practice, LNCS, vol. 1883, pp. 298–375, 2000.
Google Scholar
[UME 91] UMEYAMA S., “Least-squares estimation of transformation parameters between two point patterns”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, pp. 376–380, 1991.
10.1109/34.88573
Web of Science® Google Scholar
[VAN 07] VAN DEN HENGEL A., DICK A., THORMÄHLEN T., et al., “VideoTrace: rapid interactive scene modelling from video”, ACM SIGGRAPH 2007 Papers, ACM New York, NY, USA, p. 86–91, 2007.
10.1145/1275808.1276485
Google Scholar
[VIG 03] VIGUERAS F., BERGER M.-O., SIMON G., “Iterative multi-planar camera calibration: improving stability using model selection”, Vision, Video and Graphics (VVG '03), Bath, UK, July 2003.
Google Scholar
[WIL 02] WILCZKOWIAK M., BOYER E., STURM P., “3D modeling using geometric constraints: a parallelepiped based approach”, 7th European Conference on Computer Vision (ECCV '02), vol. 4, Springer-Verlag, Copenhagen, Denmark, pp. 221–237, 2002.
Web of Science® Google Scholar
[ZHA 97] ZHANG Z., “Parameter estimation techniques; a tutorial with application to conic fitting”, Image and Vision Computing Journal, vol. 15, no. 1, pp. 59–76, 1997.
10.1016/S0262-8856(96)01112-2
Web of Science® Google Scholar

3D Video: From Capture to Diffusion