3D Video Coding and Standards

With the advancement of 2D video compression technology, end users can experience better visual quality at much lower bit rate. To take advantage of the successful development from existing 2D video coding tools and their fast deployment in the marketplace, the 3D video codecs are often built based on the popular 2D video codecs, especially with consideration of the availability of hardware platforms. This chapter overviews the fundamentals of 2D video coding and the important components adopted in H.264. It presents a short overview of the next generation video codec, high efficiency video coding (HEVC). The chapter discusses the early development of 3D video coding technologies based on existing single-view video codecs. There are two basic coding methodologies: individual view coding and inter-view coding. The chapter discusses the two different types of frame-compatible stereo coding according to the picture resolution.

Controlled Vocabulary Terms

video coding; video compression

References

T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “Overview of the H.264/AVC Video Coding Standard”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, July 2003, pp. 560–576.
10.1109/TCSVT.2003.815165
Web of Science® Google Scholar
D. Marpe, H. Schwarz, and T. Wiegand, “Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, July, 2003, pp. 620–636.
10.1109/TCSVT.2003.815173
Web of Science® Google Scholar
V. Lappalainen, A. Hallapuro, and T.D. Hämäläinen, “Complexity of optimized H.26L video decoder implementation”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, July, 2003, pp. 717–725,
10.1109/TCSVT.2003.814968
Web of Science® Google Scholar
H. Schwarz, D. Marpe and T. Wiegand, “Analysis of Hierarchical B Pictures and MCTF”, IEEE International Conference on Multimedia and Expo, 2006, pp. 1929–1932.
Google Scholar
M. Karczewicz and R. Kurceren, “The SP- and SI-frames design for H.264/AVC”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 637–644, Jul. 2003.
10.1109/TCSVT.2003.814969
Web of Science® Google Scholar
JCTVC-H1003 High Efficiency Video Coding (HEVC) text specification Working Draft 6 (B. Bross, W.-J. Han, G. J. Sullivan, J.-R. Ohm,T. Wiegand) (CD).
Google Scholar
G. Saygili, C. G. Gurler, and A. M. Tekalp, “Evaluation of asymmetric stereo video coding and rate scaling for adaptive 3D video streaming”, IEEE Trans. on Broadcasting, vol. 57, no. 2, June 2011, pp. 593–601.
10.1109/TBC.2011.2131450
Web of Science® Google Scholar
L. Stelmach, W. J. Tam, D. Meegan, and A. Vincent. “Stereo image quality: effects of mixed spatio-temporal resolution”. IEEE Trans. on Circuits and Systems for Video Technology, vol. 10, no. 2 , Feb 2000, pp. 188–193.
10.1109/76.825717
Web of Science® Google Scholar
H. Brust, A. Smolic, K. Mueller, G. Tech, and T. Wiegand. “ Mixed resolution coding of stereoscopic video for Mobile devices”. IEEE 3DTV Conference, 2009.
Google Scholar
W. J. Tam, “Image and depth quality of asymmetrically coded stereoscopic video for 3D-TV”, JVT-W094, San Jose, CA, April 2007.
Google Scholar
A. Aksay, C. Bilen, E. Kurutepe, T. Ozcelebi, G. B. Akar, R. Civanlar, and A. M. Tekalp, “Temporal and spatial scaling for stereoscopic video compression” in Proc. EUSIPCO, Florence, Italy, 2006, pp. 4–8.
Google Scholar
H. Lee, S. Cho, K. Yun, N. Hur, and J. Kim., “ A backward-compatible, mobile, personalized 3DTV broadcasting system based on T-DMB” in Three-Dimensional Television Capture, Transmission, Display. New York: Springer, 2008.
Google Scholar
A. Vetro, T. Wiegand, and G. J. Sullivan, “Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard”, Proceedings of the IEEE, vol. 99, no. 4, April 2011. pp. 626–642.
10.1109/JPROC.2010.2098830
Web of Science® Google Scholar
A. Vetro, A. M. Tourapis, K. Muller, and T. Chen, “3D-TV content storage and transmission”, IEEE Trans. on Broadcasting, vol. 57, no. 2, June 2011, pp. 384–394.
10.1109/TBC.2010.2102950
Web of Science® Google Scholar
C.A. Segall and G.J. Sullivan, “Spatial scalability within the H.264/AVC scalable video coding extension”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 9, Sept. 2007, pp. 1121–1135.
10.1109/TCSVT.2007.906824
Web of Science® Google Scholar
H. Schwarz, D. Marpe, and T. Wiegand, “Overview of the scalable video coding extension of the H.264/AVC standard”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 9, September 2007, pp. 1103–1120.
Web of Science® Google Scholar
A. Tourapis, et. al., “A frame compatible system for 3D delivery”, m17925, ISO/IEC JTCl/SC29/WGll, Geneva, Switzerland, July 2010.
Google Scholar
Y. Chen, R. Zhang, and M. Karczewicz, “MVC based scalable codec enhancing frame-compatible stereoscopic video”, IEEE International Conference on Multimedia and Expo, 2011.
Google Scholar
C. Fehn, P. Kauff, M. O. d. Beeck, F. Ernst, W. IJsselsteijn, M. Pollefeys, L. Van Gool, E. Ofek, and I. Sexton. “ An evolutionary and optimized approach on 3D-TV”, Proc. Int. Broadcast Conf., Amsterdam, The Netherlands, September 2002, pp. 357–365.
Google Scholar
Fehn C. “ 3D-TV using depth-image-based rendering (DIBR)”, Proc. Picture Coding Symp., San Francisco, CA, USA, December 2004.
Google Scholar
K. Müller, P. Merkle, and T. Wiegand, “3D Video Representation Using Depth Maps”, Proceedings of the IEEE, vol. 99, no. 4, April 2011, pp. 643–656.
10.1109/JPROC.2010.2091090
Web of Science® Google Scholar
A. Bourge, J. Gobert, and F. Bruls. “ MPEG-C part 3: Enabling the introduction of video plus depth contents”, Proc. of IEEE Workshop on Content Generation and Coding for 3D Television, 2006.
Google Scholar
MSR 3D Video Download, available at http://research.microsoft.com/en-us/um/people/sbkang/3dvideodownload/.
Google Scholar
C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High-quality video view interpolation using a layered representation”, ACM SIGGRAPH and ACM Trans. on Graphics, Los Angeles, CA, Aug. 2004, pp. 600–608.
Web of Science® Google Scholar
A. Smolic K. Mueller, N. Stefanoski, J. Ostermann, A. Gotchev, G. B. Akar, G. Triantafyllidis, and A. Koz, “Coding algorithms for 3DTV – A Survey.”, IEEE Trans. on Circuits and Systems for Video Technology, 2007; 17(11): 1606–1621.
10.1109/TCSVT.2007.909972
Web of Science® Google Scholar
L. Zhang, C. Vázquez, and S. Knorr, “3D-TV Content Creation: Automatic 2D-to-3D Video Conversion”, IEEE Trans. on Broadcasting, vol. 57, no. 2, June 2011, pp. 372–383.
10.1109/TBC.2011.2122930
Web of Science® Google Scholar
C. Fehn, “Depth-image-based rendering (DIBR), compression and transmission for a new approach on 3D-TV”, SPIE Conf. Stereoscopic Displays Virtual Reality System XI. CA, Jan 2004, vol. 5291, pp. 93–104.
10.1117/12.524762
Web of Science® Google Scholar
C. Vazquez, W. J. Tam, and F. Speranza, “Stereoscopic imaging: filling disoccluded areas in depth image-based rendering”, SPIE Conf. 3DTV, Video, Display V, 2006, vol. 6392, 63920D.
Web of Science® Google Scholar
R. Klein Gunnewiek, R.-P. M. Berretty, B. Barenbrug, and J. P. Magalhães, “Coherent spatial and temporal occlusion generation” in Proc. SPIE 7237, 2009, p. 723713.
Google Scholar
W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, “ Smoothing depth maps for improved stereoscopic image quality” in SPIE Conf. 3–D TV, Video, Display III, Philadelphia, PA, Oct. 2004, vol. 5599, pp. 162–172.
Web of Science® Google Scholar
P. Lai, A. Ortega, C. C. Dorea, P. Yin, and C. Gomila, “Improving view rendering quality and coding efficiency by suppressing compression artifacts in depth-image coding”, Proc. SPIE VCIP, San Jose, CA,Jan. 2009.
Google Scholar
K.-J. Oh, A. Vetro, and Y.-S. Ho,, “Depth coding using a boundary reconstruction filter for 3D video systems”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 21, no. 3, March 2011, pp. 350–359.
10.1109/TCSVT.2011.2116590
Web of Science® Google Scholar
S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, “Bilateral Filtering: Theory and Applications”, Foundations and Trends in Computer Graphics and Vision, vol. 4, no. 1 2008, pp. 1–73.
10.1561/0600000020
Google Scholar
S. Liu, P. Lai, D. Tina, and C. W. Chen, “New depth coding techniques with utilization of corresponding video”, IEEE Trans. on Broadcasting, vol. 57, no. 2, June 2011, pp. 551–561.
10.1109/TBC.2011.2120750
Web of Science® Google Scholar
C.-M. Cheng, S.-J. Lin, and S.-H. Lai, “Spatio-temporally consistent novel view synthesis algorithm from video-plus-depth sequences for autostereoscopic displays”, IEEE Trans. on Broadcasting, vol. 57, no. 2, June 2011, pp. 523–532.
10.1109/TBC.2011.2139090
Web of Science® Google Scholar
P. Merkle, Y. Morvan, A. Smolic, D. Farin, K. Muller, P.H.N. deWith, and T. Wiegand, “The effects of multiview depth video compression on multiview Rendering”, Signal Processing: Image Communication, vol. 24, no. 1, 2009, pp. 73–88.
10.1016/j.image.2008.10.010
Web of Science® Google Scholar
M.-K. Kang, Y.-S. Ho, “Depth video coding using adaptive geometry based intra-prediction for 3D video system”, IEEE Transactions on Multimedia, vol. 14, no. 1, February 2012.
10.1109/TMM.2011.2169238
Web of Science® Google Scholar
M.-K. Kang, J. Lee, I. Lim, Y.-S. Ho, “ Object-adaptive depth compensated interprediction for depth video coding in 3D video system,”, IS&T/SPIE Electronic Imaging, pp. 1–12, Jan. 2011.
Google Scholar
A. Smolic, K. Mueller, N. Stefanoski, J. Ostermann, A. Gotchev, G. B. Akar, G. Triantafyllidis, and A. Koz, “Coding algorithms for 3DTV – A Survey”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, Nov. 2007, pp. 1606–1621.
10.1109/TCSVT.2007.909972
Web of Science® Google Scholar
M. Flierl and B. Girod, “Multi-view video compression”, IEEE Signal Processing Magazine, vol. 24, no. 6, December 2007, pp. 66–76.
10.1109/MSP.2007.905699
Web of Science® Google Scholar
K. Muller, P. Merkle, and T. Wiegand, “Compressing time-varying visual content”, IEEE Signal Processing Magazine, vol. 24, no. 6, December 2007, pp. 58–65.
10.1109/MSP.2007.905697
Google Scholar
P. Kerkle, A. Smolic, K. Mueller, and T. Wiegand, “Efficient prediction structures for multi-view video”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, Nov. 2007, pp. 1461–1473.
Google Scholar
J.-H. Hur, S. Cho, and Y.-L. Lee, “Adaptive local illumination change compensation method for H.264/AVC-based multiview video”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, November 2007, pp. 1496–1505.
10.1109/TCSVT.2007.903774
Web of Science® Google Scholar
K. Yamamoto, M. Kitahara, H. Kimata, T. Yendo, T. Fujii, M. Tanimoto, S. Shimizu, K. Kamikura, and Y. Yashima, “Multiview video coding using view interpolation and color correction”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, Nov. 2007, pp. 1436–1449.
10.1109/TCSVT.2007.903802
Web of Science® Google Scholar
J. H. Kim, P. Lai, J. Lopez, A. Ortega, Y. Su, P. Yin, and C. Gomila, “New coding tools for illumination and focus mismatch compensation in multiview video coding”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, November 2007, pp. 1519–1535.
10.1109/TCSVT.2007.909976
Web of Science® Google Scholar
J. Lu, H. Cai, J.-G. Lou, and J. Li, “An epipolar geometry-based fast disparity estimation algorithm for multiview image and video coding”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 6, 2007, pp. 737–750.
10.1109/TCSVT.2007.896659
Web of Science® Google Scholar
L. Ding, P.-K. Tsung, S.-Y. Chien, W.-Y. Chen, and L.-G. Chen, “Content-aware prediction algorithm with inter-view mode decision for multiview video coding”, IEEE Transactions on Multimedia, vol. 10, no. 8, pp. 1553–1563, December 2008.
10.1109/TMM.2008.2007314
Web of Science® Google Scholar
L. Shen; Z. Liu, P. An, R. Ma, and Z. Zhang, “Low-complexity mode decision for MVC”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no. 6, 2011, pp. 837–843.
10.1109/TCSVT.2011.2130310
Web of Science® Google Scholar
L. Shen, Z. Liu, S. Liu, Z. Zhang, and P. An, “Selective disparity estimation and variable size motion estimation based on motion homogeneity for multi-view coding”, IEEE Transactions on Broadcast, vol. 55, no. 4, pp. 761–766, December 2009.
10.1109/TBC.2009.2030453
Web of Science® Google Scholar
P. Merkle, A. Smolic, K. Muller, and T. Wiegand, “Multi-view video plus depth representation and coding”, Proc. IEEE ICIP, vol. I, 2007, pp. 201–205.
Google Scholar
H. Oh and Y.-S. Ho, “ H.264-based depth map coding using motion information of corresponding texture video”, Adv. Image Video Technology, vol. 4319, 2006.
Google Scholar
P. Kauff, N. Atzpadin, C. Fehn, C. Muller, O. Schreer, A. Smolic, and R. Tanger, “Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability”, Signal Processing: Image Communication, vol. 22, no. 2, 2007, pp. 217–234.
10.1016/j.image.2006.11.013
Web of Science® Google Scholar
S. Yea and A. Vetro, “View synthesis prediction for multiview video coding”, Signal Processing: Image Communication, vol. 24, no. 1, 2009, pp. 89–100.
10.1016/j.image.2008.10.007
Web of Science® Google Scholar
K. Muller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, “ View synthesis for advanced 3D video systems”, EURASIP Journal on Image and Video Processing, vol. 2008, article ID 438148, 2008.
Google Scholar
X. Xiu, D. Pang, and J. Liang, “Rectification-based view interpolation and extrapolation for multi-view video coding”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 21, no. 6, June 2011, pp. 693–707.
10.1109/TCSVT.2011.2129230
Web of Science® Google Scholar
K. Muller K, A. Smolic, K. Dix, P. Kauff, and T. Wiegand, “ Reliability-based generation and view synthesis in layered depth video”, IEEE Workshop on Multimedia Signal Processing, 2008, pp. 34–39.
Google Scholar
M. Bourges-Sevenier and E. S. Jang, “An introduction to the MPEG-4 animation framework eXtension”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, no. 7, July 2004, pp. 928–936.
10.1109/TCSVT.2004.830662
Web of Science® Google Scholar
K. Diepold, F. Pereira, and W. Chang, “MPEG-A: multimedia application formats”, IEEE Multimedia, vol. 12, no. 4, Oct/Dec 2005, pp. 34–41.
10.1109/MMUL.2005.79
Web of Science® Google Scholar
A. Smolic and P. Kauff, “Interactive 3D Video Representation and Coding Technologies”, Proceedings of the IEEE, vol. 93, no. 1, January 2005, pp. 98–110.
10.1109/JPROC.2004.839608
Web of Science® Google Scholar
E. Adelson and J. Bergen, “ The plenoptic function and the elements of early vision”, Computational Models of Visual Processing. Cambridge, MA: MIT Press, 1991, pp. 3–20.
10.7551/mitpress/2002.003.0004
Google Scholar
A. A. Alatan, Y. Yemez, U. Gudukbay, X. Zabulis, K. Muller, C. E. Erdem, C. Weigel, and A. Smolic, “Scene Representation Technologies for 3DTV – A Survey”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 17, no. 11, Nov. 2007, pp. 1587–1605.
10.1109/TCSVT.2007.909974
Web of Science® Google Scholar
M. Tanimoto, M. P. Tehrani, T. Fujii, and T. Yendo “FTV for 3D spatial communication”, Proceedings of the IEEE, vol. 100, no. 4, April 2012, pp. 905–917.
10.1109/JPROC.2011.2182101
Web of Science® Google Scholar
Call for Proposals on 3D Video Coding Technology, ISO/IEC JTC1/SC29/WG11 MPEG, N12036, March 2011.
Google Scholar

Citing Literature

3D Visual Communications