Functionality Representations and Applications for Shape Analysis
Corresponding Author
R. Hu
Shenzhen University
All the authors have equal contribution.
Corresponding author: R. Hu ([email protected])Search for more papers by this authorM. Savva
Princeton University
All the authors have equal contribution.
Search for more papers by this authorO. van Kaick
Carleton University
All the authors have equal contribution.
Search for more papers by this authorCorresponding Author
R. Hu
Shenzhen University
All the authors have equal contribution.
Corresponding author: R. Hu ([email protected])Search for more papers by this authorM. Savva
Princeton University
All the authors have equal contribution.
Search for more papers by this authorO. van Kaick
Carleton University
All the authors have equal contribution.
Search for more papers by this authorAbstract
A central goal of computer graphics is to provide tools for designing and simulating real or imagined artifacts. An understanding of functionality is important in enabling such modeling tools. Given that the majority of man-made artifacts are designed to serve a certain function, the functionality of objects is often reflected by their geometry, the way that they are organized in an environment, and their interaction with other objects or agents. Thus, in recent years, a variety of methods in shape analysis have been developed to extract functional information about objects and scenes from these different types of cues. In this report, we discuss recent developments that incorporate functionality aspects into the analysis of 3D shapes and scenes. We provide a summary of the state-of-the-art in this area, including a discussion of key ideas and an organized review of the relevant literature. More specifically, the report is structured around a general definition of functionality from which we derive criteria for classifying the body of prior work. This definition also facilitates a comparative view of methods for functionality analysis. We focus on studying the inference of functionality from a geometric perspective, and pose functionality analysis as a process involving both the geometry and interactions of a functional entity. In addition, we discuss a variety of applications that benefit from an analysis of functionality, and conclude the report with a discussion of current challenges and potential future works.
References
- Bar-Aviv E., Rivlin E.: Functional 3d object classification using simulation of embodied agent. In British Machine Vision Conference (2006), pp. 307–316. 7, 13
- Bogoni L., Bajcsy R.: Interactive recognition and representation of functionality. Computer Vision and Image Understanding 62, 2 (1995), 194–214. 2
- Chao Y.-W., Wang Z., Mihalcea R., Deng J.: Mining semantic affordances of visual object categories. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 4259–4267. 2
- Desai C., Ramanan D.: Predicting functional regions on objects. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops (2013), pp. 968–975. 2
- Fish N., Averkiou M., van Kaick O., Sorkine-Hornung O., Cohen-Or D., Mitra N. J.: Meta-representation of shape families. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 34:1–11. 4, 5, 7, 10, 18
- Fu Q., Chen X., Su X., Fu H.: Pose-inspired shape synthesis and functional hybrid. IEEE Trans. Visualization & Computer Graphics 23, 12 (2017), 2574–2585. 18
- Fouhey D. F., Delaitre V., Gupta A., Efros A. A., Laptev I., Sivic J.: People watching: Human actions as a cue for single view geometry. International journal of computer vision 110, 3 (2014), 259–274. 2
- Fisher M., Ritchie D., Savva M., Funkhouser T., Hanrahan P.: Example-based synthesis of 3D object arrangements. ACM Trans. on Graph (SIGGRAPH Asia) 31, 6 (2012), 135:1–11. 7, 8, 17
- Fisher M., Savva M., Hanrahan P.: Characterizing structural relationships in scenes using graph kernels. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 34:1–12. 4, 7, 9, 12, 17
- Fisher M., Savva M., Li Y., Hanrahan P., Niessner M.: Activity-centric scene synthesis for functional 3D scene modeling. ACM Trans. on Graph (SIGGRAPH Asia) 34, 6 (2015), 179:1–13. 4, 7, 13, 14, 17
- Fish N., van Kaick O., Bermano A., Cohen-Or D.: Structure-oriented networks of shape collections. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 171:1–14. 6, 7
- Golovinskiy A., Funkhouser T.: Consistent segmentation of 3D models. Comput. Graph. (Proc. SMI) 33, 3 (2009), 262–269. 6
- Gelfand N., Guibas L. J.: Shape segmentation using local slippage analysis. In Proc. SGP (2004), pp. 214–223. 5, 7, 10, 15
- Grabner H., Gall J., Van Gool L.: What makes a chair a chair? In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2011), IEEE, pp. 1529–1536. 7, 13, 14, 16
- Guerrero P., Mitra N. J., Wonka P.: RAID: A relation-augmented image descriptor. ACM Trans. on Graph (SIGGRAPH) 35, 4 (2016), 46:1–12. 4, 10, 16, 17
- Gupta A., Satkin S., Efros A. A., Hebert M.: From 3D scene geometry to human workspace. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2011), IEEE, pp. 1961–1968. 2
- Hu R., Fan L., Liu L.: Co-segmentation of 3D shapes via subspace clustering. Computer Graphics Forum (SGP) 31, 5 (2012), 1703–1713. 6
- Hu R., Li W., Kaick O. V., Shamir A., Zhang H., Huang H.: Learning to predict part mobility from a single static snapshot. ACM Trans. on Graph (SIGGRAPH Asia) 36, 6 (2017), 217:1–13. 7, 11, 12, 13, 16
- Hu R., van Kaick O., Wu B., Huang H., Shamir A., Zhang H.: Learning how objects function via co-analysis of interactions. ACM Trans. on Graph (SIGGRAPH) 35, 4 (2016), 47:1–13. 7, 12, 16
- Huang Q., Wang F., Guibas L.: Functional map networks for analyzing and exploring large shape collections. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 36:1–11. 6
- Hu R., Zhu C., van Kaick O., Liu L., Shamir A., Zhang H.: Interaction context (ICON): Towards a geometric functionality descriptor. ACM Trans. on Graph (SIGGRAPH) 34, 4 (2015), 83:1–12. 4, 5, 6, 7, 10, 11, 12, 17
- Jiang Y., Koppula H., Saxena A.: Hallucinated humans as the hidden context for labeling 3D scenes. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2013), IEEE, pp. 2993–3000. 7, 13, 14, 16
- Kim V. G., Chaudhuri S., Guibas L., Funkhouser T.: Shape2pose: Human-centric shape analysis. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 120:1–12. 5, 7, 14, 17
- Kalogerakis E., Hertzmann A., Singh K.: Learning 3D mesh segmentation and labeling. ACM Trans. on Graph (SIGGRAPH) 29, 3 (2010), 102:1–12. 8
- Kim V. G., Li W., Mitra N. J., Chaudhuri S., DiVerdi S., Funkhouser T.: Learning part-based templates from large collections of 3D shapes. ACM Trans. on Graph (SIGGRAPH) 32, 4 (2013), 70:1–12. 6
- Koo B., Li W., Yao J., Agrawala M., Mitra N. J.: Creating works-like prototypes of mechanical objects. ACM Trans. on Graph (SIGGRAPH Asia) 33, 6 (2014), 217:1–9. 4, 18
- Kim D. I., Sukhatme G. S.: Semantic labeling of 3D point clouds with object affordance for robot manipulation. In Int. Conf. Robotics and Automation (ICRA) (2014), pp. 5578–5584. 7, 12, 15
- Lee K. H., Choi M. G., Lee J.: Motion patches: building blocks for virtual environments annotated with motion data. ACM Trans. on Graph (SIGGRAPH) 25, 3 (2006), 898–906. 7, 14, 15
- Lun Z., Kalogerakis E., Wang R., Sheffer A.: Functionality preserving shape style transfer. ACM Trans. on Graph 35, 6 (2016), 209. 18
- Laga H., Mortara M., Spagnuolo M.: Geometry and context for semantic correspondences and functionality recognition in man-made 3D shapes. ACM Trans. on Graph 32, 5 (2013), 150:1–16. 7, 12, 15
- Lau M., Ohgawara A., Mitani J., Igarashi T.: Converting 3D furniture models to fabricatable parts and connectors. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 85:1–6. 4, 18
- Li H., Wan G., Li H., Sharf A., Xu K., Chen B.: Mobility fitting using 4D RANSAC. Computer Graphics Forum (SGP) 35, 5 (2016), 79–88. 12
- Mitra N. J., Guibas L., Pauly M.: Partial and approximate symmetry detection for 3D geometry. ACM Trans. on Graph (SIGGRAPH) 25, 3 (2006), 560–568. 8
- Ma R., Li H., Zou C., Liao Z., Tong X., Zhang H.: Action-driven 3D indoor scene evolution. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 173:1–13. 4, 7, 14, 15, 17
- Mitra N. J., Pauly M., Wand M., Ceylan D.: Symmetry in 3D geometry: Extraction and applications. In Eurographics State-of-the-art Report (STAR) (2012), Eurographics. 8
- Merrell P., Schkufza E., Li Z., Agrawala M., Koltun V.: Interactive furniture layout using interior design guidelines. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 87:1–10. 7, 8, 18
- Myers A., Teo C. L., C., Aloimonos Y.: Affordance detection of tool parts from geometric features. In Robotics and Automation (ICRA), 2015 IEEE International Conference on (2015), IEEE, pp. 1374–1381. 7, 12
- Mitra N., Wand M., Zhang H., Cohen-Or D., Bokeloh M.: Structure-aware shape processing. In Eurographics State-of-the-art Report (STAR) (2013), Eurographics. 2, 6
- Meng M., Xia J., Luo J., He Y.: Unsupervised co-segmentation for 3D shapes using iterative multi-label optimization. Computer-Aided Design 45 (2013), 312–320. 6
- Mitra N. J., Yang Y.-L., Yan D.-M., Li W., Agrawala M.: Illustrating how mechanical assemblies work. ACM Trans. on Graph (SIGGRAPH) 29, 4 (2010), 58:1–12. 4, 7, 9, 18
- Pirk S., Krs V., Hu K., Rajasekaran S. D., Kang H., Yoshiyasu Y., Benes B., Guibas L. J.: Understanding and exploiting object interaction landscapes. ACM Trans. on Graph 36, 3 (2017), 31:1–14. 4, 5, 7, 11, 16
- Pauly M., Mitra N. J., Wallner J., Pottmann H., Guibas L. J.: Discovering structural regularity in 3D geometry. ACM Trans. on Graph (SIGGRAPH) 27, 3 (2008), 43:1–11. 8
- Pechuk M., Soldea O., Rivlin E.: Learning function-based object classification from 3D imagery. Computer Vision and Image Understanding 110, 2 (2008), 173–191. 7, 10, 15
- Rivlin E., Dickinson S. J., Rosenfeld A.: Recognition by functional parts. Computer Vision and Image Understanding 62, 2 (1995), 164–176. 10
- Roy A., Todorovic S.: A multi-scale CNN for affordance segmentation in RGB images. In European Conf. on Computer Vision (2016), Springer, pp. 186–201. 2
- Savva M., Chang A. X., Hanrahan P., Fisher M., Niessner M.: SceneGrok: Inferring action maps in 3D environments. ACM Trans. on Graph (SIGGRAPH Asia) 33, 6 (2014), 212:1–10. 4, 5, 6, 7, 13, 14, 16, 17
- Savva M., Chang A. X., Hanrahan P., Fisher M., Niessner M.: PiGraphs: Learning Interaction Snapshots from Observations. ACM Trans. on Graph (SIGGRAPH) 35, 4 (2016), 139:1–12. 4, 5, 7, 14, 17
- Sharf A., Huang H., Liang C., Zhang J., Chen B., Gong M.: Mobility-trees for indoor scenes manipulation. Computer Graphics Forum 33, 1 (2014), 2–14. 12, 18
- Shao T., Li D., Rong Y., Zheng C., Zhou K.: Dynamic furniture modeling through assembly instructions. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 172:1–15. 18
- Shu Z., Qi C., Xin S., Hu C., Wang L., Zhang Y., Liu L.: Unsupervised 3D shape segmentation and co-segmentation via deep learning. Computer Aided Geometric Design 43 (2016), 39–52. 6
- Sullivan L. H.: The tall office building artistically considered. Lippincott's Magazine (Mar. 1896), 403–409. 3
- Sidi O., van Kaick O., Kleiman Y., Zhang H., Cohen-Or D.: Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering. ACM Trans. on Graph (SIGGRAPH Asia) 30, 6 (2011), 126:1–10. 6
- Tevs A., Huang Q., Wand M., Seidel H.-P., Guibas L.: Relating shapes via geometric symmetries and regularities. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 119:1–12. 8
- Umetani N., Igarashi T., Mitra N. J.: Guided exploration of physically valid shapes for furniture design. ACM Trans. on Graph 31, 4 (2012), 86–1. 18
- van Kaick O., Tagliasacchi A., Sidi O., Zhang H., Cohen-Or D., Wolf L., Hamarneh G.: Prior knowledge for part correspondence. Computer Graphics Forum (Eurographics) 30, 2 (2011), 553–562. 8
- van Kaick O., Xu K., Zhang H., Wang Y., Sun S., Shamir A., Cohen-Or D.: Co-hierarchical analysis of shape structures. ACM Trans. on Graph (SIGGRAPH) 32, 4 (2013), 69:1–10. 6
- Wang H., Liang W., Yu L.-F.: Transferring objects: Joint inference of container and human pose. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2017), pp. 2933–2941. 7, 13
- Wu Z., Wang Y., Shou R., Chen B., Liu X.: Unsupervised co-segmentation of 3D shapes via affinity aggregation spectral clustering. Computers & Graphics (Proc. SMI) 37 (2013), 628–637. 6
- Wang Y., Xu K., Li J., Zhang H., Shamir A., Liu L., Cheng Z., Xiong Y.: Symmetry hierarchy of man-made objects. Computer Graphics Forum (Eurographics) 30, 2 (2011), 287–296. 4, 8, 18
- Xu K., Kim V. G., Huang Q., Kalogerakis E.: Data-driven shape analysis and processing. Computer Graphics Forum 36, 1 (2017), 101–132. 2, 6, 19
- Xu M., Li M., Xu W., Deng Z., Yang Y., Zhou K.: Interactive mechanism modeling from multi-view images. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 236:1–13. 4, 7, 9, 18
- Xu K., Ma R., Zhang H., Zhu C., Shamir A., Cohen-Or D., Huang H.: Organizing heterogeneous scene collection through contextual focal points. ACM Trans. on Graph (SIGGRAPH) 33, 4 (2014), 35:1–12. 5, 9, 17
- Xu K., Stewart J., Fiume E.: Constraint-based automatic placement for scene composition. In Proc. Graphics Interface (2002), Vol. 2, pp. 25–34. 7, 8, 17
- Yao B., Fei-Fei L.: Recognizing human-object interactions in still images by modeling the mutual context of objects and human poses. Trans. on Pattern Analysis and Machine Intelligence 34, 9 (2012), 1691–1703. 2
- Yumer M. E., Kara L. B.: Co-constrained handles for deformation in shape collections. ACM Trans. on Graph (SIGGRAPH Asia) 33, 6 (2014), 187: 1–11. 4, 7, 10, 18
- Yao B., Ma J., Fei-Fei L.: Discovering object functionality. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2013), pp. 2512–2519. 2
- Yu L.-F., Yeung S. K., Tang C.-K., Terzopoulos D., Chan T. F., Osher S.: Make it home: automatic optimization of furniture arrangement. ACM Trans. on Graph (SIGGRAPH) 30, 4 (2011), 86:1–12. 7, 8, 17
- Zhao X., Choi M. G., Komura T.: Character-object interaction retrieval using the interaction bisector surface. Computer Graphics Forum (Eurographics) 36, 2 (2017), 119–129. 7, 14, 15
- Zheng Y., Cohen-Or D., Averkiou M., Mitra N. J.: Recurring part arrangements in shape collections. Computer Graphics Forum 33, 2 (2014), 115–124. 6
- Zheng Y., Cohen-Or D., Mitra N. J.: Smart variations: Functional substructures for part compatibility. Computer Graphics Forum (Eurographics) 32, 2pt2 (2013), 195–204. 7, 9, 10, 18
- Zhu Y., Fathi A., Fei-Fei L.: Reasoning about object affordances in a knowledge base representation. In European Conf. on Computer Vision (2014), Springer, pp. 408–424. 2
- Zhao X., Hu R., Guerrero P., Mitra N., Komura T.: Relationship templates for creating scene variations. ACM Trans. on Graph (SIGGRAPH Asia) 35, 6 (2016), 207:1–13. 7, 9, 18
- Zhu Y., Jiang C., Zhao Y., Terzopoulos D., Zhu S.-C.: Inferring forces and learning human utilities from videos. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2016), IEEE, pp. 3823–3833. 7, 13, 14, 16
- Zheng Y., Liu H., Dorsey J., Mitra N. J.: Ergonomics-inspired reshaping and exploration ofcollections ofmodels. IEEE Trans. Visualization & Computer Graphics 22, 6 (2016), 1732–1744. 7, 14, 18
- Zhao X., Wang H., Komura T.: Indexing 3D scenes using the interaction bisector surface. ACM Trans. on Graph 33, 3 (2014), 22:1–14. 4, 6, 7, 9, 10, 17
- Zhao Y., Zhu S.-C.: Scene parsing by integrating function, geometry and appearance models. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2013), pp. 3119–3126. 2
- Zhu Y., Zhao Y., Chun Zhu S.: Understanding tools: Task-oriented object modeling, learning and recognition. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR) (2015), pp. 2855–2864. 7, 13