Models and Frameworks
VIBE: A Design Space for VIsual Belief Elicitation in Data Journalism
Shambhavi Mahajan, Bonnie Chen, Alireza Karduni, Yea-Seul Kim, Emily Wall,
Shambhavi Mahajan, Bonnie Chen, Alireza Karduni, Yea-Seul Kim, Emily Wall,
Abstract
The process of forming, expressing, and updating beliefs from data plays a critical role in data-driven decision making. Effectively eliciting those beliefs has potential for high impact across a broad set of applications, including increased engagement with data and visualizations, personalizing visualizations, and understanding users' visual reasoning processes, which can inform improved data analysis and decision making strategies (e.g., via bias mitigation). Recently, belief-driven visualizations have been used to elicit and visualize readers' beliefs in a visualization alongside data in narrative media and data journalism platforms such as the New York Times and FiveThirtyEight. However, there is little research on different aspects that constitute designing an effective belief-driven visualization. In this paper, we synthesize a design space for belief-driven visualizations based on formative and summative interviews with designers and visualization experts. The design space includes 7 main design considerations, beginning with an assumed data set, then structured according to: from who, why, when, what, and how the belief is elicited, and the possible feedback about the belief that may be provided to the visualization viewer. The design space covers considerations such as the type of data parameter with optional uncertainty being elicited, interaction techniques, and visual feedback, among others. Finally, we describe how more than 24 existing belief-driven visualizations from popular news media outlets span the design space and discuss trends and opportunities within this space.
Supporting Information
| Filename | Description |
|---|---|
| cgf14556-sup-0001-S1.xlsx6.2 KB | Supporting Information |
| cgf14556-sup-0001-S2.xlsx55.4 KB | Supporting Information |
| cgf14556-sup-0001-S3.pdf1.7 MB | Supporting Information |
| cgf14556-sup-0001-S4.zip517.9 KB | Supporting Information |
| cgf14556-sup-0001-S5.zip6.9 MB | Supporting Information |
| cgf14556-sup-0001-S6.pdf324.7 KB | Supporting Information |
| cgf14556-sup-0001-S7.zip2.1 MB | Supporting Information |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Aisch G., Cox A., Quealy K.: You draw it: How family income predicts children's college chances - new york times. https://www.nytimes.com/interactive/2015/05/28/upshot/you-draw-it-how-family-income-affects-childrens-college-chances.html, May 2015. (Accessed on 11/20/2021). 1, 2, 3, 7, 9
- Armona L., Fuster A., Zafar B.: Home price expectations and behaviour: Evidence from a randomized information experiment. The Review of Economic Studies 86, 4 (2019), 1371–1410. 2
- Aschwanden C., Koeze E.: How fast would you run a marathon? - fivethirtyeight. https://projects.fivethirtyeight.com/marathon-calculator/, November 2018. Accessed on 07/27/2021. 6
- ASHKENAS J.: Can you live on the minimum wage? - new york times. https://www.nytimes.com/interactive/2014/02/09/opinion/minimum-wage.html, February 2014. Accessed on 07/27/2021. 6
- Bailey M., Dávila E., Kuchler T., Stroebel J.: House price beliefs and mortgage leverage choice. The Review of Economic Studies 86, 6 (2019), 2403–2452. 2
- Bickel J. E.: Some comparisons among quadratic, spherical, and logarithmic scoring rules. Decision Analysis 4, 2 (2007), 49–65. 10
10.1287/deca.1070.0089 Google Scholar
- Badger E., Quealy K.: Where is america's heartland? pick your map - new york times. https://www.nytimes.com/interactive/2017/01/03/upshot/where-is-americas-heartland-pick-your-map.html, January 2017. (Accessed on 11/20/2021). 3, 5
- Beach L. R., Swenson R. G.: Intuitive estimation of means. Psychonomic Science 5, 4 (1966), 161–162. 10
10.3758/BF03328331 Google Scholar
- Borkin M. A., Vo A. A., Bylinskii Z., Isola P., Sunkavalli S., Oliva A., Pfister H.: What makes a visualization memorable? IEEE transactions on visualization and computer graphics 19, 12 (2013), 2306–2315. 6
- Clarke S., Hulley-Jones F.: Find out which brexit deal is right for you - guardian. https://www.theguardian.com/politics/ng-interactive/2019/jan/18/find-out-which-brexit-deal-is-right-for-you, January 2019. Accessed on 07/27/2021. 6
- CNN: Cnn interactives. https://www.cnn.com/specials/multimedia/cnn-interactives, 2021. Accessed on 07/27/2021. 6
- Chen L., Pu P.: Survey of preference elicitation methods. Tech. rep., 2004. 2
- Explorable: Explorable explanations. https://explorabl.es/, 2021. Accessed on 07/27/2021. 6
- FiveThirtyEight: Fivethirtyeight interactives. https://projects.fivethirtyeight.com/, 2021. Accessed on 07/27/2021. 6
- Fodor J. A.: Representations: Philosophical essays on the foundations of cognitive science. Mit Press, 1983. 2
- Fan M., Shi S., Truong K. N.: Practices and challenges of using think-aloud protocols in industry: An international survey. Journal of Usability Studies 15, 2 (2020). 10
- Gavrilova T.: Choice of knowledge elicitation technique: the psychological aspect. International Journal on Information Theory and Applications 1, 8 (1993), 20–26. 2
- Garthwaite P. H., Kadane J. B., O'Hagan A.: Statistical methods for eliciting probability distributions. Journal of the American Statistical Association 100, 470 (2005), 680–701. 2, 3, 10
- Company salaries. Glassdoor https://www.glassdoor.com/Salaries/index.htm, 2008-2021. 5
- Goldstein D. G., Rothschild D.: Lay understanding of probability distributions. Judgment & Decision Making 9, 1 (2014). 9
- Guardian T.: Guardian interactives. https://www.theguardian.com/interactive, 2021. Accessed on 07/27/2021. 6
- Hohman F., Conlen M., Heer J., Chau D. H.: Communicating with interactive articles - distill. https://distill.pub/2020/communicating-with-interactive-articles/, 2020. Accessed on 07/27/2021. 6
- Hullman J., Kay M., Kim Y.-S., Shrestha S.: Imagining replications: Graphical prediction & discrete visualizations improve recall & estimation of effect uncertainty. IEEE transactions on visualization and computer graphics 24, 1 (2018), 446–456. 2
- Heyer J., Raveendranath N. K., Reda K.: Pushing the (visual) narrative: the effects of prior knowledge elicitation in provocative topics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020), pp. 1–14. 2
- Hoffman R. R., Shadbolt N. R., Burton A. M., Klein G.: Eliciting knowledge from experts: A methodological analysis. Organizational behavior and human decision processes 62, 2 (1995), 129–158. 2
- Hunter D.: On the relation between categorical and probabilistic belief. Noûs 30, 1 (1996), 75–98. 2
- Janis I. L.: Groupthink. IEEE Engineering Management Review 36, 1 (2008), 36. 4
10.1109/EMR.2008.4490137 Google Scholar
- Johnson S. R., Tomlinson G. A., Hawker G. A., Granton J. T., Feldman B. M.: Methods to elicit beliefs for bayesian priors: a systematic review. Journal of clinical epidemiology 63, 4 (2010), 355–369. 10
- Koonchanok R., Baser P., Sikharam A., Raveendranath N. K., Reda K.: Data prophecy: Exploring the effects of belief elicitation in visual analytics. 3
- Kim Y.: Exploring the effects of source credibility and others' comments on online news evaluation. Electronic News 9, 3 (2015), 160–176. 9
10.1177/1931243115593318 Google Scholar
- Kim Y.-S., Kayongo P., Grunde-McLaughlin M., Hullman J.: Bayesian-assisted inference from visualized data. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 989–999. 2, 3
- Karduni A., Markant D., Wesslen R., Dou W.: A bayesian cognition approach for belief updating of correlation judgement through uncertainty visualizations. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 978–988. 3, 9, 10
- Kim Y.-S., Reinecke K., Hullman J.: Explaining the gap: Visualizing one's predictions improves recall and comprehension of data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2017), ACM, pp. 1375–1386. 1, 2, 9
- Kim Y.-S., Reinecke K., Hullman J.: Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation. IEEE transactions on visualization and computer graphics 24, 1 (2018), 760–769. 9
- King R., Silver N.: Which flight will get you there fastest? - fivethirtyeight. https://projects.fivethirtyeight.com/flights/, November 2015. Accessed on 07/27/2021. 6
- Kim Y.-S., Walls L. A., Krafft P., Hullman J.: A bayesian cognition approach to improve data visualization. In Proceedings of the 2019 CHI conference on human factors in computing systems (2019), pp. 1–14. 1, 2, 3
- Manski C. F.: Measuring expectations. Econometrica 72, 5 (2004), 1329–1376. 2
- Moody J. W., Blanton J. E., Will R. P.: Capturing expertise from experts: The need to match knowledge elicitation techniques with expert system types. Journal of Computer Information Systems 39, 2 (1999), 89–95. 2
- McHugh M. L.: Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282. 7
- Manski C. F., Neri C.: First-and second-order subjective expectations in strategic decision-making: Experimental evidence. Games and Economic Behavior 81 (2013), 232–254. 5
- Moskin J., Plumer B., Lieberman R., Weingart E.: Your questions about food and climate change, answered - new york times. https://www.nytimes.com/interactive/2019/04/30/dining/climate-change-food-eating-habits.html, April 2019. (Accessed on 11/20/2021). 3
- Mehta D., Wolfe J.: Do you know where america stands on guns? - fivethirtyeight. https://projects.fivethirtyeight.com/guns-parkland-polling-quiz/, March 2018. (Accessed on 11/20/2021). 3
- Nguyen F.: Belief-driven data journalism. Computation+Journalism (2019). 3, 5
- Nickerson R. S.: Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175–220. 9
10.1037/1089-2680.2.2.175 Google Scholar
- O'Hagan A., Buck C. E., Daneshkhah A., Eiser J. R., Garthwaite P. H., Jenkinson D. J., Oakley J. E., Rakow T.: Uncertain judgements: eliciting experts' probabilities. John Wiley & Sons, 2006. 2, 5
- Payne J. W., Bettman J. R., Schkade D. A., Schwarz N., Gregory R.: Measuring constructed preferences: Towards a building code. In Elicitation of preferences. Springer, 1999, pp. 243–275. 2
- Pommeranz A., Broekens J., Wiggers P., Brinkman W.-P., Jonker C. M.: Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Modeling and User-Adapted Interaction 22, 4 (2012), 357–397. 2
- Pu P., Chen L., Hu R.: Evaluating recommender systems from the user's perspective: survey of the state of the art. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 317–355. 2
- Pang A. T., Wittenbrink C. M., Lodha S. K., et al.: Approaches to uncertainty visualization. The Visual Computer 13, 8 (1997), 370–390. 5
- Stevens S. S., et al.: On the theory of scales of measurement. 5
- Schmitt-Beck R.: Bandwagon effect. The international encyclopedia of political communication (2015), 1–5. 4
- Singer P., Helic D., Hotho A., Strohmaier M.: Hyptrails: A bayesian approach for comparing hypotheses about human trails on the web. In Proceedings of the 24th International Conference on World Wide Web (2015), pp. 1003–1013. 6, 10
- Shuford E. H.: Percentage estimation of proportion as a function of element type, exposure time, and task. Journal of Experimental Psychology 61, 5 (1961), 430. 10
10.1037/h0043335 Google Scholar
- Simon D., Krawczyk D. C., Holyoak K. J.: Construction of preferences by constraint satisfaction. Psychological Science 15, 5 (2004), 331–336. 2
- Surowiecki J.: The wisdom of crowds. Anchor, 2005. 4
- Swamidass P. M.: Forecast Errors: Encyclopedia of production and manufacturing management. Springer Science & Business Media, 2000. 5
- Times N. Y.: 2020: The year in visual stories and graphics. https://www.nytimes.com/interactive/2020/12/30/us/2020-year-in-graphics.html, December 2020. Accessed on 07/27/2021. 6
- Tversky A., Kahneman D.: Judgment under uncertainty: Heuristics and biases. science 185, 4157 (1974), 1124–1131. 10
- Vul E., Goodman N., Griffiths T. L., Tenenbaum J. B.: One and done? optimal decisions from very few samples. Cognitive science 38, 4 (2014), 599–637. 3
- Wang S. W.: Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters 111, 1 (2011), 30–33. 10
- Waterman D. A.: A guide to expert systems. Addison-Wesley Longman Publishing Co., Inc., 1985. 2
- Wall E., Blaha L. M., Franklin L., Endert A.: Warning, bias may occur: A proposed approach to detecting cognitive bias in interactive visual analytics. In 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) (2017), IEEE, pp. 104–115. 9
- Weber E. U., Johnson E. J.: Constructing preferences from memory. The Construction of Preference, Lichtenstein, S. & Slovic, P.,(eds.) (2006), 397–410. 2
- Wall E., Stasko J., Endert A.: Toward a design space for mitigating cognitive bias in vis. In 2019 IEEE Visualization Conference (VIS) (2019), IEEE, pp. 111–115. 9
- What is a zestimate? zillow's zestimate accuracy. Zillow, May 27, 2021, https://www.zillow.com/z/zestimate/, 2021. 5
- Zacks J., Tversky B.: Bars and lines: A study of graphic communication. Memory & cognition 27, 6 (1999), 1073–1079. 6
