Systematic review of spatial abilities and virtual reality: The role of interaction

1 INTRODUCTION

The importance of spatial abilities, especially for individuals' success in science, technology, engineering, and mathematics (STEM) domains is well established (Buckley et al., 2018; Wai et al., 2009) because these abilities facilitate the identification, analysis, and mental representation of objects and shapes in the field of view (Carroll, 2009). Individual levels of spatial ability differ and are influenced by gender (Maeda & Yoon, 2013) sports-related activities (Morawietz & Muehlbauer, 2021), general childhood activities (Peterson et al., 2020), or video games (Bediou et al., 2018). This underscores the need to train individuals for these abilities, especially those who pursue studies or careers in STEM (Sorby, 2009). Spatial abilities are malleable, particularly for low-spatial-ability students; however, research on how to best support these students is lacking (Uttal et al., 2013).

Focusing on engineering, researchers have emphasized the importance of spatial abilities for individuals' success in education and careers. They found that spatial abilities have positively influenced individuals' creativity and problem-solving skills and have predicted their selection of engineering-related studies (Duffy et al., 2018). Through supplementary interventions, engineering students' spatial skills can be improved, and higher grades of those who are doing introductory courses can be achieved. Research on the influence of spatial ability on academic performance has provided positive results.

The use of virtual reality (VR) in engineering education has become popular. By providing an immersive and interactive learning environment, VR can enhance learning (Han et al., 2023). However, research on the use of VR in engineering education lacks a theoretical and pedagogical background. Therefore, VR research based on educational and multimedia learning theories is required (Oje et al., 2023).

This article addresses the need for further understanding of the interplay between spatial abilities and interaction in immersive VR (IVR), and vice versa. For example, studies have only started to focus on the influence of users' spatial abilities on object manipulation (Drey et al., 2023).

Recent reviews have indicated the usefulness of virtual technologies, particularly VR, in training learners for spatial abilities (Di & Zheng, 2022; Uz Bilgin et al., 2021). Several studies have focused on visual aspects and on the influence of VR on different levels of spatial abilities (Höffler, 2010; Sun et al., 2019). One feature of the broadly accessible IVR devices is that they render new forms of representation for designing three-dimensional settings, incorporating immersive environments, and free interactions. Earlier, virtual interaction had been limited to pointing and clicking and was thus “disconnected from the rich array of sensorimotor experiences with tight spatial relationships experienced in real life” (Clifton et al., 2016, p. 2). VR incorporates physical movement into embodied cognition. A review of the interplay between embodied interaction and spatial abilities indicates that the former may support spatial interactions. However, further research is needed on design decisions with a general focus on VR, rather than on the specific IVR (Lee-Cultura & Giannakos, 2020).

Studies on the general effect of IVR using head-mounted displays (HMDs) on learning have shown greater effectiveness of IVR than that of non-immersive learning, with some exceptions that reported a negative effect. IVR has been effectively used in the field of science education and specific-ability training and is a “promising complement that can diversify learning experiences,” (Wu et al., 2020, p. 2001) although it cannot replace traditional learning altogether (Wu et al., 2020). Simultaneously, IVR is promising both for K–6 (Villena-Taranilla et al., 2022) and K–12 learners (Wu et al., 2020). Makransky and Petersen's (2021) cognitive affective model of immersive learning (CAMIL) highlights the importance of control factors that influence the learning process in IVR. Petersen et al. (2022, p. 13) describe “two indirect paths from interactivity and immersion to learning: the first involving situational interest and the second involving embodied learning” and point to a gap in knowledge about how physical actions with virtual objects—in other words, interactions—shape and influence learning. Among many influences of IVR on learning, as described by Helmke's supply use model (Helmke & Weinert, 1997), covering educational and cultural backgrounds, didactic context, and class atmosphere (Dengel & Magdefrau, 2018), this review focuses on the influence of IVR on individuals' spatial abilities. The study aimed to address the above-mentioned gap by reviewing available research on spatial abilities and interactions in IVR. We specially focused on the bibliographic features of the research; the measurement and characteristics of interaction, VR, and spatial abilities; and the potential contributions of VR to improving spatial abilities.

2 METHOD

A literature search was conducted according to the guidelines Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA-P; 2015) and suggestions on the effective conduct of a systematic review in engineering education (Borrego et al., 2014). The search was performed on three different databases—Scopus, Web of Science, and PsychINFO—to obtain a general overview. The psychology-specific database (PsychINFO) was included as a large part of previous and recent research in the field of spatial ability, and VR stems from this area of expertise (Uz Bilgin et al., 2021). The search string comprised three key concepts listed in Table 1. Inconsistencies in the terminology used to describe each concept resulted in a large number of search terms.

The timeframe ranged from 2012 to 2021. This follows Hamilton et al., who identified an increase in VR literature after 2013 when the Oculus Rift Development Kit 1 (DK-1) was released as “economically viable and high quality [sic] HMDs that could be used both within educational institutions and at home” (Hamilton et al., 2021, p. 6). The year 2012 was included to cover the period before this increase.

2.1 Inclusion and exclusion criteria

To identify articles, categorize them, and compare their different research findings, we used the following inclusion criteria:

1. Virtual reality: The article discusses the use of VR. As discussed above, this includes all types of glasses, HMDs, and similar devices that facilitate a stereoscopic view.
2. Spatial abilities: The article must make a quantitative assessment of spatial abilities and include both the test and the results.
3. Interaction: There must be an interactive part of research that extends beyond passive viewing of the stimulus.
4. Articles published in English.

We adopted the following exclusion criteria to review selected articles again and exclude those that were not suitable.

1. Type of article that is not appropriate (reviews, theoretical articles).
2. Articles that do not use stereoscopic VR conditions.
3. Lack of exact information on the used spatial ability test.
4. Lack of description of the used device and interaction.

A total of 352 articles were identified using the search string. After removing duplicates, we retained 262 articles for the initial screening. We screened titles and abstracts for the selection of all relevant articles. After applying the inclusion and exclusion criteria, we selected 91 studies. Thereafter, we screened the full text of these articles; finally, 24 articles were included in the review. The flow diagram shows the process according to the PRISMA guidelines (Figure 1). For example, Back et al.'s (2020) article was excluded because its description of the spatial ability test was appraised as too vague, though it describes an interactive collaborative learning scenario and the influence of spatial abilities on learning. Also, the article defined interaction, and its setting was a VR environment. Another study that was excluded is Yilmaz et al. (2015). Although it explained spatial ability tests used and the interactions in VR, the article does not mention the device used to interact with the VR environment.

3 RESULTS

The findings of the study are presented based on the research questions, starting with Q1: What are the bibliographic features of the research on interaction, VR, and spatial ability?

No relevant articles were found for the years, 2012, 2014, and 2015. However, two articles were published in 2013. Since 2016 onward, there has been an increasing number of articles that focus on spatial abilities and interaction in a VR setting. The period 2019–2021 witnessed an overall small number of five articles per year; however, a general rise in interest in VR was reflected between 2016 and 2019, although the trend did not continue after 2019 (Figure 2).

Most publications were by authors from North America, followed by those from Europe, Asia, and South America (Figure 3). In North America, a significant number of publications came from the United States, whereas in Europe, Germany stood out as the source of most publications.

The research characteristics were categorized based on participants, used interaction devices, and actual interactions (Table 2). The most frequently used device is the “HTC Vive HMD” and its associated controllers. For interactive training in the medical domain, two studies used training tools specifically designed for practice in that field (Jang et al., 2017; Sommer et al., 2021). Three other studies used interactive devices designed only for specific VR training (Barrett & Hegarty, 2016; Chang et al., 2017; Gehrke et al., 2018). Lukačević et al. (2020) enhanced the possibilities of the “HTC Vive” with additional tools. However, two studies merely stated that controllers were used, without offering further explanations (Brown et al., 2019; Chao & Chang, 2020). To ease orientation throughout our findings, we referenced each covered study using a serial number.

When focusing on interaction, we identified several themes as common in all reviewed studies. The first is a direct interaction, which is similar to a specific task reproduced in VR. This includes the movement of physical blocks to align fences (Chang et al., 2017), driving simulations (Smyth et al., 2021) or scope orientation, and cholecystectomy dissection training (Sommer et al., 2021). Another idea of using interaction in VR is the exploration of entities that are too small or large to be identified in the analog world. Barrett and Hegarty (2016) used VR to allow manipulation of molecular models to match orientation, Jang et al. (2017) included the rotation of an ear model, and Safadel and White (2020) implemented the grabbing and attaching of molecules. In Kleanthous and Matsi's (2019) study, participants had to navigate and pick planets inside the solar system. The third theme was the manipulation and combination of abstract objects and shapes. This includes the rotation and combination of surfaces and solids (Brown et al., 2019), the rotation, flipping, and stacking of building blocks (Chao & Chang, 2020), or the development of a 3D arrangement by combining regular geometric forms (Obeid & Demirkan, 2020). As the last idea, we included orientation and walking inside the VR, as mentioned by several studies. This covered the comparison of teleportation or walking (Cherep et al., 2020), navigation inside a warzone (Parsons et al., 2013), and finding locations by pointing (Whitney, 2019).

Most studies covered only one of these aspects, and only a few of them had combined these aspects into more complex interactions. Gehrke et al. (2018) combined walking (walking inside an invisible maze) with visual feedback when participants' hands touched an invisible wall. Alternatively, Lages and Bowman (2018) compared rotating an object while walking around it. The details of this interaction are interesting. Some studies described every possible interaction, whereas others stated only the general idea behind the interaction without going into details.

To compare and summarize the spatial ability tests used, we categorized the tests used in each study into general types of spatial ability tests and then into the four categories of intrinsic/extrinsic and static/dynamic properties used, for example, by Uttal et al. (2013) (see Table 3). Mental rotation mostly consists of the mental rotation test and the Purdue spatial visualization test: visualization of rotations (PSVT: R), the building block rotation test, and the tube figure test. The numbers in the third column refer to the following categories: 1 = intrinsic, static; 2 = intrinsic, dynamic; 3 = extrinsic, static; and 4 = extrinsic, dynamic.

To comparatively analyze the findings of the research on training, we categorized them based on the influence of spatial-ability characteristics on the interactive VR learning experience and the influence of interactive VR on the tested spatial abilities. Fifteen studies reported the influence of spatial abilities (Table 4) and 11 considered the influence on spatial abilities (Table 5).

Most of these findings show an increase in participants' performance in a given task based on spatial abilities, with high-spatial-ability learners benefiting the most (Cherep et al., 2020; Kleanthous & Matsi, 2019; Sajjadi et al., 2021). Higher spatial-ability scores led to less motion sickness (Smyth et al., 2021) and less disorientation in participants when they were confronted with missing cues (Cherep et al., 2020) and better use of spatial strategies (Whitney, 2019). No linear relationships were observed between mental rotation and the perception of spatial properties (Lukačević et al., 2020), motivation (Obeid & Demirkan, 2020), or general post-test measurements (Jang et al., 2017). No significant correlation between the sense of direction and other measures was observed, either (Gehrke et al., 2018). One study reported the negative influence of low spatial ability with a large effect size (Wainman et al., 2021). Two studies reported a positive effect of interactive IVR on the performance of low-spatial-ability learners (Jang et al., 2017; Safadel & White, 2020). Two other studies described the supporting effects of IVR on low-spatial-ability learners who used overlapping haptic and visual information together (Barrett & Hegarty, 2016) and gaming experience in the case of interaction using a controller (Lages & Bowman, 2018).

Summarizing the effect of interactive VR on spatial abilities, we could observe several reports of a general increase in the spatial abilities of mental rotation (Brown et al., 2019; Chao & Chang, 2020; Molina-Carmona et al., 2018; Parsons et al., 2013) and perspective taking (Chang et al., 2017). In addition to the overall fostering of spatial abilities, two studies reported support, especially for low-spatial-ability learners (Chao & Chang, 2020; Seabra & Santos, 2013). One study reported no significant improvement in high-spatial-ability learners, as only low-spatial-ability learners showed significantly increased spatial abilities. Although they did not completely close the gap, they reached the pre-experiment level of high-spatial-ability learners (Sommer et al., 2021). Contrary to these findings, no strong evidence of an increase in spatial abilities was found after participants played Parong and Mayer (2020), a VR brain-training game, for 1.5 h.

When we focused on spatial-ability performance in VR, we found that performance in perspective-taking tasks was increased by cues that included interactivity and/or agency, but not directionality alone (Gunalp et al., 2019). The level of immersion impacted spatial learning (Chao & Chang, 2020), whereas a stressful environment led to decreased navigational skills in VR (Parsons et al., 2013). Although gender and the spatial-ability-test type influenced the time needed to fulfill spatial tasks, users' primary hand did not exert any influence, contrary to differences in pen-and-paper formats (Guzsvinecz et al., 2021).

4 DISCUSSION

The research characteristics revealed similarities between the interaction devices used in the studies and the interaction itself. Most authors explained which common tools were used, for example, a specific brand of HMD and its associated controllers. Two extremes can be identified. On one end, studies implemented new devices designed for a specific embodied task. Barrett and Hegarty (2016), for example, used their self-made controllers of the same size as virtual models, and Chang et al. (2017) used wooden blocks with leap sensors to facilitate direct interaction with virtual fences. On the other hand, some studies stated that they used Bluetooth controllers or VR controllers without providing further details. The type of system and controller used to conduct the study should be defined to clearly analyze the interaction and to compare and reproduce findings with the goal of facilitating a general understanding of the underlying processes. The same applies to the analysis of the descriptions of interactions. A few common trends and themes were observed in the characteristics of the environments offered. These themes can be used as a basis for the creation of frameworks and guidelines for the use of common features and for research and education. However, as in the description of interaction devices, some studies describe only the bare minimum of the possible interactions, complicating the reproduction and generalization of these settings.

Regarding the tests used, the most noticeable feature was the large number of studies that used only mental rotation tests to sort participants into low- or high-spatial ability learners. Although it is one of the spatial abilities on which researchers agree, this raises two questions. The first question concerns the validity of using only one factor or part (intrinsic-dynamic) of a multifaceted ability to determine the full extent of that ability. Would a broader approach for measuring several spatial abilities (such as Whitney (2019)) and distinguishing individuals based on different skill levels not lead to a better understanding of the connection between spatial abilities and interactions? Based on the different results obtained in these studies, there may be more underlying mechanisms than mental rotation ability.

The results on the influence of spatial ability on VR interaction show a clear advantage for high-spatial-ability learners. Low-spatial-ability learners may be hindered by the use of interactive VR. This supports the ability-as-enhancer hypothesis and is counterproductive if the goal is to support low-spatial-ability learners, as it might widen the gap instead of closing it (Lee et al., 2009). The secondary effects of spatial abilities on interactive VR can aggravate this problem. Considering Brown et al.'s (2019) report that learners stopped using VR glasses as they suffered from motion sickness and Smyth et al.'s (2021) finding on the reduced motion sickness with increased spatial abilities, we found that the threshold for low-spatial-ability learners might even increase.

Few studies have described the influence of VR on spatial ability. These studies reported a positive effect, especially for low-spatial-ability learners. A study that did not find significant changes argued that the cause might be the participants' short exposure to 1.5 h (Parong & Mayer, 2020). Considering the findings of Di and Zheng (2022) that describe an increased gain in spatial abilities in learning periods of more than 1 month, we found this to be a valid point. Di and Zheng (2022) reported significant improvements but also asked how these short-term developments could be transferred into long-term gains. Although we did not discuss it in detail, we found that some studies did not list exact durations; some conducted experiments in a relatively short period, sometimes taking only 15–20 min (Chang et al., 2017) or 2 weeks (Molina-Carmona et al., 2018; Sommer et al., 2021). This may affect the degree of change in learners' spatial abilities. Further, several factors complicate the incorporation of these findings into a more general context. Some groups were relatively small because of the limited number of devices and experimental design. Furthermore, as discussed by Jang et al. (2017)), the control group did not use VR, in most cases. This was understandable, as several studies did not specifically focus on the interaction. However, some studies focused on describing the effects of interactions using desktop VR in a control group. As the effect of different visualization devices and their interplay with spatial abilities is yet to be fully understood, to simultaneously change two variables—“interaction” and “visualization”—complicates the deriving of definite conclusions. The study by Lages and Bowman (2018) was one of a few studies that merely changed the interaction device (walking vs. controller), stayed in VR, and described how different levels of spatial abilities led to different performances of the interactive device.

5 CONCLUSION

Research on spatial abilities and interactions in stereoscopic VR is emerging, and some studies have already described possible implications, indicating the requirement for further research. As interactive VR benefits high-spatial-ability learners and probably hinders low-spatial-ability learners, its implementation should be accompanied by rigorous research on its underlying effects. This includes ways to support low-spatial-ability learners, lower the threshold for users, especially engineering students, and find the best fit between interaction and spatial abilities, as described by Lages and Bowman (2018)). An increase in the group size should also be considered, as some treatment groups consisted of only 20 subjects, complicating the generalization of the findings.

Research on the effects of VR on spatial ability has yielded promising results. Not only can researchers generally improve learners' spatial abilities but low-spatial-ability learners can benefit from interactive VR.

To fully support the implementation of interactive VR, more research is needed to understand how different spatial-ability characteristics lead to different performances. Further, the effects of different interaction devices on spatial abilities must be investigated. Another point is whether direct improvements can lead to the long-term development of spatial abilities. Ideally, interaction is considered not merely as a by-product of research focusing on visualization, but as the focus of research itself.

By doing so, not only could interactive VR be used to support learners as they proceeded in their learning but its misuse could also be avoided, which would otherwise widen the gap between learners with low and high spatial ability.