Design and Development of Mixed Reality (MR) Laboratory Tools to Improve Spatial Cognition, Student Engagement, and Employee Safety

Introduction

VR has recently developed as a prevalent technique adopted in numerous fields of scientific experiments. The thriving concept of simulating an interaction environment broadens the researchers’ competence to design the ideal virtual experimental conditions and sufficiently manipulate the environment’s layouts. 

VR is a broad term incorporating a wide range of computer simulations [1] that provides an immersive experience to the user via a head-mounted display (HMD). Although the VR technology varies, the common purpose of VR is to enable a human-computer interface that simulates an alternate three-dimensional (3D) environment and augments audio-visual graphics, referred to as AR, in this study. In addition, AR provides on-demand data, thus facilitating and enriching the employees’ work [2] in real-time applications. This leads to redesigning the training material for the manufacturing and safety industry [1] and is expected to become more flexible. 

The MR, on the other hand, combines both VR and AR technologies and merges real and virtual worlds, where physical and digital objects interact in real-time to provide enhanced 3D information to the users [3,4]. This enables the users to visualize the immersive virtual world and the real world and interact with those objects. A study by Richter, et al. in 2022 discussed the challenges of video materials in education,  

especially during the Covid-19 pandemic, and proposed VR simulations for immersive experiences for  

teachers and students (Richter, et al., 2022). Additionally, the authors argued that VR makes it easy to generate materials from a first-person perspective as opposed to a third-person perspective. However, a systematic literature review [5] discussed the most frequent adverse events after a VR exposition could be disorientation and nausea. Therefore, this study focuses on developing AR/VR/MR simulation tools to limit extended exposure in the virtual environment. These simulated endeavors aim to generate simulated trainers, limiting the simulation use time to 20 minutes or less; or activities to develop muscle memory for students in STEM fields and create potential AR applications for workers frequently exposed to hazardous materials. 

This study discusses six case studies with custom-developed AR, VR, and MR tools to improve spatial cognition, student engagement, and employee safety and provide services to the COVID-19-impacted communities as part of relief efforts. This study aims to broaden the emerging technologies and generate discussion on the potential positive outcomes that these novel technologies may offer to STEM fields. The researchers discuss six case studies supporting the conversation with the discovered outcomes from each AR, VR, or MR-related activity. The researchers use IBM’s Statistical Package for Social Sciences (SPSS) statistical analysis tool to conduct comparative analyses using t-Tests for dependent and independent groups, multivariate analyses, and Tukey’s post-hoc tests [6]. 

Methods and Results 

Case Study 1

This study involved the introduction of an AR tool in primary school students. It investigated the effectiveness of an AR tool for promoting engaged learning in technological literacy for STEM education [7]. Forty-two children (25 female and 17 male) from three different first-grade classes with diverse socio-ethnic backgrounds participated in the study, as seen in Table 1. One of the classes experienced a conventional teaching method by their class teacher, the second class experienced the integration of the AR tool in their learning, and the third class participated in both modes. This way, the researchers achieved two groups for study and attempted to attain homogenous participation with limited interruptions of the outcomes. 

Table 1. Frequency distribution of Participant Gender ^a 

The research team separated the participating students into two groups, a control group (Standard=24) and an experimental group (Experimental AR=18), to explore the developed AR tool’s effectiveness in students’ learning and engagement, as shown in Figure 1. Both male and female students participated in the standard and experimental groups in a randomized fashion. 

An independent sample t-Test is conducted on the response of the students that participated in the study. The control and experimental groups have a total of 24 and 18 participants, respectively as shown in Table 1. The descriptive statistics table (Table 2) illustrates that the mean value for the Experimental AR group increased by 4.736 points, where the mean value for the standard group was Mean 1 = 11.875 and the mean value for the Experimental AR group was Mean 2 = 16.11. 

Table 2. Descriptive Statistics for Participating Groups (Standard & Experimental AR) 

The independent sample t-Test (Table 3) showed a significant difference between the two groups, p = 0.001< p = 0.05 alpha level. Therefore, the student’s involvement and participation in academic learning with the integration of the AR tool improved significantly. 

Table 3. Independent Sample t-Test Analysis for the Standard and Experimental AR Groups 

Student learning success is highly dependent on the availability and literacy of experiential technological tools provided to them. The researchers introduced the first-grade teachers to the AR tool, as seen in Figure 2, used in this study. Once the teachers became familiar with the application of the AR tool, they provided demonstrative education to their classes. Overall, the investigators observed that cognitive reasoning [4] and interest in learning improved with the integrating of AR tools in the classrooms. Although the first-grade teachers commented that uncertainties of technological devices might hinder technological literacy in students, the AR and VR tools could benefit the student’s retention of the newly introduced material at an early age. 

Case Study 2 

The effects of disaster training with and without the Augmented Reality Mobile (ARM) tool, as seen in Figure 3, were explored to study the experimental design. The study subjects, Undergraduate Engineering Technology students from first-year to senior year, participated in the disaster response activity by designing and constructing a Mobile Renewable Response Trailer (MRRT). 

None of the participants previously had any operation in emergency response vehicles; therefore, the control and treatment groups were randomly selected to participate in the study. Two participating control groups (CG1 & CG2) were given a scenario where they needed to operate the MRRT from a non-operational stage to assist the disaster victims and provide power without any assistance from the trained first responders. In contrast, the treatment group (TG) received ARM to repeat the same tasks on the spot. The researchers compared the timeliness of the equipment setup between the three groups, where CG1 was asked to set up the trailer equipment, CG2 was asked to set up the trailer equipment with a step-by-step instructional operation manual, and the TG was asked to set up the trailer using the ARM application on a tablet or phone [4,8]. 

Case Study 3 

The ability to visualize 3D objects and their orthogonal views and manipulate those images is a cognitive skill that is vital to many STEM fields, especially those requiring work with computer-aided design (CAD) tools [9]. In addition, research suggests that well-developed spatial skills of this type are critical to successfully advancing in engineering and many other fields [10]. These spatial skills involve visualizing 3D objects and perceiving their different orthogonal viewpoints if they were rotated in space. 

A group of CAD students (freshmen and sophomores) participated in this study. A total number of thirty-five (4 female and 31 male) students completed a spatial orientation test. The spatial orientation test was used to explore students’ spatial cognition and was comprised of 10 questions in which the participants must select the correct orientation of a given part. In addition, in each class, instructors provided a 3D representation of a part of the virtual environment for students to analyze before class, as seen in Figure 4. 

Paired sample t-Test resulted in the students finding the correct orientation after they analyzed the part (Pretest Mean = 1.42 < Posttest Mean = 8.58) within the virtual environment, as seen in Table 4. 

Table 4. Paired Sample t-Test Descriptives for Freshmen and Sophomore Engineering Students 

Additionally, SPSS t-Test results indicate a significant difference (p value=0.00 < alpha level 0.05) between the test results when the students were exposed to the virtual analysis of the 3D part model with a 95% confidence interval, as reported in Table 5. 

Table 5. Paired Sample t-Test Mean Comparison to Compare the Influence of a VR 

Although the t-Test analysis showed significant differences between the pretest and posttest average means of the scores, the correlation table indicates a slight correlation between the two scores, as seen in Table 6. 

Table 6. Correlation between the Pretest and Posttest outcomes for Spatial Orientation 

Moreover, the researchers needed to investigate more to see if the involvement of a VR tool in the 3D modeling class would improve the spatial orientation skills of freshman and sophomore engineering students, as seen in Table 7. 

Table 7. Paired Sample t-Test Descriptives for Freshmen and Sophomore Engineering Students 

The descriptive statistics in Table 8 show that students’ post-test mean values were slightly higher than the pretest, meaning that the interjection of VR into the model analysis improved students’ spatial visualizations. 

Table 8. Paired Sample t-Test Analysis for the Influence of a VR in Fall 2017 Semester 

The paired sample t-Test between the mean values at the beginning of the semester, with and without the introduction of a VR environment to analyze virtual parts, indicated a significant difference (p value=0.00 < 0.05 alpha level) between the test scores. This result shows that the students understand the structures of the 3D model when they inspect it in the virtual environment. It should also be noted that the descriptive statistics produced a higher mean value for the posttest boxplot as well as one extreme outlier for the posttest for the spring 2017 data points. The reason for this outlier was calculated inaccurately since SPSS multiplied by 1.5 IQR [11]. Therefore, the researchers decided to review the data point and concluded the histogram of the data was normal for the posttest data. The data score of 6 was not an outlier. Therefore, the researchers believe that restructuring an introductory CAD course could help instructors engage and motivate students and train skilled drafters/modelers with less effort. 

Case Study 4 

The AR_GHS (Augmented Reality Globally Harmonized System) safety tool was developed for employee awareness and training on OSHA-determined nine pictograms, as seen in Figure 5 [8]. Employee awareness of pictograms requires workers to retrieve and understand potential hazards in proximity (OSHA) as they perform janitorial tasks on the premises. The assembly of OSHA determined nine pictograms to be selected since these pictograms are internationalized and exist in almost all manufacturing companies [12]. 

A total of 21 untrained janitorial and carpentry workers (age 18-45, male=11 and female=10) participated in the study from two countries (Nigeria and Kyrgyzstan). Two independent groups were randomly recruited to participate in the study. The Globally Harmonized System (GHS) pictograms were printed and posted on the wall. The first group was invited to observe and study the pictograms and fill out a brief survey prepared by the researchers. The room where the participants conducted their experimental study was an empty room with a regular table and a chair. A similar setting was practiced in both countries. The second  group of participants, who were never exposed to the pictograms, were invited to the room with the pictograms. 

Based on the Shapiro-Wilk test, the normality of data frequency was violated for pre- (p = 0.01) and post- (p < 0.01) data. Therefore, a non-parametric statistical test (Wilcoxon test) was used to analyze the data. The Wilcoxon test results show that the participants’ scores significantly increased after using the AR app (p < 0.01). However, the independent test shows no significant influence of gender differences (p = 0.10) or geographical location (p = 0.15). 

The preliminary study results show that the innovative AR application can enhance the understanding of the GHS pictograms for non-English speaking employees, contributing to their safety awareness of hazardous materials. However, further data collection is necessary to validate these critical discoveries and study additional variables that AR technology can introduce to improve safety. 

The MR-Lab application enables students to visualize the isometric product from its orthogonal views and provides short tutorial clips of how a specific feature was developed and what tools were used [13]. The MR applications have become crucial laboratory pedagogy for STEM fields [14] as conventional lab courses require dedicated equipment. In this regard, the MR applications can provide virtual labs for situational learning and involve the participants with learning tasks. In addition, the students can perform basic modifications on the 3D part in the ARCADE, such as section views, details views, scale, rotation, and explode assembly views. Although this project is a work in progress, the initial pretest and posttest results show a significant improvement in students’ spatial cognition when the proposed tool is used to assist the course. 

Case Study 5

Firefighters are on the front lines, protecting lives and creating emergency preparedness to ensure our homeland is safe, secure, and resilient against different hazards (Dakeev, et al., 2020). Firefighters respond to a wide range of unexpected incidents, including medical calls, motor vehicle accidents, fires, technical rescues, explosions, hazardous material incidents, terrorism, mass casualty incidents, and anything else involving a call to 911. 

A total of 7 subjects participated from student population to be exposed to three VR environments (Figure 6) with three levels of mental challenge activities for each environment: 1- Fighting Fire, 2- Rescue, and 3- Escape the Scene to investigate firefighters’ physical behavior represent by the following physiological responses: heart rate, body posture, as well as the respiratory rate during VR simulation. Participants with a history of motion sickness, claustrophobia, anxiety issues, neck issues, elevated blood pressure, or epilepsy were ineligible for participation. 

The virtual environment contained three burning facilities with low to high-intensity levels of fire. The researchers visited the local fire station to receive feedback and consultancy on the fire scene development to make the scenes as realistic as possible. The received feedback and suggestions were incorporated into the scenes, where the environments gradually get darker as time progresses within the scene. The researchers initially started with a real burning room fire scene and incorporated it into the VR simulation; however, the rarity of training facilities, resources, and the equipment’s heat resistance factors, steered the VR simulation to be developed in animation and game engines. In contrast, the number of VR headsets is increasing [15] as more industries find positive impacts on their business activities. 

A bio harness monitor from BioHarnessTM III, Zephyr Technology Corporation, Annapolis, MD was used to measure physiological responses. The bio harness was strapped to the chest of each participant before starting the experimental session of virtual simulation and was worn until the end of the session. 

A total of five students (age: 20.2 ± 1.4, BMI: 25 ± 1.4) participated, from the untrained firefighters’ population, in the fire-simulating VR project. Figure 8 shows the heart rate comparison between the fire intensity levels for sixty readings from the bio harness. 

The researchers conducted a One-way Analysis of Variance (ANOVA) in SPSS to compare outcomes from each scene (Cabin, Morgue, and Old House) and to analyze whether there is any significant difference between the scene fire intensity levels’ impact on the participants in the virtual environment. The descriptive statistics show that the means were relatively similar for Morgue =82.867 and Old House=82.267, where the Old House scored 0.6 lower average than the Morgue scene, and both greater than the Cabin mean value of 76.371, as shown in Table 9. 

Table 9. Paired Sample t-Test Analysis for the Influence of a VR in Fall 2017 Semester 

One-way ANOVA Table 10 shows a significant difference between the fire scenes at p=0.024< 0.05=alpha level from 95 total observations, as seen in Table 10. 

Table 10. One-way ANOVA report for three intensity-level fire scenes in VR 

To further investigate how the virtual fire scenes impacted participants’ physiological conditions, further Tukey post hoc analysis is reported in Table 11. 

Table 11. Paired Sample t-Test Analysis for the Influence of a VR in Fall 2017 Semester 

Table 11 shows that the mean difference is significant at 0.05 alpha level, which indicates that the participants were highly impacted by the Morgue = 6.495 vs Cabin most, followed by the Old House = 5.895 vs. Cabin, and the Cabin impacting the least among the three levels of fire scenes. 

Of the three variables: 1-heart rate, 2-respiratory rate, and 3-posture from the participants, the latter two did not yield reportable data due to the limited number of participants. However, the researchers discovered that the intensity of VR fire simulation fire scenes, with variable intensity levels of fire, resulted in a significant heart rate elevation between scenes. A negative reaction of untrained/first-timer firefighters may lead to delays in decision-making during critical times and may lead to injuries during duties. 

Case Study 6 

The researchers developed an Engineering Building floor plan and an immersive simulation to investigate the effectiveness of safety escape zones as seen in Figure 9. The participants with no prior experience with either the VR simulations or the building itself participated in visualizing the VR environment to locate the exit signs. Similarly, an independent group of participants located the physical exit signs on the experimental floor. 

Independent sample t-Test analysis showed that (p-value 0.03 < p=0.05 for 15 participants) the VR simulation provides a significantly shorter time to locate the exit signs, which could be a matter of life and death. 

Discussion 

The authors of this study discussed six case studies from their previous findings and publications. Each case study presents generalized outcomes, from kindergarten children to adults at workplaces, to investigate the effectiveness of VR and AR applications to a specific group of participants in this study. Case study 1 started with the provided AR tool’s positive impact on kindergarten student learning and engagement compared to the conventional way of teaching styles. In case study two, the college students developed an AR tool for an existing community resilience trailer to support community engagement. Case study three discussed and provided outcomes on the impact of AR tools to support student engagement and spatial cognition improvement in early CAD students in colleges. Case study four examined additional custom-developed AR tools for industry workers, specifically those who do not receive formal safety training on hazardous materials and their labels to prevent major workplace safety accidents. These four AR-related case studies showed that supplementary AR applications can significantly impact various persons from diverse groups and ages in engaging with the content material, learning about safety hazards, retaining information, and staying motivated with the new knowledge they are acquiring. Cases five and six focused on VR applications, where case study five discussed physiological changes in new firefighter trainees when they are exposed to different intensity levels of fire, and case study six discussed muscle memory  

they are exposed to different intensity levels of fire, and case study six discussed muscle memory development in finding safety exits from unknown premises. Both VR case studies showed significant improvement when a VR tool is used as a simulator. 

Conclusion 

Generation z is an electronic tycoon due to the abundance of computerized devices available. Therefore, participants who had previously played role-playing games were more enthusiastic about trying VR. Researchers needed to develop measures to deal with motion sickness that the participants would experience during the immersive experiences in VR. VR time was limited to a maximum of 20 minutes as part of those measures. Furthermore, the researchers used smooth locomotion control with a thumb stick instead of teleportation. Motion sickness may be further reduced by using omni pods to enable physical movement. 

Further, this study examined the effects of AR applications on student learning, engagement, motivation, and spatial cognition in CAD courses. The paired sample t-tests and independent t-tests showed that AR tools significantly improved understanding of 2D blueprints and spatial cognition skills, regardless of previous 3D modeling experience. A t-test of independent samples showed that students could comprehend 3D models significantly better when using the AR tool. Engaging students in the MR learning process and motivating them can improve their engagement and enthusiasm toward learning. 

Acknowledgments. The authors would like to thank the participants of all the research projects discussed in this paper. 

Disclosures. The authors declare no conflicts of interest.