The Stanford Virtual Heart is transforming how patients and medical trainees learn about congenital heart defects. Its co-creator Dr. David Axelrod explains how virtual reality is bringing medical education to life.

With an immense knowledge base and content in three (and often four) dimensions, medicine and science lend themselves extremely well to the developing fields of augmented and virtual reality. A stunning array of Extended Reality (XR) applications has been released since the mainstream acceptance of VR and AR hardware; from clinical training programs that teach surgical skills to scientific explorations of the natural world. Our approach to VR development at Lighthaus Inc, highlights a few key features of this growing industry.

Virtual Reality for Training and Education: The Stanford Virtual Heart

Initially created for a Continuing Medical Education (CME) symposium, The Stanford Virtual Heart represents a collaboration between The Betty Irene Moore Children’s Heart Center at Stanford and Lighthaus (funded in part by OculusVR, Menlo Park, CA).

We use VR to provide an immersive and interactive congenital heart experience in various educational settings, and the unique VR visualizations are matched with four-dimensional “spatial audio” created by VR audio engineers (Delta Soundworks). Medical students, pediatric residents, and advanced pediatric trainees (fellows in cardiology/critical care) have all been the focus of clinical VR training programs and medical education research studies as well. In 2018 we formed The Stanford Virtual Heart Pilot Program, a consortium of which has now grown to 35 centers across the world using the application for training and clinical care. These participating centers collaborate on research studies, provide vital feedback for ongoing development, and often use the program in novel and unanticipated ways (e.g. art museum installation).

In 2019, with increasing use of the program, we developed significant upgrades (‘version 2.0’) including: a simplified user interface, accurate realistic blood flows (color coded to indicate intracardiac shunts), and addition of an ‘audio-player guide’ requested by a Pilot Program engaging in educational research. Our collaborations have branched beyond strictly clinical or educational medicine; we recently joined forces with Dr. Alison Marsden, a leader in bioengineering and creator of The Simvascular flow modelling application. With grant funding in process, we are currently implementing a VR based flow modeling program for use in presurgical planning.

Using VR for Patient Care: Experiencing Complex Surgical Procedures, Improving Informed Consent, and Highlighting Unique Clinical Programs

VR offers tremendous potential to educate patients and the lay public, as evidenced by our collaboration with The Johnson Center for MaternalFetal Medicine at Stanford. The Stanford Fetal Therapy VR experience provides an immersive and interactive experience to explain novel fetoscopic surgical procedures.

These techniques treat complex fetal disorders – specifically twintwin transfusion disorder and spina bifida – that require a multidisciplinary medical/surgical team; the VR experience provides an additional tool for patient understanding of these intricate procedures. In addition, demonstration of the VR experience at national conferences has improved programmatic visibility; a formal assessment of referral patterns remains to be performed.

VR Software Development to Create Instructional Materials

In addition to creation of standalone VR applications, we have also included VR content in undergraduate educational curricula with Stanford University professors. Dr. Anne Friedlander, professor of Human Biology, has previously used online videos to enhance her digital curriculum and teach physiology.

In 2018, Lighthaus leveraged the existing Stanford Virtual Heart software to create a virtual representation of Dr. Friedlander’s curriculum on exercise and aging. Complete with virtual depictions of coronary artery plaque rupture, this educational tool was successfully implemented in the physiology course and remains a core aspect of the curriculum. Although the experience is not yet provided for students to experience in VR, this example illustrates the power of VR software development tools in online learning. Future developments may also include multi-user VR learning with this online educational experience.

VR-Cardiopulmonary Resuscitation: Hardware Innovation and Ongoing Development

A number of promising XR applications have targeted improved medical simulation, with cardiopulmonary resuscitation (CPR) offering a natural environment for VR development. Due to the high-stakes, low-frequency nature of CPR events, in vivo training is impractical and costly; students train (and are certified) based on performance on a CPR mannequin.

Our approach to VR-CPR integrates the ‘off the shelf’ mannequin with an optical ‘time of flight’ sensor and hand tracking gloves. The student therefore becomes immersed in the virtual world and responds to the VR representation of cardiac arrest, while performing compressions on the real-life mannequin that is physically linked to the virtual patient. This hybridized simulation provides real time feedback on the
quality of CPR (compression depth and rate, hand positioning, and chest recoil).

Students initially train on the VRCPR module, created with support from the American Heart Association, and subsequently perform an in-VR examination. The initial pilot of more than 120 U.S. high-school students was presented at the 2019 American Heart Association Resuscitation Science Sessions in Philadelphia.

VR Cellular Biology and the Sciences

Lifecraft VR represents our initial experience with VR science training. This immersive VR journey into plant, bacterial, and human (neuronal) cells requires students to perform cellular processes such as producing energy (ATP) from a pyruvate molecule.

Initially created for the U.S. Department of Education’s 2017 ‘EdSim Challenge, Lifecraft provided an exciting environment to test VR education before VR hardware was widely distributed to the public. With the interval development of more efficient and less costly head mounted displays, the virtual cellular processes created for Lifecraft continue to inform our development of educational VR programs for the sciences.

Color Space: A Clinical Tool with (Unexpected) Commercial Success

In 2018, Lighthaus began production of a VR coloring book application for patients receiving chemotherapy. ‘Color Space’ was initially conceived to provide a relaxing and engaging experience while maintaining simplicity and requiring minimal effort. The specific patient needs — low complexity, intuitive user interface, and calming content informed the VR software engineering.

Specifically, users are guided through a simple tutorial and then use only one button to control the application of color to the scene (analogous to ‘paint by numbers‘). This clinical tool was piloted at the sponsoring institution’s chemotherapy clinic and a clinical trial is currently in design. Anecdotally, the initial feedback revealed that the VR coloring book can reduce anxiety and shorten the perceived duration of chemotherapy administration.

Lighthaus then expanded the audience to the general public in 2020. While the release of Color Space (on the Oculus Store) had been planned for months, it became available in March 2020 coincident with the U.S. peak of the coronavirus pandemic. The release of this calming VR experience (during a period of unprecedented anxiety) provided an unexpected in vivo observational study; user reviews and comments confirmed the powerful potential of a simple VR coloring book application.

Interestingly, ongoing development must maintain the single-button user interface while also addressing suggestions from non-medical users. The first iteration, for example, included a markedly increased color palate – a consistent suggestion from VR users ‘sheltered in place.’

Future Directions

The examples detailed above reflect a broad sample of our healthcare and science VR work at Lighthaus. By no means does this represent a comprehensive description of the existing XR applications; many programs have established exciting VR and AR experiences for clinical, educational, and research purposes. As hardware becomes more readily available, it seems likely that students will ultimately demand these immersive and interactive tools. Ultimately, technological developments may render the choice between ‘AR versus VR’ moot and advances in medical XR are sure to continue the trend of immersive visualization and interaction.

Dr. David Axelrod is co-creator of The Stanford Virtual Heart, the Lead Medical Advisor at Lighthaus Inc. and a pediatric cardiologist at Stanford University’s School of Medicine. The work described from Lighthaus Inc. has been supported by The Betty Irene Moore Children’s Heart Center at Stanford and OculusVR (a Facebook subsidiary, Menlo Park, CA).