Renowned neurosurgeon Benjamin Warf of Boston Children's Hospital stands in the MIT.nano Immersion Lab. More than 3,000 miles away, his virtual avatar stands next to Matheus Vasconcelos in Brazil, as a resident performs delicate surgery on a doll-like model of a baby's brain.
Using virtual reality goggles, Vasconcelos can watch Warf's avatar demonstrate a brain surgery procedure before replicating the technology himself and questioning Warf's digital twin.
“It was almost an out-of-body experience,” says Warf, watching his avatar interact with residents. “Maybe this is what it feels like to have identical twins?”
This is the goal. Warf's digital twin bridged the distance, allowing it to functionally exist in two places at the same time. “This was my first training using this model, and it performed very well,” said Vasconcelos, a neurosurgery resident at Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil. “As a resident, I now feel more confident and comfortable applying these techniques to real patients under the guidance of my professors.”
Warf's avatar arrived via a new project launched by medical simulator and augmented reality (AR) company EDUCSIM. The company is part of the 2023 cohort of START.nano, MIT.nano's deep technology accelerator that provides early-stage startups with discounted access to the MIT.nano lab.
In March 2023, Giselle Coelho, scientific director of EDUCSIM and pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children's Hospital, began collaborating with technical staff at the MIT.nano Immersion Lab to create an avatar of Warf. By November, Avatar was training future surgeons like Vasconcelos.
“I had the idea of creating an avatar of Dr. Warf as a proof of concept and asked, ‘Where in the world would they be researching such technology?’” Coelho says: “Then I discovered MIT.nano.”
surgeon arrested
Coelho, a neurosurgery resident, was frustrated with the lack of practical training options for complex surgeries as he built his own model of the baby's brain. Her physical model includes all the structures of the brain and even bleeding, so she “simulates every step of the surgery, from the incision to the suturing of the skin,” she says.
She soon learned that simulators and virtual reality (VR) demonstrations reduced the learning curve for residents. Coelho launched EDUCSIM in 2017 to expand the diversity and scope of training for residents and specialists looking to learn new skills.
These techniques include a procedure to treat infantile hydrocephalus pioneered by Warf, director of neonatal and congenital neurosurgery at Boston Children's Hospital. Coelho learned the technique directly from Warf and thought his avatar could be a way for surgeons unable to travel to Boston to utilize his expertise.
Coelho collaborated with Talis Reks, an AR/VR/Gaming/Big Data IT technician at Immersion Lab, to create the avatars.
“A lot of the technology and hardware can be very expensive for startups to access when they begin their corporate journey,” Reks explains. “START.nano is one way to leverage and make affordable the tools and technologies we have in MIT.nano’s Immersion Lab.”
Coelho and her colleagues needed high-definition, high-resolution motion capture technology, volumetric video capture, and a variety of other VR/AR technologies to capture Warf's deft finger movements and facial expressions. Warf made several trips to MIT.nano to “capture” them digitally, including wearing special gloves and clothing embedded with sensors and working on real baby models.
“These technologies were primarily used for entertainment and VFX. [visual effects] or CGI [computer-generated imagery]“But this is a unique project because we are applying it to real-world medical practice and real-world learning,” says Reks.
Reks said one of the biggest challenges is what Coelho calls “holoportation,” a technology that transmits 3D volumetric video captures of Warf in real time over the Internet to help his avatar appear in transcontinental medical training. It was.
Warf avatars have synchronous and asynchronous modes. The training Vasconcelos received was in an asynchronous mode where residents could observe the avatar's demonstration and ask questions. The answers, available in multiple languages, come from AI algorithms based on previous research and a wide range of questions and answers provided by Warf.
In synchronous mode, Warf manipulates his avatar from a distance in real time, Coelho says. “He could move around the room, he could talk to me, he could give me directions. surprising.”
Coelho, Warf, Reks and other team members demonstrated their mode combinations during a second session in late December. The demo consisted of volumetric live video capture between the Immersion Lab and Brazil, spatialized and displayed in real time via an AR headset. This is a significant expansion over previous demonstrations that streamed volumetric data in only one direction through a two-dimensional display.
powerful influence
Warf has a long history of training desperately needed pediatric neurosurgeons around the world, most recently through the nonprofit organization Neurokids. He says remote and simulation training has become an increasingly part of larger training since the pandemic, but he doesn't think it will ever fully replace in-person hands-on training and collaboration.
“But I think if one day we could actually be like Giselle's Avatar and be able to show people from afar how to do things and answer their questions, there would be no travel costs or time costs or anything like that. It can be really powerful,” says Warf.
Coelho said the Avatar project is especially important for surgeons working in remote and underserved areas, such as Brazil's Amazon region. “This is a way to give them the same level of education they would receive elsewhere and the same opportunity to have contact with Dr. Warf.”
One baby recently treated for hydrocephalus at an Amazon clinic traveled 30 hours by boat for surgery, according to Coelho.
She says that training surgeons through avatars “could change the reality and future of these babies.”