Mixed reality in the operating room
Published on 07/02/2022 in Innovate
In the Imelda Hospital in Bonheiden, Dr. Tim Tollens performed a complex incisional hernia operation, supported by 3D visualization. The surgeon used the HoloLens headset by Microsoft and an application designed by Mr. Watts.
IT in the care sector
Tim Tollens is the head of surgery at the Imelda Hospital. He is himself a specialist in abdominal surgery and, more specifically, complex incisional hernia surgery. “It is true that the development of IT is visible in our sector but, in my opinion, it is going too slowly,” he notes. “Corona may provide an acceleration in that area. The patient is now much more familiar with video consultations and remote advice.” The doctor himself is particularly interested in the use of IT in his profession and is always looking for new applications. That led to a Flemish first.
Livestream with smartglasses
“Through the medical technology company Becton Dickinson, I came into contact with the HoloLens, Microsoft's mixed reality headset,” he says. “In that context, I also became acquainted with Mr. Watts, a Belgian company that develops applications for the HoloLens.” Dr. Tollens already had experience filming surgical procedures for training and research.
With the HoloLens, a camera was added, which allows the others to follow the procedure through the surgeon's eyes, as it were. Dr. Tollens performed a complex incisional hernia operation with the smartglasses on. Doctors followed the live stream from 280 locations in 49 countries.
Via the HoloLens, the surgeon at the operating table has access to all kinds of information and medical images.
Dr. Tim Tollens
Projection on the patient
“The real added value is in the application Mr. Watts developed,” continues Dr. Tollens. “It creates a virtual space that you can access via the HoloLens. This way, as a surgeon, you can open the patient's medical file in that virtual space.” The app is controlled by voice, gestures or eye movements. “I don't have to leave the operating table for that and I don't have to take off my gloves to check something on the PC. Via the HoloLens, all information is immediately available in a safe manner at the operating table.”
The biggest advantage, however, lies in the use of mixed reality (MR). The application displays medical images – for example, from a CT scan – via the HoloLens and projects them onto the patient. In this way, the surgeon can see in the MR image where the exact location of the problem is in the patient. It allows the surgeon to determine the location of the procedure in a very specific way, without having to be invasive. “That technique – 3D visualization and superimposition – is really the future as far as I'm concerned,” says Dr. Tollens.
The HoloLens is also used in other sectors. Construction company Hooyberghs has already deployed the HoloLens application of Mr. Watts to visualize 3D models on site. With the smartglasses there is a better insight into the plans, because they are projected on the location itself, with the site as a kind of 3D canvas.
Advantage in the operating room
The surgery that the doctor performed, supported by the HoloLens and Mr. Watts, seemed to have some immediate and important practical advantages. “In the operating theater there is no longer a need for expensive video screens with all kinds of complex cabling,” he says.
“The images the surgeon needs appear on his smartglasses. That not only makes it more convenient for the entire team's work, but also saves a lot of money when setting up an operating room.”
Created for 5G
Dr. Tollens looks back enthusiastically on his first experience with the headset and MR application, although there is still room for improvement. “Rendering 3D structures and positioning them in a correctly calibrated way remains a challenge. But still, it’s only a matter of time before that is further refined.” When using the HoloLens, large volumes of data move between the smartglasses and the application running in the cloud. “It’s the kind of application that 5G is made for.” Every time the doctor moves his head, the application adjusts the image in real time, without delay.
It is not the only application where 5G will make a difference in the future. “I am looking with great interest at Microsoft Mesh, which will enable remote holographic collaboration,” says Dr. Tollens. “In the hospital, we want to focus on a holographic training and consultation platform, among other things.” There is every indication that the use of holographic applications – such as VR and MR – will grow rapidly, also in the medical sector, thanks to 5G.
Do you have plans with 5G? Proximus has launched the 5G innovation platform where you can test your 5G projects in practice.
“The HoloLens offers numerous possibilities,” says Dr. Tollens. “We are only at the beginning of this technological revolution. More applications will appear soon. It is a development that takes medicine to a higher level and offers opportunities to the new generation of doctors.”
Dr. Tim Tollens is a general and abdominal surgeon specializing in colorectal surgery and incisional hernia surgery. In addition to coordinating internship as a supervisor and trainer, Dr. Tollens is also head of surgery at the Imelda Hospital in Bonheiden.