La Trobe University has used CT scans of a coronavirus-infected patient’s lungs to build a virtual 3D model of the infected organs
La Trobe University has used CT scans of a coronavirus-infected patient’s lungs to build a virtual 3D model of the infected organs, providing unique insight of the disease viewed through a HoloLens 2 mixed reality (MR) headset.
Believed to be a world first, the visualisations enable enhanced exploration inside the patient’s lungs, allowing the user to have a closer interaction with particular areas of interest.
The system has been developed over eight weeks by a team led by Professor Henry Duh, La Trobe’s Head of Computer Science and Information Technology.
Professor Duh said 3D modelling can show the transformation from healthy lungs to those infected by coronavirus – a sequence which is hard to pick up with 2D biomedical images – and provide a better understanding of how the virus invades the body.
“Medical imaging is indispensable to the delivery of healthcare and an essential component of clinical research. However, there is currently a lack of appropriate software to assimilate and derive maximum benefit from multimodal interactions with medical images in mixed reality environments.”
La Trobe University seeks to overcome that by building immersive medical tools using AR – the ability to superimpose 3D graphics over real-world views seen through a camera lens.
Leveraging Microsoft resources including Windows10 and HoloLens2, the University hopes the technology will one day be adopted in clinical diagnosis.
The team is now working toward using AI to automatically segment the lungs – or indeed any organs of interest – and render 3D scans from 2D medical images, to provide a deeper analysis of what it reveals.
Tiffany Wright, education director for Microsoft Australia said: “It is an absolute privilege for Microsoft to be able to support the work of pioneering technologists like Professor Duh and his team at La Trobe.
“AI and the HoloLens2 are combined here to provide insights that would be hard to grasp in any other way – and the speed at which La Trobe was able to spin up this application is extremely impressive. Starting in 2021 La Trobe is going to offer a new course – a Masters in Digital Media which will explore issues such as user experience, interaction design and virtual and mixed reality environments and this project will serve as an important beacon for new students and researchers alike.”
According to Professor Duh: “The ultimate goal of our project is to visualise the infected lungs and at the same time link to the outcomes of AI analysis. Clinicians will be able to see the analytics and interact with the model to gain insights from the system.”