King’s College London Study Suggests Machine Learning Currently Ineffective For Detecting Brain Aneurysms

First systematic review and meta-analysis looking at feasability of machine learning algorithms to detect brain aneurysms

Researchers from the School of Biomedical Engineering & Imaging Sciences have published a systematic review and meta-analysis showing that the quality of evidence is poor from studies using machine learning algorithms to detect brain aneurysms, indicating that if used at all, then assistance would be best in the form as a second reader, such as with computer assisted diagnosis (CAD). Their findings were published in Journal of Neurointerventional Surgery.

Aneurysm detection using machine learning has been described as a primary focus in the field of neurointervention, but there has been no comprehensive systematic review or meta-analysis of relevant studies to assess their suitability for clinical use.

The quality of evidence is poor for studies using machine learning algorithms to detect aneurysms. The studies analysed were largely problematic as they appear to be biased and there were concerns about whether these can be used in the clinic.

These concerns related to flawed methods regarding how patients were selected in the studies. Additionally, many studies did not show whether the algorithms would work in hospitals outside of where the study was performed.

The systematic review shows that assistance would best be in the form as a second reader – like with the computer assisted diagnosis (CAD) that has been used as a second reader for mammography for decades, albeit with variable success.

However, it remains unproven whether there is clinical benefit in brain aneurysm detection given the high false-positive rate – in other words lots of flagged aneurysms would need to be reviewed and then ignored as they are not aneurysms.

Aneurysms are blisters arising from arteries. In the head, a bleed from an aneurysm (known as a subarachnoid haemorrhage) yields a poor prognosis, with a mortality rate of nearly 50 percent.

Aneurysms are common with an estimated prevalence of 3 percent in the general population.

The detection of brain aneurysms is required in two scenarios. One is following a subarachnoid haemorrhage where the ruptured aneurysm needs to be detected to allow treatment.

Another is when an unruptured aneurysm is an incidental finding—for example, during vascular imaging which routinely takes place following a stroke.

Screening may also occur in high-risk populations. Early aneurysm identification, aided by accurate automated systems, may improve patient outcomes.

In current routine clinical practice for detection of cerebral aneurysms, cross-sectional angiography (i.e. using a CT or MRI scan with vascular sequences) is an acceptable first-line diagnostic biomarker for patients with a spontaneous subarachnoid haemorrhage because it is non-invasive with good accuracy.

In almost any case of spontaneous subarachnoid haemorrhage, definitive digital subtraction angiography is also mandated as it is the most accurate imaging modality.

Cross-sectional angiography is also used as vascular imaging for screening in populations who have a high-risk of aneurysm development, and also for patients following a stroke to look for clots – when aneurysm might be discovered incidentally.

But the relentless increase in medical imaging in the clinic is outpacing the recruitment of new radiologists.

Theoretically, AI might be used to replace or assist radiologists in tasks such as the detection of aneurysms during cross-sectional angiography or digital subtraction angiography.

There have been a couple of interesting review articles but none have systematically applied systematic review methodology used for diagnostic accuracy studies. This means that important conclusions have been missed, such as the high false-positive rate causing each examination to produce several aneurysm candidates requiring review. A revealing example is that in a model with 97 per cent sensitivity (pick up rate), for every true aneurysm detected, 14 highlighted regions have to be examined by the radiologist where there is actually no aneurysm. The implications for workflow are considerable.
– Dr Tom Booth, Reader in Neuroimaging at the School of Biomedical Engineering & Imaging Sciences, King’s College London
Additionally, the previous review articles confine themselves to deep learning only, however, there are some important machine learning studies that have not used deep learning which need to be analysed for a comprehensive review.

Further studies where AI CAD tools are trained on a large and representative dataset and evaluated on a prospective multicenter cohort are needed to clinically validate the efficacy of these tools. If this advice is taken up by the research community, the positive new developments in machine learning algorithms that are nearly ready for clinical adoption, will become enablers of better clinical care.