Scientists in Moscow conducted an assessment of the diagnostic accuracy metrics for both Russian and International AI services
Moscow scientists have assessed the diagnostic accuracy of various artificial intelligence (AI) services for radiology. In a recent study, five AI algorithms were evaluated, including three developed in Russia and two from India and South Korea. The study revealed that the Russian AI service demonstrated the highest accuracy in detecting pulmonary nodules, outperforming its foreign counterparts. The results were published in the Chinese journal “Quantitative Imaging in Medicine and Surgery.”
Yuri Vasiliev, CEO of the Center for Diagnostics and Telemedicine of the Moscow City Health Department, highlighted the importance of this achievement. “In a recent study involving both Russian and international artificial intelligence (AI) services, our in-house AI solution demonstrated the highest accuracy in identifying lung nodules, outperforming similar solutions from India and South Korea. This achievement marks a significant advancement in our efforts to enhance medical care quality. Radiologists currently utilize over 50 AI services to interpret medical image studies, and more than 13 million studies in Moscow have been analyzed using neural networks. The growing market for AI services is continuously introducing solutions that streamline the workload of healthcare professionals while maintaining high standards of performance.” said Vasiliev.
“We aim to provide physicians with tools that not only facilitate their work but also enhance the overall quality of medical care. To achieve this, we have developed a maturity matrix—a comprehensive tool designed to assess and benchmark the performance of various AI services. Since the end of 2022, our AI service has consistently ranked as a leader in the field of chest organ radiography,” as stated by Yuri Vasilev.
The AI service, designed for the automatic analysis of chest X-rays, currently identifies 14 signs of various pathologies. Additionally, it calculates the cardiothoracic ratio and generates a comprehensive radiology report.
“We prepared a unique dataset consisting of 100 X-ray studies, including 50 with confirmed pulmonary nodules and 50 without pulmonary nodules. Of those, 25 cases involved nodules that were initially doubted by radiologists, but confirmed by CT scans. However, the presence of pulmonary nodules in all 50 studies was confirmed by CT scans,” explained Kirill Arzamasov, Head of the Department of Health Informatics, Radiomics, and Radiogenomics at the Center for Diagnostics and Telemedicine.
“The evaluation of AI services was conducted in three stages, with results at each stage compared against a reference standard confirmed by CT studies. The findings demonstrated that the Russian AI service outperformed its international counterparts across all diagnostic accuracy metrics. The dataset is publicly accessible on the website, allowing developers to independently assess the quality of AI services,” said Kirill Arzamasov, Head of the Department of Health Informatics, Radiomics, and Radiogenomics at the Center for Diagnostics and Telemedicine, Moscow Healthcare Department.”
This study is part of a broader experiment launched in 2020 to integrate computer vision technologies into Moscow’s healthcare system. Supported by the Moscow Social Development Complex and the Department of Information Technologies, the project is designed to foster innovation in AI, particularly by supporting developers.
The Diagnostics and Telemedicine Centre, established in 1996, plays a leading role in introducing AI technology to medicine in Russia. It focuses on advancing AI in medicine, developing diagnostic imaging, management of medical departments, conducting research, and the training of healthcare professionals.