Siberian Federal University: SibFU Student Has Taught Artificial Intelligence to Understand Sign Language

Vladislava Zhukovskaya, a student of the School of Space and Information Technology at Siberian Federal University, has developed a method for recognizing Russian sign language gestures from video sequences, taking into account the language dialect of the Siberian region.

Sign language is not a universal language for all deaf people in the world. Depending on the location of the hard-of-hearing, sign language dialects differ. To ensure communicative interaction, Vladislava developed methods for recognizing sign language from visual data using neural networks.

«In order to implement the gesture recognition software, I used a modified LSTM neural network model that can highlight the key points of the palm then tracks the movements of these points. The system can reliably understand 6 different gestures. Currently, I continue filling the database of gestures so that the system can understand more words. Then I will have to figure out how to recognize sentences, because often pauses in sign speech are minimal or absent. I am going to continue my work during my master’s degree training,” said Vladislava.

This work was carried out within the framework of interdisciplinary cooperation between the School of Space and Information Technology and the School of Philology and Language Communication of Siberian Federal University.

“Lyudmila Kulikova, director of the School of Philology and Language Communication, contacted me in September. We discussed possible interdisciplinary cooperation between our schools. Among all the tasks, Vladislava was interested in gesture recognition based on visual data. It turned out that sign language differs significantly in different regions of Russia, and our Siberian region is no exception, so gesture recognition is a complex and interesting task,” shared Anna Pyataeva, supervisor, assistant professor of the Department of Artificial Intelligence Systems at the School of Space and Information Technology.

The thesis titled “Development of methods for recognizing the Russian language from video sequences on the basis of the language dialect of the Siberian region” by Vladislava Zhukovskaya became a winner of the All-Russian Engineering Competition in the Best Bachelor’s Thesis nomination.