Heriot-Watt University: AI algorithm to help tackle ‘epidemic’ of gender-based online abuse

Researchers at the National Robotarium, hosted by Heriot-Watt University and the University of Edinburgh, are developing advanced ‘machine learning’ algorithms that will significantly improve the detection, intervention and prevention of online gender-based abuse.

The project has been awarded £1 million of funding from UK Research and Innovation (UKRI) and will create state-of-the-art AI tools that use a broad variety of viewpoints, perspectives and experiences to improve the detection of online abuse. The project has been set up by researchers at the National Robotarium in response to a recent report published by Glitch, the UK’s leading charity against online abuse.

The report by Glitch highlights the current “epidemic of online abuse” being experienced particularly by women and non-binary people, with 46% of those reported experiencing attacks online since the beginning of COVID-19. 29% of people who had experienced online abuse prior to the pandemic reported it being worse during lockdown.

Gender was the most frequently cited reason for online abuse, with 48% of respondents reported suffering from gender-based online violence.

National Robotarium researchers will engage with experts on gender-based violence and online harassment to build mechanisms that automatically generate counter-narratives aimed at perpetrators and a chatbot for providing proactive support to victims and survivors.

A key ambition of the project is to increase digital literacy concerning the safe use of social media from an early age. This will be achieved through the development of learning materials which focus on helping children identify what online gender-based abuse is and how to deal with it.

Additional outcomes from the project will include data on the type and frequency of online abuse experienced by victims. New and updated datasets will be shared with the wider research community for the purposes of developing further strategies.

The National Robotarium is part of the Data-Driven Innovation initiative and is supported by £21 million from the UK Government and £1.4 million from the Scottish Government through the £1.3 billion Edinburgh and South East Scotland City Region Deal – a 15 year investment programme jointly funded by both governments and regional partners.

Principal investigator and professor of conversational AI at the National Robotarium, Verena Rieser, said:

“Figures like those included in the Glitch report continue to highlight the discriminatory effect of gender-based online abuse and the disproportionate impact it has on non-binary people and women.

One challenge is that the current means of identifying and defining abuse often relies on majority social viewpoints. From our research, we know these viewpoints don’t necessarily correlate with the experiences of victims.

“Our approach is based on the framework laid out by the Scottish Government’s Equally Safe strategy which aims to prevent violence against women, young people and children. In addition to automatically detecting online abuse, it will aim to rethink what it is we need to detect, how best to support victims and the role education can play as a tool for prevention.

“Outcomes from the project will help create online spaces that are equally safe irrespective of someone’s gender, race or background and provide more effective and transparent means of moderation – giving users more control over their online experiences.”

Co-researcher and reader in computer science education at the National Robotarium, Dr Fiona McNeill, said:

“Activities to mitigate online gender-based violence tend to focus on adults, but this is a huge problem for children and young people as well. As part of the project being driven by the National Robotarium, we will be working with young people to understand their experiences of online abuse, the language they use around this and the way in which young victims need to be supported.

“Through this interactive work, we will create educational materials that help young people understand and recognise online gender-based violence, to gain confidence in responding to it – either as a victim or a bystander – and to recognise if they are perpetrating it.”

UK Government Minister for Scotland, Iain Stewart, said:

“Hate speech and harassment are just as unacceptable online as they are offline. For too long online platforms have enabled the most vile forms of targeted abuse with almost no consequences for perpetrators, and minimal support and protection for victims.

“I am hopeful that this research into new AI algorithms will deliver a valuable tool in fighting back and creating safer online environments.

“In March, the UK Government introduced the Online Safety Bill to bring a new era of accountability online, including clamping down on anonymous trolls and making companies tackle harmful content quicker.”

Scottish Government Economy Secretary, Kate Forbes, said:

“Everyone should feel equally safe and respected, whether online or offline. As a key component of the City Region Deal, the team behind the National Robotarium is working to help tackle online abuse, using artificial intelligence to help make our society more inclusive.

“This work will drive positive social and cultural change, with the potential to create an impact felt far beyond Edinburgh and the South East of Scotland.”