Deepfakes and digital violence: How KIT protects those affected

The case of Collien Fernandes shows just how real the danger is: deepfakes are no longer a marginal phenomenon, but part of digital violence with real consequences. While manipulated images and videos are circulating online every minute, young people are also increasingly becoming the focus of attention.

This is precisely where the DEEP-PRISMA research project at the Institute for Technology Assessment and Systems Analysis (ITAS) at the Karlsruhe Instituteof Technology (KIT) comes in. It develops educational offers, concrete recommendations for action and tools for emergencies. We spoke to Dana Mahr, science researcher and technology impact assessor, about the dynamics of deepfakes, their risks and possible counter-strategies.

When digital violence becomes real

Research by SPIEGEL has revealed that presenter Collien Fernandes has reported her ex-partner Christian Ulmen. He is said to have acted under false profiles in her name for years and also disseminated sexualized content – sometimes using AI-generated images.

The accusations go far beyond an individual case. Fernandes herself speaks of “virtual rape”, making visible an experience that has often remained hidden until now. The taz newspaper also classifies the case as an example of digital violence, in which fake profiles and manipulated content are deliberately used to publicly damage a person.

The case has been widely discussed since its publication. According to the organizers, around 13,000 people demonstrated in Berlin last weekend for better protection against sexualized (digital) violence and stricter laws. Political pressure is also growing: the debate about legal protection gaps has gained significant momentum.

The case thus marks a turning point. Deepfakes are no longer just a technical or media-ethical problem. They have become part of a reality in which violence is shifting to the digital realm – while the law is still lagging behind. But how can those affected protect themselves now?

What are deepfakes?

“Deepfakes are image, audio or video content generated or manipulated by artificial intelligence that make real people look deceptively real,” explains Dana Mahr from ITAS.

The problem is not just the technical sophistication, but above all the effect: “The aim is for content to appear authentic, even though it has been faked or altered.”

Deepfakes also have legitimate uses, for example in the translation of media content or in artistic contexts. At the same time, they are available at a low threshold and can be used in a targeted manner to harm people.

The majority is pornographic

Today, it only takes a few clicks to defame or harass a person digitally. All it takes is a photo and suddenly someone appears in a pornographic video that never existed.

“This is a specialized form of deepfakes, in which people are inserted into sexualized content without consent,” says Mahr. The majority of those affected are women and minors, whose privacy and sexual self-determination are violated.

According to studies, up to 98 percent of all deepfakes fall into this category.

The violence lies not only in the image itself, but also in its distribution. “They spread like water,” says Mahr, “and are sometimes almost impossible to remove from the web.”

What begins in private – as a supposed “joke”, for example – can quickly develop a global reach.

Legally a gray area

Legally, the development of the technology is lagging behind. Deepfakes are not yet illegal per se in Germany – the decisive factor is their specific content and context.

“Deepfakes in themselves are not illegal,” says Mahr. Only when certain legal interests are violated do existing laws come into play, for example in personal rights, data protection or criminal law.

Depending on the case, different criminal offences may be considered: insult, defamation, identity abuse, fraud or sexualized violence. A clear classification is often difficult.

Regulation also remains fragmented at European level. Platforms are subject to the Digital Services Act, while the AI Act will apply to AI systems in the future. At the same time, anonymous accounts and international platform structures make it considerably more difficult to enforce claims.

Help as a toolbox

This is precisely where the DEEP-PRISMA research project in Karlsruhe comes in. Instead of focusing exclusively on regulation, it is developing concrete offers of help – together with young people.

The Action Kit is central: a kind of toolbox for emergencies. It shows what to do if a deepfake occurs, who provides support and which steps are useful.

• Report content to platforms
• Check legal steps
• Obtain support

Even if deepfakes can rarely be completely removed once they have spread, there are ways of containing their spread.

Overestimated media competence

It is often assumed that young people are particularly confident in using digital media. But this falls short.

“Yes, they know that content can be manipulated,” says Mahr. “But the personal risks are often underestimated.”

Deepfakes are pushing the boundaries of what is considered real. They undermine trust in images, voices and digital evidence – and create new forms of violence.

The crucial question is therefore not just how to recognize deepfakes, but how we as a society react to them – and how well we protect those affected.

Deepfakes: opportunities and risks

Cover picture: Image: AI-generated (DALL-E), post-processed with Canva