The commission examines AI tools that could generate non-consensual sexual images of women and children.

The Commission states that it is analyzing the findings of an investigation regarding artificial intelligence models hosted on an online platform and is discussing with the parties involved. The institution has not yet announced the opening of a formal procedure, but indicates requests for information, investigations, and fines among the measures that can be used under the Artificial Intelligence Regulation.

The European Commission is analyzing information according to which some artificial intelligence tools hosted on an online platform could generate non-consensual sexual images of women and materials involving children. The institution claims to be aware of the investigation that flagged these tools, is discussing with the parties involved, and is examining the findings to determine whether enforcement measures under the Artificial Intelligence Regulation are necessary.

In short

The Commission states that it has received information about AI tools that could generate sexual images of women without consent and materials involving sexual abuse of children.

The institution is in contact with the parties involved and is analyzing the findings, but has not confirmed the opening of a formal investigation and has not publicly identified the tools under examination.

Starting from August 2, the Commission has enforcement powers regarding providers of general-purpose AI models and the relevant prohibitions set out in the Artificial Intelligence Regulation.

Initial measures may include discussions with providers and requests for information. If indications of a violation arise, the Commission may adopt binding requests, open an investigation, and impose fines.

Enforcement is shared between the Commission and national authorities. The Commission oversees providers of general-purpose AI models, while market surveillance authorities in member states control other systems that fall under their jurisdiction.

The case concerns tools that would use generative artificial intelligence to produce or modify sexual images without the consent of the individuals represented. Such services may allow users to input a photograph, describe the desired outcome, or use a trained model for the automatic generation of content.

The issue is not limited to completely artificial images. A tool may use a photograph of a real person and create a sexual representation that has never existed, without the respective person having consented to the use of the image or the outcome produced.

In the case of children, the risk is more severe as the tools can be used to create sexual representations of minors. The Commission has stated that systems producing materials regarding sexual abuse of children or sexual images of women against their will have no place in the European Union.

The institution has not published a list of the models analyzed, their number, the identity of the developers, or the volume of content that may have been generated. Nor is the platform on which they would be hosted identified clearly enough in the available transcript to be named without further verification.

The Commission has confirmed, however, that it is aware of the report and is examining the findings for possible measures. It also stated that it is in contact with all relevant parties, without specifying whether these include the platform, the developers of the models, French authorities, or the organization that conducted the investigation.

This preliminary stage must be distinguished from the opening of a formal procedure. Analyzing a report allows the institution to verify the information, determine what technology is involved, and identify the responsible provider before deciding whether there are sufficient elements for an investigation.

The Commission has not confirmed that it has already identified a violation. The fact that a tool is presented in an external report does not alone demonstrate that it operates in the described manner, that it is accessible to users in the EU, or that legal responsibility lies with the platform hosting it.

The assessment must distinguish between the model developer, the person who published it, the service that makes it available to users, and the platform that provides the hosting infrastructure. Depending on how it operates, each actor may have a different role and distinct legal obligations.

A platform may host the files of a model without having developed it. In other cases, it may integrate the model into its own service, control access, recommend the tool to users, or generate revenue from its use.

For the enforcement of the Artificial Intelligence Regulation, the Commission must determine whether the technology analyzed is a general-purpose AI model, a system built on such a model, or a service that uses multiple components. The classification determines the competent authority and applicable obligations.

The Commission has indicated August 2 as an important date for its new enforcement powers. From that moment, the institution can intervene directly with providers of general-purpose AI models and in matters related to the prohibitions set out in the regulation.

General-purpose AI models can be used for multiple tasks and integrated into numerous applications. The provider develops the base model, and other companies or users can adapt it for generating images, texts, audio recordings, or other outputs.

A tool explicitly intended to generate non-consensual sexual images can be built by modifying a broader model. In such a situation, the assessment must determine what capabilities come from the original model and what functions have been added by the application developer.

The Commission explained that the enforcement of the rules will follow a gradual approach. The first step may consist of dialogue with providers and a voluntary request for explanations regarding the operation of the tool, safety measures, and user access.

The institution may request information about the data used, the filters applied, safety tests, reporting mechanisms, and measures to prevent the generation of prohibited content. It may also request explanations about how the model was published and distributed.

An initial request for information does not represent a finding of guilt. It allows the authority to verify whether the external report accurately describes the product and whether the provider has implemented sufficient measures to prevent abusive uses.

If the responses are incomplete or the issue requires a formal verification, the Commission may adopt a binding decision requesting information. Failure to provide data or providing incorrect information may have separate consequences from any violation caused by the operation of the tool.

The Commission may subsequently open an investigation. This may analyze the provider's responsibility, the accessibility of the service in the Union, the technical characteristics, and the compliance of the measures introduced to prevent prohibited content.

An investigation may lead to requests for modifications to the system, limiting access to certain functions, withdrawing a tool, or applying financial penalties, depending on the identified violation and the powers provided by legislation.

The spokesperson explicitly mentioned the possibility of fines but did not indicate the maximum applicable value in this case. The level of a penalty depends on the violated norm, the responsible actor, the severity and duration of the situation, and the cooperation of the company with the authorities.

The Commission compared the process to the enforcement of the Digital Services Regulation, where oversight can begin with contacts with the platform, continue with requests for information, and lead to a formal investigation.

The two regulations are not, however, identical. The Artificial Intelligence Regulation aims at the development and use of AI systems, while the Digital Services Regulation establishes the responsibilities of online services regarding content, systemic risks, and user protection.

A case may fall under both frameworks. The model that generates the images may be analyzed under the Artificial Intelligence Regulation, while the platform distributing it may have separate obligations under the Digital Services Regulation.

The Commission stated that there are already clear rules both in the DSA and in the Artificial Intelligence Regulation. The choice of legal instrument depends on the operation of the service, the role of each company, and the nature of the content.

The DSA does not require a platform to pre-check every piece of information or tool uploaded by users. Once informed about potentially illegal content or activities, the platform may, however, have the obligation to analyze the notification and act in accordance with the applicable rules.

Very large platforms must also assess the systemic risks created by the operation of their services. These risks may concern the distribution of illegal content, the protection of minors, gender-based violence, and the effects on fundamental rights.

The Commission did not specify whether the targeted platform is designated as a very large online platform or whether it is already conducting a separate assessment under the DSA.

At the level of the Artificial Intelligence Regulation, the powers are shared. The Commission has direct responsibility regarding providers of general-purpose AI models, while member states must designate supervisory authorities for systems and uses that fall under national jurisdiction.

The spokesperson stated that the 27 national market surveillance authorities will be able to control and apply the rules and will cooperate with the Commission. It does not result that all 27 authorities would simultaneously investigate the same service.

The competent authority may depend on where the provider is established, how the product is introduced to the European market, and the category of technology. Cooperation is necessary when a service is accessible in multiple states.

The Commission has not provided data on the number of users who may have accessed the tools, the countries in which they are available, or the existence of identified victims. The external report and its findings must be verified before such information can be officially established.

It is also unclear whether the platform has already removed some models, restricted access, or introduced additional measures following the emergence of the investigation. The Commission has not provided information about the responses of the companies involved.

For individuals represented in the generated images, harm may exist even if the material is artificial. The image may be distributed as if it were authentic, may affect reputation and privacy, and may be used for harassment, blackmail, or intimidation.

In the case of children, the generation and circulation of sexual materials can normalize abuse and may be used in criminal communities. Authorities must distinguish the assessment of technology from the criminal investigation of individuals who create or distribute illegal content.

The Artificial Intelligence Regulation does not replace criminal legislation, child protection rules, data protection, or the right to one's own image. The same situation may generate different procedures against the technology provider and users who produced or distributed the materials.

Image generation tools may introduce filters that block certain requests, detect sexual content, limit the uploading of photographs of real people, or mark the results as artificial. The effectiveness of these measures depends on their design and the ability of users to circumvent them.

The mere existence of rules in the terms of use is not always sufficient. Authorities may analyze whether the provider effectively applies the restrictions, whether it responds to reports, and whether it removes models or functions created in a way that is clearly accessible to prohibited users.

The Commission did not indicate what measure it would consider appropriate in the analyzed case. The decision may depend on the existence of a violation, the cooperation of providers, and the possibility of quickly remedying the risk.

The institution has used a categorical formulation regarding the desired outcome, stating that it will not accept or tolerate in Europe tools that generate materials regarding sexual abuse of children or sexual images of women without their consent.

This political statement does not replace the legal procedure. Before a penalty, the Commission must establish the facts, the applicable norm, and the responsibility of each company.

The Artificial Intelligence Regulation distributes supervisory responsibilities between the European Commission and national authorities. The European Office for Artificial Intelligence coordinates the enforcement of rules regarding general-purpose AI models and cooperates with bodies in member states.

The reported case represents one of the first tests for the enforcement of prohibitions and European powers over generative tools that can produce abusive sexual content. The Commission is still in the stage of examining the information and has not announced a conclusion, a formal investigation, or a penalty.

https://2eu.brussels/en/news/the-commission-is-examining-ai-tools-that-could-generate-non-consensual-sexual-images-of-women-and-children