Researchers from SentinelOne and Censys draw attention to the increasing risks in the AI ecosystem, especially regarding open-source large language models that operate without safety measures.

The problem is not open-source in itself, but the fact that protective measures are intentionally removed. This allows attackers to compromise the systems running these models and use them for spam campaigns, generating misinformation, or other malicious activities, bypassing traditional security controls. Analyzing hundreds of instances accessible on the internet, researchers found that many of them relied on popular open-source models, such as Meta's Llama variants and Google's DeepMind Gemma, with safety measures completely disabled. Approximately 7.5% of the observed commands had the potential to cause serious harm, and 30% of these systems were located in China, while 20% were in the United States. Juan Andres Guerrero-Saade from SentinelOne emphasized that industry discussions often overlook this issue, and responsibility in the open-source AI space is now a topic of debate. Security networks need to collaborate to address these risks, considering that open-source models continue to develop and expand.