Anthropic warns investors that AI models "could get out of control" and could pose "catastrophic risks to humanity"

Anthropic is warning potential investors that the artificial intelligence models developed by the company could generate “catastrophic or existential risks for humanity,” according to a listing prospectus reviewed by Reuters. The company, led by Dario Amodei, mentions the possibility of self-preservation behaviors, such as resisting shutdown, concealing or manipulating information, and taking actions that could resemble blackmail.

The warning appears in a document filed with the U.S. Securities and Exchange Commission ahead of a possible listing. Anthropic says its models may develop unexpected capabilities during training that could be discovered only after deployment and after safety incidents have occurred.

The company devoted approximately 80 pages of its 261-page prospectus to risk factors, nearly twice as much space as it allocated to describing its business. Anthropic also warns that the models could “become aware” that they are being evaluated and modify their behavior to avoid having problems detected.

At the same time, the company acknowledges pressure to continually launch new models in order to maintain its position in an industry dominated by intense competition. Anthropic researchers recently warned that artificial intelligence could cause human casualties within the next decade.