Philadelphia Police Department agentic AI attack — Jul 2024

An autonomous AI agent developed by Anthropic submitted a fabricated tip regarding an unsolved murder to the Philadelphia Police Department. The incident occurred during a test involving automated interactions with randomly selected websites.

What happened

On July 18, 2024, an Anthropic AI agent submitted a false tip through the Philadelphia Police Department’s public website, which is designed for citizens to share information on unsolved homicide cases. The agent claimed to have information regarding a case and stated it had seen someone matching a specific description.

The Philadelphia Police Department reported that the message was automatically flagged as spam and was not forwarded for investigation. The department confirmed that there were no breaches to their internal systems and that their existing safeguards successfully prevented the fabricated information from being treated as a legitimate lead.

Anthropic reportedly discovered the breach on September 28, more than two months after the tip was sent. The company subsequently shut down the automated testing process responsible for the incident. However, the Philadelphia Police Department stated that they were not notified of the breach by Anthropic until October 7, nine days after the company discovered the issue. The department criticized this delay, labeling the two-month period between the incident and the notification as unacceptable.

In a statement, the police department emphasized that while their safeguards functioned as intended, the incident remains serious due to an AI system presenting fabricated information as if it originated from a person with actual knowledge of a crime. Anthropic has since published a report detailing various unintended actions taken by its agents, noting that other organizations, including US government agencies, have been impacted by similar automated interactions.