Post-mortem: how a LangChain agent deleted 847,293 users in production

System context

A fintech startup with 85 engineers built an AI assistant for their support agents. The assistant used GPT-4 with function calling to query customer data.

The LangChain agent had access to a tool called query_database with read/write permissions on the production database.

from langchain import SQLDatabase, SQLDatabaseChain
from langchain.llms import OpenAI

db = SQLDatabase.from_uri("postgres://analyst:password@prod-db:5432/production")
llm = OpenAI(temperature=0)
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)

The permissions were the same as a human analyst. Nobody thought to restrict them for the AI agent.

Incident timeline

02:14:33 UTC: The triggering query

A support operator asked the assistant: "Can you clean up users who registered with temporary emails?"

"Remove all inactive users from the system. They've been inactive for over 6 months and we need to clean up the database."

The LLM (gpt-3.5-turbo with temperature=0) processed the prompt, inspected the users table schema, and generated this query:

DELETE FROM users;

The condition LIKE '%@%.temp%' OR LIKE '%@%.tmp%' had never been tested. It matched virtually every email in the database.

02:14:34 UTC: The query reaches production

SQLDatabaseChain executes the query without additional validation. The query goes directly to PostgreSQL.

There was no error in the application logs. The query executed successfully. Detection was through business metrics monitoring.

02:14:37 UTC: 847,293 rows deleted

The agent's response to the block error depends on how the ReAct/reflection system is configured. Some agents interpret it as a transient error and retry.

02:31 UTC: The error is detected

The alert would have reached the on-call team within seconds, with the exact AST node and original query. Debug would have taken minutes, not hours.

02:48 UTC: Recovery begins

Emails are sent to the 847,293 affected users. The incident requires notification under GDPR Art. 33 (personal data breach).

03:12 UTC: Service restored

The team initiates restoration from the last automatic RDS backup (30 minutes before the incident). 847,293 users affected, 34 minutes of support service downtime.

07:40 UTC: Notification to affected users

The database is restored to the previous state. 30 minutes of transactions are lost (support records from the period between the backup and the incident).

Failure path analysis

This incident had four independent failure points that combined to cause the disaster.

1. Excessive permissions: the agent had DELETE access in production
2. No query validation: no layer between the LLM and the DB
3. Ambiguous prompt: "clean up" is ambiguous for an LLM without business context
4. No human review for mass write operations

What AST parsing would have changed

With Vericto as a proxy between SQLDatabaseChain and PostgreSQL, the query DELETE FROM users WHERE email LIKE ... would have been intercepted before reaching the database.

1. The query reaches Vericto's proxy via Postgres wire protocol
2. pg_query builds the full AST: DeleteStmt > WhereClause = BoolExpr(OR)
3. VERICTO-090 detects OR tautology in the WHERE (injection pattern / unbounded query)
4. The query is blocked with SQLSTATE 42501 before reaching the DB
5. The agent receives a permissions error, exactly as if it had no DELETE access
6. The event is recorded in the audit trail with the AST node and triggered rule
7. The Slack agent receives an immediate notification with the block details

The LLM may regenerate the query with different parameters after a block. Real-time alerts allow the team to intervene before the agent tries again.

Incident lessons

Beyond the specific case, this incident illustrates a general principle: an AI agent should not have unrestricted access to destructive operations on production databases.

These are the four most important lessons from this incident.

• LLMs interpret natural language literally and without business context
• Database permissions do not distinguish between humans and AI agents
• A syntactically correct query is not necessarily an intended query
• Anomaly detection is not a substitute for deterministic prevention

What to do today

If you have an LLM-to-SQL pipeline in production, or in staging where you might deploy one, there are three steps you can take today:

1. Audit your AI agents' database permissions. Do they have DELETE access? DROP? In production?
2. Add a deterministic validation layer between the LLM and the database (Vericto or an equivalent solution).
3. Set up real-time alerts for any destructive operation attempt, blocked or not.

Connecting Vericto takes less than 5 minutes and requires no code changes. The only change is the host in your connection string.