Rule simulator.

Overview

Writing a custom rule takes minutes. Enforcing it is the delicate part: if some code path of your application issues the query the rule forbids, the proxy refuses it in production. The simulator answers the question that matters before you enforce: what of my own traffic would this change have blocked last month?

It takes a candidate rule change, evaluates it against the queries recorded in the window and compares it with what your rules decide today. The result is a report with a verdict, a sentence ready to paste in the PR (“Would have blocked 2 legitimate flows (9,442 executions) in the last 30 days.”) and every affected flow, with its executions, identities and masked samples.

Nothing is executed against your database, nothing is blocked, and nothing is written to the query history or to your quota.

How it works

A candidate change can combine up to 20 of these items. Each YAML is validated with the same checker as when you create the rule:

• A new custom rule (custom_new): name, severity, dialect and its YAML condition.
• An edit of an existing custom rule (custom_edit): another severity, another YAML or both.
• Disabling a custom rule (custom_disable).
• A standard rule (standard_override): enable it, disable it or change its severity.

The simulation runs in the background, in four phases:

1. Reading events. The queries the TCP proxy and the Runtime API recorded in the window, for the whole workspace or for one database.
2. Grouping by flow. Each query is grouped by its fingerprint: the text with its literals replaced. DELETE FROM cart_cartitem WHERE id = 41 and … WHERE id = 97 are the same flow.
3. Evaluation. First one representative statement of each flow, then the others, up to the plan's budget (50,000 evaluations on Builder, 200,000 on Team and Enterprise). Each statement is evaluated with today's engine, with your current rules and with the candidate ones. Statements left unevaluated inherit their flow's outcome and are counted as inferred.
4. Comparison. Candidate versus current is the effect of your change, and it is the only thing that counts for the verdict. Current versus what was recorded at the time is drift (for example, a new engine version): it is reported apart and never attributed to your change.

From the dashboard

• In the custom rule editor, Simulate against production simulates the rule you are writing or editing, without saving it. You choose how many days to simulate (7, 30, 90 or 180, or a number of your own), up to your plan's maximum; 30 by default.
• In the rule list, Simulate on a rule's row simulates enabling or disabling it.
• In Rules → Simulations you follow each simulation's progress (reading, grouping, evaluating, comparing), can cancel it and open the report when it finishes. Simulations from the last 7 days are kept there.
• When you choose the window, the dialog shows an estimate (for example, “≈ 3,200 distinct statements, ~2 min”): it counts the events and distinct statements of that window without reading any query (exact up to 100,000 events, extrapolated from a sample beyond that) and converts them to time with the speed of your recent simulations. It is an estimate: the run can take more or less.
• In Rules → Simulations you can tick two finished runs, typically of the same rule before and after an edit, and Compare runs: each run's verdict, executions, window and databases; per category, the counts and the flows added, resolved or unchanged (matched by fingerprint); and the diff of the rules' YAML. When the rules, windows or databases differ, the comparison says so.

One simulation runs at a time per workspace, with up to 10 a day on Builder and 30 on Team and Enterprise.

From CI through the API

To fail the PR that changes a rule file when it would break a legitimate flow, create in API Keys a key with the rules:simulate scope and store it as a CI secret. That scope is never assigned by default and is not included in ci_dryrun:execute, because it lets Vericto read the production history: it has to be granted on purpose. The key only sees its own workspace's simulations.

1. Start the simulation

curl -s -X POST https://api.vericto.com/api/v1/ci/rules/simulate \
  -H "X-API-Key: $VERICTO_SIM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "days": 30,
    "candidates": [
      { "kind": "custom_new", "name": "no-physical-deletes", "severity": "high", "dialect": "postgres",
        "ast_condition_yaml": "rule: no-physical-deletes\nnode_type: DeleteStmt" },
      { "kind": "custom_new", "name": "ledger-append-only", "severity": "critical", "dialect": "postgres",
        "ast_condition_yaml": "rule: ledger-append-only\nnode_type: UpdateStmt\ncondition:\n  relation: \"billing_ledgerentry\"" }
    ],
    "thresholds": { "fail_on_break": 0, "min_executions": 3, "min_days": 2 }
  }'

// 202 Accepted · Retry-After: 5
{
  "simulation_id": "7f3c9a2e-5b1d-4c8e-9f60-2a7d41e0b9c3",
  "status": "queued",
  "channel": "cli",
  "requested_days": 30,
  "progress": { "phase": "queued", "done": 0, "total": 0 },
  "summary": null
}

Optional: database_id to simulate a single database (without it, all of them) and thresholds. days goes from 1 to your plan's maximum: 30 on Builder, 90 on Team and your contract's on Enterprise (up to 180). It defaults to 30 (or the plan's maximum, if lower). A value above the maximum is rejected with an error that states your plan's maximum; it is never cut silently. POST /ci/rules/simulate and the dashboard follow the same rules.

2. Wait for the result and decide

while :; do
  r=$(curl -s https://api.vericto.com/api/v1/ci/rules/simulations/$SIMULATION_ID \
        -H "X-API-Key: $VERICTO_SIM_KEY")
  s=$(echo "$r" | jq -r .status)          # queued | running | succeeded | failed | cancelled
  [ "$s" = queued ] || [ "$s" = running ] || break
  sleep 5
done
echo "$r" | jq -r .summary.headline
[ "$(echo "$r" | jq -r .summary.verdict)" != fail ]   # the job fails on FAIL

When it finishes, the same response carries the summary and the findings that need attention, legitimate newly blocked flows first (trimmed):

{
  "simulation_id": "7f3c9a2e-5b1d-4c8e-9f60-2a7d41e0b9c3",
  "status": "succeeded",
  "summary": {
    "verdict": "fail",
    "headline": "Would have blocked 2 legitimate flows (9,442 executions) in the last 30 days.",
    "window": { "start": "2026-09-10T14:02:11Z", "end": "2026-10-10T14:02:11Z", "days": 30,
                "requested_days": 30, "cut": false, "retention_days": 90 },
    "events": 1842310,
    "fingerprints": 2914,
    "evaluated_statements": 41206,
    "budget": 200000,
    "budget_exhausted": false,
    "coverage": { "ratio": 0.996, "evaluated_executions": 1207553, "inferred_executions": 627312,
                  "not_replayable_executions": 7445, "not_covered_executions": 0 },
    "categories": {
      "newly_blocked": { "fingerprints": 3, "executions": 9443,
                         "legitimate_fingerprints": 2, "legitimate_executions": 9442 },
      "newly_flagged": { "fingerprints": 0, "executions": 0 },
      "no_longer_blocked": { "fingerprints": 0, "executions": 0 },
      "unchanged": { "fingerprints": 2911, "executions": 1825422 },
      "drift": { "fingerprints": 3, "executions": 12 }
    },
    "candidates": [
      { "code": "SIM-001", "kind": "custom_new", "name": "no-physical-deletes", "matched_executions": 9443 },
      { "code": "SIM-002", "kind": "custom_new", "name": "ledger-append-only", "matched_executions": 0 }
    ],
    "query_text_mode": "raw",
    "note": null
  },
  "findings": [
    {
      "category": "newly_blocked",
      "fingerprint": "4be1c07a9d3f6e21",
      "rule_codes": ["SIM-001"],
      "statement_kind": "DELETE",
      "tables": ["cart_cartitem"],
      "executions": 9412,
      "days_seen": 30,
      "sources": { "tcp": 9412 },
      "identities": [{ "type": "db_user", "ref": "app_web", "name": "app_web", "executions": 9412 }],
      "recorded": { "ALLOWED": 9412 },
      "outcomes": { "current": "ALLOWED", "candidate": "BLOCKED" },
      "legitimate": true,
      "confidence": "structural",
      "samples": [{ "event_id": "…", "created_at": "2026-10-10T13:58:40Z",
                    "text": "DELETE FROM \"cart_cartitem\" WHERE \"cart_cartitem\".\"id\" = $1" }]
    }
  ]
}

API limits: 6 simulations an hour per key, on top of the workspace's daily limit. Errors: 400 (an invalid YAML or rule code), 403 PLAN_UPGRADE_REQUIRED (Free plan), 409 SIMULATION_ALREADY_ACTIVE (one is already running; the response carries its simulation_id), 429 (limit reached) and 503 SIMULATOR_UNAVAILABLE (the simulator is not available at that moment).

Reading the report

A flow is legitimate when every recorded execution ran (none blocked), it ran at least 3 times on at least 2 different days (adjustable with min_executions and min_days) and at least half of its executions come from application identities: the database users (TCP proxy) and API keys (Runtime API) seen on at least half of the days of the window.

The verdict:

• FAIL: the legitimate newly blocked flows exceed fail_on_break (0 by default), whatever the coverage.
• INCONCLUSIVE: coverage is below 95% of executions, the window ended up shorter than requested (because of the plan's retention or the workspace's age), or the budget ran out with mixed flows among the newly blocked.
• WARN: does not exceed fail_on_break, but something changes: newly flagged, newly blocked flows that are not legitimate, or rules that stop blocking.
• PASS: no decision changes in the window, apart from the legitimate blocks fail_on_break tolerates.

Each finding gives the rules that decide it, the statement kind and tables, the executions (evaluated and inferred), the days it appeared on, the channel (tcp or http), the identities that issued it (up to 10), the recorded outcome and the confidence: structural, literal-dependent (the rule depends on literals) or mixed (the flow's statements disagree). It includes up to 3 samples, always with literals replaced, each linked to its event in the Audit Trail.

Example: tightening DELETEs without breaking the application

Illustrative example with test data: the application, tables and figures are not a customer's.

A DevOps team runs a Django application on PostgreSQL behind the TCP proxy. The standard rules already block DELETE and UPDATE without WHERE. Since orders and payments are soft-deleted, the team wants to go further: block every DELETE and make the ledger append-only (a rule on any UPDATE of billing_ledgerentry).

The 30-day simulation returns FAIL: “Would have blocked 2 legitimate flows (9,442 executions)”. Both flows are real application behaviour: removing an item from the cart (app_web, 9,412 executions on 30 of 30 days) and Django's daily session cleanup (app_cron, 30 executions). The third newly blocked flow, a manual DELETE of an order run once by an on-call user, is not classified legitimate: it is exactly what the team wants to stop. The ledger rule matched nothing in 30 days, which is the evidence it wanted.

The team adjusts the YAML: it scopes the DELETE rule to the soft-deleted tables (relation: "shop_order" and relation: "shop_payment") and keeps the ledger rule. The new simulation returns WARN with 0 legitimate flows blocked: only the manual DELETE is left. Then the team creates the rules and enforces them.

Limits

• Window. The one you choose on each run, up to 30 days on Builder, 90 on Team and per contract (up to 180) on Enterprise, cut by your plan's history retention and by the workspace's age. If it ends up shorter than requested, the verdict is INCONCLUSIVE.
• History is evidence, not proof. Only flows that ran in the window are covered: a monthly job can fall outside it.
• What traffic is replayed. The TCP proxy's and the Runtime API's. CLI, Validation API and MCP checks validate SQL before it runs: they are not production executions and are not replayed.
• Telemetry in sanitized mode. Literals arrived already replaced, so rules that depend on them (where_always_true, VERICTO-002, 003, 030 and 090) can under-report. The report says so (query_text_mode: "sanitized") and labels those findings literal-dependent.
• Events that cannot be replayed. Texts that cannot be decrypted, evaluations that fail closed and events of deleted databases are counted with their executions (not_replayable_executions), but not evaluated.
• Size and duration. Up to 20 changes per simulation and 15 minutes per run. Results are kept for 7 days.

Permissions, privacy and audit

• Starting and cancelling a simulation needs the member role or above (the rules.simulate permission) on a Builder plan or above.
• Viewing reports needs the viewer role (rules.view). Samples need member, as in the Audit Trail: a viewer sees counts, not statements.
• From CI, an API key with the rules:simulate scope, created by the workspace's admins.
• Privacy. Texts are decrypted only in Vericto's backend, during the simulation. Results store fingerprints, counts, identities and samples with literals always replaced, whatever the telemetry mode: no value, including one from a sensitive column, is stored or shown.
• Audit. Every simulation is recorded in the workspace's audit log as rule.simulation.requested (channel, API key, hash of the changes, window and database), and every cancellation as rule.simulation.cancelled.

See also

• Custom rules
• API Keys
• Telemetry privacy
• Audit log