Data pipelines: dbt in CI and every Airflow SQL task

What it protects

• VERICTO-001 and 042: DELETE and UPDATE without WHERE in a maintenance task.
• VERICTO-010 and 011: DROP TABLE and TRUNCATE outside a migration.
• VERICTO-040 and 084: INSERT INTO … SELECT without a filter, and CREATE TABLE AS SELECT.
• VERICTO-061: batch INSERT of more than 10,000 rows, which is flagged.
• VERICTO-091 to 097 for Snowflake, in the CI check with the Snowflake dialect.

Latency and availability

In CI, the check runs outside the runtime and does not affect how the pipeline runs. The CLI sends SQL in batches of up to 500 statements, and each statement counts as one validation.

In Airflow with the proxy, each statement adds the hop to the proxy and its in-process evaluation.

For parsing and rule evaluation, the design target is p99 < 2 ms. It is a target, not a measurement: the proxy latency benchmark is not published yet.

How to implement it

1. dbt: install the CLI in the CI job (installation) and store an API key with the ci_dryrun:execute scope as a secret.
2. Compile the project (dbt compile writes the SQL to target/compiled) and run it through vericto check with your warehouse's dialect: PostgreSQL, MySQL, Oracle, SQL Server, Snowflake or SQLite.
3. Deploy (dbt run) only if the check passes: exit code 1 fails the job.
4. Airflow: deploy the proxy next to the workers or as a service and point the Airflow connection's host and port (Postgres or MySQL) at the proxy. The DAGs do not change.
5. Give the pipeline its own database user, to see its traffic separately and, on Team and Enterprise, give it its own allowlist.

Coming soon: vericto-dbt check, which will read the dbt project directly, and for Airflow an operator and a cluster policy that will evaluate every SQL task through the Runtime API, also on dialects with no proxy.

When not to use this option

• Transformations that are not SQL (Spark, pandas): Vericto evaluates SQL statements, nothing else.
• Materializing tables through the proxy. If the pipeline creates tables with CREATE TABLE AS SELECT through the proxy, VERICTO-084 blocks them by default: disable it or change its severity in that pipeline's workspace, and simulate the change first.
• Airflow tasks on Snowflake through the proxy. Snowflake has no proxy: until the Airflow operator and cluster policy arrive (coming soon), call the Runtime API from the task. See data warehouse.

Plan availability

Prices and every limit on Pricing.

Other architectures

• All reference architectures
• Web application
• Microservices and Kubernetes
• AI agents and text-to-SQL
• Data warehouse and analytics
• CI/CD and migrations
• Serverless and edge
• Multi-tenant SaaS
• Enterprise and regulated