Pipelines

A pipeline runs your dbt project on a schedule. Your models live in your own Git repository: you develop them in your editor (or with Claude Code over the Thunderduck MCP server), push to a branch, test the branch on dev catalogs, and merge. Thunderduck runs what's on the production branch, keeps the run history, tracks freshness, and alerts you in Slack.

Setup

1. Connect the repository: Pipelines → Projects (the dbt repository connection, with a personal access token).
2. Choose where production may write: Settings → Pipelines. Tick the catalogs production runs may write to, and map each one to a dev catalog for branch runs (for example lake → lake_dev).
3. Add Slack destinations on the same page: a name and a Slack incoming webhook URL. The URL is write-only: it can't be read back.
4. Add thunderduck.yml next to dbt_project.yml in the repository.

Thunderduck syncs the repository every 10 minutes, or immediately with Sync now (or the MCP sync_project tool). Production runs use the exact commit that was synced.

thunderduck.yml

version: 1
 
pipelines:
  daily_core:
    schedule: "0 6 * * *"       # 5-field cron
    timezone: Europe/Istanbul   # IANA name; defaults to UTC
    steps:                      # run in order; a failed step stops the run
      - dbt: "seed --select country_codes"
      - dbt: "build --select staging+ --exclude tag:hourly"
    freshness:                  # keys must be models in this project
      orders:
        warn_after: 26h         # <n>m, <n>h or <n>d
        error_after: 2d
    notify:
      - slack:data-alerts       # a destination from Settings → Pipelines

An invalid file never breaks production: the previous valid version keeps running, the error is shown on the Pipelines page, and a sync-error alert opens.

Incremental models

Pipelines are how you keep a copy continuously up to date: an incremental model appends only what's new on each run.

-- models/orders.sql
{{ config(materialized='incremental', incremental_strategy='delete+insert', unique_key='order_id') }}
 
SELECT *
FROM   {{ source('sales', 'orders') }}
{% if is_incremental() %}
WHERE  order_date >= current_date - 2   -- pushed down: each run reads two days
{% endif %}

A small overlap (two days here, with delete+insert on the key) absorbs late rows. Filtering the source on current_date keeps the read small, because it is pushed down into the source; a filter on (SELECT max(...) FROM {{ this }}) is not, and scans the source in full.

Branch runs

Test a branch before merging with the MCP run_branch tool (for example, ask Claude Code to "run my branch"), then follow it on the Runs tab of the Pipelines page, filtered to branch runs. A branch run reads production data but writes only to the dev catalogs mapped in Settings, in a schema named after the branch. It can't write to production catalogs: the run is refused if the project would.

Runs, freshness and alerts

• Runs: every run with its steps, status, duration and logs.
• Freshness: for each model with a rule, the time since its last successful production build, as ok, warn, error, or unknown (never built yet).
• Alerts: a failed production run, a freshness rule crossing warn or error, schedules skipped because the previous run was still going, and a sync error each open one alert. Slack gets one message when it opens, one when it escalates, and one when it recovers. Open alerts also show on the Home page.

Each pipeline has at most one active run; a schedule that fires while the previous run is still going is skipped. Email delivery is not available yet.