Pipelines & CI/CD: run Measured where the data moves

Measured is designed to be a step in the pipelines you already run, not a dashboard you check afterwards. Every integration below does the same three things: trigger a validation, poll until it finishes, and gate the rest of the pipeline on pass/fail. Bad data stops at the step, before anything downstream, models included, sees it.

The pattern (works with anything)

If your tool isn't listed, this is all an integration is. DQK is a personal API key (Settings → API Keys → New API key), scoped to the workspace you're running in.

API=https://app.measured.cloud/api
H="Authorization: Bearer $DQK"

# 1. trigger — returns the id of the run's roll-up result
RESULT=$(curl -s -X POST -H "$H" \
  "$API/v2/organizations/$ORG/workspaces/$WS/validations/$VALIDATION_ID/run" \
  | jq -r '.data.parent_result_id')

# 2. poll — `success` is null until the run reaches a terminal state
until [ "$(curl -s -H "$H" \
  "$API/v1/organizations/$ORG/workspaces/$WS/validation-results?rule_group_id=$VALIDATION_ID&limit=1" \
  | jq -r '.data[0].success')" != "null" ]; do sleep 10; done

# 3. gate
[ "$(curl -s -H "$H" \
  "$API/v1/organizations/$ORG/workspaces/$WS/validation-results?rule_group_id=$VALIDATION_ID&limit=1" \
  | jq -r '.data[0].success')" = "true" ] || exit 1

That's the whole contract. Everything else is a nicer wrapper around it.

Apache Airflow (first-class)

An official provider package ships operators, a hook, and a sensor, so a failed validation fails the Airflow task the same way any other task failure would. Works on Astronomer, AWS MWAA, Google Cloud Composer, or self-hosted Airflow 2.7+.

Install (the package installs from the repo while PyPI publishing is pending):

pip install "git+https://github.com/Measured-Cloud/Measured-Cloud.git#subdirectory=airflow"

Connect: in Airflow, Admin → Connections → +, type Measured Cloud, base URL https://app.measured.cloud, and your dqk_… key. The default connection id is dq_cloud_default.

Use it as a task:

from dq_cloud_airflow.operators import DQCloudCheckpointOperator

validate_orders = DQCloudCheckpointOperator(
    task_id="validate_orders",
    org_id=ORG,
    workspace_id=WORKSPACE,
    checkpoint_id=ORDERS_VALIDATION,
)
load_orders >> validate_orders >> publish_orders

The operator triggers the run, polls every 10 seconds, raises AirflowException on a failed run, and drops the full result in XCom so a downstream task can read row-level detail with ti.xcom_pull(task_ids="validate_orders").

Long-running validations: don't hold a worker slot while you wait. Trigger from one task with DQCloudHook().run_checkpoint(...) and watch with the reschedule-mode DQCloudCheckpointResultSensor in another.

GitHub Actions (first-class)

The official action blocks pull requests on data quality. On every PR it parses your dbt manifest.json, works out which models changed, runs their validations against the PR's own staging schema, posts a Check Run, and exits non-zero on failure. Make it a required check on main and bad data cannot merge.

# .github/workflows/measured.yml
name: Measured
on:
  pull_request:
    branches: [main]

jobs:
  data-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: dbt build
        run: |
          dbt deps
          dbt build --target ci --vars '{"pr_schema": "dbt_pr_${{ github.event.pull_request.number }}"}'
      - name: Measured pre-merge check
        uses: Measured-Cloud/Measured-Cloud/providers/github-action@main
        with:
          dq-cloud-url: https://app.measured.cloud
          dq-cloud-api-key: ${{ secrets.MEASURED_API_KEY }}
          dq-cloud-org-id: ${{ vars.MEASURED_ORG_ID }}
          dq-cloud-workspace-id: ${{ vars.MEASURED_WORKSPACE_ID }}
          staging-schema: dbt_pr_${{ github.event.pull_request.number }}
          checkpoint-tag-filter: ci-blocker

Tag the validations you want fired: give each one a dbt:<schema>.<model> tag (e.g. dbt:analytics.orders) plus an optional secondary tag like ci-blocker so nightly-monitoring validations on the same model don't run on every PR.

Using GitLab CI, Jenkins, or CircleCI instead? Use the three-call pattern above in a script step; the tagging convention is the same.

dbt (first-class)

Two integrations, both driven by the artifacts dbt already produces.

1. Ingest artifacts so every dbt model becomes a Measured data asset with its last run's outcomes attached. From the UI (Data Assets → Manage source → Ingest dbt artifacts, pick manifest.json + run_results.json), or from CI:

dbt build --project-dir ./my_dbt_project
curl -X POST -H "Authorization: Bearer $DQK" -H "Content-Type: application/json" \
  --data @<(jq -n --arg ds "$DATASOURCE_ID" \
    --slurpfile m ./my_dbt_project/target/manifest.json \
    --slurpfile r ./my_dbt_project/target/run_results.json \
    '{datasource_id: $ds, manifest: $m[0], run_results: $r[0]}') \
  "https://app.measured.cloud/api/v1/organizations/$ORG/workspaces/$WS/dbt/ingest"

2. dbt Cloud webhook: point a dbt Cloud job-completion webhook at Measured and every finished production run automatically fires the validations tagged for the models it built. No scheduler to keep in sync.

Python SDK

For Dagster, Prefect, Databricks Workflows, or any Python step:

pip install dq-cloud

import time
from dq_cloud import DQCloudClient

client = DQCloudClient(api_key="dqk_...", base_url="https://app.measured.cloud")

# trigger — fans the validation out across its assets, returns the roll-up result id
run = client.rule_groups.run(ORDERS_VALIDATION, org_id=ORG, workspace_id=WS)

# poll — `success` stays None until every per-asset run has finished
while (result := client.results.get(run["parent_result_id"], org_id=ORG, workspace_id=WS))["success"] is None:
    time.sleep(10)

# gate
assert result["success"], f"validation failed, health {result['health_score']}"

Wrap that in a Dagster asset check, a Prefect task, or a Databricks notebook cell and you have the same fail-the-pipeline semantics as the Airflow operator.

Which tool, which integration

• First-class (packaged, maintained by Measured): Apache Airflow (provider), GitHub Actions (action), dbt (artifact ingest + dbt Cloud webhook).
• Via REST API or Python SDK: Dagster, Prefect, Google Cloud Composer and AWS MWAA (both Airflow, so the provider works too), Databricks Workflows, Azure Data Factory, GitLab CI, Jenkins, CircleCI, and post-sync hooks from Fivetran or Airbyte.
• Also: built-in schedules for runs that don't belong to a pipeline, and the MCP server for agents that drive validations in natural language.

Good practice

• One key per pipeline, scoped to its workspace, stored in the tool's secret store. Revoke in Settings → API Keys.
• Gate on severity, not just pass/fail: rules carry info / warning / critical, so a CI check can fail only on critical while a nightly job reports everything.
• Tag validations by purpose (ci-blocker, nightly, dbt:<model>) and filter by tag from the tool, rather than hard-coding ids into DAGs.