BigQuery

Measured connects to BigQuery using a Google Cloud service account.

Prerequisites

• A Google Cloud project with BigQuery enabled
• A service account with BigQuery read access
• A service account JSON key file

Create a service account

Via the Google Cloud Console

1. Go to IAM & Admin → Service Accounts
2. Click Create Service Account
3. Name it (e.g. dq-cloud-svc) and click Create
4. Assign the role BigQuery Data Viewer (roles/bigquery.dataViewer)
5. Also assign BigQuery Job User (roles/bigquery.jobUser), required to run queries
6. Click Done
7. Click the service account → Keys → Add Key → Create new key → JSON
8. Download the JSON key file

Via gcloud

# Create service account
gcloud iam service-accounts create dq-cloud-svc \
  --display-name="Measured Service Account"

# Assign roles
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:dq-cloud-svc@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataViewer"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:dq-cloud-svc@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/bigquery.jobUser"

# Create key
gcloud iam service-accounts keys create dq-cloud-key.json \
  --iam-account="dq-cloud-svc@YOUR_PROJECT_ID.iam.gserviceaccount.com"

Configure Measured

In the UI: Data Sources → Add data source → BigQuery

You'll need to provide:

• Project ID, your GCP project ID
• Dataset, the BigQuery dataset to monitor (optional; can be selected per-asset)
• Service account key, paste the contents of your JSON key file

Or via the API:

curl -X POST https://dq.your-company.com/api/v1/datasources \
  -H "Authorization: Bearer $DQ_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "type": "datasource",
      "attributes": {
        "name": "BigQuery Analytics",
        "type": "bigquery",
        "project": "my-gcp-project",
        "credentials": { ...service account JSON key... }
      }
    }
  }'

Dataset-level access

For tighter security, grant access at the dataset level rather than the project level:

bq add-iam-policy-binding \
  --member="serviceAccount:dq-cloud-svc@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataViewer" \
  YOUR_PROJECT_ID:YOUR_DATASET

Note: the bigquery.jobUser role must still be granted at the project level.

Cost management

Measured runs SQL queries against BigQuery at checkpoint run time. Each query is billed at standard BigQuery on-demand rates (or deducted from flat-rate slots if you have a reservation).

To minimize costs:

• Schedule checkpoints during off-peak hours
• Use table partitioning and add partition filters to your data assets
• Use expect_column_values_to_be_between with mostly threshold rather than full-scan row-level checks

Troubleshooting

403 Access Denied: BigQuery BigQuery: Permission denied
Check that both bigquery.dataViewer and bigquery.jobUser are assigned to the service account.

404 Not found: Dataset
Ensure the dataset exists in the specified project and region. BigQuery datasets are region-specific.

Billing account not set
The project must have a billing account enabled to run BigQuery jobs.