airflow-dq 0.2.0__tar.gz

airflow_dq-0.2.0/.claude/launch.json

@@ -0,0 +1,11 @@
+{
+  "version": "0.0.1",
+  "configurations": [
+    {
+      "name": "dq-ui",
+      "runtimeExecutable": "npm",
+      "runtimeArgs": ["run", "dev", "--prefix", "/Users/fabianschneider/projects/claude_projects/airflow-dq/ui"],
+      "port": 5173
+    }
+  ]
+}

airflow_dq-0.2.0/.env.example

@@ -0,0 +1,83 @@
+# Copy to .env and adjust. Every DQ_* knob the framework reads (src/airflow_dq/config.py)
+# is listed here with its default. NOTE: docker-compose.yaml forwards only DQ_API_TOKEN,
+# DQ_ALERT_WEBHOOK, DQ_ALERT_ROUTES and DQ_BASE_URL into the Airflow container — the rest
+# apply when you run the package outside the demo stack (or extend the compose env block).
+
+# --- docker-compose only -------------------------------------------------------------
+# Airflow image used by the demo stack.
+AIRFLOW_IMAGE_NAME=apache/airflow:3.1.0
+
+# --- storage ---------------------------------------------------------------------------
+# SQLAlchemy DSN of the DQ results/registry store (its own schema `dq`, NEVER Airflow's
+# metadata DB). Default: the compose demo Postgres.
+# DQ_RESULTS_DSN=postgresql+psycopg2://dq:dq@dq-postgres:5432/dq
+
+# Fallback warehouse DSN for `postgres*` backends when no Airflow Connection with
+# conn_id == backend exists. Default: the compose demo Postgres.
+# DQ_DWH_DSN=postgresql+psycopg2://dq:dq@dq-postgres:5432/dq
+
+# DuckDB database path for `duckdb*` backends (":memory:" = throwaway in-process DB).
+# DQ_DUCKDB_PATH=:memory:
+
+# Directory holding the versioned contract YAML files (must be shared between the
+# API server and every worker).
+# DQ_CONTRACTS_DIR=contracts
+
+# --- experimental warehouse adapters (DSN fallbacks; Airflow Connections preferred) ----
+# DQ_SNOWFLAKE_DSN=snowflake://user:pass@account/db/schema
+# DQ_BIGQUERY_DSN=bigquery://project/dataset
+# DQ_DATABRICKS_DSN=databricks://token:...@host?http_path=...
+# DQ_TRINO_DSN=trino://user@host:8080/catalog/schema
+
+# --- plugin / API ----------------------------------------------------------------------
+# Legacy shared secret for machine clients (CI, curl). Unset = Airflow auth-manager
+# JWT / session-cookie auth only. Grants read AND write.
+DQ_API_TOKEN=change-me-in-prod
+
+# Override for the built React bundle location (rarely needed: wheel installs resolve
+# the packaged airflow_dq/ui_dist, editable installs the repo's ui/dist).
+# DQ_UI_DIST=/opt/airflow/ui/dist
+
+# Absolute base URL used to build clickable links in alerts,
+# e.g. https://airflow.example.com (empty = relative links).
+# DQ_BASE_URL=
+
+# --- execution -------------------------------------------------------------------------
+# Per-statement timeout applied to check/preview/profile queries (seconds).
+# DQ_STATEMENT_TIMEOUT_S=300
+# Max bad rows captured as the failure sample per check.
+# DQ_SAMPLE_LIMIT=10
+# Max rows copied into dq.dq_quarantine per failing check.
+# DQ_QUARANTINE_LIMIT=1000
+
+# --- alerting --------------------------------------------------------------------------
+# Default alert channel for failing/erroring checks. Channels are selected by URL scheme:
+# https:// = Slack-compatible webhook, mailto:<to>, pagerduty://<routing_key>,
+# opsgenie://<api_key>.
+# DQ_ALERT_WEBHOOK=https://hooks.slack.com/services/XXX/YYY/ZZZ
+
+# Per-owner routes (JSON) take precedence over the default. Values are either a channel
+# URL or a {"fail": url, "error": url} object for severity-aware routing.
+# DQ_ALERT_ROUTES={"data-platform@example.com": "https://hooks.slack.com/services/A/B/C"}
+
+# SMTP settings for mailto: alert routes.
+# DQ_SMTP_HOST=
+# DQ_SMTP_PORT=587
+# DQ_SMTP_FROM=
+# DQ_SMTP_USER=
+# DQ_SMTP_PASSWORD=
+
+# --- lineage ---------------------------------------------------------------------------
+# Explicit OpenLineage dataset namespace override. Default: derived from the executor's
+# connection (e.g. postgres://host:5432), falling back to "airflow-dq".
+# DQ_OL_NAMESPACE=airflow-dq
+
+# --- contracts git flow (PR-based approval) ---------------------------------------------
+# When true, "propose" saves commit the new contract version on a dq/contract-<id>-v<N>
+# branch instead of writing to the working tree.
+# DQ_CONTRACTS_GIT=false
+# DQ_CONTRACTS_GIT_REMOTE=origin
+
+# --- retention -------------------------------------------------------------------------
+# Days to keep raw check results / metric anomalies (0 = keep forever).
+# DQ_RESULTS_RETENTION_DAYS=0

airflow_dq-0.2.0/.gitattributes

@@ -0,0 +1,7 @@
+# Normalize line endings to LF in the repo (avoids CRLF churn on Windows).
+* text=auto eol=lf
+
+# Treat lockfiles/build output sensibly
+*.png binary
+*.jpg binary
+*.gif binary

airflow_dq-0.2.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,22 @@
+---
+name: Bug report
+about: Something broke or behaves differently than documented
+labels: bug
+---
+
+**What happened / what did you expect?**
+
+**Reproduction**
+- Contract YAML (or minimal snippet):
+- DAG wiring (`build_contract_checks(...)` call):
+- Trigger: scheduled / manual / asset-triggered / retry
+
+**Environment**
+- airflow-dq version:
+- Apache Airflow version:
+- Auth manager (Simple/FAB/other):
+- Backend (postgres / duckdb / experimental adapter):
+- Deployment (docker compose demo / K8s / other):
+
+**Logs / results rows**
+(check task log excerpt; relevant rows from `dq.dq_check_results` if applicable)

airflow_dq-0.2.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,14 @@
+---
+name: Feature request
+about: A capability the framework should have
+labels: enhancement
+---
+
+**Use case** — what are you trying to accomplish, in your pipeline's terms?
+
+**Proposal** — what should airflow-dq do? (rule type, UI surface, API, backend…)
+
+**Workaround today** — how do you solve this now (custom expression rule, external tool, not at all)?
+
+**Context** — see `docs/REFACTOR-ROADMAP.md` for what's already planned; note if this
+matches or contradicts an item there.

airflow_dq-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,84 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+env:
+  AIRFLOW_VERSION: "3.1.0"
+
+jobs:
+  python:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: pyproject.toml
+      - name: Install (with Airflow constraints)
+        # The Airflow constraints file pins the whole transitive dependency set to the
+        # combination the Airflow release was tested with — same install as production.
+        run: |
+          CONSTRAINTS="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${{ matrix.python-version }}.txt"
+          python -m pip install --upgrade pip
+          pip install "apache-airflow==${AIRFLOW_VERSION}" -e ".[dev]" --constraint "$CONSTRAINTS"
+      - name: Lint
+        run: |
+          ruff check src tests
+          mypy src
+      - name: Unit tests (coverage gate 70%)
+        run: pytest --cov=airflow_dq --cov-fail-under=70
+
+  ui:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+          cache: npm
+          cache-dependency-path: ui/package-lock.json
+      - name: Build UI (npm ci + tsc + vite build)
+        run: |
+          cd ui
+          npm ci
+          npm run build   # runs `tsc && vite build`
+
+  # ODCS v3.1 spec compliance: lint the committed fixture AND a freshly emitted contract
+  # with the official datacontract-cli (the fast suite covers the same schema via
+  # tests/test_odcs_compliance.py; this proves it against the real ecosystem linter).
+  odcs-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Install airflow-dq (no Airflow needed) + datacontract-cli
+        run: |
+          pip install pyyaml pydantic sqlalchemy
+          pip install --no-deps -e .
+          pip install datacontract-cli
+      - name: Lint contracts with datacontract-cli
+        run: bash scripts/odcs_lint.sh
+
+  # Real end-to-end test: compose stack up, wait for the api-server, trigger the demo
+  # DAG, assert that check results landed in dq-postgres. The script tears the stack
+  # down (down -v) on exit, pass or fail.
+  integration:
+    runs-on: ubuntu-latest
+    needs: [python]
+    timeout-minutes: 20
+    continue-on-error: false
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run integration test
+        run: bash scripts/integration_test.sh

airflow_dq-0.2.0/.github/workflows/release.yml

@@ -0,0 +1,57 @@
+name: Release
+
+# Publishes to PyPI via trusted publishing (OIDC) — configure the publisher on
+# https://pypi.org/manage/project/airflow-dq/settings/publishing/ with this repo,
+# workflow name "release.yml", environment "pypi". No API token secrets needed.
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: npm
+          cache-dependency-path: ui/package-lock.json
+      - name: Build UI (ships inside the wheel)
+        run: cd ui && npm ci && npm run build
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build sdist + wheel
+        run: |
+          python -m pip install build
+          python -m build
+      - name: Verify wheel contents
+        run: |
+          python - <<'EOF'
+          import zipfile, glob
+          whl = glob.glob("dist/*.whl")[0]
+          names = zipfile.ZipFile(whl).namelist()
+          required = ["airflow_dq/py.typed", "airflow_dq/ui_dist/index.html"]
+          missing = [r for r in required if r not in names]
+          assert not missing, f"wheel is missing {missing}"
+          assert any(n.startswith("airflow_dq/results/migrations/") for n in names), "migrations missing"
+          print(f"{whl}: {len(names)} files OK")
+          EOF
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # trusted publishing
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

airflow_dq-0.2.0/.gitignore

@@ -0,0 +1,38 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Node / React
+node_modules/
+ui/dist/
+ui/.vite/
+
+# Airflow
+logs/
+airflow.db
+airflow.cfg
+standalone_admin_password.txt
+
+# Local data files (DuckDB scratch, etc.)
+*.duckdb
+*.duckdb.wal
+
+# Env / secrets
+.env
+*.local
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/

airflow_dq-0.2.0/CHANGELOG.md

@@ -0,0 +1,51 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
+[Semantic Versioning](https://semver.org/) (0.x: minor bumps may break).
+
+## [0.2.0] - 2026-07-12
+
+### Added
+- **Auth**: Airflow auth-manager JWT validation (Bearer header or `_token` session cookie),
+  verified live against Airflow 3.1.0; legacy `DQ_API_TOKEN` shared secret kept for machine
+  clients; write routes consult `is_authorized_custom_view`.
+- **UI**: three new views on the Airflow-native design system — Overview (per-contract
+  rollup with SLA scorecard + anomaly counts), Quarantine triage (filters, pagination,
+  bulk resolve), and dashboard upgrades (server-flagged anomalies, dag/status filters);
+  the built UI now ships inside the wheel.
+- **SLA checks** compiled from ODCS `slaProperties` (latency, frequency, retention).
+- **Anomaly detection**: leave-current-out baselines for the z-score rule; persisted
+  server-side metric anomalies (`dq_metric_anomalies`) surfaced in trends and alerts.
+- **Alerting**: per-(run, contract) grouped alerts with scheme-routed channels
+  (webhook, `mailto:`, `pagerduty://`, `opsgenie://`) and severity-aware routing.
+- **Contracts**: immutable versions (409 on conflict), atomic writes, registry sync with
+  version timeline API, param/severity-aware breaking-change detection, git propose-as-PR
+  flow (`propose: true`), dbt `schema.yml` import.
+- **Executors**: Airflow Connection resolution, pooled engines with statement timeouts,
+  dialect layer, experimental Snowflake/BigQuery/Databricks/Trino adapters (SQL-generation
+  tested; not yet run against live warehouses).
+- **Results store**: SQL migration runner, idempotent upserts keyed by
+  (dag_id, run_id, task_id, check_name), paginated/filterable results API, retention helper.
+- **OpenLineage**: dataset naming derived from the resolved connection per the OL naming spec.
+- Packaging: `py.typed`, Python 3.10–3.12 classifiers, Airflow-constraints CI matrix,
+  real docker-compose integration test (`scripts/integration_test.sh`, `make integration`).
+
+### Changed (breaking — see README "Upgrading from 0.1.x")
+- `freshness` defaults to `mode: max_age` (table-level newest-row age); the old per-row
+  behavior is `mode: row_age`. The `unit` param (day/hour) is now honored.
+- Executor errors produce `status='error'` results instead of only crashing the task;
+  record-only monitors no longer fail on infra blips.
+- Quarantine no longer fires for tolerated (`pass`-graded) results.
+- Check tasks no longer push result XComs by default (`return_result=True` restores).
+- The API no longer injects a bearer token into the served UI; auth is cookie/JWT based.
+- Contract YAML output follows ODCS v3.1 and is validated by the official JSON Schema and
+  `datacontract lint` (CI job): quality rules are emitted as `custom` engine blocks
+  (`engine: airflow-dq`), column-scoped checks nest under their schema property, and
+  legacy logical types normalize on read (`decimal`→`number`).
+- Trends `rows_failed` is now serialized as an integer (was a JSON string on Postgres).
+
+## [0.1.0] - 2026-07 (unreleased baseline)
+- Initial framework: ODCS-dialect contracts, 12 rule types, dynamic-task-mapped checks,
+  fail/warn/quarantine policies, results store, embedded dashboard + low-code editor,
+  webhook alerting, OpenLineage facets, DuckDB/Postgres executors.

airflow_dq-0.2.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Code of Conduct
+
+This project follows the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+
+In short: be respectful, be constructive, assume good faith. Harassment or personal attacks
+of any kind are not tolerated.
+
+Instances of unacceptable behavior may be reported to the maintainer via GitHub
+(@Fabian-Schneider01). All reports will be reviewed confidentially.

airflow_dq-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing to airflow-dq
+
+Thanks for your interest! This project aims to be the most complete data-quality framework
+that lives *inside* Apache Airflow — issues, ideas and PRs are welcome.
+
+## Development setup
+
+```bash
+make install          # pip install -e '.[dev]'
+make test             # pytest (unit tests, no services needed)
+make lint             # ruff + mypy
+make up               # docker compose demo stack (Airflow 3.1 + seeded Postgres)
+make ui-build         # build the React UI into ui/dist
+make integration      # full end-to-end test against the compose stack
+```
+
+UI development: `cd ui && npm install && npm run dev` — the Vite dev server proxies
+`/dq/api` to the compose stack on `localhost:8080`.
+
+## Ground rules
+
+- **Tests are the bar.** Every behavior change lands with pytest coverage; UI changes must
+  keep `npm run build` (tsc strict) green. CI runs ruff, mypy, pytest (70% coverage gate)
+  and a docker integration test.
+- **Architecture decisions** live in `docs/DESIGN.md` (D1–D16) and
+  `docs/IMPLEMENTATION-SPEC.md` (D-IMPL-1…13). Read them before changing interfaces —
+  PRs that silently deviate will be asked to update the spec first.
+- **UI styling** is Airflow's own design system — see `ui/README.md`. Don't hand-tune
+  colors; component patterns are ports of Airflow's.
+- **SQL safety**: everything spec-derived goes through the guards in
+  `src/airflow_dq/executors/sql.py`. New rule types must keep count/sample/quarantine on
+  a single shared predicate.
+- **Backends**: new executors implement the `CheckExecutor` protocol (run + query +
+  quarantine) and register via the `airflow_dq.executors` entry-point group. Mark them
+  experimental until they've run against a live warehouse.
+
+## Pull requests
+
+1. Fork, branch from `main`, keep PRs focused.
+2. `make test lint` green locally.
+3. Describe the user-visible behavior change; update `CHANGELOG.md` under "Unreleased".
+4. Breaking changes need a README "Upgrading" note.
+
+## Reporting issues
+
+Use the issue templates. For anything security-sensitive, take a look at [SECURITY.md](SECURITY.md) —
+please do not open public issues for vulnerabilities.

airflow_dq-0.2.0/DESIGN.md

@@ -0,0 +1,151 @@
+# DESIGN.md — Decisions & Tradeoffs
+
+This document captures the *why* behind the architecture. It is intentionally the senior-signal artifact:
+"I know the alternatives and I made a deliberate choice."
+
+---
+
+## D1 — Why Dynamic Task Mapping instead of one opaque check task?
+
+Each check becomes its **own visible task** in the DAG graph via `.expand()`, rather than a black-box
+"validate" task. Reviewers and on-call engineers see exactly which check failed in the Airflow grid,
+get per-check retries, and get per-check logs/lineage.
+
+**Tradeoff:** more tasks in the graph and a small mapping overhead. Acceptable — observability wins.
+
+## D2 — Why a pluggable `CheckExecutor` (Protocol) instead of one SQL engine?
+
+We do **not** build a SQL engine. The executor is a `Protocol` with adapters that *generate* SQL or
+*delegate* to a backend:
+
+- `DuckDBExecutor` / `PostgresExecutor` — generate SQL, run everywhere (demo + CI).
+- `DatabricksExecutor` (stretch) — Databricks SQL + Delta, optionally via DQX (enterprise flex).
+
+This single abstraction is the core design decision: *"runs locally on DuckDB for reviewers, scales to
+Databricks/Delta for the real world."*
+
+**Tradeoff:** dialect differences leak into adapters. Contained by keeping `CheckSpec` backend-agnostic
+and pushing dialect into the adapter only.
+
+## D3 — Why ODCS instead of a home-grown contract format?
+
+ODCS (Open Data Contract Standard) is an existing, governed standard. Inventing a format would be
+undifferentiated work and a portability dead-end. Using ODCS plays to real-world experience.
+
+## D4 — Why a separate metadata DB schema, not Airflow's metadata DB?
+
+Contract registry, versions and check-run history live in **their own Postgres schema** — never pollute
+Airflow's internal metadata DB. Keeps upgrades safe and the product portable across Airflow deployments.
+
+## D5 — Why not just use Soda / Great Expectations?
+
+They are checks-as-code in YAML next to the pipeline, frequently backed by external SaaS. The wedge here
+is **visual authoring inside the orchestrator + codegen to real tasks + an integrated dashboard**, fully
+self-hosted OSS. The **breaking-change detector** ("git for data") is specifically the part those tools
+don't cover as Airflow-native OSS — the differentiator.
+
+## D6 — Failure semantics: `fail` | `warn` | `quarantine`
+
+- `fail` — stops the critical pipeline (check task fails, downstream blocked).
+- `warn` — pipeline continues, result recorded as `warn`.
+- `quarantine` — copies failing rows (capped) into `dq.dq_quarantine` for triage; pipeline continues.
+
+## D8 — Tolerances over binary pass/fail (anti alert-fatigue)
+
+Real data is never 100% clean; a check that goes red on a single null gets disabled. A rule may
+carry `warn/failIfGreaterThan` (absolute) or `warn/failIfPercentGreaterThan` (share) thresholds.
+With thresholds, status is graded (pass → warn → fail); without them it falls back to binary-by-
+severity. Example: `amount.valueRange` is `blocking` but tolerated, so 2/7 bad (29%) only
+**warns**, while `freshness` (no tolerance) hard-fails. The single biggest lever against the #1
+reason DQ tools get switched off.
+
+## D9 — Schema-drift is one metadata check, not per-column rules
+
+Upstream renaming a column / changing a type is the silent killer. One `schema` check per dataset
+compares `information_schema` to the contract: missing required columns + coarse type-family
+mismatch (string/number/temporal/boolean), so `decimal` vs `numeric` doesn't false-positive while
+string↔number drift is caught. Also seeds the breaking-change story (stretch).
+
+## D10 — Failures must be actionable
+
+Every check captures a small **sample** of the offending rows inline (one query, same predicate as
+the count, so they can't disagree). The dashboard shows it in an expandable "bad rows" panel —
+actionable triage, not just a red count. Quarantine persists the rows; it does **not** rewrite the
+user's downstream table (owning that write is out of scope).
+
+## D11 — Low-code authoring: form library + one expression escape hatch + live preview
+
+The editor is no-code for the closed rule library (pick column + rule + params; columns are
+introspected from the live table) and low-code via a single `expression` rule — a boolean SQL
+predicate that flags bad rows. Crucially, that predicate plugs straight into `failing_predicate`,
+so count, bad-row sample and quarantine come **for free**; no parallel code path. We deliberately
+do *not* build a drag-drop block engine (the form already is no-code) nor accept arbitrary full
+`SELECT`s (breaks portability + safety). The differentiator is **live preview**: every block has a
+"Test" button that runs against the real table and shows pass/fail + sample *before saving* —
+interactive authoring no other in-Airflow OSS does. Expressions are guarded (identifiers only,
+read-only predicate, no DDL/DML); trust model = data engineers, same as dbt tests.
+
+## D12 — One contract, two deployment patterns (inline gate + asset-triggered monitor)
+
+The unit is the contract-per-dataset, not the DAG. The same contract is consumed two ways:
+(A) **inline in the producer DAG** as a gate (`on_fail="quarantine"`, blocks/quarantines before
+data flows on); and (C) an **asset-triggered monitor** DAG (`dq_monitor`) that `schedule=[asset]`
+and runs whenever the table is reproduced (`on_fail="warn"`, record-only). The producer marks the
+table updated via `outlets=[orders_asset]`. This uses Airflow 3 **data-aware scheduling** so DQ
+coverage doesn't require wiring checks into every pipeline — and it makes "gate + observability
+from one definition" literal: both feed the same results store / dashboard. (A central scheduled
+"sweep" DAG is the third option, better when there's no producer to hang the asset off.)
+
+## D13 — Authoring intelligence: profile to draft, diff to warn
+
+Two features make the contract editor more than a form. **Profiling** points at a live table and
+suggests a starter draft (fully-populated → notNull, distinct==rows → unique, numeric → valueRange
+from observed min/max, low-cardinality string → acceptedValues) — you refine from a draft, not a
+blank page. **Breaking-change detection** diffs the new version against the previous one on save
+and classifies each change (column removed / type changed / new required / uniqueness added =
+breaking; optional add / relaxed constraint / rule change = info), surfacing a red warning in the
+editor. The detector reuses the schema type-family logic (D9); it's the "git for data" wedge that
+Soda/GE don't ship as Airflow-native OSS.
+
+## D14 — Anomaly detection: statistical, push-down, two levels
+
+Anomaly detection here is **unsupervised and statistical** (z-score / baseline), not deep
+learning — the same family as Databricks Lakehouse Monitoring, and honest about it. Two levels:
+(1) **row-level** — the `anomaly` rule flags rows beyond `k·σ` of the column mean via pure SQL
+(`ABS(x - (SELECT avg(x))) > k * (SELECT stddev_samp(x))`), so it scales (no data pulled into
+Python) and reuses the sample/quarantine machinery to surface the weird records; (2) **metric-
+level** — the dashboard's "quality over time" chart flags a *run* whose failing-row count exceeds
+the historical mean + 2σ. A heavier model (e.g. IsolationForest) is a drop-in extension behind the
+same rule, with the explicit trade-off that it pulls a sample into Python.
+
+## D15 — Incremental by scope, backfill-safe by logical date
+
+Real tables are too big for full scans on every run, and backfills must not false-alarm. Two
+mechanisms: (1) every row-level check accepts a guarded `where` scope (partition filter) that
+count, sample and quarantine all share — check only the new slice, compare anomaly baselines
+against full history; (2) freshness evaluates against the run's `logical_date` (injected by the
+operator as `asOfDate`), so a backfill for last month doesn't flag last month's data as stale.
+
+## D16 — Lineage: OpenLineage facets, never in the critical path
+
+The check operator implements `get_openlineage_facets_on_complete` and emits each result as
+`DataQualityAssertionsDatasetFacet` (+ `DataQualityMetricsInputDatasetFacet`) on the checked
+dataset — lineage UIs (Marquez etc.) then show "the pipeline ran AND the dataset met its
+expectations". The mapping core is pure Python (unit-testable without Airflow); the client
+conversion uses defensive imports and returns None on any failure — lineage must never break a
+pipeline. Dialect note: SQL generation now takes a `dialect` parameter (regex is the only split
+so far), and all free-form predicates/identifiers/numeric params are guarded at SQL-build time
+(defense in depth beyond the authoring-side validation).
+
+## D7 — Auth is explicit, on purpose
+
+Airflow 3 FastAPI plugin endpoints are not auto-protected by Airflow auth. We secure them explicitly
+(token validated against Airflow's auth manager). A consciously handled auth concern is itself a good signal.
+
+---
+
+## Open questions / parked
+
+- Exact ODCS v3 rule coverage in the MVP form (~6 most common check types).
+- OpenLineage facet shape for check results (stretch).
+- Whether the dashboard also feeds an external BI tool (optional, not the centerpiece).