det-elt 0.1.0__tar.gz

det_elt-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kadeem
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

det_elt-0.1.0/PKG-INFO

@@ -0,0 +1,349 @@
+Metadata-Version: 2.4
+Name: det-elt
+Version: 0.1.0
+Summary: DET — Data Extract Tool: config-driven extraction runtime with pluggable ingestion
+Author-email: Kadeem <kletts2000@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/codekid/det
+Project-URL: Repository, https://github.com/codekid/det
+Project-URL: Issues, https://github.com/codekid/det/issues
+Project-URL: Documentation, https://github.com/codekid/det/blob/main/docs/getting-started-library.md
+Keywords: etl,data-extraction,pipeline,iceberg,bronze
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: <3.14,>=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: jsonschema>=4.23.0
+Requires-Dist: pendulum>=3.0.0
+Requires-Dist: pydantic>=2.10.0
+Requires-Dist: pyyaml>=6.0.0
+Requires-Dist: requests>=2.32.0
+Requires-Dist: structlog>=25.1.0
+Requires-Dist: typer>=0.15.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.3.0; extra == "dev"
+Requires-Dist: pytest-cov>=6.0.0; extra == "dev"
+Requires-Dist: ruff>=0.9.0; extra == "dev"
+Provides-Extra: examples
+Requires-Dist: beautifulsoup4>=4.12.0; extra == "examples"
+Requires-Dist: dlt[filesystem]>=1.24.0; extra == "examples"
+Provides-Extra: duckdb
+Requires-Dist: duckdb>=1.5.1; extra == "duckdb"
+Provides-Extra: scaffold
+Requires-Dist: jinja2>=3.1.0; extra == "scaffold"
+Provides-Extra: dbt
+Requires-Dist: dbt-core>=1.9; extra == "dbt"
+Requires-Dist: dbt-duckdb>=1.9; extra == "dbt"
+Provides-Extra: bigquery
+Requires-Dist: dbt-bigquery>=1.9; extra == "bigquery"
+Requires-Dist: google-cloud-bigquery>=3.0; extra == "bigquery"
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
+Provides-Extra: s3
+Requires-Dist: s3fs>=2024.0.0; extra == "s3"
+Provides-Extra: gcs
+Requires-Dist: gcsfs>=2024.0.0; extra == "gcs"
+Provides-Extra: iceberg
+Requires-Dist: pyiceberg>=0.8; extra == "iceberg"
+Requires-Dist: pyarrow>=15; extra == "iceberg"
+Provides-Extra: mcp
+Requires-Dist: mcp<2,>=1.2; extra == "mcp"
+Dynamic: license-file
+
+# DET — Data Extract Tool
+
+Extract → **raw** (wire bytes) → **bronze** (typed Iceberg). **dbt** owns silver and gold.
+**dlt never lands bronze** — it may help HTTP; DET owns validation, meta, and writers.
+
+## Choose your path
+
+| Audience | Start here |
+| --- | --- |
+| **Operator** (this monorepo: CLI, examples, dbt, Airflow, MCP) | [Try it](#try-it-fixtures-no-noaa-download) below |
+| **Library** (embed extract→raw→bronze in your app) | **[docs/getting-started-library.md](docs/getting-started-library.md)** · [docs/api.md](docs/api.md) |
+
+This checkout is the operator product **and** the library source. Embedders install
+the PyPI package **`det-elt`** (import `det`) and keep pipelines under
+*their* `project_root` — they do not need this monorepo’s dbt/Airflow layout.
+
+---
+
+## Try it (fixtures, no NOAA download)
+
+Python 3.12+ and [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv venv && make install          # recommended extras (includes iceberg + examples)
+export DET_LAKE_PATH="$PWD/data/lake"
+
+uv run det run -p noaa.storm_events -s 2026-08-06 \
+  --set source.overrides.local_csv_dir=fixtures/storm_events \
+  --set source.overrides.filename_substr=details
+
+uv run det dbt -p noaa.storm_events
+duckdb data/analytics.duckdb -c "select * from gold.gold_yearly_damage"
+```
+
+You should see:
+
+- Raw: `data/lake/raw/noaa/storm_events_v1/…/data/` + `meta/manifest.json`
+- Bronze: Iceberg table at `data/lake/bronze/noaa/storm_events_v1/`
+- Silver/gold in `data/analytics.duckdb`
+
+`make run-local` is a **thin JSONL** smoke (`destination.type=filesystem`). It does not
+feed Iceberg `iceberg_scan` in dbt. Use the commands above for the full path, or
+`make all` only if you intend JSONL bronze.
+
+Use `uv run det …` (or `make …`). If macOS hides the editable `.pth`, `make unhide` or
+`PYTHONPATH=src .venv/bin/det …`.
+
+---
+
+## Mental model
+
+```mermaid
+flowchart LR
+  src[Source plugin] --> extract
+  extract --> raw[Raw lake]
+  raw --> load
+  load --> bronze[Iceberg bronze]
+  bronze --> stg[dbt stg]
+  stg --> silver --> gold
+```
+
+| Layer | Owner | Rule |
+| --- | --- | --- |
+| Raw | DET | Append extract runs; rebuild source of truth |
+| Bronze | DET | Typed landing; prune bronze only, never raw |
+| Silver / gold | dbt | Dedupe and marts; latest extract wins |
+
+**Interval:** `-s` inclusive start, `-e` exclusive end (default start + 1 day).
+`det run` shares one `__extract_run_datetime` across raw, bronze, and rows.
+
+---
+
+## Install extras
+
+Base `det` is the runtime + CLI (no Iceberg / DuckDB / example HTTP deps).
+**Recommended first line:** install with the `iceberg` extra.
+
+| Extra | For |
+| --- | --- |
+| `iceberg` | **Recommended** — default lake bronze |
+| `examples` | In-tree HTTP sources (`dlt`, BeautifulSoup) |
+| `duckdb` | DuckDB destination / DuckDB-backed prune |
+| `scaffold` | Jinja2 dbt/pipeline scaffolding |
+| `dbt` | `det dbt` |
+| `postgres` | SQL serving destination |
+| `s3` / `gcs` | Object-store lake URI |
+| `mcp` | Cursor inspect / dry-run server |
+| `dev` | pytest, ruff |
+
+```bash
+# App / embed (PyPI):
+uv pip install "det-elt[iceberg]"
+uv pip install "det-elt[iceberg,duckdb,dbt]"
+# This checkout (editable):
+uv pip install -e ".[iceberg]"              # recommended
+uv pip install -e ".[iceberg,duckdb,dbt]"   # typical app repo
+make install                                # full extras for operator dev
+make test
+uv run det check
+```
+
+---
+
+## Pipeline YAML
+
+Canonical id is `provider.source` (`-p noaa.storm_events`). Defaults live in the
+plugin; YAML wires schema, destination, and optional dbt knobs. Plugins are
+discovered from `src/det/sources/<provider>/<source>.py` (`name` must equal
+`provider.source`) — they are not listed in `plugins.py`. Out-of-tree packages may
+use entry points `det.sources` / `det.mappers`.
+
+```yaml
+name: noaa.storm_events
+source:
+  type: noaa.storm_events
+destination:
+  type: iceberg          # default lake; filesystem = JSONL; duckdb / postgres = SQL
+  partition: extract_run # Iceberg only; omit = extract_run; small tables: none
+wire_version: 1          # lake id is always {name}_vN (including _v1)
+```
+
+Schema defaults to `schemas/<provider>/<source>/<source>.schema.yaml`.
+Greenfield: `det init-pipeline --name example_api.events --source-type example_api.events`.
+
+---
+
+## Destinations
+
+Lake root: `DET_LAKE_PATH` / `--lake-path` (default `./data/lake`). Same hive under
+one root (`raw/` + `bronze/` prefixes) — dual buckets are not supported. There is
+no `destination.type: s3`.
+
+**`DET_LAKE_MODE`** (policy around the URI; unset → `local`):
+
+| Mode | Allowed lake | Typical use |
+| --- | --- | --- |
+| `local` | filesystem path or `memory://` (tests) | laptop, CI default suite, Compose default |
+| `cloud` | `s3://…` or `gs://…` / `gcs://…` | object store; CI MinIO soak covers S3 extract→Iceberg→`iceberg_scan`/`det dbt`; GCS soak covers extract→Iceberg (PyIceberg) |
+
+`--lake-path` cannot bypass mode. `det check` errors on mismatch and warns when
+`mode=cloud`. Compose: `DET_LAKE_MODE` + overridable `DET_LAKE_PATH` (see
+`airflow/.env.example`). Local analytics/ops DuckDB stay on the worker filesystem.
+Prod on GCS: BigQuery reads Iceberg bronze via **BigLake** — see
+[docs/gcp-biglake.md](docs/gcp-biglake.md) (architecture C).
+
+| `destination.type` | Bronze |
+| --- | --- |
+| **`iceberg`** | Default. Parquet table at `<lake>/bronze/<provider>/<source>_vN/` |
+| `filesystem` | Hive JSONL (thin / fixtures). Cannot share that path with Iceberg |
+| `duckdb` | `bronze_{provider}.{source}_vN` — needs `connection` |
+| `postgres` | Same SQL names — `connection_env: DET_POSTGRES_DSN` (never a DSN in YAML) |
+
+There is **no** `destination.type: bigquery`. On `gs://`, bronze stays Iceberg;
+BQ is a reader (BigLake), not a DET lander.
+
+Iceberg-only: `destination.partition` is `extract_run` (default — identity on
+`__extract_run_datetime` for load replace / silver watermark) or `none`
+(unpartitioned; use for small tables). Spec applies on **create** only. Live
+mismatch **hard-fails** until `det migrate … --recreate-iceberg` (full table
+purge, then rewrite `-s`/`-e` or `--all-raw`; latest raw per interval unless
+`--all-raw-runs`) or a manual wipe. Not the raw hive layout.
+
+`det dbt -p …` sets `DET_BRONZE_SOURCE` from the pipeline (`iceberg` → `iceberg_scan`
+in `sources.yml` for DuckDB). On `s3://` lakes, `det dbt` auto-selects profile
+target `duckdb_s3` (httpfs + S3 secret from the same `AWS_*` as extract/load).
+On `gs://`, it does **not** force DuckDB S3 — set `DET_DBT_TARGET=bigquery` /
+`--target bigquery` for BigLake-backed silver/gold/ops ([docs/gcp-biglake.md](docs/gcp-biglake.md)).
+Register BigLake tables with `det biglake-register` (dry-run → `det approve` → `--apply`).
+See [docs/gcp-biglake.md](docs/gcp-biglake.md) for connection IAM and sandbox teardown.
+Layout contract: [docs/lake-layout.md](docs/lake-layout.md).
+
+---
+
+## Everyday CLI
+
+`-p` is a canonical id, `provider/source`, or a YAML path. Project root:
+`--project-root` > `DET_PROJECT_ROOT` > cwd.
+
+```bash
+det run -p noaa.storm_events -s 2026-08-06
+det extract -p noaa.storm_events -s 2026-08-06
+det load -p noaa.storm_events -s 2026-08-06
+det dbt -p noaa.storm_events
+det check          # CI runs det check --strict (warnings fail the build)
+det list-pipelines
+det list-sources
+```
+
+| Command | When |
+| --- | --- |
+| `det prune -p … -s … --keep 1 --dry-run` then `--apply` | Drop old bronze extract-run siblings (never raw) |
+| `det migrate -p … --to-bronze … --schema … --mapper identity -s … -e …` | Rebuild bronze from raw after a contract change |
+| `det scaffold-dbt -p …` | Emit stg/silver + `sources.yml` |
+| `det runs` / `det runs-materialize` | Attempt receipts; optional ops Iceberg + `det dbt --select tag:ops` |
+| `det lock-show` / `lock-release --force` | Lake lease on `(pipeline, interval)` if a worker died |
+
+Logs: console on a laptop TTY, JSON off TTY (`DET_LOG_FORMAT`). Secrets: names in
+YAML (`auth_env`, `connection_env`); values in env. `det check` fails passwordful DSNs
+in committed config.
+
+---
+
+## dbt
+
+```bash
+det dbt -p noaa.storm_events    # stg_<provider>__<source>+ ; sets lake env
+det dbt                         # full analytics project (excludes tag:ops)
+```
+
+Scaffolded `stg_*` use `det_bronze_from`. Gold is hand-written. Nested flatten /
+relations: `dbt.stg` in pipeline YAML, then `det scaffold-dbt -p …`. Contract
+triangle (schema → sources.yml → dbt.stg): [docs/contract-triangle.md](docs/contract-triangle.md).
+More: [`.cursor/skills/det-dbt/SKILL.md`](.cursor/skills/det-dbt/SKILL.md).
+
+---
+
+## Local Airflow (dev only)
+
+```bash
+make airflow-up          # http://localhost:8080  — airflow / airflow
+make airflow-down
+```
+
+Compose is **LocalExecutor**, not production. DAGs: extract → load, silver/gold dbt,
+ops receipts — decoupled. Env: `airflow/.env.example`. Guide:
+[`.cursor/skills/det-airflow/SKILL.md`](.cursor/skills/det-airflow/SKILL.md).
+
+---
+
+## Cube (local semantic layer)
+
+Gold and ops metrics for agents. Not Cube Cloud MCP.
+
+```bash
+make cube-up             # http://localhost:4000
+make cube-down
+```
+
+Compose reads `data/analytics.duckdb` (`yearly_damage`) and `data/det_ops.duckdb`
+(`run_daily`). Copy `cube/.env.example` → `cube/.env`. MCP: `cube_meta` /
+`cube_load` (`DET_CUBE_BASE_URL`, `DET_CUBE_API_SECRET`). Do not run `det dbt`
+against a DuckDB file Cube has open. Certified metrics go through Cube; silver
+or ops row detail uses MCP `query_analytics`.
+
+---
+
+## MCP (Cursor)
+
+Read-only inspect + dry-run (no extract/load/prune-apply/DagRuns).
+`uv pip install -e ".[mcp]"`; [`.cursor/mcp.json`](.cursor/mcp.json) already launches it.
+Agent contract: [AGENTS.md](AGENTS.md). Tools + policy:
+[`.cursor/rules/det-mcp.mdc`](.cursor/rules/det-mcp.mdc).
+
+---
+
+## Repo map
+
+```text
+src/det/                 CLI, runtime, example sources, writers, det.testing
+configs/pipelines/       provider.source YAML
+schemas/                 bronze JSON Schema
+docs/api.md              public Python API (SemVer / __all__)
+docs/getting-started-library.md  embedder first hour
+docs/lake-layout.md      hive / SQL compatibility
+docs/gcp-biglake.md      gs:// Iceberg + BigLake + dbt-BQ (architecture C)
+dbt/                     silver + gold + ops
+cube/                    local Cube Core (gold + ops metrics)
+dags/ + airflow/         local Compose
+fixtures/                offline NOAA CSVs
+```
+
+Example sources implement `extract_to_raw` / `records_from_raw`. Land near-wire bytes;
+unexpected fields fail JSON Schema. Analytics renames live in `dbt.stg`. True wire
+breaks bump `wire_version`. dlt: `RESTClient` / `@dlt.resource` as iterators only —
+never `dlt.pipeline` for landing.
+
+---
+
+## Troubleshooting
+
+| Symptom | Fix |
+| --- | --- |
+| `No module named 'det'` | `make unhide` or `PYTHONPATH=src .venv/bin/det` (macOS hidden `.pth`) |
+| `dbt CLI not found` | `uv run det dbt` so the venv `dbt` is on PATH |
+| `No raw partitions` | Load the **same** `-s`/`-e` you extracted |
+| Iceberg partition YAML ≠ live | `det migrate … --recreate-iceberg` (or wipe the bronze table path); plain load/migrate hard-fails |
+| Iceberg dbt vs JSONL lake | Don’t mix `make run-local` (filesystem) with `det dbt -p` (pipeline is iceberg) |
+| DuckDB lock in Airflow | Absolute `DET_ANALYTICS_DUCKDB`; one dbt process at a time |
+| Cube MCP `cube_unavailable` | `make cube-up`; copy `cube/.env.example` → `cube/.env` |
+| DuckDB lock with Cube | Do not `det dbt` while Cube has that file open |
+| `DET_LAKE_MODE=local forbids…` / `requires an s3://` | Align mode and path: local + filesystem, or cloud + `s3://`/`gs://` |
+| GCS soak / localgcp | Set `STORAGE_EMULATOR_HOST` (`http://127.0.0.1:4443`), `GOOGLE_CLOUD_PROJECT`, `DET_GCS_BUCKET`; `pytest -m gcs`. BigLake needs a real GCP project — see [docs/gcp-biglake.md](docs/gcp-biglake.md) |

det_elt-0.1.0/README.md

@@ -0,0 +1,292 @@
+# DET — Data Extract Tool
+
+Extract → **raw** (wire bytes) → **bronze** (typed Iceberg). **dbt** owns silver and gold.
+**dlt never lands bronze** — it may help HTTP; DET owns validation, meta, and writers.
+
+## Choose your path
+
+| Audience | Start here |
+| --- | --- |
+| **Operator** (this monorepo: CLI, examples, dbt, Airflow, MCP) | [Try it](#try-it-fixtures-no-noaa-download) below |
+| **Library** (embed extract→raw→bronze in your app) | **[docs/getting-started-library.md](docs/getting-started-library.md)** · [docs/api.md](docs/api.md) |
+
+This checkout is the operator product **and** the library source. Embedders install
+the PyPI package **`det-elt`** (import `det`) and keep pipelines under
+*their* `project_root` — they do not need this monorepo’s dbt/Airflow layout.
+
+---
+
+## Try it (fixtures, no NOAA download)
+
+Python 3.12+ and [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv venv && make install          # recommended extras (includes iceberg + examples)
+export DET_LAKE_PATH="$PWD/data/lake"
+
+uv run det run -p noaa.storm_events -s 2026-08-06 \
+  --set source.overrides.local_csv_dir=fixtures/storm_events \
+  --set source.overrides.filename_substr=details
+
+uv run det dbt -p noaa.storm_events
+duckdb data/analytics.duckdb -c "select * from gold.gold_yearly_damage"
+```
+
+You should see:
+
+- Raw: `data/lake/raw/noaa/storm_events_v1/…/data/` + `meta/manifest.json`
+- Bronze: Iceberg table at `data/lake/bronze/noaa/storm_events_v1/`
+- Silver/gold in `data/analytics.duckdb`
+
+`make run-local` is a **thin JSONL** smoke (`destination.type=filesystem`). It does not
+feed Iceberg `iceberg_scan` in dbt. Use the commands above for the full path, or
+`make all` only if you intend JSONL bronze.
+
+Use `uv run det …` (or `make …`). If macOS hides the editable `.pth`, `make unhide` or
+`PYTHONPATH=src .venv/bin/det …`.
+
+---
+
+## Mental model
+
+```mermaid
+flowchart LR
+  src[Source plugin] --> extract
+  extract --> raw[Raw lake]
+  raw --> load
+  load --> bronze[Iceberg bronze]
+  bronze --> stg[dbt stg]
+  stg --> silver --> gold
+```
+
+| Layer | Owner | Rule |
+| --- | --- | --- |
+| Raw | DET | Append extract runs; rebuild source of truth |
+| Bronze | DET | Typed landing; prune bronze only, never raw |
+| Silver / gold | dbt | Dedupe and marts; latest extract wins |
+
+**Interval:** `-s` inclusive start, `-e` exclusive end (default start + 1 day).
+`det run` shares one `__extract_run_datetime` across raw, bronze, and rows.
+
+---
+
+## Install extras
+
+Base `det` is the runtime + CLI (no Iceberg / DuckDB / example HTTP deps).
+**Recommended first line:** install with the `iceberg` extra.
+
+| Extra | For |
+| --- | --- |
+| `iceberg` | **Recommended** — default lake bronze |
+| `examples` | In-tree HTTP sources (`dlt`, BeautifulSoup) |
+| `duckdb` | DuckDB destination / DuckDB-backed prune |
+| `scaffold` | Jinja2 dbt/pipeline scaffolding |
+| `dbt` | `det dbt` |
+| `postgres` | SQL serving destination |
+| `s3` / `gcs` | Object-store lake URI |
+| `mcp` | Cursor inspect / dry-run server |
+| `dev` | pytest, ruff |
+
+```bash
+# App / embed (PyPI):
+uv pip install "det-elt[iceberg]"
+uv pip install "det-elt[iceberg,duckdb,dbt]"
+# This checkout (editable):
+uv pip install -e ".[iceberg]"              # recommended
+uv pip install -e ".[iceberg,duckdb,dbt]"   # typical app repo
+make install                                # full extras for operator dev
+make test
+uv run det check
+```
+
+---
+
+## Pipeline YAML
+
+Canonical id is `provider.source` (`-p noaa.storm_events`). Defaults live in the
+plugin; YAML wires schema, destination, and optional dbt knobs. Plugins are
+discovered from `src/det/sources/<provider>/<source>.py` (`name` must equal
+`provider.source`) — they are not listed in `plugins.py`. Out-of-tree packages may
+use entry points `det.sources` / `det.mappers`.
+
+```yaml
+name: noaa.storm_events
+source:
+  type: noaa.storm_events
+destination:
+  type: iceberg          # default lake; filesystem = JSONL; duckdb / postgres = SQL
+  partition: extract_run # Iceberg only; omit = extract_run; small tables: none
+wire_version: 1          # lake id is always {name}_vN (including _v1)
+```
+
+Schema defaults to `schemas/<provider>/<source>/<source>.schema.yaml`.
+Greenfield: `det init-pipeline --name example_api.events --source-type example_api.events`.
+
+---
+
+## Destinations
+
+Lake root: `DET_LAKE_PATH` / `--lake-path` (default `./data/lake`). Same hive under
+one root (`raw/` + `bronze/` prefixes) — dual buckets are not supported. There is
+no `destination.type: s3`.
+
+**`DET_LAKE_MODE`** (policy around the URI; unset → `local`):
+
+| Mode | Allowed lake | Typical use |
+| --- | --- | --- |
+| `local` | filesystem path or `memory://` (tests) | laptop, CI default suite, Compose default |
+| `cloud` | `s3://…` or `gs://…` / `gcs://…` | object store; CI MinIO soak covers S3 extract→Iceberg→`iceberg_scan`/`det dbt`; GCS soak covers extract→Iceberg (PyIceberg) |
+
+`--lake-path` cannot bypass mode. `det check` errors on mismatch and warns when
+`mode=cloud`. Compose: `DET_LAKE_MODE` + overridable `DET_LAKE_PATH` (see
+`airflow/.env.example`). Local analytics/ops DuckDB stay on the worker filesystem.
+Prod on GCS: BigQuery reads Iceberg bronze via **BigLake** — see
+[docs/gcp-biglake.md](docs/gcp-biglake.md) (architecture C).
+
+| `destination.type` | Bronze |
+| --- | --- |
+| **`iceberg`** | Default. Parquet table at `<lake>/bronze/<provider>/<source>_vN/` |
+| `filesystem` | Hive JSONL (thin / fixtures). Cannot share that path with Iceberg |
+| `duckdb` | `bronze_{provider}.{source}_vN` — needs `connection` |
+| `postgres` | Same SQL names — `connection_env: DET_POSTGRES_DSN` (never a DSN in YAML) |
+
+There is **no** `destination.type: bigquery`. On `gs://`, bronze stays Iceberg;
+BQ is a reader (BigLake), not a DET lander.
+
+Iceberg-only: `destination.partition` is `extract_run` (default — identity on
+`__extract_run_datetime` for load replace / silver watermark) or `none`
+(unpartitioned; use for small tables). Spec applies on **create** only. Live
+mismatch **hard-fails** until `det migrate … --recreate-iceberg` (full table
+purge, then rewrite `-s`/`-e` or `--all-raw`; latest raw per interval unless
+`--all-raw-runs`) or a manual wipe. Not the raw hive layout.
+
+`det dbt -p …` sets `DET_BRONZE_SOURCE` from the pipeline (`iceberg` → `iceberg_scan`
+in `sources.yml` for DuckDB). On `s3://` lakes, `det dbt` auto-selects profile
+target `duckdb_s3` (httpfs + S3 secret from the same `AWS_*` as extract/load).
+On `gs://`, it does **not** force DuckDB S3 — set `DET_DBT_TARGET=bigquery` /
+`--target bigquery` for BigLake-backed silver/gold/ops ([docs/gcp-biglake.md](docs/gcp-biglake.md)).
+Register BigLake tables with `det biglake-register` (dry-run → `det approve` → `--apply`).
+See [docs/gcp-biglake.md](docs/gcp-biglake.md) for connection IAM and sandbox teardown.
+Layout contract: [docs/lake-layout.md](docs/lake-layout.md).
+
+---
+
+## Everyday CLI
+
+`-p` is a canonical id, `provider/source`, or a YAML path. Project root:
+`--project-root` > `DET_PROJECT_ROOT` > cwd.
+
+```bash
+det run -p noaa.storm_events -s 2026-08-06
+det extract -p noaa.storm_events -s 2026-08-06
+det load -p noaa.storm_events -s 2026-08-06
+det dbt -p noaa.storm_events
+det check          # CI runs det check --strict (warnings fail the build)
+det list-pipelines
+det list-sources
+```
+
+| Command | When |
+| --- | --- |
+| `det prune -p … -s … --keep 1 --dry-run` then `--apply` | Drop old bronze extract-run siblings (never raw) |
+| `det migrate -p … --to-bronze … --schema … --mapper identity -s … -e …` | Rebuild bronze from raw after a contract change |
+| `det scaffold-dbt -p …` | Emit stg/silver + `sources.yml` |
+| `det runs` / `det runs-materialize` | Attempt receipts; optional ops Iceberg + `det dbt --select tag:ops` |
+| `det lock-show` / `lock-release --force` | Lake lease on `(pipeline, interval)` if a worker died |
+
+Logs: console on a laptop TTY, JSON off TTY (`DET_LOG_FORMAT`). Secrets: names in
+YAML (`auth_env`, `connection_env`); values in env. `det check` fails passwordful DSNs
+in committed config.
+
+---
+
+## dbt
+
+```bash
+det dbt -p noaa.storm_events    # stg_<provider>__<source>+ ; sets lake env
+det dbt                         # full analytics project (excludes tag:ops)
+```
+
+Scaffolded `stg_*` use `det_bronze_from`. Gold is hand-written. Nested flatten /
+relations: `dbt.stg` in pipeline YAML, then `det scaffold-dbt -p …`. Contract
+triangle (schema → sources.yml → dbt.stg): [docs/contract-triangle.md](docs/contract-triangle.md).
+More: [`.cursor/skills/det-dbt/SKILL.md`](.cursor/skills/det-dbt/SKILL.md).
+
+---
+
+## Local Airflow (dev only)
+
+```bash
+make airflow-up          # http://localhost:8080  — airflow / airflow
+make airflow-down
+```
+
+Compose is **LocalExecutor**, not production. DAGs: extract → load, silver/gold dbt,
+ops receipts — decoupled. Env: `airflow/.env.example`. Guide:
+[`.cursor/skills/det-airflow/SKILL.md`](.cursor/skills/det-airflow/SKILL.md).
+
+---
+
+## Cube (local semantic layer)
+
+Gold and ops metrics for agents. Not Cube Cloud MCP.
+
+```bash
+make cube-up             # http://localhost:4000
+make cube-down
+```
+
+Compose reads `data/analytics.duckdb` (`yearly_damage`) and `data/det_ops.duckdb`
+(`run_daily`). Copy `cube/.env.example` → `cube/.env`. MCP: `cube_meta` /
+`cube_load` (`DET_CUBE_BASE_URL`, `DET_CUBE_API_SECRET`). Do not run `det dbt`
+against a DuckDB file Cube has open. Certified metrics go through Cube; silver
+or ops row detail uses MCP `query_analytics`.
+
+---
+
+## MCP (Cursor)
+
+Read-only inspect + dry-run (no extract/load/prune-apply/DagRuns).
+`uv pip install -e ".[mcp]"`; [`.cursor/mcp.json`](.cursor/mcp.json) already launches it.
+Agent contract: [AGENTS.md](AGENTS.md). Tools + policy:
+[`.cursor/rules/det-mcp.mdc`](.cursor/rules/det-mcp.mdc).
+
+---
+
+## Repo map
+
+```text
+src/det/                 CLI, runtime, example sources, writers, det.testing
+configs/pipelines/       provider.source YAML
+schemas/                 bronze JSON Schema
+docs/api.md              public Python API (SemVer / __all__)
+docs/getting-started-library.md  embedder first hour
+docs/lake-layout.md      hive / SQL compatibility
+docs/gcp-biglake.md      gs:// Iceberg + BigLake + dbt-BQ (architecture C)
+dbt/                     silver + gold + ops
+cube/                    local Cube Core (gold + ops metrics)
+dags/ + airflow/         local Compose
+fixtures/                offline NOAA CSVs
+```
+
+Example sources implement `extract_to_raw` / `records_from_raw`. Land near-wire bytes;
+unexpected fields fail JSON Schema. Analytics renames live in `dbt.stg`. True wire
+breaks bump `wire_version`. dlt: `RESTClient` / `@dlt.resource` as iterators only —
+never `dlt.pipeline` for landing.
+
+---
+
+## Troubleshooting
+
+| Symptom | Fix |
+| --- | --- |
+| `No module named 'det'` | `make unhide` or `PYTHONPATH=src .venv/bin/det` (macOS hidden `.pth`) |
+| `dbt CLI not found` | `uv run det dbt` so the venv `dbt` is on PATH |
+| `No raw partitions` | Load the **same** `-s`/`-e` you extracted |
+| Iceberg partition YAML ≠ live | `det migrate … --recreate-iceberg` (or wipe the bronze table path); plain load/migrate hard-fails |
+| Iceberg dbt vs JSONL lake | Don’t mix `make run-local` (filesystem) with `det dbt -p` (pipeline is iceberg) |
+| DuckDB lock in Airflow | Absolute `DET_ANALYTICS_DUCKDB`; one dbt process at a time |
+| Cube MCP `cube_unavailable` | `make cube-up`; copy `cube/.env.example` → `cube/.env` |
+| DuckDB lock with Cube | Do not `det dbt` while Cube has that file open |
+| `DET_LAKE_MODE=local forbids…` / `requires an s3://` | Align mode and path: local + filesystem, or cloud + `s3://`/`gs://` |
+| GCS soak / localgcp | Set `STORAGE_EMULATOR_HOST` (`http://127.0.0.1:4443`), `GOOGLE_CLOUD_PROJECT`, `DET_GCS_BUCKET`; `pytest -m gcs`. BigLake needs a real GCP project — see [docs/gcp-biglake.md](docs/gcp-biglake.md) |