local-data-platform 0.1.3__tar.gz → 0.1.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (143) hide show
  1. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/PKG-INFO +535 -34
  2. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/README.md +534 -33
  3. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/pyproject.toml +1 -1
  4. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/__init__.py +1 -1
  5. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/cli.py +36 -13
  6. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/config.py +32 -3
  7. local_data_platform-0.1.5/src/local_data_platform/contracts.py +479 -0
  8. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/datasets.py +74 -8
  9. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/build.py +2 -2
  10. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/project.py +14 -1
  11. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/publish.py +73 -3
  12. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/sources.py +41 -3
  13. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/engine/duckdb/__init__.py +4 -4
  14. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/events.py +235 -34
  15. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/exceptions.py +17 -0
  16. local_data_platform-0.1.5/src/local_data_platform/gold/__init__.py +9 -0
  17. local_data_platform-0.1.5/src/local_data_platform/gold/build.py +430 -0
  18. local_data_platform-0.1.5/src/local_data_platform/gold/catalog.py +595 -0
  19. local_data_platform-0.1.5/src/local_data_platform/gold/cli.py +428 -0
  20. local_data_platform-0.1.5/src/local_data_platform/gold/release.py +698 -0
  21. local_data_platform-0.1.5/src/local_data_platform/gold/render.py +414 -0
  22. local_data_platform-0.1.5/src/local_data_platform/gold/serve.py +860 -0
  23. local_data_platform-0.1.5/src/local_data_platform/gold/status.py +274 -0
  24. local_data_platform-0.1.5/src/local_data_platform/gold/sync.py +133 -0
  25. local_data_platform-0.1.5/src/local_data_platform/html.py +294 -0
  26. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/lake.py +51 -11
  27. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/maintenance/__init__.py +8 -4
  28. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/maintenance/cli.py +145 -5
  29. local_data_platform-0.1.5/src/local_data_platform/maintenance/system.py +213 -0
  30. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/__init__.py +1 -1
  31. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/cli.py +29 -6
  32. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/discovery.py +72 -16
  33. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/guard.py +64 -10
  34. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/sandbox.py +99 -5
  35. local_data_platform-0.1.5/src/local_data_platform/mcp_server/sla.py +146 -0
  36. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/tools.py +197 -181
  37. local_data_platform-0.1.5/src/local_data_platform/metrics/__init__.py +61 -0
  38. local_data_platform-0.1.5/src/local_data_platform/metrics/check.py +221 -0
  39. local_data_platform-0.1.5/src/local_data_platform/metrics/cli.py +180 -0
  40. local_data_platform-0.1.5/src/local_data_platform/metrics/compiler.py +427 -0
  41. local_data_platform-0.1.5/src/local_data_platform/metrics/load.py +258 -0
  42. local_data_platform-0.1.5/src/local_data_platform/metrics/model.py +621 -0
  43. local_data_platform-0.1.5/src/local_data_platform/metrics/render.py +180 -0
  44. local_data_platform-0.1.5/src/local_data_platform/metrics/runner.py +177 -0
  45. local_data_platform-0.1.5/src/local_data_platform/metrics/sql.py +213 -0
  46. local_data_platform-0.1.5/src/local_data_platform/monitoring/__init__.py +18 -0
  47. local_data_platform-0.1.5/src/local_data_platform/monitoring/alerts.py +189 -0
  48. local_data_platform-0.1.5/src/local_data_platform/monitoring/cli.py +201 -0
  49. local_data_platform-0.1.5/src/local_data_platform/monitoring/configs.py +71 -0
  50. local_data_platform-0.1.5/src/local_data_platform/monitoring/evaluation.py +78 -0
  51. local_data_platform-0.1.5/src/local_data_platform/monitoring/freshness.py +136 -0
  52. local_data_platform-0.1.5/src/local_data_platform/monitoring/history.py +200 -0
  53. local_data_platform-0.1.5/src/local_data_platform/monitoring/ledger.py +197 -0
  54. local_data_platform-0.1.5/src/local_data_platform/monitoring/metric.py +430 -0
  55. local_data_platform-0.1.5/src/local_data_platform/monitoring/results.py +136 -0
  56. local_data_platform-0.1.5/src/local_data_platform/monitoring/schema.py +216 -0
  57. local_data_platform-0.1.5/src/local_data_platform/monitoring/sla.py +709 -0
  58. local_data_platform-0.1.5/src/local_data_platform/monitoring/status.py +333 -0
  59. local_data_platform-0.1.5/src/local_data_platform/monitoring/status_page.py +117 -0
  60. local_data_platform-0.1.5/src/local_data_platform/monitoring/target.py +70 -0
  61. local_data_platform-0.1.5/src/local_data_platform/monitoring/volume.py +204 -0
  62. local_data_platform-0.1.5/src/local_data_platform/notify.py +680 -0
  63. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/cli.py +13 -1
  64. local_data_platform-0.1.5/src/local_data_platform/orchestration/gold_task.py +197 -0
  65. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/tasks.py +48 -22
  66. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/__init__.py +91 -9
  67. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/builders.py +13 -7
  68. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/duckdb_to_iceberg/__init__.py +3 -2
  69. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/spec.py +205 -19
  70. local_data_platform-0.1.5/src/local_data_platform/store/source/duckdb/__init__.py +269 -0
  71. local_data_platform-0.1.3/src/local_data_platform/store/source/duckdb/__init__.py +0 -155
  72. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/LICENSE +0 -0
  73. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/catalog/__init__.py +0 -0
  74. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/catalog/local/__init__.py +0 -0
  75. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/catalog/local/iceberg/__init__.py +0 -0
  76. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/catalog/provider.py +0 -0
  77. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/__init__.py +0 -0
  78. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/dbt_layer/cli.py +0 -0
  79. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/demo.py +0 -0
  80. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/engine/__init__.py +0 -0
  81. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/engine/duckdb/row_filter.py +0 -0
  82. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/engine/router.py +0 -0
  83. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/engine/spark/__init__.py +0 -0
  84. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/etl.py +0 -0
  85. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/filelock.py +0 -0
  86. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/__init__.py +0 -0
  87. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/csv/__init__.py +0 -0
  88. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/excel/__init__.py +0 -0
  89. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/iceberg/__init__.py +0 -0
  90. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/iceberg/commit.py +0 -0
  91. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/format/parquet/__init__.py +0 -0
  92. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/fs.py +0 -0
  93. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/github/__init__.py +0 -0
  94. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/integrations/__init__.py +0 -0
  95. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/integrations/airflow.py +0 -0
  96. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/issue/__init__.py +0 -0
  97. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/logger.py +0 -0
  98. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/maintenance/_common.py +0 -0
  99. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/maintenance/orphans.py +0 -0
  100. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/maintenance/snapshots.py +0 -0
  101. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/__main__.py +0 -0
  102. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/audit.py +0 -0
  103. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/errors.py +0 -0
  104. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/mcp_server/server.py +0 -0
  105. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/__init__.py +0 -0
  106. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/backfills.py +0 -0
  107. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/dbt_task.py +0 -0
  108. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/ledger.py +0 -0
  109. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/recipes.py +0 -0
  110. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/retries.py +0 -0
  111. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/runner.py +0 -0
  112. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/schedule.py +0 -0
  113. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/schedule_cli.py +0 -0
  114. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/windows.py +0 -0
  115. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/orchestration/workflow.py +0 -0
  116. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/paths.py +0 -0
  117. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/builtin.py +0 -0
  118. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/egression/__init__.py +0 -0
  119. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/egression/csv_to_iceberg/__init__.py +0 -0
  120. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/egression/iceberg_to_csv/__init__.py +0 -0
  121. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/egression/iceberg_to_parquet/__init__.py +0 -0
  122. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/__init__.py +0 -0
  123. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/bigquery_to_csv/__init__.py +0 -0
  124. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/csv_to_iceberg/__init__.py +0 -0
  125. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/excel_to_csv/__init__.py +0 -0
  126. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/excel_to_iceberg/__init__.py +0 -0
  127. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/parquet_to_iceberg/__init__.py +0 -0
  128. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/ingestion/pyarrow/__init__.py +0 -0
  129. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/pipeline/registry.py +0 -0
  130. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/quality/__init__.py +0 -0
  131. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/quality/checks.py +0 -0
  132. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/quality/report.py +0 -0
  133. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/quality/runner.py +0 -0
  134. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/quality/temporal.py +0 -0
  135. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/__init__.py +0 -0
  136. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/__init__.py +0 -0
  137. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/gcp/__init__.py +0 -0
  138. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/gcp/bigquery/__init__.py +0 -0
  139. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/json/__init__.py +0 -0
  140. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/near/__init__.py +0 -0
  141. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/source/parquet/__init__.py +0 -0
  142. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/target/__init__.py +0 -0
  143. {local_data_platform-0.1.3 → local_data_platform-0.1.5}/src/local_data_platform/store/target/iceberg/__init__.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: local-data-platform
3
- Version: 0.1.3
3
+ Version: 0.1.5
4
4
  Summary: A Python API layer for running a complete Iceberg lakehouse data platform on your laptop.
5
5
  License: MIT
6
6
  License-File: LICENSE
@@ -67,10 +67,18 @@ It's for learning how a lakehouse works, for building and testing pipelines loca
67
67
  for cloud infrastructure, and for giving agents and notebooks safe, reproducible access to the
68
68
  tables you build.
69
69
 
70
- > **Status.** This is version 0.1.3, the Orchestration milestone. It adds windows and idempotency keys
71
- > to `ldp run`, workflow files with `ldp workflow plan` and `ldp workflow run`, `ldp backfill`, retries,
72
- > `ldp schedule` for cron and launchd, and an Airflow 3 operator (see [Orchestration](#orchestration)),
73
- > with a [guide](docs/orchestration.md). Version 0.1.2 added `ldp dbt`, which runs a dbt project over
70
+ > **Status.** This is version 0.1.5, the Monitoring milestone. It adds an SLA in the config of a table,
71
+ > `ldp monitor` with freshness, volume, schema and metric monitors, alerts to Slack, a webhook or a file
72
+ > when a monitor changes state and when it recovers, `ldp status` as one screen for every pipeline, the
73
+ > SLA verdict of a table for agents in `ldp mcp`, and `ldp maintain --system` to keep the `_ldp` history
74
+ > small (see [Monitoring](#monitoring)), with a [guide](docs/monitoring.md). Version 0.1.4 added gold
75
+ > tables that carry an owner, a meaning and a description of every column, `ldp gold build` and one
76
+ > release of all the gold tables, `ldp gold list`, `show` and `docs` as a catalog, metrics defined once
77
+ > in `metrics.json`, and Excel, CSV and DuckDB files for people (see [Gold layer](#gold-layer)), with a
78
+ > [guide](docs/gold.md). Version 0.1.3 added windows and idempotency keys to `ldp run`, workflow files
79
+ > with `ldp workflow plan` and `ldp workflow run`, `ldp backfill`, retries, `ldp schedule` for cron and
80
+ > launchd, and an Airflow 3 operator (see [Orchestration](#orchestration)), with a
81
+ > [guide](docs/orchestration.md). Version 0.1.2 added `ldp dbt`, which runs a dbt project over
74
82
  > your Iceberg tables and publishes the models you mark back to Iceberg (see
75
83
  > [Transform with dbt](#transform-with-dbt)), the `DUCKDB` and `XLSX` source formats, time travel and
76
84
  > folders of tables in `ldp query`, retries for concurrent appends, and a restaurant example with a
@@ -86,7 +94,7 @@ tables you build.
86
94
  `ldp` needs Python 3.12 or newer. Install it from PyPI, preferably in a virtual environment:
87
95
 
88
96
  ```bash
89
- pip install "local-data-platform[duckdb]>=0.1.3"
97
+ pip install "local-data-platform[duckdb]>=0.1.5"
90
98
 
91
99
  ldp demo --workdir ldp_demo
92
100
  ```
@@ -126,10 +134,12 @@ More demos, each one a `make` target (see [Development](#development)):
126
134
  | `make demo-robotics` | Humanoid-robot episodes and sensor frames in bronze, silver and gold Iceberg tables, temporal quality checks, a pinned training split and DuckDB research queries |
127
135
  | `make demo-agent` | The read-only MCP server on the demo table, driven by a scripted agent client |
128
136
  | `make demo-warehouse` | A restaurant chain from Excel and CSV exports to dbt-tested Iceberg tables with monthly customer churn, a late batch and time travel (needs the `dbt` and `excel` extras) |
129
- | `make demo-orchestration` | The restaurant's nightly drops run as a workflow: a backfill that stops at a bad row and resumes after the fix, a third run that skips every day, and a scheduler tick (runs its dbt models task when the `dbt` extra is installed) |
137
+ | `make demo-orchestration` | The restaurant's nightly drops run as a workflow: a backfill that stops at a bad row and resumes after the fix, a third run that skips every day, and a scheduler tick (runs its dbt models task and its gold task, which cuts a release for each night, when the `dbt` extra is installed) |
138
+ | `make demo-gold` | The restaurant data mart: a faulty batch that the gate stops with each failed check named, dbt models whose gold tables have an owner, a meaning and a gate, a release of the gold tables, a metric, an Excel file, a DuckDB file and a catalog page (needs the `dbt` and `excel` extras) |
139
+ | `make demo-monitor` | The restaurant's orders watched through five nights: a late table, a short batch, a new column and a revenue drop raise alerts, a fresh night resolves them, and `ldp status` shows every pipeline on one screen (the revenue step runs when the `dbt` extra is installed) |
130
140
  | `make demo-spark` | The Scala Spark job reading the demo table and writing an aggregate that pyiceberg reads back (needs scala-cli) |
131
141
  | `make demo-rest` | The demo config against an Iceberg REST catalog started locally on scala-cli's JVM: `ldp catalog test`, two runs, the snapshots and a query (needs scala-cli) |
132
- | `make demo-all` | All of them; `demo-warehouse` is skipped when dbt or the Excel reader isn't installed and the two JVM demos when scala-cli isn't |
142
+ | `make demo-all` | All of them; `demo-warehouse` and `demo-gold` are skipped when dbt or the Excel reader isn't installed and the two JVM demos when scala-cli isn't |
133
143
 
134
144
  ### Extras
135
145
 
@@ -239,6 +249,7 @@ ldp query rides.json "SELECT * FROM rides" --max-rows 20 # print at most 20 ro
239
249
  ldp query rides.json "SELECT count(*) FROM rides" --snapshot 4437 # an older snapshot, from ldp snapshots
240
250
  ldp query rides.json "SELECT count(*) FROM rides" --as-of "2026-07-31 23:59:59" # the newest snapshot at or before that time (UTC)
241
251
  ldp query config/ "SELECT o.id FROM raw.orders o JOIN raw.customers c ON c.id = o.id" # a folder of configs: every table, joinable
252
+ ldp query config/ "SELECT count(*) FROM gold.daily_sales" --release r2 # the gold tables as release r2 names them
242
253
  ldp demo --workdir /tmp/ldp_demo # also --rows N (default 1000) and --seed N (default 42)
243
254
  ```
244
255
 
@@ -260,8 +271,8 @@ Every command follows one exit-code table.
260
271
  |---|---|
261
272
  | 0 | success, including a run that published and a run that skipped a duplicate |
262
273
  | 1 | an error: a bad config, a usage error, a failed write |
263
- | 2 | reserved for an SLA breach that `ldp monitor` reports in 0.1.5 |
264
- | 3 | a data gate held the run back, such as a quality check that blocked a write |
274
+ | 2 | an SLA breach, which `ldp monitor` reports when a monitor is in error |
275
+ | 3 | a data gate held the run back, such as a quality check that blocked a write, or a gold table whose contract does not hold |
265
276
 
266
277
  A scheduler retries a run that exits with 1, because the cause may pass, and never retries one that exits with 3,
267
278
  because the data has to change first. Add `-v` to see a traceback. The commands
@@ -289,10 +300,24 @@ call:
289
300
  | `ldp plan CONFIG [--window WINDOW] [--json]` | Validate a config (or a folder of files, each as its kind) and print its spec hash, route, target, sinks, schedule, and the source path and idempotency key of a window | `spec.validate_spec(data)`, `spec.plan(config, window=...)` |
290
301
  | `ldp runs CONFIG [--run RUN_ID] [--events] [--attempts]` | List the runs a config's sinks recorded, the last attempt of a run that was retried, or every attempt with `--attempts` | `events.summarize_runs(events.read_jsonl(path))` or `read_iceberg_events(catalog)` |
291
302
  | `ldp commits CONFIG [--key KEY] [--branches]` | List the staged publishes on a table, newest first | `format.iceberg.commit.commit_log(iceberg.table())` and `staged_branches(...)` |
292
- | `ldp datasets pin\|list\|export` | Pin, list and export dataset versions | `datasets.pin`, `list_versions`, `export` |
293
- | `ldp maintain CONFIG [--expire DAYS] [--orphans] [--apply]` | Expire snapshots and find orphan files; a dry run unless `--apply` | `maintenance.expire_snapshots`, `find_orphans`, `remove_orphans` |
303
+ | `ldp datasets pin\|list\|export` | Pin, list and export dataset versions, as Parquet, JSONL or CSV | `datasets.pin`, `list_versions`, `export` |
304
+ | `ldp maintain CONFIG [--system [--keep-days N]] [--expire DAYS] [--orphans] [--apply]` | Expire snapshots and find orphan files; a dry run unless `--apply`. With `--system` it trims the `_ldp` history tables of the config's catalog instead, deleting the rows older than N days (90 by default) first | `maintenance.expire_snapshots`, `find_orphans`, `remove_orphans`, `delete_old_rows`, `maintain_system_tables` |
294
305
  | `ldp mcp --config CONFIG ...` | Serve tables to an agent over MCP (stdio) | `python -m local_data_platform.mcp_server ...`, same options |
295
306
  | `ldp dbt init\|build\|test\|docs\|exec PROJECT_DIR` | Run a dbt project over your Iceberg tables; `build` writes the models marked `ldp_publish` back to Iceberg (needs the `dbt` extra) | `dbt_layer.register_sources(...)`, `dbt_layer.publish.plan_publish(...)` and `publish(...)` |
307
+ | `ldp gold sync CONFIG [--dbt-manifest FILE] [--dry-run] [--json]` | Write the contract of a config (its `metadata.docs`) into its existing Iceberg table, with no new data. `--dbt-manifest` reads it from a dbt manifest, for a table that dbt built, and `--dry-run` only says what would change. See [Table contracts](#table-contracts) | `gold.sync.sync_config(path)`, `contracts.apply_contract(table, docs)` |
308
+ | `ldp gold build DIR [--no-release] [--duckdb FILE] [--json]` | Build the gold tables of a folder in the order that their `inputs` say, and check the ones that are built elsewhere. One line for each table, and a gate that blocks a table stops the build with exit status 3. When every table passed, and the metrics of the folder fit them, it cuts a release, unless `--no-release` says not to. `--duckdb FILE` then writes the DuckDB file of the gold layer. See [Build the gold tables](#build-the-gold-tables) | `gold.build.build_gold(folder, release=True)`, `gold.serve.refresh_after_build(folder, out, result)` |
309
+ | `ldp gold releases DIR [--show RELEASE] [--json]` | List the releases of the gold tables of a folder, newest first, or the tables of one release. See [Release the gold tables](#release-the-gold-tables) | `gold.release.list_releases(folder)`, `current_release(folder)`, `snapshot_for(table, release)` |
310
+ | `ldp gold list DIR [--all-layers] [--json] [--strict]` | One row for each gold table of a folder, with what it means, who owns it, what one row is, its rows, when its data last changed, the status of its last run, the verdict of its last quality check and the release it is in. A table with no contract shows `no description`. `--all-layers` adds the other tables, and `--strict` exits with 1 when a gold table lacks part of its contract, for CI. See [Browse the gold tables](#browse-the-gold-tables) | `gold.catalog.collect(folder)` |
311
+ | `ldp gold export DIR TABLE DEST [--format csv\|parquet\|jsonl\|xlsx] [--release RN\|--live] [--json]` | Write one gold table, as the current release holds it, as a CSV, Parquet, JSONL or Excel file, which opens with no Python and no Iceberg. An Excel file has a second sheet, `about`, with the contract of the table, and needs the `excel` extra. See [Open the gold tables in a spreadsheet or a BI tool](#open-the-gold-tables-in-a-spreadsheet-or-a-bi-tool) | `gold.serve.export_table(folder, table, dest)` |
312
+ | `ldp gold duckdb DIR --out FILE [--release RN\|--live] [--json]` | Write the whole gold layer as one DuckDB file, with a copy of each table of the current release, a comment on each table and column, a `_catalog` table and a `_metrics` table. It opens in the DuckDB command line, the DuckDB UI and the BI tools that read DuckDB. See [Open the gold tables in a spreadsheet or a BI tool](#open-the-gold-tables-in-a-spreadsheet-or-a-bi-tool) | `gold.serve.write_duckdb_file(folder, out)` |
313
+ | `ldp gold show DIR TABLE [--runs N] [--no-samples] [--json]` | Describe one table: its contract, every column with its type and description, its last runs, the releases that include it, a few sample rows and an `ldp query` line that reads it | `gold.catalog.collect(folder, only=table, detail=True)` |
314
+ | `ldp gold docs DIR --out FILE [--format html\|md] [--all-layers] [--no-samples]` | Write the catalog as one self-contained page with a search box, or as Markdown for a MkDocs site | `gold.render.html_page(catalog)`, `gold.render.markdown_page(catalog)` |
315
+ | `ldp metrics list DIR [--json]` | One row for each metric of the `metrics.json` files of a folder, with its description, owner, format, table, dimensions and grains. See [Define metrics](#define-metrics) | `metrics.load_metrics(folder)` |
316
+ | `ldp metrics validate DIR [--json]` | Check every name and expression of the metrics against the schema of the gold tables, list every problem, and exit with 1 if there is one. No row is read, and `ldp gold build` runs the same check | `metrics.validate_folder(folder)` |
317
+ | `ldp metrics query DIR METRICS [--by DIMS] [--grain GRAIN] [--filter DIM=VALUE] [--from DAY] [--to DAY] [--release RN\|--live] [--sql] [--out FILE] [--raw] [--json]` | Compute metrics from the current release of the gold tables, grouped and filtered, and print them in their format. `--sql` prints the SQL and runs nothing, and `--out` writes CSV, Parquet or JSONL | `metrics.query(folder, metrics, by=..., grain=..., filters=[...])` returns a `pyarrow.Table` |
318
+ | `ldp monitor CONFIG [--json] [--now TIME]` | Check the SLA of a config, or of every dataset config of a folder, and print one line for each monitor. The freshness monitor measures the age of a table from the metadata of its last commit, or from the newest value of `freshness.column`, and scans no rows. The volume monitor compares the rows of the latest run with `min_rows` and `max_rows`, and with the median of the runs before it, and the schema monitor compares the columns of the table, and of its source in the latest run, with the last evaluation. A metric monitor computes a metric of the metrics file, or one query, for the latest complete period of its grain, and compares it with `min` and `max` and with the median of the periods before it in the history of the monitors, see the [monitoring guide](docs/monitoring.md). The command writes to no table and creates no catalog. It records the results in `_ldp.monitor_results` when the config has an `iceberg` sink and its catalog exists, and in `.ldp/monitor.jsonl` next to the config otherwise. A config that lists channels in `metadata.observability.alerts` is told when a monitor changes state and when it recovers, see [Alerts](docs/alerts.md). The exit status is 0 when every monitor is `ok` or `warn`, 2 when one is `error` and 1 when `ldp` itself fails | `monitoring.evaluate(config, now=...)` returns a `MonitorResult` for each monitor, and `monitoring.history.record` and `read_results` write and read the history |
319
+ | `ldp status CONFIG [--json] [--html FILE] [--now TIME]` | Print one row for each dataset config of a file or a folder, the worst first, with the last run and how long ago it started, how long ago a run succeeded, the rows of the table, what the freshness, volume, schema and metrics monitors say now, the checks that failed in the last 7 days and the open alerts. It evaluates the monitors in memory and records and sends nothing, reads each `_ldp` table once for the configs that share a catalog, and says how far back the history goes. The exit status is 0 unless `ldp` itself fails. `--json` prints the rows, and `--html FILE` also writes one self-contained page with the table, the monitors and the last 20 runs of each pipeline | `monitoring.status.rows(path, now=None)` returns a `StatusRow` for each config, and `monitoring.ledger.Ledger` reads the history of many configs with one read of each table |
320
+ | `ldp alert test CONFIG` | Send one test message to each channel of `metadata.observability.alerts`, whatever its `on` says, and print which ones got it. It sends real messages. The exit status is 0 when every channel got it and 1 otherwise | `notify.notifier_for(config).test()` returns a `Delivery` for each channel, and `notify.NotifySink` is the event sink that alerts when a run fails or is blocked |
296
321
  | `ldp spark CONFIG` | Run the Scala Spark job on a config's table (needs scala-cli) | `engine.spark.ScalaSparkJob` |
297
322
 
298
323
  ## Config schema
@@ -306,17 +331,21 @@ and a `metadata` block:
306
331
  | `metadata.target` | The same, plus for Iceberg `write_mode` (`append`, `overwrite` or `upsert`), `join_cols` for upserts, and `partition_by` |
307
332
  | `metadata.target.catalog` | `type` (`local` by default, `sql`, `rest` or `glue`) and that type's keys; see [Catalogs](#catalogs-and-object-storage) |
308
333
  | `metadata.quality` | `on_failure` (`fail` or `warn`) and a list of `checks`: `row_count`, `not_null`, `unique`, `accepted_values`, `range`, `freshness`, `schema`, and the temporal checks `monotonic`, `max_skew`, `rate_below` and `max_gap` |
309
- | `metadata.observability` | Where run events go, for example `{"sinks": ["iceberg", "jsonl"]}`. The default sends them nowhere |
334
+ | `metadata.observability` | Where run events go, for example `{"sinks": ["iceberg", "jsonl"]}`, and `alerts`, the channels that are told when a run fails or a monitor breaks and recovers, see [Alerts](docs/alerts.md). The default sends events nowhere and alerts to nobody |
310
335
  | `metadata.schedule` | When the config runs: a 5-field cron expression, or `@hourly`, `@daily`, `@weekly` or `@monthly`. Without it, a `when` of `hourly`, `daily`, `weekly` or `monthly` means the alias. Editing it does not change the spec hash |
311
336
  | `metadata.window` | `"none"`, or `size` (`day`, `hour` or `15m`), an optional `column` that selects the rows of a window, and an optional `timezone` (UTC by default); see [Windows](#windows) |
312
337
  | `metadata.retries` | How a workflow tries the run again after a failure: `max_attempts` (1 by default, which is no retry), `delay_s` (30) and `backoff` (2.0); see [Workflow files](#workflow-files). Editing it does not change the spec hash |
338
+ | `metadata.lineage` | The Iceberg tables that a source was made from, as `ICEBERG` blocks, for the run events only. `ldp dbt build` writes it for each model that it publishes, from the manifest. A `DUCKDB` source with `inputs` needs none. Editing it does not change the spec hash |
339
+ | `metadata.docs` | The contract of the table: `layer` (`bronze`, `silver` or `gold`), `description`, `owner` (the config's `who` by default), `grain`, `primary_key`, `columns` (column name to description), `tags` and `refresh` (the config's `when` by default); see [Table contracts](#table-contracts). Editing it does not change the spec hash |
340
+ | `metadata.sla` | How fresh, how big and how stable the table should be, and which business numbers to watch, in up to four sections. `freshness` has `warn_after` and `error_after`, each a `count` of `minute`, `hour` or `day` periods such as `{"count": 26, "period": "hour"}`, and an optional `column`. `volume` has `min_rows`, `max_rows`, `max_change_pct` and `lookback_runs` (7 by default). `schema` has `on_change` (`ignore`, `warn` or `error`, `warn` by default) and `allow`, a list of `added_column` and `widened_type`. `metrics` is a list of entries that each name a `metric` of the metrics file next to the config, or give a `sql` query and a `name`, with a `grain`, `min`, `max`, `max_change_pct`, an optional `filter` and a `lookback` (7 by default). A malformed value is an error that names its path, `ldp plan` prints the SLA and reports a metric that the folder does not define, and `ldp schema` describes it. `ldp monitor` checks all four sections. Editing it does not change the spec hash |
313
341
 
314
342
  A config may also say `"apiVersion": "ldp/v1"` and `"kind": "Pipeline"`. `kind` says what a JSON file is. `Pipeline`
315
343
  is the default, so every config here is one, and a config has the same spec hash with and without it. `Workflow` is
316
- the second kind. A kind is written exactly as shown, so `workflow` fails with a message that names `Workflow`, and
317
- `Metrics`, `Report` and `Profiles` are reserved for later milestones. `ldp plan` validates each file of a folder as its
318
- kind, and `ldp query`, `ldp mcp --config` and `ldp dbt` read the tables of the `Pipeline` files only, so a folder can
319
- hold workflow files next to its configs. `ldp schema --kind Workflow` prints the schema of another kind.
344
+ the second kind and `Metrics` the third, see [Define metrics](#define-metrics). A kind is written exactly as shown, so
345
+ `workflow` fails with a message that names `Workflow`, and `Report` and `Profiles` are reserved for later milestones.
346
+ `ldp plan` validates each file of a folder as its kind, and `ldp query`, `ldp mcp --config` and `ldp dbt` read the
347
+ tables of the `Pipeline` files only, so a folder can hold workflow and metrics files next to its configs.
348
+ `ldp schema --kind Metrics` prints the schema of another kind.
320
349
 
321
350
  Partition transforms are `identity`, `year`, `month`, `day`, `hour`, `bucket[N]` and
322
351
  `truncate[W]`. On pyiceberg 0.10 and later, writing with any transform except `identity` needs the
@@ -388,9 +417,10 @@ ldp runs .ldp/events.jsonl # the workf
388
417
  ```
389
418
 
390
419
  - Each task has an `id`, an optional `depends_on`, and exactly one kind: `pipeline` (the path of a dataset config,
391
- relative to the workflow file), `dbt` (a dbt project, built for the window, see below) or `maintain` (a config's Iceberg
392
- table and the options of `ldp maintain`, a dry run unless `apply` is true). Later milestones add kinds without changing the
393
- file format.
420
+ relative to the workflow file), `dbt` (a dbt project, built for the window, see below), `maintain` (a config's Iceberg
421
+ table and the options of `ldp maintain`, a dry run unless `apply` is true) or `gold` (the gold tables of a folder, built
422
+ for the window, with a release cut at the end, see [Release the gold tables](#release-the-gold-tables)). Later
423
+ milestones add kinds without changing the file format.
394
424
  - Tasks run one at a time, in dependency order. A pipeline task runs with the window of the workflow, so its idempotency key
395
425
  is the one `ldp plan CONFIG --window` prints.
396
426
  - A task that fails, or that a quality check blocks, makes the tasks that depend on it `upstream_failed`, and they do not run.
@@ -539,10 +569,23 @@ snapshots. It is how tables built with SQL (or with dbt, through `ldp dbt`) reac
539
569
  }
540
570
  ```
541
571
 
542
- Use `relation` (`table`, `schema.table` or `catalog.schema.table`) or `sql`, not both. The file is
543
- opened read-only and closed before the Iceberg write starts, and a `sql` source must be exactly one
544
- `SELECT` or `WITH` query. It needs the `duckdb` extra. `ldp plan` reports a missing file or a
545
- source with both or neither of `relation` and `sql`.
572
+ Use `relation` (`table`, `schema.table` or `catalog.schema.table`), `sql` or `query_file`, one of them. The file is
573
+ opened read-only and closed before the Iceberg write starts, and a query must be exactly one `SELECT` or `WITH` query.
574
+ A `query_file` is a file that holds the query, in place of `sql`. It needs the `duckdb` extra. `ldp plan` reports a
575
+ missing file or a source with more than one or none of the three.
576
+
577
+ A source can also have `inputs` in place of `path`, which is a list of dataset configs. It then reads no file. It
578
+ opens an in-memory DuckDB, registers the Iceberg table that each input config writes under that table's name, with the
579
+ session time zone set to UTC, and runs the query on them. This is how a gold table is one SQL file over silver tables,
580
+ and [`ldp gold build`](#build-the-gold-tables) builds them in order.
581
+
582
+ ```json
583
+ "source": {"name": "revenue_by_restaurant", "format": "DUCKDB", "query_file": "models/revenue_by_restaurant.sql",
584
+ "inputs": ["silver_orders.json"]}
585
+ ```
586
+
587
+ A query that reads a table that is not in `inputs` fails with DuckDB's own unknown table error, so `inputs` is the whole
588
+ lineage of the table, and every input is an input dataset in the run's OpenLineage events.
546
589
 
547
590
  ### An Excel sheet as a source
548
591
 
@@ -572,6 +615,415 @@ sheet. It needs the `excel` extra, and `ldp plan` reports a missing file or a `.
572
615
  atomically, so a folder that Google Drive syncs never holds a half-written file; see the recipe
573
616
  [Excel to CSV in a Google Drive folder](docs/recipes.md#excel-to-csv-in-a-google-drive-folder).
574
617
 
618
+ ### Table contracts
619
+
620
+ A table says what it means, who owns it, what one row is and what each column holds, in a `metadata.docs` block next to
621
+ the pipeline that builds it. After a run publishes, LDP writes the block into the Iceberg table, as the properties
622
+ `comment`, `ldp.owner`, `ldp.layer`, `ldp.grain` and `ldp.primary-key` and the doc of each column. Anyone who finds the
623
+ table reads it there, in `ldp mcp`, in DuckDB or in any other client.
624
+
625
+ ```json
626
+ "docs": {
627
+ "layer": "gold",
628
+ "description": "Orders and revenue for each restaurant on each day it had an order.",
629
+ "owner": "restaurant operations",
630
+ "grain": "one row for each restaurant on each day it had an order",
631
+ "primary_key": ["restaurant_id", "day"],
632
+ "columns": {"restaurant_id": "The restaurant.", "day": "The day.", "orders": "How many orders.",
633
+ "revenue": "What they cost."}
634
+ }
635
+ ```
636
+
637
+ - A `gold` table has to keep its promise. It needs a `description`, a `grain` and an owner (`owner`, or the `who` of the
638
+ config). `ldp run` stops it before anything is written when a column has no description, or when `columns` describes
639
+ a column that the table does not have, and the exit status is 3, as for a failed quality check. A table of another
640
+ layer may document as much or as little as it likes.
641
+ - The contract is one metadata-only commit, so it adds no snapshot, and a table that already has it gets no commit. It
642
+ only adds and changes text, and a run that wrote nothing does not touch the table.
643
+ - `ldp gold sync CONFIG` writes the contract of a config into a table that exists, with no new data. With
644
+ `--dbt-manifest target/manifest.json` it reads the contract from the manifest, for a table that dbt built.
645
+ - `ldp dbt build` fills the block from the dbt manifest. A model that is published to the `gold` namespace is a gold
646
+ table, so it needs `owner` and `grain` in its meta, a description, and a description of each column.
647
+ - Editing the block does not change the spec hash. The design is in [the v0.1.4 contract](docs/design/v0_1_4.md).
648
+
649
+ ### Build the gold tables
650
+
651
+ A gold table is one SQL file over other tables. Its config has a `DUCKDB` source with `inputs` and a `query_file`, and
652
+ a `metadata.docs` block with `"layer": "gold"`. `ldp gold build DIR` finds every gold config of a folder, orders them by
653
+ their `inputs`, since a gold table may read another, and builds them one at a time.
654
+
655
+ ```bash
656
+ ldp gold build examples/robot_episodes/data
657
+ ```
658
+
659
+ ```text
660
+ gold_episode_stats built 96 rows snapshot 3844274472678295321 4 of 4 checks passed
661
+ gold_robot_daily built 16 rows snapshot 1985638678580193697 2 of 2 checks passed
662
+ gold build examples/robot_episodes/data: 2 built, 0 checked, 0 blocked, 0 failed, 0 not run in 0.6s
663
+ metrics: 1 semantic model and 5 metrics fit the gold tables
664
+ release r1 cut with 2 tables and 112 rows in robots
665
+ ```
666
+
667
+ The last line is the release that the build cuts, see [Release the gold tables](#release-the-gold-tables).
668
+
669
+ - A cycle between configs is refused before anything runs, and the message names it.
670
+ - The build stops at the first table that a gate blocks or that fails, and the tables after it are `not run`, because
671
+ they could read what it did not write. The exit status is 3 when a gate held a table back, and 1 on any other
672
+ failure. A table whose contract does not hold is held back by a gate too.
673
+ - A gold config with no `inputs`, such as a dbt model that `ldp dbt build` published or a table that Spark or a script
674
+ wrote, is built elsewhere. The build does not run it. It checks that the table exists and runs its quality checks on
675
+ the current snapshot, and it writes nothing. A failing check blocks the build, unless the config says
676
+ `on_failure` `warn`.
677
+ - The runs are recorded in `_ldp.runs` and `_ldp.quality_results`, through the `iceberg` sink, unless the config names
678
+ sinks of its own, so the freshness and the gate status of a gold table show in `describe_table` of `ldp mcp`.
679
+ - A folder with a `metrics.json` has its metrics checked, see [Define metrics](#define-metrics). A file that is not
680
+ valid stops the build before anything is built. When every table was built or checked, the names and expressions are
681
+ checked against the tables, and the build says so with a `metrics:` line. Metrics that do not fit cut no release and
682
+ the exit status is 1, and the readers still see the last release.
683
+
684
+ ### Release the gold tables
685
+
686
+ A gold table is rebuilt on its own, so two tables that are read at the same moment can come from different builds. A
687
+ release fixes that. When every gold table of a build was built or checked, `ldp gold build` cuts release `r1`, and then
688
+ `r2` and so on. A release names the snapshot that each gold table ended the build with, so a reader that reads the
689
+ current release sees tables that agree. A build that a gate blocks cuts no release, and the last release stays current.
690
+ The build says which release readers still see.
691
+
692
+ ```bash
693
+ ldp gold build examples/robot_episodes/data # builds, and then cuts a release
694
+ ldp gold releases examples/robot_episodes/data # the releases, newest first
695
+ ldp gold releases examples/robot_episodes/data --show r1
696
+ ldp query examples/robot_episodes/data "SELECT count(*) FROM robots.gold_episode_stats" --release r1
697
+ ```
698
+
699
+ ```text
700
+ gold_episode_stats built 96 rows snapshot 7902942047508908069 4 of 4 checks passed
701
+ gold_robot_daily built 16 rows snapshot 7453607785734378588 2 of 2 checks passed
702
+ gold build examples/robot_episodes/data: 2 built, 0 checked, 0 blocked, 0 failed, 0 not run in 0.6s
703
+ metrics: 1 semantic model and 5 metrics fit the gold tables
704
+ release r1 cut with 2 tables and 112 rows in robots
705
+ ```
706
+
707
+ - A release is a file in the warehouse, `<warehouse>/.ldp/releases/<namespace>/r1.json`. It lists each table with its
708
+ snapshot, the dataset version that pins it, the run that built it and its rows. The file is written to a temporary
709
+ path and renamed, so a reader sees all of it or none of it.
710
+ - Each table is pinned with `datasets.pin` as the dataset `<namespace>.<table>`, with the property `release=r1`. A pin
711
+ tags its snapshot, and snapshot expiry never removes a tagged snapshot, so `ldp maintain` keeps every released
712
+ snapshot. That keeps its files too, so a table that is rebuilt in full for each release keeps a copy for each of them.
713
+ - A release covers one namespace of one warehouse, because that is where its file is. A folder whose gold tables are in
714
+ more than one is refused before anything is built. Give each its own folder, or build with `--no-release`.
715
+ - A build that finds the tables at the snapshots of the current release, as a workflow does when it runs a window that
716
+ it already ran, cuts no new release.
717
+ - `ldp query DIR SQL --release r1` reads the gold tables as r1 names them. A table that the release does not hold, and
718
+ a table in a namespace that has no releases, is read at its current snapshot. Without `--release`, `ldp query` reads the
719
+ tables as they are now.
720
+ - Code that reads the gold layer asks for the release with `local_data_platform.gold.release`. `current_release(dir)`
721
+ gives the release readers see, `snapshot_for(table, release)` gives the snapshot to read, and `register_release(engine,
722
+ dir)` registers every gold table in a `DuckDBEngine` at its snapshot in the release. Pass `release="r1"` for an older
723
+ one and `live=True` to read the tables as they are.
724
+ - `ldp runs` shows a release in the history of each table, as a run with the status `released`.
725
+
726
+ A workflow can build and release the gold layer as one of its tasks, after the pipelines and the dbt project that
727
+ feed it. The task is one line.
728
+
729
+ ```json
730
+ {"id": "gold", "depends_on": ["orders", "dbt"], "gold": {"dir": "config"}}
731
+ ```
732
+
733
+ The task builds the gold tables of `config` for the window of the run, as a pipeline task runs a config, so running the
734
+ same window again finds the tables already published and cuts no second release. It ends `blocked_quality` when a gate
735
+ holds a table back, and then no release is cut.
736
+
737
+ ### Browse the gold tables
738
+
739
+ A person who is not an engineer should find the right gold table without reading pipeline code, and so should an agent
740
+ that works for them. `ldp gold list` prints one row for each gold table of a folder, and `ldp gold show` describes one.
741
+
742
+ ```bash
743
+ ldp gold list examples/robot_episodes/data
744
+ ```
745
+
746
+ ```text
747
+ 2 gold tables in examples/robot_episodes/data
748
+ table description owner grain rows refreshed_utc run quality release
749
+ ------------------------- ------------------------------------------------ -------------------------- ------------------------- ---- ------------------- --------- ------------- -------
750
+ robots.gold_episode_stats One row for each promoted episode, with its o... robot learning researchers one row per episode 96 2026-10-16 09:17:03 published 4 of 4 passed r1
751
+ robots.gold_robot_daily Fleet health for each robot on each day: how ... fleet operations one row per robot per day 16 2026-10-16 09:17:03 published 2 of 2 passed r1
752
+ (2 rows)
753
+ ```
754
+
755
+ ```bash
756
+ ldp gold show examples/robot_episodes/data gold_robot_daily
757
+ ```
758
+
759
+ ```text
760
+ robots.gold_robot_daily (gold)
761
+ Fleet health for each robot on each day: how many episodes it recorded, how many succeeded, and how healthy its frames were.
762
+ Owner fleet operations
763
+ Grain one row per robot per day
764
+ Primary key robot_id, recorded_on
765
+ Refresh after each promotion
766
+ Tags robotics, fleet
767
+ Config gold_robot_daily.json
768
+ Rows 16
769
+ Refreshed 2026-10-16 09:17:03 UTC
770
+ Last run published, run 01a14400-2856-7074-a459-21ae4afcae71, 2026-10-16 09:17:03 UTC
771
+ Quality 2 of 2 passed
772
+ Release r1
773
+
774
+ Columns (9)
775
+ ...
776
+ Metrics
777
+ ...
778
+ Releases that include it (1)
779
+ ...
780
+ Read it
781
+ ldp query examples/robot_episodes/data "SELECT * FROM robots.gold_robot_daily LIMIT 10"
782
+ ```
783
+
784
+ `show` also prints every column with its type and description, the metrics defined on the table, the last 5 runs, the
785
+ releases that include the table and 5 sample rows. The last line is an `ldp query` line that reads the table, ready to copy. `TABLE` is a table with its
786
+ namespace, or its bare name when only one table has it, and it can be a table of any layer.
787
+
788
+ - The tables are the Iceberg tables that the dataset configs of the folder write. A gold table is one whose
789
+ `metadata.docs.layer` is `gold`, or whose Iceberg table says so, or that is in a namespace called `gold`.
790
+ `--all-layers` adds the bronze and silver tables and the tables that have no layer, and `--json` prints the same as
791
+ data.
792
+ - The contract is the one in the `metadata.docs` of the config and the one that the Iceberg table carries. Where both
793
+ have a value the config wins, because it is what the next run writes, and `show` says so when the table does not carry
794
+ it yet. A table with no contract shows `no description` and does not fail the listing, and its description is `null`
795
+ in JSON.
796
+ - `--strict` exits with 1 when a gold table has no description, no owner or no grain, or has a column without a
797
+ description, and it names the table and what it lacks. A table of another layer is never held to it. Run it in CI, so
798
+ that a gold table that nobody documented does not get in. A table that does not exist yet is listed as `not built`.
799
+ - The last run and the last quality check are what `describe_table` of `ldp mcp` reports, from the same code. A release is
800
+ not a run, so a table that was released still shows the status of its last build.
801
+ - `release` is the release that readers see now. `r1 (behind live)` says that the table was built again after r1, so
802
+ readers of the release do not see the newer data yet. The rows and the time are the table as it is now, and the time
803
+ leaves out the snapshots that `ldp maintain` makes.
804
+ - Nothing is written, not even a catalog or a namespace that does not exist yet.
805
+
806
+ `ldp gold docs` writes the same content as one page, or as Markdown for a MkDocs site.
807
+
808
+ ```bash
809
+ ldp gold docs examples/robot_episodes/data --out catalog.html
810
+ ldp gold docs examples/robot_episodes/data --out docs/catalog.md --format md
811
+ ```
812
+
813
+ ```text
814
+ wrote catalog.html, 2 tables, 18.8 KB. Open it as a page. It needs no network.
815
+ wrote docs/catalog.md, 2 tables, 7.4 KB. Open it as a Markdown file.
816
+ ```
817
+
818
+ - The page is one file with a search box, a light and a dark theme, and its CSS and its one script inline. It asks for
819
+ nothing from the network, and its Content-Security-Policy makes the browser refuse to try, so it opens from a folder or
820
+ from an e-mail. It is built with `local_data_platform.html`, which later pages share.
821
+ - Every value that a person wrote, such as the description of a table, is escaped. It can never become markup in the page,
822
+ and the Markdown escapes it too.
823
+ - The sample rows of each table are in the file. Leave them out with `--no-samples` when the file will be shared and the
824
+ rows must not be.
825
+ - In code, `local_data_platform.gold.catalog.collect(folder)` returns one entry for each table, and
826
+ `entry.document()` is the text of an entry for a search to index.
827
+
828
+ `ldp mcp` says the same things to an agent. `list_tables` returns the layer, description, owner and grain of each table, and
829
+ `describe_table` returns the contract. `ldp mcp --config DIR --layer gold` serves only the gold tables, so no tool and no
830
+ SQL can reach the tables upstream of them.
831
+
832
+ ### Define metrics
833
+
834
+ Revenue, orders and average ticket should mean one thing in every report, export and answer of an agent. A
835
+ `metrics.json` file next to the configs of the gold tables defines each number once, with what it means, who answers
836
+ for it and how a person reads it. The robot example has one, `examples/robot_episodes/configs/metrics.json`, and so has the
837
+ restaurant example, `examples/restaurant/config/metrics.json`, which defines `churn_rate` and `active_customers` on the
838
+ churn model of the warehouse demo.
839
+
840
+ ```json
841
+ {
842
+ "apiVersion": "ldp/v1", "kind": "Metrics", "identifier": "robot_fleet_metrics", "who": "fleet operations",
843
+ "semantic_models": [{
844
+ "name": "robot_daily", "table": "robots.gold_robot_daily",
845
+ "time_dimension": {"column": "recorded_on", "grains": ["day", "week", "month", "quarter", "year"]},
846
+ "dimensions": [{"name": "robot", "column": "robot_id"}],
847
+ "measures": [{"name": "episode_sum", "agg": "sum", "expr": "episodes"},
848
+ {"name": "success_sum", "agg": "sum", "expr": "successes"}]}],
849
+ "metrics": [
850
+ {"name": "episodes", "type": "simple", "measure": "episode_sum",
851
+ "description": "How many episodes the robots recorded."},
852
+ {"name": "success_rate", "type": "ratio", "numerator": "success_sum", "denominator": "episode_sum",
853
+ "format": "percent", "description": "The share of episodes that succeeded."}]
854
+ }
855
+ ```
856
+
857
+ A semantic model names one gold table, its `time_dimension` (a date column and the grains that its numbers may be
858
+ reported at), its `dimensions` and its `measures`. A measure is an aggregate, `sum`, `count`, `count_distinct`, `avg`,
859
+ `min` or `max`, over a SQL expression of the columns of the table. A metric is `simple`, which is one measure, or a
860
+ `ratio`, which is one measure over another. It has a `description`, an `owner` (its own, or the `who` of the file) and
861
+ a `format` of `number`, `percent` or `currency`. The terms follow MetricFlow, so an exporter can follow later.
862
+ `ldp plan DIR` validates the file as `Metrics`, and `ldp schema --kind Metrics` prints its schema.
863
+
864
+ ```bash
865
+ ldp metrics list examples/robot_episodes/data
866
+ ldp metrics query examples/robot_episodes/data episodes,success_rate,frame_drop_rate --by robot --grain week
867
+ ```
868
+
869
+ ```text
870
+ metric description owner format table dimensions grains
871
+ ----------------- ------------------------------------------------ -------------------------- ------- ----------------------- ---------- -------------------------------
872
+ episodes How many episodes the robots recorded. fleet operations number robots.gold_robot_daily robot day, week, month, quarter, year
873
+ success_rate The share of episodes that succeeded. fleet operations percent robots.gold_robot_daily robot day, week, month, quarter, year
874
+ frame_drop_rate The share of frames that were dropped. fleet operations percent robots.gold_robot_daily robot day, week, month, quarter, year
875
+ worst_skew_ms The largest RGB to depth gap of any frame, in... fleet operations number robots.gold_robot_daily robot day, week, month, quarter, year
876
+ active_robot_days Days with a recording, added up over robots. robot learning researchers number robots.gold_robot_daily robot day, week, month, quarter, year
877
+ (5 rows)
878
+
879
+ week robot episodes success_rate frame_drop_rate
880
+ ---------- ----- -------- ------------ ---------------
881
+ 2026-08-31 h01 12 83.3% 0.3%
882
+ 2026-08-31 h02 12 75.0% 0.5%
883
+ 2026-08-31 h03 12 83.3% 0.3%
884
+ 2026-08-31 h04 12 91.7% 0.4%
885
+ 2026-08-31 h05 12 41.7% 0.4%
886
+ 2026-08-31 h06 12 66.7% 0.3%
887
+ 2026-08-31 h07 12 75.0% 0.4%
888
+ 2026-08-31 h08 12 66.7% 0.0%
889
+ (8 rows)
890
+ read release r1, cut 2026-10-20 08:34:32 UTC
891
+ ```
892
+
893
+ `--filter dimension=value` keeps the rows of a dimension, `--from` and `--to` take days that are both included, and
894
+ `--sql` prints what would run, with the value of each parameter, and runs nothing.
895
+
896
+ ```bash
897
+ ldp metrics query examples/robot_episodes/data success_rate --filter robot=h05 --grain month --from 2026-09-01 --sql
898
+ ```
899
+
900
+ ```text
901
+ SELECT
902
+ CAST(date_trunc('month', "recorded_on") AS DATE) AS "month",
903
+ sum(successes) / NULLIF(sum(episodes), 0) AS "success_rate"
904
+ FROM "robots"."gold_robot_daily"
905
+ WHERE "robot_id" = $1 AND "recorded_on" >= $2
906
+ GROUP BY 1
907
+ ORDER BY 1
908
+ -- $1 = "h05"
909
+ -- $2 = DATE '2026-09-01'
910
+ ```
911
+
912
+ - **A ratio is a ratio of sums.** `success_rate` is the sum of the successes over the sum of the episodes of the group,
913
+ and never the average of the daily rates, which is a different number when the days have different counts. A group
914
+ with no episodes has no value and not infinity.
915
+ - **A value is never SQL.** A filter value, a date and every other value that a person gives is bound as a parameter of
916
+ the type of its column, so `--filter robot="h05' OR '1'='1"` matches no robot. Names go into the SQL only after they
917
+ were checked against the definitions, and an unknown metric, dimension or grain is refused with the valid names and the
918
+ closest one.
919
+ - **A query reads one table.** Metrics of two semantic models are refused with the table of each, because that would need
920
+ a join, which this version does not have.
921
+ - **`grains` is where the numbers are right.** A number that counts customers at the start of a month is wrong at any
922
+ other grain, so its model allows only `month`, and another grain is refused. A query with no grain adds the whole
923
+ period into one row, which is the same step as adding up years, so it needs `year` to be allowed. The churn of the
924
+ restaurant example is such a number. After `make demo-warehouse`,
925
+ `ldp metrics query ldp_demo/restaurant/config churn_rate,active_customers --grain month` prints it for every month,
926
+ and the [recipe](docs/recipes.md) for monthly churn from an orders table shows the definition in SQL.
927
+ - **The format is for people.** `percent` shows `0.5` as `50.0%` and `currency` has two decimals, on a terminal. `--raw`
928
+ shows what was computed, and `--out FILE`, `--json` and the Python function always give the values as computed.
929
+ `--out` writes CSV, Parquet or JSONL by the ending of the name.
930
+ - **The tables are read as the current release holds them.** `--release r1` reads an older one and `--live` reads the
931
+ tables as they are now, as `ldp query` does. The line under the table says which was read. A release pins the data and
932
+ not the definitions, so an old release that is read with a definition that needs a column that it lacks fails with a
933
+ message that says so.
934
+ - **The definitions are checked against the tables.** `ldp metrics validate DIR` checks that every table is a gold table
935
+ that exists, that every dimension and the time column are columns, that the time column is a `DATE` or a `TIMESTAMP`,
936
+ that each expression is one plain SQL expression that works on the table, and that every metric is a number. It lists
937
+ every problem and exits with 1. `ldp gold build` runs the same check and cuts no release when metrics do not fit.
938
+ - **In code**, `local_data_platform.metrics.query(folder, "episodes,success_rate", by="robot", grain="week")` returns a
939
+ `pyarrow.Table`, and `metrics.compile_for(...)` returns the SQL and its parameters without running anything.
940
+
941
+ `ldp mcp` has two more tools for the same numbers, `list_metrics` and `query_metric`, see
942
+ [MCP server for agents](#mcp-server-for-agents). The gold catalog shows the metrics of each table, in `ldp gold show`,
943
+ on the page of `ldp gold docs`, and in its search.
944
+
945
+ ### Open the gold tables in a spreadsheet or a BI tool
946
+
947
+ A person who works in a spreadsheet or a desktop SQL tool should be able to use a gold table without Python, Iceberg or an
948
+ engineer. `ldp gold export` writes one table as a file, and `ldp gold duckdb` writes the whole gold layer as one DuckDB file.
949
+ Both read the gold layer as every reader does, so by default it is the current release, `--release r1` reads an older one
950
+ and `--live` reads the tables as they are now.
951
+
952
+ ```bash
953
+ ldp gold export examples/robot_episodes/data gold_robot_daily daily.csv
954
+ ldp gold export examples/robot_episodes/data gold_robot_daily daily.xlsx
955
+ ```
956
+
957
+ ```text
958
+ wrote daily.csv, 16 rows of robots.gold_robot_daily, read from release r1, cut 2026-10-23 11:26:36 UTC, snapshot 1317331787740633770
959
+ wrote daily.xlsx, 16 rows of robots.gold_robot_daily, read from release r1, cut 2026-10-23 11:26:36 UTC, snapshot 1317331787740633770
960
+ ```
961
+
962
+ The format comes from the end of the name, or from `--format`.
963
+
964
+ - **CSV, Parquet and JSONL.** The three are written by `datasets.write_rows`, the same function as `ldp datasets export`,
965
+ which has `--format csv` too. A CSV has a header and a line for each row. A list, a struct or a map is written as JSON text,
966
+ bytes as hexadecimal and a number that is not finite as no value. A Parquet file has the table, release and snapshot in its
967
+ metadata, under `ldp.gold`. A name that starts with `s3://` or `gs://` is written there.
968
+ - **Excel.** The file needs the `excel` extra, `pip install "local-data-platform[excel]"`, and says which if it is missing.
969
+ The first sheet has the rows, with a bold header that stays in view and a filter. The second sheet is called `about` and
970
+ has the description, owner, grain and key of the table, its release, snapshot id and build time, its rows, and every
971
+ column with a word for its type and its description. A text is always a text and never a formula, even when it starts
972
+ with `=`. A time with a zone is written in UTC, because a sheet has no zones. An integer beyond 2^53 is written as text,
973
+ because a sheet would round it. A table with more than 1,048,575 rows does not fit a sheet, with its header row, and is
974
+ refused before it is read, with a hint to use CSV. A table with more than 16,384 columns and a value of more than 32,767
975
+ characters are refused too, with the same hint, when the file is written, and no file is left.
976
+ - **A CSV and a formula.** Excel runs a CSV value that starts with `=`, `+`, `-` or `@` as a formula when a person opens
977
+ the file. LDP writes the values as they are, so use Excel for text that people typed and that you do not trust.
978
+ - **A table that the release does not hold.** When the current release was cut before the table existed, the table is read
979
+ as it is, and the line says why. `--release r1` for a table that r1 does not hold is refused.
980
+
981
+ ```bash
982
+ ldp gold duckdb examples/robot_episodes/data --out gold.duckdb
983
+ ```
984
+
985
+ ```text
986
+ wrote gold.duckdb, 2 tables and 5 metrics, from release r1, cut 2026-10-23 11:26:36 UTC, 1292.0 KB
987
+ ```
988
+
989
+ The file has a copy of every gold table of the release, in the `main` schema under its own name, so `SELECT * FROM
990
+ gold_robot_daily` works. It has no schema called `gold`, because DuckDB names a file's database after the file, and
991
+ `gold.daily_sales` in a file called `gold.duckdb` would be ambiguous. Three more things are in it.
992
+
993
+ - **Comments.** `COMMENT ON TABLE` has the description of each table and `COMMENT ON COLUMN` the description of each column,
994
+ from the contract, so a tool that shows comments shows what the data means.
995
+ - **`_catalog`.** One row for each table with `table_name`, `description`, `owner`, `grain`, `release`, `snapshot_id` and
996
+ `built_at`, which is when that snapshot was written, in UTC.
997
+ - **`_metrics`.** One row for each metric of `metrics.json` with its `metric`, `description`, `owner`, `type`, `format`,
998
+ `table_name`, `definition`, `dimensions`, `time_column` and `grains`. The `definition` is the SQL that computes it from
999
+ its table, such as `sum(dropped_frames) / NULLIF(sum(frames), 0)`. The table is empty when the folder has no metrics file.
1000
+
1001
+ Open it in the DuckDB command line, with `duckdb gold.duckdb`, or in the DuckDB UI, with `duckdb -ui gold.duckdb`. The UI
1002
+ fetches its web pages from ui.duckdb.org, so it needs the network.
1003
+
1004
+ ```sql
1005
+ SELECT table_name, owner, grain, release FROM _catalog;
1006
+ SELECT metric, definition FROM _metrics;
1007
+ SELECT robot_id, round(avg(success_rate), 2) AS success_rate FROM gold_robot_daily GROUP BY 1 ORDER BY 1;
1008
+ ```
1009
+
1010
+ - **It needs no extension.** The file has plain tables, so a DuckDB with no extension reads it, and no `iceberg_scan` is in
1011
+ it. It is written in the storage version of DuckDB 1.0, so a DuckDB 1.x opens it as well as a DuckDB 2.0. DuckDB 2.0
1012
+ writes a newer version by default, which a DuckDB 1.x cannot open.
1013
+ - **A reader never sees half a file.** It is written to a temporary path next to the file and renamed. A failure leaves the
1014
+ file that was there, and no temporary file.
1015
+ - **It holds one release.** A gold table that the release does not hold is left out, and the output names it, so that the
1016
+ file is as consistent as the release is. With `--live` it holds every table as it is now and `_catalog.release` has no
1017
+ value.
1018
+ - **`ldp gold build DIR --duckdb gold.duckdb`** writes the file after the build cuts a release. When the build cuts none,
1019
+ because the tables are at the snapshots of the current release, the file is written only if it is missing or holds
1020
+ another release. A build that a gate blocks or that fails leaves the file alone, so the file that readers have is the
1021
+ one of the last good release. A file that cannot be written after the release was cut is an error with exit status 1,
1022
+ and the release stays cut. `--duckdb` cannot be used with `--no-release`.
1023
+ - **Windows.** Replacing a file that a program has open fails on Windows. `{release}` in the name is replaced by the release,
1024
+ so `--out "gold_{release}.duckdb"` writes a new file for each release and never replaces one.
1025
+ - **Memory.** A table is read whole before it is written, as `ldp datasets export` does, so a gold table has to fit in memory.
1026
+
575
1027
  ## Catalogs and object storage
576
1028
 
577
1029
  The catalog block's `type` picks the catalog. It defaults to `local`, so a catalog block without
@@ -783,6 +1235,12 @@ A table or view model overwrites its table, and an incremental model with a `uni
783
1235
  published when dbt fails. Everything after the project folder goes to dbt unchanged, and the options of `ldp dbt`
784
1236
  itself go before it.
785
1237
 
1238
+ A published model also carries what the project says about it. Its description, the description of each column, its
1239
+ tags, its `owner` and `grain` meta and its layer become the table's [contract](docs/design/v0_1_4.md) and go into the
1240
+ Iceberg table. A model that is published to the `gold` namespace is a gold table, so it needs an owner, a grain, a
1241
+ description and a description of every column, and `ldp dbt build` stops before it publishes anything if one is
1242
+ missing.
1243
+
786
1244
  - The [warehousing guide](docs/warehousing.md) walks through it on a restaurant chain's Excel and CSV exports, from
787
1245
  the raw tables to tested models, churn, an incremental load and time travel. `make demo-warehouse` runs the same
788
1246
  example offline in about 25 seconds.
@@ -797,9 +1255,33 @@ make demo-orchestration # the restaurant's nightly drops, a bad row, a backf
797
1255
  ```
798
1256
 
799
1257
  - The [orchestration guide](docs/orchestration.md) walks through it on the restaurant example, command by command, from a window and its key to a workflow file, a backfill that stops at a bad row, a replaced day, a retried task, launchd and cron, and Airflow, Dagster and Prefect.
800
- - `make demo-orchestration` runs `examples/restaurant/run.py --layer orchestration`. It drops three nights of orders into a folder, with a bad row in the third. `ldp backfill` publishes two days and a data gate holds the third back, with exit status 3. After the fix, the same command runs only the third, and a third run skips every day. `ldp schedule tick` then loads the fourth night, and `ldp runs`, `ldp commits`, `ldp schedule list` and `ldp schedule install` show what happened. It needs no extra, runs the `models` task when the `dbt` extra is installed, and prints a skip line when it is not.
1258
+ - `make demo-orchestration` runs `examples/restaurant/run.py --layer orchestration`. It drops three nights of orders into a folder, with a bad row in the third. `ldp backfill` publishes two days and a data gate holds the third back, with exit status 3. After the fix, the same command runs only the third, and a third run skips every day. `ldp schedule tick` then loads the fourth night, and `ldp runs`, `ldp commits`, `ldp schedule list` and `ldp schedule install` show what happened. It needs no extra, runs the `models` and `gold` tasks when the `dbt` extra is installed, and prints a skip line when it is not.
801
1259
  - The pieces are described above, under [Windows](#windows), [Workflow files](#workflow-files), [Backfill a range of windows](#backfill-a-range-of-windows), [Run on a schedule](#run-on-a-schedule) and [Run from Airflow](#run-from-airflow), and the exit statuses are in the [table](#use-it-from-the-command-line) of the command line section.
802
1260
 
1261
+ ## Gold layer
1262
+
1263
+ A gold table is a table that people read. It says who owns it, what it means, what one row is and what each column holds, it is checked before it is written, and all the gold tables are published together as one release, so every reader sees the same build. A metric such as the average ticket is defined once. People who do not write code get the tables as a catalog page, an Excel file and one DuckDB file, and an AI agent can be limited to the gold tables.
1264
+
1265
+ ```bash
1266
+ make demo-gold # the restaurant's data mart, a faulty batch that the gate stops, a release, a metric and files for people
1267
+ ```
1268
+
1269
+ - The [gold layer guide](docs/gold.md) walks through it on the restaurant example, command by command, from the three layers and a gate that stops a bad batch to the contract of a gold table, a release that a gate can hold back, the catalog, metrics, files for people and `ldp mcp --layer gold`. It has a section for people who are not engineers, and every command in it runs in the tests.
1270
+ - `make demo-gold` runs `examples/restaurant/run.py --layer gold`. It loads the raw tables, stops a faulty week of orders at the gate with each failed check named, loads the week without the faults, builds the gold tables with dbt, cuts a release with `ldp gold build`, asks for a metric with `ldp metrics query`, and writes an Excel file, a DuckDB file and a catalog page. It needs the `dbt` and `excel` extras, and takes about 10 seconds.
1271
+ - The commands are described above, under [Table contracts](#table-contracts), [Build the gold tables](#build-the-gold-tables), [Release the gold tables](#release-the-gold-tables), [Browse the gold tables](#browse-the-gold-tables), [Define metrics](#define-metrics) and [Open the gold tables in a spreadsheet or a BI tool](#open-the-gold-tables-in-a-spreadsheet-or-a-bi-tool).
1272
+
1273
+ ## Monitoring
1274
+
1275
+ A table can be wrong for days before a person notices. A config says in `metadata.sla` how fresh, how big and how stable its table should be, and which business number must stay above a floor. `ldp monitor` checks the table against it between runs, an alert goes to Slack, a webhook or a file when a monitor changes state and again when it recovers, and `ldp status` shows every pipeline on one screen. Nothing needs a server, and a laptop can run it from cron or launchd.
1276
+
1277
+ ```bash
1278
+ make demo-monitor # a late table, a short batch, a new column and a revenue drop raise alerts, and a fresh night resolves them
1279
+ ```
1280
+
1281
+ - The [monitoring guide](docs/monitoring.md) walks through it on the restaurant example, command by command, from an SLA to a late table, a short batch, a new column, a day of low revenue, alerts that say each problem once, the status screen, and a cron or launchd job that looks every 15 minutes with a Healthchecks.io heartbeat. It says plainly what a monitor on a laptop cannot do, and every command in it runs in the tests.
1282
+ - `make demo-monitor` runs `examples/restaurant/run.py --layer monitoring`. It loads the orders of one night and runs `ldp monitor --now` 50 hours later, which reports a freshness error with exit status 2 and writes one alert to `.ldp/alerts.jsonl`. A short batch raises a volume error, a batch with a new column raises a schema warning, and a day with a tenth of the revenue raises an error on the `revenue` metric. A fresh night resolves them, a second look sends nothing, and `ldp status` shows the result. With `LDP_SLACK_WEBHOOK` set the same alerts reach Slack, and without it the demo is fully offline. It needs no extra, runs the revenue step when the `dbt` extra is installed, and prints a skip line when it is not. It takes about 15 seconds with dbt and about a second without it.
1283
+ - The commands are in the [platform commands](#platform-commands) table, and the [alerts page](docs/alerts.md) describes the channels and the events.
1284
+
803
1285
  ## MCP server for agents
804
1286
 
805
1287
  `local_data_platform.mcp_server` serves your tables to an AI agent over the Model Context Protocol,
@@ -825,6 +1307,8 @@ on stdio and read-only. Install the `mcp` extra, then point your MCP client at `
825
1307
  | `sample_rows(table, n)` | A sample of rows |
826
1308
  | `table_history(table)` | The table's snapshot history |
827
1309
  | `get_dataset(name)` | A pinned dataset version |
1310
+ | `list_metrics()` | The metrics that the `metrics.json` files of the `--config` folders define, each with what it means, its owner, format, table, dimensions and grains |
1311
+ | `query_metric(metrics, by, grain, filters, start, end, max_rows)` | Some metrics of one table, computed from the current release of the gold tables as the definitions say, with every value bound as a parameter |
828
1312
 
829
1313
  The guardrails are on by default. A query must be exactly one `SELECT` or `WITH` statement, as
830
1314
  DuckDB's parser sees it, so `COPY`, `ATTACH`, `INSTALL`, `SET`, `read_csv('/etc/passwd')` and
@@ -834,7 +1318,10 @@ Results are capped in rows and time, `--allow` limits the tables, and every call
834
1318
  tool, the SQL, row counts, duration and any error, never the result data. Records go to a JSONL file
835
1319
  as each call ends, and to the `_ldp.audit` Iceberg table in the first served catalog when the
836
1320
  server stops (`--no-iceberg-audit` turns that off).
837
- `make demo-agent` runs a scripted client against the demo table. See
1321
+ `query_metric` runs in the same sandbox as `query`, with the same row cap, timeout and audit, and its audit record
1322
+ has the arguments and the compiled SQL. It reads the current release of the gold tables, and `query` reads the tables
1323
+ as they are now. `python examples/agent_client.py` builds a small lake with a gold table, three metrics and a release,
1324
+ and calls both tools. `make demo-agent` runs a scripted client against the demo table. See
838
1325
  [MCP server for agents](docs/agents.md).
839
1326
 
840
1327
  ## Reproducible datasets and robotics
@@ -851,7 +1338,7 @@ episodes = Iceberg("gold_episode_stats", {"identifier": "robots", "warehouse_pat
851
1338
  train = datasets.pin(episodes, "humanoid_train", row_filter="success = true",
852
1339
  selected_fields=["episode_id", "robot_id", "task", "frame_count"])
853
1340
  rows = datasets.load(train) # the same rows at any later time
854
- datasets.export(train, "exports/humanoid_train_v1.parquet") # or format="jsonl"
1341
+ datasets.export(train, "exports/humanoid_train_v1.parquet") # or format="jsonl" or "csv"
855
1342
  ```
856
1343
 
857
1344
  Each pin also tags the snapshot (`ldp_ds_<name>_v<N>`), so snapshot expiry keeps it.
@@ -895,7 +1382,10 @@ Expiry always keeps the current snapshot, branch and tag heads and `main`'s last
895
1382
  table's idempotency horizon (7 days by default), because re-runs rely on finding it. Orphans are
896
1383
  files under the table's location that no metadata references and that are older than 72 hours
897
1384
  (`older_than_hours`), so the files of a write still in progress are left alone. `ldp maintain
898
- CONFIG` runs both from the command line, as a dry run unless `--apply` is given. There is no
1385
+ CONFIG` runs both from the command line, as a dry run unless `--apply` is given. `ldp maintain
1386
+ CONFIG --system` does the same for the `_ldp` history tables of the config's catalog, after it
1387
+ deletes their rows older than `--keep-days` days (90 by default), a whole UTC day at a time, which
1388
+ drops data files and rewrites none. Run it once a week where `ldp monitor` runs often. There is no
899
1389
  compaction yet.
900
1390
 
901
1391
  ## Spark (experimental)
@@ -972,8 +1462,10 @@ make install # create .venv and install the package with the dev and docs e
972
1462
  make lint # flake8
973
1463
  make test # pytest, offline: the Spark and REST integration tests skip
974
1464
  make demo # run the demo into ./ldp_demo
975
- make demo-all # every demo; demo-warehouse needs the dbt and excel extras, demo-spark and demo-rest need scala-cli
1465
+ make demo-all # every demo; demo-warehouse and demo-gold need the dbt and excel extras, demo-spark and demo-rest need scala-cli
976
1466
  make demo-orchestration # the restaurant's nightly workflow, a backfill and a scheduler tick
1467
+ make demo-gold # the restaurant data mart: a blocked batch, gold tables, a release, a metric and files for people
1468
+ make demo-monitor # the restaurant's orders watched through a late table, a short batch, a new column and a revenue drop
977
1469
  make spark-test # opt-in Spark integration tests (LDP_RUN_SPARK=1, scala-cli)
978
1470
  make rest-test # opt-in REST catalog integration tests (LDP_RUN_REST=1, scala-cli)
979
1471
  make dbt-test # the tests that need the dbt extra (pip install -e ".[dev,dbt]" first)
@@ -1019,8 +1511,8 @@ in each release, and the [contributing guide](docs/contributing.md) to get invol
1019
1511
  with no rows, leaves the models before it published. A model published with an upsert that gains or loses
1020
1512
  a column needs one `ldp run CONFIG --mode overwrite`.
1021
1513
  - **The restaurant example is in the repo, not in the package.** `pip install` does not include
1022
- `examples/`, so the warehousing and orchestration guides and `make demo-warehouse` and `make demo-orchestration`
1023
- need a clone.
1514
+ `examples/`, so the warehousing, orchestration and monitoring guides and `make demo-warehouse`, `make demo-orchestration`,
1515
+ `make demo-gold` and `make demo-monitor` need a clone.
1024
1516
  - **Tasks run one at a time.** A workflow runs its tasks in order and a backfill its windows in order, which also
1025
1517
  respects DuckDB's single writer. There are no parallel tasks.
1026
1518
  - **The scheduler is a tick, not a daemon.** `ldp schedule tick` has to be called by cron or launchd, and there is no
@@ -1030,17 +1522,26 @@ in each release, and the [contributing guide](docs/contributing.md) to get invol
1030
1522
  model that reads them is limited to the window, and the models of the restaurant example are not.
1031
1523
  - **Airflow runs one config for each task.** DAGs built from workflow files, an `LdpTaskOperator` and managed Airflow
1032
1524
  recipes are planned for 0.2.0.
1525
+ - **A release covers one namespace of one warehouse.** `ldp gold build` refuses a folder whose gold tables are in more
1526
+ than one place, and says to give each place its own folder of configs.
1527
+ - **A metrics query reads one table.** `ldp metrics query` and the `query_metric` tool refuse metrics of two tables in one
1528
+ question, so ask for them in separate queries.
1529
+ - **A monitor looks at one moment, on a machine that is running.** A table that was late and was fixed between two looks
1530
+ never alerts, and nothing alerts while the laptop is asleep or off. A heartbeat to an outside service such as
1531
+ Healthchecks.io is the only cover, and the [monitoring guide](docs/monitoring.md) says how to set it up.
1532
+ - **A change is measured against the median of the periods before it.** It needs 3 earlier values and has no sense of the
1533
+ weekday, so a business that is busier at weekends needs a wide `max_change_pct`, or a `min` and a `max`.
1534
+ - **A local catalog keeps its history on one machine.** Two machines that look at the same table each keep a history of
1535
+ their own, and each one alerts.
1033
1536
 
1034
1537
  ## Roadmap
1035
1538
 
1036
1539
  From 0.1.1 on, the package version equals the milestone number. Each milestone adds the next layer
1037
- of a data platform, and every layer runs on a laptop first. 0.1.3 Orchestration is released, and these
1540
+ of a data platform, and every layer runs on a laptop first. 0.1.5 Monitoring is released, and these
1038
1541
  are the milestones after it:
1039
1542
 
1040
1543
  | Version | Milestone | What it adds |
1041
1544
  |---|---|---|
1042
- | 0.1.4 | Self Serving Gold Layer | Gold tables with owners and docs, releases, metrics, a catalog and exports |
1043
- | 0.1.5 | Monitoring | SLAs, freshness, volume, schema and metric monitors, alerts and `ldp status` |
1044
1545
  | 0.1.6 | Business Intelligence Reporting Dashboarding | Offline HTML reports, Slack delivery, a live dashboard and BI tool recipes |
1045
1546
  | 0.1.7 | Data Science Insights | Churn, retention, funnels, sessions, training sets and a notebook |
1046
1547
  | 0.1.8 | LLM | Catalog search and `ldp ask`, text to SQL that corrects itself |