local-data-platform 0.1.2__tar.gz → 0.1.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/PKG-INFO +278 -13
  2. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/README.md +276 -12
  3. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/pyproject.toml +4 -1
  4. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/__init__.py +1 -1
  5. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/catalog/provider.py +38 -4
  6. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/cli.py +105 -18
  7. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/config.py +50 -3
  8. local_data_platform-0.1.3/src/local_data_platform/dbt_layer/build.py +349 -0
  9. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/dbt_layer/cli.py +13 -95
  10. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/dbt_layer/project.py +7 -3
  11. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/dbt_layer/publish.py +14 -3
  12. local_data_platform-0.1.3/src/local_data_platform/etl.py +275 -0
  13. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/events.py +99 -11
  14. local_data_platform-0.1.3/src/local_data_platform/filelock.py +99 -0
  15. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/iceberg/__init__.py +50 -40
  16. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/iceberg/commit.py +115 -8
  17. local_data_platform-0.1.3/src/local_data_platform/integrations/__init__.py +6 -0
  18. local_data_platform-0.1.3/src/local_data_platform/integrations/airflow.py +249 -0
  19. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/cli.py +60 -13
  20. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/discovery.py +8 -2
  21. local_data_platform-0.1.3/src/local_data_platform/orchestration/__init__.py +73 -0
  22. local_data_platform-0.1.3/src/local_data_platform/orchestration/backfills.py +541 -0
  23. local_data_platform-0.1.3/src/local_data_platform/orchestration/cli.py +282 -0
  24. local_data_platform-0.1.3/src/local_data_platform/orchestration/dbt_task.py +395 -0
  25. local_data_platform-0.1.3/src/local_data_platform/orchestration/ledger.py +164 -0
  26. local_data_platform-0.1.3/src/local_data_platform/orchestration/recipes.py +134 -0
  27. local_data_platform-0.1.3/src/local_data_platform/orchestration/retries.py +182 -0
  28. local_data_platform-0.1.3/src/local_data_platform/orchestration/runner.py +394 -0
  29. local_data_platform-0.1.3/src/local_data_platform/orchestration/schedule.py +681 -0
  30. local_data_platform-0.1.3/src/local_data_platform/orchestration/schedule_cli.py +219 -0
  31. local_data_platform-0.1.3/src/local_data_platform/orchestration/tasks.py +581 -0
  32. local_data_platform-0.1.3/src/local_data_platform/orchestration/windows.py +646 -0
  33. local_data_platform-0.1.3/src/local_data_platform/orchestration/workflow.py +593 -0
  34. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/__init__.py +212 -25
  35. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/pyarrow/__init__.py +3 -2
  36. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/registry.py +8 -2
  37. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/quality/checks.py +6 -3
  38. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/spec.py +464 -45
  39. local_data_platform-0.1.2/src/local_data_platform/etl.py +0 -141
  40. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/LICENSE +0 -0
  41. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/catalog/__init__.py +0 -0
  42. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/catalog/local/__init__.py +0 -0
  43. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/catalog/local/iceberg/__init__.py +0 -0
  44. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/datasets.py +0 -0
  45. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/dbt_layer/__init__.py +0 -0
  46. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/dbt_layer/sources.py +0 -0
  47. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/demo.py +0 -0
  48. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/engine/__init__.py +0 -0
  49. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/engine/duckdb/__init__.py +0 -0
  50. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/engine/duckdb/row_filter.py +0 -0
  51. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/engine/router.py +0 -0
  52. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/engine/spark/__init__.py +0 -0
  53. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/exceptions.py +0 -0
  54. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/__init__.py +0 -0
  55. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/csv/__init__.py +0 -0
  56. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/excel/__init__.py +0 -0
  57. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/format/parquet/__init__.py +0 -0
  58. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/fs.py +0 -0
  59. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/github/__init__.py +0 -0
  60. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/issue/__init__.py +0 -0
  61. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/lake.py +0 -0
  62. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/logger.py +0 -0
  63. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/__init__.py +0 -0
  64. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/_common.py +0 -0
  65. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/orphans.py +0 -0
  66. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/snapshots.py +0 -0
  67. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/__init__.py +0 -0
  68. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/__main__.py +0 -0
  69. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/audit.py +0 -0
  70. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/cli.py +0 -0
  71. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/errors.py +0 -0
  72. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/guard.py +0 -0
  73. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/sandbox.py +0 -0
  74. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/server.py +0 -0
  75. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/tools.py +0 -0
  76. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/paths.py +0 -0
  77. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/builders.py +0 -0
  78. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/builtin.py +0 -0
  79. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/__init__.py +0 -0
  80. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/csv_to_iceberg/__init__.py +0 -0
  81. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/iceberg_to_csv/__init__.py +0 -0
  82. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/iceberg_to_parquet/__init__.py +0 -0
  83. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/__init__.py +0 -0
  84. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/bigquery_to_csv/__init__.py +0 -0
  85. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/csv_to_iceberg/__init__.py +0 -0
  86. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/duckdb_to_iceberg/__init__.py +0 -0
  87. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/excel_to_csv/__init__.py +0 -0
  88. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/excel_to_iceberg/__init__.py +0 -0
  89. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/parquet_to_iceberg/__init__.py +0 -0
  90. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/quality/__init__.py +0 -0
  91. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/quality/report.py +0 -0
  92. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/quality/runner.py +0 -0
  93. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/quality/temporal.py +0 -0
  94. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/__init__.py +0 -0
  95. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/__init__.py +0 -0
  96. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/duckdb/__init__.py +0 -0
  97. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/gcp/__init__.py +0 -0
  98. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/gcp/bigquery/__init__.py +0 -0
  99. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/json/__init__.py +0 -0
  100. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/near/__init__.py +0 -0
  101. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/source/parquet/__init__.py +0 -0
  102. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/target/__init__.py +0 -0
  103. {local_data_platform-0.1.2 → local_data_platform-0.1.3}/src/local_data_platform/store/target/iceberg/__init__.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: local-data-platform
3
- Version: 0.1.2
3
+ Version: 0.1.3
4
4
  Summary: A Python API layer for running a complete Iceberg lakehouse data platform on your laptop.
5
5
  License: MIT
6
6
  License-File: LICENSE
@@ -24,6 +24,7 @@ Provides-Extra: s3
24
24
  Provides-Extra: spark
25
25
  Requires-Dist: boto3 (>=1.34) ; extra == "s3"
26
26
  Requires-Dist: build (>=1.2) ; extra == "dev"
27
+ Requires-Dist: cronsim (>=2.6,<3)
27
28
  Requires-Dist: dbt-core (>=1.12,<2) ; extra == "dbt"
28
29
  Requires-Dist: dbt-duckdb (>=1.11,<2) ; extra == "dbt"
29
30
  Requires-Dist: defusedxml (>=0.7) ; extra == "dev"
@@ -66,8 +67,11 @@ It's for learning how a lakehouse works, for building and testing pipelines loca
66
67
  for cloud infrastructure, and for giving agents and notebooks safe, reproducible access to the
67
68
  tables you build.
68
69
 
69
- > **Status.** This is version 0.1.2, the Warehousing milestone. It adds `ldp dbt`, which runs a dbt
70
- > project over your Iceberg tables and publishes the models you mark back to Iceberg (see
70
+ > **Status.** This is version 0.1.3, the Orchestration milestone. It adds windows and idempotency keys
71
+ > to `ldp run`, workflow files with `ldp workflow plan` and `ldp workflow run`, `ldp backfill`, retries,
72
+ > `ldp schedule` for cron and launchd, and an Airflow 3 operator (see [Orchestration](#orchestration)),
73
+ > with a [guide](docs/orchestration.md). Version 0.1.2 added `ldp dbt`, which runs a dbt project over
74
+ > your Iceberg tables and publishes the models you mark back to Iceberg (see
71
75
  > [Transform with dbt](#transform-with-dbt)), the `DUCKDB` and `XLSX` source formats, time travel and
72
76
  > folders of tables in `ldp query`, retries for concurrent appends, and a restaurant example with a
73
77
  > [guide](docs/warehousing.md). Version 0.1.1 was the first release on PyPI since 0.1.0. It carries
@@ -82,7 +86,7 @@ tables you build.
82
86
  `ldp` needs Python 3.12 or newer. Install it from PyPI, preferably in a virtual environment:
83
87
 
84
88
  ```bash
85
- pip install "local-data-platform[duckdb]>=0.1.2"
89
+ pip install "local-data-platform[duckdb]>=0.1.3"
86
90
 
87
91
  ldp demo --workdir ldp_demo
88
92
  ```
@@ -122,9 +126,10 @@ More demos, each one a `make` target (see [Development](#development)):
122
126
  | `make demo-robotics` | Humanoid-robot episodes and sensor frames in bronze, silver and gold Iceberg tables, temporal quality checks, a pinned training split and DuckDB research queries |
123
127
  | `make demo-agent` | The read-only MCP server on the demo table, driven by a scripted agent client |
124
128
  | `make demo-warehouse` | A restaurant chain from Excel and CSV exports to dbt-tested Iceberg tables with monthly customer churn, a late batch and time travel (needs the `dbt` and `excel` extras) |
129
+ | `make demo-orchestration` | The restaurant's nightly drops run as a workflow: a backfill that stops at a bad row and resumes after the fix, a third run that skips every day, and a scheduler tick (runs its dbt models task when the `dbt` extra is installed) |
125
130
  | `make demo-spark` | The Scala Spark job reading the demo table and writing an aggregate that pyiceberg reads back (needs scala-cli) |
126
131
  | `make demo-rest` | The demo config against an Iceberg REST catalog started locally on scala-cli's JVM: `ldp catalog test`, two runs, the snapshots and a query (needs scala-cli) |
127
- | `make demo-all` | All of them; `demo-warehouse` is skipped when dbt isn't installed and the two JVM demos when scala-cli isn't |
132
+ | `make demo-all` | All of them; `demo-warehouse` is skipped when dbt or the Excel reader isn't installed and the two JVM demos when scala-cli isn't |
128
133
 
129
134
  ### Extras
130
135
 
@@ -223,6 +228,11 @@ ldp --version
223
228
  ldp pipelines # list the registered source/target routes
224
229
  ldp run rides.json # run the config's pipeline
225
230
  ldp run rides.json --mode overwrite # override the config's write_mode
231
+ ldp run rides.json --window 2026-08-01/2026-08-02 # staged and exactly once: a second run for the window skips
232
+ ldp run orders.json --window yesterday # a window by name: also last-week and last-month, in the config's time zone
233
+ ldp run rides.json --window 2026-08-01/2026-08-02 --run-id RUN_ID --attempt 2 # a retry of that run
234
+ ldp run rides.json --idempotency-key sales-2026-08-01 # a key of your own, for an orchestrator
235
+ ldp run rides.json --window 2026-08-01/2026-08-02 --json # the result as JSON on stdout, for scripts
226
236
  ldp snapshots rides.json # list the target table's snapshots
227
237
  ldp query rides.json "SELECT count(*) FROM rides" # SQL over the target, named after target.name
228
238
  ldp query rides.json "SELECT * FROM rides" --max-rows 20 # print at most 20 rows (default 100)
@@ -232,7 +242,29 @@ ldp query config/ "SELECT o.id FROM raw.orders o JOIN raw.customers c ON c.id =
232
242
  ldp demo --workdir /tmp/ldp_demo # also --rows N (default 1000) and --seed N (default 42)
233
243
  ```
234
244
 
235
- Every command exits with 0 on success and 1 on error. Add `-v` to see a traceback. The commands
245
+ With `--window` or `--idempotency-key` a run is staged and exactly once. It prints its window, its
246
+ idempotency key, its run id and attempt, its status (`published` or `skipped_duplicate`) and the snapshot that holds
247
+ its rows, and running it again for the same window finds that snapshot and writes nothing. `ldp plan CONFIG --window W`
248
+ prints the same key, and `--idempotency-key auto` asks for it. The key is looked for in the whole history that the table
249
+ still holds, before the source is read, so a window that ran weeks ago is skipped without being read too. A retry
250
+ reuses the run id with the next attempt number,
251
+ and the new attempt fences the older ones, so a staged branch that an earlier attempt left behind never reaches `main`.
252
+ `--attempt` needs `--run-id`, and both need a window or a key. Without these options `ldp run` writes directly, as it
253
+ always did. A window is `START/END`, or `yesterday`, `last-week` (the last full Monday-to-Monday week) or `last-month`,
254
+ counted in the time zone of the config, and `ldp run` prints the window it stands for. What a window reads, and how a
255
+ config says so, is in [Windows](#windows).
256
+
257
+ Every command follows one exit-code table.
258
+
259
+ | Exit | Meaning |
260
+ |---|---|
261
+ | 0 | success, including a run that published and a run that skipped a duplicate |
262
+ | 1 | an error: a bad config, a usage error, a failed write |
263
+ | 2 | reserved for an SLA breach that `ldp monitor` reports in 0.1.5 |
264
+ | 3 | a data gate held the run back, such as a quality check that blocked a write |
265
+
266
+ A scheduler retries a run that exits with 1, because the cause may pass, and never retries one that exits with 3,
267
+ because the data has to change first. Add `-v` to see a traceback. The commands
236
268
  that read a table (`snapshots`, `query`, `commits`, `maintain`, `datasets pin`, `catalog test`) fail
237
269
  without creating anything when the config's catalog is a SQLite file that doesn't exist yet (a
238
270
  `local` catalog, or `sql` on SQLite). `ldp run`, `ldp snapshots` and `ldp query` also work with the
@@ -247,9 +279,15 @@ call:
247
279
  | Command | What it does | From Python |
248
280
  |---|---|---|
249
281
  | `ldp catalog test SPEC` | Connect to a catalog (a spec file, or a config's catalog block) and list its namespaces and tables | `catalog.provider.create_catalog(spec).list_namespaces()` |
250
- | `ldp schema` | Print the `ldp/v1` JSON Schema for configs | `spec.json_schema()` |
251
- | `ldp plan CONFIG [--window START/END] [--json]` | Validate a config (or a folder of them) and print its spec hash, route, target, sinks and idempotency key | `spec.validate_spec(data)`, `spec.plan(config, window=...)` |
252
- | `ldp runs CONFIG [--run RUN_ID] [--events]` | List the runs a config's sinks recorded | `events.summarize_runs(events.read_jsonl(path))` or `read_iceberg_events(catalog)` |
282
+ | `ldp schema [--kind KIND]` | Print the `ldp/v1` JSON Schema of a kind of file, the dataset config by default | `spec.json_schema()`, `spec.kind_schema(kind)` |
283
+ | `ldp workflow plan FILE [--window W] [--json]` | Check a workflow file, or a dataset config with a schedule, and every config it names, and print its tasks in the order they run with each pipeline task's idempotency key for the window | `orchestration.plan_workflow(path, window=...)` |
284
+ | `ldp backfill FILE --from START [--to END] [--dry-run] [--replace] [--run-backwards] [--max-windows N] [--stop-on-failure] [--json]` | Run a workflow, or a config with a window, for every window of a range, oldest first, each pipeline under the key of its window. A window that is done is skipped without being read, so running it again resumes where it stopped; `--dry-run` lists the windows with their keys and states, and `--replace` writes them again under a replace key; exit status 0, 1, or 3 for a data gate | `orchestration.backfill(path, start, end, ...)` |
285
+ | `ldp workflow run FILE --window W [--json]` | Run the tasks of a workflow one at a time for a window. A failed or blocked task stops the tasks that depend on it and the rest still run, and a second run skips what published or was built; exit status 0, 1, or 3 for a data gate | `orchestration.run_workflow(path, window=...)` |
286
+ | `ldp schedule tick DIR [--now TIME] [--max-windows N] [--json]` | Run every window that is due and not done, for every workflow and config under DIR that has a schedule, then exit. cron or launchd calls it every 5 minutes. A window that failed is carried on by the next tick as the same run with the next attempt, and a tick that starts while another runs does nothing; exit status 0, 1, or 3 for a data gate | `orchestration.schedule.tick(dir, now=...)` |
287
+ | `ldp schedule list DIR [--now TIME] [--max-windows N] [--json]` | List every scheduled workflow and config under DIR with its cron expression, time zone, next fire in UTC and in local time, last window and how it ended, the windows that are due, and the windows it missed with the `ldp backfill` command that fills them | `orchestration.schedule.list_scheduled(dir)` |
288
+ | `ldp schedule install --print {cron,launchd} DIR [--ldp PATH]` | Print the crontab line or the launchd plist that ticks DIR every 5 minutes. It never writes to a system location | `orchestration.recipes.recipe(kind, dir)` |
289
+ | `ldp plan CONFIG [--window WINDOW] [--json]` | Validate a config (or a folder of files, each as its kind) and print its spec hash, route, target, sinks, schedule, and the source path and idempotency key of a window | `spec.validate_spec(data)`, `spec.plan(config, window=...)` |
290
+ | `ldp runs CONFIG [--run RUN_ID] [--events] [--attempts]` | List the runs a config's sinks recorded, the last attempt of a run that was retried, or every attempt with `--attempts` | `events.summarize_runs(events.read_jsonl(path))` or `read_iceberg_events(catalog)` |
253
291
  | `ldp commits CONFIG [--key KEY] [--branches]` | List the staged publishes on a table, newest first | `format.iceberg.commit.commit_log(iceberg.table())` and `staged_branches(...)` |
254
292
  | `ldp datasets pin\|list\|export` | Pin, list and export dataset versions | `datasets.pin`, `list_versions`, `export` |
255
293
  | `ldp maintain CONFIG [--expire DAYS] [--orphans] [--apply]` | Expire snapshots and find orphan files; a dry run unless `--apply` | `maintenance.expire_snapshots`, `find_orphans`, `remove_orphans` |
@@ -264,11 +302,21 @@ and a `metadata` block:
264
302
 
265
303
  | Key | What it holds |
266
304
  |---|---|
267
- | `metadata.source` | `name`, `format` (`CSV`, `PARQUET`, `ICEBERG`, `JSON`, `DUCKDB` or `XLSX`), and a `path` (local, `s3://` or `gs://`) or, for Iceberg, a `catalog`. A `DUCKDB` source also has one of `relation` (a table or view) and `sql` (one `SELECT` query), see [A DuckDB file as a source](#a-duckdb-file-as-a-source). An `XLSX` source may have `sheet` and `header_row`, see [An Excel sheet as a source](#an-excel-sheet-as-a-source) |
305
+ | `metadata.source` | `name`, `format` (`CSV`, `PARQUET`, `ICEBERG`, `JSON`, `DUCKDB` or `XLSX`), and a `path` (local, `s3://` or `gs://`, and it may have `{window_start:FORMAT}` placeholders, see [Windows](#windows)) or, for Iceberg, a `catalog`. A `DUCKDB` source also has one of `relation` (a table or view) and `sql` (one `SELECT` query), see [A DuckDB file as a source](#a-duckdb-file-as-a-source). An `XLSX` source may have `sheet` and `header_row`, see [An Excel sheet as a source](#an-excel-sheet-as-a-source) |
268
306
  | `metadata.target` | The same, plus for Iceberg `write_mode` (`append`, `overwrite` or `upsert`), `join_cols` for upserts, and `partition_by` |
269
307
  | `metadata.target.catalog` | `type` (`local` by default, `sql`, `rest` or `glue`) and that type's keys; see [Catalogs](#catalogs-and-object-storage) |
270
308
  | `metadata.quality` | `on_failure` (`fail` or `warn`) and a list of `checks`: `row_count`, `not_null`, `unique`, `accepted_values`, `range`, `freshness`, `schema`, and the temporal checks `monotonic`, `max_skew`, `rate_below` and `max_gap` |
271
309
  | `metadata.observability` | Where run events go, for example `{"sinks": ["iceberg", "jsonl"]}`. The default sends them nowhere |
310
+ | `metadata.schedule` | When the config runs: a 5-field cron expression, or `@hourly`, `@daily`, `@weekly` or `@monthly`. Without it, a `when` of `hourly`, `daily`, `weekly` or `monthly` means the alias. Editing it does not change the spec hash |
311
+ | `metadata.window` | `"none"`, or `size` (`day`, `hour` or `15m`), an optional `column` that selects the rows of a window, and an optional `timezone` (UTC by default); see [Windows](#windows) |
312
+ | `metadata.retries` | How a workflow tries the run again after a failure: `max_attempts` (1 by default, which is no retry), `delay_s` (30) and `backoff` (2.0); see [Workflow files](#workflow-files). Editing it does not change the spec hash |
313
+
314
+ A config may also say `"apiVersion": "ldp/v1"` and `"kind": "Pipeline"`. `kind` says what a JSON file is. `Pipeline`
315
+ is the default, so every config here is one, and a config has the same spec hash with and without it. `Workflow` is
316
+ the second kind. A kind is written exactly as shown, so `workflow` fails with a message that names `Workflow`, and
317
+ `Metrics`, `Report` and `Profiles` are reserved for later milestones. `ldp plan` validates each file of a folder as its
318
+ kind, and `ldp query`, `ldp mcp --config` and `ldp dbt` read the tables of the `Pipeline` files only, so a folder can
319
+ hold workflow files next to its configs. `ldp schema --kind Workflow` prints the schema of another kind.
272
320
 
273
321
  Partition transforms are `identity`, `year`, `month`, `day`, `hour`, `bucket[N]` and
274
322
  `truncate[W]`. On pyiceberg 0.10 and later, writing with any transform except `identity` needs the
@@ -278,6 +326,201 @@ example of every check, is in the [v0.1.1 hardening contract](docs/design/v0_1_1
278
326
  `local_data_platform.spec.json_schema()` returns it as JSON Schema, and `ldp schema` prints it. The configs in
279
327
  [`examples/`](examples/README.md) are real ones you can run.
280
328
 
329
+ ### Windows
330
+
331
+ A run can be responsible for one window of time, `[start, end)`, which is what `--window` gives it. A config says how a
332
+ window picks what the run reads, and when the config runs:
333
+
334
+ ```json
335
+ {
336
+ "identifier": "daily_orders",
337
+ "when": "daily",
338
+ "metadata": {
339
+ "schedule": "5 2 * * *",
340
+ "window": {"size": "day", "column": "ordered_at", "timezone": "Europe/Paris"},
341
+ "source": {"name": "orders", "format": "CSV", "path": "drops/orders_{window_start:%Y-%m-%d}.csv"},
342
+ "target": {"name": "orders", "format": "ICEBERG", "write_mode": "overwrite",
343
+ "catalog": {"identifier": "shop", "warehouse_path": "warehouse"}}
344
+ }
345
+ }
346
+ ```
347
+
348
+ - `{window_start:FORMAT}` and `{window_end:FORMAT}` in `metadata.source.path` name the file of the window, with a Python
349
+ `strftime` FORMAT shown in the time zone of the window. A run without a window on such a path fails and names the
350
+ placeholder.
351
+ - `window.column` keeps only the rows at or after the start of the window and before its end, before the transforms and
352
+ the quality checks. `run.extracted` reports the rows read and the rows kept.
353
+ - With `window.column` and the write mode `overwrite`, a run replaces only the rows of its window, so one changed day can
354
+ be redone and the other days stay. A published window is skipped when it runs again, so to redo a day that changed at
355
+ the source, replace it with `ldp backfill daily_orders.json --from 2026-08-02 --to 2026-08-03 --replace`, or run it with a
356
+ key of your own: `ldp run daily_orders.json --window 2026-08-02/2026-08-03 --idempotency-key orders-2026-08-02-v2`.
357
+ - A `freshness` check in a windowed run measures the age of the data at the end of the window.
358
+ - A config with a schedule that appends, and that has neither a placeholder nor a `window.column`, would load the same
359
+ rows in every window. `ldp plan` reports that as an error and `ldp run --window` logs a warning.
360
+
361
+ The [contract](docs/design/v0_1_1.md#windows-and-schedules-013) has the full rules, including how each column type is
362
+ compared.
363
+
364
+ ### Workflow files
365
+
366
+ A workflow file lists the pipelines, dbt models and maintenance that run for each window, and in what order:
367
+
368
+ ```json
369
+ {
370
+ "apiVersion": "ldp/v1",
371
+ "kind": "Workflow",
372
+ "identifier": "daily_orders",
373
+ "metadata": {"schedule": "5 2 * * *", "window": {"size": "day"}, "retries": {"max_attempts": 3, "delay_s": 30}},
374
+ "tasks": [
375
+ {"id": "orders", "pipeline": "config/orders.json"},
376
+ {"id": "customers", "pipeline": "config/customers.json"},
377
+ {"id": "gold", "dbt": {"project": "dbt", "select": "+daily_revenue"}, "depends_on": ["orders", "customers"]},
378
+ {"id": "export", "pipeline": "config/export.json", "depends_on": ["gold"]},
379
+ {"id": "tidy", "maintain": {"config": "config/orders.json", "expire": 7}, "depends_on": ["export"]}
380
+ ]
381
+ }
382
+ ```
383
+
384
+ ```bash
385
+ ldp workflow plan daily_orders.json --window 2026-08-01/2026-08-02 # the tasks in run order, with each key
386
+ ldp workflow run daily_orders.json --window 2026-08-01/2026-08-02 # run them for that day
387
+ ldp runs .ldp/events.jsonl # the workflow run and the run of each task
388
+ ```
389
+
390
+ - Each task has an `id`, an optional `depends_on`, and exactly one kind: `pipeline` (the path of a dataset config,
391
+ relative to the workflow file), `dbt` (a dbt project, built for the window, see below) or `maintain` (a config's Iceberg
392
+ table and the options of `ldp maintain`, a dry run unless `apply` is true). Later milestones add kinds without changing the
393
+ file format.
394
+ - Tasks run one at a time, in dependency order. A pipeline task runs with the window of the workflow, so its idempotency key
395
+ is the one `ldp plan CONFIG --window` prints.
396
+ - A task that fails, or that a quality check blocks, makes the tasks that depend on it `upstream_failed`, and they do not run.
397
+ The tasks that do not depend on it still run. The exit status is 1 when a task failed and 3 when the only problem is a
398
+ data gate.
399
+ - Run the same window again after fixing the cause: the pipeline tasks that published are skipped by their keys, without
400
+ reading their sources, and only the tasks that failed or did not run do their work.
401
+ - A task that fails with an error that another attempt may fix runs again by itself, as `retries` says, in the workflow
402
+ or in the task, and a key of the task overrides the same key of the workflow. These are retried: a dropped connection,
403
+ a timeout, a file that has not landed yet (any `OSError`) and the commit errors of pyiceberg. A data gate, a wrong
404
+ config and a fenced attempt never are. A retry is the next attempt of the same run, with the same run id and the same
405
+ idempotency key, so the new attempt fences the old one and a load that already landed is found and not written
406
+ twice. The wait is a random time from 0 to `delay_s * backoff ** (n - 1)` seconds after attempt n. `ldp runs` lists the
407
+ last attempt of each run, and `ldp runs LEDGER --attempts` lists them all.
408
+ - A `dbt` task does what [`ldp dbt build`](#transform-with-dbt) does, for the window: dbt gets `window_start` and
409
+ `window_end` as variables (`'{{ var("window_start") }}'::timestamp` in a model), and the models marked `ldp_publish` are
410
+ written to Iceberg. Failing data tests are a data gate, so the task is `blocked_quality` and the exit status is 3. Run the
411
+ same window again and the task is skipped when its last build succeeded with the same project and no task before it may
412
+ have written data. List the pipelines that load the tables a dbt project reads in its `depends_on`, and name the dbt task
413
+ in the `depends_on` of every task that reads the tables it publishes.
414
+ - Every event of a run goes to `.ldp/events.jsonl` next to the workflow file, with the `workflow`, `workflow_run_id` and
415
+ `task_id` of the task in its payload. A dataset config that has a `metadata.schedule` runs as a workflow of one task.
416
+
417
+ The [contract](docs/design/v0_1_1.md#workflow-files-013) has the full file format. `ldp schema --kind Workflow` prints its JSON
418
+ Schema.
419
+
420
+ ### Backfill a range of windows
421
+
422
+ `ldp backfill` runs a workflow, or a config that has a window, for every window of a range. It is how a new pipeline loads
423
+ its history, and how a load that stopped halfway finishes:
424
+
425
+ ```bash
426
+ ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 --dry-run # every window, with its key and its state
427
+ ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 # load them, oldest first
428
+ ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 # again: it skips every window that is done
429
+ ldp backfill daily_orders.json --from 2026-08-02 --to 2026-08-03 --replace # redo a day that changed at the source
430
+ ```
431
+
432
+ - The windows are the windows of `metadata.window.size` (`day`, `hour` or `15m`), or of the size the schedule means, a day for
433
+ `@daily` and an hour for `@hourly`. `--from` and `--to` are dates or datetimes in the time zone of the workflow. Each of them
434
+ has to start a window, and the window that starts at `--to` is not part of the range, so `--from 2026-06-01 --to
435
+ 2026-06-04` is the three days of June 1st, 2nd and 3rd. Without `--to` the range ends with the last complete window. A
436
+ range of more than `--max-windows` windows, 1000 by default, is refused, to catch a typo.
437
+ - Each window runs as `ldp workflow run` runs it, so each pipeline publishes under the key that `ldp plan CONFIG --window W`
438
+ prints, and a task with `retries` is retried. A window whose keys are on the table is skipped without being read, planned
439
+ or written to the ledger, so a finished backfill that runs again does nothing, and one that stopped at window 40 starts
440
+ again at window 40. A workflow with a task that a key cannot vouch for, such as a dbt task or a pipeline that writes a file,
441
+ also needs the last run of the window in the ledger to have succeeded for every task.
442
+ - A window that fails, or that a quality check blocks, does not stop the others, and it is listed with what went wrong,
443
+ such as the check that failed. The summary counts the windows that published, were skipped, were blocked and failed.
444
+ The exit status is 0 when no window failed or was blocked, 3 when a data gate is the only problem, and 1 otherwise.
445
+ `--stop-on-failure` stops at the first window that failed instead, and a window that a quality check blocked does not
446
+ stop it.
447
+ - `--dry-run` prints every window with its key and its state, which is `published` (done, so it is skipped), `partial`,
448
+ `missing`, `failed` or `blocked` (the last run of it in the ledger failed or was held back by a data gate), and runs
449
+ nothing, not even the creation of a table. `--run-backwards` runs the newest window first, and `--json` prints the result.
450
+ - `--replace` writes each window again under a replace key, the sha256 of its key, the word `replace` and the spec hash of
451
+ its config, with a window overwrite. The rows that a window has are replaced by the rows the source has now, a window that
452
+ has none is loaded, a window that was replaced under the same spec is skipped, and one that was replaced under an older
453
+ spec is replaced again. It needs a `metadata.window.column` in every pipeline that writes to Iceberg, and without one it
454
+ exits with 1 and changes nothing. A window that is replaced has to have rows in the source. An empty one is not written,
455
+ because the run fails with "No data to write", or is blocked by a `row_count` check, and the table keeps its rows.
456
+ - From Python, `from local_data_platform.orchestration import backfill` runs the same, and `backfill(path, start, end,
457
+ dry_run=True)` returns the windows with their keys and states. The [contract](docs/design/v0_1_1.md#backfills-013) has the
458
+ rest.
459
+
460
+ ### Run on a schedule
461
+
462
+ `ldp schedule tick` is a scheduler with no server and no daemon. cron or launchd calls it every 5 minutes, it runs every
463
+ window that is due and not done, and it exits, so a laptop that was asleep catches up at the first tick after it wakes:
464
+
465
+ ```bash
466
+ ldp schedule list ldp_demo/restaurant # what is scheduled, when it fires, what it missed
467
+ ldp schedule tick ldp_demo/restaurant --now 2026-08-05T02:05:00Z # run what is due at that time, then exit
468
+ ldp schedule install --print launchd ldp_demo/restaurant > ~/Library/LaunchAgents/restaurant.plist # macOS
469
+ (crontab -l 2>/dev/null; ldp schedule install --print cron ldp_demo/restaurant) | crontab - # Linux
470
+ ```
471
+
472
+ - A tick looks at every `.json` file under the folder that is a workflow file, or a dataset config, and has a schedule in
473
+ `metadata.schedule` or in a `when` such as `daily`. A config with a schedule is a workflow of one task. A file that has a
474
+ schedule and is not valid is printed with its problems and makes the tick exit with 1, and the other files still run.
475
+ - The schedule is a cron expression read in `metadata.window.timezone`, UTC without one. The window a fire owns is the
476
+ interval between the previous fire and this fire. When the workflow has a window size, in `metadata.window.size` or implied
477
+ by `@daily` and `@hourly`, it is the last complete window of that size at or before the fire, so the fire of `5 2 * * *` on
478
+ 5 August owns 4 August. A fixed time that a clock change skips fires right after the gap, and one that happens twice fires
479
+ once.
480
+ - Without `catchup`, a tick runs the window of the latest fire if it is not done, and `ldp schedule list` prints the
481
+ `ldp backfill` command for any windows before it that were missed. With `"catchup": true` and a `"start"` in the metadata of
482
+ the workflow, a tick runs every window since `start` that is not done, oldest first, up to `--max-windows` (1000) a tick.
483
+ - A window that failed, was blocked by a data gate, or never finished is run again by the next tick, as the next attempt of
484
+ the same run. The workflow and each task keep their run ids and the attempts go up by one, so the first commit of the new
485
+ attempt fences every older one and the window is loaded once however many ticks it takes. `retries` are the retries within
486
+ one tick.
487
+ - A tick takes the lock `DIR/.ldp/locks/schedule.lock` without waiting for it. A tick that starts while another runs does
488
+ nothing and exits with 0, and a crash leaves no lock. The exit status is 0 when nothing failed or was blocked, 3 when a data
489
+ gate is the only problem, and 1 otherwise.
490
+ - The cron line and the plist call `ldp` by its absolute path and write a log, `DIR/.ldp/schedule.log` or a file in
491
+ `~/Library/Logs`. cron does not run while a Mac sleeps, and launchd runs an interval that passed during sleep once when the
492
+ Mac wakes, so launchd is the recipe for macOS and cron the recipe for Linux.
493
+ - From Python, `from local_data_platform.orchestration.schedule import tick, list_scheduled` does the same. The
494
+ [contract](docs/design/v0_1_1.md#schedules-013) has the rest.
495
+
496
+ ### Run from Airflow
497
+
498
+ A team that already runs Apache Airflow 3 can run a config as a task. `LdpRunOperator` takes the data interval of the DAG run
499
+ as the window, `ti.try_number` as the attempt, and a run id made of the DAG id, the run id of the DAG run and the task id. So
500
+ a retry by Airflow is the next attempt of the same run, a backfill over a day that is in the table finds the key and writes
501
+ nothing, and a day is loaded once however Airflow got to it. The [guide](docs/airflow.md) installs Airflow, runs the daily
502
+ drops of the restaurant example from `examples/restaurant/airflow_dag.py`, and retries a late drop. Its commands, in short, are these.
503
+
504
+ ```bash
505
+ export LDP_AIRFLOW_DEMO="$PWD/ldp_demo/airflow/data"
506
+ python examples/restaurant/airflow_dag.py --prepare "$LDP_AIRFLOW_DEMO" # the seven daily drops and a windowed orders config
507
+ airflow standalone # terminal 1, with the DAG file in its dags folder
508
+ airflow dags unpause restaurant_orders # terminal 2, catches up seven days one after the other
509
+ ldp commits "$LDP_AIRFLOW_DEMO/config/orders.json" # terminal 2, one publish for each day
510
+ ```
511
+
512
+ - The window is `(data_interval_start, data_interval_end)`, so the key of an interval is the one
513
+ `ldp plan CONFIG --window START/END` prints for it.
514
+ - A data gate that holds a run back, and a config that cannot run, fail the task with `AirflowFailException`, so Airflow does
515
+ not retry them. A skipped duplicate succeeds, and the `PipelineResult` as a dict is the XCom of the task. Any other error is
516
+ left to the `retries` of the task.
517
+ - Give the DAG a `CronDataIntervalTimetable("@daily", timezone="UTC")` as its schedule and not the string `"@daily"`. Airflow 3
518
+ makes a trigger timetable for a cron string, whose runs have an empty data interval, and the operator fails them and says why.
519
+ - The mapping is plain Python and Airflow is imported when the operator is used, so `local_data_platform.integrations.airflow`
520
+ imports without it. There is no extra for it, because Airflow has to be installed with its own constraints file. Airflow 3.1.7
521
+ and later install next to LDP, and a team that cannot do that runs `ldp run CONFIG --window ... --run-id ... --attempt ...`
522
+ from a BashOperator, which the guide shows. The [contract](docs/design/v0_1_1.md#airflow-013) has the rest.
523
+
281
524
  ### A DuckDB file as a source
282
525
 
283
526
  A `DUCKDB` source reads one table, or the result of one query, from a DuckDB database file and
@@ -545,6 +788,18 @@ itself go before it.
545
788
  example offline in about 25 seconds.
546
789
  - [How `ldp dbt` works](docs/design/warehousing.md) has the contract, including the meta keys and the schema rules.
547
790
 
791
+ ## Orchestration
792
+
793
+ A workflow file declares the pipelines, dbt models and maintenance of a night once. LDP runs it for the right window, backfills a range of windows, retries what failed, and runs it on a schedule from cron or launchd, with no service. Every run goes by the idempotency key of its window, so a window is never loaded twice, whoever starts it.
794
+
795
+ ```bash
796
+ make demo-orchestration # the restaurant's nightly drops, a bad row, a backfill, a fix and a scheduler tick
797
+ ```
798
+
799
+ - The [orchestration guide](docs/orchestration.md) walks through it on the restaurant example, command by command, from a window and its key to a workflow file, a backfill that stops at a bad row, a replaced day, a retried task, launchd and cron, and Airflow, Dagster and Prefect.
800
+ - `make demo-orchestration` runs `examples/restaurant/run.py --layer orchestration`. It drops three nights of orders into a folder, with a bad row in the third. `ldp backfill` publishes two days and a data gate holds the third back, with exit status 3. After the fix, the same command runs only the third, and a third run skips every day. `ldp schedule tick` then loads the fourth night, and `ldp runs`, `ldp commits`, `ldp schedule list` and `ldp schedule install` show what happened. It needs no extra, runs the `models` task when the `dbt` extra is installed, and prints a skip line when it is not.
801
+ - The pieces are described above, under [Windows](#windows), [Workflow files](#workflow-files), [Backfill a range of windows](#backfill-a-range-of-windows), [Run on a schedule](#run-on-a-schedule) and [Run from Airflow](#run-from-airflow), and the exit statuses are in the [table](#use-it-from-the-command-line) of the command line section.
802
+
548
803
  ## MCP server for agents
549
804
 
550
805
  `local_data_platform.mcp_server` serves your tables to an AI agent over the Model Context Protocol,
@@ -718,6 +973,7 @@ make lint # flake8
718
973
  make test # pytest, offline: the Spark and REST integration tests skip
719
974
  make demo # run the demo into ./ldp_demo
720
975
  make demo-all # every demo; demo-warehouse needs the dbt and excel extras, demo-spark and demo-rest need scala-cli
976
+ make demo-orchestration # the restaurant's nightly workflow, a backfill and a scheduler tick
721
977
  make spark-test # opt-in Spark integration tests (LDP_RUN_SPARK=1, scala-cli)
722
978
  make rest-test # opt-in REST catalog integration tests (LDP_RUN_REST=1, scala-cli)
723
979
  make dbt-test # the tests that need the dbt extra (pip install -e ".[dev,dbt]" first)
@@ -763,17 +1019,26 @@ in each release, and the [contributing guide](docs/contributing.md) to get invol
763
1019
  with no rows, leaves the models before it published. A model published with an upsert that gains or loses
764
1020
  a column needs one `ldp run CONFIG --mode overwrite`.
765
1021
  - **The restaurant example is in the repo, not in the package.** `pip install` does not include
766
- `examples/`, so the warehousing guide and `make demo-warehouse` need a clone.
1022
+ `examples/`, so the warehousing and orchestration guides and `make demo-warehouse` and `make demo-orchestration`
1023
+ need a clone.
1024
+ - **Tasks run one at a time.** A workflow runs its tasks in order and a backfill its windows in order, which also
1025
+ respects DuckDB's single writer. There are no parallel tasks.
1026
+ - **The scheduler is a tick, not a daemon.** `ldp schedule tick` has to be called by cron or launchd, and there is no
1027
+ foreground loop and no systemd or Windows Task Scheduler recipe. A window that failed is run again by every tick, every
1028
+ 5 minutes under the recipes, with no wait between ticks. `retries` are the retries inside one tick.
1029
+ - **A dbt task builds its whole selection for every window.** dbt gets `window_start` and `window_end`, but only a
1030
+ model that reads them is limited to the window, and the models of the restaurant example are not.
1031
+ - **Airflow runs one config for each task.** DAGs built from workflow files, an `LdpTaskOperator` and managed Airflow
1032
+ recipes are planned for 0.2.0.
767
1033
 
768
1034
  ## Roadmap
769
1035
 
770
1036
  From 0.1.1 on, the package version equals the milestone number. Each milestone adds the next layer
771
- of a data platform, and every layer runs on a laptop first. 0.1.2 Warehousing is released, and these
1037
+ of a data platform, and every layer runs on a laptop first. 0.1.3 Orchestration is released, and these
772
1038
  are the milestones after it:
773
1039
 
774
1040
  | Version | Milestone | What it adds |
775
1041
  |---|---|---|
776
- | 0.1.3 | Orchestration | Windows, backfills, retries and a laptop scheduler, plus an Airflow operator |
777
1042
  | 0.1.4 | Self Serving Gold Layer | Gold tables with owners and docs, releases, metrics, a catalog and exports |
778
1043
  | 0.1.5 | Monitoring | SLAs, freshness, volume, schema and metric monitors, alerts and `ldp status` |
779
1044
  | 0.1.6 | Business Intelligence Reporting Dashboarding | Offline HTML reports, Slack delivery, a live dashboard and BI tool recipes |