local-data-platform 0.1.1__tar.gz → 0.1.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- local_data_platform-0.1.3/PKG-INFO +1065 -0
- local_data_platform-0.1.3/README.md +1007 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/pyproject.toml +14 -1
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/__init__.py +3 -1
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/catalog/provider.py +38 -4
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/cli.py +152 -29
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/config.py +50 -3
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/__init__.py +16 -0
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/build.py +349 -0
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/cli.py +217 -0
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/project.py +386 -0
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/publish.py +394 -0
- local_data_platform-0.1.3/src/local_data_platform/dbt_layer/sources.py +366 -0
- local_data_platform-0.1.3/src/local_data_platform/etl.py +275 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/events.py +105 -11
- local_data_platform-0.1.3/src/local_data_platform/filelock.py +99 -0
- local_data_platform-0.1.3/src/local_data_platform/format/excel/__init__.py +284 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/format/iceberg/__init__.py +111 -53
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/format/iceberg/commit.py +118 -10
- local_data_platform-0.1.3/src/local_data_platform/integrations/__init__.py +6 -0
- local_data_platform-0.1.3/src/local_data_platform/integrations/airflow.py +249 -0
- local_data_platform-0.1.3/src/local_data_platform/lake.py +202 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/cli.py +60 -13
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/discovery.py +8 -2
- local_data_platform-0.1.3/src/local_data_platform/orchestration/__init__.py +73 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/backfills.py +541 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/cli.py +282 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/dbt_task.py +395 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/ledger.py +164 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/recipes.py +134 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/retries.py +182 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/runner.py +394 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/schedule.py +681 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/schedule_cli.py +219 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/tasks.py +581 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/windows.py +646 -0
- local_data_platform-0.1.3/src/local_data_platform/orchestration/workflow.py +593 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/__init__.py +212 -25
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/builders.py +36 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/builtin.py +10 -3
- local_data_platform-0.1.3/src/local_data_platform/pipeline/ingestion/duckdb_to_iceberg/__init__.py +35 -0
- local_data_platform-0.1.3/src/local_data_platform/pipeline/ingestion/excel_to_csv/__init__.py +31 -0
- local_data_platform-0.1.3/src/local_data_platform/pipeline/ingestion/excel_to_iceberg/__init__.py +31 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/pyarrow/__init__.py +3 -2
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/registry.py +8 -2
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/quality/checks.py +6 -3
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/spec.py +515 -38
- local_data_platform-0.1.3/src/local_data_platform/store/source/duckdb/__init__.py +155 -0
- local_data_platform-0.1.1/PKG-INFO +0 -680
- local_data_platform-0.1.1/README.md +0 -632
- local_data_platform-0.1.1/src/local_data_platform/etl.py +0 -141
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/LICENSE +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/catalog/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/catalog/local/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/catalog/local/iceberg/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/datasets.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/demo.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/engine/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/engine/duckdb/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/engine/duckdb/row_filter.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/engine/router.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/engine/spark/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/exceptions.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/format/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/format/csv/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/format/parquet/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/fs.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/github/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/issue/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/logger.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/_common.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/orphans.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/maintenance/snapshots.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/__main__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/audit.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/cli.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/errors.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/guard.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/sandbox.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/server.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/mcp_server/tools.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/paths.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/csv_to_iceberg/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/iceberg_to_csv/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/egression/iceberg_to_parquet/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/bigquery_to_csv/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/csv_to_iceberg/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/pipeline/ingestion/parquet_to_iceberg/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/quality/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/quality/report.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/quality/runner.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/quality/temporal.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/gcp/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/gcp/bigquery/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/json/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/near/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/source/parquet/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/target/__init__.py +0 -0
- {local_data_platform-0.1.1 → local_data_platform-0.1.3}/src/local_data_platform/store/target/iceberg/__init__.py +0 -0
|
@@ -0,0 +1,1065 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: local-data-platform
|
|
3
|
+
Version: 0.1.3
|
|
4
|
+
Summary: A Python API layer for running a complete Iceberg lakehouse data platform on your laptop.
|
|
5
|
+
License: MIT
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Author: Tushar Choudhary
|
|
8
|
+
Requires-Python: >=3.12,<4
|
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
10
|
+
Classifier: Programming Language :: Python :: 3
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.15
|
|
15
|
+
Provides-Extra: bigquery
|
|
16
|
+
Provides-Extra: dbt
|
|
17
|
+
Provides-Extra: dev
|
|
18
|
+
Provides-Extra: docs
|
|
19
|
+
Provides-Extra: duckdb
|
|
20
|
+
Provides-Extra: excel
|
|
21
|
+
Provides-Extra: glue
|
|
22
|
+
Provides-Extra: mcp
|
|
23
|
+
Provides-Extra: s3
|
|
24
|
+
Provides-Extra: spark
|
|
25
|
+
Requires-Dist: boto3 (>=1.34) ; extra == "s3"
|
|
26
|
+
Requires-Dist: build (>=1.2) ; extra == "dev"
|
|
27
|
+
Requires-Dist: cronsim (>=2.6,<3)
|
|
28
|
+
Requires-Dist: dbt-core (>=1.12,<2) ; extra == "dbt"
|
|
29
|
+
Requires-Dist: dbt-duckdb (>=1.11,<2) ; extra == "dbt"
|
|
30
|
+
Requires-Dist: defusedxml (>=0.7) ; extra == "dev"
|
|
31
|
+
Requires-Dist: defusedxml (>=0.7) ; extra == "excel"
|
|
32
|
+
Requires-Dist: duckdb (>=1.1) ; extra == "dbt"
|
|
33
|
+
Requires-Dist: duckdb (>=1.1) ; extra == "dev"
|
|
34
|
+
Requires-Dist: duckdb (>=1.1) ; extra == "duckdb"
|
|
35
|
+
Requires-Dist: duckdb (>=1.1) ; extra == "mcp"
|
|
36
|
+
Requires-Dist: flake8 (>=7) ; extra == "dev"
|
|
37
|
+
Requires-Dist: google-auth (>=2.20) ; extra == "bigquery"
|
|
38
|
+
Requires-Dist: google-cloud-bigquery (>=3.20) ; extra == "bigquery"
|
|
39
|
+
Requires-Dist: mcp (>=2.2,<3) ; extra == "dev"
|
|
40
|
+
Requires-Dist: mcp (>=2.2,<3) ; extra == "mcp"
|
|
41
|
+
Requires-Dist: mkdocs (>=1.6.0) ; extra == "docs"
|
|
42
|
+
Requires-Dist: mkdocs-material (==9.5.21) ; extra == "docs"
|
|
43
|
+
Requires-Dist: mkdocstrings[python] (>=0.25.0) ; extra == "docs"
|
|
44
|
+
Requires-Dist: moto[server] (>=5.0) ; extra == "dev"
|
|
45
|
+
Requires-Dist: openpyxl (>=3.1) ; extra == "dev"
|
|
46
|
+
Requires-Dist: openpyxl (>=3.1) ; extra == "excel"
|
|
47
|
+
Requires-Dist: pyarrow (>=17)
|
|
48
|
+
Requires-Dist: pyiceberg[glue] (>=0.11,<0.13) ; extra == "glue"
|
|
49
|
+
Requires-Dist: pyiceberg[pyarrow,pyiceberg-core,sql-sqlite] (>=0.11,<0.13)
|
|
50
|
+
Requires-Dist: pymdown-extensions (>=10.8.1) ; extra == "docs"
|
|
51
|
+
Requires-Dist: pyspark (>=4.0,<4.2) ; extra == "spark"
|
|
52
|
+
Requires-Dist: pytest (>=8) ; extra == "dev"
|
|
53
|
+
Requires-Dist: requests (>=2.31)
|
|
54
|
+
Project-URL: Documentation, https://local-data-platform.readthedocs.io
|
|
55
|
+
Project-URL: Repository, https://github.com/tusharchou/local-data-platform
|
|
56
|
+
Description-Content-Type: text/markdown
|
|
57
|
+
|
|
58
|
+
# Local Data Platform
|
|
59
|
+
|
|
60
|
+
**local-data-platform** (`ldp`) is a Python library for running an Apache Iceberg lakehouse on your
|
|
61
|
+
laptop, and for taking the same pipelines to a shared catalog and object storage when you need to.
|
|
62
|
+
You describe a dataset in a JSON config, and `ldp` loads it into an Iceberg table, runs data quality
|
|
63
|
+
checks before the write, and lets you query the table with DuckDB SQL and read earlier snapshots.
|
|
64
|
+
The default setup needs no server and no cloud account.
|
|
65
|
+
|
|
66
|
+
It's for learning how a lakehouse works, for building and testing pipelines locally before you pay
|
|
67
|
+
for cloud infrastructure, and for giving agents and notebooks safe, reproducible access to the
|
|
68
|
+
tables you build.
|
|
69
|
+
|
|
70
|
+
> **Status.** This is version 0.1.3, the Orchestration milestone. It adds windows and idempotency keys
|
|
71
|
+
> to `ldp run`, workflow files with `ldp workflow plan` and `ldp workflow run`, `ldp backfill`, retries,
|
|
72
|
+
> `ldp schedule` for cron and launchd, and an Airflow 3 operator (see [Orchestration](#orchestration)),
|
|
73
|
+
> with a [guide](docs/orchestration.md). Version 0.1.2 added `ldp dbt`, which runs a dbt project over
|
|
74
|
+
> your Iceberg tables and publishes the models you mark back to Iceberg (see
|
|
75
|
+
> [Transform with dbt](#transform-with-dbt)), the `DUCKDB` and `XLSX` source formats, time travel and
|
|
76
|
+
> folders of tables in `ldp query`, retries for concurrent appends, and a restaurant example with a
|
|
77
|
+
> [guide](docs/warehousing.md). Version 0.1.1 was the first release on PyPI since 0.1.0. It carries
|
|
78
|
+
> the hardening work (contract in [`docs/design/v0_1_1.md`](docs/design/v0_1_1.md)) and the platform
|
|
79
|
+
> work ("multi-writer, any catalog, object storage, agent-ready", contract in
|
|
80
|
+
> [`docs/design/v0_1_1_platform.md`](docs/design/v0_1_1_platform.md)), which works from Python and
|
|
81
|
+
> through the [platform `ldp` commands](#platform-commands). The version number follows the
|
|
82
|
+
> milestones in the [roadmap](docs/roadmap.md).
|
|
83
|
+
|
|
84
|
+
## Quickstart
|
|
85
|
+
|
|
86
|
+
`ldp` needs Python 3.12 or newer. Install it from PyPI, preferably in a virtual environment:
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
pip install "local-data-platform[duckdb]>=0.1.3"
|
|
90
|
+
|
|
91
|
+
ldp demo --workdir ldp_demo
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
To work on the library itself, install it from a clone of this repo. `make install` creates
|
|
95
|
+
`.venv` and installs the package in editable mode with the `dev` and `docs` extras:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
git clone https://github.com/tusharchou/local-data-platform.git
|
|
99
|
+
cd local-data-platform
|
|
100
|
+
make install
|
|
101
|
+
source .venv/bin/activate
|
|
102
|
+
|
|
103
|
+
ldp demo --workdir ldp_demo
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`ldp demo` generates 1,000 synthetic taxi rides (always the same ones), then shows each feature
|
|
107
|
+
using only the public API:
|
|
108
|
+
|
|
109
|
+
1. It loads a config from JSON.
|
|
110
|
+
2. It appends twice, so you can see the duplicates, then overwrites twice to show that the load is
|
|
111
|
+
idempotent.
|
|
112
|
+
3. It upserts on a key column.
|
|
113
|
+
4. It writes a day-partitioned table.
|
|
114
|
+
5. It runs quality checks that pass, then shows a bad batch being blocked before the write.
|
|
115
|
+
6. It queries the table with DuckDB SQL.
|
|
116
|
+
7. It reads an earlier snapshot (time travel).
|
|
117
|
+
8. It exports the table back to CSV.
|
|
118
|
+
|
|
119
|
+
Everything it writes goes into the `--workdir` folder, so you can delete it afterwards. The
|
|
120
|
+
[quickstart guide](docs/quickstart.md) explains each step and how to write your own config.
|
|
121
|
+
|
|
122
|
+
More demos, each one a `make` target (see [Development](#development)):
|
|
123
|
+
|
|
124
|
+
| Target | What it shows |
|
|
125
|
+
|---|---|
|
|
126
|
+
| `make demo-robotics` | Humanoid-robot episodes and sensor frames in bronze, silver and gold Iceberg tables, temporal quality checks, a pinned training split and DuckDB research queries |
|
|
127
|
+
| `make demo-agent` | The read-only MCP server on the demo table, driven by a scripted agent client |
|
|
128
|
+
| `make demo-warehouse` | A restaurant chain from Excel and CSV exports to dbt-tested Iceberg tables with monthly customer churn, a late batch and time travel (needs the `dbt` and `excel` extras) |
|
|
129
|
+
| `make demo-orchestration` | The restaurant's nightly drops run as a workflow: a backfill that stops at a bad row and resumes after the fix, a third run that skips every day, and a scheduler tick (runs its dbt models task when the `dbt` extra is installed) |
|
|
130
|
+
| `make demo-spark` | The Scala Spark job reading the demo table and writing an aggregate that pyiceberg reads back (needs scala-cli) |
|
|
131
|
+
| `make demo-rest` | The demo config against an Iceberg REST catalog started locally on scala-cli's JVM: `ldp catalog test`, two runs, the snapshots and a query (needs scala-cli) |
|
|
132
|
+
| `make demo-all` | All of them; `demo-warehouse` is skipped when dbt or the Excel reader isn't installed and the two JVM demos when scala-cli isn't |
|
|
133
|
+
|
|
134
|
+
### Extras
|
|
135
|
+
|
|
136
|
+
| Extra | Adds | You need it for |
|
|
137
|
+
|---|---|---|
|
|
138
|
+
| `duckdb` | DuckDB | SQL over Iceberg tables (`DuckDBEngine`, `ldp query`, the demo's SQL step) |
|
|
139
|
+
| `mcp` | The MCP SDK and DuckDB | The read-only MCP server for agents (`ldp mcp`) |
|
|
140
|
+
| `s3` | boto3 | SigV4-signed REST catalogs on AWS, such as S3 Tables. `s3://` IO itself needs nothing extra: pyarrow does it |
|
|
141
|
+
| `glue` | `pyiceberg[glue]` (boto3) | The AWS Glue catalog |
|
|
142
|
+
| `bigquery` | Google Cloud BigQuery client | BigQuery as a source |
|
|
143
|
+
| `spark` | PySpark | The experimental `SparkEngine`, which also needs a JDK 17+ |
|
|
144
|
+
| `excel` | openpyxl and defusedxml | Excel workbooks (`.xlsx`) as a source, for `ExcelToCSV` and `ExcelToIceberg` |
|
|
145
|
+
| `dbt` | dbt-core, the dbt-duckdb adapter and DuckDB | `ldp dbt`, which runs a dbt project over your Iceberg tables and publishes the models you mark back to Iceberg ([how it works](docs/design/warehousing.md)), and the tests marked `dbt` |
|
|
146
|
+
| `dev` | pytest, flake8, build, DuckDB, `moto[server]`, the MCP SDK, openpyxl | Working on the library |
|
|
147
|
+
| `docs` | MkDocs and plugins | Building the docs |
|
|
148
|
+
|
|
149
|
+
## Use it from Python
|
|
150
|
+
|
|
151
|
+
A config names a source, a target and optional quality checks. Paths are relative to the config
|
|
152
|
+
file's folder. Save this as `rides.json`, next to a `data/rides.csv` with `ride_id`, `pickup_ts`
|
|
153
|
+
and `fare` columns:
|
|
154
|
+
|
|
155
|
+
```json
|
|
156
|
+
{
|
|
157
|
+
"identifier": "rides",
|
|
158
|
+
"metadata": {
|
|
159
|
+
"source": {"name": "rides", "format": "CSV", "path": "data/rides.csv"},
|
|
160
|
+
"target": {
|
|
161
|
+
"name": "rides",
|
|
162
|
+
"format": "ICEBERG",
|
|
163
|
+
"catalog": {"identifier": "nyc", "warehouse_path": "warehouse"},
|
|
164
|
+
"write_mode": "upsert",
|
|
165
|
+
"join_cols": ["ride_id"],
|
|
166
|
+
"partition_by": [{"column": "pickup_ts", "transform": "day"}]
|
|
167
|
+
},
|
|
168
|
+
"quality": {
|
|
169
|
+
"on_failure": "fail",
|
|
170
|
+
"checks": [
|
|
171
|
+
{"check": "not_null", "columns": ["ride_id", "pickup_ts"]},
|
|
172
|
+
{"check": "unique", "columns": ["ride_id"]},
|
|
173
|
+
{"check": "range", "column": "fare", "min": 0}
|
|
174
|
+
]
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Run it, query it and time-travel:
|
|
181
|
+
|
|
182
|
+
```python
|
|
183
|
+
from local_data_platform import Config
|
|
184
|
+
from local_data_platform.engine.duckdb import DuckDBEngine
|
|
185
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
186
|
+
from local_data_platform.pipeline.registry import create_pipeline
|
|
187
|
+
|
|
188
|
+
config = Config.from_json("rides.json")
|
|
189
|
+
result = create_pipeline(config).run() # picks CSVToIceberg from the config's formats
|
|
190
|
+
print(result.rows_read, result.rows_written, result.quality.passed)
|
|
191
|
+
|
|
192
|
+
table = Iceberg("rides", config.target["catalog"], base_dir=config.base_dir)
|
|
193
|
+
|
|
194
|
+
with DuckDBEngine() as duck:
|
|
195
|
+
duck.register_iceberg(table, "rides")
|
|
196
|
+
print(duck.query("SELECT count(*) AS n, avg(fare) AS avg_fare FROM rides").to_pylist())
|
|
197
|
+
|
|
198
|
+
first = table.snapshots()[0]["snapshot_id"]
|
|
199
|
+
print(table.get(snapshot_id=first).num_rows) # the table as of its first snapshot
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
If a check fails and `on_failure` is `"fail"`, `run()` raises `DataQualityError` and writes
|
|
203
|
+
nothing; the failing checks are in `error.report`. With `"warn"` it logs them and writes anyway.
|
|
204
|
+
|
|
205
|
+
You can also build a pipeline in code, with your own transforms:
|
|
206
|
+
|
|
207
|
+
```python
|
|
208
|
+
import pyarrow.compute as pc
|
|
209
|
+
|
|
210
|
+
from local_data_platform.format.csv import CSV
|
|
211
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
212
|
+
from local_data_platform.pipeline import Pipeline
|
|
213
|
+
from local_data_platform.quality import NotNull, Unique
|
|
214
|
+
|
|
215
|
+
pipeline = Pipeline(
|
|
216
|
+
source=CSV("rides", "data/rides.csv"),
|
|
217
|
+
target=Iceberg("rides", {"identifier": "nyc", "warehouse_path": "warehouse"}, write_mode="overwrite"),
|
|
218
|
+
transforms=[lambda df: df.filter(pc.field("fare") >= 0)],
|
|
219
|
+
checks=[NotNull(["ride_id"]), Unique(["ride_id"])],
|
|
220
|
+
)
|
|
221
|
+
print(pipeline.run())
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
## Use it from the command line
|
|
225
|
+
|
|
226
|
+
```bash
|
|
227
|
+
ldp --version
|
|
228
|
+
ldp pipelines # list the registered source/target routes
|
|
229
|
+
ldp run rides.json # run the config's pipeline
|
|
230
|
+
ldp run rides.json --mode overwrite # override the config's write_mode
|
|
231
|
+
ldp run rides.json --window 2026-08-01/2026-08-02 # staged and exactly once: a second run for the window skips
|
|
232
|
+
ldp run orders.json --window yesterday # a window by name: also last-week and last-month, in the config's time zone
|
|
233
|
+
ldp run rides.json --window 2026-08-01/2026-08-02 --run-id RUN_ID --attempt 2 # a retry of that run
|
|
234
|
+
ldp run rides.json --idempotency-key sales-2026-08-01 # a key of your own, for an orchestrator
|
|
235
|
+
ldp run rides.json --window 2026-08-01/2026-08-02 --json # the result as JSON on stdout, for scripts
|
|
236
|
+
ldp snapshots rides.json # list the target table's snapshots
|
|
237
|
+
ldp query rides.json "SELECT count(*) FROM rides" # SQL over the target, named after target.name
|
|
238
|
+
ldp query rides.json "SELECT * FROM rides" --max-rows 20 # print at most 20 rows (default 100)
|
|
239
|
+
ldp query rides.json "SELECT count(*) FROM rides" --snapshot 4437 # an older snapshot, from ldp snapshots
|
|
240
|
+
ldp query rides.json "SELECT count(*) FROM rides" --as-of "2026-07-31 23:59:59" # the newest snapshot at or before that time (UTC)
|
|
241
|
+
ldp query config/ "SELECT o.id FROM raw.orders o JOIN raw.customers c ON c.id = o.id" # a folder of configs: every table, joinable
|
|
242
|
+
ldp demo --workdir /tmp/ldp_demo # also --rows N (default 1000) and --seed N (default 42)
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
With `--window` or `--idempotency-key` a run is staged and exactly once. It prints its window, its
|
|
246
|
+
idempotency key, its run id and attempt, its status (`published` or `skipped_duplicate`) and the snapshot that holds
|
|
247
|
+
its rows, and running it again for the same window finds that snapshot and writes nothing. `ldp plan CONFIG --window W`
|
|
248
|
+
prints the same key, and `--idempotency-key auto` asks for it. The key is looked for in the whole history that the table
|
|
249
|
+
still holds, before the source is read, so a window that ran weeks ago is skipped without being read too. A retry
|
|
250
|
+
reuses the run id with the next attempt number,
|
|
251
|
+
and the new attempt fences the older ones, so a staged branch that an earlier attempt left behind never reaches `main`.
|
|
252
|
+
`--attempt` needs `--run-id`, and both need a window or a key. Without these options `ldp run` writes directly, as it
|
|
253
|
+
always did. A window is `START/END`, or `yesterday`, `last-week` (the last full Monday-to-Monday week) or `last-month`,
|
|
254
|
+
counted in the time zone of the config, and `ldp run` prints the window it stands for. What a window reads, and how a
|
|
255
|
+
config says so, is in [Windows](#windows).
|
|
256
|
+
|
|
257
|
+
Every command follows one exit-code table.
|
|
258
|
+
|
|
259
|
+
| Exit | Meaning |
|
|
260
|
+
|---|---|
|
|
261
|
+
| 0 | success, including a run that published and a run that skipped a duplicate |
|
|
262
|
+
| 1 | an error: a bad config, a usage error, a failed write |
|
|
263
|
+
| 2 | reserved for an SLA breach that `ldp monitor` reports in 0.1.5 |
|
|
264
|
+
| 3 | a data gate held the run back, such as a quality check that blocked a write |
|
|
265
|
+
|
|
266
|
+
A scheduler retries a run that exits with 1, because the cause may pass, and never retries one that exits with 3,
|
|
267
|
+
because the data has to change first. Add `-v` to see a traceback. The commands
|
|
268
|
+
that read a table (`snapshots`, `query`, `commits`, `maintain`, `datasets pin`, `catalog test`) fail
|
|
269
|
+
without creating anything when the config's catalog is a SQLite file that doesn't exist yet (a
|
|
270
|
+
`local` catalog, or `sql` on SQLite). `ldp run`, `ldp snapshots` and `ldp query` also work with the
|
|
271
|
+
other [catalog types](#catalogs-and-object-storage); they have been run against `sql` (SQLite) and
|
|
272
|
+
`rest` catalogs.
|
|
273
|
+
|
|
274
|
+
### Platform commands
|
|
275
|
+
|
|
276
|
+
`ldp --help` lists them, and `ldp <command> --help` shows each one's options. Each is also a Python
|
|
277
|
+
call:
|
|
278
|
+
|
|
279
|
+
| Command | What it does | From Python |
|
|
280
|
+
|---|---|---|
|
|
281
|
+
| `ldp catalog test SPEC` | Connect to a catalog (a spec file, or a config's catalog block) and list its namespaces and tables | `catalog.provider.create_catalog(spec).list_namespaces()` |
|
|
282
|
+
| `ldp schema [--kind KIND]` | Print the `ldp/v1` JSON Schema of a kind of file, the dataset config by default | `spec.json_schema()`, `spec.kind_schema(kind)` |
|
|
283
|
+
| `ldp workflow plan FILE [--window W] [--json]` | Check a workflow file, or a dataset config with a schedule, and every config it names, and print its tasks in the order they run with each pipeline task's idempotency key for the window | `orchestration.plan_workflow(path, window=...)` |
|
|
284
|
+
| `ldp backfill FILE --from START [--to END] [--dry-run] [--replace] [--run-backwards] [--max-windows N] [--stop-on-failure] [--json]` | Run a workflow, or a config with a window, for every window of a range, oldest first, each pipeline under the key of its window. A window that is done is skipped without being read, so running it again resumes where it stopped; `--dry-run` lists the windows with their keys and states, and `--replace` writes them again under a replace key; exit status 0, 1, or 3 for a data gate | `orchestration.backfill(path, start, end, ...)` |
|
|
285
|
+
| `ldp workflow run FILE --window W [--json]` | Run the tasks of a workflow one at a time for a window. A failed or blocked task stops the tasks that depend on it and the rest still run, and a second run skips what published or was built; exit status 0, 1, or 3 for a data gate | `orchestration.run_workflow(path, window=...)` |
|
|
286
|
+
| `ldp schedule tick DIR [--now TIME] [--max-windows N] [--json]` | Run every window that is due and not done, for every workflow and config under DIR that has a schedule, then exit. cron or launchd calls it every 5 minutes. A window that failed is carried on by the next tick as the same run with the next attempt, and a tick that starts while another runs does nothing; exit status 0, 1, or 3 for a data gate | `orchestration.schedule.tick(dir, now=...)` |
|
|
287
|
+
| `ldp schedule list DIR [--now TIME] [--max-windows N] [--json]` | List every scheduled workflow and config under DIR with its cron expression, time zone, next fire in UTC and in local time, last window and how it ended, the windows that are due, and the windows it missed with the `ldp backfill` command that fills them | `orchestration.schedule.list_scheduled(dir)` |
|
|
288
|
+
| `ldp schedule install --print {cron,launchd} DIR [--ldp PATH]` | Print the crontab line or the launchd plist that ticks DIR every 5 minutes. It never writes to a system location | `orchestration.recipes.recipe(kind, dir)` |
|
|
289
|
+
| `ldp plan CONFIG [--window WINDOW] [--json]` | Validate a config (or a folder of files, each as its kind) and print its spec hash, route, target, sinks, schedule, and the source path and idempotency key of a window | `spec.validate_spec(data)`, `spec.plan(config, window=...)` |
|
|
290
|
+
| `ldp runs CONFIG [--run RUN_ID] [--events] [--attempts]` | List the runs a config's sinks recorded, the last attempt of a run that was retried, or every attempt with `--attempts` | `events.summarize_runs(events.read_jsonl(path))` or `read_iceberg_events(catalog)` |
|
|
291
|
+
| `ldp commits CONFIG [--key KEY] [--branches]` | List the staged publishes on a table, newest first | `format.iceberg.commit.commit_log(iceberg.table())` and `staged_branches(...)` |
|
|
292
|
+
| `ldp datasets pin\|list\|export` | Pin, list and export dataset versions | `datasets.pin`, `list_versions`, `export` |
|
|
293
|
+
| `ldp maintain CONFIG [--expire DAYS] [--orphans] [--apply]` | Expire snapshots and find orphan files; a dry run unless `--apply` | `maintenance.expire_snapshots`, `find_orphans`, `remove_orphans` |
|
|
294
|
+
| `ldp mcp --config CONFIG ...` | Serve tables to an agent over MCP (stdio) | `python -m local_data_platform.mcp_server ...`, same options |
|
|
295
|
+
| `ldp dbt init\|build\|test\|docs\|exec PROJECT_DIR` | Run a dbt project over your Iceberg tables; `build` writes the models marked `ldp_publish` back to Iceberg (needs the `dbt` extra) | `dbt_layer.register_sources(...)`, `dbt_layer.publish.plan_publish(...)` and `publish(...)` |
|
|
296
|
+
| `ldp spark CONFIG` | Run the Scala Spark job on a config's table (needs scala-cli) | `engine.spark.ScalaSparkJob` |
|
|
297
|
+
|
|
298
|
+
## Config schema
|
|
299
|
+
|
|
300
|
+
A config has an `identifier`, optional descriptive fields (`who`, `what`, `where`, `when`, `how`)
|
|
301
|
+
and a `metadata` block:
|
|
302
|
+
|
|
303
|
+
| Key | What it holds |
|
|
304
|
+
|---|---|
|
|
305
|
+
| `metadata.source` | `name`, `format` (`CSV`, `PARQUET`, `ICEBERG`, `JSON`, `DUCKDB` or `XLSX`), and a `path` (local, `s3://` or `gs://`, and it may have `{window_start:FORMAT}` placeholders, see [Windows](#windows)) or, for Iceberg, a `catalog`. A `DUCKDB` source also has one of `relation` (a table or view) and `sql` (one `SELECT` query), see [A DuckDB file as a source](#a-duckdb-file-as-a-source). An `XLSX` source may have `sheet` and `header_row`, see [An Excel sheet as a source](#an-excel-sheet-as-a-source) |
|
|
306
|
+
| `metadata.target` | The same, plus for Iceberg `write_mode` (`append`, `overwrite` or `upsert`), `join_cols` for upserts, and `partition_by` |
|
|
307
|
+
| `metadata.target.catalog` | `type` (`local` by default, `sql`, `rest` or `glue`) and that type's keys; see [Catalogs](#catalogs-and-object-storage) |
|
|
308
|
+
| `metadata.quality` | `on_failure` (`fail` or `warn`) and a list of `checks`: `row_count`, `not_null`, `unique`, `accepted_values`, `range`, `freshness`, `schema`, and the temporal checks `monotonic`, `max_skew`, `rate_below` and `max_gap` |
|
|
309
|
+
| `metadata.observability` | Where run events go, for example `{"sinks": ["iceberg", "jsonl"]}`. The default sends them nowhere |
|
|
310
|
+
| `metadata.schedule` | When the config runs: a 5-field cron expression, or `@hourly`, `@daily`, `@weekly` or `@monthly`. Without it, a `when` of `hourly`, `daily`, `weekly` or `monthly` means the alias. Editing it does not change the spec hash |
|
|
311
|
+
| `metadata.window` | `"none"`, or `size` (`day`, `hour` or `15m`), an optional `column` that selects the rows of a window, and an optional `timezone` (UTC by default); see [Windows](#windows) |
|
|
312
|
+
| `metadata.retries` | How a workflow tries the run again after a failure: `max_attempts` (1 by default, which is no retry), `delay_s` (30) and `backoff` (2.0); see [Workflow files](#workflow-files). Editing it does not change the spec hash |
|
|
313
|
+
|
|
314
|
+
A config may also say `"apiVersion": "ldp/v1"` and `"kind": "Pipeline"`. `kind` says what a JSON file is. `Pipeline`
|
|
315
|
+
is the default, so every config here is one, and a config has the same spec hash with and without it. `Workflow` is
|
|
316
|
+
the second kind. A kind is written exactly as shown, so `workflow` fails with a message that names `Workflow`, and
|
|
317
|
+
`Metrics`, `Report` and `Profiles` are reserved for later milestones. `ldp plan` validates each file of a folder as its
|
|
318
|
+
kind, and `ldp query`, `ldp mcp --config` and `ldp dbt` read the tables of the `Pipeline` files only, so a folder can
|
|
319
|
+
hold workflow files next to its configs. `ldp schema --kind Workflow` prints the schema of another kind.
|
|
320
|
+
|
|
321
|
+
Partition transforms are `identity`, `year`, `month`, `day`, `hour`, `bucket[N]` and
|
|
322
|
+
`truncate[W]`. On pyiceberg 0.10 and later, writing with any transform except `identity` needs the
|
|
323
|
+
`pyiceberg-core` package, which `ldp` depends on, so a normal install has it. If it's missing, the
|
|
324
|
+
demo's step 4 says so and partitions by `identity(pickup_date)` instead. The full schema, with an
|
|
325
|
+
example of every check, is in the [v0.1.1 hardening contract](docs/design/v0_1_1.md#config-schema).
|
|
326
|
+
`local_data_platform.spec.json_schema()` returns it as JSON Schema, and `ldp schema` prints it. The configs in
|
|
327
|
+
[`examples/`](examples/README.md) are real ones you can run.
|
|
328
|
+
|
|
329
|
+
### Windows
|
|
330
|
+
|
|
331
|
+
A run can be responsible for one window of time, `[start, end)`, which is what `--window` gives it. A config says how a
|
|
332
|
+
window picks what the run reads, and when the config runs:
|
|
333
|
+
|
|
334
|
+
```json
|
|
335
|
+
{
|
|
336
|
+
"identifier": "daily_orders",
|
|
337
|
+
"when": "daily",
|
|
338
|
+
"metadata": {
|
|
339
|
+
"schedule": "5 2 * * *",
|
|
340
|
+
"window": {"size": "day", "column": "ordered_at", "timezone": "Europe/Paris"},
|
|
341
|
+
"source": {"name": "orders", "format": "CSV", "path": "drops/orders_{window_start:%Y-%m-%d}.csv"},
|
|
342
|
+
"target": {"name": "orders", "format": "ICEBERG", "write_mode": "overwrite",
|
|
343
|
+
"catalog": {"identifier": "shop", "warehouse_path": "warehouse"}}
|
|
344
|
+
}
|
|
345
|
+
}
|
|
346
|
+
```
|
|
347
|
+
|
|
348
|
+
- `{window_start:FORMAT}` and `{window_end:FORMAT}` in `metadata.source.path` name the file of the window, with a Python
|
|
349
|
+
`strftime` FORMAT shown in the time zone of the window. A run without a window on such a path fails and names the
|
|
350
|
+
placeholder.
|
|
351
|
+
- `window.column` keeps only the rows at or after the start of the window and before its end, before the transforms and
|
|
352
|
+
the quality checks. `run.extracted` reports the rows read and the rows kept.
|
|
353
|
+
- With `window.column` and the write mode `overwrite`, a run replaces only the rows of its window, so one changed day can
|
|
354
|
+
be redone and the other days stay. A published window is skipped when it runs again, so to redo a day that changed at
|
|
355
|
+
the source, replace it with `ldp backfill daily_orders.json --from 2026-08-02 --to 2026-08-03 --replace`, or run it with a
|
|
356
|
+
key of your own: `ldp run daily_orders.json --window 2026-08-02/2026-08-03 --idempotency-key orders-2026-08-02-v2`.
|
|
357
|
+
- A `freshness` check in a windowed run measures the age of the data at the end of the window.
|
|
358
|
+
- A config with a schedule that appends, and that has neither a placeholder nor a `window.column`, would load the same
|
|
359
|
+
rows in every window. `ldp plan` reports that as an error and `ldp run --window` logs a warning.
|
|
360
|
+
|
|
361
|
+
The [contract](docs/design/v0_1_1.md#windows-and-schedules-013) has the full rules, including how each column type is
|
|
362
|
+
compared.
|
|
363
|
+
|
|
364
|
+
### Workflow files
|
|
365
|
+
|
|
366
|
+
A workflow file lists the pipelines, dbt models and maintenance that run for each window, and in what order:
|
|
367
|
+
|
|
368
|
+
```json
|
|
369
|
+
{
|
|
370
|
+
"apiVersion": "ldp/v1",
|
|
371
|
+
"kind": "Workflow",
|
|
372
|
+
"identifier": "daily_orders",
|
|
373
|
+
"metadata": {"schedule": "5 2 * * *", "window": {"size": "day"}, "retries": {"max_attempts": 3, "delay_s": 30}},
|
|
374
|
+
"tasks": [
|
|
375
|
+
{"id": "orders", "pipeline": "config/orders.json"},
|
|
376
|
+
{"id": "customers", "pipeline": "config/customers.json"},
|
|
377
|
+
{"id": "gold", "dbt": {"project": "dbt", "select": "+daily_revenue"}, "depends_on": ["orders", "customers"]},
|
|
378
|
+
{"id": "export", "pipeline": "config/export.json", "depends_on": ["gold"]},
|
|
379
|
+
{"id": "tidy", "maintain": {"config": "config/orders.json", "expire": 7}, "depends_on": ["export"]}
|
|
380
|
+
]
|
|
381
|
+
}
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
```bash
|
|
385
|
+
ldp workflow plan daily_orders.json --window 2026-08-01/2026-08-02 # the tasks in run order, with each key
|
|
386
|
+
ldp workflow run daily_orders.json --window 2026-08-01/2026-08-02 # run them for that day
|
|
387
|
+
ldp runs .ldp/events.jsonl # the workflow run and the run of each task
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
- Each task has an `id`, an optional `depends_on`, and exactly one kind: `pipeline` (the path of a dataset config,
|
|
391
|
+
relative to the workflow file), `dbt` (a dbt project, built for the window, see below) or `maintain` (a config's Iceberg
|
|
392
|
+
table and the options of `ldp maintain`, a dry run unless `apply` is true). Later milestones add kinds without changing the
|
|
393
|
+
file format.
|
|
394
|
+
- Tasks run one at a time, in dependency order. A pipeline task runs with the window of the workflow, so its idempotency key
|
|
395
|
+
is the one `ldp plan CONFIG --window` prints.
|
|
396
|
+
- A task that fails, or that a quality check blocks, makes the tasks that depend on it `upstream_failed`, and they do not run.
|
|
397
|
+
The tasks that do not depend on it still run. The exit status is 1 when a task failed and 3 when the only problem is a
|
|
398
|
+
data gate.
|
|
399
|
+
- Run the same window again after fixing the cause: the pipeline tasks that published are skipped by their keys, without
|
|
400
|
+
reading their sources, and only the tasks that failed or did not run do their work.
|
|
401
|
+
- A task that fails with an error that another attempt may fix runs again by itself, as `retries` says, in the workflow
|
|
402
|
+
or in the task, and a key of the task overrides the same key of the workflow. These are retried: a dropped connection,
|
|
403
|
+
a timeout, a file that has not landed yet (any `OSError`) and the commit errors of pyiceberg. A data gate, a wrong
|
|
404
|
+
config and a fenced attempt never are. A retry is the next attempt of the same run, with the same run id and the same
|
|
405
|
+
idempotency key, so the new attempt fences the old one and a load that already landed is found and not written
|
|
406
|
+
twice. The wait is a random time from 0 to `delay_s * backoff ** (n - 1)` seconds after attempt n. `ldp runs` lists the
|
|
407
|
+
last attempt of each run, and `ldp runs LEDGER --attempts` lists them all.
|
|
408
|
+
- A `dbt` task does what [`ldp dbt build`](#transform-with-dbt) does, for the window: dbt gets `window_start` and
|
|
409
|
+
`window_end` as variables (`'{{ var("window_start") }}'::timestamp` in a model), and the models marked `ldp_publish` are
|
|
410
|
+
written to Iceberg. Failing data tests are a data gate, so the task is `blocked_quality` and the exit status is 3. Run the
|
|
411
|
+
same window again and the task is skipped when its last build succeeded with the same project and no task before it may
|
|
412
|
+
have written data. List the pipelines that load the tables a dbt project reads in its `depends_on`, and name the dbt task
|
|
413
|
+
in the `depends_on` of every task that reads the tables it publishes.
|
|
414
|
+
- Every event of a run goes to `.ldp/events.jsonl` next to the workflow file, with the `workflow`, `workflow_run_id` and
|
|
415
|
+
`task_id` of the task in its payload. A dataset config that has a `metadata.schedule` runs as a workflow of one task.
|
|
416
|
+
|
|
417
|
+
The [contract](docs/design/v0_1_1.md#workflow-files-013) has the full file format. `ldp schema --kind Workflow` prints its JSON
|
|
418
|
+
Schema.
|
|
419
|
+
|
|
420
|
+
### Backfill a range of windows
|
|
421
|
+
|
|
422
|
+
`ldp backfill` runs a workflow, or a config that has a window, for every window of a range. It is how a new pipeline loads
|
|
423
|
+
its history, and how a load that stopped halfway finishes:
|
|
424
|
+
|
|
425
|
+
```bash
|
|
426
|
+
ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 --dry-run # every window, with its key and its state
|
|
427
|
+
ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 # load them, oldest first
|
|
428
|
+
ldp backfill daily_orders.json --from 2026-06-01 --to 2026-09-01 # again: it skips every window that is done
|
|
429
|
+
ldp backfill daily_orders.json --from 2026-08-02 --to 2026-08-03 --replace # redo a day that changed at the source
|
|
430
|
+
```
|
|
431
|
+
|
|
432
|
+
- The windows are the windows of `metadata.window.size` (`day`, `hour` or `15m`), or of the size the schedule means, a day for
|
|
433
|
+
`@daily` and an hour for `@hourly`. `--from` and `--to` are dates or datetimes in the time zone of the workflow. Each of them
|
|
434
|
+
has to start a window, and the window that starts at `--to` is not part of the range, so `--from 2026-06-01 --to
|
|
435
|
+
2026-06-04` is the three days of June 1st, 2nd and 3rd. Without `--to` the range ends with the last complete window. A
|
|
436
|
+
range of more than `--max-windows` windows, 1000 by default, is refused, to catch a typo.
|
|
437
|
+
- Each window runs as `ldp workflow run` runs it, so each pipeline publishes under the key that `ldp plan CONFIG --window W`
|
|
438
|
+
prints, and a task with `retries` is retried. A window whose keys are on the table is skipped without being read, planned
|
|
439
|
+
or written to the ledger, so a finished backfill that runs again does nothing, and one that stopped at window 40 starts
|
|
440
|
+
again at window 40. A workflow with a task that a key cannot vouch for, such as a dbt task or a pipeline that writes a file,
|
|
441
|
+
also needs the last run of the window in the ledger to have succeeded for every task.
|
|
442
|
+
- A window that fails, or that a quality check blocks, does not stop the others, and it is listed with what went wrong,
|
|
443
|
+
such as the check that failed. The summary counts the windows that published, were skipped, were blocked and failed.
|
|
444
|
+
The exit status is 0 when no window failed or was blocked, 3 when a data gate is the only problem, and 1 otherwise.
|
|
445
|
+
`--stop-on-failure` stops at the first window that failed instead, and a window that a quality check blocked does not
|
|
446
|
+
stop it.
|
|
447
|
+
- `--dry-run` prints every window with its key and its state, which is `published` (done, so it is skipped), `partial`,
|
|
448
|
+
`missing`, `failed` or `blocked` (the last run of it in the ledger failed or was held back by a data gate), and runs
|
|
449
|
+
nothing, not even the creation of a table. `--run-backwards` runs the newest window first, and `--json` prints the result.
|
|
450
|
+
- `--replace` writes each window again under a replace key, the sha256 of its key, the word `replace` and the spec hash of
|
|
451
|
+
its config, with a window overwrite. The rows that a window has are replaced by the rows the source has now, a window that
|
|
452
|
+
has none is loaded, a window that was replaced under the same spec is skipped, and one that was replaced under an older
|
|
453
|
+
spec is replaced again. It needs a `metadata.window.column` in every pipeline that writes to Iceberg, and without one it
|
|
454
|
+
exits with 1 and changes nothing. A window that is replaced has to have rows in the source. An empty one is not written,
|
|
455
|
+
because the run fails with "No data to write", or is blocked by a `row_count` check, and the table keeps its rows.
|
|
456
|
+
- From Python, `from local_data_platform.orchestration import backfill` runs the same, and `backfill(path, start, end,
|
|
457
|
+
dry_run=True)` returns the windows with their keys and states. The [contract](docs/design/v0_1_1.md#backfills-013) has the
|
|
458
|
+
rest.
|
|
459
|
+
|
|
460
|
+
### Run on a schedule
|
|
461
|
+
|
|
462
|
+
`ldp schedule tick` is a scheduler with no server and no daemon. cron or launchd calls it every 5 minutes, it runs every
|
|
463
|
+
window that is due and not done, and it exits, so a laptop that was asleep catches up at the first tick after it wakes:
|
|
464
|
+
|
|
465
|
+
```bash
|
|
466
|
+
ldp schedule list ldp_demo/restaurant # what is scheduled, when it fires, what it missed
|
|
467
|
+
ldp schedule tick ldp_demo/restaurant --now 2026-08-05T02:05:00Z # run what is due at that time, then exit
|
|
468
|
+
ldp schedule install --print launchd ldp_demo/restaurant > ~/Library/LaunchAgents/restaurant.plist # macOS
|
|
469
|
+
(crontab -l 2>/dev/null; ldp schedule install --print cron ldp_demo/restaurant) | crontab - # Linux
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
- A tick looks at every `.json` file under the folder that is a workflow file, or a dataset config, and has a schedule in
|
|
473
|
+
`metadata.schedule` or in a `when` such as `daily`. A config with a schedule is a workflow of one task. A file that has a
|
|
474
|
+
schedule and is not valid is printed with its problems and makes the tick exit with 1, and the other files still run.
|
|
475
|
+
- The schedule is a cron expression read in `metadata.window.timezone`, UTC without one. The window a fire owns is the
|
|
476
|
+
interval between the previous fire and this fire. When the workflow has a window size, in `metadata.window.size` or implied
|
|
477
|
+
by `@daily` and `@hourly`, it is the last complete window of that size at or before the fire, so the fire of `5 2 * * *` on
|
|
478
|
+
5 August owns 4 August. A fixed time that a clock change skips fires right after the gap, and one that happens twice fires
|
|
479
|
+
once.
|
|
480
|
+
- Without `catchup`, a tick runs the window of the latest fire if it is not done, and `ldp schedule list` prints the
|
|
481
|
+
`ldp backfill` command for any windows before it that were missed. With `"catchup": true` and a `"start"` in the metadata of
|
|
482
|
+
the workflow, a tick runs every window since `start` that is not done, oldest first, up to `--max-windows` (1000) a tick.
|
|
483
|
+
- A window that failed, was blocked by a data gate, or never finished is run again by the next tick, as the next attempt of
|
|
484
|
+
the same run. The workflow and each task keep their run ids and the attempts go up by one, so the first commit of the new
|
|
485
|
+
attempt fences every older one and the window is loaded once however many ticks it takes. `retries` are the retries within
|
|
486
|
+
one tick.
|
|
487
|
+
- A tick takes the lock `DIR/.ldp/locks/schedule.lock` without waiting for it. A tick that starts while another runs does
|
|
488
|
+
nothing and exits with 0, and a crash leaves no lock. The exit status is 0 when nothing failed or was blocked, 3 when a data
|
|
489
|
+
gate is the only problem, and 1 otherwise.
|
|
490
|
+
- The cron line and the plist call `ldp` by its absolute path and write a log, `DIR/.ldp/schedule.log` or a file in
|
|
491
|
+
`~/Library/Logs`. cron does not run while a Mac sleeps, and launchd runs an interval that passed during sleep once when the
|
|
492
|
+
Mac wakes, so launchd is the recipe for macOS and cron the recipe for Linux.
|
|
493
|
+
- From Python, `from local_data_platform.orchestration.schedule import tick, list_scheduled` does the same. The
|
|
494
|
+
[contract](docs/design/v0_1_1.md#schedules-013) has the rest.
|
|
495
|
+
|
|
496
|
+
### Run from Airflow
|
|
497
|
+
|
|
498
|
+
A team that already runs Apache Airflow 3 can run a config as a task. `LdpRunOperator` takes the data interval of the DAG run
|
|
499
|
+
as the window, `ti.try_number` as the attempt, and a run id made of the DAG id, the run id of the DAG run and the task id. So
|
|
500
|
+
a retry by Airflow is the next attempt of the same run, a backfill over a day that is in the table finds the key and writes
|
|
501
|
+
nothing, and a day is loaded once however Airflow got to it. The [guide](docs/airflow.md) installs Airflow, runs the daily
|
|
502
|
+
drops of the restaurant example from `examples/restaurant/airflow_dag.py`, and retries a late drop. Its commands, in short, are these.
|
|
503
|
+
|
|
504
|
+
```bash
|
|
505
|
+
export LDP_AIRFLOW_DEMO="$PWD/ldp_demo/airflow/data"
|
|
506
|
+
python examples/restaurant/airflow_dag.py --prepare "$LDP_AIRFLOW_DEMO" # the seven daily drops and a windowed orders config
|
|
507
|
+
airflow standalone # terminal 1, with the DAG file in its dags folder
|
|
508
|
+
airflow dags unpause restaurant_orders # terminal 2, catches up seven days one after the other
|
|
509
|
+
ldp commits "$LDP_AIRFLOW_DEMO/config/orders.json" # terminal 2, one publish for each day
|
|
510
|
+
```
|
|
511
|
+
|
|
512
|
+
- The window is `(data_interval_start, data_interval_end)`, so the key of an interval is the one
|
|
513
|
+
`ldp plan CONFIG --window START/END` prints for it.
|
|
514
|
+
- A data gate that holds a run back, and a config that cannot run, fail the task with `AirflowFailException`, so Airflow does
|
|
515
|
+
not retry them. A skipped duplicate succeeds, and the `PipelineResult` as a dict is the XCom of the task. Any other error is
|
|
516
|
+
left to the `retries` of the task.
|
|
517
|
+
- Give the DAG a `CronDataIntervalTimetable("@daily", timezone="UTC")` as its schedule and not the string `"@daily"`. Airflow 3
|
|
518
|
+
makes a trigger timetable for a cron string, whose runs have an empty data interval, and the operator fails them and says why.
|
|
519
|
+
- The mapping is plain Python and Airflow is imported when the operator is used, so `local_data_platform.integrations.airflow`
|
|
520
|
+
imports without it. There is no extra for it, because Airflow has to be installed with its own constraints file. Airflow 3.1.7
|
|
521
|
+
and later install next to LDP, and a team that cannot do that runs `ldp run CONFIG --window ... --run-id ... --attempt ...`
|
|
522
|
+
from a BashOperator, which the guide shows. The [contract](docs/design/v0_1_1.md#airflow-013) has the rest.
|
|
523
|
+
|
|
524
|
+
### A DuckDB file as a source
|
|
525
|
+
|
|
526
|
+
A `DUCKDB` source reads one table, or the result of one query, from a DuckDB database file and
|
|
527
|
+
publishes it to Iceberg like any other source, with the same quality checks, run events and
|
|
528
|
+
snapshots. It is how tables built with SQL (or with dbt, through `ldp dbt`) reach the lakehouse:
|
|
529
|
+
|
|
530
|
+
```json
|
|
531
|
+
{
|
|
532
|
+
"identifier": "churn_by_month",
|
|
533
|
+
"metadata": {
|
|
534
|
+
"source": {"name": "churn_by_month", "format": "DUCKDB", "path": "target/build.duckdb",
|
|
535
|
+
"relation": "gold.churn_by_month"},
|
|
536
|
+
"target": {"name": "churn_by_month", "format": "ICEBERG", "write_mode": "overwrite",
|
|
537
|
+
"catalog": {"identifier": "gold", "warehouse_path": "warehouse"}}
|
|
538
|
+
}
|
|
539
|
+
}
|
|
540
|
+
```
|
|
541
|
+
|
|
542
|
+
Use `relation` (`table`, `schema.table` or `catalog.schema.table`) or `sql`, not both. The file is
|
|
543
|
+
opened read-only and closed before the Iceberg write starts, and a `sql` source must be exactly one
|
|
544
|
+
`SELECT` or `WITH` query. It needs the `duckdb` extra. `ldp plan` reports a missing file or a
|
|
545
|
+
source with both or neither of `relation` and `sql`.
|
|
546
|
+
|
|
547
|
+
### An Excel sheet as a source
|
|
548
|
+
|
|
549
|
+
An `XLSX` source reads one sheet of an `.xlsx` workbook and writes it to a CSV file (`ExcelToCSV`) or an
|
|
550
|
+
Iceberg table (`ExcelToIceberg`), so a spreadsheet that someone downloads from a website stops being converted
|
|
551
|
+
by hand:
|
|
552
|
+
|
|
553
|
+
```json
|
|
554
|
+
{
|
|
555
|
+
"identifier": "sales",
|
|
556
|
+
"metadata": {
|
|
557
|
+
"source": {"name": "sales", "format": "XLSX", "path": "downloads/sales.xlsx", "sheet": "Sales",
|
|
558
|
+
"header_row": 1},
|
|
559
|
+
"target": {"name": "sales", "format": "CSV", "path": "drive/sales.csv"}
|
|
560
|
+
}
|
|
561
|
+
}
|
|
562
|
+
```
|
|
563
|
+
|
|
564
|
+
`sheet` is the sheet's name or its position from 0, and the first sheet by default. `header_row` is the row
|
|
565
|
+
that holds the column names, counted from 1, and the rows above it are skipped. Dates and datetimes become
|
|
566
|
+
timestamps, whole numbers `int64`, other numbers `float64`, TRUE and FALSE `bool`, and text `string`. A column
|
|
567
|
+
that mixes kinds becomes `string`, and a warning names it. A formula cell gives the value Excel saved with it,
|
|
568
|
+
and empty rows at the end of the sheet and empty columns at its right edge are dropped. Formatting, charts and
|
|
569
|
+
macros are ignored, and a merged range gives its value to its first cell and nulls to the rest. A `.xls` file, a
|
|
570
|
+
sheet that does not exist, and an empty or repeated header cell fail with a message that names the file and the
|
|
571
|
+
sheet. It needs the `excel` extra, and `ldp plan` reports a missing file or a `.xls` path. The CSV is replaced
|
|
572
|
+
atomically, so a folder that Google Drive syncs never holds a half-written file; see the recipe
|
|
573
|
+
[Excel to CSV in a Google Drive folder](docs/recipes.md#excel-to-csv-in-a-google-drive-folder).
|
|
574
|
+
|
|
575
|
+
## Catalogs and object storage
|
|
576
|
+
|
|
577
|
+
The catalog block's `type` picks the catalog. It defaults to `local`, so a catalog block without
|
|
578
|
+
`type` keeps working unchanged.
|
|
579
|
+
|
|
580
|
+
| `type` | Keys | What you get |
|
|
581
|
+
|---|---|---|
|
|
582
|
+
| `local` (alias `LocalIceberg`) | `identifier`, `warehouse_path` | A SQLite catalog file and a warehouse folder |
|
|
583
|
+
| `sql` (alias `sqlite`) | `uri` (a SQLAlchemy URI such as `sqlite:///…` or `postgresql+psycopg://…`), `warehouse`, `name`, `password_env` | pyiceberg's `SqlCatalog`. Its tables are the ones Iceberg's Java `JdbcCatalog` uses, so Spark can share it |
|
|
584
|
+
| `rest` | `uri`, `warehouse`, `name`, `token_env`, `credential_env`, and optional `properties` (such as `s3.endpoint`) | pyiceberg's `RestCatalog`, for any Iceberg REST catalog, such as Polaris, Nessie or Lakekeeper |
|
|
585
|
+
| `glue` | `name`, `warehouse`, and optional `properties` | pyiceberg's `GlueCatalog` (the `glue` extra). AWS credentials come from boto3's default chain |
|
|
586
|
+
|
|
587
|
+
What has been tested: `local` and `sql` on SQLite; `rest` against Apache Iceberg's REST catalog
|
|
588
|
+
test server (`tools/rest_fixture`, opt-in); `glue` against a local moto server. Postgres and
|
|
589
|
+
hosted REST catalogs (Polaris, S3 Tables, Unity and so on) are not tested yet.
|
|
590
|
+
|
|
591
|
+
The namespace tables live in comes from `identifier` for `local` catalogs, and from `namespace`,
|
|
592
|
+
falling back to `identifier`, for every other type.
|
|
593
|
+
`properties` are passed through to pyiceberg, and `properties_env` maps a property to the
|
|
594
|
+
environment variable holding its value. A REST target looks like this:
|
|
595
|
+
|
|
596
|
+
```json
|
|
597
|
+
"catalog": {
|
|
598
|
+
"type": "rest",
|
|
599
|
+
"name": "lake",
|
|
600
|
+
"uri": "https://catalog.example.com/api/catalog",
|
|
601
|
+
"warehouse": "analytics",
|
|
602
|
+
"namespace": "rides",
|
|
603
|
+
"token_env": "LAKE_TOKEN",
|
|
604
|
+
"properties": {"s3.endpoint": "http://127.0.0.1:9000"}
|
|
605
|
+
}
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
**Secrets never go in configs.** A config names the environment variable that holds a token,
|
|
609
|
+
credential or password (`token_env`, `credential_env`, `password_env`, `properties_env`). A key or
|
|
610
|
+
property whose name contains `token`, `secret`, `password` or `credential` and holds a literal value
|
|
611
|
+
is rejected, and so is a password inside a `uri`; log lines mask them. A `postgresql+psycopg://`
|
|
612
|
+
URI also needs a SQLAlchemy driver, such as `pip install "psycopg[binary]"`.
|
|
613
|
+
|
|
614
|
+
`create_catalog` builds the same catalog a config would. For a read-only check before running
|
|
615
|
+
anything, use `ldp catalog test SPEC`; calling `create_catalog` on an `sql` spec creates the SQLite
|
|
616
|
+
file if it does not exist yet:
|
|
617
|
+
|
|
618
|
+
```python
|
|
619
|
+
from local_data_platform.catalog.provider import catalog_namespace, create_catalog
|
|
620
|
+
|
|
621
|
+
spec = {"type": "sql", "name": "lake", "uri": "sqlite:////tmp/lake/catalog.db", "warehouse": "file:///tmp/lake",
|
|
622
|
+
"namespace": "rides"}
|
|
623
|
+
catalog = create_catalog(spec)
|
|
624
|
+
print(catalog_namespace(spec), catalog.list_namespaces())
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
`register_catalog_type(name)` is a decorator that adds your own type. `Iceberg(...,
|
|
628
|
+
catalog_obj=catalog)` writes through a catalog object you already have.
|
|
629
|
+
|
|
630
|
+
**Object storage.** CSV, Parquet and JSON read and write through `local_data_platform.fs`, so their
|
|
631
|
+
paths can be local, `file://`, `s3://` or `gs://`, all through `pyarrow.fs`. Settings come from the
|
|
632
|
+
environment, never from a config: S3 credentials from the AWS default chain, `AWS_ENDPOINT_URL_S3`
|
|
633
|
+
or `AWS_ENDPOINT_URL` for MinIO, LocalStack or moto, and `AWS_REGION`; `STORAGE_EMULATOR_HOST` for
|
|
634
|
+
a GCS emulator. A URI with credentials in it (`s3://key:secret@bucket/...`) is rejected. Writes are
|
|
635
|
+
atomic: locally a temp file is renamed into place, and on an object store the data is uploaded only
|
|
636
|
+
once it is complete, so readers see the old object or the new one. The S3 tests run against a local
|
|
637
|
+
moto server; `gs://` reads and writes have no test against a GCS emulator yet.
|
|
638
|
+
|
|
639
|
+
## Exactly-once writes
|
|
640
|
+
|
|
641
|
+
A plain `Iceberg.put(df, mode)` is **direct mode**. It keeps the semantics of the
|
|
642
|
+
[hardening contract](docs/design/v0_1_1.md) (appending twice gives duplicates, overwriting twice
|
|
643
|
+
doesn't) with three fixes:
|
|
644
|
+
|
|
645
|
+
- **One commit per write.** Adding new columns and writing the data happen in one transaction.
|
|
646
|
+
- **Counts from metadata.** `rows_before` and `rows_after` come from the snapshots' `total-records`,
|
|
647
|
+
not from a scan.
|
|
648
|
+
- **A lock on local catalogs.** `overwrite` and `upsert` take an exclusive file lock,
|
|
649
|
+
`<warehouse>/.ldp/locks/<namespace>.<table>.lock`, so two overwrites or upserts on one laptop
|
|
650
|
+
can't interleave. Appends take no lock. An append that loses a commit race reloads the table
|
|
651
|
+
and runs again with a backoff (`CommitPolicy` sets the limits), so several processes can append
|
|
652
|
+
to one table at once.
|
|
653
|
+
|
|
654
|
+
Direct `overwrite` and `upsert` on a shared remote catalog are single-writer. For several
|
|
655
|
+
writers, pass a `CommitContext` and `put` switches to the **staged publish protocol**: it writes
|
|
656
|
+
to a private branch, checks the staged snapshot, then fast-forwards `main` in one catalog
|
|
657
|
+
compare-and-swap that asserts both `main` and the branch. The idempotency key is stored on the
|
|
658
|
+
snapshot as `ldp.idempotency-key` (with `ldp.run-id` and `ldp.attempt`), so re-running a key finds
|
|
659
|
+
the published snapshot and writes nothing:
|
|
660
|
+
|
|
661
|
+
```python
|
|
662
|
+
import pyarrow as pa
|
|
663
|
+
|
|
664
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
665
|
+
from local_data_platform.format.iceberg.commit import CommitContext
|
|
666
|
+
|
|
667
|
+
table = Iceberg("fares", {"identifier": "nyc", "warehouse_path": "warehouse"}, write_mode="upsert",
|
|
668
|
+
join_cols=["ride_id"])
|
|
669
|
+
batch = pa.table({"ride_id": [1, 2, 3], "fare": [12.5, 8.0, 30.25]})
|
|
670
|
+
context = CommitContext.create("fares/2024-01-01") # a new run id; looks for the key 7 days back
|
|
671
|
+
|
|
672
|
+
first = table.put(batch, commit=context)
|
|
673
|
+
again = table.put(batch, commit=context) # the same key: finds the published snapshot
|
|
674
|
+
print(first.attempts, again.skipped_duplicate) # 1 True
|
|
675
|
+
```
|
|
676
|
+
|
|
677
|
+
If another writer moves `main` first, the attempt rebases and retries within its `CommitPolicy`; an
|
|
678
|
+
upsert is recomputed against the new rows, never blindly rebased. A superseded attempt is fenced by
|
|
679
|
+
removing its branch, so it can't publish afterwards. `WriteResult` gains `branch`,
|
|
680
|
+
`idempotency_key`, `attempts` and `skipped_duplicate`. What this guarantees is at most one effect
|
|
681
|
+
on `main` per key, which with retries means effectively once. It doesn't remove duplicates a
|
|
682
|
+
producer sent twice upstream. `ldp commits
|
|
683
|
+
CONFIG` lists the publishes on a table with their keys.
|
|
684
|
+
[Exactly-once writes](docs/exactly_once.md) has the protocol and its tests, including the
|
|
685
|
+
multi-process upsert race.
|
|
686
|
+
|
|
687
|
+
## Run events and the `_ldp` namespace
|
|
688
|
+
|
|
689
|
+
`Pipeline.run(mode=None, *, commit=None, sink=None)` emits a `RunEvent` at each step:
|
|
690
|
+
`run.started`, `run.extracted`, `quality.evaluated`, then one of `run.published`,
|
|
691
|
+
`run.skipped_duplicate`, `run.blocked_quality` or `run.failed`, and finally `run.finished`.
|
|
692
|
+
`PipelineResult` gains `run_id` (a uuid7), `idempotency_key` and `published_snapshot_id`.
|
|
693
|
+
|
|
694
|
+
| Sink | Writes |
|
|
695
|
+
|---|---|
|
|
696
|
+
| `NullSink` | Nothing (the default, so a run emits no events unless you choose a sink) |
|
|
697
|
+
| `JsonlSink(path)` | One JSON event per line |
|
|
698
|
+
| `OpenLineageSink(path_or_url)` | OpenLineage 1.x `RunEvent`s with schema, data-quality and output-statistics facets |
|
|
699
|
+
| `IcebergSink(catalog_spec, base_dir)` | Batched appends to `_ldp.runs` and `_ldp.quality_results`, both day-partitioned, in your own catalog (the MCP server also uses it for `_ldp.audit`) |
|
|
700
|
+
| `MultiSink` | Several of the above |
|
|
701
|
+
|
|
702
|
+
```python
|
|
703
|
+
from local_data_platform import Config
|
|
704
|
+
from local_data_platform.events import JsonlSink
|
|
705
|
+
from local_data_platform.pipeline.registry import create_pipeline
|
|
706
|
+
|
|
707
|
+
result = create_pipeline(Config.from_json("rides.json")).run(sink=JsonlSink("runs.jsonl"))
|
|
708
|
+
print(result.run_id, result.published_snapshot_id)
|
|
709
|
+
```
|
|
710
|
+
|
|
711
|
+
Or set the default sinks in the config with `metadata.observability`, for example
|
|
712
|
+
`{"sinks": ["iceberg", "jsonl"]}`; by default the `jsonl` sink writes `.ldp/events.jsonl` in the
|
|
713
|
+
config's folder, and `ldp run` uses these sinks too. Because `_ldp.runs` and
|
|
714
|
+
`_ldp.quality_results` are ordinary Iceberg tables in your catalog, you can query your run history
|
|
715
|
+
with any engine; `ldp runs CONFIG` lists it, and `events.read_iceberg_events` and `summarize_runs`
|
|
716
|
+
read it from Python. `local_data_platform.spec` gives each config a stable `spec_hash` and an
|
|
717
|
+
`idempotency_key(config, window)`, and validates configs against the `ldp/v1` JSON Schema
|
|
718
|
+
(`ldp plan CONFIG` shows all three). See
|
|
719
|
+
[Run events and the `_ldp` namespace](docs/observability.md).
|
|
720
|
+
|
|
721
|
+
## SQL with DuckDB
|
|
722
|
+
|
|
723
|
+
`DuckDBEngine.register_iceberg(table, alias, snapshot_id=None, row_filter=None, native=None)`
|
|
724
|
+
uses DuckDB's `iceberg` extension when it can: the table becomes a view over `iceberg_scan` at the
|
|
725
|
+
chosen snapshot, which streams and pushes filters down instead of loading the table into memory.
|
|
726
|
+
If the extension can't load, it falls back to the in-memory scan with a warning. It never
|
|
727
|
+
downloads the extension on its own, so it works offline; `DuckDBEngine(install_extensions=True)` allows
|
|
728
|
+
that. `native=True` or `native=False` forces one path.
|
|
729
|
+
|
|
730
|
+
```python
|
|
731
|
+
from local_data_platform.engine.duckdb import DuckDBEngine
|
|
732
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
733
|
+
|
|
734
|
+
table = Iceberg("rides", {"identifier": "nyc", "warehouse_path": "warehouse"})
|
|
735
|
+
with DuckDBEngine() as duck:
|
|
736
|
+
duck.register_iceberg(table, "rides") # native iceberg_scan when available
|
|
737
|
+
duck.register_iceberg(table, "rides_in_memory", native=False)
|
|
738
|
+
print(duck.query("SELECT count(*) FROM rides").to_pylist())
|
|
739
|
+
print(duck.snapshots("rides")) # from iceberg_snapshots
|
|
740
|
+
```
|
|
741
|
+
|
|
742
|
+
`open_lake(path)` registers every Iceberg table that some dataset configs name, so SQL can join them. `path` is a
|
|
743
|
+
config file, a folder of configs, or a list of them, and `ldp query` uses it when it is given a folder or more than
|
|
744
|
+
one config. Each table is a view named `namespace.table` and, when no other table has the same name, a view with the
|
|
745
|
+
bare name. A namespace of several parts, such as `a.b`, is one schema named `a.b`, so SQL quotes it: `"a.b".events`.
|
|
746
|
+
`snapshot_id=` or `as_of=` read one table as it was, and refuse a lake of several. Nothing is written and
|
|
747
|
+
nothing is downloaded, and the MCP SDK is not needed.
|
|
748
|
+
|
|
749
|
+
```python
|
|
750
|
+
from local_data_platform.lake import open_lake
|
|
751
|
+
|
|
752
|
+
with open_lake("config") as lake:
|
|
753
|
+
lake.query("SELECT o.id, c.label FROM raw.orders o JOIN raw.customers c ON c.id = o.id")
|
|
754
|
+
with open_lake("config/raw.orders.json", as_of="2026-07-31 23:59:59") as lake: # UTC unless it has an offset
|
|
755
|
+
lake.query("SELECT count(*) FROM orders")
|
|
756
|
+
```
|
|
757
|
+
|
|
758
|
+
`attach_rest(name, uri, warehouse, token_env=None)` attaches a whole REST catalog to DuckDB.
|
|
759
|
+
`engine.router` estimates how many bytes a scan reads from the manifests, after partition pruning,
|
|
760
|
+
and `choose_engine` picks DuckDB below 50 GiB by default, and PySpark above that when it's
|
|
761
|
+
installed. No pipeline calls the router yet.
|
|
762
|
+
|
|
763
|
+
## Transform with dbt
|
|
764
|
+
|
|
765
|
+
`ldp dbt` runs a [dbt](https://www.getdbt.com/) project over your Iceberg tables and writes the models you mark back
|
|
766
|
+
to Iceberg, so what dbt builds is versioned, can be read as it was, and is there for `ldp query`, `ldp runs` and
|
|
767
|
+
`ldp mcp` like any other table. It needs the `dbt` extra, and the `excel` extra when a source is an Excel sheet.
|
|
768
|
+
|
|
769
|
+
```bash
|
|
770
|
+
pip install "local-data-platform[dbt,excel]"
|
|
771
|
+
ldp dbt init my_project --sources config --warehouse warehouse # a project that builds as soon as its sources exist
|
|
772
|
+
ldp dbt build my_project # register the sources, run dbt, then publish the models marked ldp_publish
|
|
773
|
+
ldp dbt test my_project # register the sources and run the data tests, and publish nothing
|
|
774
|
+
ldp dbt docs my_project --serve # generate the dbt docs and serve them on http://127.0.0.1:8080
|
|
775
|
+
ldp dbt exec my_project compile # any other dbt command, with the sources registered
|
|
776
|
+
```
|
|
777
|
+
|
|
778
|
+
Before dbt runs, every Iceberg table that the configs in `--sources` describe becomes a source in a DuckDB file inside
|
|
779
|
+
the project, so a model reads it with `{{ source('raw', 'orders') }}`. After a build in which every model and every
|
|
780
|
+
data test passed, each model whose config has `meta={'ldp_publish': true}` is written to Iceberg through the
|
|
781
|
+
[`DUCKDB` source](#a-duckdb-file-as-a-source), with the same quality checks, run events and snapshots as any other load.
|
|
782
|
+
A table or view model overwrites its table, and an incremental model with a `unique_key` upserts on it. Nothing is
|
|
783
|
+
published when dbt fails. Everything after the project folder goes to dbt unchanged, and the options of `ldp dbt`
|
|
784
|
+
itself go before it.
|
|
785
|
+
|
|
786
|
+
- The [warehousing guide](docs/warehousing.md) walks through it on a restaurant chain's Excel and CSV exports, from
|
|
787
|
+
the raw tables to tested models, churn, an incremental load and time travel. `make demo-warehouse` runs the same
|
|
788
|
+
example offline in about 25 seconds.
|
|
789
|
+
- [How `ldp dbt` works](docs/design/warehousing.md) has the contract, including the meta keys and the schema rules.
|
|
790
|
+
|
|
791
|
+
## Orchestration
|
|
792
|
+
|
|
793
|
+
A workflow file declares the pipelines, dbt models and maintenance of a night once. LDP runs it for the right window, backfills a range of windows, retries what failed, and runs it on a schedule from cron or launchd, with no service. Every run goes by the idempotency key of its window, so a window is never loaded twice, whoever starts it.
|
|
794
|
+
|
|
795
|
+
```bash
|
|
796
|
+
make demo-orchestration # the restaurant's nightly drops, a bad row, a backfill, a fix and a scheduler tick
|
|
797
|
+
```
|
|
798
|
+
|
|
799
|
+
- The [orchestration guide](docs/orchestration.md) walks through it on the restaurant example, command by command, from a window and its key to a workflow file, a backfill that stops at a bad row, a replaced day, a retried task, launchd and cron, and Airflow, Dagster and Prefect.
|
|
800
|
+
- `make demo-orchestration` runs `examples/restaurant/run.py --layer orchestration`. It drops three nights of orders into a folder, with a bad row in the third. `ldp backfill` publishes two days and a data gate holds the third back, with exit status 3. After the fix, the same command runs only the third, and a third run skips every day. `ldp schedule tick` then loads the fourth night, and `ldp runs`, `ldp commits`, `ldp schedule list` and `ldp schedule install` show what happened. It needs no extra, runs the `models` task when the `dbt` extra is installed, and prints a skip line when it is not.
|
|
801
|
+
- The pieces are described above, under [Windows](#windows), [Workflow files](#workflow-files), [Backfill a range of windows](#backfill-a-range-of-windows), [Run on a schedule](#run-on-a-schedule) and [Run from Airflow](#run-from-airflow), and the exit statuses are in the [table](#use-it-from-the-command-line) of the command line section.
|
|
802
|
+
|
|
803
|
+
## MCP server for agents
|
|
804
|
+
|
|
805
|
+
`local_data_platform.mcp_server` serves your tables to an AI agent over the Model Context Protocol,
|
|
806
|
+
on stdio and read-only. Install the `mcp` extra, then point your MCP client at `ldp mcp`
|
|
807
|
+
(`python -m local_data_platform.mcp_server` takes the same options):
|
|
808
|
+
|
|
809
|
+
```json
|
|
810
|
+
{
|
|
811
|
+
"mcpServers": {
|
|
812
|
+
"ldp": {
|
|
813
|
+
"command": "/abs/path/to/.venv/bin/ldp",
|
|
814
|
+
"args": ["mcp", "--config", "/abs/path/to/rides.json", "--max-rows", "500"]
|
|
815
|
+
}
|
|
816
|
+
}
|
|
817
|
+
}
|
|
818
|
+
```
|
|
819
|
+
|
|
820
|
+
| Tool | Returns |
|
|
821
|
+
|---|---|
|
|
822
|
+
| `list_tables()` | Every table |
|
|
823
|
+
| `describe_table(table)` | Schema, partition spec, row count, snapshots, the latest `_ldp` run and quality status, and freshness |
|
|
824
|
+
| `query(sql, max_rows)` | The result of one read-only SQL statement |
|
|
825
|
+
| `sample_rows(table, n)` | A sample of rows |
|
|
826
|
+
| `table_history(table)` | The table's snapshot history |
|
|
827
|
+
| `get_dataset(name)` | A pinned dataset version |
|
|
828
|
+
|
|
829
|
+
The guardrails are on by default. A query must be exactly one `SELECT` or `WITH` statement, as
|
|
830
|
+
DuckDB's parser sees it, so `COPY`, `ATTACH`, `INSTALL`, `SET`, `read_csv('/etc/passwd')` and
|
|
831
|
+
multi-statement SQL are rejected. DuckDB runs with external access off and file access limited to
|
|
832
|
+
the served tables' folders, and its configuration is locked once the tables are registered.
|
|
833
|
+
Results are capped in rows and time, `--allow` limits the tables, and every call is audited: the
|
|
834
|
+
tool, the SQL, row counts, duration and any error, never the result data. Records go to a JSONL file
|
|
835
|
+
as each call ends, and to the `_ldp.audit` Iceberg table in the first served catalog when the
|
|
836
|
+
server stops (`--no-iceberg-audit` turns that off).
|
|
837
|
+
`make demo-agent` runs a scripted client against the demo table. See
|
|
838
|
+
[MCP server for agents](docs/agents.md).
|
|
839
|
+
|
|
840
|
+
## Reproducible datasets and robotics
|
|
841
|
+
|
|
842
|
+
A dataset version pins a table to one snapshot, with an optional row filter and column list, and
|
|
843
|
+
is saved as a JSON manifest under `<warehouse>/.ldp/datasets/`. Loading it later returns exactly
|
|
844
|
+
the same rows, however the table has changed since:
|
|
845
|
+
|
|
846
|
+
```python
|
|
847
|
+
from local_data_platform import datasets
|
|
848
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
849
|
+
|
|
850
|
+
episodes = Iceberg("gold_episode_stats", {"identifier": "robots", "warehouse_path": "warehouse"})
|
|
851
|
+
train = datasets.pin(episodes, "humanoid_train", row_filter="success = true",
|
|
852
|
+
selected_fields=["episode_id", "robot_id", "task", "frame_count"])
|
|
853
|
+
rows = datasets.load(train) # the same rows at any later time
|
|
854
|
+
datasets.export(train, "exports/humanoid_train_v1.parquet") # or format="jsonl"
|
|
855
|
+
```
|
|
856
|
+
|
|
857
|
+
Each pin also tags the snapshot (`ldp_ds_<name>_v<N>`), so snapshot expiry keeps it.
|
|
858
|
+
`datasets.list_versions(name, warehouse="warehouse")` and
|
|
859
|
+
`datasets.get_version("name@vN", warehouse="warehouse")` find versions again. From the command
|
|
860
|
+
line, `ldp datasets pin CONFIG NAME`, `ldp datasets list` and `ldp datasets export` do the same.
|
|
861
|
+
|
|
862
|
+
The temporal quality checks suit sensor and event data:
|
|
863
|
+
|
|
864
|
+
| `check` | Keys | Passes when |
|
|
865
|
+
|---|---|---|
|
|
866
|
+
| `monotonic` | `column`, `group_by`, `strict`, `order_by` | The column never decreases (or, with `strict`, always increases) within each group |
|
|
867
|
+
| `max_skew` | `column_a`, `column_b`, `max_ms` | Two timestamps on each row are at most `max_ms` apart |
|
|
868
|
+
| `rate_below` | `predicate_column`, `max_rate`, `group_by` | The share of rows where the boolean column is true stays below `max_rate` |
|
|
869
|
+
| `max_gap` | `column`, `max_ms`, `group_by` | Consecutive timestamps within each group are at most `max_ms` apart |
|
|
870
|
+
|
|
871
|
+
[`examples/robot_episodes/`](examples/robot_episodes/) puts it together: deterministic
|
|
872
|
+
humanoid-robot episodes and sensor frames, bronze, silver and gold tables (episodes partitioned by
|
|
873
|
+
`day(start_ts)` and `bucket(8, robot_id)`), quality configs for the frame-drop rate, RGB/depth sync
|
|
874
|
+
skew, monotonic frame timestamps and accepted tasks, a pinned training split, DuckDB research
|
|
875
|
+
queries and an optional Scala Spark per-robot aggregate. It runs offline in about a second with
|
|
876
|
+
`make demo-robotics`. See [Robot-episode datasets](docs/robotics.md).
|
|
877
|
+
|
|
878
|
+
## Table maintenance
|
|
879
|
+
|
|
880
|
+
```python
|
|
881
|
+
import datetime as dt
|
|
882
|
+
|
|
883
|
+
from local_data_platform import maintenance
|
|
884
|
+
from local_data_platform.format.iceberg import Iceberg
|
|
885
|
+
|
|
886
|
+
table = Iceberg("rides", {"identifier": "nyc", "warehouse_path": "warehouse"})
|
|
887
|
+
plan = maintenance.expire_snapshots(table, older_than=dt.timedelta(days=7), dry_run=True)
|
|
888
|
+
print(plan["expired_snapshot_ids"], plan["protected_snapshot_ids"]["idempotency_key"])
|
|
889
|
+
print(maintenance.find_orphans(table)) # files no metadata references; deletes nothing
|
|
890
|
+
report = maintenance.remove_orphans(table) # a dry run unless dry_run=False
|
|
891
|
+
```
|
|
892
|
+
|
|
893
|
+
Expiry always keeps the current snapshot, branch and tag heads and `main`'s last 20 snapshots
|
|
894
|
+
(`retain_last`), and it never expires a snapshot that carries an idempotency key newer than the
|
|
895
|
+
table's idempotency horizon (7 days by default), because re-runs rely on finding it. Orphans are
|
|
896
|
+
files under the table's location that no metadata references and that are older than 72 hours
|
|
897
|
+
(`older_than_hours`), so the files of a write still in progress are left alone. `ldp maintain
|
|
898
|
+
CONFIG` runs both from the command line, as a dry run unless `--apply` is given. There is no
|
|
899
|
+
compaction yet.
|
|
900
|
+
|
|
901
|
+
## Spark (experimental)
|
|
902
|
+
|
|
903
|
+
`engine.spark` reads and writes the same tables from Spark, in two ways: `ScalaSparkJob`, which runs
|
|
904
|
+
the Scala job in [`spark/`](spark/README.md) with scala-cli (it fetches its own JDK, Spark and
|
|
905
|
+
Iceberg), and `SparkEngine`, a PySpark engine (the `spark` extra plus a JDK 17+). Both open the same
|
|
906
|
+
SQLite catalog as pyiceberg, and pyiceberg reads back what Spark writes. `spark_catalog_conf_for`
|
|
907
|
+
turns a catalog block into Spark settings for the `local`, `sql` and `rest` types:
|
|
908
|
+
|
|
909
|
+
```python
|
|
910
|
+
from local_data_platform.engine.spark import spark_catalog_conf_for
|
|
911
|
+
|
|
912
|
+
conf = spark_catalog_conf_for({"identifier": "nyc", "warehouse_path": "warehouse"})
|
|
913
|
+
print(conf["spark.sql.catalog.nyc.catalog-impl"]) # org.apache.iceberg.jdbc.JdbcCatalog
|
|
914
|
+
```
|
|
915
|
+
|
|
916
|
+
`make demo-spark` runs the Scala job on the demo table, and `ldp spark CONFIG` on any config's
|
|
917
|
+
table. The integration tests are opt-in
|
|
918
|
+
(`make spark-test`, which sets `LDP_RUN_SPARK=1`). No pipeline uses Spark, and DuckDB is the
|
|
919
|
+
supported SQL engine. [Spark](docs/spark.md) covers versions, the shared catalog and the limits.
|
|
920
|
+
|
|
921
|
+
## Architecture
|
|
922
|
+
|
|
923
|
+
```text
|
|
924
|
+
rides.json ──► Config.from_json ──► registry.create_pipeline ──► Pipeline.run(mode, commit=, sink=)
|
|
925
|
+
│
|
|
926
|
+
┌──────────────────────────────────────────────────────────┘
|
|
927
|
+
▼
|
|
928
|
+
source.get() ──► transforms ──► quality checks ──► target.put(mode, commit=)
|
|
929
|
+
CSV · Parquet · your pyarrow fail: raise direct: one transaction, file lock
|
|
930
|
+
JSON (local, s3, callables DataQualityError, staged: branch ─► checks ─► publish (CAS)
|
|
931
|
+
gs via fs.py) · write nothing CSV · Parquet targets
|
|
932
|
+
Iceberg · BigQuery │
|
|
933
|
+
▼
|
|
934
|
+
catalog provider: local (SQLite + folder) · sql · rest · glue ── warehouse: file · s3 · gs
|
|
935
|
+
│
|
|
936
|
+
RunEvents ──► JSONL · OpenLineage · Iceberg (_ldp.runs, _ldp.quality_results)
|
|
937
|
+
│
|
|
938
|
+
read by: DuckDB (iceberg_scan) · Spark (experimental) · MCP server (read-only, audited to _ldp.audit) · datasets.pin
|
|
939
|
+
```
|
|
940
|
+
|
|
941
|
+
Data moves between steps as `pyarrow.Table`s. The package lives in `src/local_data_platform/`:
|
|
942
|
+
|
|
943
|
+
| Module | Purpose |
|
|
944
|
+
|---|---|
|
|
945
|
+
| `config`, `paths` | `Config` and path resolution relative to the config file |
|
|
946
|
+
| `spec` | The `ldp/v1` JSON Schema, config validation, `spec_hash` and `idempotency_key` |
|
|
947
|
+
| `fs` | Local and object-storage IO (`file://`, `s3://`, `gs://`) with atomic writes |
|
|
948
|
+
| `format` | `CSV`, `Parquet` and `Iceberg` tables (`get` / `put`), and `WriteResult` |
|
|
949
|
+
| `format.iceberg.commit` | The staged publish protocol: `CommitContext`, `write_once`, `stage`, `publish`, `fence` |
|
|
950
|
+
| `catalog` | `create_catalog` and the `local`, `sql`, `rest` and `glue` types; `LocalIcebergCatalog` |
|
|
951
|
+
| `store` | Sources: BigQuery (`[bigquery]` extra), JSON files, and older import paths kept for compatibility |
|
|
952
|
+
| `quality` | Checks, including the temporal ones in `quality.temporal`, `run_checks` and `QualityReport` |
|
|
953
|
+
| `events` | `RunEvent` and the event sinks |
|
|
954
|
+
| `engine.duckdb` | `DuckDBEngine` for SQL over Iceberg and Arrow tables (`[duckdb]` extra) |
|
|
955
|
+
| `engine.router` | Scan-size estimates and engine choice |
|
|
956
|
+
| `engine.spark` | Experimental: `SparkEngine`, `ScalaSparkJob` and `spark_catalog_conf_for`; see [Spark](docs/spark.md) |
|
|
957
|
+
| `mcp_server` | The read-only MCP server (`[mcp]` extra) |
|
|
958
|
+
| `datasets` | Snapshot-pinned dataset versions |
|
|
959
|
+
| `maintenance` | Snapshot expiry with the idempotency floor, and orphan-file detection |
|
|
960
|
+
| `pipeline` | `Pipeline`, the built-in pipelines and the registry (`create_pipeline`) |
|
|
961
|
+
| `cli`, `demo` | The `ldp` command and the demo |
|
|
962
|
+
| `github`, `issue` | GitHub REST helpers used by the project's docs tooling |
|
|
963
|
+
|
|
964
|
+
[Why a registry instead of if/else](docs/design/factory_registry.md) explains how pipelines are
|
|
965
|
+
chosen. The [SaaS architecture](docs/design/saas_architecture.md) is a proposal for where this
|
|
966
|
+
could go; none of its control plane exists.
|
|
967
|
+
|
|
968
|
+
## Development
|
|
969
|
+
|
|
970
|
+
```bash
|
|
971
|
+
make install # create .venv and install the package with the dev and docs extras
|
|
972
|
+
make lint # flake8
|
|
973
|
+
make test # pytest, offline: the Spark and REST integration tests skip
|
|
974
|
+
make demo # run the demo into ./ldp_demo
|
|
975
|
+
make demo-all # every demo; demo-warehouse needs the dbt and excel extras, demo-spark and demo-rest need scala-cli
|
|
976
|
+
make demo-orchestration # the restaurant's nightly workflow, a backfill and a scheduler tick
|
|
977
|
+
make spark-test # opt-in Spark integration tests (LDP_RUN_SPARK=1, scala-cli)
|
|
978
|
+
make rest-test # opt-in REST catalog integration tests (LDP_RUN_REST=1, scala-cli)
|
|
979
|
+
make dbt-test # the tests that need the dbt extra (pip install -e ".[dev,dbt]" first)
|
|
980
|
+
make docs # mkdocs build --strict
|
|
981
|
+
make build # build the sdist and wheel into dist/
|
|
982
|
+
make smoke # install the wheel in a fresh venv and run ldp --version and ldp demo
|
|
983
|
+
```
|
|
984
|
+
|
|
985
|
+
The JVM demos and tests use [scala-cli](https://scala-cli.virtuslab.org), which provisions its own
|
|
986
|
+
Temurin 17 JDK; the first run downloads it with Spark, Iceberg and their jars (about 500 MB).
|
|
987
|
+
`tests/test_rest_catalog.py` and `make demo-rest` start the Iceberg REST fixture in
|
|
988
|
+
`tools/rest_fixture` (`make demo-rest` on port 8181; `REST_PORT=` changes it) and stop it
|
|
989
|
+
afterwards, also when a step fails.
|
|
990
|
+
|
|
991
|
+
CI runs lint, tests on Python 3.12 and 3.13 against both pyiceberg 0.11.0 and the newest supported
|
|
992
|
+
release, the wheel smoke test and a strict docs build. The Spark and REST jobs are in a separate,
|
|
993
|
+
opt-in workflow (`.github/workflows/jvm.yml`): they run when started by hand, weekly, or on a pull
|
|
994
|
+
request labelled `jvm`, and never block a merge. See the [changelog](CHANGELOG.md) for what changed
|
|
995
|
+
in each release, and the [contributing guide](docs/contributing.md) to get involved.
|
|
996
|
+
|
|
997
|
+
## Known limitations
|
|
998
|
+
|
|
999
|
+
- **The engine router is unused.** Nothing calls `engine.router` yet.
|
|
1000
|
+
- **Remote catalogs are lightly tested.** `rest` is tested against Iceberg's REST test server and
|
|
1001
|
+
`glue` against moto. Postgres, hosted REST catalogs and GCS have no integration tests yet.
|
|
1002
|
+
- **Direct overwrite and upsert on a shared catalog are single-writer.** Several writers need the
|
|
1003
|
+
staged protocol (`commit=`). The local file lock only covers processes on one machine.
|
|
1004
|
+
- **Exactly once means at most one effect per key.** Duplicates a producer sent upstream are not
|
|
1005
|
+
removed.
|
|
1006
|
+
- **An upsert batch needs every column of the table.** An upsert replaces whole rows, so a batch
|
|
1007
|
+
that lacks one of the table's columns is rejected with a `ConfigError` that names them (appends
|
|
1008
|
+
accept it).
|
|
1009
|
+
- **Mostly in memory.** Reads return a whole `pyarrow.Table`. DuckDB's native `iceberg_scan`
|
|
1010
|
+
streams, but pipelines don't; there are no streaming or incremental reads.
|
|
1011
|
+
- **Spark is experimental.** It needs a JDK (or scala-cli) and downloads jars on first use, no
|
|
1012
|
+
pipeline uses it, and its tests are opt-in.
|
|
1013
|
+
- **The MCP server is read-only and local.** It speaks MCP over stdio only. Its `_ldp.audit` rows
|
|
1014
|
+
are appended when it stops, so a server that is killed leaves them only in the JSONL audit file.
|
|
1015
|
+
- **No compaction.** Maintenance expires snapshots and finds orphans; it doesn't rewrite data files.
|
|
1016
|
+
- **BigQuery is read-only.** It can be a source, not a target.
|
|
1017
|
+
- **`ldp dbt` publishes one model at a time.** Nothing is published when dbt fails, but the writes of a
|
|
1018
|
+
successful build are separate commits, so a write that fails, such as a blocked quality check or a model
|
|
1019
|
+
with no rows, leaves the models before it published. A model published with an upsert that gains or loses
|
|
1020
|
+
a column needs one `ldp run CONFIG --mode overwrite`.
|
|
1021
|
+
- **The restaurant example is in the repo, not in the package.** `pip install` does not include
|
|
1022
|
+
`examples/`, so the warehousing and orchestration guides and `make demo-warehouse` and `make demo-orchestration`
|
|
1023
|
+
need a clone.
|
|
1024
|
+
- **Tasks run one at a time.** A workflow runs its tasks in order and a backfill its windows in order, which also
|
|
1025
|
+
respects DuckDB's single writer. There are no parallel tasks.
|
|
1026
|
+
- **The scheduler is a tick, not a daemon.** `ldp schedule tick` has to be called by cron or launchd, and there is no
|
|
1027
|
+
foreground loop and no systemd or Windows Task Scheduler recipe. A window that failed is run again by every tick, every
|
|
1028
|
+
5 minutes under the recipes, with no wait between ticks. `retries` are the retries inside one tick.
|
|
1029
|
+
- **A dbt task builds its whole selection for every window.** dbt gets `window_start` and `window_end`, but only a
|
|
1030
|
+
model that reads them is limited to the window, and the models of the restaurant example are not.
|
|
1031
|
+
- **Airflow runs one config for each task.** DAGs built from workflow files, an `LdpTaskOperator` and managed Airflow
|
|
1032
|
+
recipes are planned for 0.2.0.
|
|
1033
|
+
|
|
1034
|
+
## Roadmap
|
|
1035
|
+
|
|
1036
|
+
From 0.1.1 on, the package version equals the milestone number. Each milestone adds the next layer
|
|
1037
|
+
of a data platform, and every layer runs on a laptop first. 0.1.3 Orchestration is released, and these
|
|
1038
|
+
are the milestones after it:
|
|
1039
|
+
|
|
1040
|
+
| Version | Milestone | What it adds |
|
|
1041
|
+
|---|---|---|
|
|
1042
|
+
| 0.1.4 | Self Serving Gold Layer | Gold tables with owners and docs, releases, metrics, a catalog and exports |
|
|
1043
|
+
| 0.1.5 | Monitoring | SLAs, freshness, volume, schema and metric monitors, alerts and `ldp status` |
|
|
1044
|
+
| 0.1.6 | Business Intelligence Reporting Dashboarding | Offline HTML reports, Slack delivery, a live dashboard and BI tool recipes |
|
|
1045
|
+
| 0.1.7 | Data Science Insights | Churn, retention, funnels, sessions, training sets and a notebook |
|
|
1046
|
+
| 0.1.8 | LLM | Catalog search and `ldp ask`, text to SQL that corrects itself |
|
|
1047
|
+
| 0.1.9 | Launch Documentation | The Learn path, generated reference pages, onboarding and contributor setup |
|
|
1048
|
+
| 0.2.0 | Cloud Integration | Profiles, cloud catalogs and storage, Snowflake and one container image |
|
|
1049
|
+
|
|
1050
|
+
The [roadmap](docs/roadmap.md) has the goal, issues and demo of each milestone.
|
|
1051
|
+
|
|
1052
|
+
## References
|
|
1053
|
+
|
|
1054
|
+
- [PyIceberg](https://py.iceberg.apache.org)
|
|
1055
|
+
- [DuckDB Iceberg extension](https://duckdb.org/docs/extensions/iceberg.html)
|
|
1056
|
+
- [Model Context Protocol](https://modelcontextprotocol.io)
|
|
1057
|
+
- [OpenLineage](https://openlineage.io)
|
|
1058
|
+
- [NEAR Lake Framework](https://docs.near.org/concepts/advanced/near-lake-framework)
|
|
1059
|
+
- [Reliable change data capture using Iceberg](https://medium.com/@tushar.choudhary.de/reliable-cdc-apache-spark-ingestion-pipeline-using-iceberg-5d8f0fee6fd6)
|
|
1060
|
+
- [Internals of Apache PyIceberg](https://medium.com/@tushar.choudhary.de/internals-of-apache-pyiceberg-10c2302a5c8b)
|
|
1061
|
+
|
|
1062
|
+
## License
|
|
1063
|
+
|
|
1064
|
+
MIT. See [LICENSE](LICENSE).
|
|
1065
|
+
|