fdl-cli 0.25.0__tar.gz → 0.27.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CHANGELOG.md +76 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/PKG-INFO +16 -16
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/README.md +15 -15
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/blume.config.ts +5 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/meta.ts +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/repositories.md +4 -4
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/versions-and-refs.md +8 -8
- fdl_cli-0.27.0/docs/concepts/working-metadata.md +49 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/getting-started.md +13 -13
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/building-a-host.md +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/mirrors.md +6 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/public-and-private.md +4 -4
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/reading-without-duckdb.md +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/serving-over-http.md +6 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/index.md +8 -8
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/ducklake-formats.md +15 -15
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/from-write-to-read.md +22 -22
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/inside-a-repository.md +16 -16
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/meta.ts +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/repositories.md +4 -4
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/versions-and-refs.md +8 -8
- fdl_cli-0.27.0/docs/ja/concepts/working-metadata.md +49 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/getting-started.md +13 -13
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/building-a-host.md +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/mirrors.md +6 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/public-and-private.md +4 -4
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/reading-without-duckdb.md +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/serving-over-http.md +6 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/index.md +8 -8
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/ducklake-formats.md +15 -15
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/from-write-to-read.md +22 -22
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/inside-a-repository.md +16 -16
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/cli.md +25 -25
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/python.md +9 -9
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/data-files.md +11 -11
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/http.md +5 -5
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/index.md +7 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/layout.md +9 -9
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/reading.md +14 -14
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/versions.md +49 -47
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/writing.md +19 -19
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/stability.md +3 -3
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/cli.md +25 -25
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/python.md +9 -9
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/data-files.md +11 -11
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/http.md +5 -5
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/index.md +7 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/layout.md +9 -9
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/reading.md +13 -13
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/versions.md +48 -46
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/writing.md +19 -19
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/stability.md +3 -3
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/pyproject.toml +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/cli.py +36 -35
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/layout.py +15 -6
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/profiles.py +28 -9
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/repo.py +91 -67
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/version.py +16 -13
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_cli.py +1 -1
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_http.py +4 -4
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_merge.py +2 -2
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_profiles.py +65 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_repo.py +24 -10
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/uv.lock +1 -1
- fdl_cli-0.25.0/docs/concepts/working-catalogs.md +0 -49
- fdl_cli-0.25.0/docs/ja/concepts/working-catalogs.md +0 -49
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/docs.yml +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/publish.yml +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/release.yml +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/test.yml +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.gitignore +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.node-version +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.python-version +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CLAUDE.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CONTRIBUTING.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/LICENSE +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/SECURITY.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/cases/collection.json +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/stores.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/stores.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/collection-cases.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/refs.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/collection-cases.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/meta.ts +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/refs.md +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/package-lock.json +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/package.json +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/public/CNAME +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/__init__.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/catalog.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/dataset.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/diff.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/lineage.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/merge.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/objects.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/py.typed +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/refs.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/store.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/stores.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/http_repo.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_collection_cases.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_dataset.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_objects.py +0 -0
- {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_stores.py +0 -0
|
@@ -1,6 +1,82 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
3
|
|
|
4
|
+
## v0.27.0 (2026-11-15)
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
- Name the frozen DuckLake metadata metadata.db and record it as metadata
|
|
9
|
+
([#61](https://github.com/queria-io/fdl/pull/61),
|
|
10
|
+
[`babd57f`](https://github.com/queria-io/fdl/commit/babd57fe57a45468019f00042e9d52f011afe15d))
|
|
11
|
+
|
|
12
|
+
* feat!: name the frozen DuckLake metadata metadata.db and record it as metadata
|
|
13
|
+
|
|
14
|
+
A version's metadata file is now versions/<ID>/metadata.db, recorded under "metadata" in
|
|
15
|
+
version.json; other DuckLake formats go under metadata/, and fdl.json declares them as
|
|
16
|
+
metadata_formats. Readers open the name the record gives, so versions written by 0.26 and earlier
|
|
17
|
+
(ducklake.duckdb, recorded as "catalog") are still read.
|
|
18
|
+
|
|
19
|
+
BREAKING CHANGE: version.json records the metadata file as "metadata", and fdl.json's
|
|
20
|
+
catalog_formats is now metadata_formats.
|
|
21
|
+
|
|
22
|
+
* docs: redirect the old working catalog page
|
|
23
|
+
|
|
24
|
+
* fix(expire): also drop the other formats that fdl-cli 0.26 and earlier kept under catalogs/
|
|
25
|
+
|
|
26
|
+
### Breaking Changes
|
|
27
|
+
|
|
28
|
+
- Version.json records the metadata file as "metadata", and fdl.json's catalog_formats is now
|
|
29
|
+
metadata_formats.
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
## v0.26.0 (2026-11-14)
|
|
33
|
+
|
|
34
|
+
### Features
|
|
35
|
+
|
|
36
|
+
- Profile views, and measure wide tables a hundred columns at a time
|
|
37
|
+
([#60](https://github.com/queria-io/fdl/pull/60),
|
|
38
|
+
[`e6c3442`](https://github.com/queria-io/fdl/commit/e6c34429590eedfcee7b568f6c030b497c5ef668))
|
|
39
|
+
|
|
40
|
+
* feat!: profile tables with fdl profile instead of at commit
|
|
41
|
+
|
|
42
|
+
fdl commit no longer writes profiles. fdl profile <working> profiles the version committed there,
|
|
43
|
+
for fdl push to upload with it; fdl profile <remote> adds the profiles a pushed version lacks. The
|
|
44
|
+
spec lets a version get a missing profile later.
|
|
45
|
+
|
|
46
|
+
BREAKING CHANGE: fdl commit writes no profiles. Run fdl profile between fdl commit and fdl push to
|
|
47
|
+
keep writing them.
|
|
48
|
+
|
|
49
|
+
* feat: profile tables at push instead of commit
|
|
50
|
+
|
|
51
|
+
fdl push profiles the version it uploads (--no-profile skips it), so the steps stay commit and push.
|
|
52
|
+
fdl profile <remote> is left for versions pushed without profiles. Also warns of tables that could
|
|
53
|
+
not be read, lists the version's profiles in one request, and updates the spec wording.
|
|
54
|
+
|
|
55
|
+
* fix: keep fdl push working when profiling cannot read the working catalog
|
|
56
|
+
|
|
57
|
+
Read dataset.json and the working catalog inside the guarded block, so a missing or unreadable
|
|
58
|
+
catalog only warns. Profile again when a stopped push left some tables without profiles.
|
|
59
|
+
|
|
60
|
+
* feat: profile views, and measure wide tables a hundred columns at a time
|
|
61
|
+
|
|
62
|
+
A view's profile is measured from the rows its query returns, with the columns DESCRIBE gives. It
|
|
63
|
+
keeps the parent's profile only while its sql is the parent's and no table's data files changed.
|
|
64
|
+
|
|
65
|
+
The column statistics now run in queries of at most 100 columns. One query over hundreds of
|
|
66
|
+
count(DISTINCT) ran out of memory, as DuckDB holds a hash table for each at once.
|
|
67
|
+
|
|
68
|
+
* fix: re-profile a view when any view's query changed
|
|
69
|
+
|
|
70
|
+
A view may read another view, so a view keeps its parent's profile only when every view's sql and
|
|
71
|
+
every table's data files are the parent's. Fetch the parent's dataset.json only when there are
|
|
72
|
+
views, list more of DuckDB's temporal types, and describe view profiling in push --help.
|
|
73
|
+
|
|
74
|
+
### Breaking Changes
|
|
75
|
+
|
|
76
|
+
- Fdl commit writes no profiles. Run fdl profile between fdl commit and fdl push to keep writing
|
|
77
|
+
them.
|
|
78
|
+
|
|
79
|
+
|
|
4
80
|
## v0.25.0 (2026-11-14)
|
|
5
81
|
|
|
6
82
|
### Bug Fixes
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: fdl-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.27.0
|
|
4
4
|
Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
|
|
5
5
|
Project-URL: Homepage, https://fdl.queria.io/
|
|
6
6
|
Project-URL: Documentation, https://fdl.queria.io/
|
|
@@ -24,9 +24,9 @@ Description-Content-Type: text/markdown
|
|
|
24
24
|
[](https://pypi.org/project/fdl-cli/)
|
|
25
25
|
[](LICENSE)
|
|
26
26
|
|
|
27
|
-
Versions of
|
|
27
|
+
Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
|
|
28
28
|
|
|
29
|
-
FDL is a storage layout and a tool for it. Each version is a frozen
|
|
29
|
+
FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
|
|
30
30
|
|
|
31
31
|
```
|
|
32
32
|
<root>/
|
|
@@ -35,7 +35,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
|
|
|
35
35
|
refs/heads/main 019a3c4e-…
|
|
36
36
|
versions/019a3c4e-…/
|
|
37
37
|
version.json parents, author, message, sha256
|
|
38
|
-
|
|
38
|
+
metadata.db the frozen metadata
|
|
39
39
|
data/main/towns/ducklake-….parquet
|
|
40
40
|
```
|
|
41
41
|
|
|
@@ -52,9 +52,9 @@ fdl checkout s3://my-bucket/towns/ work/lake.sqlite
|
|
|
52
52
|
|
|
53
53
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
54
54
|
|
|
55
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working
|
|
55
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `checkout` makes, a SQLite file.
|
|
56
56
|
|
|
57
|
-
Write to the working
|
|
57
|
+
Write to the working metadata with DuckDB, attached the way `checkout` prints:
|
|
58
58
|
|
|
59
59
|
```sql
|
|
60
60
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -64,9 +64,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
64
64
|
CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
|
|
65
65
|
```
|
|
66
66
|
|
|
67
|
-
Rows have to go to Parquet, not into the
|
|
67
|
+
Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
|
|
68
68
|
|
|
69
|
-
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the
|
|
69
|
+
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
|
|
70
70
|
|
|
71
71
|
```yaml
|
|
72
72
|
name: towns
|
|
@@ -88,7 +88,7 @@ fdl url s3://my-bucket/towns/
|
|
|
88
88
|
```
|
|
89
89
|
|
|
90
90
|
```sql
|
|
91
|
-
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/
|
|
91
|
+
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
92
92
|
DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
93
93
|
);
|
|
94
94
|
```
|
|
@@ -100,18 +100,18 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
100
100
|
| Command | What it does |
|
|
101
101
|
| --- | --- |
|
|
102
102
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
103
|
-
| `fdl checkout <remote> <working>` | Make a working
|
|
104
|
-
| `fdl commit <working> -m … --author …` | Freeze the working
|
|
103
|
+
| `fdl checkout <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
104
|
+
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
105
105
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
106
106
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
107
|
-
| `fdl status <working>` | What the working
|
|
107
|
+
| `fdl status <working>` | What the working metadata changed since the checkout |
|
|
108
108
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
109
109
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
110
|
-
| `fdl url <remote> [--attach]` | Print the
|
|
110
|
+
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
111
111
|
| `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
|
|
112
112
|
| `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
|
|
113
|
-
| `fdl merge <ref> <working> --author …` | Merge another branch into the working
|
|
114
|
-
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their
|
|
113
|
+
| `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
|
|
114
|
+
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
|
|
115
115
|
| `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
|
|
116
116
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
117
117
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
@@ -133,7 +133,7 @@ for version in repo.log(store):
|
|
|
133
133
|
|
|
134
134
|
## Earlier releases
|
|
135
135
|
|
|
136
|
-
Releases up to 0.11 were a different tool for managing a single frozen
|
|
136
|
+
Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
|
|
137
137
|
|
|
138
138
|
## License
|
|
139
139
|
|
|
@@ -4,9 +4,9 @@
|
|
|
4
4
|
[](https://pypi.org/project/fdl-cli/)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
|
-
Versions of
|
|
7
|
+
Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
|
|
8
8
|
|
|
9
|
-
FDL is a storage layout and a tool for it. Each version is a frozen
|
|
9
|
+
FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
|
|
10
10
|
|
|
11
11
|
```
|
|
12
12
|
<root>/
|
|
@@ -15,7 +15,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
|
|
|
15
15
|
refs/heads/main 019a3c4e-…
|
|
16
16
|
versions/019a3c4e-…/
|
|
17
17
|
version.json parents, author, message, sha256
|
|
18
|
-
|
|
18
|
+
metadata.db the frozen metadata
|
|
19
19
|
data/main/towns/ducklake-….parquet
|
|
20
20
|
```
|
|
21
21
|
|
|
@@ -32,9 +32,9 @@ fdl checkout s3://my-bucket/towns/ work/lake.sqlite
|
|
|
32
32
|
|
|
33
33
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
34
34
|
|
|
35
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working
|
|
35
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `checkout` makes, a SQLite file.
|
|
36
36
|
|
|
37
|
-
Write to the working
|
|
37
|
+
Write to the working metadata with DuckDB, attached the way `checkout` prints:
|
|
38
38
|
|
|
39
39
|
```sql
|
|
40
40
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -44,9 +44,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
44
44
|
CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
|
|
45
45
|
```
|
|
46
46
|
|
|
47
|
-
Rows have to go to Parquet, not into the
|
|
47
|
+
Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
|
|
48
48
|
|
|
49
|
-
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the
|
|
49
|
+
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
|
|
50
50
|
|
|
51
51
|
```yaml
|
|
52
52
|
name: towns
|
|
@@ -68,7 +68,7 @@ fdl url s3://my-bucket/towns/
|
|
|
68
68
|
```
|
|
69
69
|
|
|
70
70
|
```sql
|
|
71
|
-
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/
|
|
71
|
+
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
72
72
|
DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
73
73
|
);
|
|
74
74
|
```
|
|
@@ -80,18 +80,18 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
80
80
|
| Command | What it does |
|
|
81
81
|
| --- | --- |
|
|
82
82
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
83
|
-
| `fdl checkout <remote> <working>` | Make a working
|
|
84
|
-
| `fdl commit <working> -m … --author …` | Freeze the working
|
|
83
|
+
| `fdl checkout <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
84
|
+
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
85
85
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
86
86
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
87
|
-
| `fdl status <working>` | What the working
|
|
87
|
+
| `fdl status <working>` | What the working metadata changed since the checkout |
|
|
88
88
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
89
89
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
90
|
-
| `fdl url <remote> [--attach]` | Print the
|
|
90
|
+
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
91
91
|
| `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
|
|
92
92
|
| `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
|
|
93
|
-
| `fdl merge <ref> <working> --author …` | Merge another branch into the working
|
|
94
|
-
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their
|
|
93
|
+
| `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
|
|
94
|
+
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
|
|
95
95
|
| `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
|
|
96
96
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
97
97
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
@@ -113,7 +113,7 @@ for version in repo.log(store):
|
|
|
113
113
|
|
|
114
114
|
## Earlier releases
|
|
115
115
|
|
|
116
|
-
Releases up to 0.11 were a different tool for managing a single frozen
|
|
116
|
+
Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
|
|
117
117
|
|
|
118
118
|
## License
|
|
119
119
|
|
|
@@ -24,5 +24,9 @@ export default defineConfig({
|
|
|
24
24
|
fonts: ["Noto Sans JP"],
|
|
25
25
|
},
|
|
26
26
|
},
|
|
27
|
-
redirects: [
|
|
27
|
+
redirects: [
|
|
28
|
+
{ from: "/cli", to: "/reference/cli" },
|
|
29
|
+
{ from: "/concepts/working-catalogs", to: "/concepts/working-metadata" },
|
|
30
|
+
{ from: "/ja/concepts/working-catalogs", to: "/ja/concepts/working-metadata" },
|
|
31
|
+
],
|
|
28
32
|
});
|
|
@@ -9,11 +9,11 @@ A repository is everything under one prefix of a store: `s3://bucket/lake/` on S
|
|
|
9
9
|
|
|
10
10
|
| Operation | Used by | For |
|
|
11
11
|
| --- | --- | --- |
|
|
12
|
-
| read an object | readers and writers | refs, `version.json`,
|
|
13
|
-
| write an object | writers | data files,
|
|
12
|
+
| read an object | readers and writers | refs, `version.json`, metadata files, data files |
|
|
13
|
+
| write an object | writers | data files, metadata files, `version.json` |
|
|
14
14
|
| write on condition | writers | refs: create only if absent, replace only if unchanged |
|
|
15
15
|
| list a prefix | maintenance | finding data files that no version names |
|
|
16
|
-
| delete | maintenance |
|
|
16
|
+
| delete | maintenance | metadata files of versions past retention, and unused data files |
|
|
17
17
|
|
|
18
18
|
Conditional writes are what make two writers safe. A ref is created with `If-None-Match: *` and moved with `If-Match: <the ETag that was read>`. If someone else moved it in between, the store refuses the write and nothing changes. S3 and R2 support this. The folder store checks the condition without a lock, so it suits one writer at a time (see [Stores](/guides/stores)). A store without conditional writes can still be read from, but must not be pushed to: the last writer would silently replace someone else's version.
|
|
19
19
|
|
|
@@ -32,7 +32,7 @@ Readers only read. A static HTTP server in front of the repository is enough for
|
|
|
32
32
|
assets/<sha256> files a version points at, such as PDFs, images and its README
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
The repository holds no
|
|
35
|
+
The repository holds no metadata outside a version folder, and never a working metadata file. Either would hand readers a state that was never published.
|
|
36
36
|
|
|
37
37
|
`fdl.json` carries an `id`, a UUID written once by `fdl init`. Copies of the repository on other hosts keep the same `id`, so a tool can tell that two URLs hold the same repository. A service that hosts repositories may write its own ID for the dataset there.
|
|
38
38
|
|
|
@@ -1,17 +1,17 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Versions and refs
|
|
3
|
-
description: A version is a frozen
|
|
3
|
+
description: A version is a frozen metadata file with a record; a ref is a name that points at one. How they relate to DuckLake snapshots and to Git commits.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
## Versions
|
|
7
7
|
|
|
8
|
-
A version is one frozen DuckLake
|
|
8
|
+
A version is one frozen DuckLake metadata and a record of how it came to be. It lives in its own folder and is never changed once written.
|
|
9
9
|
|
|
10
10
|
```
|
|
11
11
|
versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/
|
|
12
12
|
version.json the record: parents, author, message, checksums
|
|
13
13
|
dataset.json what it holds: tables, columns, license
|
|
14
|
-
|
|
14
|
+
metadata.db the frozen metadata
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
The record says:
|
|
@@ -20,10 +20,10 @@ The record says:
|
|
|
20
20
|
- `seq`: the number shown to people, the parent's `seq` plus one
|
|
21
21
|
- `parents`: the version it was built on. Empty for the first
|
|
22
22
|
- `author`, `message` and `created_at`
|
|
23
|
-
- `
|
|
24
|
-
- `ducklake`: the DuckLake format of the
|
|
23
|
+
- `metadata`: the size and sha256 of the metadata file, so a reader can check what it fetched
|
|
24
|
+
- `ducklake`: the DuckLake format of the metadata and the snapshot it was frozen at
|
|
25
25
|
|
|
26
|
-
The ID is not a hash of the content, as a Git commit's is. The same
|
|
26
|
+
The ID is not a hash of the content, as a Git commit's is. The same metadata can be written as different bytes by two DuckDB releases, so a content hash would not be stable. The sha256 in `metadata` checks the content instead.
|
|
27
27
|
|
|
28
28
|
The full list of fields is in [Versions](/spec/versions) in the specification.
|
|
29
29
|
|
|
@@ -31,7 +31,7 @@ The full list of fields is in [Versions](/spec/versions) in the specification.
|
|
|
31
31
|
|
|
32
32
|
DuckLake makes a snapshot on every write: each `INSERT`, `CREATE TABLE` or `ALTER` in its own transaction. A day of work can make hundreds. A version is made when you decide the work is ready, and it takes the last of those snapshots.
|
|
33
33
|
|
|
34
|
-
A snapshot is to a version what a commit on a work branch is to the squashed commit that lands on `main`. The snapshots stay inside the
|
|
34
|
+
A snapshot is to a version what a commit on a work branch is to the squashed commit that lands on `main`. The snapshots stay inside the metadata, with their `commit_message` and `commit_extra_info`, so the detail is not lost. `version.json` records the range of snapshots the version adds after its parent, but does not copy them.
|
|
35
35
|
|
|
36
36
|
## Refs
|
|
37
37
|
|
|
@@ -94,4 +94,4 @@ Moving a branch back, to undo a version, is allowed only when asked for explicit
|
|
|
94
94
|
First towns
|
|
95
95
|
```
|
|
96
96
|
|
|
97
|
-
Versions that were taken out of retention stay in the history. Their record is kept, only their
|
|
97
|
+
Versions that were taken out of retention stay in the history. Their record is kept, only their metadata is gone. See [Retention](/internals/inside-a-repository#taking-versions-out-of-retention).
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Working metadata files
|
|
3
|
+
description: "Where writing happens: a DuckLake metadata file in SQLite, checked out from a version and frozen into the next one."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
A working metadata file is a DuckLake metadata file in a SQLite file, on the writer's own disk or in a container. It is the only place anyone writes to. It is to FDL what a working tree is to Git: made from a version, changed freely, and turned into the next version when the change is ready.
|
|
7
|
+
|
|
8
|
+
## The cycle
|
|
9
|
+
|
|
10
|
+
| Step | Command | What happens |
|
|
11
|
+
| --- | --- | --- |
|
|
12
|
+
| check out | `fdl checkout` | Copy the metadata of the branch's version into SQLite, and remember that version |
|
|
13
|
+
| write | any DuckDB client | DuckLake writes Parquet into the repository and bookkeeping into SQLite |
|
|
14
|
+
| commit | `fdl commit` | Freeze the last snapshot into a new version, kept beside the working metadata |
|
|
15
|
+
| push | `fdl push` | Upload the version and move the branch to it |
|
|
16
|
+
|
|
17
|
+
Checking out copies DuckLake's bookkeeping tables from the frozen DuckDB file into SQLite as they are. Freezing does the reverse. Neither rewrites a Parquet file, so both are fast even for a large dataset: the metadata is small and the data stays where it is.
|
|
18
|
+
|
|
19
|
+
## Why SQLite
|
|
20
|
+
|
|
21
|
+
A working metadata file has to be written by DuckDB and kept between sessions. SQLite does both without a database server, works the same on a laptop and in a container, and copies in and out of DuckDB's format without loss.
|
|
22
|
+
|
|
23
|
+
There is one kind of working metadata. A host that writes on behalf of its users (a scheduled import, a web form) checks out, writes, commits and pushes the same way a person does on a laptop. When two of them push at once, the conditional write on the ref settles it: one wins, and the other checks out again.
|
|
24
|
+
|
|
25
|
+
## Attaching
|
|
26
|
+
|
|
27
|
+
`fdl checkout` prints the statement to attach the working metadata with:
|
|
28
|
+
|
|
29
|
+
```sql
|
|
30
|
+
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
31
|
+
DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true,
|
|
32
|
+
META_TYPE 'sqlite', DATA_INLINING_ROW_LIMIT 0,
|
|
33
|
+
META_JOURNAL_MODE 'WAL', BUSY_TIMEOUT 5000
|
|
34
|
+
);
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
- `DATA_PATH` is the repository's data folder, so new Parquet files land where readers will look for them. For a repository over HTTP it is a folder beside the working metadata instead, and `fdl push` uploads the new files (see [Pushing over HTTP](/spec/http#writing-to-it))
|
|
38
|
+
- `DATA_INLINING_ROW_LIMIT 0` keeps every row out of the metadata. A version must not carry inlined rows, and `fdl commit` refuses a working metadata file that holds some
|
|
39
|
+
- `META_TYPE 'sqlite'` has to be stated. Left to guess from the file, DuckDB passes an option to SQLite that SQLite rejects
|
|
40
|
+
|
|
41
|
+
## Data files written before a push
|
|
42
|
+
|
|
43
|
+
DuckLake writes Parquet as you go, so data files reach the repository before any version names them. Readers never see them: a reader only opens metadata files of versions, and no version names those files yet.
|
|
44
|
+
|
|
45
|
+
A file that stays unnamed for longer than the grace period (7 days by default) may be deleted by maintenance. Push within that time. If a push is refused because a file it names is missing, rebuild the table in the working metadata, commit and push again. A version committed and never pushed names its files only on your disk, so they are deleted in the same way.
|
|
46
|
+
|
|
47
|
+
## Snapshots in the working metadata
|
|
48
|
+
|
|
49
|
+
Do not run `ducklake_expire_snapshots` or `ducklake_cleanup_old_files` on a working metadata file. They only see the working metadata, not the repository, and would delete files that other versions still name. Removing old versions and files is done on the repository as a whole (see [Retention](/internals/inside-a-repository#taking-versions-out-of-retention)).
|
|
@@ -29,17 +29,17 @@ towns/
|
|
|
29
29
|
|
|
30
30
|
`HEAD` names the default branch, `main`. The branch itself does not exist yet: there is no version for it to point at.
|
|
31
31
|
|
|
32
|
-
`fdl init` makes the repository where readers will find it, as `git init --bare` does on a server. Your own place to write is the working
|
|
32
|
+
`fdl init` makes the repository where readers will find it, as `git init --bare` does on a server. Your own place to write is the working metadata, made next.
|
|
33
33
|
|
|
34
|
-
## Check out a working
|
|
34
|
+
## Check out a working metadata file
|
|
35
35
|
|
|
36
|
-
You never write to the repository directly. You write to a working
|
|
36
|
+
You never write to the repository directly. You write to a working metadata file, a DuckLake metadata file in a SQLite file on your own disk, and turn it into a version when you are done.
|
|
37
37
|
|
|
38
38
|
```bash
|
|
39
39
|
fdl checkout ./towns/ work/lake.sqlite
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
Before the first version, `checkout` makes an empty
|
|
42
|
+
Before the first version, `checkout` makes an empty metadata. Later it copies the metadata of the version `main` holds. It prints the statement to attach the working metadata with:
|
|
43
43
|
|
|
44
44
|
```sql
|
|
45
45
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -49,7 +49,7 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
49
49
|
);
|
|
50
50
|
```
|
|
51
51
|
|
|
52
|
-
Beside the working
|
|
52
|
+
Beside the working metadata, `work/lake.sqlite.fdl.json` remembers the repository, the branch and the version the checkout was made from.
|
|
53
53
|
|
|
54
54
|
## Write
|
|
55
55
|
|
|
@@ -60,7 +60,7 @@ CREATE TABLE lake.main.towns AS
|
|
|
60
60
|
SELECT * FROM 'towns.csv';
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
-
`DATA_INLINING_ROW_LIMIT 0` makes DuckLake write every row to Parquet. Left at its default, DuckLake keeps small tables inside the
|
|
63
|
+
`DATA_INLINING_ROW_LIMIT 0` makes DuckLake write every row to Parquet. Left at its default, DuckLake keeps small tables inside the metadata, and a version has to carry its rows in data files that a reader can fetch.
|
|
64
64
|
|
|
65
65
|
## Describe
|
|
66
66
|
|
|
@@ -74,7 +74,7 @@ Every version carries a `dataset.json` that says what it holds. Write one next t
|
|
|
74
74
|
}
|
|
75
75
|
```
|
|
76
76
|
|
|
77
|
-
Only `name` and `title` are required. `fdl commit` checks the file against the
|
|
77
|
+
Only `name` and `title` are required. `fdl commit` checks the file against the metadata and fills in each table's row count and each column's type.
|
|
78
78
|
|
|
79
79
|
## Commit and push
|
|
80
80
|
|
|
@@ -83,7 +83,7 @@ fdl commit work/lake.sqlite -m "First towns" --author alice
|
|
|
83
83
|
fdl push work/lake.sqlite
|
|
84
84
|
```
|
|
85
85
|
|
|
86
|
-
`commit` freezes the working
|
|
86
|
+
`commit` freezes the working metadata into a version on your disk. `push` measures each table into a small summary (a profile) that readers can show without reading the data, uploads the version with them and moves `main` to it, and prints `refs/heads/main -> 019a3c4e-… (version 1)`. The repository now holds:
|
|
87
87
|
|
|
88
88
|
```
|
|
89
89
|
towns/
|
|
@@ -91,7 +91,7 @@ towns/
|
|
|
91
91
|
HEAD
|
|
92
92
|
refs/heads/main 019a3c4e-…
|
|
93
93
|
versions/019a3c4e-…/
|
|
94
|
-
|
|
94
|
+
metadata.db
|
|
95
95
|
dataset.json
|
|
96
96
|
files.json
|
|
97
97
|
profiles/main/towns.json
|
|
@@ -114,16 +114,16 @@ fdl log ./towns/
|
|
|
114
114
|
fdl url ./towns/
|
|
115
115
|
```
|
|
116
116
|
|
|
117
|
-
prints the
|
|
117
|
+
prints the metadata of the version `main` holds. Attach it read-only, and tell DuckDB where the data files are:
|
|
118
118
|
|
|
119
119
|
```sql
|
|
120
|
-
ATTACH 'ducklake:towns/versions/019a3c4e-…/
|
|
120
|
+
ATTACH 'ducklake:towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
121
121
|
DATA_PATH 'towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
122
122
|
);
|
|
123
123
|
SELECT count(*) FROM towns.main.towns;
|
|
124
124
|
```
|
|
125
125
|
|
|
126
|
-
A version's
|
|
126
|
+
A version's metadata never changes, so a reader may cache it forever. Only `HEAD` and the files under `refs/` change.
|
|
127
127
|
|
|
128
128
|
## Make a second version
|
|
129
129
|
|
|
@@ -140,6 +140,6 @@ fdl commit work/lake.sqlite -m "Add 2026" --author alice
|
|
|
140
140
|
fdl push work/lake.sqlite
|
|
141
141
|
```
|
|
142
142
|
|
|
143
|
-
The new version lists the first as its parent. Both
|
|
143
|
+
The new version lists the first as its parent. Both metadata files name the first Parquet file; only the second names the new one.
|
|
144
144
|
|
|
145
145
|
If someone else pushed in between, the push is refused, because the new version would not be built on what `main` holds now. Check out again and redo the change. [Versions and refs](/concepts/versions-and-refs) explains why.
|
|
@@ -50,4 +50,4 @@ Access rules, approvals and billing stay in the host's own database. A repositor
|
|
|
50
50
|
|
|
51
51
|
## Retention
|
|
52
52
|
|
|
53
|
-
`repo.prune` takes versions out of retention and `repo.expire` takes out one. Deleting data files that no live version names is left to the host, which can list the whole repository and read every live
|
|
53
|
+
`repo.prune` takes versions out of retention and `repo.expire` takes out one. Deleting data files that no live version names is left to the host, which can list the whole repository and read every live metadata. See [Data files](/spec/data-files) for when a file may be deleted.
|
|
@@ -30,24 +30,24 @@ A mirror is for reading. Pushing to both the origin and a mirror would give them
|
|
|
30
30
|
|
|
31
31
|
## Reading from a mirror
|
|
32
32
|
|
|
33
|
-
The
|
|
33
|
+
The metadata records where the writer published its data files: the origin's data folder, as an absolute URL. A reader of the mirror gives DuckDB the mirror's data folder instead:
|
|
34
34
|
|
|
35
35
|
```sql
|
|
36
|
-
ATTACH 'ducklake:https://mirror.example.org/towns/versions/019a3c4e-…/
|
|
36
|
+
ATTACH 'ducklake:https://mirror.example.org/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
37
37
|
DATA_PATH 'https://mirror.example.org/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
38
38
|
);
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
Without `OVERRIDE_DATA_PATH true`, DuckDB refuses to use a data path other than the one the
|
|
41
|
+
Without `OVERRIDE_DATA_PATH true`, DuckDB refuses to use a data path other than the one the metadata records. Readers always pass both options, at the origin too, so the same statement works anywhere with only the root changed.
|
|
42
42
|
|
|
43
43
|
## Checking what was copied
|
|
44
44
|
|
|
45
|
-
`version.json` records the size and sha256 of the
|
|
45
|
+
`version.json` records the size and sha256 of the metadata. A mirror can check each metadata file after copying it. Parquet files are named by a UUID and carry their own checksums, and DuckLake records each file's size in the metadata.
|
|
46
46
|
|
|
47
47
|
Anyone can write a `version.json`, so a checksum shows the copy matches the record, not that the record came from the origin. A version signed by the origin can be checked on a mirror: see [Signatures](/spec/reading#signatures).
|
|
48
48
|
|
|
49
49
|
## Expired versions and withdrawn data
|
|
50
50
|
|
|
51
|
-
When the origin takes a version out of retention, the mirror sees `expired.json` appear and the
|
|
51
|
+
When the origin takes a version out of retention, the mirror sees `expired.json` appear and the metadata disappear on the next copy. Copy `expired.json` and delete the metadata files the origin deleted. A mirror that keeps them serves versions its origin no longer does.
|
|
52
52
|
|
|
53
|
-
An `expired.json` with `"reason": "withdrawn"` means the origin removed data that must not be published. Delete the
|
|
53
|
+
An `expired.json` with `"reason": "withdrawn"` means the origin removed data that must not be published. Delete the metadata files, the data files and the assets the origin deleted, without waiting.
|
|
@@ -17,7 +17,7 @@ s3://public/towns/ the tables chosen for publication
|
|
|
17
17
|
A version of the public repository is made from a version of the private one:
|
|
18
18
|
|
|
19
19
|
1. Check out the public repository
|
|
20
|
-
2. Copy the chosen tables from the private version into the working
|
|
20
|
+
2. Copy the chosen tables from the private version into the working metadata. The Parquet files are copied into the public repository's data folder
|
|
21
21
|
3. Commit, and name the private version in `inputs`
|
|
22
22
|
|
|
23
23
|
```json
|
|
@@ -26,13 +26,13 @@ A version of the public repository is made from a version of the private one:
|
|
|
26
26
|
]
|
|
27
27
|
```
|
|
28
28
|
|
|
29
|
-
`sha256` is the private version's
|
|
29
|
+
`sha256` is the private version's metadata checksum. Readers of the public repository see that the version was made from another one, without being able to open it.
|
|
30
30
|
|
|
31
31
|
## Why not one repository
|
|
32
32
|
|
|
33
|
-
Putting a public
|
|
33
|
+
Putting a public metadata file next to a private one in a single repository looks simpler, but:
|
|
34
34
|
|
|
35
35
|
- the server would have to decide file by file who can read which Parquet file, and get it right for every file of every version
|
|
36
|
-
- the private
|
|
36
|
+
- the private metadata's snapshot history names the private tables, their columns and their commit messages. A public metadata file copied out of it would carry that history unless it was rewritten
|
|
37
37
|
|
|
38
38
|
Two repositories keep the rule simple. A reader can read everything in a repository or nothing in it.
|
|
@@ -22,7 +22,7 @@ const rows = await readWhere(await openParquet(store, file), "town_code", "13101
|
|
|
22
22
|
|
|
23
23
|
## What it can read
|
|
24
24
|
|
|
25
|
-
It follows [Reading](/spec/reading) up to the
|
|
25
|
+
It follows [Reading](/spec/reading) up to the metadata and then reads `files.json` in its place:
|
|
26
26
|
|
|
27
27
|
- A version without `files.json` cannot be read. Writers before fdl-cli 0.16 did not make one
|
|
28
28
|
- A table whose files have `deletes`, or whose `renamed_columns` is `true`, is refused (see [files.json](/spec/versions#filesjson))
|
|
@@ -3,18 +3,18 @@ title: Serving over HTTP
|
|
|
3
3
|
description: Letting readers attach a repository from a plain HTTP server or a CDN, and the caching each kind of file allows.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
Readers only read files, so a repository can be served by any HTTP server, a bucket's public URL or a CDN in front of one. DuckDB fetches the
|
|
6
|
+
Readers only read files, so a repository can be served by any HTTP server, a bucket's public URL or a CDN in front of one. DuckDB fetches the metadata and the Parquet files with range requests.
|
|
7
7
|
|
|
8
8
|
## What a reader fetches
|
|
9
9
|
|
|
10
10
|
1. `HEAD`, or the ref it was asked for
|
|
11
11
|
2. `refs/heads/main`, if `HEAD` pointed there
|
|
12
12
|
3. `versions/<ID>/version.json`, to check the version is not expired and that it can open the format
|
|
13
|
-
4. `versions/<ID>/
|
|
13
|
+
4. `versions/<ID>/metadata.db`, with range requests
|
|
14
14
|
5. the Parquet files the query needs, with range requests
|
|
15
15
|
|
|
16
16
|
```sql
|
|
17
|
-
ATTACH 'ducklake:https://data.example.org/towns/versions/019a3c4e-…/
|
|
17
|
+
ATTACH 'ducklake:https://data.example.org/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
18
18
|
DATA_PATH 'https://data.example.org/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
19
19
|
);
|
|
20
20
|
```
|
|
@@ -38,11 +38,11 @@ The server must answer range requests (`Accept-Ranges: bytes`, `206 Partial Cont
|
|
|
38
38
|
|
|
39
39
|
Readers do not list. A reader that knows a ref name gets everything else by following IDs, and one that does not reads `refs.json` for the names. Leave directory listings off; nothing needs them.
|
|
40
40
|
|
|
41
|
-
## Serving the latest
|
|
41
|
+
## Serving the latest metadata at a fixed URL
|
|
42
42
|
|
|
43
|
-
Some readers do not know FDL and expect one
|
|
43
|
+
Some readers do not know FDL and expect one metadata file URL. A host may serve the metadata of the version `HEAD` holds at `<root>/metadata.db`. It must then send the version ID in an `FDL-Version` response header, so a reader can tell which version it got, and must not let it be cached longer than the ref.
|
|
44
44
|
|
|
45
|
-
This is a convenience for readers, not part of the repository: the store holds no
|
|
45
|
+
This is a convenience for readers, not part of the repository: the store holds no metadata outside `versions/`.
|
|
46
46
|
|
|
47
47
|
## CORS
|
|
48
48
|
|