fdl-cli 0.27.0__tar.gz → 0.29.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/CHANGELOG.md +51 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/CONTRIBUTING.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/PKG-INFO +8 -8
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/README.md +7 -7
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/cases/collection.json +104 -0
- fdl_cli-0.29.0/cases/data-dir.json +211 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/concepts/working-metadata.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/getting-started.md +8 -6
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/building-a-host.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/public-and-private.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/internals/from-write-to-read.md +2 -2
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/internals/inside-a-repository.md +2 -2
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/concepts/versions-and-refs.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/concepts/working-metadata.md +5 -5
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/getting-started.md +8 -6
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/building-a-host.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/public-and-private.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/internals/from-write-to-read.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/internals/inside-a-repository.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/reference/cli.md +16 -14
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/reference/python.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/collection-cases.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/data-files.md +7 -2
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/http.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/layout.md +19 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/refs.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/writing.md +10 -8
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/reference/cli.md +16 -14
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/reference/python.md +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/collection-cases.md +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/data-files.md +7 -2
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/layout.md +19 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/writing.md +9 -7
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/pyproject.toml +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/cli.py +97 -19
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/diff.py +3 -3
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/layout.py +17 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/repo.py +20 -18
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_cli.py +137 -21
- fdl_cli-0.29.0/tests/test_data_dir_cases.py +24 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_http.py +10 -10
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_merge.py +2 -2
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_objects.py +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_profiles.py +1 -1
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_repo.py +11 -5
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.github/workflows/docs.yml +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.github/workflows/publish.yml +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.github/workflows/release.yml +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.github/workflows/test.yml +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.gitignore +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.node-version +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/.python-version +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/CLAUDE.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/LICENSE +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/SECURITY.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/blume.config.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/concepts/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/concepts/repositories.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/concepts/versions-and-refs.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/mirrors.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/reading-without-duckdb.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/serving-over-http.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/guides/stores.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/index.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/internals/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/concepts/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/concepts/repositories.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/mirrors.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/reading-without-duckdb.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/serving-over-http.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/guides/stores.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/index.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/internals/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/reference/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/index.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/reading.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/spec/versions.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/ja/stability.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/reference/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/http.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/index.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/meta.ts +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/reading.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/refs.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/spec/versions.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/docs/stability.md +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/package-lock.json +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/package.json +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/public/CNAME +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/__init__.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/catalog.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/dataset.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/lineage.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/merge.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/objects.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/profiles.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/py.typed +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/refs.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/store.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/stores.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/src/fdl/version.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/http_repo.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_collection_cases.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_dataset.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/tests/test_stores.py +0 -0
- {fdl_cli-0.27.0 → fdl_cli-0.29.0}/uv.lock +0 -0
|
@@ -1,6 +1,57 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
3
|
|
|
4
|
+
## v0.29.0 (2026-11-15)
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
- Record who takes a version out in expired.json, and keep version.json on withdrawal
|
|
9
|
+
([#65](https://github.com/queria-io/fdl/pull/65),
|
|
10
|
+
[`1d3159b`](https://github.com/queria-io/fdl/commit/1d3159b4a50d309424b4b25a756477f7c225934b))
|
|
11
|
+
|
|
12
|
+
A withdrawal no longer deletes version.json: it rewrites the fields that hold the data, so every
|
|
13
|
+
repository holds its history back to the first version. expire and withdraw take --by, recorded in
|
|
14
|
+
expired.json.
|
|
15
|
+
|
|
16
|
+
- Write the version's dataset.json, README and docs/ on clone
|
|
17
|
+
([#64](https://github.com/queria-io/fdl/pull/64),
|
|
18
|
+
[`1f37b04`](https://github.com/queria-io/fdl/commit/1f37b047866e662ff967e4adda561ed5ba5018e1))
|
|
19
|
+
|
|
20
|
+
A clone made only the working metadata, so whoever cloned a dataset had to write its description
|
|
21
|
+
again before they could commit. Clone now writes the version's dataset.json (without what commit
|
|
22
|
+
fills in) and the files of its tree.json into the current folder, unless a description is already
|
|
23
|
+
there.
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
## v0.28.0 (2026-11-15)
|
|
27
|
+
|
|
28
|
+
### Bug Fixes
|
|
29
|
+
|
|
30
|
+
- Refuse a data_dir that is not a folder under the root, and a build checked out before data_dir
|
|
31
|
+
moved ([#62](https://github.com/queria-io/fdl/pull/62),
|
|
32
|
+
[`fba88fc`](https://github.com/queria-io/fdl/commit/fba88fcc54b4cb0ad7106edb4b99009d4f6a9f43))
|
|
33
|
+
|
|
34
|
+
- The spec states what data_dir may be (segments of [a-z0-9_.-], ending in /, not under versions/,
|
|
35
|
+
refs/ or assets/), that it is used as written and never corrected, and cases/data-dir.json lists
|
|
36
|
+
values every implementation runs - read_config refuses other values, so "data" no longer writes
|
|
37
|
+
datamain/... while other readers read data/main/... - Layout describes how a host changes
|
|
38
|
+
data_dir; Data files says files outside data_dir are reached by no version and are the host's to
|
|
39
|
+
delete - checkout remembers data_dir, and commit and push refuse when fdl.json names another one,
|
|
40
|
+
saying so, instead of failing on missing data files - Two collection cases: a former data_dir is
|
|
41
|
+
left alone, and an invalid data_dir deletes nothing
|
|
42
|
+
|
|
43
|
+
BREAKING CHANGE: a repository whose data_dir breaks the rules is refused instead of read.
|
|
44
|
+
|
|
45
|
+
### Features
|
|
46
|
+
|
|
47
|
+
- Rename fdl checkout to fdl clone ([#63](https://github.com/queria-io/fdl/pull/63),
|
|
48
|
+
[`e4213e8`](https://github.com/queria-io/fdl/commit/e4213e8b6c9499367004e2fba9deaf100b49d94a))
|
|
49
|
+
|
|
50
|
+
### Breaking Changes
|
|
51
|
+
|
|
52
|
+
- A repository whose data_dir breaks the rules is refused instead of read.
|
|
53
|
+
|
|
54
|
+
|
|
4
55
|
## v0.27.0 (2026-11-15)
|
|
5
56
|
|
|
6
57
|
### Features
|
|
@@ -27,7 +27,7 @@ src/fdl/
|
|
|
27
27
|
├── dataset.py — dataset.json: reading the description and checking it against the catalog
|
|
28
28
|
├── objects.py — assets and tree.json
|
|
29
29
|
├── profiles.py — the profile of each table
|
|
30
|
-
├── repo.py — init,
|
|
30
|
+
├── repo.py — init, clone, commit, push, log, expire, withdraw
|
|
31
31
|
├── store.py — what a store has to provide
|
|
32
32
|
├── stores.py — a folder, S3 and a host over HTTP as stores
|
|
33
33
|
└── cli.py — the fdl command
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: fdl-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.29.0
|
|
4
4
|
Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
|
|
5
5
|
Project-URL: Homepage, https://fdl.queria.io/
|
|
6
6
|
Project-URL: Documentation, https://fdl.queria.io/
|
|
@@ -47,14 +47,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
|
|
|
47
47
|
pip install 'fdl-cli[s3]'
|
|
48
48
|
|
|
49
49
|
fdl init s3://my-bucket/towns/
|
|
50
|
-
fdl
|
|
50
|
+
fdl clone s3://my-bucket/towns/ work/lake.sqlite
|
|
51
51
|
```
|
|
52
52
|
|
|
53
53
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
54
54
|
|
|
55
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `
|
|
55
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
|
|
56
56
|
|
|
57
|
-
Write to the working metadata with DuckDB, attached the way `
|
|
57
|
+
Write to the working metadata with DuckDB, attached the way `clone` prints:
|
|
58
58
|
|
|
59
59
|
```sql
|
|
60
60
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -100,11 +100,11 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
100
100
|
| Command | What it does |
|
|
101
101
|
| --- | --- |
|
|
102
102
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
103
|
-
| `fdl
|
|
103
|
+
| `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
104
104
|
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
105
105
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
106
106
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
107
|
-
| `fdl status <working>` | What the working metadata changed since the
|
|
107
|
+
| `fdl status <working>` | What the working metadata changed since the clone |
|
|
108
108
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
109
109
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
110
110
|
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
@@ -116,7 +116,7 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
116
116
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
117
117
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
118
118
|
|
|
119
|
-
A push that is not built on what `main` holds now is refused:
|
|
119
|
+
A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
|
|
120
120
|
|
|
121
121
|
## Python
|
|
122
122
|
|
|
@@ -129,7 +129,7 @@ for version in repo.log(store):
|
|
|
129
129
|
print(version.seq, version.id, version.message)
|
|
130
130
|
```
|
|
131
131
|
|
|
132
|
-
`fdl.repo` has the operations without a terminal (`init`, `
|
|
132
|
+
`fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
|
|
133
133
|
|
|
134
134
|
## Earlier releases
|
|
135
135
|
|
|
@@ -27,14 +27,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
|
|
|
27
27
|
pip install 'fdl-cli[s3]'
|
|
28
28
|
|
|
29
29
|
fdl init s3://my-bucket/towns/
|
|
30
|
-
fdl
|
|
30
|
+
fdl clone s3://my-bucket/towns/ work/lake.sqlite
|
|
31
31
|
```
|
|
32
32
|
|
|
33
33
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
34
34
|
|
|
35
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `
|
|
35
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
|
|
36
36
|
|
|
37
|
-
Write to the working metadata with DuckDB, attached the way `
|
|
37
|
+
Write to the working metadata with DuckDB, attached the way `clone` prints:
|
|
38
38
|
|
|
39
39
|
```sql
|
|
40
40
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -80,11 +80,11 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
80
80
|
| Command | What it does |
|
|
81
81
|
| --- | --- |
|
|
82
82
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
83
|
-
| `fdl
|
|
83
|
+
| `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
84
84
|
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
85
85
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
86
86
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
87
|
-
| `fdl status <working>` | What the working metadata changed since the
|
|
87
|
+
| `fdl status <working>` | What the working metadata changed since the clone |
|
|
88
88
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
89
89
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
90
90
|
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
@@ -96,7 +96,7 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
96
96
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
97
97
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
98
98
|
|
|
99
|
-
A push that is not built on what `main` holds now is refused:
|
|
99
|
+
A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
|
|
100
100
|
|
|
101
101
|
## Python
|
|
102
102
|
|
|
@@ -109,7 +109,7 @@ for version in repo.log(store):
|
|
|
109
109
|
print(version.seq, version.id, version.message)
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
-
`fdl.repo` has the operations without a terminal (`init`, `
|
|
112
|
+
`fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
|
|
113
113
|
|
|
114
114
|
## Earlier releases
|
|
115
115
|
|
|
@@ -433,6 +433,110 @@
|
|
|
433
433
|
"assets/abc": "2026-11-01T00:00:00Z"
|
|
434
434
|
},
|
|
435
435
|
"deleted": []
|
|
436
|
+
},
|
|
437
|
+
{
|
|
438
|
+
"name": "leaves alone the files of a former data_dir",
|
|
439
|
+
"files": {
|
|
440
|
+
"fdl.json": {
|
|
441
|
+
"fdl": "0.1",
|
|
442
|
+
"data_dir": "data/"
|
|
443
|
+
},
|
|
444
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
445
|
+
"fdl": "0.1",
|
|
446
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
447
|
+
"seq": 1,
|
|
448
|
+
"parents": [],
|
|
449
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
450
|
+
"author": {
|
|
451
|
+
"name": "example"
|
|
452
|
+
},
|
|
453
|
+
"message": "",
|
|
454
|
+
"catalog": {
|
|
455
|
+
"path": "ducklake.duckdb",
|
|
456
|
+
"storage": "duckdb-v1.5.0",
|
|
457
|
+
"bytes": 0,
|
|
458
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
459
|
+
},
|
|
460
|
+
"ducklake": {
|
|
461
|
+
"format": "1.0",
|
|
462
|
+
"snapshot_id": 1,
|
|
463
|
+
"snapshots": [
|
|
464
|
+
1
|
|
465
|
+
]
|
|
466
|
+
},
|
|
467
|
+
"files": {
|
|
468
|
+
"path": "files.json",
|
|
469
|
+
"bytes": 0,
|
|
470
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
471
|
+
}
|
|
472
|
+
},
|
|
473
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
474
|
+
"tables": {
|
|
475
|
+
"main.t": {
|
|
476
|
+
"files": []
|
|
477
|
+
}
|
|
478
|
+
},
|
|
479
|
+
"named": []
|
|
480
|
+
}
|
|
481
|
+
},
|
|
482
|
+
"objects": {
|
|
483
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
484
|
+
"ducklake.duckdb.files/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
|
|
485
|
+
},
|
|
486
|
+
"deleted": [
|
|
487
|
+
"data/main/t/ducklake-a.parquet"
|
|
488
|
+
]
|
|
489
|
+
},
|
|
490
|
+
{
|
|
491
|
+
"name": "deletes nothing while data_dir is not a folder under the root",
|
|
492
|
+
"files": {
|
|
493
|
+
"fdl.json": {
|
|
494
|
+
"fdl": "0.1",
|
|
495
|
+
"data_dir": "data"
|
|
496
|
+
},
|
|
497
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
498
|
+
"fdl": "0.1",
|
|
499
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
500
|
+
"seq": 1,
|
|
501
|
+
"parents": [],
|
|
502
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
503
|
+
"author": {
|
|
504
|
+
"name": "example"
|
|
505
|
+
},
|
|
506
|
+
"message": "",
|
|
507
|
+
"catalog": {
|
|
508
|
+
"path": "ducklake.duckdb",
|
|
509
|
+
"storage": "duckdb-v1.5.0",
|
|
510
|
+
"bytes": 0,
|
|
511
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
512
|
+
},
|
|
513
|
+
"ducklake": {
|
|
514
|
+
"format": "1.0",
|
|
515
|
+
"snapshot_id": 1,
|
|
516
|
+
"snapshots": [
|
|
517
|
+
1
|
|
518
|
+
]
|
|
519
|
+
},
|
|
520
|
+
"files": {
|
|
521
|
+
"path": "files.json",
|
|
522
|
+
"bytes": 0,
|
|
523
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
524
|
+
}
|
|
525
|
+
},
|
|
526
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
527
|
+
"tables": {
|
|
528
|
+
"main.t": {
|
|
529
|
+
"files": []
|
|
530
|
+
}
|
|
531
|
+
},
|
|
532
|
+
"named": []
|
|
533
|
+
}
|
|
534
|
+
},
|
|
535
|
+
"objects": {
|
|
536
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
537
|
+
"datamain/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
|
|
538
|
+
},
|
|
539
|
+
"deleted": []
|
|
436
540
|
}
|
|
437
541
|
]
|
|
438
542
|
}
|
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
{
|
|
2
|
+
"cases": [
|
|
3
|
+
{
|
|
4
|
+
"name": "absent: the default",
|
|
5
|
+
"fdl_json": {
|
|
6
|
+
"fdl": "0.1"
|
|
7
|
+
},
|
|
8
|
+
"data_dir": "data/"
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"name": "the default, written out",
|
|
12
|
+
"fdl_json": {
|
|
13
|
+
"fdl": "0.1",
|
|
14
|
+
"data_dir": "data/"
|
|
15
|
+
},
|
|
16
|
+
"data_dir": "data/"
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"name": "dots inside a segment",
|
|
20
|
+
"fdl_json": {
|
|
21
|
+
"fdl": "0.1",
|
|
22
|
+
"data_dir": "ducklake.duckdb.files/"
|
|
23
|
+
},
|
|
24
|
+
"data_dir": "ducklake.duckdb.files/"
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"name": "more than one segment",
|
|
28
|
+
"fdl_json": {
|
|
29
|
+
"fdl": "0.1",
|
|
30
|
+
"data_dir": "lake/data/"
|
|
31
|
+
},
|
|
32
|
+
"data_dir": "lake/data/"
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"name": "digits, _ and -",
|
|
36
|
+
"fdl_json": {
|
|
37
|
+
"fdl": "0.1",
|
|
38
|
+
"data_dir": "data_2-x/"
|
|
39
|
+
},
|
|
40
|
+
"data_dir": "data_2-x/"
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"name": "no trailing slash",
|
|
44
|
+
"fdl_json": {
|
|
45
|
+
"fdl": "0.1",
|
|
46
|
+
"data_dir": "data"
|
|
47
|
+
},
|
|
48
|
+
"data_dir": null
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
"name": "empty",
|
|
52
|
+
"fdl_json": {
|
|
53
|
+
"fdl": "0.1",
|
|
54
|
+
"data_dir": ""
|
|
55
|
+
},
|
|
56
|
+
"data_dir": null
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"name": "the root itself",
|
|
60
|
+
"fdl_json": {
|
|
61
|
+
"fdl": "0.1",
|
|
62
|
+
"data_dir": "/"
|
|
63
|
+
},
|
|
64
|
+
"data_dir": null
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
"name": "starts with a slash",
|
|
68
|
+
"fdl_json": {
|
|
69
|
+
"fdl": "0.1",
|
|
70
|
+
"data_dir": "/data/"
|
|
71
|
+
},
|
|
72
|
+
"data_dir": null
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
"name": "an empty segment",
|
|
76
|
+
"fdl_json": {
|
|
77
|
+
"fdl": "0.1",
|
|
78
|
+
"data_dir": "data//x/"
|
|
79
|
+
},
|
|
80
|
+
"data_dir": null
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
"name": "a dot segment",
|
|
84
|
+
"fdl_json": {
|
|
85
|
+
"fdl": "0.1",
|
|
86
|
+
"data_dir": "./data/"
|
|
87
|
+
},
|
|
88
|
+
"data_dir": null
|
|
89
|
+
},
|
|
90
|
+
{
|
|
91
|
+
"name": "a parent segment",
|
|
92
|
+
"fdl_json": {
|
|
93
|
+
"fdl": "0.1",
|
|
94
|
+
"data_dir": "../data/"
|
|
95
|
+
},
|
|
96
|
+
"data_dir": null
|
|
97
|
+
},
|
|
98
|
+
{
|
|
99
|
+
"name": "a parent segment inside",
|
|
100
|
+
"fdl_json": {
|
|
101
|
+
"fdl": "0.1",
|
|
102
|
+
"data_dir": "data/../refs/"
|
|
103
|
+
},
|
|
104
|
+
"data_dir": null
|
|
105
|
+
},
|
|
106
|
+
{
|
|
107
|
+
"name": "dots alone",
|
|
108
|
+
"fdl_json": {
|
|
109
|
+
"fdl": "0.1",
|
|
110
|
+
"data_dir": ".../"
|
|
111
|
+
},
|
|
112
|
+
"data_dir": null
|
|
113
|
+
},
|
|
114
|
+
{
|
|
115
|
+
"name": "percent-encoded dots",
|
|
116
|
+
"fdl_json": {
|
|
117
|
+
"fdl": "0.1",
|
|
118
|
+
"data_dir": "%2e%2e/"
|
|
119
|
+
},
|
|
120
|
+
"data_dir": null
|
|
121
|
+
},
|
|
122
|
+
{
|
|
123
|
+
"name": "an absolute URL",
|
|
124
|
+
"fdl_json": {
|
|
125
|
+
"fdl": "0.1",
|
|
126
|
+
"data_dir": "https://example.org/data/"
|
|
127
|
+
},
|
|
128
|
+
"data_dir": null
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"name": "another storage",
|
|
132
|
+
"fdl_json": {
|
|
133
|
+
"fdl": "0.1",
|
|
134
|
+
"data_dir": "s3://bucket/data/"
|
|
135
|
+
},
|
|
136
|
+
"data_dir": null
|
|
137
|
+
},
|
|
138
|
+
{
|
|
139
|
+
"name": "a backslash",
|
|
140
|
+
"fdl_json": {
|
|
141
|
+
"fdl": "0.1",
|
|
142
|
+
"data_dir": "data\\x/"
|
|
143
|
+
},
|
|
144
|
+
"data_dir": null
|
|
145
|
+
},
|
|
146
|
+
{
|
|
147
|
+
"name": "a space",
|
|
148
|
+
"fdl_json": {
|
|
149
|
+
"fdl": "0.1",
|
|
150
|
+
"data_dir": "data files/"
|
|
151
|
+
},
|
|
152
|
+
"data_dir": null
|
|
153
|
+
},
|
|
154
|
+
{
|
|
155
|
+
"name": "upper case",
|
|
156
|
+
"fdl_json": {
|
|
157
|
+
"fdl": "0.1",
|
|
158
|
+
"data_dir": "Data/"
|
|
159
|
+
},
|
|
160
|
+
"data_dir": null
|
|
161
|
+
},
|
|
162
|
+
{
|
|
163
|
+
"name": "non-ASCII",
|
|
164
|
+
"fdl_json": {
|
|
165
|
+
"fdl": "0.1",
|
|
166
|
+
"data_dir": "データ/"
|
|
167
|
+
},
|
|
168
|
+
"data_dir": null
|
|
169
|
+
},
|
|
170
|
+
{
|
|
171
|
+
"name": "versions",
|
|
172
|
+
"fdl_json": {
|
|
173
|
+
"fdl": "0.1",
|
|
174
|
+
"data_dir": "versions/"
|
|
175
|
+
},
|
|
176
|
+
"data_dir": null
|
|
177
|
+
},
|
|
178
|
+
{
|
|
179
|
+
"name": "under refs",
|
|
180
|
+
"fdl_json": {
|
|
181
|
+
"fdl": "0.1",
|
|
182
|
+
"data_dir": "refs/x/"
|
|
183
|
+
},
|
|
184
|
+
"data_dir": null
|
|
185
|
+
},
|
|
186
|
+
{
|
|
187
|
+
"name": "assets",
|
|
188
|
+
"fdl_json": {
|
|
189
|
+
"fdl": "0.1",
|
|
190
|
+
"data_dir": "assets/"
|
|
191
|
+
},
|
|
192
|
+
"data_dir": null
|
|
193
|
+
},
|
|
194
|
+
{
|
|
195
|
+
"name": "not a string",
|
|
196
|
+
"fdl_json": {
|
|
197
|
+
"fdl": "0.1",
|
|
198
|
+
"data_dir": 1
|
|
199
|
+
},
|
|
200
|
+
"data_dir": null
|
|
201
|
+
},
|
|
202
|
+
{
|
|
203
|
+
"name": "null",
|
|
204
|
+
"fdl_json": {
|
|
205
|
+
"fdl": "0.1",
|
|
206
|
+
"data_dir": null
|
|
207
|
+
},
|
|
208
|
+
"data_dir": null
|
|
209
|
+
}
|
|
210
|
+
]
|
|
211
|
+
}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: Working metadata files
|
|
3
|
-
description: "Where writing happens: a DuckLake metadata file in SQLite,
|
|
3
|
+
description: "Where writing happens: a DuckLake metadata file in SQLite, cloned from a version and frozen into the next one."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
A working metadata file is a DuckLake metadata file in a SQLite file, on the writer's own disk or in a container. It is the only place anyone writes to. It is to FDL what a working tree is to Git: made from a version, changed freely, and turned into the next version when the change is ready.
|
|
@@ -9,7 +9,7 @@ A working metadata file is a DuckLake metadata file in a SQLite file, on the wri
|
|
|
9
9
|
|
|
10
10
|
| Step | Command | What happens |
|
|
11
11
|
| --- | --- | --- |
|
|
12
|
-
|
|
|
12
|
+
| clone | `fdl clone` | Copy the metadata of the branch's version into SQLite, and remember that version |
|
|
13
13
|
| write | any DuckDB client | DuckLake writes Parquet into the repository and bookkeeping into SQLite |
|
|
14
14
|
| commit | `fdl commit` | Freeze the last snapshot into a new version, kept beside the working metadata |
|
|
15
15
|
| push | `fdl push` | Upload the version and move the branch to it |
|
|
@@ -24,7 +24,7 @@ There is one kind of working metadata. A host that writes on behalf of its users
|
|
|
24
24
|
|
|
25
25
|
## Attaching
|
|
26
26
|
|
|
27
|
-
`fdl
|
|
27
|
+
`fdl clone` prints the statement to attach the working metadata with:
|
|
28
28
|
|
|
29
29
|
```sql
|
|
30
30
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -31,15 +31,15 @@ towns/
|
|
|
31
31
|
|
|
32
32
|
`fdl init` makes the repository where readers will find it, as `git init --bare` does on a server. Your own place to write is the working metadata, made next.
|
|
33
33
|
|
|
34
|
-
##
|
|
34
|
+
## Clone a working metadata file
|
|
35
35
|
|
|
36
36
|
You never write to the repository directly. You write to a working metadata file, a DuckLake metadata file in a SQLite file on your own disk, and turn it into a version when you are done.
|
|
37
37
|
|
|
38
38
|
```bash
|
|
39
|
-
fdl
|
|
39
|
+
fdl clone ./towns/ work/lake.sqlite
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
-
Before the first version, `
|
|
42
|
+
Before the first version, `clone` makes an empty metadata. Later it copies the metadata of the version `main` holds. It prints the statement to attach the working metadata with:
|
|
43
43
|
|
|
44
44
|
```sql
|
|
45
45
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -49,7 +49,7 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
49
49
|
);
|
|
50
50
|
```
|
|
51
51
|
|
|
52
|
-
Beside the working metadata, `work/lake.sqlite.fdl.json` remembers the repository, the branch and the version the
|
|
52
|
+
Beside the working metadata, `work/lake.sqlite.fdl.json` remembers the repository, the branch and the version the clone was made from.
|
|
53
53
|
|
|
54
54
|
## Write
|
|
55
55
|
|
|
@@ -128,7 +128,7 @@ A version's metadata never changes, so a reader may cache it forever. Only `HEAD
|
|
|
128
128
|
## Make a second version
|
|
129
129
|
|
|
130
130
|
```bash
|
|
131
|
-
fdl
|
|
131
|
+
fdl clone ./towns/ work/lake.sqlite
|
|
132
132
|
```
|
|
133
133
|
|
|
134
134
|
```sql
|
|
@@ -140,6 +140,8 @@ fdl commit work/lake.sqlite -m "Add 2026" --author alice
|
|
|
140
140
|
fdl push work/lake.sqlite
|
|
141
141
|
```
|
|
142
142
|
|
|
143
|
+
The current folder already holds `dataset.json`, so the clone leaves it as it is. In an empty folder, the clone also writes the version's `dataset.json`, `README.md` and `docs/` there, so whoever clones the repository can change the description as well as the data.
|
|
144
|
+
|
|
143
145
|
The new version lists the first as its parent. Both metadata files name the first Parquet file; only the second names the new one.
|
|
144
146
|
|
|
145
|
-
If someone else pushed in between, the push is refused, because the new version would not be built on what `main` holds now.
|
|
147
|
+
If someone else pushed in between, the push is refused, because the new version would not be built on what `main` holds now. Clone again and redo the change. [Versions and refs](/concepts/versions-and-refs) explains why.
|
|
@@ -36,7 +36,7 @@ The host checks that the version is built on what `main` holds now (`repo.is_anc
|
|
|
36
36
|
|
|
37
37
|
## Writing on behalf of users
|
|
38
38
|
|
|
39
|
-
A scheduled import or a form in a web app writes the same way a person does:
|
|
39
|
+
A scheduled import or a form in a web app writes the same way a person does: clone into a SQLite file in a container, write with DuckDB, commit. Two jobs that commit at once are settled by the conditional write on the ref. The loser checks out again and reruns its SQL.
|
|
40
40
|
|
|
41
41
|
## Keeping the host out of the format
|
|
42
42
|
|
|
@@ -16,7 +16,7 @@ s3://public/towns/ the tables chosen for publication
|
|
|
16
16
|
|
|
17
17
|
A version of the public repository is made from a version of the private one:
|
|
18
18
|
|
|
19
|
-
1.
|
|
19
|
+
1. Clone the public repository
|
|
20
20
|
2. Copy the chosen tables from the private version into the working metadata. The Parquet files are copied into the public repository's data folder
|
|
21
21
|
3. Commit, and name the private version in `inputs`
|
|
22
22
|
|
|
@@ -34,9 +34,9 @@ Three things differ on purpose:
|
|
|
34
34
|
|
|
35
35
|
## Writing a version
|
|
36
36
|
|
|
37
|
-
###
|
|
37
|
+
### Clone
|
|
38
38
|
|
|
39
|
-
`fdl
|
|
39
|
+
`fdl clone` reads `HEAD`, then the ref it names, then that version's `version.json`, and copies its metadata into a SQLite file on the writer's machine. This is the working metadata. If the version has `expired.json`, clone stops.
|
|
40
40
|
|
|
41
41
|
The working metadata's data path depends on the store:
|
|
42
42
|
|
|
@@ -28,7 +28,7 @@ ref: refs/heads/main
|
|
|
28
28
|
|
|
29
29
|
## While writing
|
|
30
30
|
|
|
31
|
-
After `fdl
|
|
31
|
+
After `fdl clone` and a `CREATE TABLE`, DuckLake has written Parquet into the repository:
|
|
32
32
|
|
|
33
33
|
```
|
|
34
34
|
fdl.json
|
|
@@ -95,7 +95,7 @@ data/main/towns/ducklake-019a3b….parquet named by the second only
|
|
|
95
95
|
}
|
|
96
96
|
```
|
|
97
97
|
|
|
98
|
-
`main` is moved with `If-Match: <the ETag read at the start of the push>`. Had Alice moved it in the meantime, the ETag would not match and the store would refuse the write. Bob's version folder would stay, unreferenced, and Bob would
|
|
98
|
+
`main` is moved with `If-Match: <the ETag read at the start of the push>`. Had Alice moved it in the meantime, the ETag would not match and the store would refuse the write. Bob's version folder would stay, unreferenced, and Bob would clone again.
|
|
99
99
|
|
|
100
100
|
The first Parquet file is not copied. Both metadata files name it at the same path, which is why data files must never be overwritten.
|
|
101
101
|
|
|
@@ -60,11 +60,11 @@ HEAD ref: refs/heads/main
|
|
|
60
60
|
|
|
61
61
|
```
|
|
62
62
|
main → A
|
|
63
|
-
Alice が A
|
|
64
|
-
Bob も A
|
|
63
|
+
Alice が A をクローンし、B(親 A)をコミットして push: main → B
|
|
64
|
+
Bob も A をクローンしていて、C(親 A)をコミットして push: 断られる
|
|
65
65
|
```
|
|
66
66
|
|
|
67
|
-
C は B の上に積まれていないので、Bob の push は断られます。受け付けると、誰も決めていないのに B が `main` の履歴から消えてしまいます。Bob は B
|
|
67
|
+
C は B の上に積まれていないので、Bob の push は断られます。受け付けると、誰も決めていないのに B が `main` の履歴から消えてしまいます。Bob は B をクローンし、変更をやり直します。新しいバージョンの上でのやり直しは、たいてい同じ SQL を実行し直すだけで済みます。
|
|
68
68
|
|
|
69
69
|
## マージ
|
|
70
70
|
|