fdl-cli 0.26.0__tar.gz → 0.28.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CHANGELOG.md +57 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CONTRIBUTING.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/PKG-INFO +19 -19
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/README.md +18 -18
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/blume.config.ts +5 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/cases/collection.json +104 -0
- fdl_cli-0.28.0/cases/data-dir.json +211 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/meta.ts +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/repositories.md +4 -4
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/versions-and-refs.md +8 -8
- fdl_cli-0.28.0/docs/concepts/working-metadata.md +49 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/getting-started.md +16 -16
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/building-a-host.md +2 -2
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/mirrors.md +6 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/public-and-private.md +5 -5
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/reading-without-duckdb.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/serving-over-http.md +6 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/index.md +8 -8
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/ducklake-formats.md +15 -15
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/from-write-to-read.md +23 -23
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/inside-a-repository.md +18 -18
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/meta.ts +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/repositories.md +4 -4
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/versions-and-refs.md +11 -11
- fdl_cli-0.28.0/docs/ja/concepts/working-metadata.md +49 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/getting-started.md +16 -16
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/building-a-host.md +2 -2
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/mirrors.md +6 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/public-and-private.md +5 -5
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/reading-without-duckdb.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/serving-over-http.md +6 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/index.md +8 -8
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/ducklake-formats.md +15 -15
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/from-write-to-read.md +24 -24
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/inside-a-repository.md +19 -19
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/cli.md +29 -29
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/python.md +11 -11
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/collection-cases.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/data-files.md +15 -11
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/http.md +6 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/index.md +7 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/layout.md +28 -9
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/reading.md +14 -14
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/refs.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/versions.md +48 -46
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/writing.md +23 -21
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/stability.md +3 -3
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/cli.md +29 -29
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/python.md +11 -11
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/collection-cases.md +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/data-files.md +15 -11
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/http.md +5 -5
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/index.md +7 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/layout.md +28 -9
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/reading.md +13 -13
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/versions.md +47 -45
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/writing.md +23 -21
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/stability.md +3 -3
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/pyproject.toml +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/cli.py +77 -45
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/diff.py +3 -3
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/layout.py +32 -6
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/repo.py +70 -70
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/version.py +16 -13
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_cli.py +85 -22
- fdl_cli-0.28.0/tests/test_data_dir_cases.py +24 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_http.py +14 -14
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_merge.py +4 -4
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_objects.py +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_profiles.py +1 -1
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_repo.py +28 -14
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/uv.lock +1 -1
- fdl_cli-0.26.0/docs/concepts/working-catalogs.md +0 -49
- fdl_cli-0.26.0/docs/ja/concepts/working-catalogs.md +0 -49
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/docs.yml +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/publish.yml +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/release.yml +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/test.yml +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.gitignore +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.node-version +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.python-version +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CLAUDE.md +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/LICENSE +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/SECURITY.md +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/stores.md +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/stores.md +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/meta.ts +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/refs.md +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/package-lock.json +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/package.json +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/public/CNAME +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/__init__.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/catalog.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/dataset.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/lineage.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/merge.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/objects.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/profiles.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/py.typed +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/refs.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/store.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/stores.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/http_repo.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_collection_cases.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_dataset.py +0 -0
- {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_stores.py +0 -0
|
@@ -1,6 +1,63 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
3
|
|
|
4
|
+
## v0.28.0 (2026-11-15)
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
- Refuse a data_dir that is not a folder under the root, and a build checked out before data_dir
|
|
9
|
+
moved ([#62](https://github.com/queria-io/fdl/pull/62),
|
|
10
|
+
[`fba88fc`](https://github.com/queria-io/fdl/commit/fba88fcc54b4cb0ad7106edb4b99009d4f6a9f43))
|
|
11
|
+
|
|
12
|
+
- The spec states what data_dir may be (segments of [a-z0-9_.-], ending in /, not under versions/,
|
|
13
|
+
refs/ or assets/), that it is used as written and never corrected, and cases/data-dir.json lists
|
|
14
|
+
values every implementation runs - read_config refuses other values, so "data" no longer writes
|
|
15
|
+
datamain/... while other readers read data/main/... - Layout describes how a host changes
|
|
16
|
+
data_dir; Data files says files outside data_dir are reached by no version and are the host's to
|
|
17
|
+
delete - checkout remembers data_dir, and commit and push refuse when fdl.json names another one,
|
|
18
|
+
saying so, instead of failing on missing data files - Two collection cases: a former data_dir is
|
|
19
|
+
left alone, and an invalid data_dir deletes nothing
|
|
20
|
+
|
|
21
|
+
BREAKING CHANGE: a repository whose data_dir breaks the rules is refused instead of read.
|
|
22
|
+
|
|
23
|
+
### Features
|
|
24
|
+
|
|
25
|
+
- Rename fdl checkout to fdl clone ([#63](https://github.com/queria-io/fdl/pull/63),
|
|
26
|
+
[`e4213e8`](https://github.com/queria-io/fdl/commit/e4213e8b6c9499367004e2fba9deaf100b49d94a))
|
|
27
|
+
|
|
28
|
+
### Breaking Changes
|
|
29
|
+
|
|
30
|
+
- A repository whose data_dir breaks the rules is refused instead of read.
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
## v0.27.0 (2026-11-15)
|
|
34
|
+
|
|
35
|
+
### Features
|
|
36
|
+
|
|
37
|
+
- Name the frozen DuckLake metadata metadata.db and record it as metadata
|
|
38
|
+
([#61](https://github.com/queria-io/fdl/pull/61),
|
|
39
|
+
[`babd57f`](https://github.com/queria-io/fdl/commit/babd57fe57a45468019f00042e9d52f011afe15d))
|
|
40
|
+
|
|
41
|
+
* feat!: name the frozen DuckLake metadata metadata.db and record it as metadata
|
|
42
|
+
|
|
43
|
+
A version's metadata file is now versions/<ID>/metadata.db, recorded under "metadata" in
|
|
44
|
+
version.json; other DuckLake formats go under metadata/, and fdl.json declares them as
|
|
45
|
+
metadata_formats. Readers open the name the record gives, so versions written by 0.26 and earlier
|
|
46
|
+
(ducklake.duckdb, recorded as "catalog") are still read.
|
|
47
|
+
|
|
48
|
+
BREAKING CHANGE: version.json records the metadata file as "metadata", and fdl.json's
|
|
49
|
+
catalog_formats is now metadata_formats.
|
|
50
|
+
|
|
51
|
+
* docs: redirect the old working catalog page
|
|
52
|
+
|
|
53
|
+
* fix(expire): also drop the other formats that fdl-cli 0.26 and earlier kept under catalogs/
|
|
54
|
+
|
|
55
|
+
### Breaking Changes
|
|
56
|
+
|
|
57
|
+
- Version.json records the metadata file as "metadata", and fdl.json's catalog_formats is now
|
|
58
|
+
metadata_formats.
|
|
59
|
+
|
|
60
|
+
|
|
4
61
|
## v0.26.0 (2026-11-14)
|
|
5
62
|
|
|
6
63
|
### Features
|
|
@@ -27,7 +27,7 @@ src/fdl/
|
|
|
27
27
|
├── dataset.py — dataset.json: reading the description and checking it against the catalog
|
|
28
28
|
├── objects.py — assets and tree.json
|
|
29
29
|
├── profiles.py — the profile of each table
|
|
30
|
-
├── repo.py — init,
|
|
30
|
+
├── repo.py — init, clone, commit, push, log, expire, withdraw
|
|
31
31
|
├── store.py — what a store has to provide
|
|
32
32
|
├── stores.py — a folder, S3 and a host over HTTP as stores
|
|
33
33
|
└── cli.py — the fdl command
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: fdl-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.28.0
|
|
4
4
|
Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
|
|
5
5
|
Project-URL: Homepage, https://fdl.queria.io/
|
|
6
6
|
Project-URL: Documentation, https://fdl.queria.io/
|
|
@@ -24,9 +24,9 @@ Description-Content-Type: text/markdown
|
|
|
24
24
|
[](https://pypi.org/project/fdl-cli/)
|
|
25
25
|
[](LICENSE)
|
|
26
26
|
|
|
27
|
-
Versions of
|
|
27
|
+
Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
|
|
28
28
|
|
|
29
|
-
FDL is a storage layout and a tool for it. Each version is a frozen
|
|
29
|
+
FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
|
|
30
30
|
|
|
31
31
|
```
|
|
32
32
|
<root>/
|
|
@@ -35,7 +35,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
|
|
|
35
35
|
refs/heads/main 019a3c4e-…
|
|
36
36
|
versions/019a3c4e-…/
|
|
37
37
|
version.json parents, author, message, sha256
|
|
38
|
-
|
|
38
|
+
metadata.db the frozen metadata
|
|
39
39
|
data/main/towns/ducklake-….parquet
|
|
40
40
|
```
|
|
41
41
|
|
|
@@ -47,14 +47,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
|
|
|
47
47
|
pip install 'fdl-cli[s3]'
|
|
48
48
|
|
|
49
49
|
fdl init s3://my-bucket/towns/
|
|
50
|
-
fdl
|
|
50
|
+
fdl clone s3://my-bucket/towns/ work/lake.sqlite
|
|
51
51
|
```
|
|
52
52
|
|
|
53
53
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
54
54
|
|
|
55
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working
|
|
55
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
|
|
56
56
|
|
|
57
|
-
Write to the working
|
|
57
|
+
Write to the working metadata with DuckDB, attached the way `clone` prints:
|
|
58
58
|
|
|
59
59
|
```sql
|
|
60
60
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -64,9 +64,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
64
64
|
CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
|
|
65
65
|
```
|
|
66
66
|
|
|
67
|
-
Rows have to go to Parquet, not into the
|
|
67
|
+
Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
|
|
68
68
|
|
|
69
|
-
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the
|
|
69
|
+
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
|
|
70
70
|
|
|
71
71
|
```yaml
|
|
72
72
|
name: towns
|
|
@@ -88,7 +88,7 @@ fdl url s3://my-bucket/towns/
|
|
|
88
88
|
```
|
|
89
89
|
|
|
90
90
|
```sql
|
|
91
|
-
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/
|
|
91
|
+
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
92
92
|
DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
93
93
|
);
|
|
94
94
|
```
|
|
@@ -100,23 +100,23 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
100
100
|
| Command | What it does |
|
|
101
101
|
| --- | --- |
|
|
102
102
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
103
|
-
| `fdl
|
|
104
|
-
| `fdl commit <working> -m … --author …` | Freeze the working
|
|
103
|
+
| `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
104
|
+
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
105
105
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
106
106
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
107
|
-
| `fdl status <working>` | What the working
|
|
107
|
+
| `fdl status <working>` | What the working metadata changed since the clone |
|
|
108
108
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
109
109
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
110
|
-
| `fdl url <remote> [--attach]` | Print the
|
|
110
|
+
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
111
111
|
| `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
|
|
112
112
|
| `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
|
|
113
|
-
| `fdl merge <ref> <working> --author …` | Merge another branch into the working
|
|
114
|
-
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their
|
|
113
|
+
| `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
|
|
114
|
+
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
|
|
115
115
|
| `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
|
|
116
116
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
117
117
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
118
118
|
|
|
119
|
-
A push that is not built on what `main` holds now is refused:
|
|
119
|
+
A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
|
|
120
120
|
|
|
121
121
|
## Python
|
|
122
122
|
|
|
@@ -129,11 +129,11 @@ for version in repo.log(store):
|
|
|
129
129
|
print(version.seq, version.id, version.message)
|
|
130
130
|
```
|
|
131
131
|
|
|
132
|
-
`fdl.repo` has the operations without a terminal (`init`, `
|
|
132
|
+
`fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
|
|
133
133
|
|
|
134
134
|
## Earlier releases
|
|
135
135
|
|
|
136
|
-
Releases up to 0.11 were a different tool for managing a single frozen
|
|
136
|
+
Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
|
|
137
137
|
|
|
138
138
|
## License
|
|
139
139
|
|
|
@@ -4,9 +4,9 @@
|
|
|
4
4
|
[](https://pypi.org/project/fdl-cli/)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
|
|
7
|
-
Versions of
|
|
7
|
+
Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
|
|
8
8
|
|
|
9
|
-
FDL is a storage layout and a tool for it. Each version is a frozen
|
|
9
|
+
FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
|
|
10
10
|
|
|
11
11
|
```
|
|
12
12
|
<root>/
|
|
@@ -15,7 +15,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
|
|
|
15
15
|
refs/heads/main 019a3c4e-…
|
|
16
16
|
versions/019a3c4e-…/
|
|
17
17
|
version.json parents, author, message, sha256
|
|
18
|
-
|
|
18
|
+
metadata.db the frozen metadata
|
|
19
19
|
data/main/towns/ducklake-….parquet
|
|
20
20
|
```
|
|
21
21
|
|
|
@@ -27,14 +27,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
|
|
|
27
27
|
pip install 'fdl-cli[s3]'
|
|
28
28
|
|
|
29
29
|
fdl init s3://my-bucket/towns/
|
|
30
|
-
fdl
|
|
30
|
+
fdl clone s3://my-bucket/towns/ work/lake.sqlite
|
|
31
31
|
```
|
|
32
32
|
|
|
33
33
|
The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
|
|
34
34
|
|
|
35
|
-
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working
|
|
35
|
+
`fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
|
|
36
36
|
|
|
37
|
-
Write to the working
|
|
37
|
+
Write to the working metadata with DuckDB, attached the way `clone` prints:
|
|
38
38
|
|
|
39
39
|
```sql
|
|
40
40
|
ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
@@ -44,9 +44,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
|
|
|
44
44
|
CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
|
|
45
45
|
```
|
|
46
46
|
|
|
47
|
-
Rows have to go to Parquet, not into the
|
|
47
|
+
Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
|
|
48
48
|
|
|
49
|
-
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the
|
|
49
|
+
Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
|
|
50
50
|
|
|
51
51
|
```yaml
|
|
52
52
|
name: towns
|
|
@@ -68,7 +68,7 @@ fdl url s3://my-bucket/towns/
|
|
|
68
68
|
```
|
|
69
69
|
|
|
70
70
|
```sql
|
|
71
|
-
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/
|
|
71
|
+
ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
|
|
72
72
|
DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
|
|
73
73
|
);
|
|
74
74
|
```
|
|
@@ -80,23 +80,23 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
|
|
|
80
80
|
| Command | What it does |
|
|
81
81
|
| --- | --- |
|
|
82
82
|
| `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
|
|
83
|
-
| `fdl
|
|
84
|
-
| `fdl commit <working> -m … --author …` | Freeze the working
|
|
83
|
+
| `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
|
|
84
|
+
| `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
|
|
85
85
|
| `fdl push <working>` | Upload the committed version and move `main` to it |
|
|
86
86
|
| `fdl log <remote>` | List versions from `main` back to the first |
|
|
87
|
-
| `fdl status <working>` | What the working
|
|
87
|
+
| `fdl status <working>` | What the working metadata changed since the clone |
|
|
88
88
|
| `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
|
|
89
89
|
| `fdl show <remote> [version]` | A version's record, its tables and its changes |
|
|
90
|
-
| `fdl url <remote> [--attach]` | Print the
|
|
90
|
+
| `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
|
|
91
91
|
| `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
|
|
92
92
|
| `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
|
|
93
|
-
| `fdl merge <ref> <working> --author …` | Merge another branch into the working
|
|
94
|
-
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their
|
|
93
|
+
| `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
|
|
94
|
+
| `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
|
|
95
95
|
| `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
|
|
96
96
|
| `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
|
|
97
97
|
| `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
|
|
98
98
|
|
|
99
|
-
A push that is not built on what `main` holds now is refused:
|
|
99
|
+
A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
|
|
100
100
|
|
|
101
101
|
## Python
|
|
102
102
|
|
|
@@ -109,11 +109,11 @@ for version in repo.log(store):
|
|
|
109
109
|
print(version.seq, version.id, version.message)
|
|
110
110
|
```
|
|
111
111
|
|
|
112
|
-
`fdl.repo` has the operations without a terminal (`init`, `
|
|
112
|
+
`fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
|
|
113
113
|
|
|
114
114
|
## Earlier releases
|
|
115
115
|
|
|
116
|
-
Releases up to 0.11 were a different tool for managing a single frozen
|
|
116
|
+
Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
|
|
117
117
|
|
|
118
118
|
## License
|
|
119
119
|
|
|
@@ -24,5 +24,9 @@ export default defineConfig({
|
|
|
24
24
|
fonts: ["Noto Sans JP"],
|
|
25
25
|
},
|
|
26
26
|
},
|
|
27
|
-
redirects: [
|
|
27
|
+
redirects: [
|
|
28
|
+
{ from: "/cli", to: "/reference/cli" },
|
|
29
|
+
{ from: "/concepts/working-catalogs", to: "/concepts/working-metadata" },
|
|
30
|
+
{ from: "/ja/concepts/working-catalogs", to: "/ja/concepts/working-metadata" },
|
|
31
|
+
],
|
|
28
32
|
});
|
|
@@ -433,6 +433,110 @@
|
|
|
433
433
|
"assets/abc": "2026-11-01T00:00:00Z"
|
|
434
434
|
},
|
|
435
435
|
"deleted": []
|
|
436
|
+
},
|
|
437
|
+
{
|
|
438
|
+
"name": "leaves alone the files of a former data_dir",
|
|
439
|
+
"files": {
|
|
440
|
+
"fdl.json": {
|
|
441
|
+
"fdl": "0.1",
|
|
442
|
+
"data_dir": "data/"
|
|
443
|
+
},
|
|
444
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
445
|
+
"fdl": "0.1",
|
|
446
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
447
|
+
"seq": 1,
|
|
448
|
+
"parents": [],
|
|
449
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
450
|
+
"author": {
|
|
451
|
+
"name": "example"
|
|
452
|
+
},
|
|
453
|
+
"message": "",
|
|
454
|
+
"catalog": {
|
|
455
|
+
"path": "ducklake.duckdb",
|
|
456
|
+
"storage": "duckdb-v1.5.0",
|
|
457
|
+
"bytes": 0,
|
|
458
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
459
|
+
},
|
|
460
|
+
"ducklake": {
|
|
461
|
+
"format": "1.0",
|
|
462
|
+
"snapshot_id": 1,
|
|
463
|
+
"snapshots": [
|
|
464
|
+
1
|
|
465
|
+
]
|
|
466
|
+
},
|
|
467
|
+
"files": {
|
|
468
|
+
"path": "files.json",
|
|
469
|
+
"bytes": 0,
|
|
470
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
471
|
+
}
|
|
472
|
+
},
|
|
473
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
474
|
+
"tables": {
|
|
475
|
+
"main.t": {
|
|
476
|
+
"files": []
|
|
477
|
+
}
|
|
478
|
+
},
|
|
479
|
+
"named": []
|
|
480
|
+
}
|
|
481
|
+
},
|
|
482
|
+
"objects": {
|
|
483
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
484
|
+
"ducklake.duckdb.files/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
|
|
485
|
+
},
|
|
486
|
+
"deleted": [
|
|
487
|
+
"data/main/t/ducklake-a.parquet"
|
|
488
|
+
]
|
|
489
|
+
},
|
|
490
|
+
{
|
|
491
|
+
"name": "deletes nothing while data_dir is not a folder under the root",
|
|
492
|
+
"files": {
|
|
493
|
+
"fdl.json": {
|
|
494
|
+
"fdl": "0.1",
|
|
495
|
+
"data_dir": "data"
|
|
496
|
+
},
|
|
497
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
498
|
+
"fdl": "0.1",
|
|
499
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
500
|
+
"seq": 1,
|
|
501
|
+
"parents": [],
|
|
502
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
503
|
+
"author": {
|
|
504
|
+
"name": "example"
|
|
505
|
+
},
|
|
506
|
+
"message": "",
|
|
507
|
+
"catalog": {
|
|
508
|
+
"path": "ducklake.duckdb",
|
|
509
|
+
"storage": "duckdb-v1.5.0",
|
|
510
|
+
"bytes": 0,
|
|
511
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
512
|
+
},
|
|
513
|
+
"ducklake": {
|
|
514
|
+
"format": "1.0",
|
|
515
|
+
"snapshot_id": 1,
|
|
516
|
+
"snapshots": [
|
|
517
|
+
1
|
|
518
|
+
]
|
|
519
|
+
},
|
|
520
|
+
"files": {
|
|
521
|
+
"path": "files.json",
|
|
522
|
+
"bytes": 0,
|
|
523
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
524
|
+
}
|
|
525
|
+
},
|
|
526
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
527
|
+
"tables": {
|
|
528
|
+
"main.t": {
|
|
529
|
+
"files": []
|
|
530
|
+
}
|
|
531
|
+
},
|
|
532
|
+
"named": []
|
|
533
|
+
}
|
|
534
|
+
},
|
|
535
|
+
"objects": {
|
|
536
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
537
|
+
"datamain/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
|
|
538
|
+
},
|
|
539
|
+
"deleted": []
|
|
436
540
|
}
|
|
437
541
|
]
|
|
438
542
|
}
|
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
{
|
|
2
|
+
"cases": [
|
|
3
|
+
{
|
|
4
|
+
"name": "absent: the default",
|
|
5
|
+
"fdl_json": {
|
|
6
|
+
"fdl": "0.1"
|
|
7
|
+
},
|
|
8
|
+
"data_dir": "data/"
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"name": "the default, written out",
|
|
12
|
+
"fdl_json": {
|
|
13
|
+
"fdl": "0.1",
|
|
14
|
+
"data_dir": "data/"
|
|
15
|
+
},
|
|
16
|
+
"data_dir": "data/"
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"name": "dots inside a segment",
|
|
20
|
+
"fdl_json": {
|
|
21
|
+
"fdl": "0.1",
|
|
22
|
+
"data_dir": "ducklake.duckdb.files/"
|
|
23
|
+
},
|
|
24
|
+
"data_dir": "ducklake.duckdb.files/"
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"name": "more than one segment",
|
|
28
|
+
"fdl_json": {
|
|
29
|
+
"fdl": "0.1",
|
|
30
|
+
"data_dir": "lake/data/"
|
|
31
|
+
},
|
|
32
|
+
"data_dir": "lake/data/"
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"name": "digits, _ and -",
|
|
36
|
+
"fdl_json": {
|
|
37
|
+
"fdl": "0.1",
|
|
38
|
+
"data_dir": "data_2-x/"
|
|
39
|
+
},
|
|
40
|
+
"data_dir": "data_2-x/"
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"name": "no trailing slash",
|
|
44
|
+
"fdl_json": {
|
|
45
|
+
"fdl": "0.1",
|
|
46
|
+
"data_dir": "data"
|
|
47
|
+
},
|
|
48
|
+
"data_dir": null
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
"name": "empty",
|
|
52
|
+
"fdl_json": {
|
|
53
|
+
"fdl": "0.1",
|
|
54
|
+
"data_dir": ""
|
|
55
|
+
},
|
|
56
|
+
"data_dir": null
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"name": "the root itself",
|
|
60
|
+
"fdl_json": {
|
|
61
|
+
"fdl": "0.1",
|
|
62
|
+
"data_dir": "/"
|
|
63
|
+
},
|
|
64
|
+
"data_dir": null
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
"name": "starts with a slash",
|
|
68
|
+
"fdl_json": {
|
|
69
|
+
"fdl": "0.1",
|
|
70
|
+
"data_dir": "/data/"
|
|
71
|
+
},
|
|
72
|
+
"data_dir": null
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
"name": "an empty segment",
|
|
76
|
+
"fdl_json": {
|
|
77
|
+
"fdl": "0.1",
|
|
78
|
+
"data_dir": "data//x/"
|
|
79
|
+
},
|
|
80
|
+
"data_dir": null
|
|
81
|
+
},
|
|
82
|
+
{
|
|
83
|
+
"name": "a dot segment",
|
|
84
|
+
"fdl_json": {
|
|
85
|
+
"fdl": "0.1",
|
|
86
|
+
"data_dir": "./data/"
|
|
87
|
+
},
|
|
88
|
+
"data_dir": null
|
|
89
|
+
},
|
|
90
|
+
{
|
|
91
|
+
"name": "a parent segment",
|
|
92
|
+
"fdl_json": {
|
|
93
|
+
"fdl": "0.1",
|
|
94
|
+
"data_dir": "../data/"
|
|
95
|
+
},
|
|
96
|
+
"data_dir": null
|
|
97
|
+
},
|
|
98
|
+
{
|
|
99
|
+
"name": "a parent segment inside",
|
|
100
|
+
"fdl_json": {
|
|
101
|
+
"fdl": "0.1",
|
|
102
|
+
"data_dir": "data/../refs/"
|
|
103
|
+
},
|
|
104
|
+
"data_dir": null
|
|
105
|
+
},
|
|
106
|
+
{
|
|
107
|
+
"name": "dots alone",
|
|
108
|
+
"fdl_json": {
|
|
109
|
+
"fdl": "0.1",
|
|
110
|
+
"data_dir": ".../"
|
|
111
|
+
},
|
|
112
|
+
"data_dir": null
|
|
113
|
+
},
|
|
114
|
+
{
|
|
115
|
+
"name": "percent-encoded dots",
|
|
116
|
+
"fdl_json": {
|
|
117
|
+
"fdl": "0.1",
|
|
118
|
+
"data_dir": "%2e%2e/"
|
|
119
|
+
},
|
|
120
|
+
"data_dir": null
|
|
121
|
+
},
|
|
122
|
+
{
|
|
123
|
+
"name": "an absolute URL",
|
|
124
|
+
"fdl_json": {
|
|
125
|
+
"fdl": "0.1",
|
|
126
|
+
"data_dir": "https://example.org/data/"
|
|
127
|
+
},
|
|
128
|
+
"data_dir": null
|
|
129
|
+
},
|
|
130
|
+
{
|
|
131
|
+
"name": "another storage",
|
|
132
|
+
"fdl_json": {
|
|
133
|
+
"fdl": "0.1",
|
|
134
|
+
"data_dir": "s3://bucket/data/"
|
|
135
|
+
},
|
|
136
|
+
"data_dir": null
|
|
137
|
+
},
|
|
138
|
+
{
|
|
139
|
+
"name": "a backslash",
|
|
140
|
+
"fdl_json": {
|
|
141
|
+
"fdl": "0.1",
|
|
142
|
+
"data_dir": "data\\x/"
|
|
143
|
+
},
|
|
144
|
+
"data_dir": null
|
|
145
|
+
},
|
|
146
|
+
{
|
|
147
|
+
"name": "a space",
|
|
148
|
+
"fdl_json": {
|
|
149
|
+
"fdl": "0.1",
|
|
150
|
+
"data_dir": "data files/"
|
|
151
|
+
},
|
|
152
|
+
"data_dir": null
|
|
153
|
+
},
|
|
154
|
+
{
|
|
155
|
+
"name": "upper case",
|
|
156
|
+
"fdl_json": {
|
|
157
|
+
"fdl": "0.1",
|
|
158
|
+
"data_dir": "Data/"
|
|
159
|
+
},
|
|
160
|
+
"data_dir": null
|
|
161
|
+
},
|
|
162
|
+
{
|
|
163
|
+
"name": "non-ASCII",
|
|
164
|
+
"fdl_json": {
|
|
165
|
+
"fdl": "0.1",
|
|
166
|
+
"data_dir": "データ/"
|
|
167
|
+
},
|
|
168
|
+
"data_dir": null
|
|
169
|
+
},
|
|
170
|
+
{
|
|
171
|
+
"name": "versions",
|
|
172
|
+
"fdl_json": {
|
|
173
|
+
"fdl": "0.1",
|
|
174
|
+
"data_dir": "versions/"
|
|
175
|
+
},
|
|
176
|
+
"data_dir": null
|
|
177
|
+
},
|
|
178
|
+
{
|
|
179
|
+
"name": "under refs",
|
|
180
|
+
"fdl_json": {
|
|
181
|
+
"fdl": "0.1",
|
|
182
|
+
"data_dir": "refs/x/"
|
|
183
|
+
},
|
|
184
|
+
"data_dir": null
|
|
185
|
+
},
|
|
186
|
+
{
|
|
187
|
+
"name": "assets",
|
|
188
|
+
"fdl_json": {
|
|
189
|
+
"fdl": "0.1",
|
|
190
|
+
"data_dir": "assets/"
|
|
191
|
+
},
|
|
192
|
+
"data_dir": null
|
|
193
|
+
},
|
|
194
|
+
{
|
|
195
|
+
"name": "not a string",
|
|
196
|
+
"fdl_json": {
|
|
197
|
+
"fdl": "0.1",
|
|
198
|
+
"data_dir": 1
|
|
199
|
+
},
|
|
200
|
+
"data_dir": null
|
|
201
|
+
},
|
|
202
|
+
{
|
|
203
|
+
"name": "null",
|
|
204
|
+
"fdl_json": {
|
|
205
|
+
"fdl": "0.1",
|
|
206
|
+
"data_dir": null
|
|
207
|
+
},
|
|
208
|
+
"data_dir": null
|
|
209
|
+
}
|
|
210
|
+
]
|
|
211
|
+
}
|
|
@@ -9,11 +9,11 @@ A repository is everything under one prefix of a store: `s3://bucket/lake/` on S
|
|
|
9
9
|
|
|
10
10
|
| Operation | Used by | For |
|
|
11
11
|
| --- | --- | --- |
|
|
12
|
-
| read an object | readers and writers | refs, `version.json`,
|
|
13
|
-
| write an object | writers | data files,
|
|
12
|
+
| read an object | readers and writers | refs, `version.json`, metadata files, data files |
|
|
13
|
+
| write an object | writers | data files, metadata files, `version.json` |
|
|
14
14
|
| write on condition | writers | refs: create only if absent, replace only if unchanged |
|
|
15
15
|
| list a prefix | maintenance | finding data files that no version names |
|
|
16
|
-
| delete | maintenance |
|
|
16
|
+
| delete | maintenance | metadata files of versions past retention, and unused data files |
|
|
17
17
|
|
|
18
18
|
Conditional writes are what make two writers safe. A ref is created with `If-None-Match: *` and moved with `If-Match: <the ETag that was read>`. If someone else moved it in between, the store refuses the write and nothing changes. S3 and R2 support this. The folder store checks the condition without a lock, so it suits one writer at a time (see [Stores](/guides/stores)). A store without conditional writes can still be read from, but must not be pushed to: the last writer would silently replace someone else's version.
|
|
19
19
|
|
|
@@ -32,7 +32,7 @@ Readers only read. A static HTTP server in front of the repository is enough for
|
|
|
32
32
|
assets/<sha256> files a version points at, such as PDFs, images and its README
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
The repository holds no
|
|
35
|
+
The repository holds no metadata outside a version folder, and never a working metadata file. Either would hand readers a state that was never published.
|
|
36
36
|
|
|
37
37
|
`fdl.json` carries an `id`, a UUID written once by `fdl init`. Copies of the repository on other hosts keep the same `id`, so a tool can tell that two URLs hold the same repository. A service that hosts repositories may write its own ID for the dataset there.
|
|
38
38
|
|