fdl-cli 0.26.0__tar.gz → 0.28.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (113) hide show
  1. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CHANGELOG.md +57 -0
  2. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CONTRIBUTING.md +1 -1
  3. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/PKG-INFO +19 -19
  4. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/README.md +18 -18
  5. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/blume.config.ts +5 -1
  6. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/cases/collection.json +104 -0
  7. fdl_cli-0.28.0/cases/data-dir.json +211 -0
  8. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/meta.ts +1 -1
  9. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/repositories.md +4 -4
  10. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/concepts/versions-and-refs.md +8 -8
  11. fdl_cli-0.28.0/docs/concepts/working-metadata.md +49 -0
  12. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/getting-started.md +16 -16
  13. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/building-a-host.md +2 -2
  14. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/mirrors.md +6 -6
  15. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/public-and-private.md +5 -5
  16. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/reading-without-duckdb.md +1 -1
  17. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/serving-over-http.md +6 -6
  18. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/index.md +8 -8
  19. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/ducklake-formats.md +15 -15
  20. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/from-write-to-read.md +23 -23
  21. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/inside-a-repository.md +18 -18
  22. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/meta.ts +1 -1
  23. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/repositories.md +4 -4
  24. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/concepts/versions-and-refs.md +11 -11
  25. fdl_cli-0.28.0/docs/ja/concepts/working-metadata.md +49 -0
  26. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/getting-started.md +16 -16
  27. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/building-a-host.md +2 -2
  28. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/mirrors.md +6 -6
  29. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/public-and-private.md +5 -5
  30. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/reading-without-duckdb.md +1 -1
  31. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/serving-over-http.md +6 -6
  32. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/index.md +8 -8
  33. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/ducklake-formats.md +15 -15
  34. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/from-write-to-read.md +24 -24
  35. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/inside-a-repository.md +19 -19
  36. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/cli.md +29 -29
  37. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/python.md +11 -11
  38. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/collection-cases.md +1 -1
  39. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/data-files.md +15 -11
  40. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/http.md +6 -6
  41. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/index.md +7 -6
  42. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/layout.md +28 -9
  43. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/reading.md +14 -14
  44. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/refs.md +1 -1
  45. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/versions.md +48 -46
  46. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/writing.md +23 -21
  47. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/stability.md +3 -3
  48. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/cli.md +29 -29
  49. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/python.md +11 -11
  50. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/collection-cases.md +1 -1
  51. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/data-files.md +15 -11
  52. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/http.md +5 -5
  53. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/index.md +7 -6
  54. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/layout.md +28 -9
  55. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/reading.md +13 -13
  56. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/versions.md +47 -45
  57. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/writing.md +23 -21
  58. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/stability.md +3 -3
  59. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/pyproject.toml +1 -1
  60. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/cli.py +77 -45
  61. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/diff.py +3 -3
  62. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/layout.py +32 -6
  63. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/repo.py +70 -70
  64. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/version.py +16 -13
  65. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_cli.py +85 -22
  66. fdl_cli-0.28.0/tests/test_data_dir_cases.py +24 -0
  67. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_http.py +14 -14
  68. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_merge.py +4 -4
  69. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_objects.py +1 -1
  70. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_profiles.py +1 -1
  71. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_repo.py +28 -14
  72. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/uv.lock +1 -1
  73. fdl_cli-0.26.0/docs/concepts/working-catalogs.md +0 -49
  74. fdl_cli-0.26.0/docs/ja/concepts/working-catalogs.md +0 -49
  75. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/docs.yml +0 -0
  76. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/publish.yml +0 -0
  77. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/release.yml +0 -0
  78. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.github/workflows/test.yml +0 -0
  79. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.gitignore +0 -0
  80. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.node-version +0 -0
  81. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/.python-version +0 -0
  82. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/CLAUDE.md +0 -0
  83. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/LICENSE +0 -0
  84. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/SECURITY.md +0 -0
  85. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/meta.ts +0 -0
  86. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/guides/stores.md +0 -0
  87. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/internals/meta.ts +0 -0
  88. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/meta.ts +0 -0
  89. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/guides/stores.md +0 -0
  90. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/internals/meta.ts +0 -0
  91. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/reference/meta.ts +0 -0
  92. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/ja/spec/meta.ts +0 -0
  93. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/reference/meta.ts +0 -0
  94. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/meta.ts +0 -0
  95. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/docs/spec/refs.md +0 -0
  96. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/package-lock.json +0 -0
  97. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/package.json +0 -0
  98. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/public/CNAME +0 -0
  99. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/__init__.py +0 -0
  100. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/catalog.py +0 -0
  101. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/dataset.py +0 -0
  102. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/lineage.py +0 -0
  103. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/merge.py +0 -0
  104. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/objects.py +0 -0
  105. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/profiles.py +0 -0
  106. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/py.typed +0 -0
  107. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/refs.py +0 -0
  108. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/store.py +0 -0
  109. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/src/fdl/stores.py +0 -0
  110. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/http_repo.py +0 -0
  111. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_collection_cases.py +0 -0
  112. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_dataset.py +0 -0
  113. {fdl_cli-0.26.0 → fdl_cli-0.28.0}/tests/test_stores.py +0 -0
@@ -1,6 +1,63 @@
1
1
  # CHANGELOG
2
2
 
3
3
 
4
+ ## v0.28.0 (2026-11-15)
5
+
6
+ ### Bug Fixes
7
+
8
+ - Refuse a data_dir that is not a folder under the root, and a build checked out before data_dir
9
+ moved ([#62](https://github.com/queria-io/fdl/pull/62),
10
+ [`fba88fc`](https://github.com/queria-io/fdl/commit/fba88fcc54b4cb0ad7106edb4b99009d4f6a9f43))
11
+
12
+ - The spec states what data_dir may be (segments of [a-z0-9_.-], ending in /, not under versions/,
13
+ refs/ or assets/), that it is used as written and never corrected, and cases/data-dir.json lists
14
+ values every implementation runs - read_config refuses other values, so "data" no longer writes
15
+ datamain/... while other readers read data/main/... - Layout describes how a host changes
16
+ data_dir; Data files says files outside data_dir are reached by no version and are the host's to
17
+ delete - checkout remembers data_dir, and commit and push refuse when fdl.json names another one,
18
+ saying so, instead of failing on missing data files - Two collection cases: a former data_dir is
19
+ left alone, and an invalid data_dir deletes nothing
20
+
21
+ BREAKING CHANGE: a repository whose data_dir breaks the rules is refused instead of read.
22
+
23
+ ### Features
24
+
25
+ - Rename fdl checkout to fdl clone ([#63](https://github.com/queria-io/fdl/pull/63),
26
+ [`e4213e8`](https://github.com/queria-io/fdl/commit/e4213e8b6c9499367004e2fba9deaf100b49d94a))
27
+
28
+ ### Breaking Changes
29
+
30
+ - A repository whose data_dir breaks the rules is refused instead of read.
31
+
32
+
33
+ ## v0.27.0 (2026-11-15)
34
+
35
+ ### Features
36
+
37
+ - Name the frozen DuckLake metadata metadata.db and record it as metadata
38
+ ([#61](https://github.com/queria-io/fdl/pull/61),
39
+ [`babd57f`](https://github.com/queria-io/fdl/commit/babd57fe57a45468019f00042e9d52f011afe15d))
40
+
41
+ * feat!: name the frozen DuckLake metadata metadata.db and record it as metadata
42
+
43
+ A version's metadata file is now versions/<ID>/metadata.db, recorded under "metadata" in
44
+ version.json; other DuckLake formats go under metadata/, and fdl.json declares them as
45
+ metadata_formats. Readers open the name the record gives, so versions written by 0.26 and earlier
46
+ (ducklake.duckdb, recorded as "catalog") are still read.
47
+
48
+ BREAKING CHANGE: version.json records the metadata file as "metadata", and fdl.json's
49
+ catalog_formats is now metadata_formats.
50
+
51
+ * docs: redirect the old working catalog page
52
+
53
+ * fix(expire): also drop the other formats that fdl-cli 0.26 and earlier kept under catalogs/
54
+
55
+ ### Breaking Changes
56
+
57
+ - Version.json records the metadata file as "metadata", and fdl.json's catalog_formats is now
58
+ metadata_formats.
59
+
60
+
4
61
  ## v0.26.0 (2026-11-14)
5
62
 
6
63
  ### Features
@@ -27,7 +27,7 @@ src/fdl/
27
27
  ├── dataset.py — dataset.json: reading the description and checking it against the catalog
28
28
  ├── objects.py — assets and tree.json
29
29
  ├── profiles.py — the profile of each table
30
- ├── repo.py — init, checkout, commit, push, log, expire, withdraw
30
+ ├── repo.py — init, clone, commit, push, log, expire, withdraw
31
31
  ├── store.py — what a store has to provide
32
32
  ├── stores.py — a folder, S3 and a host over HTTP as stores
33
33
  └── cli.py — the fdl command
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: fdl-cli
3
- Version: 0.26.0
3
+ Version: 0.28.0
4
4
  Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
5
5
  Project-URL: Homepage, https://fdl.queria.io/
6
6
  Project-URL: Documentation, https://fdl.queria.io/
@@ -24,9 +24,9 @@ Description-Content-Type: text/markdown
24
24
  [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/fdl-cli/)
25
25
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
26
26
 
27
- Versions of a [DuckLake](https://ducklake.select/) catalog on object storage, kept the way Git keeps commits.
27
+ Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
28
28
 
29
- FDL is a storage layout and a tool for it. Each version is a frozen catalog with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
29
+ FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
30
30
 
31
31
  ```
32
32
  <root>/
@@ -35,7 +35,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
35
35
  refs/heads/main 019a3c4e-…
36
36
  versions/019a3c4e-…/
37
37
  version.json parents, author, message, sha256
38
- ducklake.duckdb the frozen catalog
38
+ metadata.db the frozen metadata
39
39
  data/main/towns/ducklake-….parquet
40
40
  ```
41
41
 
@@ -47,14 +47,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
47
47
  pip install 'fdl-cli[s3]'
48
48
 
49
49
  fdl init s3://my-bucket/towns/
50
- fdl checkout s3://my-bucket/towns/ work/lake.sqlite
50
+ fdl clone s3://my-bucket/towns/ work/lake.sqlite
51
51
  ```
52
52
 
53
53
  The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
54
54
 
55
- `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working catalog that `checkout` makes, a SQLite file.
55
+ `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
56
56
 
57
- Write to the working catalog with DuckDB, attached the way `checkout` prints:
57
+ Write to the working metadata with DuckDB, attached the way `clone` prints:
58
58
 
59
59
  ```sql
60
60
  ATTACH 'ducklake:work/lake.sqlite' AS lake (
@@ -64,9 +64,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
64
64
  CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
65
65
  ```
66
66
 
67
- Rows have to go to Parquet, not into the catalog (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
67
+ Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
68
68
 
69
- Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the catalog:
69
+ Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
70
70
 
71
71
  ```yaml
72
72
  name: towns
@@ -88,7 +88,7 @@ fdl url s3://my-bucket/towns/
88
88
  ```
89
89
 
90
90
  ```sql
91
- ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
91
+ ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
92
92
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
93
93
  );
94
94
  ```
@@ -100,23 +100,23 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
100
100
  | Command | What it does |
101
101
  | --- | --- |
102
102
  | `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
103
- | `fdl checkout <remote> <working>` | Make a working catalog from the version `main` holds |
104
- | `fdl commit <working> -m … --author …` | Freeze the working catalog into a version |
103
+ | `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
104
+ | `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
105
105
  | `fdl push <working>` | Upload the committed version and move `main` to it |
106
106
  | `fdl log <remote>` | List versions from `main` back to the first |
107
- | `fdl status <working>` | What the working catalog changed since the checkout |
107
+ | `fdl status <working>` | What the working metadata changed since the clone |
108
108
  | `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
109
109
  | `fdl show <remote> [version]` | A version's record, its tables and its changes |
110
- | `fdl url <remote> [--attach]` | Print the catalog URL of the version `main` holds, or the whole `ATTACH` statement |
110
+ | `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
111
111
  | `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
112
112
  | `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
113
- | `fdl merge <ref> <working> --author …` | Merge another branch into the working catalog's branch, rows matched by primary key |
114
- | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their catalogs |
113
+ | `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
114
+ | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
115
115
  | `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
116
116
  | `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
117
117
  | `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
118
118
 
119
- A push that is not built on what `main` holds now is refused: check out again and redo the work, or work on a branch and merge it.
119
+ A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
120
120
 
121
121
  ## Python
122
122
 
@@ -129,11 +129,11 @@ for version in repo.log(store):
129
129
  print(version.seq, version.id, version.message)
130
130
  ```
131
131
 
132
- `fdl.repo` has the operations without a terminal (`init`, `checkout`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
132
+ `fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
133
133
 
134
134
  ## Earlier releases
135
135
 
136
- Releases up to 0.11 were a different tool for managing a single frozen catalog, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
136
+ Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
137
137
 
138
138
  ## License
139
139
 
@@ -4,9 +4,9 @@
4
4
  [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/fdl-cli/)
5
5
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
6
6
 
7
- Versions of a [DuckLake](https://ducklake.select/) catalog on object storage, kept the way Git keeps commits.
7
+ Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
8
8
 
9
- FDL is a storage layout and a tool for it. Each version is a frozen catalog with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
9
+ FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
10
10
 
11
11
  ```
12
12
  <root>/
@@ -15,7 +15,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
15
15
  refs/heads/main 019a3c4e-…
16
16
  versions/019a3c4e-…/
17
17
  version.json parents, author, message, sha256
18
- ducklake.duckdb the frozen catalog
18
+ metadata.db the frozen metadata
19
19
  data/main/towns/ducklake-….parquet
20
20
  ```
21
21
 
@@ -27,14 +27,14 @@ The documentation and the specification are at [fdl.queria.io](https://fdl.queri
27
27
  pip install 'fdl-cli[s3]'
28
28
 
29
29
  fdl init s3://my-bucket/towns/
30
- fdl checkout s3://my-bucket/towns/ work/lake.sqlite
30
+ fdl clone s3://my-bucket/towns/ work/lake.sqlite
31
31
  ```
32
32
 
33
33
  The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
34
34
 
35
- `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working catalog that `checkout` makes, a SQLite file.
35
+ `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `clone` makes, a SQLite file.
36
36
 
37
- Write to the working catalog with DuckDB, attached the way `checkout` prints:
37
+ Write to the working metadata with DuckDB, attached the way `clone` prints:
38
38
 
39
39
  ```sql
40
40
  ATTACH 'ducklake:work/lake.sqlite' AS lake (
@@ -44,9 +44,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
44
44
  CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
45
45
  ```
46
46
 
47
- Rows have to go to Parquet, not into the catalog (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
47
+ Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
48
48
 
49
- Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the catalog:
49
+ Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
50
50
 
51
51
  ```yaml
52
52
  name: towns
@@ -68,7 +68,7 @@ fdl url s3://my-bucket/towns/
68
68
  ```
69
69
 
70
70
  ```sql
71
- ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
71
+ ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
72
72
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
73
73
  );
74
74
  ```
@@ -80,23 +80,23 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
80
80
  | Command | What it does |
81
81
  | --- | --- |
82
82
  | `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
83
- | `fdl checkout <remote> <working>` | Make a working catalog from the version `main` holds |
84
- | `fdl commit <working> -m … --author …` | Freeze the working catalog into a version |
83
+ | `fdl clone <remote> <working>` | Make a working metadata file from the version `main` holds |
84
+ | `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
85
85
  | `fdl push <working>` | Upload the committed version and move `main` to it |
86
86
  | `fdl log <remote>` | List versions from `main` back to the first |
87
- | `fdl status <working>` | What the working catalog changed since the checkout |
87
+ | `fdl status <working>` | What the working metadata changed since the clone |
88
88
  | `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
89
89
  | `fdl show <remote> [version]` | A version's record, its tables and its changes |
90
- | `fdl url <remote> [--attach]` | Print the catalog URL of the version `main` holds, or the whole `ATTACH` statement |
90
+ | `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
91
91
  | `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
92
92
  | `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
93
- | `fdl merge <ref> <working> --author …` | Merge another branch into the working catalog's branch, rows matched by primary key |
94
- | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their catalogs |
93
+ | `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
94
+ | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
95
95
  | `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
96
96
  | `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
97
97
  | `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
98
98
 
99
- A push that is not built on what `main` holds now is refused: check out again and redo the work, or work on a branch and merge it.
99
+ A push that is not built on what `main` holds now is refused: clone again and redo the work, or work on a branch and merge it.
100
100
 
101
101
  ## Python
102
102
 
@@ -109,11 +109,11 @@ for version in repo.log(store):
109
109
  print(version.seq, version.id, version.message)
110
110
  ```
111
111
 
112
- `fdl.repo` has the operations without a terminal (`init`, `checkout`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
112
+ `fdl.repo` has the operations without a terminal (`init`, `clone`, `commit`, `push`, `log`, `expire`, `prune`), for tools and services that build on FDL.
113
113
 
114
114
  ## Earlier releases
115
115
 
116
- Releases up to 0.11 were a different tool for managing a single frozen catalog, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
116
+ Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
117
117
 
118
118
  ## License
119
119
 
@@ -24,5 +24,9 @@ export default defineConfig({
24
24
  fonts: ["Noto Sans JP"],
25
25
  },
26
26
  },
27
- redirects: [{ from: "/cli", to: "/reference/cli" }],
27
+ redirects: [
28
+ { from: "/cli", to: "/reference/cli" },
29
+ { from: "/concepts/working-catalogs", to: "/concepts/working-metadata" },
30
+ { from: "/ja/concepts/working-catalogs", to: "/ja/concepts/working-metadata" },
31
+ ],
28
32
  });
@@ -433,6 +433,110 @@
433
433
  "assets/abc": "2026-11-01T00:00:00Z"
434
434
  },
435
435
  "deleted": []
436
+ },
437
+ {
438
+ "name": "leaves alone the files of a former data_dir",
439
+ "files": {
440
+ "fdl.json": {
441
+ "fdl": "0.1",
442
+ "data_dir": "data/"
443
+ },
444
+ "versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
445
+ "fdl": "0.1",
446
+ "id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
447
+ "seq": 1,
448
+ "parents": [],
449
+ "created_at": "2026-11-01T00:00:00Z",
450
+ "author": {
451
+ "name": "example"
452
+ },
453
+ "message": "",
454
+ "catalog": {
455
+ "path": "ducklake.duckdb",
456
+ "storage": "duckdb-v1.5.0",
457
+ "bytes": 0,
458
+ "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
459
+ },
460
+ "ducklake": {
461
+ "format": "1.0",
462
+ "snapshot_id": 1,
463
+ "snapshots": [
464
+ 1
465
+ ]
466
+ },
467
+ "files": {
468
+ "path": "files.json",
469
+ "bytes": 0,
470
+ "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
471
+ }
472
+ },
473
+ "versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
474
+ "tables": {
475
+ "main.t": {
476
+ "files": []
477
+ }
478
+ },
479
+ "named": []
480
+ }
481
+ },
482
+ "objects": {
483
+ "data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
484
+ "ducklake.duckdb.files/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
485
+ },
486
+ "deleted": [
487
+ "data/main/t/ducklake-a.parquet"
488
+ ]
489
+ },
490
+ {
491
+ "name": "deletes nothing while data_dir is not a folder under the root",
492
+ "files": {
493
+ "fdl.json": {
494
+ "fdl": "0.1",
495
+ "data_dir": "data"
496
+ },
497
+ "versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
498
+ "fdl": "0.1",
499
+ "id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
500
+ "seq": 1,
501
+ "parents": [],
502
+ "created_at": "2026-11-01T00:00:00Z",
503
+ "author": {
504
+ "name": "example"
505
+ },
506
+ "message": "",
507
+ "catalog": {
508
+ "path": "ducklake.duckdb",
509
+ "storage": "duckdb-v1.5.0",
510
+ "bytes": 0,
511
+ "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
512
+ },
513
+ "ducklake": {
514
+ "format": "1.0",
515
+ "snapshot_id": 1,
516
+ "snapshots": [
517
+ 1
518
+ ]
519
+ },
520
+ "files": {
521
+ "path": "files.json",
522
+ "bytes": 0,
523
+ "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
524
+ }
525
+ },
526
+ "versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
527
+ "tables": {
528
+ "main.t": {
529
+ "files": []
530
+ }
531
+ },
532
+ "named": []
533
+ }
534
+ },
535
+ "objects": {
536
+ "data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
537
+ "datamain/t/ducklake-a.parquet": "2026-11-01T00:00:00Z"
538
+ },
539
+ "deleted": []
436
540
  }
437
541
  ]
438
542
  }
@@ -0,0 +1,211 @@
1
+ {
2
+ "cases": [
3
+ {
4
+ "name": "absent: the default",
5
+ "fdl_json": {
6
+ "fdl": "0.1"
7
+ },
8
+ "data_dir": "data/"
9
+ },
10
+ {
11
+ "name": "the default, written out",
12
+ "fdl_json": {
13
+ "fdl": "0.1",
14
+ "data_dir": "data/"
15
+ },
16
+ "data_dir": "data/"
17
+ },
18
+ {
19
+ "name": "dots inside a segment",
20
+ "fdl_json": {
21
+ "fdl": "0.1",
22
+ "data_dir": "ducklake.duckdb.files/"
23
+ },
24
+ "data_dir": "ducklake.duckdb.files/"
25
+ },
26
+ {
27
+ "name": "more than one segment",
28
+ "fdl_json": {
29
+ "fdl": "0.1",
30
+ "data_dir": "lake/data/"
31
+ },
32
+ "data_dir": "lake/data/"
33
+ },
34
+ {
35
+ "name": "digits, _ and -",
36
+ "fdl_json": {
37
+ "fdl": "0.1",
38
+ "data_dir": "data_2-x/"
39
+ },
40
+ "data_dir": "data_2-x/"
41
+ },
42
+ {
43
+ "name": "no trailing slash",
44
+ "fdl_json": {
45
+ "fdl": "0.1",
46
+ "data_dir": "data"
47
+ },
48
+ "data_dir": null
49
+ },
50
+ {
51
+ "name": "empty",
52
+ "fdl_json": {
53
+ "fdl": "0.1",
54
+ "data_dir": ""
55
+ },
56
+ "data_dir": null
57
+ },
58
+ {
59
+ "name": "the root itself",
60
+ "fdl_json": {
61
+ "fdl": "0.1",
62
+ "data_dir": "/"
63
+ },
64
+ "data_dir": null
65
+ },
66
+ {
67
+ "name": "starts with a slash",
68
+ "fdl_json": {
69
+ "fdl": "0.1",
70
+ "data_dir": "/data/"
71
+ },
72
+ "data_dir": null
73
+ },
74
+ {
75
+ "name": "an empty segment",
76
+ "fdl_json": {
77
+ "fdl": "0.1",
78
+ "data_dir": "data//x/"
79
+ },
80
+ "data_dir": null
81
+ },
82
+ {
83
+ "name": "a dot segment",
84
+ "fdl_json": {
85
+ "fdl": "0.1",
86
+ "data_dir": "./data/"
87
+ },
88
+ "data_dir": null
89
+ },
90
+ {
91
+ "name": "a parent segment",
92
+ "fdl_json": {
93
+ "fdl": "0.1",
94
+ "data_dir": "../data/"
95
+ },
96
+ "data_dir": null
97
+ },
98
+ {
99
+ "name": "a parent segment inside",
100
+ "fdl_json": {
101
+ "fdl": "0.1",
102
+ "data_dir": "data/../refs/"
103
+ },
104
+ "data_dir": null
105
+ },
106
+ {
107
+ "name": "dots alone",
108
+ "fdl_json": {
109
+ "fdl": "0.1",
110
+ "data_dir": ".../"
111
+ },
112
+ "data_dir": null
113
+ },
114
+ {
115
+ "name": "percent-encoded dots",
116
+ "fdl_json": {
117
+ "fdl": "0.1",
118
+ "data_dir": "%2e%2e/"
119
+ },
120
+ "data_dir": null
121
+ },
122
+ {
123
+ "name": "an absolute URL",
124
+ "fdl_json": {
125
+ "fdl": "0.1",
126
+ "data_dir": "https://example.org/data/"
127
+ },
128
+ "data_dir": null
129
+ },
130
+ {
131
+ "name": "another storage",
132
+ "fdl_json": {
133
+ "fdl": "0.1",
134
+ "data_dir": "s3://bucket/data/"
135
+ },
136
+ "data_dir": null
137
+ },
138
+ {
139
+ "name": "a backslash",
140
+ "fdl_json": {
141
+ "fdl": "0.1",
142
+ "data_dir": "data\\x/"
143
+ },
144
+ "data_dir": null
145
+ },
146
+ {
147
+ "name": "a space",
148
+ "fdl_json": {
149
+ "fdl": "0.1",
150
+ "data_dir": "data files/"
151
+ },
152
+ "data_dir": null
153
+ },
154
+ {
155
+ "name": "upper case",
156
+ "fdl_json": {
157
+ "fdl": "0.1",
158
+ "data_dir": "Data/"
159
+ },
160
+ "data_dir": null
161
+ },
162
+ {
163
+ "name": "non-ASCII",
164
+ "fdl_json": {
165
+ "fdl": "0.1",
166
+ "data_dir": "データ/"
167
+ },
168
+ "data_dir": null
169
+ },
170
+ {
171
+ "name": "versions",
172
+ "fdl_json": {
173
+ "fdl": "0.1",
174
+ "data_dir": "versions/"
175
+ },
176
+ "data_dir": null
177
+ },
178
+ {
179
+ "name": "under refs",
180
+ "fdl_json": {
181
+ "fdl": "0.1",
182
+ "data_dir": "refs/x/"
183
+ },
184
+ "data_dir": null
185
+ },
186
+ {
187
+ "name": "assets",
188
+ "fdl_json": {
189
+ "fdl": "0.1",
190
+ "data_dir": "assets/"
191
+ },
192
+ "data_dir": null
193
+ },
194
+ {
195
+ "name": "not a string",
196
+ "fdl_json": {
197
+ "fdl": "0.1",
198
+ "data_dir": 1
199
+ },
200
+ "data_dir": null
201
+ },
202
+ {
203
+ "name": "null",
204
+ "fdl_json": {
205
+ "fdl": "0.1",
206
+ "data_dir": null
207
+ },
208
+ "data_dir": null
209
+ }
210
+ ]
211
+ }
@@ -3,5 +3,5 @@ import { defineMeta } from "blume";
3
3
  export default defineMeta({
4
4
  title: "Concepts",
5
5
  order: 2,
6
- pages: ["repositories", "versions-and-refs", "working-catalogs"],
6
+ pages: ["repositories", "versions-and-refs", "working-metadata"],
7
7
  });
@@ -9,11 +9,11 @@ A repository is everything under one prefix of a store: `s3://bucket/lake/` on S
9
9
 
10
10
  | Operation | Used by | For |
11
11
  | --- | --- | --- |
12
- | read an object | readers and writers | refs, `version.json`, catalogs, data files |
13
- | write an object | writers | data files, catalogs, `version.json` |
12
+ | read an object | readers and writers | refs, `version.json`, metadata files, data files |
13
+ | write an object | writers | data files, metadata files, `version.json` |
14
14
  | write on condition | writers | refs: create only if absent, replace only if unchanged |
15
15
  | list a prefix | maintenance | finding data files that no version names |
16
- | delete | maintenance | catalogs of versions past retention, and unused data files |
16
+ | delete | maintenance | metadata files of versions past retention, and unused data files |
17
17
 
18
18
  Conditional writes are what make two writers safe. A ref is created with `If-None-Match: *` and moved with `If-Match: <the ETag that was read>`. If someone else moved it in between, the store refuses the write and nothing changes. S3 and R2 support this. The folder store checks the condition without a lock, so it suits one writer at a time (see [Stores](/guides/stores)). A store without conditional writes can still be read from, but must not be pushed to: the last writer would silently replace someone else's version.
19
19
 
@@ -32,7 +32,7 @@ Readers only read. A static HTTP server in front of the repository is enough for
32
32
  assets/<sha256> files a version points at, such as PDFs, images and its README
33
33
  ```
34
34
 
35
- The repository holds no catalog outside a version folder, and never a working catalog. Either would hand readers a state that was never published.
35
+ The repository holds no metadata outside a version folder, and never a working metadata file. Either would hand readers a state that was never published.
36
36
 
37
37
  `fdl.json` carries an `id`, a UUID written once by `fdl init`. Copies of the repository on other hosts keep the same `id`, so a tool can tell that two URLs hold the same repository. A service that hosts repositories may write its own ID for the dataset there.
38
38