fdl-cli 0.25.0__tar.gz → 0.27.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (111) hide show
  1. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CHANGELOG.md +76 -0
  2. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/PKG-INFO +16 -16
  3. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/README.md +15 -15
  4. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/blume.config.ts +5 -1
  5. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/meta.ts +1 -1
  6. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/repositories.md +4 -4
  7. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/concepts/versions-and-refs.md +8 -8
  8. fdl_cli-0.27.0/docs/concepts/working-metadata.md +49 -0
  9. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/getting-started.md +13 -13
  10. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/building-a-host.md +1 -1
  11. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/mirrors.md +6 -6
  12. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/public-and-private.md +4 -4
  13. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/reading-without-duckdb.md +1 -1
  14. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/serving-over-http.md +6 -6
  15. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/index.md +8 -8
  16. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/ducklake-formats.md +15 -15
  17. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/from-write-to-read.md +22 -22
  18. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/inside-a-repository.md +16 -16
  19. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/meta.ts +1 -1
  20. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/repositories.md +4 -4
  21. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/concepts/versions-and-refs.md +8 -8
  22. fdl_cli-0.27.0/docs/ja/concepts/working-metadata.md +49 -0
  23. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/getting-started.md +13 -13
  24. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/building-a-host.md +1 -1
  25. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/mirrors.md +6 -6
  26. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/public-and-private.md +4 -4
  27. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/reading-without-duckdb.md +1 -1
  28. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/serving-over-http.md +6 -6
  29. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/index.md +8 -8
  30. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/ducklake-formats.md +15 -15
  31. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/from-write-to-read.md +22 -22
  32. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/inside-a-repository.md +16 -16
  33. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/cli.md +25 -25
  34. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/python.md +9 -9
  35. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/data-files.md +11 -11
  36. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/http.md +5 -5
  37. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/index.md +7 -6
  38. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/layout.md +9 -9
  39. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/reading.md +14 -14
  40. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/versions.md +49 -47
  41. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/writing.md +19 -19
  42. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/stability.md +3 -3
  43. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/cli.md +25 -25
  44. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/python.md +9 -9
  45. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/data-files.md +11 -11
  46. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/http.md +5 -5
  47. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/index.md +7 -6
  48. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/layout.md +9 -9
  49. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/reading.md +13 -13
  50. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/versions.md +48 -46
  51. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/writing.md +19 -19
  52. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/stability.md +3 -3
  53. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/pyproject.toml +1 -1
  54. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/cli.py +36 -35
  55. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/layout.py +15 -6
  56. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/profiles.py +28 -9
  57. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/repo.py +91 -67
  58. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/version.py +16 -13
  59. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_cli.py +1 -1
  60. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_http.py +4 -4
  61. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_merge.py +2 -2
  62. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_profiles.py +65 -0
  63. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_repo.py +24 -10
  64. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/uv.lock +1 -1
  65. fdl_cli-0.25.0/docs/concepts/working-catalogs.md +0 -49
  66. fdl_cli-0.25.0/docs/ja/concepts/working-catalogs.md +0 -49
  67. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/docs.yml +0 -0
  68. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/publish.yml +0 -0
  69. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/release.yml +0 -0
  70. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.github/workflows/test.yml +0 -0
  71. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.gitignore +0 -0
  72. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.node-version +0 -0
  73. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/.python-version +0 -0
  74. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CLAUDE.md +0 -0
  75. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/CONTRIBUTING.md +0 -0
  76. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/LICENSE +0 -0
  77. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/SECURITY.md +0 -0
  78. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/cases/collection.json +0 -0
  79. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/meta.ts +0 -0
  80. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/guides/stores.md +0 -0
  81. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/internals/meta.ts +0 -0
  82. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/meta.ts +0 -0
  83. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/guides/stores.md +0 -0
  84. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/internals/meta.ts +0 -0
  85. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/reference/meta.ts +0 -0
  86. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/collection-cases.md +0 -0
  87. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/meta.ts +0 -0
  88. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/ja/spec/refs.md +0 -0
  89. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/reference/meta.ts +0 -0
  90. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/collection-cases.md +0 -0
  91. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/meta.ts +0 -0
  92. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/docs/spec/refs.md +0 -0
  93. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/package-lock.json +0 -0
  94. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/package.json +0 -0
  95. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/public/CNAME +0 -0
  96. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/__init__.py +0 -0
  97. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/catalog.py +0 -0
  98. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/dataset.py +0 -0
  99. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/diff.py +0 -0
  100. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/lineage.py +0 -0
  101. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/merge.py +0 -0
  102. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/objects.py +0 -0
  103. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/py.typed +0 -0
  104. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/refs.py +0 -0
  105. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/store.py +0 -0
  106. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/src/fdl/stores.py +0 -0
  107. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/http_repo.py +0 -0
  108. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_collection_cases.py +0 -0
  109. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_dataset.py +0 -0
  110. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_objects.py +0 -0
  111. {fdl_cli-0.25.0 → fdl_cli-0.27.0}/tests/test_stores.py +0 -0
@@ -1,6 +1,82 @@
1
1
  # CHANGELOG
2
2
 
3
3
 
4
+ ## v0.27.0 (2026-11-15)
5
+
6
+ ### Features
7
+
8
+ - Name the frozen DuckLake metadata metadata.db and record it as metadata
9
+ ([#61](https://github.com/queria-io/fdl/pull/61),
10
+ [`babd57f`](https://github.com/queria-io/fdl/commit/babd57fe57a45468019f00042e9d52f011afe15d))
11
+
12
+ * feat!: name the frozen DuckLake metadata metadata.db and record it as metadata
13
+
14
+ A version's metadata file is now versions/<ID>/metadata.db, recorded under "metadata" in
15
+ version.json; other DuckLake formats go under metadata/, and fdl.json declares them as
16
+ metadata_formats. Readers open the name the record gives, so versions written by 0.26 and earlier
17
+ (ducklake.duckdb, recorded as "catalog") are still read.
18
+
19
+ BREAKING CHANGE: version.json records the metadata file as "metadata", and fdl.json's
20
+ catalog_formats is now metadata_formats.
21
+
22
+ * docs: redirect the old working catalog page
23
+
24
+ * fix(expire): also drop the other formats that fdl-cli 0.26 and earlier kept under catalogs/
25
+
26
+ ### Breaking Changes
27
+
28
+ - Version.json records the metadata file as "metadata", and fdl.json's catalog_formats is now
29
+ metadata_formats.
30
+
31
+
32
+ ## v0.26.0 (2026-11-14)
33
+
34
+ ### Features
35
+
36
+ - Profile views, and measure wide tables a hundred columns at a time
37
+ ([#60](https://github.com/queria-io/fdl/pull/60),
38
+ [`e6c3442`](https://github.com/queria-io/fdl/commit/e6c34429590eedfcee7b568f6c030b497c5ef668))
39
+
40
+ * feat!: profile tables with fdl profile instead of at commit
41
+
42
+ fdl commit no longer writes profiles. fdl profile <working> profiles the version committed there,
43
+ for fdl push to upload with it; fdl profile <remote> adds the profiles a pushed version lacks. The
44
+ spec lets a version get a missing profile later.
45
+
46
+ BREAKING CHANGE: fdl commit writes no profiles. Run fdl profile between fdl commit and fdl push to
47
+ keep writing them.
48
+
49
+ * feat: profile tables at push instead of commit
50
+
51
+ fdl push profiles the version it uploads (--no-profile skips it), so the steps stay commit and push.
52
+ fdl profile <remote> is left for versions pushed without profiles. Also warns of tables that could
53
+ not be read, lists the version's profiles in one request, and updates the spec wording.
54
+
55
+ * fix: keep fdl push working when profiling cannot read the working catalog
56
+
57
+ Read dataset.json and the working catalog inside the guarded block, so a missing or unreadable
58
+ catalog only warns. Profile again when a stopped push left some tables without profiles.
59
+
60
+ * feat: profile views, and measure wide tables a hundred columns at a time
61
+
62
+ A view's profile is measured from the rows its query returns, with the columns DESCRIBE gives. It
63
+ keeps the parent's profile only while its sql is the parent's and no table's data files changed.
64
+
65
+ The column statistics now run in queries of at most 100 columns. One query over hundreds of
66
+ count(DISTINCT) ran out of memory, as DuckDB holds a hash table for each at once.
67
+
68
+ * fix: re-profile a view when any view's query changed
69
+
70
+ A view may read another view, so a view keeps its parent's profile only when every view's sql and
71
+ every table's data files are the parent's. Fetch the parent's dataset.json only when there are
72
+ views, list more of DuckDB's temporal types, and describe view profiling in push --help.
73
+
74
+ ### Breaking Changes
75
+
76
+ - Fdl commit writes no profiles. Run fdl profile between fdl commit and fdl push to keep writing
77
+ them.
78
+
79
+
4
80
  ## v0.25.0 (2026-11-14)
5
81
 
6
82
  ### Bug Fixes
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: fdl-cli
3
- Version: 0.25.0
3
+ Version: 0.27.0
4
4
  Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
5
5
  Project-URL: Homepage, https://fdl.queria.io/
6
6
  Project-URL: Documentation, https://fdl.queria.io/
@@ -24,9 +24,9 @@ Description-Content-Type: text/markdown
24
24
  [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/fdl-cli/)
25
25
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
26
26
 
27
- Versions of a [DuckLake](https://ducklake.select/) catalog on object storage, kept the way Git keeps commits.
27
+ Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
28
28
 
29
- FDL is a storage layout and a tool for it. Each version is a frozen catalog with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
29
+ FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
30
30
 
31
31
  ```
32
32
  <root>/
@@ -35,7 +35,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
35
35
  refs/heads/main 019a3c4e-…
36
36
  versions/019a3c4e-…/
37
37
  version.json parents, author, message, sha256
38
- ducklake.duckdb the frozen catalog
38
+ metadata.db the frozen metadata
39
39
  data/main/towns/ducklake-….parquet
40
40
  ```
41
41
 
@@ -52,9 +52,9 @@ fdl checkout s3://my-bucket/towns/ work/lake.sqlite
52
52
 
53
53
  The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
54
54
 
55
- `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working catalog that `checkout` makes, a SQLite file.
55
+ `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `checkout` makes, a SQLite file.
56
56
 
57
- Write to the working catalog with DuckDB, attached the way `checkout` prints:
57
+ Write to the working metadata with DuckDB, attached the way `checkout` prints:
58
58
 
59
59
  ```sql
60
60
  ATTACH 'ducklake:work/lake.sqlite' AS lake (
@@ -64,9 +64,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
64
64
  CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
65
65
  ```
66
66
 
67
- Rows have to go to Parquet, not into the catalog (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
67
+ Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
68
68
 
69
- Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the catalog:
69
+ Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
70
70
 
71
71
  ```yaml
72
72
  name: towns
@@ -88,7 +88,7 @@ fdl url s3://my-bucket/towns/
88
88
  ```
89
89
 
90
90
  ```sql
91
- ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
91
+ ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
92
92
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
93
93
  );
94
94
  ```
@@ -100,18 +100,18 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
100
100
  | Command | What it does |
101
101
  | --- | --- |
102
102
  | `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
103
- | `fdl checkout <remote> <working>` | Make a working catalog from the version `main` holds |
104
- | `fdl commit <working> -m … --author …` | Freeze the working catalog into a version |
103
+ | `fdl checkout <remote> <working>` | Make a working metadata file from the version `main` holds |
104
+ | `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
105
105
  | `fdl push <working>` | Upload the committed version and move `main` to it |
106
106
  | `fdl log <remote>` | List versions from `main` back to the first |
107
- | `fdl status <working>` | What the working catalog changed since the checkout |
107
+ | `fdl status <working>` | What the working metadata changed since the checkout |
108
108
  | `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
109
109
  | `fdl show <remote> [version]` | A version's record, its tables and its changes |
110
- | `fdl url <remote> [--attach]` | Print the catalog URL of the version `main` holds, or the whole `ATTACH` statement |
110
+ | `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
111
111
  | `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
112
112
  | `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
113
- | `fdl merge <ref> <working> --author …` | Merge another branch into the working catalog's branch, rows matched by primary key |
114
- | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their catalogs |
113
+ | `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
114
+ | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
115
115
  | `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
116
116
  | `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
117
117
  | `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
@@ -133,7 +133,7 @@ for version in repo.log(store):
133
133
 
134
134
  ## Earlier releases
135
135
 
136
- Releases up to 0.11 were a different tool for managing a single frozen catalog, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
136
+ Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
137
137
 
138
138
  ## License
139
139
 
@@ -4,9 +4,9 @@
4
4
  [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](https://pypi.org/project/fdl-cli/)
5
5
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
6
6
 
7
- Versions of a [DuckLake](https://ducklake.select/) catalog on object storage, kept the way Git keeps commits.
7
+ Versions of [DuckLake](https://ducklake.select/) metadata on object storage, kept the way Git keeps commits.
8
8
 
9
- FDL is a storage layout and a tool for it. Each version is a frozen catalog with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
9
+ FDL is a storage layout and a tool for it. Each version is a frozen metadata file with a record of its parent, author and message; refs such as `refs/heads/main` say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.
10
10
 
11
11
  ```
12
12
  <root>/
@@ -15,7 +15,7 @@ FDL is a storage layout and a tool for it. Each version is a frozen catalog with
15
15
  refs/heads/main 019a3c4e-…
16
16
  versions/019a3c4e-…/
17
17
  version.json parents, author, message, sha256
18
- ducklake.duckdb the frozen catalog
18
+ metadata.db the frozen metadata
19
19
  data/main/towns/ducklake-….parquet
20
20
  ```
21
21
 
@@ -32,9 +32,9 @@ fdl checkout s3://my-bucket/towns/ work/lake.sqlite
32
32
 
33
33
  The package is `fdl-cli` on PyPI. It installs the `fdl` command and the `fdl` Python package (`import fdl`).
34
34
 
35
- `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working catalog that `checkout` makes, a SQLite file.
35
+ `fdl init` makes the repository in the store itself, as `git init --bare` does on a server. Your own copy to write to is the working metadata that `checkout` makes, a SQLite file.
36
36
 
37
- Write to the working catalog with DuckDB, attached the way `checkout` prints:
37
+ Write to the working metadata with DuckDB, attached the way `checkout` prints:
38
38
 
39
39
  ```sql
40
40
  ATTACH 'ducklake:work/lake.sqlite' AS lake (
@@ -44,9 +44,9 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
44
44
  CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';
45
45
  ```
46
46
 
47
- Rows have to go to Parquet, not into the catalog (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
47
+ Rows have to go to Parquet, not into the metadata (`DATA_INLINING_ROW_LIMIT 0`): a version is read over HTTP, and inlined rows would not be in it.
48
48
 
49
- Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the catalog:
49
+ Describe what the version holds in `dataset.yml`, next to where you run `fdl commit`. Only `name` and `title` are required; `fdl commit` fills in each table's columns and row count from the metadata:
50
50
 
51
51
  ```yaml
52
52
  name: towns
@@ -68,7 +68,7 @@ fdl url s3://my-bucket/towns/
68
68
  ```
69
69
 
70
70
  ```sql
71
- ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
71
+ ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/metadata.db' AS towns (
72
72
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
73
73
  );
74
74
  ```
@@ -80,18 +80,18 @@ Credentials and the endpoint come from boto3's usual places (`AWS_*` environment
80
80
  | Command | What it does |
81
81
  | --- | --- |
82
82
  | `fdl init <remote>` | Start a repository (`fdl.json` and `HEAD`) |
83
- | `fdl checkout <remote> <working>` | Make a working catalog from the version `main` holds |
84
- | `fdl commit <working> -m … --author …` | Freeze the working catalog into a version |
83
+ | `fdl checkout <remote> <working>` | Make a working metadata file from the version `main` holds |
84
+ | `fdl commit <working> -m … --author …` | Freeze the working metadata into a version |
85
85
  | `fdl push <working>` | Upload the committed version and move `main` to it |
86
86
  | `fdl log <remote>` | List versions from `main` back to the first |
87
- | `fdl status <working>` | What the working catalog changed since the checkout |
87
+ | `fdl status <working>` | What the working metadata changed since the checkout |
88
88
  | `fdl diff <remote> [version]` | What a version changed since its parent: tables, rows, columns, data files, descriptions |
89
89
  | `fdl show <remote> [version]` | A version's record, its tables and its changes |
90
- | `fdl url <remote> [--attach]` | Print the catalog URL of the version `main` holds, or the whole `ATTACH` statement |
90
+ | `fdl url <remote> [--attach]` | Print the metadata URL of the version `main` holds, or the whole `ATTACH` statement |
91
91
  | `fdl tag <remote> [name] [version]` | List tags, tag a version, or delete a tag (`-d`) |
92
92
  | `fdl branch <remote> [name] [version]` | List branches, start a branch at a version, or delete a branch (`-d`) |
93
- | `fdl merge <ref> <working> --author …` | Merge another branch into the working catalog's branch, rows matched by primary key |
94
- | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their catalogs |
93
+ | `fdl merge <ref> <working> --author …` | Merge another branch into the working metadata's branch, rows matched by primary key |
94
+ | `fdl expire <remote> --keep N` | Mark older versions as past retention and delete their metadata files |
95
95
  | `fdl gc <remote> --yes` | Delete data files no live version reaches, after the grace period |
96
96
  | `fdl withdraw <remote> <version>…` | Remove versions that hold data that must not be published |
97
97
  | `fdl assets <folder>` | List files for an asset column (path, sha256, bytes, media type) |
@@ -113,7 +113,7 @@ for version in repo.log(store):
113
113
 
114
114
  ## Earlier releases
115
115
 
116
- Releases up to 0.11 were a different tool for managing a single frozen catalog, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
116
+ Releases up to 0.11 were a different tool for managing a single frozen metadata file, published as `frozen-ducklake`. Their code is at the [`legacy`](https://github.com/queria-io/fdl/tree/legacy) branch.
117
117
 
118
118
  ## License
119
119
 
@@ -24,5 +24,9 @@ export default defineConfig({
24
24
  fonts: ["Noto Sans JP"],
25
25
  },
26
26
  },
27
- redirects: [{ from: "/cli", to: "/reference/cli" }],
27
+ redirects: [
28
+ { from: "/cli", to: "/reference/cli" },
29
+ { from: "/concepts/working-catalogs", to: "/concepts/working-metadata" },
30
+ { from: "/ja/concepts/working-catalogs", to: "/ja/concepts/working-metadata" },
31
+ ],
28
32
  });
@@ -3,5 +3,5 @@ import { defineMeta } from "blume";
3
3
  export default defineMeta({
4
4
  title: "Concepts",
5
5
  order: 2,
6
- pages: ["repositories", "versions-and-refs", "working-catalogs"],
6
+ pages: ["repositories", "versions-and-refs", "working-metadata"],
7
7
  });
@@ -9,11 +9,11 @@ A repository is everything under one prefix of a store: `s3://bucket/lake/` on S
9
9
 
10
10
  | Operation | Used by | For |
11
11
  | --- | --- | --- |
12
- | read an object | readers and writers | refs, `version.json`, catalogs, data files |
13
- | write an object | writers | data files, catalogs, `version.json` |
12
+ | read an object | readers and writers | refs, `version.json`, metadata files, data files |
13
+ | write an object | writers | data files, metadata files, `version.json` |
14
14
  | write on condition | writers | refs: create only if absent, replace only if unchanged |
15
15
  | list a prefix | maintenance | finding data files that no version names |
16
- | delete | maintenance | catalogs of versions past retention, and unused data files |
16
+ | delete | maintenance | metadata files of versions past retention, and unused data files |
17
17
 
18
18
  Conditional writes are what make two writers safe. A ref is created with `If-None-Match: *` and moved with `If-Match: <the ETag that was read>`. If someone else moved it in between, the store refuses the write and nothing changes. S3 and R2 support this. The folder store checks the condition without a lock, so it suits one writer at a time (see [Stores](/guides/stores)). A store without conditional writes can still be read from, but must not be pushed to: the last writer would silently replace someone else's version.
19
19
 
@@ -32,7 +32,7 @@ Readers only read. A static HTTP server in front of the repository is enough for
32
32
  assets/<sha256> files a version points at, such as PDFs, images and its README
33
33
  ```
34
34
 
35
- The repository holds no catalog outside a version folder, and never a working catalog. Either would hand readers a state that was never published.
35
+ The repository holds no metadata outside a version folder, and never a working metadata file. Either would hand readers a state that was never published.
36
36
 
37
37
  `fdl.json` carries an `id`, a UUID written once by `fdl init`. Copies of the repository on other hosts keep the same `id`, so a tool can tell that two URLs hold the same repository. A service that hosts repositories may write its own ID for the dataset there.
38
38
 
@@ -1,17 +1,17 @@
1
1
  ---
2
2
  title: Versions and refs
3
- description: A version is a frozen catalog with a record; a ref is a name that points at one. How they relate to DuckLake snapshots and to Git commits.
3
+ description: A version is a frozen metadata file with a record; a ref is a name that points at one. How they relate to DuckLake snapshots and to Git commits.
4
4
  ---
5
5
 
6
6
  ## Versions
7
7
 
8
- A version is one frozen DuckLake catalog and a record of how it came to be. It lives in its own folder and is never changed once written.
8
+ A version is one frozen DuckLake metadata and a record of how it came to be. It lives in its own folder and is never changed once written.
9
9
 
10
10
  ```
11
11
  versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/
12
12
  version.json the record: parents, author, message, checksums
13
13
  dataset.json what it holds: tables, columns, license
14
- ducklake.duckdb the frozen catalog
14
+ metadata.db the frozen metadata
15
15
  ```
16
16
 
17
17
  The record says:
@@ -20,10 +20,10 @@ The record says:
20
20
  - `seq`: the number shown to people, the parent's `seq` plus one
21
21
  - `parents`: the version it was built on. Empty for the first
22
22
  - `author`, `message` and `created_at`
23
- - `catalog`: the size and sha256 of the catalog file, so a reader can check what it fetched
24
- - `ducklake`: the DuckLake format of the catalog and the snapshot it was frozen at
23
+ - `metadata`: the size and sha256 of the metadata file, so a reader can check what it fetched
24
+ - `ducklake`: the DuckLake format of the metadata and the snapshot it was frozen at
25
25
 
26
- The ID is not a hash of the content, as a Git commit's is. The same catalog can be written as different bytes by two DuckDB releases, so a content hash would not be stable. The sha256 in `catalog` checks the content instead.
26
+ The ID is not a hash of the content, as a Git commit's is. The same metadata can be written as different bytes by two DuckDB releases, so a content hash would not be stable. The sha256 in `metadata` checks the content instead.
27
27
 
28
28
  The full list of fields is in [Versions](/spec/versions) in the specification.
29
29
 
@@ -31,7 +31,7 @@ The full list of fields is in [Versions](/spec/versions) in the specification.
31
31
 
32
32
  DuckLake makes a snapshot on every write: each `INSERT`, `CREATE TABLE` or `ALTER` in its own transaction. A day of work can make hundreds. A version is made when you decide the work is ready, and it takes the last of those snapshots.
33
33
 
34
- A snapshot is to a version what a commit on a work branch is to the squashed commit that lands on `main`. The snapshots stay inside the catalog, with their `commit_message` and `commit_extra_info`, so the detail is not lost. `version.json` records the range of snapshots the version adds after its parent, but does not copy them.
34
+ A snapshot is to a version what a commit on a work branch is to the squashed commit that lands on `main`. The snapshots stay inside the metadata, with their `commit_message` and `commit_extra_info`, so the detail is not lost. `version.json` records the range of snapshots the version adds after its parent, but does not copy them.
35
35
 
36
36
  ## Refs
37
37
 
@@ -94,4 +94,4 @@ Moving a branch back, to undo a version, is allowed only when asked for explicit
94
94
  First towns
95
95
  ```
96
96
 
97
- Versions that were taken out of retention stay in the history. Their record is kept, only their catalog is gone. See [Retention](/internals/inside-a-repository#taking-versions-out-of-retention).
97
+ Versions that were taken out of retention stay in the history. Their record is kept, only their metadata is gone. See [Retention](/internals/inside-a-repository#taking-versions-out-of-retention).
@@ -0,0 +1,49 @@
1
+ ---
2
+ title: Working metadata files
3
+ description: "Where writing happens: a DuckLake metadata file in SQLite, checked out from a version and frozen into the next one."
4
+ ---
5
+
6
+ A working metadata file is a DuckLake metadata file in a SQLite file, on the writer's own disk or in a container. It is the only place anyone writes to. It is to FDL what a working tree is to Git: made from a version, changed freely, and turned into the next version when the change is ready.
7
+
8
+ ## The cycle
9
+
10
+ | Step | Command | What happens |
11
+ | --- | --- | --- |
12
+ | check out | `fdl checkout` | Copy the metadata of the branch's version into SQLite, and remember that version |
13
+ | write | any DuckDB client | DuckLake writes Parquet into the repository and bookkeeping into SQLite |
14
+ | commit | `fdl commit` | Freeze the last snapshot into a new version, kept beside the working metadata |
15
+ | push | `fdl push` | Upload the version and move the branch to it |
16
+
17
+ Checking out copies DuckLake's bookkeeping tables from the frozen DuckDB file into SQLite as they are. Freezing does the reverse. Neither rewrites a Parquet file, so both are fast even for a large dataset: the metadata is small and the data stays where it is.
18
+
19
+ ## Why SQLite
20
+
21
+ A working metadata file has to be written by DuckDB and kept between sessions. SQLite does both without a database server, works the same on a laptop and in a container, and copies in and out of DuckDB's format without loss.
22
+
23
+ There is one kind of working metadata. A host that writes on behalf of its users (a scheduled import, a web form) checks out, writes, commits and pushes the same way a person does on a laptop. When two of them push at once, the conditional write on the ref settles it: one wins, and the other checks out again.
24
+
25
+ ## Attaching
26
+
27
+ `fdl checkout` prints the statement to attach the working metadata with:
28
+
29
+ ```sql
30
+ ATTACH 'ducklake:work/lake.sqlite' AS lake (
31
+ DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true,
32
+ META_TYPE 'sqlite', DATA_INLINING_ROW_LIMIT 0,
33
+ META_JOURNAL_MODE 'WAL', BUSY_TIMEOUT 5000
34
+ );
35
+ ```
36
+
37
+ - `DATA_PATH` is the repository's data folder, so new Parquet files land where readers will look for them. For a repository over HTTP it is a folder beside the working metadata instead, and `fdl push` uploads the new files (see [Pushing over HTTP](/spec/http#writing-to-it))
38
+ - `DATA_INLINING_ROW_LIMIT 0` keeps every row out of the metadata. A version must not carry inlined rows, and `fdl commit` refuses a working metadata file that holds some
39
+ - `META_TYPE 'sqlite'` has to be stated. Left to guess from the file, DuckDB passes an option to SQLite that SQLite rejects
40
+
41
+ ## Data files written before a push
42
+
43
+ DuckLake writes Parquet as you go, so data files reach the repository before any version names them. Readers never see them: a reader only opens metadata files of versions, and no version names those files yet.
44
+
45
+ A file that stays unnamed for longer than the grace period (7 days by default) may be deleted by maintenance. Push within that time. If a push is refused because a file it names is missing, rebuild the table in the working metadata, commit and push again. A version committed and never pushed names its files only on your disk, so they are deleted in the same way.
46
+
47
+ ## Snapshots in the working metadata
48
+
49
+ Do not run `ducklake_expire_snapshots` or `ducklake_cleanup_old_files` on a working metadata file. They only see the working metadata, not the repository, and would delete files that other versions still name. Removing old versions and files is done on the repository as a whole (see [Retention](/internals/inside-a-repository#taking-versions-out-of-retention)).
@@ -29,17 +29,17 @@ towns/
29
29
 
30
30
  `HEAD` names the default branch, `main`. The branch itself does not exist yet: there is no version for it to point at.
31
31
 
32
- `fdl init` makes the repository where readers will find it, as `git init --bare` does on a server. Your own place to write is the working catalog, made next.
32
+ `fdl init` makes the repository where readers will find it, as `git init --bare` does on a server. Your own place to write is the working metadata, made next.
33
33
 
34
- ## Check out a working catalog
34
+ ## Check out a working metadata file
35
35
 
36
- You never write to the repository directly. You write to a working catalog, a DuckLake catalog in a SQLite file on your own disk, and turn it into a version when you are done.
36
+ You never write to the repository directly. You write to a working metadata file, a DuckLake metadata file in a SQLite file on your own disk, and turn it into a version when you are done.
37
37
 
38
38
  ```bash
39
39
  fdl checkout ./towns/ work/lake.sqlite
40
40
  ```
41
41
 
42
- Before the first version, `checkout` makes an empty catalog. Later it copies the catalog of the version `main` holds. It prints the statement to attach the working catalog with:
42
+ Before the first version, `checkout` makes an empty metadata. Later it copies the metadata of the version `main` holds. It prints the statement to attach the working metadata with:
43
43
 
44
44
  ```sql
45
45
  ATTACH 'ducklake:work/lake.sqlite' AS lake (
@@ -49,7 +49,7 @@ ATTACH 'ducklake:work/lake.sqlite' AS lake (
49
49
  );
50
50
  ```
51
51
 
52
- Beside the working catalog, `work/lake.sqlite.fdl.json` remembers the repository, the branch and the version the checkout was made from.
52
+ Beside the working metadata, `work/lake.sqlite.fdl.json` remembers the repository, the branch and the version the checkout was made from.
53
53
 
54
54
  ## Write
55
55
 
@@ -60,7 +60,7 @@ CREATE TABLE lake.main.towns AS
60
60
  SELECT * FROM 'towns.csv';
61
61
  ```
62
62
 
63
- `DATA_INLINING_ROW_LIMIT 0` makes DuckLake write every row to Parquet. Left at its default, DuckLake keeps small tables inside the catalog, and a version has to carry its rows in data files that a reader can fetch.
63
+ `DATA_INLINING_ROW_LIMIT 0` makes DuckLake write every row to Parquet. Left at its default, DuckLake keeps small tables inside the metadata, and a version has to carry its rows in data files that a reader can fetch.
64
64
 
65
65
  ## Describe
66
66
 
@@ -74,7 +74,7 @@ Every version carries a `dataset.json` that says what it holds. Write one next t
74
74
  }
75
75
  ```
76
76
 
77
- Only `name` and `title` are required. `fdl commit` checks the file against the catalog and fills in each table's row count and each column's type.
77
+ Only `name` and `title` are required. `fdl commit` checks the file against the metadata and fills in each table's row count and each column's type.
78
78
 
79
79
  ## Commit and push
80
80
 
@@ -83,7 +83,7 @@ fdl commit work/lake.sqlite -m "First towns" --author alice
83
83
  fdl push work/lake.sqlite
84
84
  ```
85
85
 
86
- `commit` freezes the working catalog into a version on your disk. `push` measures each table into a small summary (a profile) that readers can show without reading the data, uploads the version with them and moves `main` to it, and prints `refs/heads/main -> 019a3c4e-… (version 1)`. The repository now holds:
86
+ `commit` freezes the working metadata into a version on your disk. `push` measures each table into a small summary (a profile) that readers can show without reading the data, uploads the version with them and moves `main` to it, and prints `refs/heads/main -> 019a3c4e-… (version 1)`. The repository now holds:
87
87
 
88
88
  ```
89
89
  towns/
@@ -91,7 +91,7 @@ towns/
91
91
  HEAD
92
92
  refs/heads/main 019a3c4e-…
93
93
  versions/019a3c4e-…/
94
- ducklake.duckdb
94
+ metadata.db
95
95
  dataset.json
96
96
  files.json
97
97
  profiles/main/towns.json
@@ -114,16 +114,16 @@ fdl log ./towns/
114
114
  fdl url ./towns/
115
115
  ```
116
116
 
117
- prints the catalog of the version `main` holds. Attach it read-only, and tell DuckDB where the data files are:
117
+ prints the metadata of the version `main` holds. Attach it read-only, and tell DuckDB where the data files are:
118
118
 
119
119
  ```sql
120
- ATTACH 'ducklake:towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
120
+ ATTACH 'ducklake:towns/versions/019a3c4e-…/metadata.db' AS towns (
121
121
  DATA_PATH 'towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
122
122
  );
123
123
  SELECT count(*) FROM towns.main.towns;
124
124
  ```
125
125
 
126
- A version's catalog never changes, so a reader may cache it forever. Only `HEAD` and the files under `refs/` change.
126
+ A version's metadata never changes, so a reader may cache it forever. Only `HEAD` and the files under `refs/` change.
127
127
 
128
128
  ## Make a second version
129
129
 
@@ -140,6 +140,6 @@ fdl commit work/lake.sqlite -m "Add 2026" --author alice
140
140
  fdl push work/lake.sqlite
141
141
  ```
142
142
 
143
- The new version lists the first as its parent. Both catalogs name the first Parquet file; only the second names the new one.
143
+ The new version lists the first as its parent. Both metadata files name the first Parquet file; only the second names the new one.
144
144
 
145
145
  If someone else pushed in between, the push is refused, because the new version would not be built on what `main` holds now. Check out again and redo the change. [Versions and refs](/concepts/versions-and-refs) explains why.
@@ -50,4 +50,4 @@ Access rules, approvals and billing stay in the host's own database. A repositor
50
50
 
51
51
  ## Retention
52
52
 
53
- `repo.prune` takes versions out of retention and `repo.expire` takes out one. Deleting data files that no live version names is left to the host, which can list the whole repository and read every live catalog. See [Data files](/spec/data-files) for when a file may be deleted.
53
+ `repo.prune` takes versions out of retention and `repo.expire` takes out one. Deleting data files that no live version names is left to the host, which can list the whole repository and read every live metadata. See [Data files](/spec/data-files) for when a file may be deleted.
@@ -30,24 +30,24 @@ A mirror is for reading. Pushing to both the origin and a mirror would give them
30
30
 
31
31
  ## Reading from a mirror
32
32
 
33
- The catalog records where the writer published its data files: the origin's data folder, as an absolute URL. A reader of the mirror gives DuckDB the mirror's data folder instead:
33
+ The metadata records where the writer published its data files: the origin's data folder, as an absolute URL. A reader of the mirror gives DuckDB the mirror's data folder instead:
34
34
 
35
35
  ```sql
36
- ATTACH 'ducklake:https://mirror.example.org/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
36
+ ATTACH 'ducklake:https://mirror.example.org/towns/versions/019a3c4e-…/metadata.db' AS towns (
37
37
  DATA_PATH 'https://mirror.example.org/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
38
38
  );
39
39
  ```
40
40
 
41
- Without `OVERRIDE_DATA_PATH true`, DuckDB refuses to use a data path other than the one the catalog records. Readers always pass both options, at the origin too, so the same statement works anywhere with only the root changed.
41
+ Without `OVERRIDE_DATA_PATH true`, DuckDB refuses to use a data path other than the one the metadata records. Readers always pass both options, at the origin too, so the same statement works anywhere with only the root changed.
42
42
 
43
43
  ## Checking what was copied
44
44
 
45
- `version.json` records the size and sha256 of the catalog. A mirror can check each catalog after copying it. Parquet files are named by a UUID and carry their own checksums, and DuckLake records each file's size in the catalog.
45
+ `version.json` records the size and sha256 of the metadata. A mirror can check each metadata file after copying it. Parquet files are named by a UUID and carry their own checksums, and DuckLake records each file's size in the metadata.
46
46
 
47
47
  Anyone can write a `version.json`, so a checksum shows the copy matches the record, not that the record came from the origin. A version signed by the origin can be checked on a mirror: see [Signatures](/spec/reading#signatures).
48
48
 
49
49
  ## Expired versions and withdrawn data
50
50
 
51
- When the origin takes a version out of retention, the mirror sees `expired.json` appear and the catalog disappear on the next copy. Copy `expired.json` and delete the catalogs the origin deleted. A mirror that keeps them serves versions its origin no longer does.
51
+ When the origin takes a version out of retention, the mirror sees `expired.json` appear and the metadata disappear on the next copy. Copy `expired.json` and delete the metadata files the origin deleted. A mirror that keeps them serves versions its origin no longer does.
52
52
 
53
- An `expired.json` with `"reason": "withdrawn"` means the origin removed data that must not be published. Delete the catalogs, the data files and the assets the origin deleted, without waiting.
53
+ An `expired.json` with `"reason": "withdrawn"` means the origin removed data that must not be published. Delete the metadata files, the data files and the assets the origin deleted, without waiting.
@@ -17,7 +17,7 @@ s3://public/towns/ the tables chosen for publication
17
17
  A version of the public repository is made from a version of the private one:
18
18
 
19
19
  1. Check out the public repository
20
- 2. Copy the chosen tables from the private version into the working catalog. The Parquet files are copied into the public repository's data folder
20
+ 2. Copy the chosen tables from the private version into the working metadata. The Parquet files are copied into the public repository's data folder
21
21
  3. Commit, and name the private version in `inputs`
22
22
 
23
23
  ```json
@@ -26,13 +26,13 @@ A version of the public repository is made from a version of the private one:
26
26
  ]
27
27
  ```
28
28
 
29
- `sha256` is the private version's catalog checksum. Readers of the public repository see that the version was made from another one, without being able to open it.
29
+ `sha256` is the private version's metadata checksum. Readers of the public repository see that the version was made from another one, without being able to open it.
30
30
 
31
31
  ## Why not one repository
32
32
 
33
- Putting a public catalog next to a private one in a single repository looks simpler, but:
33
+ Putting a public metadata file next to a private one in a single repository looks simpler, but:
34
34
 
35
35
  - the server would have to decide file by file who can read which Parquet file, and get it right for every file of every version
36
- - the private catalog's snapshot history names the private tables, their columns and their commit messages. A public catalog copied out of it would carry that history unless it was rewritten
36
+ - the private metadata's snapshot history names the private tables, their columns and their commit messages. A public metadata file copied out of it would carry that history unless it was rewritten
37
37
 
38
38
  Two repositories keep the rule simple. A reader can read everything in a repository or nothing in it.
@@ -22,7 +22,7 @@ const rows = await readWhere(await openParquet(store, file), "town_code", "13101
22
22
 
23
23
  ## What it can read
24
24
 
25
- It follows [Reading](/spec/reading) up to the catalog and then reads `files.json` in its place:
25
+ It follows [Reading](/spec/reading) up to the metadata and then reads `files.json` in its place:
26
26
 
27
27
  - A version without `files.json` cannot be read. Writers before fdl-cli 0.16 did not make one
28
28
  - A table whose files have `deletes`, or whose `renamed_columns` is `true`, is refused (see [files.json](/spec/versions#filesjson))
@@ -3,18 +3,18 @@ title: Serving over HTTP
3
3
  description: Letting readers attach a repository from a plain HTTP server or a CDN, and the caching each kind of file allows.
4
4
  ---
5
5
 
6
- Readers only read files, so a repository can be served by any HTTP server, a bucket's public URL or a CDN in front of one. DuckDB fetches the catalog and the Parquet files with range requests.
6
+ Readers only read files, so a repository can be served by any HTTP server, a bucket's public URL or a CDN in front of one. DuckDB fetches the metadata and the Parquet files with range requests.
7
7
 
8
8
  ## What a reader fetches
9
9
 
10
10
  1. `HEAD`, or the ref it was asked for
11
11
  2. `refs/heads/main`, if `HEAD` pointed there
12
12
  3. `versions/<ID>/version.json`, to check the version is not expired and that it can open the format
13
- 4. `versions/<ID>/ducklake.duckdb`, with range requests
13
+ 4. `versions/<ID>/metadata.db`, with range requests
14
14
  5. the Parquet files the query needs, with range requests
15
15
 
16
16
  ```sql
17
- ATTACH 'ducklake:https://data.example.org/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
17
+ ATTACH 'ducklake:https://data.example.org/towns/versions/019a3c4e-…/metadata.db' AS towns (
18
18
  DATA_PATH 'https://data.example.org/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
19
19
  );
20
20
  ```
@@ -38,11 +38,11 @@ The server must answer range requests (`Accept-Ranges: bytes`, `206 Partial Cont
38
38
 
39
39
  Readers do not list. A reader that knows a ref name gets everything else by following IDs, and one that does not reads `refs.json` for the names. Leave directory listings off; nothing needs them.
40
40
 
41
- ## Serving the latest catalog at a fixed URL
41
+ ## Serving the latest metadata at a fixed URL
42
42
 
43
- Some readers do not know FDL and expect one catalog URL. A host may serve the catalog of the version `HEAD` holds at `<root>/ducklake.duckdb`. It must then send the version ID in an `FDL-Version` response header, so a reader can tell which version it got, and must not let it be cached longer than the ref.
43
+ Some readers do not know FDL and expect one metadata file URL. A host may serve the metadata of the version `HEAD` holds at `<root>/metadata.db`. It must then send the version ID in an `FDL-Version` response header, so a reader can tell which version it got, and must not let it be cached longer than the ref.
44
44
 
45
- This is a convenience for readers, not part of the repository: the store holds no catalog outside `versions/`.
45
+ This is a convenience for readers, not part of the repository: the store holds no metadata outside `versions/`.
46
46
 
47
47
  ## CORS
48
48