fdl-cli 0.15.0__tar.gz → 0.17.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/CHANGELOG.md +72 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/PKG-INFO +2 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/guides/building-a-host.md +1 -1
- fdl_cli-0.17.0/docs/guides/meta.ts +7 -0
- fdl_cli-0.17.0/docs/guides/reading-without-duckdb.md +41 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/guides/stores.md +1 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/guides/building-a-host.md +1 -1
- fdl_cli-0.17.0/docs/ja/guides/meta.ts +7 -0
- fdl_cli-0.17.0/docs/ja/guides/reading-without-duckdb.md +41 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/guides/stores.md +1 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/reference/cli.md +16 -6
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/reference/python.md +2 -2
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/data-files.md +1 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/layout.md +1 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/versions.md +44 -4
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/writing.md +11 -2
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/reference/cli.md +16 -6
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/reference/python.md +2 -2
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/data-files.md +1 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/layout.md +1 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/versions.md +44 -4
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/writing.md +11 -2
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/pyproject.toml +2 -1
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/catalog.py +240 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/cli.py +32 -21
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/dataset.py +91 -8
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/layout.py +2 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/repo.py +52 -4
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/stores.py +65 -17
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/version.py +23 -3
- fdl_cli-0.17.0/tests/test_cli.py +170 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/tests/test_dataset.py +72 -4
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/tests/test_repo.py +42 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/tests/test_stores.py +35 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/uv.lock +3 -1
- fdl_cli-0.15.0/docs/guides/meta.ts +0 -7
- fdl_cli-0.15.0/docs/ja/guides/meta.ts +0 -7
- fdl_cli-0.15.0/tests/test_cli.py +0 -82
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.github/workflows/docs.yml +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.github/workflows/publish.yml +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.github/workflows/release.yml +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.gitignore +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.node-version +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/.python-version +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/CLAUDE.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/CONTRIBUTING.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/LICENSE +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/README.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/blume.config.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/concepts/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/concepts/repositories.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/concepts/versions-and-refs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/concepts/working-catalogs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/getting-started.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/guides/mirrors.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/guides/public-and-private.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/guides/serving-over-http.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/index.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/internals/inside-a-repository.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/internals/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/concepts/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/concepts/repositories.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/concepts/versions-and-refs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/concepts/working-catalogs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/getting-started.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/guides/mirrors.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/guides/public-and-private.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/guides/serving-over-http.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/index.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/internals/inside-a-repository.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/internals/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/reference/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/http.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/index.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/reading.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/ja/spec/refs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/reference/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/http.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/index.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/meta.ts +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/reading.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/docs/spec/refs.md +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/package-lock.json +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/package.json +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/public/CNAME +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/__init__.py +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/py.typed +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/refs.py +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/src/fdl/store.py +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/tests/http_repo.py +0 -0
- {fdl_cli-0.15.0 → fdl_cli-0.17.0}/tests/test_http.py +0 -0
|
@@ -1,6 +1,78 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
3
|
|
|
4
|
+
## v0.17.0 (2026-11-03)
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
- Compare rows by value and check the description first
|
|
9
|
+
([`5d17ae5`](https://github.com/queria-io/fdl/commit/5d17ae57ee79af2cf02ffe7ffd12f72176ecbdc0))
|
|
10
|
+
|
|
11
|
+
hash() gives NULL and empty nested values the same hash, so rows are compared with EXCEPT ALL once
|
|
12
|
+
the counts match. The dataset.json check runs before reading rows, table names are escaped, and
|
|
13
|
+
the message no longer names the checked-out version as where the branch is.
|
|
14
|
+
|
|
15
|
+
- Count a table's rows as a reader would
|
|
16
|
+
([`ca8c020`](https://github.com/queria-io/fdl/commit/ca8c02095f11763adbfb72c556fe00da1c0488a9))
|
|
17
|
+
|
|
18
|
+
dataset.json took rows from ducklake_table_stats, which keeps counting rows that were deleted or
|
|
19
|
+
updated. Rows are now the live data files' records less what their delete files remove.
|
|
20
|
+
|
|
21
|
+
### Documentation
|
|
22
|
+
|
|
23
|
+
- Add a guide to reading without DuckDB
|
|
24
|
+
([`b61328e`](https://github.com/queria-io/fdl/commit/b61328ef8ac0585dd5d547f25c8e0511370d4f28))
|
|
25
|
+
|
|
26
|
+
### Features
|
|
27
|
+
|
|
28
|
+
- Credential helper in fdl's config, views in dataset.json, and annotations
|
|
29
|
+
([#30](https://github.com/queria-io/fdl/pull/30),
|
|
30
|
+
[`e082f2b`](https://github.com/queria-io/fdl/commit/e082f2b189bae5491da0863bd1c48b1519e920f0))
|
|
31
|
+
|
|
32
|
+
BREAKING CHANGE: tokens saved by `fdl login` are no longer read. Set `FDL_TOKEN` or a credential
|
|
33
|
+
helper in fdl's config. `dataset.json` and `version.json` write `annotations` instead of
|
|
34
|
+
`extensions`.
|
|
35
|
+
|
|
36
|
+
- Make no version when nothing changed
|
|
37
|
+
([`fc08213`](https://github.com/queria-io/fdl/commit/fc08213f60195a474238f22bb1a513d903f8182e))
|
|
38
|
+
|
|
39
|
+
fdl commit compares the working catalog's last snapshot with the parent's. No new snapshot, or the
|
|
40
|
+
same tables, views, columns and rows (as a multiset) and the same dataset.json, prints "Nothing to
|
|
41
|
+
commit" and leaves the branch. Files are compared first, so rows are read only for tables written
|
|
42
|
+
to new files. --allow-empty makes the version anyway; rows that cannot be read are taken as
|
|
43
|
+
changed.
|
|
44
|
+
|
|
45
|
+
### Breaking Changes
|
|
46
|
+
|
|
47
|
+
- Tokens saved by `fdl login` are no longer read. Set `FDL_TOKEN` or a credential helper in fdl's
|
|
48
|
+
config. `dataset.json` and `version.json` write `annotations` instead of `extensions`.
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
## v0.16.0 (2026-11-01)
|
|
52
|
+
|
|
53
|
+
### Documentation
|
|
54
|
+
|
|
55
|
+
- Compare files.json with Git's reachability bitmap
|
|
56
|
+
([`181a738`](https://github.com/queria-io/fdl/commit/181a738c9b09dde2e74adea19b9b520884d76833))
|
|
57
|
+
|
|
58
|
+
### Features
|
|
59
|
+
|
|
60
|
+
- Index each table's files in files.json, not version.json
|
|
61
|
+
([`74412af`](https://github.com/queria-io/fdl/commit/74412af89913284ea48ab66f333da0725fb724b9))
|
|
62
|
+
|
|
63
|
+
files.json sits beside the catalog as an optional index made from it, like Git's commit-graph.
|
|
64
|
+
version.json names it with its size and sha256. Tables with a renamed column are marked, and
|
|
65
|
+
readers without DuckLake must not read a table with deletes or renamed columns. Expiry deletes the
|
|
66
|
+
index with the catalogs.
|
|
67
|
+
|
|
68
|
+
- List each table's files in version.json
|
|
69
|
+
([`1134b2d`](https://github.com/queria-io/fdl/commit/1134b2df324b6eb3fe0fa3598c5bf355bed64561))
|
|
70
|
+
|
|
71
|
+
Readers without DuckDB (a Worker reading Parquet by range) need to know which files make up a table
|
|
72
|
+
at a version. commit now writes tables with each data file's path under data_dir, rows, bytes,
|
|
73
|
+
footer size and its delete file.
|
|
74
|
+
|
|
75
|
+
|
|
4
76
|
## v0.15.0 (2026-11-01)
|
|
5
77
|
|
|
6
78
|
### Features
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: fdl-cli
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.17.0
|
|
4
4
|
Summary: Versions of a DuckLake catalog on object storage, kept the way Git keeps commits
|
|
5
5
|
Project-URL: Homepage, https://fdl.flo8s.com/
|
|
6
6
|
Project-URL: Documentation, https://fdl.flo8s.com/
|
|
@@ -12,6 +12,7 @@ License-File: LICENSE
|
|
|
12
12
|
Requires-Python: >=3.10
|
|
13
13
|
Requires-Dist: duckdb<2.0,>=1.5.2
|
|
14
14
|
Requires-Dist: pyyaml>=6.0
|
|
15
|
+
Requires-Dist: tomli>=1.1; python_version < '3.11'
|
|
15
16
|
Requires-Dist: typer>=0.15
|
|
16
17
|
Provides-Extra: s3
|
|
17
18
|
Requires-Dist: boto3>=1.36; extra == 's3'
|
|
@@ -43,7 +43,7 @@ A scheduled import or a form in a web app writes the same way a person does: che
|
|
|
43
43
|
What a host adds goes outside the fields FDL defines:
|
|
44
44
|
|
|
45
45
|
- extra files in a version folder (a preview), which readers ignore
|
|
46
|
-
- `
|
|
46
|
+
- `annotations` in `version.json` and `dataset.json`, keyed by the host's name
|
|
47
47
|
- its own ID for the dataset in `fdl.json`'s `id`, written at `init` and never changed
|
|
48
48
|
|
|
49
49
|
Access rules, approvals and billing stay in the host's own database. A repository copied off the host still reads correctly, with nothing in it that only the host understands.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Reading without DuckDB
|
|
3
|
+
description: Reading a table's Parquet files from JavaScript with files.json, in Workers, browsers and Node.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
A reader that cannot run DuckDB, such as a Cloudflare Worker or a web page, can still read a version: `files.json` lists each table's data files, and the Parquet files can be read by range. [`@queria-io/fdl-js`](https://www.npmjs.com/package/@queria-io/fdl-js) does this in JavaScript.
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
npm install @queria-io/fdl-js
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
```ts
|
|
13
|
+
import { httpStore, open, openParquet, readWhere, tableFiles } from "@queria-io/fdl-js";
|
|
14
|
+
|
|
15
|
+
const store = httpStore("https://data.example.org/towns/");
|
|
16
|
+
const version = await open(store); // HEAD, a branch, a tag or a version ID
|
|
17
|
+
const [file] = tableFiles(version, "main.towns") ?? [];
|
|
18
|
+
const rows = await readWhere(await openParquet(store, file), "town_code", "131016");
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
`readWhere` reads only the row groups whose min/max can hold the value, so a table sorted by the column it is read by costs a few small requests.
|
|
22
|
+
|
|
23
|
+
## What it can read
|
|
24
|
+
|
|
25
|
+
It follows [Reading](/spec/reading) up to the catalog and then reads `files.json` in its place:
|
|
26
|
+
|
|
27
|
+
- A version without `files.json` cannot be read. Writers before fdl-cli 0.16 did not make one
|
|
28
|
+
- A table whose files have `deletes`, or whose `renamed_columns` is `true`, is refused (see [files.json](/spec/versions#filesjson))
|
|
29
|
+
- A version with `expired.json` is refused
|
|
30
|
+
|
|
31
|
+
Tables written once and replaced whole by each version, as `CREATE OR REPLACE TABLE` does, have neither deletes nor renamed columns.
|
|
32
|
+
|
|
33
|
+
## Stores
|
|
34
|
+
|
|
35
|
+
| Store | Reads from |
|
|
36
|
+
| --- | --- |
|
|
37
|
+
| `httpStore(root)` | A URL, with `Range` requests |
|
|
38
|
+
| `r2Store(bucket, prefix)` | An R2 binding in a Cloudflare Worker |
|
|
39
|
+
| `fileStore(dir)` from `@queria-io/fdl-js/node` | A local folder |
|
|
40
|
+
|
|
41
|
+
Refs move, so keep the version a reader opened for a few minutes at most. Data files never change, so their Parquet footers can be cached forever.
|
|
@@ -3,7 +3,7 @@ title: Stores
|
|
|
3
3
|
description: Using a local folder, S3, R2 or another S3-compatible store as a repository, and what credentials the fdl command reads.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
A repository's URL picks the store. `s3://bucket/prefix/` is S3 or anything that speaks its API. `https://host/path/` is a host that takes [pushes over HTTP](/spec/http), with
|
|
6
|
+
A repository's URL picks the store. `s3://bucket/prefix/` is S3 or anything that speaks its API. `https://host/path/` is a host that takes [pushes over HTTP](/spec/http), with a token from `FDL_TOKEN` or a credential helper ([Tokens for HTTP](/reference/cli#tokens-for-http)). Anything else is a folder on disk.
|
|
7
7
|
|
|
8
8
|
## A folder
|
|
9
9
|
|
|
@@ -43,7 +43,7 @@ refs.write(store, "refs/heads/main", version.id, expected=current)
|
|
|
43
43
|
ホストが足すものは、FDL が決めた項目の外に置きます。
|
|
44
44
|
|
|
45
45
|
- バージョンのフォルダに置く追加のファイル(プレビューなど)。読み手は無視する
|
|
46
|
-
- `version.json` と `dataset.json` の `
|
|
46
|
+
- `version.json` と `dataset.json` の `annotations`。ホストの名前をキーにする
|
|
47
47
|
- `fdl.json` の `id` に書く、ホスト側のデータセットの ID。`init` のときに書き、変えない
|
|
48
48
|
|
|
49
49
|
権限、承認、課金はホスト自身のデータベースに置きます。ホストから持ち出したリポジトリも、ホストにしか分からないものを含まずに正しく読めます。
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: DuckDB を使わずに読む
|
|
3
|
+
description: files.json を使い、表の Parquet ファイルを JavaScript で読む方法。Workers・ブラウザ・Node で動きます。
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Cloudflare Worker や Web ページのように DuckDB を動かせない読み手も、バージョンを読めます。`files.json` に表ごとのデータファイルが並んでいるので、Parquet ファイルを範囲指定で読めば済みます。JavaScript では [`@queria-io/fdl-js`](https://www.npmjs.com/package/@queria-io/fdl-js) がこれを行います。
|
|
7
|
+
|
|
8
|
+
```bash
|
|
9
|
+
npm install @queria-io/fdl-js
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
```ts
|
|
13
|
+
import { httpStore, open, openParquet, readWhere, tableFiles } from "@queria-io/fdl-js";
|
|
14
|
+
|
|
15
|
+
const store = httpStore("https://data.example.org/towns/");
|
|
16
|
+
const version = await open(store); // HEAD、ブランチ、タグ、バージョン ID
|
|
17
|
+
const [file] = tableFiles(version, "main.towns") ?? [];
|
|
18
|
+
const rows = await readWhere(await openParquet(store, file), "town_code", "131016");
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
`readWhere` は、min/max にその値が入りうる行グループだけを読みます。読むときの列で並べた表なら、小さな読み取り数回で済みます。
|
|
22
|
+
|
|
23
|
+
## 読めるもの
|
|
24
|
+
|
|
25
|
+
[読み方](/ja/spec/reading)の手順にカタログの手前まで従い、カタログの代わりに `files.json` を読みます。
|
|
26
|
+
|
|
27
|
+
- `files.json` の無いバージョンは読めません。fdl-cli 0.16 より前の書き手は作りません
|
|
28
|
+
- ファイルに `deletes` がある表と、`renamed_columns` が `true` の表は断ります([files.json](/ja/spec/versions#filesjson) を参照)
|
|
29
|
+
- `expired.json` のあるバージョンは断ります
|
|
30
|
+
|
|
31
|
+
`CREATE OR REPLACE TABLE` のように、バージョンごとに表をまるごと作り直す書き方なら、削除も列の改名も起きません。
|
|
32
|
+
|
|
33
|
+
## 置き場
|
|
34
|
+
|
|
35
|
+
| 置き場 | 読む先 |
|
|
36
|
+
| --- | --- |
|
|
37
|
+
| `httpStore(root)` | URL(`Range` リクエスト) |
|
|
38
|
+
| `r2Store(bucket, prefix)` | Cloudflare Worker の R2 バインディング |
|
|
39
|
+
| `@queria-io/fdl-js/node` の `fileStore(dir)` | 手元のフォルダ |
|
|
40
|
+
|
|
41
|
+
ref は動くので、開いたバージョンを覚えておくのは数分までにしてください。データファイルは変わらないので、Parquet のフッターはずっとキャッシュしてかまいません。
|
|
@@ -3,7 +3,7 @@ title: 置き場
|
|
|
3
3
|
description: フォルダ、S3、R2、ほかの S3 互換ストレージをリポジトリの置き場にする方法と、fdl コマンドが読む認証情報。
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
リポジトリの URL で置き場が決まります。`s3://bucket/prefix/` は S3 か、S3 の API を話すストレージです。`https://host/path/` は [HTTP で push を受け付ける](/ja/spec/http)ホストで、`
|
|
6
|
+
リポジトリの URL で置き場が決まります。`s3://bucket/prefix/` は S3 か、S3 の API を話すストレージです。`https://host/path/` は [HTTP で push を受け付ける](/ja/spec/http)ホストで、`FDL_TOKEN` か credential helper のトークンを使います([HTTP のトークン](/ja/reference/cli#http-のトークン))。それ以外はディスク上のフォルダです。
|
|
7
7
|
|
|
8
8
|
## フォルダ
|
|
9
9
|
|
|
@@ -3,15 +3,23 @@ title: コマンドライン
|
|
|
3
3
|
description: fdl のコマンド、オプション、断られるとき。
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
`<remote>` はリポジトリで、`s3://bucket/prefix/`、`https://host/path/`、またはフォルダです。S3 の認証情報とエンドポイントは boto3 の通常の場所から読みます(`AWS_*` の環境変数、R2 などの S3 互換ストレージには `AWS_ENDPOINT_URL`)。HTTP のリポジトリには、`
|
|
6
|
+
`<remote>` はリポジトリで、`s3://bucket/prefix/`、`https://host/path/`、またはフォルダです。S3 の認証情報とエンドポイントは boto3 の通常の場所から読みます(`AWS_*` の環境変数、R2 などの S3 互換ストレージには `AWS_ENDPOINT_URL`)。HTTP のリポジトリには、`FDL_TOKEN` か credential helper でトークンを渡します([HTTP のトークン](#http-のトークン))。
|
|
7
7
|
|
|
8
|
-
##
|
|
8
|
+
## HTTP のトークン
|
|
9
9
|
|
|
10
|
-
|
|
11
|
-
|
|
10
|
+
fdl はトークンを保存せず、ログインのコマンドもありません。HTTP のリポジトリには、次の順に見つけたトークンを `Authorization: Bearer` で送ります。
|
|
11
|
+
|
|
12
|
+
1. `FDL_TOKEN` があればそれ(CI 向け)
|
|
13
|
+
2. 無ければ、fdl の設定でリポジトリの URL に設定した credential helper が答えたもの。helper が無ければトークンを送りません
|
|
14
|
+
|
|
15
|
+
設定は `~/.config/fdl/config.toml`(`$XDG_CONFIG_HOME/fdl/config.toml`、または `FDL_CONFIG` で指定したファイル)です。helper は URL の前方一致で設定し、パスが一番長く一致したものを使います。
|
|
16
|
+
|
|
17
|
+
```toml
|
|
18
|
+
[credential."https://data.example.org"]
|
|
19
|
+
helper = "example-cli fdl-credential"
|
|
12
20
|
```
|
|
13
21
|
|
|
14
|
-
|
|
22
|
+
fdl は `<helper> get` を実行し、標準入力に `protocol=`・`host=`・`path=` の行と空行を渡して、出力の `password=` の行を読みます。何も出さないか、エラーで終わったら、トークンは無いものとします。トークンの保存と helper の書き込みは、ホストの側の道具が行います。
|
|
15
23
|
|
|
16
24
|
## fdl init
|
|
17
25
|
|
|
@@ -36,13 +44,15 @@ HTTP のリポジトリでは、新しいデータファイルは `<working.sqli
|
|
|
36
44
|
## fdl commit
|
|
37
45
|
|
|
38
46
|
```bash
|
|
39
|
-
fdl commit <working.sqlite> -m "What changed" --author alice [--dataset dataset.json]
|
|
47
|
+
fdl commit <working.sqlite> -m "What changed" --author alice [--dataset dataset.json] [--allow-empty]
|
|
40
48
|
```
|
|
41
49
|
|
|
42
50
|
作業カタログを、チェックアウトしたバージョンを親とする新しいバージョンに固め、アップロードし、条件付きでブランチを動かします。ブランチ、新しいバージョン ID、`seq` を表示します。
|
|
43
51
|
|
|
44
52
|
バージョンはそれぞれ `dataset.json`([バージョン](/ja/spec/versions#datasetjson)を参照)で説明します。`--dataset` で指定したファイルか、なければいまのディレクトリの `dataset.json` か `dataset.yml` から読みます。そのフォルダより下にある `*.table.yml`(または `*.table.json`)のテーブルの説明も合わせて読みます。隠しフォルダの中は読みません。コマンドはそれをカタログと照らし合わせ、列の型、行数、カタログのコメントを書き足します。
|
|
45
53
|
|
|
54
|
+
親と同じ中身になるとき([変更がないとき](/ja/spec/writing#変更がないとき))は `Nothing to commit` と表示し、ブランチを動かさずに終了コード 0 で終わります。同じ行を新しいファイルに書き直しただけの作り直しも、変更なしとみなします。`--allow-empty` を付けると、それでもバージョンを作ります。行を読めず判定できないときは、警告を出してバージョンを作ります。
|
|
55
|
+
|
|
46
56
|
次のときは断られます。
|
|
47
57
|
|
|
48
58
|
- `dataset.json` が無いとき、カタログに無い表や列を説明しているとき、形式が決めていないキーがあるとき
|
|
@@ -49,13 +49,13 @@ repo.checkout(store, working, *, ref="HEAD") -> Version | None
|
|
|
49
49
|
`ref` が指すバージョンから、`working` に作業カタログを作ります。そのバージョンを返し、最初のバージョンの前は `None` を返します。
|
|
50
50
|
|
|
51
51
|
```python
|
|
52
|
-
repo.commit(working, out_dir, *, parent, data_path, author, message, inputs=(),
|
|
52
|
+
repo.commit(working, out_dir, *, parent, data_path, author, message, inputs=(), annotations=None, dataset=None) -> Version
|
|
53
53
|
```
|
|
54
54
|
|
|
55
55
|
作業カタログを `out_dir/<ID>/` に固めます。`dataset`(読み込んだ `dataset.json`)をカタログと照らし合わせ、カタログから書き足します。`data_path` は `repo.data_url(store)` です。アップロードはしません。
|
|
56
56
|
|
|
57
57
|
```python
|
|
58
|
-
repo.record(frozen, out_dir, *, parent, author, message, inputs=(),
|
|
58
|
+
repo.record(frozen, out_dir, *, parent, author, message, inputs=(), annotations=None, dataset=None) -> Version
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
固め終えたカタログからバージョンを作ります。自分でカタログを固める書き手向けです。
|
|
@@ -24,7 +24,7 @@ DuckLake 自身の片付け(`ducklake_cleanup_old_files`、`ducklake_expire_sn
|
|
|
24
24
|
バージョンの保持期間は、次の順に終わらせます。
|
|
25
25
|
|
|
26
26
|
1. `versions/<ID>/expired.json` を置く
|
|
27
|
-
2.
|
|
27
|
+
2. バージョンのカタログ(`ducklake.duckdb` と `catalogs/`)と `files.json` を消す
|
|
28
28
|
3. 猶予期間のあと、上の条件に合うデータファイルを消す
|
|
29
29
|
|
|
30
30
|
```json expired.json
|
|
@@ -21,6 +21,7 @@ description: リポジトリのファイル、そのパス、fdl.json の中身
|
|
|
21
21
|
version.json.sig 記録の署名(任意)
|
|
22
22
|
ducklake.duckdb 固めたカタログ
|
|
23
23
|
dataset.json バージョンの中身の説明
|
|
24
|
+
files.json カタログから作った、表ごとのデータファイルの索引(任意)
|
|
24
25
|
catalogs/<format>.duckdb 別の DuckLake 形式のカタログ(任意)
|
|
25
26
|
expired.json 保持期間を過ぎた印(過ぎたときだけ)
|
|
26
27
|
<others> ホストが足すファイル(プレビューなど)
|
|
@@ -43,6 +43,11 @@ description: バージョン ID、version.json の項目、固めたカタログ
|
|
|
43
43
|
"snapshot_id": 345,
|
|
44
44
|
"snapshots": [301, 345]
|
|
45
45
|
},
|
|
46
|
+
"files": {
|
|
47
|
+
"path": "files.json",
|
|
48
|
+
"bytes": 1536,
|
|
49
|
+
"sha256": "7a0e…"
|
|
50
|
+
},
|
|
46
51
|
"changes": [
|
|
47
52
|
{ "table": "main.population", "kind": "add_column", "column": "note", "breaking": false }
|
|
48
53
|
],
|
|
@@ -54,7 +59,7 @@ description: バージョン ID、version.json の項目、固めたカタログ
|
|
|
54
59
|
"url": "https://example.org/population-2025.csv"
|
|
55
60
|
}
|
|
56
61
|
],
|
|
57
|
-
"
|
|
62
|
+
"annotations": {}
|
|
58
63
|
}
|
|
59
64
|
```
|
|
60
65
|
|
|
@@ -70,9 +75,10 @@ description: バージョン ID、version.json の項目、固めたカタログ
|
|
|
70
75
|
| `catalog` | must | カタログのファイル名、大きさ、sha256、保存形式(`storage`) |
|
|
71
76
|
| `dataset` | must | カタログの隣の `dataset.json` のファイル名、大きさ、sha256。必須になる前に書かれたバージョンには無い |
|
|
72
77
|
| `ducklake` | must | DuckLake の形式、固めたスナップショット、親から足したスナップショットの範囲(最初と最後) |
|
|
78
|
+
| `files` | may | カタログの隣の `files.json` を書いたときの、そのファイル名、大きさ、sha256([files.json](#filesjson)) |
|
|
73
79
|
| `changes` | may | 親との列の違い。`breaking` は、親に向けて書いた SQL が失敗しうるか(列の削除・改名・作り直し) |
|
|
74
80
|
| `inputs` | may | 使った元のファイル。`source_id` の付け方は書き手が決める |
|
|
75
|
-
| `
|
|
81
|
+
| `annotations` | may | 形式が解釈しない情報。読む側(ホストやツール)の名前をキーにする |
|
|
76
82
|
|
|
77
83
|
読み手は、知らない項目を無視しなければなりません。
|
|
78
84
|
|
|
@@ -90,6 +96,38 @@ FDL 0.1 では、書き手が書く親は多くても 1 つです。`parents`
|
|
|
90
96
|
|
|
91
97
|
読み手は、`version.json` だけで、そのバージョンを開けるか判断できます。
|
|
92
98
|
|
|
99
|
+
## files.json
|
|
100
|
+
|
|
101
|
+
`files.json` は、`ducklake.snapshot_id` の時点の表ごとのデータファイルの索引です。カタログを開けない読み手のためのものです。書き手はこれをカタログから作り、カタログと食い違ったときはカタログが正です。置くかどうかは任意で、読み手はこれが無くても動かなければなりません。Git の到達可能性ビットマップ(pack の隣の `.bitmap`)に当たります。あるコミットからたどれるオブジェクトの一覧をオブジェクトから作ったもので、読むのを速くし、無くても読めます。
|
|
102
|
+
|
|
103
|
+
```json files.json
|
|
104
|
+
{
|
|
105
|
+
"tables": {
|
|
106
|
+
"main.population": {
|
|
107
|
+
"files": [
|
|
108
|
+
{
|
|
109
|
+
"path": "main/population/ducklake-019a3c4e-….parquet",
|
|
110
|
+
"rows": 1741,
|
|
111
|
+
"bytes": 52311,
|
|
112
|
+
"footer_bytes": 2048,
|
|
113
|
+
"deletes": { "path": "main/population/ducklake-019a3c4f-…-delete.parquet", "rows": 3, "bytes": 812 }
|
|
114
|
+
}
|
|
115
|
+
]
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
- `tables` には、そのスナップショットにあるすべての表を `schema.table` をキーにして入れます。行の無い表の `files` は空です
|
|
122
|
+
- `path` はリポジトリの `data_dir` からの相対パスです。ファイルはカタログの順に並べます
|
|
123
|
+
- `rows` と `bytes` は、カタログが記録している行数と大きさです。`footer_bytes` は Parquet のフッターの大きさで、記録があるときだけ書きます
|
|
124
|
+
- `deletes` は、そのデータファイルから行を消したときの削除ファイルです
|
|
125
|
+
- `renamed_columns` は、その表の列の名前を変えたことがあるときに `true` にします。名前を変える前に書いたデータファイルは、古い名前のままです
|
|
126
|
+
|
|
127
|
+
DuckLake を使わずに Parquet ファイルを読む読み手は、`deletes` を持つファイルがある表や、`renamed_columns` が `true` の表を読んではいけません。DuckLake と同じように削除と改名を当てはめる場合は読んでかまいません。当てはめずに読むと、消した行が返ったり、名前を変えた列が欠けたりします。
|
|
128
|
+
|
|
129
|
+
保持期間から外すとき、`files.json` はカタログと一緒に消します。
|
|
130
|
+
|
|
93
131
|
## 固めたカタログ
|
|
94
132
|
|
|
95
133
|
`ducklake.duckdb` は、DuckDB のファイル形式の DuckLake カタログです。
|
|
@@ -137,7 +175,7 @@ FDL 0.1 では、書き手が書く親は多くても 1 つです。`parents`
|
|
|
137
175
|
]
|
|
138
176
|
}
|
|
139
177
|
],
|
|
140
|
-
"
|
|
178
|
+
"annotations": {}
|
|
141
179
|
}
|
|
142
180
|
```
|
|
143
181
|
|
|
@@ -145,8 +183,10 @@ FDL 0.1 では、書き手が書く親は多くても 1 つです。`parents`
|
|
|
145
183
|
- 書き手は説明を YAML(`dataset.yml`)で持ってもよく、テーブルごとに別のファイル(`<name>.table.yml`、`tables` の 1 項目)に分けてもよい。バージョンには常に説明全体が 1 つの JSON で入る
|
|
146
184
|
- `licenses` の各要素は SPDX の識別子か、`id` に書き手が足したいもの(`spdx`・`url`・`title`)を添えたオブジェクト
|
|
147
185
|
- `tables` には、カタログの最後のスナップショットにある表をすべて並べる。書き手が説明しなかった表は、書き手のツールがカタログから分かることで足す。カタログに無い表や列を説明していたら断る
|
|
186
|
+
- カタログのビューは表のあとに `"kind": "view"` を付けて並べ、`sql` はカタログのものを入れる。カタログはビューの列を記録しないので、ビューの `fields` は書き手が書いたままで、`type`・`nullable`・`rows` を持たない。表には `kind` を付けない
|
|
148
187
|
- カタログから取るもの: 表ごとの `rows`、列ごとの `type`(DuckLake の型名)と `nullable`。書き手が `description` を書かなかったところには、カタログのコメント(`COMMENT ON`)を使う
|
|
149
|
-
-
|
|
188
|
+
- この仕様が決めていないキーは断る。形式が解釈しない情報は、最上位・表・列それぞれの `annotations` に、読む側の名前をキーにして置く。書き手は書かれたまま残す
|
|
189
|
+
- `annotations` という名前になる前のバージョンは、同じものを `extensions` に持つ。読み手は `extensions` を `annotations` として読む
|
|
150
190
|
- ライセンスや出典に書いたことは書き手の主張で、ホストは公開する前に確かめてよい
|
|
151
191
|
|
|
152
192
|
## ほかの形式のカタログ
|
|
@@ -17,7 +17,7 @@ description: チェックアウト、コミット、push。作業カタログ、
|
|
|
17
17
|
| --- | --- |
|
|
18
18
|
| チェックアウト | ブランチのバージョンのカタログから作業カタログを作り、そのバージョンを覚える。最初のバージョンの前は空のものを作る |
|
|
19
19
|
| コミット | 作業カタログの最後のスナップショットを新しいバージョンに固める。親は覚えておいたバージョン。中身を `dataset.json` で説明する |
|
|
20
|
-
| push | データファイルを確かめ、カタログ、`dataset.json`、`version.json` の順にアップロードし、条件付きで ref を動かす |
|
|
20
|
+
| push | データファイルを確かめ、カタログ、`dataset.json` と `files.json`、`version.json` の順にアップロードし、条件付きで ref を動かす |
|
|
21
21
|
|
|
22
22
|
チェックアウトでは、DuckLake の管理用の表を、固めたカタログから SQLite へそのまま写します。固めるときはその逆で、最後のスナップショットまでの行を残します。どちらもデータファイルは書き換えません。
|
|
23
23
|
|
|
@@ -27,11 +27,20 @@ description: チェックアウト、コミット、push。作業カタログ、
|
|
|
27
27
|
- 固めたカタログの `data_path` を `<root><data_dir>` の絶対 URL にする
|
|
28
28
|
- 絶対パスで記録されたデータファイルや、`data_dir` の外のデータファイルがあれば断る
|
|
29
29
|
|
|
30
|
+
## 変更がないとき
|
|
31
|
+
|
|
32
|
+
書き手は、親と同じ中身のバージョンを作らないようにするべきです。Git が親と同じ tree のコミットを作らないのと同じです。次のどちらかなら、親と同じ中身です。
|
|
33
|
+
|
|
34
|
+
- 最後のスナップショットが親のもの(`ducklake.snapshot_id`)と同じ
|
|
35
|
+
- 同じ表とビューがあり、どの表も列の名前と型が同じで、行が同じ(重複を数え、順番は問わない)。さらに `dataset.json` の sha256 が親と同じ
|
|
36
|
+
|
|
37
|
+
同じ入力から作り直した表は新しいデータファイルに書かれるので、ファイルやカタログのバイトを比べても判定できません。書き手はまずファイルを比べ、ファイルが違う表だけ行を読みます。行を読めないときはバージョンを作ります。頼まれたときは、`git commit --allow-empty` と同じく、変更がなくても作ってかまいません。
|
|
38
|
+
|
|
30
39
|
## 書き込みの順番
|
|
31
40
|
|
|
32
41
|
1. データファイル
|
|
33
42
|
2. `versions/<ID>/ducklake.duckdb`(と `catalogs/`)
|
|
34
|
-
3. `versions/<ID>/dataset.json`
|
|
43
|
+
3. `versions/<ID>/dataset.json` と `versions/<ID>/files.json`
|
|
35
44
|
4. `versions/<ID>/version.json`
|
|
36
45
|
5. ref(条件付き)
|
|
37
46
|
|
|
@@ -3,15 +3,23 @@ title: Command line
|
|
|
3
3
|
description: The fdl commands, their options and when they refuse.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
`<remote>` is a repository: `s3://bucket/prefix/`, `https://host/path/` or a folder. Credentials and the endpoint for S3 come from boto3's usual places (`AWS_*` environment variables, `AWS_ENDPOINT_URL` for R2 and other S3-compatible stores). A repository over HTTP takes
|
|
6
|
+
`<remote>` is a repository: `s3://bucket/prefix/`, `https://host/path/` or a folder. Credentials and the endpoint for S3 come from boto3's usual places (`AWS_*` environment variables, `AWS_ENDPOINT_URL` for R2 and other S3-compatible stores). A repository over HTTP takes a token from `FDL_TOKEN` or a credential helper (see [Tokens for HTTP](#tokens-for-http)).
|
|
7
7
|
|
|
8
|
-
##
|
|
8
|
+
## Tokens for HTTP
|
|
9
9
|
|
|
10
|
-
|
|
11
|
-
|
|
10
|
+
fdl keeps no tokens and has no login. For a repository over HTTP it sends, as `Authorization: Bearer`:
|
|
11
|
+
|
|
12
|
+
1. `FDL_TOKEN`, when set (for CI)
|
|
13
|
+
2. Otherwise what the credential helper set for the repository's URL in fdl's config answers. With no helper set, fdl sends no token
|
|
14
|
+
|
|
15
|
+
The config is `~/.config/fdl/config.toml` (`$XDG_CONFIG_HOME/fdl/config.toml`, or the file `FDL_CONFIG` names). A helper is set per URL prefix; the one with the longest matching path wins:
|
|
16
|
+
|
|
17
|
+
```toml
|
|
18
|
+
[credential."https://data.example.org"]
|
|
19
|
+
helper = "example-cli fdl-credential"
|
|
12
20
|
```
|
|
13
21
|
|
|
14
|
-
|
|
22
|
+
fdl runs `<helper> get`, writes `protocol=`, `host=` and `path=` lines and a blank line to its stdin, and reads the `password=` line it prints. A helper that prints nothing, or exits with an error, means no token. The host's own tool stores the token and writes its helper here.
|
|
15
23
|
|
|
16
24
|
## fdl init
|
|
17
25
|
|
|
@@ -36,13 +44,15 @@ Refused when the version is past retention.
|
|
|
36
44
|
## fdl commit
|
|
37
45
|
|
|
38
46
|
```bash
|
|
39
|
-
fdl commit <working.sqlite> -m "What changed" --author alice [--dataset dataset.json]
|
|
47
|
+
fdl commit <working.sqlite> -m "What changed" --author alice [--dataset dataset.json] [--allow-empty]
|
|
40
48
|
```
|
|
41
49
|
|
|
42
50
|
Freezes the working catalog into a new version whose parent is the version it was checked out at, uploads it, and moves the branch on condition. It prints the branch, the new version ID and its `seq`.
|
|
43
51
|
|
|
44
52
|
Every version is described by a `dataset.json` (see [Versions](/spec/versions#datasetjson)), read from `--dataset` or else from `dataset.json` or `dataset.yml` in the current directory. Tables described in `*.table.yml` (or `*.table.json`) files below that folder are added to it; hidden folders are skipped. The command checks it against the catalog and fills in the column types, row counts and catalog comments.
|
|
45
53
|
|
|
54
|
+
When the version would hold what its parent holds (see [When nothing changed](/spec/writing#when-nothing-changed)), it prints `Nothing to commit`, leaves the branch where it is and exits 0. A rebuild that writes the same rows to new files counts as unchanged. `--allow-empty` makes the version anyway. If the rows cannot be read to tell, it warns and makes the version.
|
|
55
|
+
|
|
46
56
|
Refused when:
|
|
47
57
|
|
|
48
58
|
- there is no `dataset.json`, or it describes a table or column the catalog lacks, or has keys the format does not define
|
|
@@ -49,13 +49,13 @@ repo.checkout(store, working, *, ref="HEAD") -> Version | None
|
|
|
49
49
|
Make a working catalog at `working` from the version `ref` holds. Returns that version, or `None` before the first one.
|
|
50
50
|
|
|
51
51
|
```python
|
|
52
|
-
repo.commit(working, out_dir, *, parent, data_path, author, message, inputs=(),
|
|
52
|
+
repo.commit(working, out_dir, *, parent, data_path, author, message, inputs=(), annotations=None, dataset=None) -> Version
|
|
53
53
|
```
|
|
54
54
|
|
|
55
55
|
Freeze the working catalog into `out_dir/<ID>/`, with `dataset` (the parsed `dataset.json`) checked against the catalog and completed from it. `data_path` is `repo.data_url(store)`. Nothing is uploaded.
|
|
56
56
|
|
|
57
57
|
```python
|
|
58
|
-
repo.record(frozen, out_dir, *, parent, author, message, inputs=(),
|
|
58
|
+
repo.record(frozen, out_dir, *, parent, author, message, inputs=(), annotations=None, dataset=None) -> Version
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
Make a version around a catalog that is already frozen, for a writer that freezes it itself.
|
|
@@ -24,7 +24,7 @@ DuckLake's own cleanup (`ducklake_cleanup_old_files`, `ducklake_expire_snapshots
|
|
|
24
24
|
A version is taken out of retention in this order:
|
|
25
25
|
|
|
26
26
|
1. Put `versions/<ID>/expired.json`
|
|
27
|
-
2. Delete the version's catalogs
|
|
27
|
+
2. Delete the version's catalogs, `ducklake.duckdb` and `catalogs/`, and its `files.json`
|
|
28
28
|
3. After the grace period, delete the data files that meet the conditions above
|
|
29
29
|
|
|
30
30
|
```json expired.json
|
|
@@ -21,6 +21,7 @@ Paths are relative to the root, lower case, and separated by `/`. The root is wh
|
|
|
21
21
|
version.json.sig a signature of the record (optional)
|
|
22
22
|
ducklake.duckdb the frozen catalog
|
|
23
23
|
dataset.json what the version holds, described
|
|
24
|
+
files.json each table's data files, indexed from the catalog (optional)
|
|
24
25
|
catalogs/<format>.duckdb the catalog in another DuckLake format (optional)
|
|
25
26
|
expired.json the past-retention mark (only once expired)
|
|
26
27
|
<others> files a host adds, such as previews
|
|
@@ -43,6 +43,11 @@ A version is the files under `versions/<version ID>/`. They must not change once
|
|
|
43
43
|
"snapshot_id": 345,
|
|
44
44
|
"snapshots": [301, 345]
|
|
45
45
|
},
|
|
46
|
+
"files": {
|
|
47
|
+
"path": "files.json",
|
|
48
|
+
"bytes": 1536,
|
|
49
|
+
"sha256": "7a0e…"
|
|
50
|
+
},
|
|
46
51
|
"changes": [
|
|
47
52
|
{ "table": "main.population", "kind": "add_column", "column": "note", "breaking": false }
|
|
48
53
|
],
|
|
@@ -54,7 +59,7 @@ A version is the files under `versions/<version ID>/`. They must not change once
|
|
|
54
59
|
"url": "https://example.org/population-2025.csv"
|
|
55
60
|
}
|
|
56
61
|
],
|
|
57
|
-
"
|
|
62
|
+
"annotations": {}
|
|
58
63
|
}
|
|
59
64
|
```
|
|
60
65
|
|
|
@@ -70,9 +75,10 @@ A version is the files under `versions/<version ID>/`. They must not change once
|
|
|
70
75
|
| `catalog` | must | The catalog's file name, size, sha256, and storage format (`storage`) |
|
|
71
76
|
| `dataset` | must | The `dataset.json` beside the catalog: its file name, size and sha256. Versions written before it was required lack it |
|
|
72
77
|
| `ducklake` | must | The DuckLake format, the snapshot frozen, and the range of snapshots this version adds after its parent (first and last) |
|
|
78
|
+
| `files` | may | The `files.json` beside the catalog, if written: its file name, size and sha256 ([files.json](#filesjson)) |
|
|
73
79
|
| `changes` | may | Column differences from the parent. `breaking` says whether SQL written for the parent may fail (a column dropped, renamed or recreated) |
|
|
74
80
|
| `inputs` | may | Source files used. How `source_id` is assigned is up to the writer |
|
|
75
|
-
| `
|
|
81
|
+
| `annotations` | may | What the format does not interpret, keyed by the name of whoever reads it (a host, a tool) |
|
|
76
82
|
|
|
77
83
|
A reader must ignore fields it does not know.
|
|
78
84
|
|
|
@@ -90,6 +96,38 @@ The description and SQL of each snapshot stay in the catalog, in DuckLake's `com
|
|
|
90
96
|
|
|
91
97
|
A reader can tell from `version.json` alone whether it can open a version.
|
|
92
98
|
|
|
99
|
+
## files.json
|
|
100
|
+
|
|
101
|
+
`files.json` is an index of each table's data files at `ducklake.snapshot_id`, for readers that cannot open the catalog. The writer makes it from the catalog, and the catalog wins where the two differ. It is optional: a reader must work without it. It plays the part of Git's reachability bitmap (`.bitmap` beside a pack): a list, made from the objects, of what one commit reaches, which speeds up reading and which a reader can do without.
|
|
102
|
+
|
|
103
|
+
```json files.json
|
|
104
|
+
{
|
|
105
|
+
"tables": {
|
|
106
|
+
"main.population": {
|
|
107
|
+
"files": [
|
|
108
|
+
{
|
|
109
|
+
"path": "main/population/ducklake-019a3c4e-….parquet",
|
|
110
|
+
"rows": 1741,
|
|
111
|
+
"bytes": 52311,
|
|
112
|
+
"footer_bytes": 2048,
|
|
113
|
+
"deletes": { "path": "main/population/ducklake-019a3c4f-…-delete.parquet", "rows": 3, "bytes": 812 }
|
|
114
|
+
}
|
|
115
|
+
]
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
- `tables` holds every table alive at the snapshot, keyed `schema.table`. A table with no rows has an empty `files`
|
|
122
|
+
- `path` is relative to the repository's `data_dir`. Files are in the catalog's order
|
|
123
|
+
- `rows` and `bytes` are the row count and size the catalog records. `footer_bytes` is the size of the Parquet footer, when recorded
|
|
124
|
+
- `deletes` is the delete file for that data file, if rows were deleted from it
|
|
125
|
+
- `renamed_columns` is `true` when a column of the table was ever renamed. Data files written before the rename carry the old name
|
|
126
|
+
|
|
127
|
+
A reader that reads the Parquet files without DuckLake must not read a table whose files have `deletes`, or whose `renamed_columns` is `true`, unless it applies them as DuckLake does. Otherwise it returns deleted rows or misses renamed columns.
|
|
128
|
+
|
|
129
|
+
When a version is taken out of retention, `files.json` is deleted with its catalogs.
|
|
130
|
+
|
|
93
131
|
## The frozen catalog
|
|
94
132
|
|
|
95
133
|
`ducklake.duckdb` is a DuckLake catalog in DuckDB's file format.
|
|
@@ -137,7 +175,7 @@ The writer describes; the writer's tool checks the description against the froze
|
|
|
137
175
|
]
|
|
138
176
|
}
|
|
139
177
|
],
|
|
140
|
-
"
|
|
178
|
+
"annotations": {}
|
|
141
179
|
}
|
|
142
180
|
```
|
|
143
181
|
|
|
@@ -145,8 +183,10 @@ The writer describes; the writer's tool checks the description against the froze
|
|
|
145
183
|
- A writer may keep its description as YAML (`dataset.yml`), and may keep each table in its own file (`<name>.table.yml`, one entry of `tables`). The version always holds the whole description as one JSON
|
|
146
184
|
- Each of `licenses` is an SPDX identifier, or an object with `id` and what the writer wants to add (`spdx`, `url`, `title`)
|
|
147
185
|
- `tables` lists every table of the catalog's last snapshot. The writer's tool adds the tables the writer did not describe, with what the catalog knows of them, and refuses a described table or column the catalog lacks
|
|
186
|
+
- A view of the catalog is listed after the tables with `"kind": "view"` and its `sql` from the catalog. The catalog records no columns for a view, so its `fields` are the writer's, without `type`, `nullable` or `rows`. `kind` is left out for a table
|
|
148
187
|
- From the catalog: each table's `rows`, and each field's `type` (DuckLake's type name) and `nullable`. Comments in the catalog (`COMMENT ON`) are used where the writer gave no `description`
|
|
149
|
-
- Keys this specification does not define are refused.
|
|
188
|
+
- Keys this specification does not define are refused. What the format does not interpret goes under `annotations`, keyed by the name of whoever reads it, at the top, in a table or in a field. Writers keep it as written
|
|
189
|
+
- Versions written before `annotations` had its name carry it as `extensions`. A reader reads `extensions` as `annotations`
|
|
150
190
|
- What a license or a source says is the writer's claim. A host may check it before it publishes
|
|
151
191
|
|
|
152
192
|
## Catalogs in other formats
|