fdl-cli 0.23.0__tar.gz → 0.24.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/CHANGELOG.md +37 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/PKG-INFO +1 -1
- fdl_cli-0.24.1/cases/collection.json +438 -0
- fdl_cli-0.24.1/docs/ja/spec/collection-cases.md +30 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/data-files.md +14 -1
- fdl_cli-0.24.1/docs/ja/spec/meta.ts +7 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/versions.md +17 -2
- fdl_cli-0.24.1/docs/spec/collection-cases.md +30 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/data-files.md +14 -1
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/meta.ts +1 -1
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/versions.md +17 -2
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/pyproject.toml +1 -1
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/catalog.py +35 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/repo.py +22 -23
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_cli.py +16 -2
- fdl_cli-0.24.1/tests/test_collection_cases.py +46 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_repo.py +74 -2
- fdl_cli-0.23.0/docs/ja/spec/meta.ts +0 -7
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.github/workflows/docs.yml +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.github/workflows/publish.yml +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.github/workflows/release.yml +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.github/workflows/test.yml +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.gitignore +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.node-version +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/.python-version +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/CLAUDE.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/CONTRIBUTING.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/LICENSE +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/README.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/SECURITY.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/blume.config.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/concepts/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/concepts/repositories.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/concepts/versions-and-refs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/concepts/working-catalogs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/getting-started.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/building-a-host.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/mirrors.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/public-and-private.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/reading-without-duckdb.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/serving-over-http.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/guides/stores.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/index.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/internals/from-write-to-read.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/internals/inside-a-repository.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/internals/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/concepts/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/concepts/repositories.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/concepts/versions-and-refs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/concepts/working-catalogs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/getting-started.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/building-a-host.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/mirrors.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/public-and-private.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/reading-without-duckdb.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/serving-over-http.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/guides/stores.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/index.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/internals/ducklake-formats.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/internals/from-write-to-read.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/internals/inside-a-repository.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/internals/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/reference/cli.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/reference/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/reference/python.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/http.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/index.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/layout.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/reading.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/refs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/spec/writing.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/ja/stability.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/reference/cli.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/reference/meta.ts +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/reference/python.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/http.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/index.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/layout.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/reading.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/refs.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/spec/writing.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/docs/stability.md +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/package-lock.json +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/package.json +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/public/CNAME +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/__init__.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/cli.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/dataset.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/diff.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/layout.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/lineage.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/merge.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/objects.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/profiles.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/py.typed +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/refs.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/store.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/stores.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/src/fdl/version.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/http_repo.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_dataset.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_http.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_merge.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_objects.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_profiles.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/tests/test_stores.py +0 -0
- {fdl_cli-0.23.0 → fdl_cli-0.24.1}/uv.lock +0 -0
|
@@ -1,6 +1,43 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
3
|
|
|
4
|
+
## v0.24.1 (2026-11-13)
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
- A version reaches every data file its catalog names
|
|
9
|
+
([#52](https://github.com/queria-io/fdl/pull/52),
|
|
10
|
+
[`88a2bbc`](https://github.com/queria-io/fdl/commit/88a2bbcfa40c71742642eaf61e394fadad781f98))
|
|
11
|
+
|
|
12
|
+
* fix: a version reaches every data file its catalog names
|
|
13
|
+
|
|
14
|
+
The spec says a version reaches a data file when its catalog names it, and fdl push checks every
|
|
15
|
+
such file. gc counted only the files in files.json's tables, which are those at the version's
|
|
16
|
+
snapshot, and so deleted files that older snapshots in a live catalog still name. The next push of
|
|
17
|
+
that catalog was refused for missing files.
|
|
18
|
+
|
|
19
|
+
files.json gains named: every data and delete file the catalog names. gc keeps those, and opens the
|
|
20
|
+
catalog for a files.json without named. The spec says so, and that a withdrawal's clean version
|
|
21
|
+
must expire the snapshots that held the data.
|
|
22
|
+
|
|
23
|
+
* docs(spec): rules for deleting shared objects safely, and shared collection cases
|
|
24
|
+
|
|
25
|
+
Deleting is the one step that cannot be undone. Write down that a process deletes nothing it cannot
|
|
26
|
+
account for, reads the versions again before deleting, deletes by key, and agrees with every other
|
|
27
|
+
implementation on the cases in cases/collection.json.
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
## v0.24.0 (2026-11-13)
|
|
31
|
+
|
|
32
|
+
### Features
|
|
33
|
+
|
|
34
|
+
- List each table's columns in files.json ([#53](https://github.com/queria-io/fdl/pull/53),
|
|
35
|
+
[`98ae9fd`](https://github.com/queria-io/fdl/commit/98ae9fde19ec660778dd653add508b77bd882b54))
|
|
36
|
+
|
|
37
|
+
Readers without DuckDB can map Parquet columns by field_id from the catalog's column IDs and types,
|
|
38
|
+
without reading every file's footer.
|
|
39
|
+
|
|
40
|
+
|
|
4
41
|
## v0.23.0 (2026-11-11)
|
|
5
42
|
|
|
6
43
|
### Bug Fixes
|
|
@@ -0,0 +1,438 @@
|
|
|
1
|
+
{
|
|
2
|
+
"now": "2026-11-12T00:00:00Z",
|
|
3
|
+
"grace_seconds": 86400,
|
|
4
|
+
"cases": [
|
|
5
|
+
{
|
|
6
|
+
"name": "keeps every data file named, not only those at the snapshot",
|
|
7
|
+
"files": {
|
|
8
|
+
"fdl.json": {
|
|
9
|
+
"fdl": "0.1",
|
|
10
|
+
"data_dir": "data/"
|
|
11
|
+
},
|
|
12
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
13
|
+
"fdl": "0.1",
|
|
14
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
15
|
+
"seq": 1,
|
|
16
|
+
"parents": [],
|
|
17
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
18
|
+
"author": {
|
|
19
|
+
"name": "example"
|
|
20
|
+
},
|
|
21
|
+
"message": "",
|
|
22
|
+
"catalog": {
|
|
23
|
+
"path": "ducklake.duckdb",
|
|
24
|
+
"storage": "duckdb-v1.5.0",
|
|
25
|
+
"bytes": 0,
|
|
26
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
27
|
+
},
|
|
28
|
+
"ducklake": {
|
|
29
|
+
"format": "1.0",
|
|
30
|
+
"snapshot_id": 1,
|
|
31
|
+
"snapshots": [
|
|
32
|
+
1
|
|
33
|
+
]
|
|
34
|
+
},
|
|
35
|
+
"files": {
|
|
36
|
+
"path": "files.json",
|
|
37
|
+
"bytes": 0,
|
|
38
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
39
|
+
}
|
|
40
|
+
},
|
|
41
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
42
|
+
"tables": {
|
|
43
|
+
"main.t": {
|
|
44
|
+
"files": [
|
|
45
|
+
{
|
|
46
|
+
"path": "main/t/ducklake-a.parquet",
|
|
47
|
+
"rows": 1,
|
|
48
|
+
"bytes": 1
|
|
49
|
+
}
|
|
50
|
+
]
|
|
51
|
+
}
|
|
52
|
+
},
|
|
53
|
+
"named": [
|
|
54
|
+
"main/t/ducklake-a.parquet",
|
|
55
|
+
"main/t/ducklake-b.parquet",
|
|
56
|
+
"main/t/ducklake-b-delete.parquet"
|
|
57
|
+
]
|
|
58
|
+
}
|
|
59
|
+
},
|
|
60
|
+
"objects": {
|
|
61
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
62
|
+
"data/main/t/ducklake-b.parquet": "2026-11-01T00:00:00Z",
|
|
63
|
+
"data/main/t/ducklake-b-delete.parquet": "2026-11-01T00:00:00Z",
|
|
64
|
+
"data/main/t/ducklake-c.parquet": "2026-11-01T00:00:00Z"
|
|
65
|
+
},
|
|
66
|
+
"deleted": [
|
|
67
|
+
"data/main/t/ducklake-c.parquet"
|
|
68
|
+
]
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
"name": "keeps what was placed within the grace period",
|
|
72
|
+
"files": {
|
|
73
|
+
"fdl.json": {
|
|
74
|
+
"fdl": "0.1",
|
|
75
|
+
"data_dir": "data/"
|
|
76
|
+
},
|
|
77
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
78
|
+
"fdl": "0.1",
|
|
79
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
80
|
+
"seq": 1,
|
|
81
|
+
"parents": [],
|
|
82
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
83
|
+
"author": {
|
|
84
|
+
"name": "example"
|
|
85
|
+
},
|
|
86
|
+
"message": "",
|
|
87
|
+
"catalog": {
|
|
88
|
+
"path": "ducklake.duckdb",
|
|
89
|
+
"storage": "duckdb-v1.5.0",
|
|
90
|
+
"bytes": 0,
|
|
91
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
92
|
+
},
|
|
93
|
+
"ducklake": {
|
|
94
|
+
"format": "1.0",
|
|
95
|
+
"snapshot_id": 1,
|
|
96
|
+
"snapshots": [
|
|
97
|
+
1
|
|
98
|
+
]
|
|
99
|
+
},
|
|
100
|
+
"files": {
|
|
101
|
+
"path": "files.json",
|
|
102
|
+
"bytes": 0,
|
|
103
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
104
|
+
}
|
|
105
|
+
},
|
|
106
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
107
|
+
"tables": {
|
|
108
|
+
"main.t": {
|
|
109
|
+
"files": [
|
|
110
|
+
{
|
|
111
|
+
"path": "main/t/ducklake-a.parquet",
|
|
112
|
+
"rows": 1,
|
|
113
|
+
"bytes": 1
|
|
114
|
+
}
|
|
115
|
+
]
|
|
116
|
+
}
|
|
117
|
+
},
|
|
118
|
+
"named": [
|
|
119
|
+
"main/t/ducklake-a.parquet"
|
|
120
|
+
]
|
|
121
|
+
}
|
|
122
|
+
},
|
|
123
|
+
"objects": {
|
|
124
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
125
|
+
"data/main/t/ducklake-b.parquet": "2026-11-11T12:00:00Z",
|
|
126
|
+
"data/main/t/ducklake-c.parquet": "2026-11-01T00:00:00Z"
|
|
127
|
+
},
|
|
128
|
+
"deleted": [
|
|
129
|
+
"data/main/t/ducklake-c.parquet"
|
|
130
|
+
]
|
|
131
|
+
},
|
|
132
|
+
{
|
|
133
|
+
"name": "deletes what only a version past retention reached",
|
|
134
|
+
"files": {
|
|
135
|
+
"fdl.json": {
|
|
136
|
+
"fdl": "0.1",
|
|
137
|
+
"data_dir": "data/"
|
|
138
|
+
},
|
|
139
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
140
|
+
"fdl": "0.1",
|
|
141
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
142
|
+
"seq": 1,
|
|
143
|
+
"parents": [],
|
|
144
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
145
|
+
"author": {
|
|
146
|
+
"name": "example"
|
|
147
|
+
},
|
|
148
|
+
"message": "",
|
|
149
|
+
"catalog": {
|
|
150
|
+
"path": "ducklake.duckdb",
|
|
151
|
+
"storage": "duckdb-v1.5.0",
|
|
152
|
+
"bytes": 0,
|
|
153
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
154
|
+
},
|
|
155
|
+
"ducklake": {
|
|
156
|
+
"format": "1.0",
|
|
157
|
+
"snapshot_id": 1,
|
|
158
|
+
"snapshots": [
|
|
159
|
+
1
|
|
160
|
+
]
|
|
161
|
+
}
|
|
162
|
+
},
|
|
163
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/expired.json": {
|
|
164
|
+
"expired_at": "2026-11-01T00:00:00Z",
|
|
165
|
+
"reason": "retention"
|
|
166
|
+
},
|
|
167
|
+
"versions/019a3d00-1111-7aaa-8bbb-2f0d8e1a4b6c/version.json": {
|
|
168
|
+
"fdl": "0.1",
|
|
169
|
+
"id": "019a3d00-1111-7aaa-8bbb-2f0d8e1a4b6c",
|
|
170
|
+
"seq": 2,
|
|
171
|
+
"parents": [
|
|
172
|
+
"019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c"
|
|
173
|
+
],
|
|
174
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
175
|
+
"author": {
|
|
176
|
+
"name": "example"
|
|
177
|
+
},
|
|
178
|
+
"message": "",
|
|
179
|
+
"catalog": {
|
|
180
|
+
"path": "ducklake.duckdb",
|
|
181
|
+
"storage": "duckdb-v1.5.0",
|
|
182
|
+
"bytes": 0,
|
|
183
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
184
|
+
},
|
|
185
|
+
"ducklake": {
|
|
186
|
+
"format": "1.0",
|
|
187
|
+
"snapshot_id": 1,
|
|
188
|
+
"snapshots": [
|
|
189
|
+
1
|
|
190
|
+
]
|
|
191
|
+
},
|
|
192
|
+
"files": {
|
|
193
|
+
"path": "files.json",
|
|
194
|
+
"bytes": 0,
|
|
195
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
196
|
+
}
|
|
197
|
+
},
|
|
198
|
+
"versions/019a3d00-1111-7aaa-8bbb-2f0d8e1a4b6c/files.json": {
|
|
199
|
+
"tables": {
|
|
200
|
+
"main.t": {
|
|
201
|
+
"files": [
|
|
202
|
+
{
|
|
203
|
+
"path": "main/t/ducklake-b.parquet",
|
|
204
|
+
"rows": 1,
|
|
205
|
+
"bytes": 1
|
|
206
|
+
}
|
|
207
|
+
]
|
|
208
|
+
}
|
|
209
|
+
},
|
|
210
|
+
"named": [
|
|
211
|
+
"main/t/ducklake-b.parquet"
|
|
212
|
+
]
|
|
213
|
+
}
|
|
214
|
+
},
|
|
215
|
+
"objects": {
|
|
216
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
217
|
+
"data/main/t/ducklake-b.parquet": "2026-11-01T00:00:00Z"
|
|
218
|
+
},
|
|
219
|
+
"deleted": [
|
|
220
|
+
"data/main/t/ducklake-a.parquet"
|
|
221
|
+
]
|
|
222
|
+
},
|
|
223
|
+
{
|
|
224
|
+
"name": "deletes no data file while a live version's files.json has no named",
|
|
225
|
+
"files": {
|
|
226
|
+
"fdl.json": {
|
|
227
|
+
"fdl": "0.1",
|
|
228
|
+
"data_dir": "data/"
|
|
229
|
+
},
|
|
230
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
231
|
+
"fdl": "0.1",
|
|
232
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
233
|
+
"seq": 1,
|
|
234
|
+
"parents": [],
|
|
235
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
236
|
+
"author": {
|
|
237
|
+
"name": "example"
|
|
238
|
+
},
|
|
239
|
+
"message": "",
|
|
240
|
+
"catalog": {
|
|
241
|
+
"path": "ducklake.duckdb",
|
|
242
|
+
"storage": "duckdb-v1.5.0",
|
|
243
|
+
"bytes": 0,
|
|
244
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
245
|
+
},
|
|
246
|
+
"ducklake": {
|
|
247
|
+
"format": "1.0",
|
|
248
|
+
"snapshot_id": 1,
|
|
249
|
+
"snapshots": [
|
|
250
|
+
1
|
|
251
|
+
]
|
|
252
|
+
},
|
|
253
|
+
"files": {
|
|
254
|
+
"path": "files.json",
|
|
255
|
+
"bytes": 0,
|
|
256
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
257
|
+
}
|
|
258
|
+
},
|
|
259
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
260
|
+
"tables": {
|
|
261
|
+
"main.t": {
|
|
262
|
+
"files": [
|
|
263
|
+
{
|
|
264
|
+
"path": "main/t/ducklake-a.parquet",
|
|
265
|
+
"rows": 1,
|
|
266
|
+
"bytes": 1
|
|
267
|
+
}
|
|
268
|
+
]
|
|
269
|
+
}
|
|
270
|
+
}
|
|
271
|
+
}
|
|
272
|
+
},
|
|
273
|
+
"objects": {
|
|
274
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
275
|
+
"data/main/t/ducklake-b.parquet": "2026-11-01T00:00:00Z"
|
|
276
|
+
},
|
|
277
|
+
"deleted": []
|
|
278
|
+
},
|
|
279
|
+
{
|
|
280
|
+
"name": "deletes nothing while a live version has no files.json and no catalog",
|
|
281
|
+
"files": {
|
|
282
|
+
"fdl.json": {
|
|
283
|
+
"fdl": "0.1",
|
|
284
|
+
"data_dir": "data/"
|
|
285
|
+
},
|
|
286
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
287
|
+
"fdl": "0.1",
|
|
288
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
289
|
+
"seq": 1,
|
|
290
|
+
"parents": [],
|
|
291
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
292
|
+
"author": {
|
|
293
|
+
"name": "example"
|
|
294
|
+
},
|
|
295
|
+
"message": "",
|
|
296
|
+
"catalog": {
|
|
297
|
+
"path": "ducklake.duckdb",
|
|
298
|
+
"storage": "duckdb-v1.5.0",
|
|
299
|
+
"bytes": 0,
|
|
300
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
301
|
+
},
|
|
302
|
+
"ducklake": {
|
|
303
|
+
"format": "1.0",
|
|
304
|
+
"snapshot_id": 1,
|
|
305
|
+
"snapshots": [
|
|
306
|
+
1
|
|
307
|
+
]
|
|
308
|
+
}
|
|
309
|
+
}
|
|
310
|
+
},
|
|
311
|
+
"objects": {
|
|
312
|
+
"data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z",
|
|
313
|
+
"assets/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": "2026-11-01T00:00:00Z"
|
|
314
|
+
},
|
|
315
|
+
"deleted": []
|
|
316
|
+
},
|
|
317
|
+
{
|
|
318
|
+
"name": "keeps the assets a live version's tree lists",
|
|
319
|
+
"files": {
|
|
320
|
+
"fdl.json": {
|
|
321
|
+
"fdl": "0.1",
|
|
322
|
+
"data_dir": "data/"
|
|
323
|
+
},
|
|
324
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
325
|
+
"fdl": "0.1",
|
|
326
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
327
|
+
"seq": 1,
|
|
328
|
+
"parents": [],
|
|
329
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
330
|
+
"author": {
|
|
331
|
+
"name": "example"
|
|
332
|
+
},
|
|
333
|
+
"message": "",
|
|
334
|
+
"catalog": {
|
|
335
|
+
"path": "ducklake.duckdb",
|
|
336
|
+
"storage": "duckdb-v1.5.0",
|
|
337
|
+
"bytes": 0,
|
|
338
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
339
|
+
},
|
|
340
|
+
"ducklake": {
|
|
341
|
+
"format": "1.0",
|
|
342
|
+
"snapshot_id": 1,
|
|
343
|
+
"snapshots": [
|
|
344
|
+
1
|
|
345
|
+
]
|
|
346
|
+
},
|
|
347
|
+
"files": {
|
|
348
|
+
"path": "files.json",
|
|
349
|
+
"bytes": 0,
|
|
350
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
351
|
+
},
|
|
352
|
+
"tree": {
|
|
353
|
+
"path": "tree.json",
|
|
354
|
+
"bytes": 0,
|
|
355
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
356
|
+
}
|
|
357
|
+
},
|
|
358
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
359
|
+
"tables": {
|
|
360
|
+
"main.t": {
|
|
361
|
+
"files": []
|
|
362
|
+
}
|
|
363
|
+
},
|
|
364
|
+
"named": []
|
|
365
|
+
},
|
|
366
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/tree.json": {
|
|
367
|
+
"entries": [
|
|
368
|
+
{
|
|
369
|
+
"path": "README.md",
|
|
370
|
+
"sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
|
371
|
+
"bytes": 1
|
|
372
|
+
}
|
|
373
|
+
]
|
|
374
|
+
}
|
|
375
|
+
},
|
|
376
|
+
"objects": {
|
|
377
|
+
"assets/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa": "2026-11-01T00:00:00Z",
|
|
378
|
+
"assets/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": "2026-11-01T00:00:00Z"
|
|
379
|
+
},
|
|
380
|
+
"deleted": [
|
|
381
|
+
"assets/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
|
382
|
+
]
|
|
383
|
+
},
|
|
384
|
+
{
|
|
385
|
+
"name": "leaves alone what is not a shared object",
|
|
386
|
+
"files": {
|
|
387
|
+
"fdl.json": {
|
|
388
|
+
"fdl": "0.1",
|
|
389
|
+
"data_dir": "data/"
|
|
390
|
+
},
|
|
391
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/version.json": {
|
|
392
|
+
"fdl": "0.1",
|
|
393
|
+
"id": "019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c",
|
|
394
|
+
"seq": 1,
|
|
395
|
+
"parents": [],
|
|
396
|
+
"created_at": "2026-11-01T00:00:00Z",
|
|
397
|
+
"author": {
|
|
398
|
+
"name": "example"
|
|
399
|
+
},
|
|
400
|
+
"message": "",
|
|
401
|
+
"catalog": {
|
|
402
|
+
"path": "ducklake.duckdb",
|
|
403
|
+
"storage": "duckdb-v1.5.0",
|
|
404
|
+
"bytes": 0,
|
|
405
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
406
|
+
},
|
|
407
|
+
"ducklake": {
|
|
408
|
+
"format": "1.0",
|
|
409
|
+
"snapshot_id": 1,
|
|
410
|
+
"snapshots": [
|
|
411
|
+
1
|
|
412
|
+
]
|
|
413
|
+
},
|
|
414
|
+
"files": {
|
|
415
|
+
"path": "files.json",
|
|
416
|
+
"bytes": 0,
|
|
417
|
+
"sha256": "0000000000000000000000000000000000000000000000000000000000000000"
|
|
418
|
+
}
|
|
419
|
+
},
|
|
420
|
+
"versions/019a3c4e-7b21-7f3a-9c55-2f0d8e1a4b6c/files.json": {
|
|
421
|
+
"tables": {
|
|
422
|
+
"main.t": {
|
|
423
|
+
"files": []
|
|
424
|
+
}
|
|
425
|
+
},
|
|
426
|
+
"named": []
|
|
427
|
+
}
|
|
428
|
+
},
|
|
429
|
+
"objects": {
|
|
430
|
+
"data/main/t/other.parquet": "2026-11-01T00:00:00Z",
|
|
431
|
+
"data/notes.txt": "2026-11-01T00:00:00Z",
|
|
432
|
+
"assets/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA": "2026-11-01T00:00:00Z",
|
|
433
|
+
"assets/abc": "2026-11-01T00:00:00Z"
|
|
434
|
+
},
|
|
435
|
+
"deleted": []
|
|
436
|
+
}
|
|
437
|
+
]
|
|
438
|
+
}
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: 収集のテストケース
|
|
3
|
+
description: 共有オブジェクトを消す処理のテストケース。すべての実装が実行し、同じ結果になります。
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
[`cases/collection.json`](https://github.com/queria-io/fdl/blob/main/cases/collection.json) には、リポジトリと、それぞれで共有オブジェクトを消す処理([データファイル](/ja/spec/data-files#共有オブジェクトを消してよいとき))が消すオブジェクトが入っています。共有オブジェクトを消す実装は、すべてのケースを実行し、ちょうどそれらを消します。エラーで止まったときは何も消さなかったものとして扱うので、何も消さないケースでだけ一致します。
|
|
7
|
+
|
|
8
|
+
```json
|
|
9
|
+
{
|
|
10
|
+
"now": "2026-11-12T00:00:00Z",
|
|
11
|
+
"grace_seconds": 86400,
|
|
12
|
+
"cases": [
|
|
13
|
+
{
|
|
14
|
+
"name": "keeps every data file named, not only those at the snapshot",
|
|
15
|
+
"files": { "fdl.json": {}, "versions/<ID>/version.json": {}, "versions/<ID>/files.json": {} },
|
|
16
|
+
"objects": { "data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z" },
|
|
17
|
+
"deleted": ["data/main/t/ducklake-c.parquet"]
|
|
18
|
+
}
|
|
19
|
+
]
|
|
20
|
+
}
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
| キー | 中身 |
|
|
24
|
+
| --- | --- |
|
|
25
|
+
| `now`、`grace_seconds` | 処理を実行する時刻と、猶予期間 |
|
|
26
|
+
| `files` | 共有オブジェクト以外のリポジトリのファイル。キーごとに、ファイルの JSON |
|
|
27
|
+
| `objects` | データディレクトリと `assets/` の下のオブジェクト。キーごとに、置いた時刻。中身は問いません |
|
|
28
|
+
| `deleted` | 消すキー |
|
|
29
|
+
|
|
30
|
+
消してよいものを変えるときは、この仕様と同じ変更でここにケースを追加します。
|
|
@@ -33,8 +33,21 @@ description: 共有オブジェクト(データファイル・アセット)
|
|
|
33
33
|
|
|
34
34
|
共有オブジェクトは、パスが上の表の形をしたファイルだけです。データディレクトリの下の `ducklake-<名前>.parquet` か、`assets/` に続く小文字 16 進 64 桁です。共有オブジェクトを消す処理は、それ以外のファイルを消してはいけません。
|
|
35
35
|
|
|
36
|
+
バージョンのカタログは、`ducklake.snapshot_id` の時点だけでなく、残しているすべてのスナップショットのデータファイルを名指しします。そのため、`files.json` の `tables` にあるファイルが到達するファイルのすべてではありません。カタログを開けない処理は `files.json` の `named` を使います。`files.json` が無いか、`named` の無い `files.json` を持つ有効なバージョンがあるときは、そのカタログを開くか、データファイルを消しません。
|
|
37
|
+
|
|
36
38
|
DuckLake 自身の片付け(`ducklake_cleanup_old_files`、`ducklake_expire_snapshots`)は 1 つのカタログしか見ないので、リポジトリから消すのに使ってはいけません。消すのは、リポジトリ全体を見る処理の役目です。
|
|
37
39
|
|
|
40
|
+
## 安全に消す
|
|
41
|
+
|
|
42
|
+
消した共有オブジェクトは戻せず、それに到達するバージョンはすべて読めなくなります。共有オブジェクトを消す処理は、次の規則に従います。
|
|
43
|
+
|
|
44
|
+
1. 判定できないときは消さない。有効なバージョンが何に到達するかを判定できないとき、つまり `version.json`・`files.json`・`tree.json` を読めないか解釈できないとき、または `files.json` に `named` が無くカタログも開けないときは、その種類のオブジェクトを消しません。どのデータファイルに到達するか判定できなければデータファイルを、どのアセットに到達するか判定できなければアセットを消しません。処理を止めて何も消さないのでもかまいません。残しすぎたオブジェクトは容量を使うだけですが、早く消しすぎたオブジェクトはバージョンを壊します
|
|
45
|
+
2. 消す直前にバージョンを読み直す。処理が判定している間に、書き手がバージョンを作ることがあります。判定のときに無かったバージョンが到達するオブジェクトは残します
|
|
46
|
+
3. 判定したオブジェクトを 1 つずつキーで消す。プレフィックスやディレクトリごとには消しません
|
|
47
|
+
4. このページの実装は、[収集のテストケース](/ja/spec/collection-cases) で同じ結果になる。そこにある各リポジトリについて、消すオブジェクトが一致します
|
|
48
|
+
|
|
49
|
+
処理は、オブジェクトを読み手が読まない場所(別のバケットなど)へ移し、決めた時間が過ぎてからそこで消してもかまいません。そうすると、間違えたときに戻せます。書き手は、そこにあるオブジェクトをあるものとして数えません。
|
|
50
|
+
|
|
38
51
|
## 保持期間を過ぎたとき
|
|
39
52
|
|
|
40
53
|
バージョンの保持期間は、次の順に終わらせます。
|
|
@@ -55,7 +68,7 @@ DuckLake 自身の片付け(`ducklake_cleanup_old_files`、`ducklake_expire_sn
|
|
|
55
68
|
|
|
56
69
|
取り下げは、間違って公開した個人情報のように、公開してはいけないデータを消すための手順です。保持期間と違い、そのデータを持つバージョンを、有効なものも含めてすべて対象にし、猶予期間を待ちません。
|
|
57
70
|
|
|
58
|
-
1.
|
|
71
|
+
1. そのデータを除いた新しいバージョンを作り、該当するバージョンを指しているブランチをすべてそこへ動かす。新しいバージョンの親はいまのバージョンなので、前へ動かすことになる。カタログは名指しするすべてのファイルに到達するので、新しいバージョンのカタログにはそのデータを持つスナップショットを残さない(作業カタログで `ducklake_expire_snapshots` を実行する)
|
|
59
72
|
2. 該当するバージョンを指すタグを消す
|
|
60
73
|
3. 個人情報を持つデータファイルや、それが写ったスクリーンショットなど、そのデータを持つ共有オブジェクトをすぐに消す。該当するバージョンから到達し、残すバージョンのどれからも到達しないものが対象。そのオブジェクトに到達するバージョンは、すべて該当するバージョンになる。該当するバージョンに保持期間を過ぎたものがあると、カタログが無いので、それがどのオブジェクトに到達していたかが分からない。その場合は、有効なバージョンのどれからも到達しない共有オブジェクトもすべてすぐに消す
|
|
61
74
|
4. 該当するバージョンそれぞれに `"reason": "withdrawn"` の `expired.json` を置き、カタログを消す
|
|
@@ -152,9 +152,22 @@ description: バージョン ID、version.json の項目、固めたカタログ
|
|
|
152
152
|
"footer_bytes": 2048,
|
|
153
153
|
"deletes": { "path": "main/population/ducklake-019a3c4f-…-delete.parquet", "rows": 3, "bytes": 812 }
|
|
154
154
|
}
|
|
155
|
+
],
|
|
156
|
+
"columns": [
|
|
157
|
+
{ "id": 1, "name": "code", "type": "varchar", "nullable": false },
|
|
158
|
+
{ "id": 2, "name": "population", "type": "int64", "nullable": true, "default": "0" },
|
|
159
|
+
{
|
|
160
|
+
"id": 3, "name": "tags", "type": "list", "nullable": true,
|
|
161
|
+
"children": [{ "id": 4, "name": "element", "type": "varchar", "nullable": true }]
|
|
162
|
+
}
|
|
155
163
|
]
|
|
156
164
|
}
|
|
157
|
-
}
|
|
165
|
+
},
|
|
166
|
+
"named": [
|
|
167
|
+
"main/population/ducklake-019a3c4e-….parquet",
|
|
168
|
+
"main/population/ducklake-019a3c4f-…-delete.parquet",
|
|
169
|
+
"main/population/ducklake-0199f2a1-….parquet"
|
|
170
|
+
]
|
|
158
171
|
}
|
|
159
172
|
```
|
|
160
173
|
|
|
@@ -162,9 +175,11 @@ description: バージョン ID、version.json の項目、固めたカタログ
|
|
|
162
175
|
- `path` はリポジトリの `data_dir` からの相対パスです。ファイルはカタログの順に並べます
|
|
163
176
|
- `rows` と `bytes` は、カタログが記録している行数と大きさです。`footer_bytes` は Parquet のフッターの大きさで、記録があるときだけ書きます
|
|
164
177
|
- `deletes` は、そのデータファイルから行を消したときの削除ファイルです
|
|
178
|
+
- `columns` は、そのスナップショットでの表の列をカタログの順に並べたものです。`id` は DuckLake の列 ID で、DuckLake はこれをどの Parquet データファイルにも列の `field_id` として書きます。列の名前を変えても `id` は変わりません。`type` は DuckLake の型名で、`nullable` は `NOT NULL` の列で `false` になります。`default` は、列を追加する前に書いたデータファイルの行がその列に持つ値を文字列にしたもので、NULL でないときだけ書きます。`struct` はフィールドを、`list` は `element` を、`map` は `key` と `value` を `children` に持ちます。fdl-cli 0.24 より前の書き手は `columns` を書きません
|
|
165
179
|
- `renamed_columns` は、その表の列の名前を変えたことがあるときに `true` にします。名前を変える前に書いたデータファイルは、古い名前のままです
|
|
180
|
+
- `named` には、カタログがどのスナップショットででも名指しするデータファイルと削除ファイルを、`data_dir` からの相対パスですべて入れます。これがそのバージョンから到達するデータファイルです([データファイル](/ja/spec/data-files))。カタログが古いスナップショットを残しているときは、`tables` より多くなります
|
|
166
181
|
|
|
167
|
-
DuckLake を使わずに Parquet ファイルを読む読み手は、`deletes` を持つファイルがある表や、`renamed_columns` が `true` の表を読んではいけません。DuckLake
|
|
182
|
+
DuckLake を使わずに Parquet ファイルを読む読み手は、`deletes` を持つファイルがある表や、`renamed_columns` が `true` の表を読んではいけません。DuckLake と同じように削除と改名を当てはめる場合は読んでかまいません。当てはめずに読むと、消した行が返ったり、名前を変えた列が欠けたりします。`columns` の `id` を使い、データファイルの列を `field_id` で探す読み手は、名前を変えた列も DuckLake と同じように読めます。
|
|
168
183
|
|
|
169
184
|
保持期間から外すとき、`files.json` はカタログと一緒に消します。
|
|
170
185
|
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Collection cases
|
|
3
|
+
description: Test cases for deleting shared objects, which every implementation runs and agrees on.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
[`cases/collection.json`](https://github.com/queria-io/fdl/blob/main/cases/collection.json) holds repositories and, for each, the shared objects a process that deletes them ([Data files](/spec/data-files#when-a-shared-object-may-be-deleted)) deletes. An implementation that deletes shared objects runs every case and deletes exactly those. Stopping with an error counts as deleting nothing, so it meets a case only when that case deletes nothing.
|
|
7
|
+
|
|
8
|
+
```json
|
|
9
|
+
{
|
|
10
|
+
"now": "2026-11-12T00:00:00Z",
|
|
11
|
+
"grace_seconds": 86400,
|
|
12
|
+
"cases": [
|
|
13
|
+
{
|
|
14
|
+
"name": "keeps every data file named, not only those at the snapshot",
|
|
15
|
+
"files": { "fdl.json": {}, "versions/<ID>/version.json": {}, "versions/<ID>/files.json": {} },
|
|
16
|
+
"objects": { "data/main/t/ducklake-a.parquet": "2026-11-01T00:00:00Z" },
|
|
17
|
+
"deleted": ["data/main/t/ducklake-c.parquet"]
|
|
18
|
+
}
|
|
19
|
+
]
|
|
20
|
+
}
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
| Key | Holds |
|
|
24
|
+
| --- | --- |
|
|
25
|
+
| `now`, `grace_seconds` | The time the process runs, and the grace period |
|
|
26
|
+
| `files` | The repository's files other than shared objects, by key. Each value is the file's JSON |
|
|
27
|
+
| `objects` | The objects under the data directory and `assets/`, by key, with the time each was placed. Their content does not matter |
|
|
28
|
+
| `deleted` | The keys deleted |
|
|
29
|
+
|
|
30
|
+
A change to what may be deleted comes with a case here, in the same change as this specification.
|
|
@@ -33,8 +33,21 @@ Only when both hold:
|
|
|
33
33
|
|
|
34
34
|
Only a file whose path has the form in the table above is a shared object: a data file named `ducklake-<name>.parquet` under the data directory, or `assets/` followed by 64 lower-case hex digits. A process that deletes shared objects must leave every other file alone.
|
|
35
35
|
|
|
36
|
+
A version's catalog names data files at every snapshot it keeps, not only at `ducklake.snapshot_id`, so the files in `tables` of `files.json` are not all it reaches. A process that cannot open the catalog uses `named` of `files.json`. When a live version has no `files.json`, or one without `named`, the process opens its catalog or deletes no data files.
|
|
37
|
+
|
|
36
38
|
DuckLake's own cleanup (`ducklake_cleanup_old_files`, `ducklake_expire_snapshots`) sees only one catalog, so it must not be used to delete from a repository. Deleting is for a process that sees the whole repository.
|
|
37
39
|
|
|
40
|
+
## Deleting safely
|
|
41
|
+
|
|
42
|
+
A deleted shared object cannot be brought back, and every version that reaches it can no longer be read. A process that deletes shared objects follows these rules:
|
|
43
|
+
|
|
44
|
+
1. Delete nothing it cannot account for. When it cannot tell what a live version reaches, because a `version.json`, `files.json` or `tree.json` cannot be read or parsed, or `files.json` has no `named` and the catalog cannot be opened, it deletes none of that kind: no data file when it cannot tell which data files the version reaches, no asset when it cannot tell which assets. Stopping without deleting anything also meets this. An object kept too long only costs storage; an object deleted too early breaks a version
|
|
45
|
+
2. Read the versions again right before deleting. A writer may make a version while the process decides. Objects a version it did not see when deciding reaches are kept
|
|
46
|
+
3. Delete the objects it decided on, one key at a time. It never deletes by prefix or a whole directory
|
|
47
|
+
4. Implementations of this page agree on the cases in [Collection cases](/spec/collection-cases): for each repository there, the objects to delete are the same
|
|
48
|
+
|
|
49
|
+
A process may move an object to a place readers never read from, such as a separate bucket, and delete it there after a set time, so that a mistake can be undone. A writer does not count an object there as present.
|
|
50
|
+
|
|
38
51
|
## Past retention
|
|
39
52
|
|
|
40
53
|
A version is taken out of retention in this order:
|
|
@@ -55,7 +68,7 @@ A version is taken out of retention in this order:
|
|
|
55
68
|
|
|
56
69
|
Withdrawal removes data that must not be published, such as personal data published by mistake. Unlike retention, it removes data from every version that holds it, live ones included, and does not wait for the grace period.
|
|
57
70
|
|
|
58
|
-
1. Make a new version without the data and move every branch that holds an affected version to it. This is a forward move: the new version's parent is the current one
|
|
71
|
+
1. Make a new version without the data and move every branch that holds an affected version to it. This is a forward move: the new version's parent is the current one. Its catalog must not keep a snapshot that holds the data (`ducklake_expire_snapshots` in the working catalog), since a catalog reaches every file it names
|
|
59
72
|
2. Delete the tags that point at affected versions
|
|
60
73
|
3. Delete the shared objects that hold the data, at once, such as a data file with personal data or a screenshot showing it: those an affected version reaches and no version kept reaches. Every version that reaches such an object is an affected version. An affected version already past retention has no catalog to say which objects it reached, so when there is one, every shared object no live version reaches is deleted at once too
|
|
61
74
|
4. For every affected version, put `expired.json` with `"reason": "withdrawn"` and delete its catalogs
|