@devultur/convex 1.5.0 → 1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +40 -3
- package/dist/chunk-AUAWZ5BD.js +65 -0
- package/dist/chunk-FOBFHC4A.js +55 -0
- package/dist/chunk-O76FU54F.js +0 -0
- package/dist/component/http.js +2 -2
- package/dist/component/library.d.ts +2 -2
- package/dist/component/library.js +3 -3
- package/dist/component/lock.js +3 -3
- package/dist/component/mediaClient.js +1 -1
- package/dist/component/metadata.js +2 -2
- package/dist/component/public.js +9 -9
- package/dist/component/uploads.js +6 -6
- package/dist/component/workerErrors.d.ts +1 -1
- package/dist/component/workerErrors.js +1 -1
- package/dist/contexts-6F8TJINX.d.ts +149 -0
- package/dist/index.d.ts +53 -16
- package/dist/index.js +1 -0
- package/dist/react/contexts.d.ts +6 -0
- package/dist/react/contexts.js +22 -0
- package/dist/react/hooks.d.ts +39 -0
- package/dist/react/hooks.js +28 -0
- package/dist/react.d.ts +125 -277
- package/dist/react.js +788 -829
- package/docs/guides/object-lock.md +517 -0
- package/docs/guides/plugins.md +205 -0
- package/docs/guides/self-host.md +523 -0
- package/docs/guides/video-player.md +105 -0
- package/package.json +10 -5
|
@@ -0,0 +1,517 @@
|
|
|
1
|
+
# Retention and holds
|
|
2
|
+
|
|
3
|
+
Keep a file from being deleted or overwritten until a date, or until you say so (WORM storage). The media Worker
|
|
4
|
+
enforces it on every route that deletes or replaces stored bytes.
|
|
5
|
+
|
|
6
|
+
- [The model](#the-model)
|
|
7
|
+
- [Compliance and governance](#compliance-and-governance)
|
|
8
|
+
- [What's enforced, and where](#whats-enforced-and-where)
|
|
9
|
+
- [Derived files](#derived-files)
|
|
10
|
+
- [Transcoding a locked video](#transcoding-a-locked-video)
|
|
11
|
+
- [Locking at upload](#locking-at-upload)
|
|
12
|
+
- [Retention vs expiry](#retention-vs-expiry)
|
|
13
|
+
- [Changing a lock](#changing-a-lock)
|
|
14
|
+
- [Holds](#holds)
|
|
15
|
+
- [Audit](#audit)
|
|
16
|
+
- [Deleting a prefix from your own code](#deleting-a-prefix-from-your-own-code)
|
|
17
|
+
- [Caveats](#caveats)
|
|
18
|
+
- [Recipes](#recipes)
|
|
19
|
+
|
|
20
|
+
## The model
|
|
21
|
+
|
|
22
|
+
```typescript
|
|
23
|
+
type ObjectLock = {
|
|
24
|
+
retention?: {
|
|
25
|
+
mode: "governance" | "compliance";
|
|
26
|
+
until?: string; // a fixed date (ISO 8601 with a time zone)
|
|
27
|
+
period?: number; // or seconds from upload, or from the hold's release
|
|
28
|
+
derivatives?: "inherit" | "original-only"; // default "inherit"
|
|
29
|
+
};
|
|
30
|
+
hold?: boolean; // retained indefinitely while true
|
|
31
|
+
};
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
- **Retention** keeps the file until `until`, or for `period` seconds (at most 100 years). Set one of the two: passing
|
|
35
|
+
both is a `400`. A `period` counts from the upload, or from the moment a hold is released.
|
|
36
|
+
- **Hold** keeps the file indefinitely, whatever its retention says. Releasing it starts the retention's `period`.
|
|
37
|
+
If there's no retention, the file is free again.
|
|
38
|
+
|
|
39
|
+
If you know S3 Object Lock or Google Cloud Storage, it's the same model:
|
|
40
|
+
|
|
41
|
+
| Here | S3 Object Lock | Google Cloud Storage |
|
|
42
|
+
| --- | --- | --- |
|
|
43
|
+
| `retention.mode: "compliance"` | Compliance mode | Object retention, `Locked` mode |
|
|
44
|
+
| `retention.mode: "governance"` | Governance mode (`s3:BypassGovernanceRetention`) | Object retention, `Unlocked` mode |
|
|
45
|
+
| `retention.until` | Retain until date | Retention expiration time |
|
|
46
|
+
| `hold` | Legal hold | Temporary hold |
|
|
47
|
+
| `hold` + `retention.period` | (no equivalent) | Event-based hold + retention period |
|
|
48
|
+
|
|
49
|
+
**Resolution order.** It works like `visibility` and the [security flags](https://github.com/devultur/devultur/blob/main/docs/guides/security.md): the resource's own lock,
|
|
50
|
+
then the Worker's default (`config.lock`), then no lock. `hold` resolves on its own. `retention` resolves as a whole:
|
|
51
|
+
its mode and its date come from the same level, so `until` and `period` never mix. `derivatives` falls back to the
|
|
52
|
+
default's, then to `inherit`. A resource can replace the default retention but can't drop it.
|
|
53
|
+
|
|
54
|
+
A past `until` is valid. It just means the file isn't locked any more, so a default with a fixed date keeps working
|
|
55
|
+
after that date passes.
|
|
56
|
+
|
|
57
|
+
## Compliance and governance
|
|
58
|
+
|
|
59
|
+
- **`compliance`**: nobody can delete or overwrite the file before its date, and the lock can only get stricter. Its
|
|
60
|
+
date can move later, it can't switch to `governance`, and it can't move from `inherit` to `original-only`. Not even
|
|
61
|
+
the Worker secret can shorten it.
|
|
62
|
+
- **`governance`**: the same, except that a privileged call can shorten or remove it. Privileged means the request
|
|
63
|
+
asks for `bypassGovernance: true` **and** an [`authorizeLockChange`](plugins.md#hooks) plugin approves it. Without
|
|
64
|
+
such a plugin, nothing can shorten a governance retention (`403 lock_privilege_required`).
|
|
65
|
+
|
|
66
|
+
Any lock can be made stricter without privilege: set a later date, a longer period, `inherit`, or a hold. An expired
|
|
67
|
+
retention can be replaced freely.
|
|
68
|
+
|
|
69
|
+
While a hold is set, a `period` can't be swapped for a fixed `until`. The period only starts at the release, so any
|
|
70
|
+
date could end before it: compliance refuses it, and governance needs the privilege. Release the hold first, or
|
|
71
|
+
extend the period.
|
|
72
|
+
|
|
73
|
+
## What's enforced, and where
|
|
74
|
+
|
|
75
|
+
The Worker checks the lock before it deletes or replaces anything:
|
|
76
|
+
|
|
77
|
+
| Path | A locked file |
|
|
78
|
+
| --- | --- |
|
|
79
|
+
| `DELETE /v1/media/<key>` | `409 retention_active`. Its image variants and encryption keys stay too. |
|
|
80
|
+
| `DELETE /v1/media/videos/<id>` | `409 retention_active` when any of the video's originals is locked. Nothing is deleted. |
|
|
81
|
+
| Deleting a file's encryption keys | Refused the same way, since deleting them destroys the file (crypto-shredding) |
|
|
82
|
+
| `PUT /v1/upload/<key>`, multipart create and complete | `409 retention_active` when they would replace a locked file |
|
|
83
|
+
| `POST /v1/upload` with `overwrite: true` | `409 retention_active` before any bytes are sent |
|
|
84
|
+
| `POST /v1/captions` | `409 retention_active` for a locale that already exists on a locked video. A new locale is allowed. |
|
|
85
|
+
| `POST /v1/transcode` | `409 retention_active` for a re-transcode (see [Transcoding a locked video](#transcoding-a-locked-video)) |
|
|
86
|
+
| `transcode.complete` with `deleteOriginalOnSuccess` | The original is kept (see [below](#transcoding-a-locked-video)) |
|
|
87
|
+
|
|
88
|
+
A write to a key that doesn't exist yet is a create, not an overwrite, so it's always allowed.
|
|
89
|
+
|
|
90
|
+
Every write is conditional on the object the check saw. If the file changes between the check and the write (for
|
|
91
|
+
example, a lock lands meanwhile), R2 refuses the write and the Worker answers `409 write_conflict`. The Worker doesn't
|
|
92
|
+
retry, because the new object may be locked: retry the request and you get the current answer.
|
|
93
|
+
|
|
94
|
+
A refused delete or overwrite answers:
|
|
95
|
+
|
|
96
|
+
```json
|
|
97
|
+
{
|
|
98
|
+
"error": "private/file/f1/deed.pdf is retained and can't be deleted",
|
|
99
|
+
"code": "retention_active",
|
|
100
|
+
"key": "private/file/f1/deed.pdf",
|
|
101
|
+
"mode": "compliance",
|
|
102
|
+
"until": "2031-01-01T00:00:00.000Z",
|
|
103
|
+
"hold": false
|
|
104
|
+
}
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
`key` names the locked original. For a derived file (a caption, an HLS segment), that's the video's original. `mode`
|
|
108
|
+
is left out for a hold with no retention, and `until` is left out while a hold keeps a `period` from starting.
|
|
109
|
+
|
|
110
|
+
All the lock error codes:
|
|
111
|
+
|
|
112
|
+
| Status | `code` | When |
|
|
113
|
+
| --- | --- | --- |
|
|
114
|
+
| `409` | `retention_active` | A delete or overwrite of a locked file |
|
|
115
|
+
| `409` | `write_conflict` | The file changed between the check and the write: retry |
|
|
116
|
+
| `409` | `compliance_extend_only` | A change that would weaken a compliance retention |
|
|
117
|
+
| `409` | `lock_conflict` | Another lock change landed first: retry |
|
|
118
|
+
| `409` | `lock_record_unreadable` | A change to a file whose lock record can't be read (see [Caveats](#caveats)) |
|
|
119
|
+
| `403` | `lock_privilege_required` | Shortening or removing a governance retention without an approved bypass |
|
|
120
|
+
| `403` | `lock_below_minimum` | An upload asks for a looser lock than its upload token requires |
|
|
121
|
+
| `400` | `invalid_lock`, `invalid_hold` | A malformed lock or hold request |
|
|
122
|
+
| `400` | `reserved_key` | A key under `__lock/` or `__lock-audit/`, which the Worker keeps for itself |
|
|
123
|
+
|
|
124
|
+
Owner reads (the Worker secret on `GET /v1/media/<key>`) also get `x-devultur-lock-until` and, while a hold is set,
|
|
125
|
+
`x-devultur-lock-hold: true`. An owner read never goes through the edge cache: it isn't answered from a public copy
|
|
126
|
+
and doesn't store one. Its `Cache-Control` is the private form of the file's own, so no shared cache keeps the lock
|
|
127
|
+
headers either.
|
|
128
|
+
|
|
129
|
+
## Derived files
|
|
130
|
+
|
|
131
|
+
A video's transcode outputs and captions live under its folder (`{visibility}/video/{id}/…`).
|
|
132
|
+
`retention.derivatives` decides whether they're retained with the original:
|
|
133
|
+
|
|
134
|
+
- **`inherit`** (the default): transcode outputs and captions can't be deleted or overwritten while the original is
|
|
135
|
+
locked. New outputs and new caption locales can still be added.
|
|
136
|
+
- **`original-only`**: only the uploaded file is retained. Its outputs and captions can be deleted or regenerated
|
|
137
|
+
freely. Deleting the whole video is still refused, since the original is in it.
|
|
138
|
+
|
|
139
|
+
Image variants and mezzanines are regenerable caches. They're never retained.
|
|
140
|
+
|
|
141
|
+
Derived files also expire with their original. An HLS playlist or segment, an MP4, a thumbnail, poster or storyboard
|
|
142
|
+
written by the transcode runner gets the earliest `expiresAt` of the video's raws, and a caption gets the one of the
|
|
143
|
+
file it was made from. Past it, they answer `410` like the original. The value is copied when the output is written
|
|
144
|
+
(the raw is deleted by `deleteOriginalOnSuccess` only after the transcode completes), so changing a raw's expiry
|
|
145
|
+
later doesn't reach outputs already written. The HLS encryption key (`.key`) has no expiry of its own: it's useless
|
|
146
|
+
without the segments.
|
|
147
|
+
|
|
148
|
+
## Transcoding a locked video
|
|
149
|
+
|
|
150
|
+
- **The first transcode** of a locked video is allowed, since it only creates outputs.
|
|
151
|
+
- **A transcode that never completed** (failed, or the runner died) can be retried, and the new job may overwrite
|
|
152
|
+
the partial outputs.
|
|
153
|
+
- **Once a transcode has completed**, re-transcoding is refused with `409 retention_active`, because it would
|
|
154
|
+
overwrite retained outputs. The Worker records the completion durably, under
|
|
155
|
+
`__lock/{visibility}/video/{id}/transcode-completed`.
|
|
156
|
+
- **While a job is in progress** (`queued` or `running`), the runner may rewrite its own outputs (HLS, MP4,
|
|
157
|
+
thumbnails, storyboard), for example to retry an upload. From `transcode.complete` on, they're protected like any
|
|
158
|
+
derived file.
|
|
159
|
+
|
|
160
|
+
**Deleting the original after transcode.** With `transcode.deleteOriginalOnSuccess`, the Worker deletes the original
|
|
161
|
+
once a transcode completes. A locked original wins: it's kept, the skip is audited (`original_retained`), and the
|
|
162
|
+
webhook still succeeds. `getRouteResult(context).originalRetained` tells a plugin it happened. The original keeps
|
|
163
|
+
taking storage until its lock ends, and in Infrequent Access (`originalStorageClass`) it also keeps that class's
|
|
164
|
+
30-day minimum and per-read retrieval charge. Once the lock ends, delete it yourself: `deleteOriginalOnSuccess`
|
|
165
|
+
doesn't come back for it.
|
|
166
|
+
|
|
167
|
+
## Locking at upload
|
|
168
|
+
|
|
169
|
+
An upload can carry its lock, so the file never exists unprotected. The lock is written with the bytes, and nothing
|
|
170
|
+
can strip it later (the Worker drops a `lock` key from `config.uploadMetadata`).
|
|
171
|
+
|
|
172
|
+
```typescript
|
|
173
|
+
const client = createWorkerClient(secret, workerUrl);
|
|
174
|
+
|
|
175
|
+
await client.createUploadUrl({
|
|
176
|
+
filename: "deed.pdf",
|
|
177
|
+
contentType: "application/pdf",
|
|
178
|
+
lock: { retention: { mode: "compliance", period: 7 * 365 * 24 * 3600 } },
|
|
179
|
+
});
|
|
180
|
+
await client.createUploadUrl({ filename: "evidence.mp4", contentType: "video/mp4", hold: true });
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
`hold: true` is shorthand for `lock: { hold: true }`. `uploadFile` and `uploads.uploadAsset` take the same options.
|
|
184
|
+
A Worker-wide default goes in `config.lock`:
|
|
185
|
+
|
|
186
|
+
```typescript
|
|
187
|
+
export default createWorker({
|
|
188
|
+
plugins: defaultPlugins({ lock: { retention: { mode: "governance", period: 30 * 24 * 3600 } } }),
|
|
189
|
+
});
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
**Upload tokens** can pin a minimum lock: `uploads.createToken({ prefix, lock })`. An upload with that token gets at
|
|
193
|
+
least that lock. It can ask for a stricter one, never a looser one (`403 lock_below_minimum`).
|
|
194
|
+
|
|
195
|
+
**Expiry.** A lock may outlive the file's `expiresAt` (`expiresIn`, the token's or the config's TTL): see
|
|
196
|
+
[Retention vs expiry](#retention-vs-expiry).
|
|
197
|
+
|
|
198
|
+
**Uploads without a pending upload.** A `PUT /v1/upload/<key>` with the Worker secret and no `POST /v1/upload` before
|
|
199
|
+
it gets `config.lock` too, unless the key is a video's derived output (those follow their original through
|
|
200
|
+
`derivatives`).
|
|
201
|
+
|
|
202
|
+
Locks need the media Worker. In direct-storage mode, `createMediaRouter` throws when an upload asks for a lock,
|
|
203
|
+
because a presigned PUT to the bucket can't enforce one.
|
|
204
|
+
|
|
205
|
+
## Retention vs expiry
|
|
206
|
+
|
|
207
|
+
Retention and expiry are separate, as in S3 Object Lock. **Retention preserves** the object: it can't be deleted or
|
|
208
|
+
overwritten. **Expiry ends access** to it: past `expiresAt`, nobody but the owner reads it, and that covers a video's
|
|
209
|
+
transcode outputs and captions (see [Derived files](#derived-files)). A lock doesn't extend access, and an expiry
|
|
210
|
+
doesn't shorten a lock. A file can be shared for 30 days and kept for 10 years.
|
|
211
|
+
|
|
212
|
+
| State | Viewers, public requests, download links | The owner (Worker secret) | The object |
|
|
213
|
+
| --- | --- | --- | --- |
|
|
214
|
+
| Before `expiresAt` | Normal access | Normal access | Kept (and protected while locked) |
|
|
215
|
+
| After `expiresAt`, still retained | `410` | Can read it, to export or audit | Not deleted: delete and overwrite answer `409 retention_active` |
|
|
216
|
+
| After `expiresAt`, free | `410` | `410` | Stays in the bucket until your expiry cleanup deletes it |
|
|
217
|
+
|
|
218
|
+
- **Download links are access.** A link never outlives the file's `expiresAt`, even for a retained file. Creating one
|
|
219
|
+
for an expired file answers `410`, and redeeming one after the file expired does too.
|
|
220
|
+
- **Edge copies.** A public file's edge copy never outlives its `expiresAt`: its TTL is capped there, and a copy found
|
|
221
|
+
past it is ignored, so the request reaches the Worker and gets `410`.
|
|
222
|
+
- **Listings.** `GET /v1/assets` (an owner route) includes an expired file that's still retained, marked
|
|
223
|
+
`expired: true` with `retainedUntil` (its retention's end) or `hold: true`. An expired free file isn't listed.
|
|
224
|
+
Telling them apart reads each expired file's lock record (one subrequest each), so a page reads at most 32
|
|
225
|
+
(`MAX_EXPIRED_LOOKUPS`): past that, it ends early with a `cursor` that resumes right after its last entry. A page
|
|
226
|
+
can hold fewer entries than `limit` while more remain; keep following the `cursor` until it's `null`.
|
|
227
|
+
- **The expiry cleanup is yours to run.** Nothing in `@devultur/self-host` deletes expired files: the Worker only
|
|
228
|
+
refuses access to them. Run your own job (a cron Worker, a script) that deletes them through the Worker
|
|
229
|
+
(`DELETE /v1/media/<key>`, or `/v1/media/videos/<id>` for a video), so each delete goes through the lock check: a
|
|
230
|
+
retained file answers `409 retention_active` and is kept until its lock ends, then the job's next run deletes it.
|
|
231
|
+
Don't use an R2 lifecycle rule for it: it doesn't go through the Worker and deletes retained files too (see
|
|
232
|
+
[Caveats](#caveats)).
|
|
233
|
+
|
|
234
|
+
## Changing a lock
|
|
235
|
+
|
|
236
|
+
`GET /v1/lock/<key>` and `PUT /v1/lock/<key>` are control routes. The default auth accepts only the Worker secret.
|
|
237
|
+
|
|
238
|
+
```typescript
|
|
239
|
+
await client.lock.get(key);
|
|
240
|
+
// { key, mode, until, period, hold, derivatives, source: "upload" | "sidecar" | "none", revision }
|
|
241
|
+
|
|
242
|
+
await client.lock.set(key, { until: "2032-01-01T00:00:00Z" }); // extend
|
|
243
|
+
await client.lock.set(key, { mode: "governance", period: 90 * 24 * 3600 }); // set on an unlocked file
|
|
244
|
+
await client.lock.set(key, { until: "2026-12-31T00:00:00Z", bypassGovernance: true }); // shorten (governance)
|
|
245
|
+
await client.lock.set(key, { remove: true, bypassGovernance: true }); // remove (governance)
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
The fields merge over the current retention, and `until` and `period` replace each other. `PUT` changes the retention
|
|
249
|
+
only. Holds have [their own route](#holds).
|
|
250
|
+
|
|
251
|
+
R2 can't edit an object's metadata in place, so a change goes to a small record at `{storagePrefix}__lock/<key>`. The
|
|
252
|
+
record is bound to the object version it was written for (R2's per-write `version`, never the etag, which repeats when
|
|
253
|
+
the same bytes are uploaded again). Deleting or overwriting the file leaves the record in place, inert: it never
|
|
254
|
+
applies to another version, so a file uploaded again at the same key starts from its own upload lock, and a lock
|
|
255
|
+
change made on the new file right after it landed can't be lost to a cleanup of the old record. A record of a deleted
|
|
256
|
+
file is a few hundred bytes and stays as evidence of its last lock. To reclaim them, delete only records whose file no
|
|
257
|
+
longer exists: deleting the record of an existing file drops every change made to its lock. Two concurrent changes
|
|
258
|
+
can't both land: the second gets `409 lock_conflict`.
|
|
259
|
+
|
|
260
|
+
## Holds
|
|
261
|
+
|
|
262
|
+
`POST /v1/lock/hold` is the one way to set or release a hold, including on a single file:
|
|
263
|
+
|
|
264
|
+
```typescript
|
|
265
|
+
await client.lock.hold([key]);
|
|
266
|
+
await client.lock.release([key]);
|
|
267
|
+
|
|
268
|
+
// Every page of a prefix, or a long list of keys
|
|
269
|
+
for await (const page of client.lock.holdAll({ prefix: "private/asset/case-17/" })) {
|
|
270
|
+
console.log(page.updated, page.skipped);
|
|
271
|
+
}
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
- A request names up to 50 keys (`HOLD_PAGE_SIZE`), or a prefix. A prefix is walked 50 objects per request, with a
|
|
275
|
+
`cursor`. Each object costs about four subrequests, so a page stays well under the Workers limit. `holdAll` and
|
|
276
|
+
`releaseAll` follow the cursor and split long key lists for you.
|
|
277
|
+
- The answer is `{ updated: string[], skipped: { key, reason }[], cursor? }`. The `reason` is `not_found`,
|
|
278
|
+
`unchanged` (already in that state), `lock_conflict` (changed concurrently: retry it) or `lock_record_unreadable`
|
|
279
|
+
(a release of a file whose lock record can't be read, see [Caveats](#caveats)).
|
|
280
|
+
- A hold never touches the retention, so it can only add protection. A hold on a compliance file is allowed, and its
|
|
281
|
+
date stays.
|
|
282
|
+
- **Releasing:** with a `retention.period`, the period starts now. With a fixed `until`, that date still applies.
|
|
283
|
+
With no retention, the file is free.
|
|
284
|
+
- Image variants under a prefix are skipped, since they're caches. The Worker's own records (`__lock/`,
|
|
285
|
+
`__lock-audit/`) are never listed or changed, whatever the prefix.
|
|
286
|
+
|
|
287
|
+
## Audit
|
|
288
|
+
|
|
289
|
+
Every lock event gets one record. That covers every change (including holds and releases), the lock set at upload,
|
|
290
|
+
every rejected delete or overwrite, and every original kept after a transcode.
|
|
291
|
+
|
|
292
|
+
```typescript
|
|
293
|
+
interface LockAuditRecord {
|
|
294
|
+
at: string; // ISO time
|
|
295
|
+
action: "set" | "extend" | "shorten" | "remove" | "hold" | "release"
|
|
296
|
+
| "delete_rejected" | "overwrite_rejected" | "original_retained";
|
|
297
|
+
keys: string[];
|
|
298
|
+
before: ObjectLock | null;
|
|
299
|
+
after: ObjectLock | null;
|
|
300
|
+
actor: PluginContext | null; // the request's plugin context; for an upload, the uploader's
|
|
301
|
+
outcome: "applied" | "rejected" | "skipped";
|
|
302
|
+
bypass?: true; // the change used the governance privilege
|
|
303
|
+
reason?: string; // why it was rejected
|
|
304
|
+
}
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
Each record goes to three places:
|
|
308
|
+
|
|
309
|
+
- a structured log line (`{"event":"lock.audit", …}`);
|
|
310
|
+
- one create-only object at `{storagePrefix}__lock-audit/{yyyy-mm-dd}/{ts}-{uuid}.json` in the media bucket;
|
|
311
|
+
- every plugin's [`onLockAudit`](plugins.md#hooks) hook, to store it elsewhere.
|
|
312
|
+
|
|
313
|
+
The object and the hooks run after the response. A failure there is logged, never surfaced. The audit isn't a general
|
|
314
|
+
access log: reads aren't recorded.
|
|
315
|
+
|
|
316
|
+
## Deleting a prefix from your own code
|
|
317
|
+
|
|
318
|
+
Code that runs in your Worker and deletes whole prefixes straight from the R2 binding (an app's teardown, a user's
|
|
319
|
+
data export cleanup, a sweeper for expired files) bypasses the routes' checks. `@devultur/self-host` exports the
|
|
320
|
+
guard's lock resolution for it, so it can skip retained files:
|
|
321
|
+
|
|
322
|
+
```typescript
|
|
323
|
+
import { deletePrefixRespectingLocks, readObjectLock, retainedSummary } from "@devultur/self-host";
|
|
324
|
+
|
|
325
|
+
// The lock in force on one stored object (its full R2 key), or null when there's no such object.
|
|
326
|
+
const lock = await readObjectLock({ mediaBucket: env.MEDIA_BUCKET }, "apps/a1/private/file/f1/deed.pdf");
|
|
327
|
+
// { locked: true, hold: false, until: "2031-01-01T00:00:00.000Z", mode: "compliance" }
|
|
328
|
+
|
|
329
|
+
// Delete what's free under a storage prefix, keep what's retained. Call again with `cursor` until it's gone.
|
|
330
|
+
let cursor: string | undefined;
|
|
331
|
+
do {
|
|
332
|
+
const result = await deletePrefixRespectingLocks({
|
|
333
|
+
mediaBucket: env.MEDIA_BUCKET,
|
|
334
|
+
keysBucket: env.KEYS_BUCKET,
|
|
335
|
+
prefix: "apps/a1/",
|
|
336
|
+
cursor,
|
|
337
|
+
});
|
|
338
|
+
// result: { deletedCount, deletedBytes, deletedKeyCount, retainedCount, retainedBytes, retained, skippedCount,
|
|
339
|
+
// cursor?, subrequests }
|
|
340
|
+
cursor = result.cursor;
|
|
341
|
+
} while (cursor);
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
- **Keys are storage keys**: the full R2 keys, your storage prefix included. Nothing needs a request. The storage
|
|
345
|
+
prefix each key lives under (where its lock record is) is read from the key: everything before its first `public`
|
|
346
|
+
or `private` segment. Pass `root` when you walk a single storage prefix that has a segment named `public` or
|
|
347
|
+
`private`. One prefix can span several storage prefixes, each read on its own.
|
|
348
|
+
- **Keys it can't place are kept.** A key with no `public` or `private` segment (or none right after the `root` you
|
|
349
|
+
passed) isn't laid out like the Worker's, so its lock record can't be found. It's never deleted: it comes back in
|
|
350
|
+
`retained` with `unrecognized: true`. A key with a segment ending in `__lock` or `__lock-audit` (a lock record or
|
|
351
|
+
audit object, even under a storage prefix with no trailing `/`) is never touched, and is counted in `skippedCount`.
|
|
352
|
+
- **Same resolution as the routes.** The upload's lock, or the lock record bound to the stored version (a record left
|
|
353
|
+
by a replaced version is ignored). An unreadable record counts as held (`unreadable: true`). Each object's own lock
|
|
354
|
+
is checked on the version read (HEAD) right before its delete. A video's files inherit from its raws, and an image's
|
|
355
|
+
variants from the image, unless the lock is `original-only`. That inherited lock is read once and reused for at
|
|
356
|
+
most 50 objects or 5 seconds. `lockedBy` names the original whose lock it is.
|
|
357
|
+
- **What goes.** A free object goes the way `DELETE /v1/media/<key>` deletes it: with its encryption keys in
|
|
358
|
+
`KEYS_BUCKET` (except those of an upload in flight over it) and its variants'. A video folder left empty also loses
|
|
359
|
+
the rest of its keys written before the call started (the HLS key) and its transcode-completed marker, as with
|
|
360
|
+
`DELETE /v1/media/videos/<id>`, when the whole folder is under the prefix. `deletedBytes` adds up the sizes of the
|
|
361
|
+
versions deleted.
|
|
362
|
+
- **Real time only.** The delete takes no `now`: locks are judged by the real clock, so no option can make a retention
|
|
363
|
+
read as expired. `readObjectLock` and `retainedSummary` take a `now`, to ask what's retained at another time.
|
|
364
|
+
- **Cursors are bound to their prefix.** A cursor works only with the `prefix` it was returned for. One that's
|
|
365
|
+
malformed, from another prefix, or pointing outside it throws a `RangeError` before any bucket call.
|
|
366
|
+
- **What stays.** Retained files, with their variants, keys and lock records. The lock records of deleted files stay
|
|
367
|
+
too, inert, as with the routes. Nothing under `__lock/` or `__lock-audit/` is touched: once a storage prefix has
|
|
368
|
+
nothing retained left, delete those yourself if you're tearing it down.
|
|
369
|
+
- **Bounded.** A Worker invocation has a subrequest limit. Every bucket call is counted, and a call never makes more
|
|
370
|
+
than `maxSubrequests`. It reads everything an object needs before deleting any of it, and only deletes if the budget
|
|
371
|
+
left covers it, so it never stops halfway through an object. It returns a `cursor` to go on from (a video folder it
|
|
372
|
+
emptied is finished by the next call). The default, 40, fits under the Workers Free plan's 50 subrequests per
|
|
373
|
+
request and leaves room for your own calls. On a paid plan, raise it (to a few hundred, say) so each call gets
|
|
374
|
+
further. A bound too small to finish even one object throws a `RangeError`. `batchSize` (default 100, at most 1000)
|
|
375
|
+
sets how many objects it lists at a time. Run a large prefix over several invocations (a queue, or a cron that
|
|
376
|
+
keeps the cursor).
|
|
377
|
+
- **Not audited.** The call is your code, not a request, so no audit record or hook runs: log `retained` if you need a
|
|
378
|
+
trail. R2 deletes aren't conditional: a lock set between an object's check and its delete isn't seen. That gap is
|
|
379
|
+
milliseconds for the object's own lock, and up to 5 seconds for an inherited one (see [Caveats](#caveats)).
|
|
380
|
+
|
|
381
|
+
`retainedSummary({ mediaBucket, prefix, cursor?, now? })` answers "can this be deleted?" without deleting anything:
|
|
382
|
+
`{ scannedCount, retainedCount, retainedBytes, latestUntil?, hasHold, cursor?, subrequests }`, with the same bounds.
|
|
383
|
+
Add up the counts of each call. With `hasHold`, something stays until it's released, not until `latestUntil`.
|
|
384
|
+
|
|
385
|
+
## Caveats
|
|
386
|
+
|
|
387
|
+
- **The check/delete window.** R2 deletes aren't conditional, so a lock set in the milliseconds between the Worker's
|
|
388
|
+
check and its delete isn't seen. Writes are conditional, and protected.
|
|
389
|
+
- **Only the Worker enforces locks.** R2 lifecycle rules, the R2 dashboard, `wrangler r2`, and any S3 or binding
|
|
390
|
+
access that doesn't go through the Worker can delete a locked file. Treat locks as compliance only as long as nobody
|
|
391
|
+
else can write to the bucket. This isn't a certified WORM store.
|
|
392
|
+
- **R2 bucket locks** can be an optional second layer: an R2 bucket lock rule refuses deletes at the storage level.
|
|
393
|
+
Anyone who can edit the bucket's configuration can remove the rule, though, so it raises the bar rather than
|
|
394
|
+
closing it. [Self-host](self-host.md#r2-bucket-lock-defense-in-depth) has the safe prefixes and what
|
|
395
|
+
changes once a rule is on.
|
|
396
|
+
- **The runner credential.** While a transcode job is in progress, a request with the runner's credential may rewrite
|
|
397
|
+
that job's outputs on a locked video. With `workerSecretAuth()`, that credential is `WORKER_AUTH_SECRET`, which your
|
|
398
|
+
backend holds too. To limit the exemption to the runner, give the runner a narrower credential from your own auth
|
|
399
|
+
plugin (`updateRequestState(context, { runnerCredential })`) and accept it only for uploads of transcode outputs.
|
|
400
|
+
- **Direct storage** (`createMediaRouter` with your own bucket and no Worker) refuses locks: nothing would enforce
|
|
401
|
+
them.
|
|
402
|
+
- **`deletePrefix`**, exported by `@devultur/self-host`, deletes everything under a prefix and bypasses locks. Use it
|
|
403
|
+
only for regenerable data. The routes delete through the guarded helpers; your own code can use
|
|
404
|
+
[`deletePrefixRespectingLocks`](#deleting-a-prefix-from-your-own-code).
|
|
405
|
+
- **Listings show retention.** An expired, retained entry of `GET /v1/assets` carries `retainedUntil` and `hold`:
|
|
406
|
+
whoever can list a prefix learns what's retained in it. With `workerSecretAuth()`, listing takes the Worker secret,
|
|
407
|
+
the same credential as `GET /v1/lock`. An auth plugin of your own that lets a narrower credential list (a sync
|
|
408
|
+
script's, say) gives it that view too. If that credential shouldn't see what's retained, keep it off
|
|
409
|
+
`GET /v1/assets`.
|
|
410
|
+
- **An unreadable lock record** (the record at `__lock/<key>`, or the upload's `lock` metadata, isn't valid JSON or
|
|
411
|
+
isn't a valid lock) fails closed: the file reads as held, for whatever version is stored at the key. Since nobody can
|
|
412
|
+
tell what it really said (it might be a compliance retention), the API never weakens it: a release is skipped with
|
|
413
|
+
`lock_record_unreadable`, `PUT /v1/lock/<key>` answers `409 lock_record_unreadable`, and deletes and overwrites
|
|
414
|
+
answer `409 retention_active`. Recovery is an operator's job, outside the Worker: find the record's last good
|
|
415
|
+
content in the audit (the `after` of the key's last `applied` record under `__lock-audit/`), write it back to
|
|
416
|
+
`__lock/<key>` as `{ revision, objectVersion, lock, updatedAt }` with the current object's R2 `version` as
|
|
417
|
+
`objectVersion` (for bad upload metadata, `lock` is the lock it was uploaded with), and check it with
|
|
418
|
+
`GET /v1/lock/<key>`.
|
|
419
|
+
|
|
420
|
+
## Recipes
|
|
421
|
+
|
|
422
|
+
### Keep a record until N years after an event
|
|
423
|
+
|
|
424
|
+
The end date depends on an event that hasn't happened yet (a contract ends, a case closes). Upload with a hold and a
|
|
425
|
+
`period`, and release the hold when the event happens: the period starts then.
|
|
426
|
+
|
|
427
|
+
```typescript
|
|
428
|
+
const SEVEN_YEARS = 7 * 365 * 24 * 3600;
|
|
429
|
+
|
|
430
|
+
// At upload: retained indefinitely until the event
|
|
431
|
+
const upload = await client.createUploadUrl({
|
|
432
|
+
filename: "record.pdf",
|
|
433
|
+
contentType: "application/pdf",
|
|
434
|
+
lock: { retention: { mode: "compliance", period: SEVEN_YEARS }, hold: true },
|
|
435
|
+
});
|
|
436
|
+
|
|
437
|
+
// When the event happens: the 7 years start now
|
|
438
|
+
await client.lock.release([upload.key]);
|
|
439
|
+
|
|
440
|
+
// Or every file of the record at once
|
|
441
|
+
for await (const page of client.lock.releaseAll({ prefix: "private/asset/record-42/" })) {
|
|
442
|
+
if (page.skipped.length > 0) console.warn(page.skipped);
|
|
443
|
+
}
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
### Share for 30 days, keep for 10 years
|
|
447
|
+
|
|
448
|
+
Access ends after 30 days, and the file stays (undeletable) for 10 years:
|
|
449
|
+
|
|
450
|
+
```typescript
|
|
451
|
+
await client.createUploadUrl({
|
|
452
|
+
filename: "statement.pdf",
|
|
453
|
+
contentType: "application/pdf",
|
|
454
|
+
expiresIn: 30 * 24 * 3600,
|
|
455
|
+
lock: { retention: { mode: "compliance", period: 10 * 365 * 24 * 3600 } },
|
|
456
|
+
});
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
After 30 days, viewers and download links get `410`. Your backend can still read the file with the Worker secret,
|
|
460
|
+
and `GET /v1/assets` lists it with `expired: true` and `retainedUntil`. Until the 10 years pass, your
|
|
461
|
+
[expiry cleanup](#retention-vs-expiry) gets `409 retention_active` for it; its first run after that deletes it.
|
|
462
|
+
|
|
463
|
+
### Compliance retention for N years from upload
|
|
464
|
+
|
|
465
|
+
```typescript
|
|
466
|
+
export default createWorker({
|
|
467
|
+
plugins: defaultPlugins({
|
|
468
|
+
lock: { retention: { mode: "compliance", period: 10 * 365 * 24 * 3600 } },
|
|
469
|
+
}),
|
|
470
|
+
});
|
|
471
|
+
```
|
|
472
|
+
|
|
473
|
+
Every upload gets the lock, unless it asks for a stricter one. To scope it to some uploads, set the lock per upload
|
|
474
|
+
instead, or hand out [upload tokens](#locking-at-upload) with a minimum `lock`.
|
|
475
|
+
|
|
476
|
+
### Convex
|
|
477
|
+
|
|
478
|
+
The [Convex component](https://github.com/devultur/devultur/blob/main/packages/convex/README.md#object-lock) exposes the same operations to your own
|
|
479
|
+
functions. A Worker refusal arrives as a `ConvexError` whose `data.code` is the code above:
|
|
480
|
+
|
|
481
|
+
```typescript
|
|
482
|
+
import { Devultur, devulturErrorOf } from "@devultur/convex";
|
|
483
|
+
import { v } from "convex/values";
|
|
484
|
+
import { components } from "./_generated/api";
|
|
485
|
+
import { action } from "./_generated/server";
|
|
486
|
+
|
|
487
|
+
const devultur = new Devultur(components.devultur);
|
|
488
|
+
|
|
489
|
+
export const extendRetention = action({
|
|
490
|
+
args: { key: v.string(), until: v.string() },
|
|
491
|
+
handler: async (ctx, { key, until }) => {
|
|
492
|
+
try {
|
|
493
|
+
return await devultur.setLock(ctx, key, { until });
|
|
494
|
+
} catch (error) {
|
|
495
|
+
const refusal = devulturErrorOf(error);
|
|
496
|
+
if (refusal?.code === "compliance_extend_only") return { ok: false, reason: "can only move later" } as const;
|
|
497
|
+
if (refusal?.code === "lock_conflict") return devultur.setLock(ctx, key, { until }); // changed meanwhile
|
|
498
|
+
throw error;
|
|
499
|
+
}
|
|
500
|
+
},
|
|
501
|
+
});
|
|
502
|
+
|
|
503
|
+
export const deleteFile = action({
|
|
504
|
+
args: { key: v.string() },
|
|
505
|
+
handler: async (ctx, { key }) => {
|
|
506
|
+
try {
|
|
507
|
+
await ctx.runAction(components.devultur.public.deleteMediaAction, { key });
|
|
508
|
+
} catch (error) {
|
|
509
|
+
const refusal = devulturErrorOf(error);
|
|
510
|
+
// { key, mode, until, hold } of the locked original
|
|
511
|
+
if (refusal?.code === "retention_active") return { deleted: false, lock: refusal.details } as const;
|
|
512
|
+
throw error;
|
|
513
|
+
}
|
|
514
|
+
return { deleted: true } as const;
|
|
515
|
+
},
|
|
516
|
+
});
|
|
517
|
+
```
|