@ultimat3/storage 30.9.0 → 31.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md CHANGED
@@ -1,6 +1,7 @@
1
1
  # @ultimat3/storage — agent notes
2
2
 
3
3
  Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
4
+ A rule marked `†` has its full text in [`docs/history/storage-rules.md`](../../docs/history/storage-rules.md).
4
5
 
5
6
  - May import: `@ultimat3/core`, `@ultimat3/schema`. Nothing else. No npm dependencies.
6
7
  - Must NOT know about HTTP, entities, actions, render — a route calls `acceptSignedUpload()`,
@@ -34,7 +35,7 @@ Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
34
35
  | Keys | every driver method starts with `assertSafeKey()`. No exceptions, no sanitising. `META_DIR` (`.meta`) is a reserved first segment on every driver — see below |
35
36
  | Time | take a `Clock`; never `Date.now()` |
36
37
  | Bytes | `Uint8Array \| ReadableStream \| Blob`. Never a base64 string, never unbounded — `toBytes(body, limit)` |
37
- | Exports | explicit in `src/index.ts`; no `export *` |
38
+ | Exports | explicit in `src/index.ts`; no `export *`. `./adapter` (`adapter.ts`): `StorageDriver`, its shapes, the errors and helpers a driver raises with — the `storage` port; no installer (the boot reads `runtime.ts`'s `storage` driver as its default disk); the `uploadScanner` port + installer |
38
39
 
39
40
  | File | Owns |
40
41
  |---|---|
@@ -67,11 +68,14 @@ Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
67
68
  | `image.ts` | deterministic variant keys; byte path over core's pipeline (png/jpeg encode only) |
68
69
  | | `VARIANT_FORMATS` — what a variant KEY can carry — and NOT `IMAGE_FORMATS`, which is core's and means what core can PROBE. See below |
69
70
  | | `variantKey` is the cache identity `@ultimat3/cli`'s `/media/*` route looks a variant up by — derived, never stored, so a request that misses transforms once and every later one is a disk read |
71
+ | `image-variant.ts` | `variant()`: one kept variant, made by an `images` driver PASSED IN (seo's, structurally) |
72
+ | `upload-scanner*.ts` / `upload-scan.ts` | the `uploadScanner` port + memory (EICAR) reference; `promoteAttachment` scans a per-attempt copy |
70
73
  | `image-warm.ts` | `warmVariants`: a stored image's variants written ahead of `/media`, under its keys, the revision off the bytes read; core's spec, so a warmed variant is the one a miss would store |
71
74
  | `storage.ts` | `defineStorage` + module-level `storage()` / `disk()` |
75
+ | `storage-instrument.ts` | each resolved disk: `storage.<verb>` bus event + client span |
72
76
  | `grant.ts` | `grantUpload` — the ONE way a presigned PUT is minted; the client never names a key |
73
77
  | `accept.ts` | `acceptSignedUpload` / `readSignedObject` — the two decisions a `/_storage` route is made of |
74
- | `attachment.ts` | `pending/` → row promotion, the `quarantine/` segment, `sweepOrphans` and `deleteVariants` |
78
+ | `attachment.ts` | `pending/` → row promotion, `sweepOrphans` and `deleteVariants` |
75
79
  | `errors-sniff.ts` | the two `X_STORAGE_TYPE_REJECTED` refusals only a bounded sniff reaches: too much leading whitespace, XML past the window |
76
80
  | `upload-client.ts` | the browser half: grant → PUT with progress → key. No `Bun.*`, no `node:` |
77
81
  | `subject-target.ts` | `storageSubjectTarget(disk, keysOf)` — structural `SubjectTarget`: the app names the keys, export lists, erase deletes |
@@ -82,21 +86,10 @@ bun run typecheck
82
86
  ```
83
87
 
84
88
  Gotchas:
85
- - **The local disk refuses a key that is another key's path, and that is pinned, not fixed**
86
- (`As of 2026-10`). `put('a')` then `put('a/b')` — or the reverse, or `a` beside `a.json/b` in the
87
- sidecar tree — is `X_STORAGE_KEY_CONFLICT`, decided by `claim()` for BOTH paths before either is
88
- touched. Suffixed on-disk names would lift it and orphan every existing root's objects without a
89
- migration; do not change the layout without one. Memory emulates it; s3 holds both. A key that
90
- is only a directory is ABSENT to `delete()` (test a refusal with `driver-local-fixture.ts`).
91
- - **The local commit order is marker → sidecar → bytes → clear marker. Do not reorder**
92
- (`As of 2026-10`). Bytes-first serves new bytes under the old type; sidecar-first alone serves
93
- old bytes under the new one. The marker holds the new etag, so `headObject` knows the one state
94
- in which the sidecar may not describe the bytes: `get`/`stat`/`copy` re-check against the bytes,
95
- `list` reports no type and `etag: ''`. A fresh key is absent until its bytes land.
96
- `driver-local-crash.test.ts` dies at every step through `WriteSteps`. Writers and `get()` of one
97
- key also queue in-process (`keyedQueue`) — unqueued, two puts interleaved 3/3. `copy()` holds BOTH
98
- keys' turns (`inTurnOfBoth`, key order): measured in one and read in the other, an overwritten
99
- source committed new bytes under the old etag.
89
+ - **The local disk refuses a key that is another key's path** (`X_STORAGE_KEY_CONFLICT`, `claim()`);
90
+ pinned, not fixed — changing the layout needs a migration. †
91
+ - **The local commit order is marker → sidecar → bytes → clear marker. Do not reorder**; one key's
92
+ writers and `get()` queue in-process (`keyedQueue`), `copy()` holds both keys' turns. †
100
93
  - **A refused write is `X_STORAGE_PUT_FAILED` and a refused read `X_STORAGE_READ_FAILED`**, on
101
94
  local and s3 (both were bare throws); a 404 mid-read stays `X_STORAGE_NOT_FOUND`; `assertPutOptions` holds `metadata`/`cacheControl`/`contentType` to their types on
102
95
  all three drivers before a byte moves.
@@ -110,21 +103,14 @@ Gotchas:
110
103
  sees a path — and `signedUrlBasePath` is how an absolute `baseUrl` is compared.
111
104
  - **`isWithinOrg` never throws**: an org id that cannot be one contains nothing.
112
105
  - **A variant key keeps the whole source key** (`hero.png@w640.webp`): two cache identities.
113
- - **A driver's semantics are pinned in ONE test with the other driver beside them.**
114
- `driver-parity.test.ts` drives `localDriver` over a temp dir and `s3Driver` over `FakeS3Client`
115
- in a single `test()` per claim, so neither disk can move alone. Where the two genuinely cannot
116
- agree it pins the DIVERGENCE, with the reason — that is the honest form, and it still fails the
117
- day either half changes.
118
- - **The local `list()` globs with `dot: true`, and the `META_DIR` skip is the real filter.**
119
- Without `dot`, dot-prefixed keys were missing from the listing while `put`/`get` handled them —
120
- and `sweepOrphans` pages through `list()`. Pinned in `driver-parity.test.ts`.
106
+ - **A driver's semantics are `storageDriverConformance`'s** (`@ultimat3/testing/conformance`);
107
+ what only two disks share is pinned in `driver-parity.test.ts`, divergences with their reason. †
108
+ - **The local `list()` globs with `dot: true`, and the `META_DIR` skip is the real filter.** †
121
109
  - **A `list({ limit })` is a positive integer or a refusal** — `resolveListLimit` at the
122
- `ListOptions` seam, so both disks answer one way. Core's `assert` (`X_INVARIANT`).
123
- - **`list()` answers core's `Page<StorageListEntry>`, and no storage-local page type exists**
124
- (`As of 25.0.0`; `ListPage` is deleted). Every driver builds it with core's `pageOf`, so
125
- `hasMore` is derived from the cursor and cannot disagree with it. The cursor is minted only when
126
- a row past the page exists — never from `rows.length === limit`, the full-last-page defect a
127
- query's cursor had — and the full-last-page case is pinned on memory, local and live s3.
110
+ `ListOptions` seam, so every disk answers one way. Core's `assert` (`X_INVARIANT`);
111
+ `storageDriverConformance` holds each driver to it.
112
+ - **`list()` answers core's `Page<StorageListEntry>`** (`pageOf`); a cursor only when a row past the
113
+ page exists. †
128
114
  - **`META_DIR` is reserved case-INSENSITIVELY** (`As of 2026-08`), exactly as `isTenantScoped`
129
115
  folds: on APFS/NTFS `.META/` IS `.meta/`. The whole SEGMENT is compared, never a prefix
130
116
  (`.metadata/a.json` is an ordinary key); `path.test.ts` pins both halves.
@@ -142,84 +128,53 @@ Gotchas:
142
128
  S3 accepts both (delete marker, new version; the locked version stays). Pinned divergence: one
143
129
  version per key. A pair in doubt never unlocks. `NoSuchObjectLockConfiguration` is a 404.
144
130
  - **A live suite signs on a wall clock** (`performance`): `Date` is frozen → `RequestTimeTooSkewed`.
145
- - **A grant binds its size on every disk** (`As of 2026-11-02`). `grantUpload` requires `request.size` and passes `contentLength`; `s3Driver` signs `content-length` through core's `presignAwsUrl`, because Bun's presign signs `host` alone, so any other length is a 403 (`driver-s3.live.test.ts`). Local and memory hold it as the `x-max` ceiling, so a shorter body passes there: `driver-parity.test.ts` pins it. `maxBytes` with no `contentLength` on s3 is `X_STORAGE_UPLOAD_SIZE_UNDECLARED`. The content type stays unsigned.
146
- - **`putStream` on s3 is signed multipart, never `S3File.writer()`.** Measured on Bun 1.4.4: `end(error)` COMPLETES the upload, so an overrun would replace the object, and the writer cannot send `x-amz-meta-*`. One part at a time; every exit but the Complete aborts; a Complete can fail with 200 and an `<Error>` body.
131
+ - **A grant binds its size on every disk**: s3 signs `content-length` (`presignAwsUrl`), local/memory
132
+ hold `x-max`; `maxBytes` with no `contentLength` on s3 is `X_STORAGE_UPLOAD_SIZE_UNDECLARED`. †
133
+ - **`putStream` on s3 is signed multipart, never `S3File.writer()`** (Bun 1.4.4: `end(error)` COMPLETES). 8 MiB parts, ≤ `STREAM_PART_BUFFERS` held; parts settle before the abort; a Complete can fail with 200 + `<Error>`.
134
+ - **s3 `copy()` is a signed CopyObject**, past 5 GiB `UploadPartCopy`: Bun's `write(S3File)` dropped `cacheControl`/metadata.
135
+ - **`stream(key, { etag })` never serves another revision**: local re-checks the etag at the first pull, never the inode (ext4 reuses it; `highWaterMark: 0`: an unread body holds no fd); s3 sends `If-Match`.
136
+ - **The upload route streams** (`accept-stream.ts`). Keys refuse `:`, a trailing `.`, Win32 device names; `contentType`/`cacheControl` are `HEADER_VALUE` on every disk.
147
137
  - **`promoteAttachment` sniffs a head window** (`validateStoredHead`, 64 KiB), then ties it to the
148
138
  copy: `ifMatch`, then size and etag. Never refuse on etag alone: SSE-KMS mints one per copy.
149
139
  - **`/media`'s variant key names its source's revision and format.** No `revision`: the old key.
150
- - **`delete()` is idempotent for an ABSENT key and for nothing else.** Both drivers used to end in
151
- `.catch(() => undefined)`, so a denied `s3:DeleteObject`, a `SlowDown`, an expired credential and
152
- a read-only mount all resolved as success — and `sweepOrphans` returned them as deleted, which
153
- is a GDPR erasure report certifying data that is still in the bucket. The s3 driver classifies
154
- structurally (`code` in `NoSuchKey`/`NotFound`/`ENOENT`, or a 404 `statusCode`/`status`; an
155
- `S3Error` is a plain `Error` with those fields and every read of one can throw), the local
156
- driver on `ENOENT`. Everything else is `X_STORAGE_DELETE_FAILED`, whose `fix` the DRIVER
157
- supplies — the command that reproduces the refusal differs per disk.
158
- - **`toBytes` takes a `ByteLimit`, and there is no unbounded variant.** `put()` buffers, so the
159
- ceiling is what keeps a route piping a 4GB body into `disk.put()` from being an OOM kill rather
160
- than a refusal. `Uint8Array`/`Blob` are refused on their declared length; a `ReadableStream` is
161
- read chunk-by-chunk and **cancelled** past the line, so at most one chunk over is ever held.
162
- Default `maxPutBytes` is `DEFAULT_MAX_UPLOAD_BYTES` — one constant, imported by both drivers
163
- from `upload.ts`, because the server-side ceiling and the upload ceiling are the same fact.
164
- - **`serverSideEncryption` is declared and refused by BOTH drivers.** Bun exposes `acl`,
165
- `storageClass` and `type` and nothing for `x-amz-server-side-encryption*`, and a POSIX file is
166
- not encrypted at all. Refusing on the dev disk too is deliberate: a `put()` that works in `x dev`
167
- and throws in production is two rules. Bucket lifecycle (the app sweeps with `expire.ts`), storage classes and replication stay out
168
- entirely — bucket-side, terraform's, axiom 7.
169
- - **Quarantine is `pending/quarantine/`, INSIDE the pending prefix.** So `sweepOrphans` collects a
170
- never-scanned upload with no second prefix to walk, and `isPendingKey` stays true for it.
171
- `promoteAttachment` refuses a quarantined key (`X_STORAGE_QUARANTINED`); `releaseQuarantine` is
172
- the app's scan verdict, and returns the ordinary pending key promotion accepts. Ultimate ships
173
- no scanner — axiom 8, and `application/zip` is an accepted OOXML container by design.
140
+ - **`delete()` is idempotent for an ABSENT key and for nothing else** — a swallowed refusal made
141
+ `sweepOrphans` certify an erasure that never happened. s3 classifies structurally (`NoSuchKey`/
142
+ `NotFound`/`ENOENT` or a 404), local on `ENOENT`; else `X_STORAGE_DELETE_FAILED`, fix per driver.
143
+ - **`toBytes` takes a `ByteLimit`, and there is no unbounded variant**; one default,
144
+ `DEFAULT_MAX_UPLOAD_BYTES` (`upload.ts`). †
145
+ - **`serverSideEncryption` is declared and refused by BOTH drivers.** Full rule: `docs/history/storage.md`.
146
+ - **The `uploadScanner` port is the ONE scan path** (`As of 31.0.0`; the `quarantine: true` segment
147
+ and `releaseQuarantine` are retired). An out-of-band decision — a review, an async vendor scan —
148
+ is the verdict `hold`, keyed by `ScanInput.pendingKey`: `X_STORAGE_QUARANTINED` (409), upload
149
+ kept pending, attempt deleted, no policy overrides it. A scanner that holds declares `holds: true`
150
+ (else its `hold` is `X_INVARIANT`), and its outage then refuses even under `'accept'`. A targeted
151
+ grant is refused while a scanner is installed (`X_STORAGE_TARGET_UNSCANNED`): it skips promotion.
152
+ Ultimate ships no scanner vendor (axiom 8).
153
+ **No verdict fails closed in every environment** (`onScanUnavailable` absent = `'refuse'`).
174
154
  - **`StorageListEntry` vs `StorageObject`.** A listing's `contentType` is optional; a `get()`'s is
175
155
  not. `ListObjectsV2` returns no Content-Type, and no driver invents one.
176
- - **`META_DIR` lives in `path.ts`, not in `driver-local.ts`.** The sidecar namespace overlapped the
177
- object namespace: `put('a/b', png)` writes `<root>/.meta/a/b.json`, and `.meta/a/b.json` was
178
- itself a legal key, so an uploader could rewrite another object's recorded `contentType` to
179
- `text/html` and have a route serve attacker HTML from the app's origin. Reserved in
180
- `assertSafeKey`, so it holds for S3 too — a key valid on one driver and refused on another is two
181
- key rules, and the reservation folds case — see above. The `list()` skip is not a second line of
182
- defence but the only one: the glob yields the sidecar tree.
156
+ - **`META_DIR` lives in `path.ts`, reserved in `assertSafeKey` on every driver** — the sidecar
157
+ namespace must never be a writable key. †
183
158
  - **`localDriver` refuses to construct outside development without a usable signing secret** (`X_ENV_MISSING`, borrowed from core). Full rule: `docs/history/storage.md`.
184
159
  - **`localDriver({ env })` and `usesDevStorageSecret({ env })` are ONE question about ONE table**:
185
160
  the secret, `isLocal()` and the refusal's environment all read `options.env ?? process.env`, so
186
161
  the boot's guard and the constructor it guards cannot answer from two tables.
187
162
  `driver-local-boot.test.ts` pins it by mutation.
188
- - **The mounted read half is `@ultimat3/cli`'s `runtime-storage.ts`, not this package.** `GET
189
- /_storage/:disk/*key` gates on `@ultimat3/policy`'s `evaluate()` (`storage:read`), which is tier
190
- 2 and unreachable from here — so a "serve this object" helper in this package could only ever be
191
- a second authz path. This package's contribution to that route is `assertSafeKey`,
192
- `isTenantScoped`/`isWithinOrg` and the driver's own `contentType`/`etag`; the `Response` is
193
- `@ultimat3/http`'s `objectResponse`, structural over those fields, as `accept.ts`'s header says.
194
- - `acceptSignedUpload` still has no mounted route: the signing secret is closed over inside
195
- `localDriver` and no `StorageDriver` method exposes it, so a host cannot verify a signed PUT
196
- through the `Storage` seam it is handed. Serving reads needed none of it (policy decides, not a
197
- signature); mounting the write half needs that seam question answered first.
163
+ - **The mounted read half is `@ultimat3/cli`'s `runtime-storage.ts`**, gated by policy; no
164
+ "serve this object" helper here (a second authz path). †
165
+ - `acceptSignedUpload` still has no mounted route: no `StorageDriver` method exposes the signing
166
+ secret. †
198
167
  - `X_NOT_IMPLEMENTED` is core's — it belongs in `STORAGE_BORROWED_ERROR_CODES` (codes, no title)
199
168
  and never in the registration call. A `hasErrorCode()` guard there would suppress the
200
169
  `X_ERROR_CODE_DUPLICATE` two owners of one code are supposed to get. Tests need `resetStorage()`
201
170
  in `beforeEach`.
202
- - `image.ts` owns no pixels: core's `transformImageBytes`/`blurDataUrl` are the only scaler.
203
- Its image failures (`X_IMAGE_UNSUPPORTED`, `X_IMAGE_DECODE_FAILED`) surface unwrapped —
204
- wrapping them in a `StorageError` would give one failure two codes. `variantKey` now RAISES
205
- core's `imageUnsupported()` as well as passing them through, for that same reason.
206
- - **`VARIANT_FORMATS` is this package's format vocabulary, and `IMAGE_FORMATS` is core's** (what
207
- core can PROBE). `as const satisfies readonly ImageFormat[]` makes it a strict subset by the
208
- compiler; `isVariantFormat` takes `string` so `probeImage(bytes).format` narrows with no cast.
209
- `scripts/render-modes.ts` holds the `IMAGE_FORMATS` row and `image.test.ts` fails if either core
210
- name reappears in `src/index.ts`.
211
- - `transformImage()` must encode at exactly `fitDimensions()`'s size, and passes `format`
212
- explicitly (`?? 'webp'`, which then rejects): bytes that disagree with the `variantKey`
213
- extension, or with the `width`/`height` `@ultimat3/seo` inlined, are the layout shift the
214
- whole path exists to prevent. That is why it derives the box itself and asks core for
215
- `fit: 'cover'` — core's `contain` letterboxes to the requested box, this API fits inside it.
171
+ - `image.ts` owns no pixels: core's `transformImageBytes` is the only scaler; image failures surface
172
+ unwrapped. †
173
+ - **`VARIANT_FORMATS` is this package's format vocabulary, and `IMAGE_FORMATS` is core's.** †
174
+ - `transformImage()` encodes at exactly `fitDimensions()`'s size, `format` explicit — anything else
175
+ is a layout shift. †
216
176
  - Bun's S3 flag is `virtualHostedStyle`; our `forcePathStyle` is its inverse.
217
- - **No `region` signs for `auto`.** R2 wants that; AWS and a gateway that checks the scope answer
218
- `AuthorizationHeaderMalformed … expecting "<region>"`. `put`/`copy`/`list`/`delete` ask
219
- `regionMismatch()` BEFORE their own verdict, so the code is `X_CONFIG_INVALID` and the fix is the
220
- region — never `X_STORAGE_LIST_FAILED` with a fix about `s3:ListBucket`. The captured region is
221
- held to `[A-Za-z0-9-]`: a provider's sentence may put nothing else into a `fix:`. A HEAD has no
222
- body, so `exists()`/`stat()` never see it. `driver-s3.live.test.ts` runs with no region on purpose.
177
+ - **No `region` signs for `auto`** (R2's; AWS refuses it) — `regionMismatch()` makes it `X_CONFIG_INVALID` naming `S3_REGION`. Full rule: `docs/history/storage.md`.
223
178
  - The signature check runs BEFORE the expiry check. Do not reorder.
224
179
  - `timingSafeEqual` is `@ultimat3/core`'s (`signed-url.ts` imports it) — the same
225
180
  implementation `@ultimat3/auth` uses, not a second copy. Add new secret comparisons through it.
@@ -230,31 +185,19 @@ Gotchas:
230
185
  always sets one, so such a URL is hand-rolled, and trusting the uploader's header instead is the
231
186
  only other option.
232
187
  - **The signed base is the disk's REGISTERED name, and it is stated once — on the driver.** Full rule: `docs/history/storage.md`.
233
- - **`accept.ts` asks the `isTenantScoped`/`isWithinOrg` PAIR, exactly as `runtime-storage.ts` does.**
234
- `isWithinOrg` alone refused every un-scoped key, so an app's own `brand/logo.png` was unreachable
235
- through a URL it had just signed. `isTenantScoped` is case-INSENSITIVE and `isWithinOrg` is not:
236
- `Org/o2/x` and `org/o2/x` are one file on APFS/NTFS, so the fold has to count as tenant-scoped
237
- and then fail the exact-case membership test. Do not "simplify" either half.
238
- - **`AcceptSignedUploadInput.checksum` is what makes `uploadPolicy({ requireChecksum: true })`
239
- reachable.** Without it that option could only ever fail, because nothing on the accept path
240
- could declare a hash. It travels like `declaredContentType`: the route reads a header and hands
241
- it over, and `validateUpload` hashes the bytes itself. The browser half does NOT send one — a
242
- custom header on an S3 presigned PUT is a signature question this package cannot answer.
188
+ - **`accept.ts` asks the `isTenantScoped`/`isWithinOrg` PAIR, as `runtime-storage.ts` does.** Do not
189
+ "simplify" either half. †
190
+ - **`AcceptSignedUploadInput.checksum` is what makes `requireChecksum: true` reachable**; the browser
191
+ half sends none. †
243
192
  - `orgId` is required on both halves of `accept.ts` and is the ACTOR's, never a request field. A
244
193
  signed URL is a capability; a leaked capability must still not cross a tenant.
245
194
  - `X_STORAGE_ORG_MISMATCH` maps to **404**, not 403 (`@ultimat3/http`'s `error-map.ts`). 403 would
246
195
  confirm a key exists to the one caller who must not learn it.
247
196
  - **Local `lastModified` is the `Clock`'s, kept in the sidecar** — one instant from `put`/`copy`
248
197
  to every read; mtime only with no trusted sidecar.
249
- - `upload-client.ts` defaults to `XMLHttpRequest`, not `fetch`: `fetch` reports no upload
250
- progress in any shipping browser, and a bar that jumps 0 → 100 is a bar that is lying. It is
251
- **the declared XHR seam** of plan 101 (decision 13): the one `XMLHttpRequest` the framework
252
- ships, and the one file slice 15's `browser-transport` guard exempts for it — the header says so.
253
- - **`fetchSignedPut` goes through `@ultimat3/core`'s `clientTransport`** (`As of 2026-09-22`,
254
- plan 101 slice 05): `rawBody` sends the `Blob` verbatim under the grant's content type with no
255
- default header, and `decodeError` keeps a refusal `X_STORAGE_UPLOAD_FAILED` with the disk's
256
- status. A request with no response at all is the transport's `X_CLIENT_TRANSPORT_FAILED` — it was a
257
- bare `TypeError` before. This file calls no `fetch`; the XHR above is the one exemption.
198
+ - `upload-client.ts` uses `XMLHttpRequest` for upload progress — the framework's one declared XHR
199
+ seam. †
200
+ - **`fetchSignedPut` goes through `@ultimat3/core`'s `clientTransport`**; it calls no `fetch`. †
258
201
  - **Bundle** (`uploadFile` only, browser, minified, `As of 2026-09-22`): **17,767 B**.
259
202
  - `exactOptionalPropertyTypes` is on — declare optional fields as `x?: T | undefined`.
260
203
 
package/README.md CHANGED
@@ -58,15 +58,20 @@ const stored = [...sessions.objects().values()].map((bytes) => new TextDecoder()
58
58
 
59
59
  Server-side only, like both other drivers: nothing an island imports reaches it.
60
60
 
61
+ Every call on a disk `defineStorage` resolves is a `storage.<verb>` event on `@ultimat3/core`'s instrumentation bus (`put`, `get`, `stat`, `copy`, `delete`, `list`; `{ disk, key, bytes?, durationMs, error? }`) inside a `storage.<verb>` client span. `disk()` hands out that instrumented view, so it is not `===` the driver you declared — compare `.name`. A driver you hold directly reports nothing.
62
+
61
63
  `region` is what requests are signed for; unset, that is `auto`. An endpoint that expects another
62
64
  region refuses the first write, list or delete with `X_CONFIG_INVALID`, and the fix names the
63
65
  `S3_REGION` to set.
64
66
 
65
67
  `copy(from, to)` is on the contract so a promotion is not a download-and-reupload:
66
68
  `promoteAttachment` used to `get()` the whole object into the app and `put()` it back, a gigabyte
67
- through the pod to rename a 500MB attachment. `localDriver` does a real file copy; `s3Driver`
68
- hands the source `S3File` to `write()` — Bun exposes no `CopyObject`, so the bytes still cross the
69
- network, but never this process's heap. Both arguments go through `assertSafeKey`.
69
+ through the pod to rename a 500MB attachment. `localDriver` does a real file copy; `s3Driver` sends
70
+ one signed `CopyObject` (directive COPY), so the bytes never leave the provider and the source's
71
+ type, `cacheControl` and `metadata` travel, as on local and memory. Past CopyObject's 5 GiB ceiling
72
+ it is a multipart `UploadPartCopy` (512 MiB parts, grown past 10,000), every part pinned to the
73
+ revision a HEAD read, carrying the same headers, aborted on any failure. Both arguments go through
74
+ `assertSafeKey`.
70
75
 
71
76
  One S3 driver covers all three backends — the difference is `endpoint` + `forcePathStyle`.
72
77
  Credentials are **env var NAMES** (`accessKeyIdEnv`, default `S3_ACCESS_KEY_ID`), never
@@ -176,6 +181,16 @@ do {
176
181
 
177
182
  Before 25.0.0 it was `ListPage { objects, truncated, cursor? }`, its own shape beside core's.
178
183
 
184
+ ### Writing a driver: `@ultimat3/storage/adapter`
185
+
186
+ `StorageDriver`, the shapes it reads and answers, the error factories a driver raises
187
+ (`objectNotFound`, `putFailed`, `readFailed`, …) and the helpers every driver refuses with the same
188
+ way (`assertPutOptions`, `toBytes`, `resolveListLimit`, `etagOf`). Hand one to the boot as
189
+ `storage: myDriver()` in `apps/web/runtime.ts`: it becomes the default disk, registered under the
190
+ driver's `name`, in place of the `S3_ENDPOINT`/embedded-disk decision (31.0 — the field took a whole
191
+ `Storage` before). Several disks stay the app's `defineStorage()`. The suite is
192
+ `@ultimat3/testing/conformance`'s `storageDriverConformance`.
193
+
179
194
  ## `put()` is for objects that fit in memory
180
195
 
181
196
  `put()` buffers the whole body — size and checksum have to be known before the object exists —
@@ -200,12 +215,17 @@ in a bucket unmeasured, so a later `get()` of it was heap growth the uploader pi
200
215
 
201
216
  `streamRange` reads no byte before `first`: s3 and local through `slice(...).stream()`, memory from
202
217
  its copy. Inclusive and clamped like `get`'s range, with no ceiling because nothing is buffered;
203
- past the end is `X_STORAGE_RANGE_UNSATISFIABLE`. `putStream` takes `contentType`, `cacheControl`,
218
+ past the end is `X_STORAGE_RANGE_UNSATISFIABLE`. `stream(key, { etag })` and
219
+ `streamRange(key, range, { etag })` serve that revision only — another is `X_STORAGE_OBJECT_CHANGED`
220
+ (409): s3 sends `If-Match`, local checks the etag at the call and again at the first read, memory compares. `/_storage`
221
+ pins every read to the etag its headers came from. `putStream` takes `contentType`, `cacheControl`,
204
222
  `metadata` and a REQUIRED `maxBytes` (`PutStreamOptions`). Past it the source is cancelled, nothing
205
223
  is stored and the key stays as it was (`X_STORAGE_TOO_LARGE`). `checksum`, `retention`, `legalHold`
206
224
  and `serverSideEncryption` need the bytes up front: the type omits them and an untyped caller gets
207
225
  `X_INVARIANT`. On s3 a body under one part is one signed PUT and past it a signed multipart upload,
208
- one part at a time (5 MiB or `maxBytes / 10000`), aborted on every exit but the Complete: Bun's
226
+ 8 MiB parts (doubled every 1,000 parts, so 10,000 reach past S3's 5 TiB; never derived from
227
+ `maxBytes`), up to `STREAM_PART_BUFFERS` (4) parts held at once — in flight plus the one filling,
228
+ so memory is at most 32 MiB per stream — aborted on every exit but the Complete: Bun's
209
229
  `S3File.writer()` completes on `end(error)` and sends no `x-amz-meta-*`. Local stages under
210
230
  `.meta/.tmp/`, then runs `put()`'s commit steps. A driver outside this package reuses
211
231
  `assertPutStream`, `pumpBounded`, `streamLimit`, `assertStreamRange`, `streamedRange` and `signedByteBound`.
@@ -418,8 +438,13 @@ Neither owns a `Request`, a `Response` or a status number — mounting is the ho
418
438
  **Step 3 ships.** `As of 2026-09-24` `@ultimat3/cli` mounts `PUT /_storage/:disk/*key` beside the
419
439
  `GET` in `x dev` and `runRole` (#523), so an app writes steps 1 and 2 only. The route calls
420
440
  `signedUploadConstraints({ url, disk, orgId })` first. That is the verified PUT constraints with
421
- no bytes, and the signed `maxBytes` caps the body read. Then it calls `acceptSignedUpload` with a
422
- policy built from the grant's own signed type and ceiling. The routes serve `definedStorage()`,
441
+ no bytes, and the signed `maxBytes` caps the body read. Then it calls `acceptSignedUploadStream`
442
+ with a policy built from the grant's own signed type and ceiling: the first 64 KiB are sniffed, the
443
+ rest piped into `disk.putStream`, so memory per upload is the window, not the body. A body that
444
+ ends inside the window, an XML type, and a required checksum nobody declared are read whole and
445
+ judged by `acceptSignedUpload`. A local or memory disk given `maxRequestBytes` (the app's
446
+ `http.maxRequestBodyBytes`) refuses at signing a single-request grant past it (`X_STORAGE_TOO_LARGE`,
447
+ the fix naming `grantMultipart`). The routes serve `definedStorage()`,
423
448
  the process's one registry, which is the app's `defineStorage` when it declared one (#524).
424
449
 
425
450
  ## Uploads in parts — `grantMultipart`, `signPart`, `listParts`, `completeUpload`, `abortUpload`
@@ -450,10 +475,8 @@ An upload happens **before** the row it belongs to exists, so it lands at
450
475
  | Call | Key |
451
476
  |---|---|
452
477
  | `pendingKey(orgId, uploadName(id, filename))` | `org/o1/pending/u-1.png` |
453
- | `quarantineKey(orgId, name)` | `org/o1/pending/quarantine/u-1.png` |
454
478
  | `attachmentKey(orgId, { entity, id, field }, name)` | `org/o1/post/p-1/cover/u-1.png` |
455
479
  | `promoteAttachment({ disk, key, orgId, target, policy })` | `stat`, `copy`, then `delete` — never the reverse. **`policy` is required**: the object is measured here (`stat().size` over `policy.maxBytes` is `X_STORAGE_TOO_LARGE`, and it stays under `pending/` for the sweep), because an s3 presign bounds no size. Then one ranged read of the first 64 KiB through `validateStoredHead`, because it binds no type or bytes either: a contradiction is `X_STORAGE_TYPE_REJECTED`, and it stays pending too. The URL stays valid after that read, so `copy(from, to, { ifMatch })` names the revision judged (`X_STORAGE_OBJECT_CHANGED`, on s3 a signed CopyObject), and the copy is checked: another size is refused and deleted, and an etag that differs or is unknown (SSE-KMS mints one per copy) means the copy's own head is judged. A call after the source is gone, with the attached key present, is `X_STORAGE_ALREADY_PROMOTED` (409, `meta.attachedKey`): an earlier call moved the bytes, so this one answers the row that holds them and never deletes that key. The key must be a **pending** one (`isPendingKey`): an attached key a client sent back is another row's file, and moving it would delete the victim's copy (`X_STORAGE_NOT_PENDING`) |
456
- | `releaseQuarantine({ disk, key, orgId })` | quarantine → pending; returns the released key |
457
480
  | `sweepOrphans({ disk, orgId, olderThanMs })` | `{ deleted, failed }` for stale `pending/` keys. `olderThanMs` is a whole number of 0 or more, refused otherwise (`X_INVARIANT`) — `NaN` read as "everything is old enough" and deleted an upload made a moment ago |
458
481
  | `deleteVariants({ disk, key })` | `{ deleted, failed }` for every stored variant of one source, every revision (`list({ prefix: `${key}@` })`). **An erasure of one object must call it**: `disk.delete` leaves the variants' bytes behind. The source is the caller's to delete; a variant keeps its prefix, so an org-wide erasure by prefix already collects them |
459
482
 
@@ -466,31 +489,80 @@ refused delete goes in `failed` with the disk's own words and the sweep keeps go
466
489
  array of "deleted" keys put every refusal in it, so an erasure sweep over a bucket whose policy
467
490
  had lost `s3:DeleteObject` reported 200 objects gone that were all still there.
468
491
 
469
- ### Quarantine is the mechanism; the scanner is your app's
492
+ ### Scanning on confirm: the `uploadScanner` port
470
493
 
471
- Magic-byte sniffing closes stored XSS. It is **not** malware scanning, and `application/zip` is
472
- accepted on purpose as the OOXML container, so a macro-laden `.docx` passes `validateUpload`
473
- exactly as a clean one does. A scanner is a business decision with a vendor, a licence and a
474
- latency budget (axiom 8), so what ships is the place to put one — one more segment in a
475
- convention that already existed:
494
+ `As of 2026-11-09`. An `uploadScanner` in `apps/<app>/runtime.ts` is asked by `promoteAttachment`
495
+ about every upload it attaches. It scans a per-attempt copy (`pending/scan/<random>-<name>`): server
496
+ named, so no presigned URL rewrites it after the scan, and never the row's key, so a refused second
497
+ confirm of a double-click cannot delete the file the first one attached. The clean attempt is what
498
+ is copied onto the row's key:
499
+
500
+ | Verdict | `promoteAttachment` |
501
+ |---|---|
502
+ | `{ verdict: 'clean' }` | promotes, as without a scanner |
503
+ | `{ verdict: 'infected', signature }` | deletes the pending upload and the attempt, throws `X_STORAGE_UPLOAD_INFECTED` (422, `meta.signature`) |
504
+ | `{ verdict: 'hold', reason }` | deletes the attempt, keeps the upload pending, throws `X_STORAGE_QUARANTINED` (409, `meta.reason`) — a decision made elsewhere has not cleared it yet. No policy overrides it; the same confirm promotes once the scanner answers clean |
505
+ | rejects (daemon down, timeout), or the attempt cannot be read | deletes the attempt, keeps the upload pending, throws `X_STORAGE_SCAN_UNAVAILABLE` (503) — **fail closed by default, in every environment**; `uploadPolicy({ onScanUnavailable: 'accept' })` promotes unscanned instead, unless the scanner declares `holds: true`. An infected or held verdict is refused under either |
476
506
 
477
507
  ```ts
478
- const grant = await grantUpload({ disk, orgId, request, quarantine: true });
479
- // → org/<orgId>/pending/quarantine/<uploadId><ext>
508
+ // apps/web/runtime.ts
509
+ import type { RuntimeOverrides } from '@ultimat3/cli/serve';
510
+ import { memoryUploadScanner } from '@ultimat3/storage';
511
+ // In production, your own UploadScanner here: e.g. one streaming `body` to clamd's INSTREAM
512
+ // over Bun.connect, mapping `OK` → clean, `<sig> FOUND` → infected, anything else → reject.
513
+ export const runtime = {
514
+ uploadScanner: memoryUploadScanner(),
515
+ } satisfies RuntimeOverrides;
516
+ ```
517
+
518
+ `UploadScanner` (`@ultimat3/storage/adapter`) is `{ name, holds?, scan({ key, pendingKey, size, contentType, body }) }`,
519
+ `body` a `ReadableStream`. `memoryUploadScanner()` is the reference: it flags the EICAR test string
520
+ (`EICAR_TEST_STRING`) anywhere in the stream, across chunk boundaries. Hold your adapter to the
521
+ contract with `behavesLike(uploadScannerConformance, () => yours)` from
522
+ `@ultimat3/testing/conformance`. No scanner ships: ClamAV, a vendor API, its licence and its
523
+ latency budget are the app's (axiom 8). No scanner installed, nothing is scanned.
524
+
525
+ ### Hold: a decision made elsewhere — a review queue, an async vendor scan
526
+
527
+ Magic-byte sniffing closes stored XSS. It is **not** malware scanning: `application/zip` is the
528
+ OOXML container, so a macro-laden `.docx` passes `validateUpload` as a clean one does. A verdict
529
+ that is not ready inside the confirm is the scanner's `hold`, keyed by `pendingKey` — the upload's
530
+ own key, stable across retries (`key` is the per-attempt copy). One scan path; the v30
531
+ `grantUpload({ quarantine: true })` + `releaseQuarantine` flow is retired.
480
532
 
481
- // promoteAttachment on that key throws X_STORAGE_QUARANTINED. Your scan job decides:
482
- const released = await releaseQuarantine({ disk, key, orgId }); // clean
483
- await promoteAttachment({ disk, key: released, orgId, target, policy });
533
+ ```ts
534
+ import type { UploadScanner } from '@ultimat3/storage/adapter';
535
+
536
+ // `approved` is your app's own store — a table a moderator's action writes.
537
+ export const reviewScanner = (approved: (key: string) => Promise<boolean>): UploadScanner => ({
538
+ name: 'review',
539
+ holds: true,
540
+ async scan({ pendingKey, body }) {
541
+ await body.cancel();
542
+ return (await approved(pendingKey))
543
+ ? { verdict: 'clean' }
544
+ : { verdict: 'hold', reason: 'awaiting a moderator' };
545
+ },
546
+ });
484
547
  ```
485
548
 
486
- Inside `pending/` deliberately: an upload nobody ever scanned is still an orphan, so
487
- `sweepOrphans` collects it with no second prefix to walk.
549
+ **`holds: true` is required of a scanner that answers `hold`**, and makes its outage always refuse:
550
+ `approved` throwing (your database is down) is `X_STORAGE_SCAN_UNAVAILABLE` even under
551
+ `onScanUnavailable: 'accept'`, because only that store knows what is held. A `hold` from a scanner
552
+ that did not declare it is `X_INVARIANT`, and nothing is attached.
553
+
554
+ A held upload is still under `pending/`, so `sweepOrphans` collects one nobody ever cleared: keep
555
+ `olderThanMs` past your review window, or spare it with `keep`.
556
+
557
+ **A grant with a `target` is refused while a scanner is installed** (`X_STORAGE_TARGET_UNSCANNED`,
558
+ `grantUpload` and `grantMultipart` both): its key never passes through `promoteAttachment`, the one
559
+ place a scan runs. Grant with no `target`, then promote.
488
560
 
489
561
  ## A person's objects, for a subject export or erase
490
562
 
491
563
  `storageSubjectTarget(() => disk(), keysOf, { name? })` is a `defineSubject({ targets })` entry
492
564
  (`@ultimat3/jobs`): `keysOf(subject)` answers the person's keys (a list, or `disk().list({ prefix })`
493
- over a prefix the app owns). The export lists each object that exists (`STORAGE_SUBJECT_COLUMNS`);
565
+ over a prefix the app owns). The export lists each object that exists (columns: `STORAGE_SUBJECT_COLUMNS` in `src/subject-target.ts`);
494
566
  the erase deletes them in bounded batches. There is no listing by owner: a key the app does not
495
567
  name is not found.
496
568
 
@@ -511,8 +583,11 @@ name is not found.
511
583
  | `X_STORAGE_UPLOAD_FAILED` | client half: the presigned `PUT` answered non-2xx or never landed |
512
584
  | `X_STORAGE_DELETE_FAILED` | the disk REFUSED a delete — denied `s3:DeleteObject`, a throttle, an expired credential, a read-only mount. An **absent** key is still not an error |
513
585
  | `X_STORAGE_LIST_FAILED` | the disk REFUSED a listing — denied `s3:ListBucket`, a throttle, an unreadable root. An **empty** disk is still not an error |
514
- | `X_STORAGE_QUARANTINED` | `promoteAttachment` on a key nothing has released from `pending/quarantine/` |
586
+ | `X_STORAGE_TARGET_UNSCANNED` | `grantUpload`/`grantMultipart` with a `target` while an `uploadScanner` is installed — the key would skip the scan |
587
+ | `X_STORAGE_QUARANTINED` | the installed `uploadScanner` answered `hold`: the upload stays pending until it answers clean; `meta.reason` is the scanner's |
515
588
  | `X_STORAGE_NOT_PENDING` | `promoteAttachment` on a key outside the org's `pending/` prefix — most often another row's attached key |
589
+ | `X_STORAGE_UPLOAD_INFECTED` | the installed `uploadScanner` found malware in an upload `promoteAttachment` was attaching; both keys are deleted, `meta.signature` names the detection |
590
+ | `X_STORAGE_SCAN_UNAVAILABLE` | the installed `uploadScanner` rejected instead of answering, under the default `onScanUnavailable: 'refuse'`; the upload stays pending for a retry |
516
591
  | `X_STORAGE_ALREADY_PROMOTED` | `promoteAttachment` on a pending key an earlier call already moved (a retried or doubled confirm); `meta.attachedKey` names where the bytes are |
517
592
  | `X_STORAGE_PUT_FAILED` | the disk REFUSED a `put()`/`copy()` — `EACCES`, `ENOSPC`, `EROFS`, a provider's refused PUT |
518
593
  | `X_STORAGE_OBJECT_LOCKED` | local, memory: a delete, overwrite or copy onto a key under retention or a legal hold; the cause names the key, the mode and `retainUntil` or the hold, and the bytes are unchanged. Never raised by s3, where the provider keeps the locked version |
@@ -538,6 +613,24 @@ a job boundary the class is gone and the `code` is what survives — match on th
538
613
  `variantKey()`, `srcsetDescriptors()`, `fitDimensions()` are pure — `@ultimat3/seo` builds
539
614
  `srcset` from them without decoding a byte.
540
615
 
616
+ `variant(key, { disk, images, width, format? })` (`image-variant.ts`) is ONE variant, kept: made by
617
+ the `images` driver you pass — `runtime.ts`'s `images` port value, `@ultimat3/seo`'s
618
+ `builtinImageDriver({ read })` or a CDN's, any `ImageTransformDriver` (storage takes it structurally;
619
+ it is tier 1 beside seo and imports none of it) — and written under
620
+ `variantKey(key, { width, format, revision })`. A second call is a `stat`, never a decode
621
+ (`{ key, object, created }`); an overwritten source is a new revision, so a new key. `format`
622
+ omitted keeps a png/jpeg/webp source's encoding. Refused with core's `X_IMAGE_UNSUPPORTED`: a source
623
+ that is not a raster image (a PDF, an SVG), a gif/avif source with no `format`, and a driver that
624
+ answers another encoding than the key names. A source overwritten while the driver read it is
625
+ `X_STORAGE_OBJECT_CHANGED`, and nothing is stored. `As of 2026-11-09`.
626
+
627
+ ```ts
628
+ import { builtinImageDriver } from '@ultimat3/seo';
629
+ import { disk, variant } from '@ultimat3/storage';
630
+ const images = builtinImageDriver({ read: async (src) => (await disk().get(src)).bytes });
631
+ const { key } = await variant('org/o1/post/p1/cover/u1.png', { disk: disk(), images, width: 640, format: 'webp' });
632
+ ```
633
+
541
634
  `warmVariants({ disk, key, widths, formats })` (`image-warm.ts`) writes every width x format of a
542
635
  stored image under the key `/media` reads — `variantKey(key, { width, format, revision })`, the
543
636
  revision off the etag of the bytes it read — so the job an upload enqueues pays the decode and no
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ultimat3/storage",
3
- "version": "30.9.0",
3
+ "version": "31.1.0",
4
4
  "description": "Named disks over Bun.file and Bun.s3: safe keys, signed URLs, sniffed uploads",
5
5
  "keywords": [
6
6
  "ultimate",
@@ -24,7 +24,8 @@
24
24
  "provenance": true
25
25
  },
26
26
  "exports": {
27
- ".": "./src/index.ts"
27
+ ".": "./src/index.ts",
28
+ "./adapter": "./src/adapter.ts"
28
29
  },
29
30
  "files": [
30
31
  "src",
@@ -42,6 +43,6 @@
42
43
  "test": "bun test"
43
44
  },
44
45
  "dependencies": {
45
- "@ultimat3/core": "30.9.0"
46
+ "@ultimat3/core": "31.1.0"
46
47
  }
47
48
  }