@ultimat3/storage 30.9.0 → 31.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +56 -113
- package/README.md +117 -24
- package/package.json +4 -3
- package/src/accept-stream.ts +125 -0
- package/src/accept.ts +24 -6
- package/src/adapter.ts +46 -0
- package/src/attachment.ts +43 -61
- package/src/driver-local-list.ts +1 -1
- package/src/driver-local-open.ts +112 -0
- package/src/driver-local-read.ts +1 -1
- package/src/driver-local-stream.ts +14 -3
- package/src/driver-local.ts +48 -16
- package/src/driver-memory.ts +39 -9
- package/src/driver-s3-copy-parts.ts +177 -0
- package/src/driver-s3-copy.ts +14 -7
- package/src/driver-s3-lock.ts +1 -3
- package/src/driver-s3-put-stream.ts +81 -17
- package/src/driver-s3-range.ts +14 -7
- package/src/driver-s3-signed.ts +1 -1
- package/src/driver-s3-stream.ts +36 -0
- package/src/driver-s3.ts +44 -27
- package/src/driver.ts +30 -5
- package/src/errors-promote.ts +3 -15
- package/src/errors-scan.ts +61 -0
- package/src/errors-stream.ts +13 -0
- package/src/errors-variant.ts +40 -0
- package/src/errors.ts +6 -0
- package/src/grant.ts +15 -10
- package/src/image-variant.ts +106 -0
- package/src/index.ts +21 -40
- package/src/path.ts +27 -1
- package/src/request-ceiling.ts +45 -0
- package/src/storage-instrument.ts +108 -0
- package/src/storage.ts +8 -2
- package/src/upload-scan.ts +91 -0
- package/src/upload-scanner-memory.ts +62 -0
- package/src/upload-scanner.ts +71 -0
- package/src/upload.ts +35 -2
package/CLAUDE.md
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# @ultimat3/storage — agent notes
|
|
2
2
|
|
|
3
3
|
Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
|
|
4
|
+
A rule marked `†` has its full text in [`docs/history/storage-rules.md`](../../docs/history/storage-rules.md).
|
|
4
5
|
|
|
5
6
|
- May import: `@ultimat3/core`, `@ultimat3/schema`. Nothing else. No npm dependencies.
|
|
6
7
|
- Must NOT know about HTTP, entities, actions, render — a route calls `acceptSignedUpload()`,
|
|
@@ -34,7 +35,7 @@ Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
|
|
|
34
35
|
| Keys | every driver method starts with `assertSafeKey()`. No exceptions, no sanitising. `META_DIR` (`.meta`) is a reserved first segment on every driver — see below |
|
|
35
36
|
| Time | take a `Clock`; never `Date.now()` |
|
|
36
37
|
| Bytes | `Uint8Array \| ReadableStream \| Blob`. Never a base64 string, never unbounded — `toBytes(body, limit)` |
|
|
37
|
-
| Exports | explicit in `src/index.ts`; no `export
|
|
38
|
+
| Exports | explicit in `src/index.ts`; no `export *`. `./adapter` (`adapter.ts`): `StorageDriver`, its shapes, the errors and helpers a driver raises with — the `storage` port; no installer (the boot reads `runtime.ts`'s `storage` driver as its default disk); the `uploadScanner` port + installer |
|
|
38
39
|
|
|
39
40
|
| File | Owns |
|
|
40
41
|
|---|---|
|
|
@@ -67,11 +68,14 @@ Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
|
|
|
67
68
|
| `image.ts` | deterministic variant keys; byte path over core's pipeline (png/jpeg encode only) |
|
|
68
69
|
| | `VARIANT_FORMATS` — what a variant KEY can carry — and NOT `IMAGE_FORMATS`, which is core's and means what core can PROBE. See below |
|
|
69
70
|
| | `variantKey` is the cache identity `@ultimat3/cli`'s `/media/*` route looks a variant up by — derived, never stored, so a request that misses transforms once and every later one is a disk read |
|
|
71
|
+
| `image-variant.ts` | `variant()`: one kept variant, made by an `images` driver PASSED IN (seo's, structurally) |
|
|
72
|
+
| `upload-scanner*.ts` / `upload-scan.ts` | the `uploadScanner` port + memory (EICAR) reference; `promoteAttachment` scans a per-attempt copy |
|
|
70
73
|
| `image-warm.ts` | `warmVariants`: a stored image's variants written ahead of `/media`, under its keys, the revision off the bytes read; core's spec, so a warmed variant is the one a miss would store |
|
|
71
74
|
| `storage.ts` | `defineStorage` + module-level `storage()` / `disk()` |
|
|
75
|
+
| `storage-instrument.ts` | each resolved disk: `storage.<verb>` bus event + client span |
|
|
72
76
|
| `grant.ts` | `grantUpload` — the ONE way a presigned PUT is minted; the client never names a key |
|
|
73
77
|
| `accept.ts` | `acceptSignedUpload` / `readSignedObject` — the two decisions a `/_storage` route is made of |
|
|
74
|
-
| `attachment.ts` | `pending/` → row promotion,
|
|
78
|
+
| `attachment.ts` | `pending/` → row promotion, `sweepOrphans` and `deleteVariants` |
|
|
75
79
|
| `errors-sniff.ts` | the two `X_STORAGE_TYPE_REJECTED` refusals only a bounded sniff reaches: too much leading whitespace, XML past the window |
|
|
76
80
|
| `upload-client.ts` | the browser half: grant → PUT with progress → key. No `Bun.*`, no `node:` |
|
|
77
81
|
| `subject-target.ts` | `storageSubjectTarget(disk, keysOf)` — structural `SubjectTarget`: the app names the keys, export lists, erase deletes |
|
|
@@ -82,21 +86,10 @@ bun run typecheck
|
|
|
82
86
|
```
|
|
83
87
|
|
|
84
88
|
Gotchas:
|
|
85
|
-
- **The local disk refuses a key that is another key's path
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
migration; do not change the layout without one. Memory emulates it; s3 holds both. A key that
|
|
90
|
-
is only a directory is ABSENT to `delete()` (test a refusal with `driver-local-fixture.ts`).
|
|
91
|
-
- **The local commit order is marker → sidecar → bytes → clear marker. Do not reorder**
|
|
92
|
-
(`As of 2026-10`). Bytes-first serves new bytes under the old type; sidecar-first alone serves
|
|
93
|
-
old bytes under the new one. The marker holds the new etag, so `headObject` knows the one state
|
|
94
|
-
in which the sidecar may not describe the bytes: `get`/`stat`/`copy` re-check against the bytes,
|
|
95
|
-
`list` reports no type and `etag: ''`. A fresh key is absent until its bytes land.
|
|
96
|
-
`driver-local-crash.test.ts` dies at every step through `WriteSteps`. Writers and `get()` of one
|
|
97
|
-
key also queue in-process (`keyedQueue`) — unqueued, two puts interleaved 3/3. `copy()` holds BOTH
|
|
98
|
-
keys' turns (`inTurnOfBoth`, key order): measured in one and read in the other, an overwritten
|
|
99
|
-
source committed new bytes under the old etag.
|
|
89
|
+
- **The local disk refuses a key that is another key's path** (`X_STORAGE_KEY_CONFLICT`, `claim()`);
|
|
90
|
+
pinned, not fixed — changing the layout needs a migration. †
|
|
91
|
+
- **The local commit order is marker → sidecar → bytes → clear marker. Do not reorder**; one key's
|
|
92
|
+
writers and `get()` queue in-process (`keyedQueue`), `copy()` holds both keys' turns. †
|
|
100
93
|
- **A refused write is `X_STORAGE_PUT_FAILED` and a refused read `X_STORAGE_READ_FAILED`**, on
|
|
101
94
|
local and s3 (both were bare throws); a 404 mid-read stays `X_STORAGE_NOT_FOUND`; `assertPutOptions` holds `metadata`/`cacheControl`/`contentType` to their types on
|
|
102
95
|
all three drivers before a byte moves.
|
|
@@ -110,21 +103,14 @@ Gotchas:
|
|
|
110
103
|
sees a path — and `signedUrlBasePath` is how an absolute `baseUrl` is compared.
|
|
111
104
|
- **`isWithinOrg` never throws**: an org id that cannot be one contains nothing.
|
|
112
105
|
- **A variant key keeps the whole source key** (`hero.png@w640.webp`): two cache identities.
|
|
113
|
-
- **A driver's semantics are
|
|
114
|
-
`driver-parity.test.ts
|
|
115
|
-
|
|
116
|
-
agree it pins the DIVERGENCE, with the reason — that is the honest form, and it still fails the
|
|
117
|
-
day either half changes.
|
|
118
|
-
- **The local `list()` globs with `dot: true`, and the `META_DIR` skip is the real filter.**
|
|
119
|
-
Without `dot`, dot-prefixed keys were missing from the listing while `put`/`get` handled them —
|
|
120
|
-
and `sweepOrphans` pages through `list()`. Pinned in `driver-parity.test.ts`.
|
|
106
|
+
- **A driver's semantics are `storageDriverConformance`'s** (`@ultimat3/testing/conformance`);
|
|
107
|
+
what only two disks share is pinned in `driver-parity.test.ts`, divergences with their reason. †
|
|
108
|
+
- **The local `list()` globs with `dot: true`, and the `META_DIR` skip is the real filter.** †
|
|
121
109
|
- **A `list({ limit })` is a positive integer or a refusal** — `resolveListLimit` at the
|
|
122
|
-
`ListOptions` seam, so
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
a row past the page exists — never from `rows.length === limit`, the full-last-page defect a
|
|
127
|
-
query's cursor had — and the full-last-page case is pinned on memory, local and live s3.
|
|
110
|
+
`ListOptions` seam, so every disk answers one way. Core's `assert` (`X_INVARIANT`);
|
|
111
|
+
`storageDriverConformance` holds each driver to it.
|
|
112
|
+
- **`list()` answers core's `Page<StorageListEntry>`** (`pageOf`); a cursor only when a row past the
|
|
113
|
+
page exists. †
|
|
128
114
|
- **`META_DIR` is reserved case-INSENSITIVELY** (`As of 2026-08`), exactly as `isTenantScoped`
|
|
129
115
|
folds: on APFS/NTFS `.META/` IS `.meta/`. The whole SEGMENT is compared, never a prefix
|
|
130
116
|
(`.metadata/a.json` is an ordinary key); `path.test.ts` pins both halves.
|
|
@@ -142,84 +128,53 @@ Gotchas:
|
|
|
142
128
|
S3 accepts both (delete marker, new version; the locked version stays). Pinned divergence: one
|
|
143
129
|
version per key. A pair in doubt never unlocks. `NoSuchObjectLockConfiguration` is a 404.
|
|
144
130
|
- **A live suite signs on a wall clock** (`performance`): `Date` is frozen → `RequestTimeTooSkewed`.
|
|
145
|
-
- **A grant binds its size on every disk
|
|
146
|
-
|
|
131
|
+
- **A grant binds its size on every disk**: s3 signs `content-length` (`presignAwsUrl`), local/memory
|
|
132
|
+
hold `x-max`; `maxBytes` with no `contentLength` on s3 is `X_STORAGE_UPLOAD_SIZE_UNDECLARED`. †
|
|
133
|
+
- **`putStream` on s3 is signed multipart, never `S3File.writer()`** (Bun 1.4.4: `end(error)` COMPLETES). 8 MiB parts, ≤ `STREAM_PART_BUFFERS` held; parts settle before the abort; a Complete can fail with 200 + `<Error>`.
|
|
134
|
+
- **s3 `copy()` is a signed CopyObject**, past 5 GiB `UploadPartCopy`: Bun's `write(S3File)` dropped `cacheControl`/metadata.
|
|
135
|
+
- **`stream(key, { etag })` never serves another revision**: local re-checks the etag at the first pull, never the inode (ext4 reuses it; `highWaterMark: 0`: an unread body holds no fd); s3 sends `If-Match`.
|
|
136
|
+
- **The upload route streams** (`accept-stream.ts`). Keys refuse `:`, a trailing `.`, Win32 device names; `contentType`/`cacheControl` are `HEADER_VALUE` on every disk.
|
|
147
137
|
- **`promoteAttachment` sniffs a head window** (`validateStoredHead`, 64 KiB), then ties it to the
|
|
148
138
|
copy: `ifMatch`, then size and etag. Never refuse on etag alone: SSE-KMS mints one per copy.
|
|
149
139
|
- **`/media`'s variant key names its source's revision and format.** No `revision`: the old key.
|
|
150
|
-
- **`delete()` is idempotent for an ABSENT key and for nothing else
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
- **`serverSideEncryption` is declared and refused by BOTH drivers.** Bun exposes `acl`,
|
|
165
|
-
`storageClass` and `type` and nothing for `x-amz-server-side-encryption*`, and a POSIX file is
|
|
166
|
-
not encrypted at all. Refusing on the dev disk too is deliberate: a `put()` that works in `x dev`
|
|
167
|
-
and throws in production is two rules. Bucket lifecycle (the app sweeps with `expire.ts`), storage classes and replication stay out
|
|
168
|
-
entirely — bucket-side, terraform's, axiom 7.
|
|
169
|
-
- **Quarantine is `pending/quarantine/`, INSIDE the pending prefix.** So `sweepOrphans` collects a
|
|
170
|
-
never-scanned upload with no second prefix to walk, and `isPendingKey` stays true for it.
|
|
171
|
-
`promoteAttachment` refuses a quarantined key (`X_STORAGE_QUARANTINED`); `releaseQuarantine` is
|
|
172
|
-
the app's scan verdict, and returns the ordinary pending key promotion accepts. Ultimate ships
|
|
173
|
-
no scanner — axiom 8, and `application/zip` is an accepted OOXML container by design.
|
|
140
|
+
- **`delete()` is idempotent for an ABSENT key and for nothing else** — a swallowed refusal made
|
|
141
|
+
`sweepOrphans` certify an erasure that never happened. s3 classifies structurally (`NoSuchKey`/
|
|
142
|
+
`NotFound`/`ENOENT` or a 404), local on `ENOENT`; else `X_STORAGE_DELETE_FAILED`, fix per driver.
|
|
143
|
+
- **`toBytes` takes a `ByteLimit`, and there is no unbounded variant**; one default,
|
|
144
|
+
`DEFAULT_MAX_UPLOAD_BYTES` (`upload.ts`). †
|
|
145
|
+
- **`serverSideEncryption` is declared and refused by BOTH drivers.** Full rule: `docs/history/storage.md`.
|
|
146
|
+
- **The `uploadScanner` port is the ONE scan path** (`As of 31.0.0`; the `quarantine: true` segment
|
|
147
|
+
and `releaseQuarantine` are retired). An out-of-band decision — a review, an async vendor scan —
|
|
148
|
+
is the verdict `hold`, keyed by `ScanInput.pendingKey`: `X_STORAGE_QUARANTINED` (409), upload
|
|
149
|
+
kept pending, attempt deleted, no policy overrides it. A scanner that holds declares `holds: true`
|
|
150
|
+
(else its `hold` is `X_INVARIANT`), and its outage then refuses even under `'accept'`. A targeted
|
|
151
|
+
grant is refused while a scanner is installed (`X_STORAGE_TARGET_UNSCANNED`): it skips promotion.
|
|
152
|
+
Ultimate ships no scanner vendor (axiom 8).
|
|
153
|
+
**No verdict fails closed in every environment** (`onScanUnavailable` absent = `'refuse'`).
|
|
174
154
|
- **`StorageListEntry` vs `StorageObject`.** A listing's `contentType` is optional; a `get()`'s is
|
|
175
155
|
not. `ListObjectsV2` returns no Content-Type, and no driver invents one.
|
|
176
|
-
- **`META_DIR` lives in `path.ts`,
|
|
177
|
-
|
|
178
|
-
itself a legal key, so an uploader could rewrite another object's recorded `contentType` to
|
|
179
|
-
`text/html` and have a route serve attacker HTML from the app's origin. Reserved in
|
|
180
|
-
`assertSafeKey`, so it holds for S3 too — a key valid on one driver and refused on another is two
|
|
181
|
-
key rules, and the reservation folds case — see above. The `list()` skip is not a second line of
|
|
182
|
-
defence but the only one: the glob yields the sidecar tree.
|
|
156
|
+
- **`META_DIR` lives in `path.ts`, reserved in `assertSafeKey` on every driver** — the sidecar
|
|
157
|
+
namespace must never be a writable key. †
|
|
183
158
|
- **`localDriver` refuses to construct outside development without a usable signing secret** (`X_ENV_MISSING`, borrowed from core). Full rule: `docs/history/storage.md`.
|
|
184
159
|
- **`localDriver({ env })` and `usesDevStorageSecret({ env })` are ONE question about ONE table**:
|
|
185
160
|
the secret, `isLocal()` and the refusal's environment all read `options.env ?? process.env`, so
|
|
186
161
|
the boot's guard and the constructor it guards cannot answer from two tables.
|
|
187
162
|
`driver-local-boot.test.ts` pins it by mutation.
|
|
188
|
-
- **The mounted read half is `@ultimat3/cli`'s `runtime-storage.ts
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
`isTenantScoped`/`isWithinOrg` and the driver's own `contentType`/`etag`; the `Response` is
|
|
193
|
-
`@ultimat3/http`'s `objectResponse`, structural over those fields, as `accept.ts`'s header says.
|
|
194
|
-
- `acceptSignedUpload` still has no mounted route: the signing secret is closed over inside
|
|
195
|
-
`localDriver` and no `StorageDriver` method exposes it, so a host cannot verify a signed PUT
|
|
196
|
-
through the `Storage` seam it is handed. Serving reads needed none of it (policy decides, not a
|
|
197
|
-
signature); mounting the write half needs that seam question answered first.
|
|
163
|
+
- **The mounted read half is `@ultimat3/cli`'s `runtime-storage.ts`**, gated by policy; no
|
|
164
|
+
"serve this object" helper here (a second authz path). †
|
|
165
|
+
- `acceptSignedUpload` still has no mounted route: no `StorageDriver` method exposes the signing
|
|
166
|
+
secret. †
|
|
198
167
|
- `X_NOT_IMPLEMENTED` is core's — it belongs in `STORAGE_BORROWED_ERROR_CODES` (codes, no title)
|
|
199
168
|
and never in the registration call. A `hasErrorCode()` guard there would suppress the
|
|
200
169
|
`X_ERROR_CODE_DUPLICATE` two owners of one code are supposed to get. Tests need `resetStorage()`
|
|
201
170
|
in `beforeEach`.
|
|
202
|
-
- `image.ts` owns no pixels: core's `transformImageBytes
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
core can PROBE). `as const satisfies readonly ImageFormat[]` makes it a strict subset by the
|
|
208
|
-
compiler; `isVariantFormat` takes `string` so `probeImage(bytes).format` narrows with no cast.
|
|
209
|
-
`scripts/render-modes.ts` holds the `IMAGE_FORMATS` row and `image.test.ts` fails if either core
|
|
210
|
-
name reappears in `src/index.ts`.
|
|
211
|
-
- `transformImage()` must encode at exactly `fitDimensions()`'s size, and passes `format`
|
|
212
|
-
explicitly (`?? 'webp'`, which then rejects): bytes that disagree with the `variantKey`
|
|
213
|
-
extension, or with the `width`/`height` `@ultimat3/seo` inlined, are the layout shift the
|
|
214
|
-
whole path exists to prevent. That is why it derives the box itself and asks core for
|
|
215
|
-
`fit: 'cover'` — core's `contain` letterboxes to the requested box, this API fits inside it.
|
|
171
|
+
- `image.ts` owns no pixels: core's `transformImageBytes` is the only scaler; image failures surface
|
|
172
|
+
unwrapped. †
|
|
173
|
+
- **`VARIANT_FORMATS` is this package's format vocabulary, and `IMAGE_FORMATS` is core's.** †
|
|
174
|
+
- `transformImage()` encodes at exactly `fitDimensions()`'s size, `format` explicit — anything else
|
|
175
|
+
is a layout shift. †
|
|
216
176
|
- Bun's S3 flag is `virtualHostedStyle`; our `forcePathStyle` is its inverse.
|
|
217
|
-
- **No `region` signs for `auto
|
|
218
|
-
`AuthorizationHeaderMalformed … expecting "<region>"`. `put`/`copy`/`list`/`delete` ask
|
|
219
|
-
`regionMismatch()` BEFORE their own verdict, so the code is `X_CONFIG_INVALID` and the fix is the
|
|
220
|
-
region — never `X_STORAGE_LIST_FAILED` with a fix about `s3:ListBucket`. The captured region is
|
|
221
|
-
held to `[A-Za-z0-9-]`: a provider's sentence may put nothing else into a `fix:`. A HEAD has no
|
|
222
|
-
body, so `exists()`/`stat()` never see it. `driver-s3.live.test.ts` runs with no region on purpose.
|
|
177
|
+
- **No `region` signs for `auto`** (R2's; AWS refuses it) — `regionMismatch()` makes it `X_CONFIG_INVALID` naming `S3_REGION`. Full rule: `docs/history/storage.md`.
|
|
223
178
|
- The signature check runs BEFORE the expiry check. Do not reorder.
|
|
224
179
|
- `timingSafeEqual` is `@ultimat3/core`'s (`signed-url.ts` imports it) — the same
|
|
225
180
|
implementation `@ultimat3/auth` uses, not a second copy. Add new secret comparisons through it.
|
|
@@ -230,31 +185,19 @@ Gotchas:
|
|
|
230
185
|
always sets one, so such a URL is hand-rolled, and trusting the uploader's header instead is the
|
|
231
186
|
only other option.
|
|
232
187
|
- **The signed base is the disk's REGISTERED name, and it is stated once — on the driver.** Full rule: `docs/history/storage.md`.
|
|
233
|
-
- **`accept.ts` asks the `isTenantScoped`/`isWithinOrg` PAIR,
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
and then fail the exact-case membership test. Do not "simplify" either half.
|
|
238
|
-
- **`AcceptSignedUploadInput.checksum` is what makes `uploadPolicy({ requireChecksum: true })`
|
|
239
|
-
reachable.** Without it that option could only ever fail, because nothing on the accept path
|
|
240
|
-
could declare a hash. It travels like `declaredContentType`: the route reads a header and hands
|
|
241
|
-
it over, and `validateUpload` hashes the bytes itself. The browser half does NOT send one — a
|
|
242
|
-
custom header on an S3 presigned PUT is a signature question this package cannot answer.
|
|
188
|
+
- **`accept.ts` asks the `isTenantScoped`/`isWithinOrg` PAIR, as `runtime-storage.ts` does.** Do not
|
|
189
|
+
"simplify" either half. †
|
|
190
|
+
- **`AcceptSignedUploadInput.checksum` is what makes `requireChecksum: true` reachable**; the browser
|
|
191
|
+
half sends none. †
|
|
243
192
|
- `orgId` is required on both halves of `accept.ts` and is the ACTOR's, never a request field. A
|
|
244
193
|
signed URL is a capability; a leaked capability must still not cross a tenant.
|
|
245
194
|
- `X_STORAGE_ORG_MISMATCH` maps to **404**, not 403 (`@ultimat3/http`'s `error-map.ts`). 403 would
|
|
246
195
|
confirm a key exists to the one caller who must not learn it.
|
|
247
196
|
- **Local `lastModified` is the `Clock`'s, kept in the sidecar** — one instant from `put`/`copy`
|
|
248
197
|
to every read; mtime only with no trusted sidecar.
|
|
249
|
-
- `upload-client.ts`
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
ships, and the one file slice 15's `browser-transport` guard exempts for it — the header says so.
|
|
253
|
-
- **`fetchSignedPut` goes through `@ultimat3/core`'s `clientTransport`** (`As of 2026-09-22`,
|
|
254
|
-
plan 101 slice 05): `rawBody` sends the `Blob` verbatim under the grant's content type with no
|
|
255
|
-
default header, and `decodeError` keeps a refusal `X_STORAGE_UPLOAD_FAILED` with the disk's
|
|
256
|
-
status. A request with no response at all is the transport's `X_CLIENT_TRANSPORT_FAILED` — it was a
|
|
257
|
-
bare `TypeError` before. This file calls no `fetch`; the XHR above is the one exemption.
|
|
198
|
+
- `upload-client.ts` uses `XMLHttpRequest` for upload progress — the framework's one declared XHR
|
|
199
|
+
seam. †
|
|
200
|
+
- **`fetchSignedPut` goes through `@ultimat3/core`'s `clientTransport`**; it calls no `fetch`. †
|
|
258
201
|
- **Bundle** (`uploadFile` only, browser, minified, `As of 2026-09-22`): **17,767 B**.
|
|
259
202
|
- `exactOptionalPropertyTypes` is on — declare optional fields as `x?: T | undefined`.
|
|
260
203
|
|
package/README.md
CHANGED
|
@@ -58,15 +58,20 @@ const stored = [...sessions.objects().values()].map((bytes) => new TextDecoder()
|
|
|
58
58
|
|
|
59
59
|
Server-side only, like both other drivers: nothing an island imports reaches it.
|
|
60
60
|
|
|
61
|
+
Every call on a disk `defineStorage` resolves is a `storage.<verb>` event on `@ultimat3/core`'s instrumentation bus (`put`, `get`, `stat`, `copy`, `delete`, `list`; `{ disk, key, bytes?, durationMs, error? }`) inside a `storage.<verb>` client span. `disk()` hands out that instrumented view, so it is not `===` the driver you declared — compare `.name`. A driver you hold directly reports nothing.
|
|
62
|
+
|
|
61
63
|
`region` is what requests are signed for; unset, that is `auto`. An endpoint that expects another
|
|
62
64
|
region refuses the first write, list or delete with `X_CONFIG_INVALID`, and the fix names the
|
|
63
65
|
`S3_REGION` to set.
|
|
64
66
|
|
|
65
67
|
`copy(from, to)` is on the contract so a promotion is not a download-and-reupload:
|
|
66
68
|
`promoteAttachment` used to `get()` the whole object into the app and `put()` it back, a gigabyte
|
|
67
|
-
through the pod to rename a 500MB attachment. `localDriver` does a real file copy; `s3Driver`
|
|
68
|
-
|
|
69
|
-
|
|
69
|
+
through the pod to rename a 500MB attachment. `localDriver` does a real file copy; `s3Driver` sends
|
|
70
|
+
one signed `CopyObject` (directive COPY), so the bytes never leave the provider and the source's
|
|
71
|
+
type, `cacheControl` and `metadata` travel, as on local and memory. Past CopyObject's 5 GiB ceiling
|
|
72
|
+
it is a multipart `UploadPartCopy` (512 MiB parts, grown past 10,000), every part pinned to the
|
|
73
|
+
revision a HEAD read, carrying the same headers, aborted on any failure. Both arguments go through
|
|
74
|
+
`assertSafeKey`.
|
|
70
75
|
|
|
71
76
|
One S3 driver covers all three backends — the difference is `endpoint` + `forcePathStyle`.
|
|
72
77
|
Credentials are **env var NAMES** (`accessKeyIdEnv`, default `S3_ACCESS_KEY_ID`), never
|
|
@@ -176,6 +181,16 @@ do {
|
|
|
176
181
|
|
|
177
182
|
Before 25.0.0 it was `ListPage { objects, truncated, cursor? }`, its own shape beside core's.
|
|
178
183
|
|
|
184
|
+
### Writing a driver: `@ultimat3/storage/adapter`
|
|
185
|
+
|
|
186
|
+
`StorageDriver`, the shapes it reads and answers, the error factories a driver raises
|
|
187
|
+
(`objectNotFound`, `putFailed`, `readFailed`, …) and the helpers every driver refuses with the same
|
|
188
|
+
way (`assertPutOptions`, `toBytes`, `resolveListLimit`, `etagOf`). Hand one to the boot as
|
|
189
|
+
`storage: myDriver()` in `apps/web/runtime.ts`: it becomes the default disk, registered under the
|
|
190
|
+
driver's `name`, in place of the `S3_ENDPOINT`/embedded-disk decision (31.0 — the field took a whole
|
|
191
|
+
`Storage` before). Several disks stay the app's `defineStorage()`. The suite is
|
|
192
|
+
`@ultimat3/testing/conformance`'s `storageDriverConformance`.
|
|
193
|
+
|
|
179
194
|
## `put()` is for objects that fit in memory
|
|
180
195
|
|
|
181
196
|
`put()` buffers the whole body — size and checksum have to be known before the object exists —
|
|
@@ -200,12 +215,17 @@ in a bucket unmeasured, so a later `get()` of it was heap growth the uploader pi
|
|
|
200
215
|
|
|
201
216
|
`streamRange` reads no byte before `first`: s3 and local through `slice(...).stream()`, memory from
|
|
202
217
|
its copy. Inclusive and clamped like `get`'s range, with no ceiling because nothing is buffered;
|
|
203
|
-
past the end is `X_STORAGE_RANGE_UNSATISFIABLE`. `
|
|
218
|
+
past the end is `X_STORAGE_RANGE_UNSATISFIABLE`. `stream(key, { etag })` and
|
|
219
|
+
`streamRange(key, range, { etag })` serve that revision only — another is `X_STORAGE_OBJECT_CHANGED`
|
|
220
|
+
(409): s3 sends `If-Match`, local checks the etag at the call and again at the first read, memory compares. `/_storage`
|
|
221
|
+
pins every read to the etag its headers came from. `putStream` takes `contentType`, `cacheControl`,
|
|
204
222
|
`metadata` and a REQUIRED `maxBytes` (`PutStreamOptions`). Past it the source is cancelled, nothing
|
|
205
223
|
is stored and the key stays as it was (`X_STORAGE_TOO_LARGE`). `checksum`, `retention`, `legalHold`
|
|
206
224
|
and `serverSideEncryption` need the bytes up front: the type omits them and an untyped caller gets
|
|
207
225
|
`X_INVARIANT`. On s3 a body under one part is one signed PUT and past it a signed multipart upload,
|
|
208
|
-
|
|
226
|
+
8 MiB parts (doubled every 1,000 parts, so 10,000 reach past S3's 5 TiB; never derived from
|
|
227
|
+
`maxBytes`), up to `STREAM_PART_BUFFERS` (4) parts held at once — in flight plus the one filling,
|
|
228
|
+
so memory is at most 32 MiB per stream — aborted on every exit but the Complete: Bun's
|
|
209
229
|
`S3File.writer()` completes on `end(error)` and sends no `x-amz-meta-*`. Local stages under
|
|
210
230
|
`.meta/.tmp/`, then runs `put()`'s commit steps. A driver outside this package reuses
|
|
211
231
|
`assertPutStream`, `pumpBounded`, `streamLimit`, `assertStreamRange`, `streamedRange` and `signedByteBound`.
|
|
@@ -418,8 +438,13 @@ Neither owns a `Request`, a `Response` or a status number — mounting is the ho
|
|
|
418
438
|
**Step 3 ships.** `As of 2026-09-24` `@ultimat3/cli` mounts `PUT /_storage/:disk/*key` beside the
|
|
419
439
|
`GET` in `x dev` and `runRole` (#523), so an app writes steps 1 and 2 only. The route calls
|
|
420
440
|
`signedUploadConstraints({ url, disk, orgId })` first. That is the verified PUT constraints with
|
|
421
|
-
no bytes, and the signed `maxBytes` caps the body read. Then it calls `
|
|
422
|
-
policy built from the grant's own signed type and ceiling
|
|
441
|
+
no bytes, and the signed `maxBytes` caps the body read. Then it calls `acceptSignedUploadStream`
|
|
442
|
+
with a policy built from the grant's own signed type and ceiling: the first 64 KiB are sniffed, the
|
|
443
|
+
rest piped into `disk.putStream`, so memory per upload is the window, not the body. A body that
|
|
444
|
+
ends inside the window, an XML type, and a required checksum nobody declared are read whole and
|
|
445
|
+
judged by `acceptSignedUpload`. A local or memory disk given `maxRequestBytes` (the app's
|
|
446
|
+
`http.maxRequestBodyBytes`) refuses at signing a single-request grant past it (`X_STORAGE_TOO_LARGE`,
|
|
447
|
+
the fix naming `grantMultipart`). The routes serve `definedStorage()`,
|
|
423
448
|
the process's one registry, which is the app's `defineStorage` when it declared one (#524).
|
|
424
449
|
|
|
425
450
|
## Uploads in parts — `grantMultipart`, `signPart`, `listParts`, `completeUpload`, `abortUpload`
|
|
@@ -450,10 +475,8 @@ An upload happens **before** the row it belongs to exists, so it lands at
|
|
|
450
475
|
| Call | Key |
|
|
451
476
|
|---|---|
|
|
452
477
|
| `pendingKey(orgId, uploadName(id, filename))` | `org/o1/pending/u-1.png` |
|
|
453
|
-
| `quarantineKey(orgId, name)` | `org/o1/pending/quarantine/u-1.png` |
|
|
454
478
|
| `attachmentKey(orgId, { entity, id, field }, name)` | `org/o1/post/p-1/cover/u-1.png` |
|
|
455
479
|
| `promoteAttachment({ disk, key, orgId, target, policy })` | `stat`, `copy`, then `delete` — never the reverse. **`policy` is required**: the object is measured here (`stat().size` over `policy.maxBytes` is `X_STORAGE_TOO_LARGE`, and it stays under `pending/` for the sweep), because an s3 presign bounds no size. Then one ranged read of the first 64 KiB through `validateStoredHead`, because it binds no type or bytes either: a contradiction is `X_STORAGE_TYPE_REJECTED`, and it stays pending too. The URL stays valid after that read, so `copy(from, to, { ifMatch })` names the revision judged (`X_STORAGE_OBJECT_CHANGED`, on s3 a signed CopyObject), and the copy is checked: another size is refused and deleted, and an etag that differs or is unknown (SSE-KMS mints one per copy) means the copy's own head is judged. A call after the source is gone, with the attached key present, is `X_STORAGE_ALREADY_PROMOTED` (409, `meta.attachedKey`): an earlier call moved the bytes, so this one answers the row that holds them and never deletes that key. The key must be a **pending** one (`isPendingKey`): an attached key a client sent back is another row's file, and moving it would delete the victim's copy (`X_STORAGE_NOT_PENDING`) |
|
|
456
|
-
| `releaseQuarantine({ disk, key, orgId })` | quarantine → pending; returns the released key |
|
|
457
480
|
| `sweepOrphans({ disk, orgId, olderThanMs })` | `{ deleted, failed }` for stale `pending/` keys. `olderThanMs` is a whole number of 0 or more, refused otherwise (`X_INVARIANT`) — `NaN` read as "everything is old enough" and deleted an upload made a moment ago |
|
|
458
481
|
| `deleteVariants({ disk, key })` | `{ deleted, failed }` for every stored variant of one source, every revision (`list({ prefix: `${key}@` })`). **An erasure of one object must call it**: `disk.delete` leaves the variants' bytes behind. The source is the caller's to delete; a variant keeps its prefix, so an org-wide erasure by prefix already collects them |
|
|
459
482
|
|
|
@@ -466,31 +489,80 @@ refused delete goes in `failed` with the disk's own words and the sweep keeps go
|
|
|
466
489
|
array of "deleted" keys put every refusal in it, so an erasure sweep over a bucket whose policy
|
|
467
490
|
had lost `s3:DeleteObject` reported 200 objects gone that were all still there.
|
|
468
491
|
|
|
469
|
-
###
|
|
492
|
+
### Scanning on confirm: the `uploadScanner` port
|
|
470
493
|
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
494
|
+
`As of 2026-11-09`. An `uploadScanner` in `apps/<app>/runtime.ts` is asked by `promoteAttachment`
|
|
495
|
+
about every upload it attaches. It scans a per-attempt copy (`pending/scan/<random>-<name>`): server
|
|
496
|
+
named, so no presigned URL rewrites it after the scan, and never the row's key, so a refused second
|
|
497
|
+
confirm of a double-click cannot delete the file the first one attached. The clean attempt is what
|
|
498
|
+
is copied onto the row's key:
|
|
499
|
+
|
|
500
|
+
| Verdict | `promoteAttachment` |
|
|
501
|
+
|---|---|
|
|
502
|
+
| `{ verdict: 'clean' }` | promotes, as without a scanner |
|
|
503
|
+
| `{ verdict: 'infected', signature }` | deletes the pending upload and the attempt, throws `X_STORAGE_UPLOAD_INFECTED` (422, `meta.signature`) |
|
|
504
|
+
| `{ verdict: 'hold', reason }` | deletes the attempt, keeps the upload pending, throws `X_STORAGE_QUARANTINED` (409, `meta.reason`) — a decision made elsewhere has not cleared it yet. No policy overrides it; the same confirm promotes once the scanner answers clean |
|
|
505
|
+
| rejects (daemon down, timeout), or the attempt cannot be read | deletes the attempt, keeps the upload pending, throws `X_STORAGE_SCAN_UNAVAILABLE` (503) — **fail closed by default, in every environment**; `uploadPolicy({ onScanUnavailable: 'accept' })` promotes unscanned instead, unless the scanner declares `holds: true`. An infected or held verdict is refused under either |
|
|
476
506
|
|
|
477
507
|
```ts
|
|
478
|
-
|
|
479
|
-
|
|
508
|
+
// apps/web/runtime.ts
|
|
509
|
+
import type { RuntimeOverrides } from '@ultimat3/cli/serve';
|
|
510
|
+
import { memoryUploadScanner } from '@ultimat3/storage';
|
|
511
|
+
// In production, your own UploadScanner here: e.g. one streaming `body` to clamd's INSTREAM
|
|
512
|
+
// over Bun.connect, mapping `OK` → clean, `<sig> FOUND` → infected, anything else → reject.
|
|
513
|
+
export const runtime = {
|
|
514
|
+
uploadScanner: memoryUploadScanner(),
|
|
515
|
+
} satisfies RuntimeOverrides;
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
`UploadScanner` (`@ultimat3/storage/adapter`) is `{ name, holds?, scan({ key, pendingKey, size, contentType, body }) }`,
|
|
519
|
+
`body` a `ReadableStream`. `memoryUploadScanner()` is the reference: it flags the EICAR test string
|
|
520
|
+
(`EICAR_TEST_STRING`) anywhere in the stream, across chunk boundaries. Hold your adapter to the
|
|
521
|
+
contract with `behavesLike(uploadScannerConformance, () => yours)` from
|
|
522
|
+
`@ultimat3/testing/conformance`. No scanner ships: ClamAV, a vendor API, its licence and its
|
|
523
|
+
latency budget are the app's (axiom 8). No scanner installed, nothing is scanned.
|
|
524
|
+
|
|
525
|
+
### Hold: a decision made elsewhere — a review queue, an async vendor scan
|
|
526
|
+
|
|
527
|
+
Magic-byte sniffing closes stored XSS. It is **not** malware scanning: `application/zip` is the
|
|
528
|
+
OOXML container, so a macro-laden `.docx` passes `validateUpload` as a clean one does. A verdict
|
|
529
|
+
that is not ready inside the confirm is the scanner's `hold`, keyed by `pendingKey` — the upload's
|
|
530
|
+
own key, stable across retries (`key` is the per-attempt copy). One scan path; the v30
|
|
531
|
+
`grantUpload({ quarantine: true })` + `releaseQuarantine` flow is retired.
|
|
480
532
|
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
533
|
+
```ts
|
|
534
|
+
import type { UploadScanner } from '@ultimat3/storage/adapter';
|
|
535
|
+
|
|
536
|
+
// `approved` is your app's own store — a table a moderator's action writes.
|
|
537
|
+
export const reviewScanner = (approved: (key: string) => Promise<boolean>): UploadScanner => ({
|
|
538
|
+
name: 'review',
|
|
539
|
+
holds: true,
|
|
540
|
+
async scan({ pendingKey, body }) {
|
|
541
|
+
await body.cancel();
|
|
542
|
+
return (await approved(pendingKey))
|
|
543
|
+
? { verdict: 'clean' }
|
|
544
|
+
: { verdict: 'hold', reason: 'awaiting a moderator' };
|
|
545
|
+
},
|
|
546
|
+
});
|
|
484
547
|
```
|
|
485
548
|
|
|
486
|
-
|
|
487
|
-
`
|
|
549
|
+
**`holds: true` is required of a scanner that answers `hold`**, and makes its outage always refuse:
|
|
550
|
+
`approved` throwing (your database is down) is `X_STORAGE_SCAN_UNAVAILABLE` even under
|
|
551
|
+
`onScanUnavailable: 'accept'`, because only that store knows what is held. A `hold` from a scanner
|
|
552
|
+
that did not declare it is `X_INVARIANT`, and nothing is attached.
|
|
553
|
+
|
|
554
|
+
A held upload is still under `pending/`, so `sweepOrphans` collects one nobody ever cleared: keep
|
|
555
|
+
`olderThanMs` past your review window, or spare it with `keep`.
|
|
556
|
+
|
|
557
|
+
**A grant with a `target` is refused while a scanner is installed** (`X_STORAGE_TARGET_UNSCANNED`,
|
|
558
|
+
`grantUpload` and `grantMultipart` both): its key never passes through `promoteAttachment`, the one
|
|
559
|
+
place a scan runs. Grant with no `target`, then promote.
|
|
488
560
|
|
|
489
561
|
## A person's objects, for a subject export or erase
|
|
490
562
|
|
|
491
563
|
`storageSubjectTarget(() => disk(), keysOf, { name? })` is a `defineSubject({ targets })` entry
|
|
492
564
|
(`@ultimat3/jobs`): `keysOf(subject)` answers the person's keys (a list, or `disk().list({ prefix })`
|
|
493
|
-
over a prefix the app owns). The export lists each object that exists (`STORAGE_SUBJECT_COLUMNS`);
|
|
565
|
+
over a prefix the app owns). The export lists each object that exists (columns: `STORAGE_SUBJECT_COLUMNS` in `src/subject-target.ts`);
|
|
494
566
|
the erase deletes them in bounded batches. There is no listing by owner: a key the app does not
|
|
495
567
|
name is not found.
|
|
496
568
|
|
|
@@ -511,8 +583,11 @@ name is not found.
|
|
|
511
583
|
| `X_STORAGE_UPLOAD_FAILED` | client half: the presigned `PUT` answered non-2xx or never landed |
|
|
512
584
|
| `X_STORAGE_DELETE_FAILED` | the disk REFUSED a delete — denied `s3:DeleteObject`, a throttle, an expired credential, a read-only mount. An **absent** key is still not an error |
|
|
513
585
|
| `X_STORAGE_LIST_FAILED` | the disk REFUSED a listing — denied `s3:ListBucket`, a throttle, an unreadable root. An **empty** disk is still not an error |
|
|
514
|
-
| `
|
|
586
|
+
| `X_STORAGE_TARGET_UNSCANNED` | `grantUpload`/`grantMultipart` with a `target` while an `uploadScanner` is installed — the key would skip the scan |
|
|
587
|
+
| `X_STORAGE_QUARANTINED` | the installed `uploadScanner` answered `hold`: the upload stays pending until it answers clean; `meta.reason` is the scanner's |
|
|
515
588
|
| `X_STORAGE_NOT_PENDING` | `promoteAttachment` on a key outside the org's `pending/` prefix — most often another row's attached key |
|
|
589
|
+
| `X_STORAGE_UPLOAD_INFECTED` | the installed `uploadScanner` found malware in an upload `promoteAttachment` was attaching; both keys are deleted, `meta.signature` names the detection |
|
|
590
|
+
| `X_STORAGE_SCAN_UNAVAILABLE` | the installed `uploadScanner` rejected instead of answering, under the default `onScanUnavailable: 'refuse'`; the upload stays pending for a retry |
|
|
516
591
|
| `X_STORAGE_ALREADY_PROMOTED` | `promoteAttachment` on a pending key an earlier call already moved (a retried or doubled confirm); `meta.attachedKey` names where the bytes are |
|
|
517
592
|
| `X_STORAGE_PUT_FAILED` | the disk REFUSED a `put()`/`copy()` — `EACCES`, `ENOSPC`, `EROFS`, a provider's refused PUT |
|
|
518
593
|
| `X_STORAGE_OBJECT_LOCKED` | local, memory: a delete, overwrite or copy onto a key under retention or a legal hold; the cause names the key, the mode and `retainUntil` or the hold, and the bytes are unchanged. Never raised by s3, where the provider keeps the locked version |
|
|
@@ -538,6 +613,24 @@ a job boundary the class is gone and the `code` is what survives — match on th
|
|
|
538
613
|
`variantKey()`, `srcsetDescriptors()`, `fitDimensions()` are pure — `@ultimat3/seo` builds
|
|
539
614
|
`srcset` from them without decoding a byte.
|
|
540
615
|
|
|
616
|
+
`variant(key, { disk, images, width, format? })` (`image-variant.ts`) is ONE variant, kept: made by
|
|
617
|
+
the `images` driver you pass — `runtime.ts`'s `images` port value, `@ultimat3/seo`'s
|
|
618
|
+
`builtinImageDriver({ read })` or a CDN's, any `ImageTransformDriver` (storage takes it structurally;
|
|
619
|
+
it is tier 1 beside seo and imports none of it) — and written under
|
|
620
|
+
`variantKey(key, { width, format, revision })`. A second call is a `stat`, never a decode
|
|
621
|
+
(`{ key, object, created }`); an overwritten source is a new revision, so a new key. `format`
|
|
622
|
+
omitted keeps a png/jpeg/webp source's encoding. Refused with core's `X_IMAGE_UNSUPPORTED`: a source
|
|
623
|
+
that is not a raster image (a PDF, an SVG), a gif/avif source with no `format`, and a driver that
|
|
624
|
+
answers another encoding than the key names. A source overwritten while the driver read it is
|
|
625
|
+
`X_STORAGE_OBJECT_CHANGED`, and nothing is stored. `As of 2026-11-09`.
|
|
626
|
+
|
|
627
|
+
```ts
|
|
628
|
+
import { builtinImageDriver } from '@ultimat3/seo';
|
|
629
|
+
import { disk, variant } from '@ultimat3/storage';
|
|
630
|
+
const images = builtinImageDriver({ read: async (src) => (await disk().get(src)).bytes });
|
|
631
|
+
const { key } = await variant('org/o1/post/p1/cover/u1.png', { disk: disk(), images, width: 640, format: 'webp' });
|
|
632
|
+
```
|
|
633
|
+
|
|
541
634
|
`warmVariants({ disk, key, widths, formats })` (`image-warm.ts`) writes every width x format of a
|
|
542
635
|
stored image under the key `/media` reads — `variantKey(key, { width, format, revision })`, the
|
|
543
636
|
revision off the etag of the bytes it read — so the job an upload enqueues pays the decode and no
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/storage",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "31.1.0",
|
|
4
4
|
"description": "Named disks over Bun.file and Bun.s3: safe keys, signed URLs, sniffed uploads",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ultimate",
|
|
@@ -24,7 +24,8 @@
|
|
|
24
24
|
"provenance": true
|
|
25
25
|
},
|
|
26
26
|
"exports": {
|
|
27
|
-
".": "./src/index.ts"
|
|
27
|
+
".": "./src/index.ts",
|
|
28
|
+
"./adapter": "./src/adapter.ts"
|
|
28
29
|
},
|
|
29
30
|
"files": [
|
|
30
31
|
"src",
|
|
@@ -42,6 +43,6 @@
|
|
|
42
43
|
"test": "bun test"
|
|
43
44
|
},
|
|
44
45
|
"dependencies": {
|
|
45
|
-
"@ultimat3/core": "
|
|
46
|
+
"@ultimat3/core": "31.1.0"
|
|
46
47
|
}
|
|
47
48
|
}
|