@ultimat3/storage 3.0.0 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +42 -6
- package/README.md +9 -3
- package/package.json +2 -2
- package/src/accept.ts +47 -12
- package/src/driver-local.ts +59 -6
- package/src/driver-s3.ts +5 -2
- package/src/driver.ts +60 -0
- package/src/errors.ts +17 -0
- package/src/index.ts +4 -0
- package/src/path.ts +8 -1
- package/src/storage.ts +25 -2
- package/src/upload.ts +14 -7
package/CLAUDE.md
CHANGED
|
@@ -50,6 +50,33 @@ Gotchas:
|
|
|
50
50
|
in a single `test()` per claim, so neither disk can move alone. Where the two genuinely cannot
|
|
51
51
|
agree it pins the DIVERGENCE, with the reason — that is the honest form, and it still fails the
|
|
52
52
|
day either half changes.
|
|
53
|
+
- **The local `list()` globs with `dot: true`, and the `META_DIR` skip is the REAL filter**
|
|
54
|
+
(`As of 2026-08`). `Bun.Glob('**/*')` matches no dot-prefixed entry, so every object whose key
|
|
55
|
+
had one — `.hidden.txt`, `org/o1/pending/.x.png`, the `.metadata/a.json` `path.test.ts` pins as
|
|
56
|
+
legal — was missing from the listing while `put`/`get`/`exists` handled it normally and
|
|
57
|
+
`s3Driver.list()` returned it: the key space and the listing disagreed by construction.
|
|
58
|
+
`sweepOrphans` pages through `list()`, so those objects were reported as erased while still on
|
|
59
|
+
disk — a false erasure report by OMISSION, the same lie a swallowed listing error tells. The
|
|
60
|
+
`.meta/` skip one line below was unreachable until this landed and this file called it "a second
|
|
61
|
+
line of defence"; it is now the only thing keeping the sidecar tree out of the object namespace,
|
|
62
|
+
and it folds case for `isSafeKey`'s reason. Pinned in `driver-parity.test.ts`.
|
|
63
|
+
- **A `list({ limit })` is a positive integer or a refusal** — `resolveListLimit` at the
|
|
64
|
+
`ListOptions` seam, so both disks answer one way. `limit: 0` used to slice `[0, 0)` on the local
|
|
65
|
+
disk and then drop its own `truncated` flag (`truncated && last !== undefined`, with an empty
|
|
66
|
+
page), so a paging caller read "complete, and there is nothing here" over a full disk, while the
|
|
67
|
+
s3 disk handed `maxKeys: 0` to the provider. Core's `assert` (`X_INVARIANT`), for the reason
|
|
68
|
+
`@ultimat3/seo`'s `chunk()` uses it: a bound with no code of its own is still a coded refusal.
|
|
69
|
+
- **`META_DIR` is reserved case-INSENSITIVELY** (`As of 2026-08`), exactly as `isTenantScoped`
|
|
70
|
+
folds and for the same filesystem: `.META/a.txt.json` was a legal key that writes
|
|
71
|
+
`<root>/.META/a.txt.json`, which on APFS and NTFS IS `<root>/.meta/a.txt.json` — the sidecar for
|
|
72
|
+
object `a.txt` — so a caller able to name a key rewrote another object's recorded `contentType`.
|
|
73
|
+
The whole SEGMENT is compared, never a prefix: `.metadata/a.json` is an ordinary key and stays
|
|
74
|
+
one, and `path.test.ts` pins both halves.
|
|
75
|
+
- **`disk(name)` resolves through a `Map`, never `config.disks[name]`.** The bracket read walked
|
|
76
|
+
the prototype chain, so `disk('constructor')` handed back the `Object` function and the next
|
|
77
|
+
`.put()` was a bare `TypeError` — `X_STORAGE_DISK_UNKNOWN` unreachable for `constructor`,
|
|
78
|
+
`toString`, `valueOf`, `hasOwnProperty` and `__proto__`, in a function whose own
|
|
79
|
+
`default:` check already read `Object.keys`.
|
|
53
80
|
- **`list()` is idempotent for an EMPTY disk and for nothing else** (`As of 2026-08`) — exactly
|
|
54
81
|
`delete()`'s rule, one call to the left, and both drivers broke it in opposite directions. The
|
|
55
82
|
local one caught EVERYTHING and answered `{ objects: [], truncated: false }`, so `EACCES` on the
|
|
@@ -113,7 +140,8 @@ Gotchas:
|
|
|
113
140
|
itself a legal key, so an uploader could rewrite another object's recorded `contentType` to
|
|
114
141
|
`text/html` and have a route serve attacker HTML from the app's origin. Reserved in
|
|
115
142
|
`assertSafeKey`, so it holds for S3 too — a key valid on one driver and refused on another is two
|
|
116
|
-
key rules. The `list()` skip
|
|
143
|
+
key rules, and the reservation folds case — see above. The `list()` skip is not a second line of
|
|
144
|
+
defence but the only one: the glob yields the sidecar tree.
|
|
117
145
|
- **`localDriver` refuses to construct outside development without a usable signing secret**
|
|
118
146
|
(`X_ENV_MISSING`, borrowed from core). Usable means neither the `signingSecret` option nor
|
|
119
147
|
`STORAGE_SIGNING_SECRET` is missing, empty **or** the published `DEV_SIGNING_SECRET` — pasting
|
|
@@ -157,11 +185,19 @@ Gotchas:
|
|
|
157
185
|
- `acceptSignedUpload` refuses a URL signed with **no** content type (`unconstrained`). `grantUpload`
|
|
158
186
|
always sets one, so such a URL is hand-rolled, and trusting the uploader's header instead is the
|
|
159
187
|
only other option.
|
|
160
|
-
- **The signed base is
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
188
|
+
- **The signed base is the disk's REGISTERED name, and it is stated once — on the driver.**
|
|
189
|
+
`defineStorage` tells each driver its registration key (`StorageDriver.registerAs`), the driver
|
|
190
|
+
rebases its URLs onto `signedUrlBaseFor(diskName)`, and `accept.ts` reads `disk.signedUrlBase`
|
|
191
|
+
rather than deriving a second one. Both halves used to derive it from `disk.name` — the DRIVER
|
|
192
|
+
kind — so they agreed with each other and disagreed with the mounted `/_storage/:disk/*key`
|
|
193
|
+
route, which resolves the segment through the registry: a disk registered as `uploads` minted
|
|
194
|
+
`/_storage/local/...` and 404'd its own signatures. Latent only because every disk in this repo
|
|
195
|
+
happens to be named `local`. An explicit `baseUrl` on `localDriver` outranks the registration
|
|
196
|
+
(the operator saying where the route is mounted); an unregistered driver still mints under
|
|
197
|
+
`local`. One driver instance under two disk names is refused at `defineStorage`
|
|
198
|
+
(`X_CONFIG_INVALID`) — it could only mint under one of them. Before that, the base was stated
|
|
199
|
+
twice (`/_storage/local` in the driver, `/_storage` in `verifySignedUrl`'s default) and NO
|
|
200
|
+
genuine URL verified at all: the key parsed as `local/<key>`. `@ultimat3/cli`'s
|
|
165
201
|
`STORAGE_BASE_PATH` is a third statement of the mount prefix and should import
|
|
166
202
|
`DEFAULT_SIGNED_URL_BASE` instead.
|
|
167
203
|
- **`accept.ts` asks the `isTenantScoped`/`isWithinOrg` PAIR, exactly as `dev-storage.ts` does.**
|
package/README.md
CHANGED
|
@@ -50,6 +50,12 @@ The etag follows the same rule: a listed object with no sidecar reports `etag: '
|
|
|
50
50
|
answering otherwise means reading and hashing the whole object — which is what the local `list()`
|
|
51
51
|
used to do, once per sidecar-less row, sequentially. `get()` hashes out of bytes it already holds.
|
|
52
52
|
|
|
53
|
+
**A listing returns every key the key rules allow**, dot-prefixed segments included
|
|
54
|
+
(`.hidden.txt`, `org/o1/pending/.x.png`) — the local glob was blind to them until 2026-08, so a
|
|
55
|
+
disk answered `exists()` true and `list()` nothing for the same object. **`limit` is a positive
|
|
56
|
+
integer or a refusal** (`X_INVARIANT`, at the `ListOptions` seam): `limit: 0` used to read back as
|
|
57
|
+
a complete, empty page on the local disk and as `maxKeys: 0` on s3.
|
|
58
|
+
|
|
53
59
|
## `put()` is for objects that fit in memory
|
|
54
60
|
|
|
55
61
|
`put()` buffers the whole body — size and checksum have to be known before the object exists —
|
|
@@ -146,9 +152,9 @@ const { key } = await uploadFile({ file, grant: (request) => api.requestUpload(r
|
|
|
146
152
|
|
|
147
153
|
// 3. server, in the route mounted at `/_storage`: take it back, or refuse
|
|
148
154
|
const object = await acceptSignedUpload({
|
|
149
|
-
url: request.url, // baseUrl defaults to
|
|
150
|
-
secret, //
|
|
151
|
-
disk: disk('uploads'), // route mounted somewhere
|
|
155
|
+
url: request.url, // baseUrl defaults to `disk.signedUrlBase` — the very base
|
|
156
|
+
secret, // the driver signed under, `/_storage/<disk name>`. Pass one
|
|
157
|
+
disk: disk('uploads'), // only for a route mounted somewhere else.
|
|
152
158
|
orgId: ctx.actor.orgId,
|
|
153
159
|
bytes, declaredContentType: request.headers.get('content-type') ?? undefined,
|
|
154
160
|
policy: uploadPolicy({ maxBytes: 5e6 }),
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/storage",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "4.0.0",
|
|
4
4
|
"description": "Named disks over Bun.file and Bun.s3: safe keys, signed URLs, sniffed uploads",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -31,6 +31,6 @@
|
|
|
31
31
|
"test": "bun test"
|
|
32
32
|
},
|
|
33
33
|
"dependencies": {
|
|
34
|
-
"@ultimat3/core": "
|
|
34
|
+
"@ultimat3/core": "4.0.0"
|
|
35
35
|
}
|
|
36
36
|
}
|
package/src/accept.ts
CHANGED
|
@@ -8,9 +8,15 @@
|
|
|
8
8
|
|
|
9
9
|
import type { Clock } from '@ultimat3/core';
|
|
10
10
|
import type { SignedUrlMethod, StorageDriver, StorageObject, StorageRead } from './driver';
|
|
11
|
-
import {
|
|
11
|
+
import {
|
|
12
|
+
orgMismatch,
|
|
13
|
+
signedUrlExpired,
|
|
14
|
+
signedUrlRejected,
|
|
15
|
+
signedUrlUnverifiable,
|
|
16
|
+
tooLarge,
|
|
17
|
+
} from './errors';
|
|
12
18
|
import { isTenantScoped, isWithinOrg } from './path';
|
|
13
|
-
import type { SignedUrlConstraints } from './signed-url';
|
|
19
|
+
import type { SignedUrlConstraints, SignedUrlVerification } from './signed-url';
|
|
14
20
|
import { signedUrlBaseFor, verifySignedUrl } from './signed-url';
|
|
15
21
|
import type { UploadPolicy } from './upload';
|
|
16
22
|
import { normalizeContentType, uploadPolicy, validateUpload } from './upload';
|
|
@@ -18,11 +24,23 @@ import { normalizeContentType, uploadPolicy, validateUpload } from './upload';
|
|
|
18
24
|
export interface SignedRequestInput {
|
|
19
25
|
/** Absolute or route-relative — `verifySignedUrl` parses both. */
|
|
20
26
|
readonly url: string;
|
|
21
|
-
readonly secret: string;
|
|
22
27
|
/**
|
|
23
|
-
*
|
|
24
|
-
*
|
|
25
|
-
*
|
|
28
|
+
* OPTIONAL since the disk can verify its own. It was required, and that requirement is why the
|
|
29
|
+
* upload half of `/_storage` has never been mounted: `localDriver` closes over the secret it
|
|
30
|
+
* mints with and exposes it through no member, so a route holding a `Storage` registry had no
|
|
31
|
+
* value to put here. A disk implementing `verifySigned()` needs none.
|
|
32
|
+
*
|
|
33
|
+
* Still honoured, and it still wins, for a caller that verifies a URL some OTHER disk signed —
|
|
34
|
+
* a migration between disks, a test signing by hand.
|
|
35
|
+
*/
|
|
36
|
+
readonly secret?: string | undefined;
|
|
37
|
+
/**
|
|
38
|
+
* Defaults to the base THIS disk signs under (`disk.signedUrlBase`), never to the bare mount
|
|
39
|
+
* prefix: a second default here made every URL `localDriver` mints a signature-mismatch,
|
|
40
|
+
* because the key parsed as `local/<key>`. Read off the driver rather than re-derived from
|
|
41
|
+
* `disk.name`, which is the driver KIND — the segment is the registered disk name, and deriving
|
|
42
|
+
* it twice is what let both halves agree with each other and disagree with the mounted route.
|
|
43
|
+
* Pass one only for a route mounted somewhere else.
|
|
26
44
|
*/
|
|
27
45
|
readonly baseUrl?: string | undefined;
|
|
28
46
|
readonly disk: StorageDriver;
|
|
@@ -53,16 +71,33 @@ export interface AcceptSignedUploadInput extends SignedRequestInput {
|
|
|
53
71
|
* the reason it exists is that a forged URL must never learn "the signature was fine, just late".
|
|
54
72
|
* The org check runs on the verified key, so an attacker cannot probe org names with a fake one.
|
|
55
73
|
*/
|
|
74
|
+
async function verify(input: SignedRequestInput): Promise<SignedUrlVerification> {
|
|
75
|
+
// An explicit secret first, so every call that shipped behaves exactly as it did — including
|
|
76
|
+
// one verifying a URL a DIFFERENT disk signed, which the driver could not answer for.
|
|
77
|
+
if (input.secret !== undefined && input.secret !== '') {
|
|
78
|
+
return verifySignedUrl({
|
|
79
|
+
url: input.url,
|
|
80
|
+
secret: input.secret,
|
|
81
|
+
baseUrl: input.baseUrl ?? input.disk.signedUrlBase ?? signedUrlBaseFor(input.disk.name),
|
|
82
|
+
...(input.clock === undefined ? {} : { clock: input.clock }),
|
|
83
|
+
});
|
|
84
|
+
}
|
|
85
|
+
// The disk that signed it verifies it, holding the key it never hands out. `baseUrl` is the
|
|
86
|
+
// driver's own here for the same reason the secret is: both are the minting half's, and stating
|
|
87
|
+
// either one a second time is what made every `localDriver` URL a signature mismatch once.
|
|
88
|
+
const own = input.disk.verifySigned;
|
|
89
|
+
if (own === undefined) throw signedUrlUnverifiable(input.disk.name);
|
|
90
|
+
return own.call(input.disk, {
|
|
91
|
+
url: input.url,
|
|
92
|
+
...(input.clock === undefined ? {} : { clock: input.clock }),
|
|
93
|
+
});
|
|
94
|
+
}
|
|
95
|
+
|
|
56
96
|
async function constraintsFor(
|
|
57
97
|
input: SignedRequestInput,
|
|
58
98
|
method: SignedUrlMethod,
|
|
59
99
|
): Promise<SignedUrlConstraints> {
|
|
60
|
-
const result = await
|
|
61
|
-
url: input.url,
|
|
62
|
-
secret: input.secret,
|
|
63
|
-
baseUrl: input.baseUrl ?? signedUrlBaseFor(input.disk.name),
|
|
64
|
-
...(input.clock === undefined ? {} : { clock: input.clock }),
|
|
65
|
-
});
|
|
100
|
+
const result = await verify(input);
|
|
66
101
|
if (!result.ok) {
|
|
67
102
|
throw result.reason === 'expired'
|
|
68
103
|
? signedUrlExpired(input.url, result.detail)
|
package/src/driver-local.ts
CHANGED
|
@@ -6,11 +6,11 @@
|
|
|
6
6
|
import { type Clock, isLocal, resolveEnvironment, stringField, systemClock } from '@ultimat3/core';
|
|
7
7
|
import {
|
|
8
8
|
DEFAULT_CONTENT_TYPE,
|
|
9
|
-
DEFAULT_LIST_LIMIT,
|
|
10
9
|
etagOf,
|
|
11
10
|
type ListOptions,
|
|
12
11
|
type ListPage,
|
|
13
12
|
type PutOptions,
|
|
13
|
+
resolveListLimit,
|
|
14
14
|
type SignedUrlOptions,
|
|
15
15
|
type StorageBody,
|
|
16
16
|
type StorageDriver,
|
|
@@ -29,7 +29,8 @@ import {
|
|
|
29
29
|
storageNotImplemented,
|
|
30
30
|
} from './errors';
|
|
31
31
|
import { assertSafeKey, META_DIR } from './path';
|
|
32
|
-
import {
|
|
32
|
+
import type { SignedUrlVerification } from './signed-url';
|
|
33
|
+
import { buildSignedUrl, signedUrlBaseFor, verifySignedUrl } from './signed-url';
|
|
33
34
|
import { DEFAULT_MAX_UPLOAD_BYTES } from './upload';
|
|
34
35
|
|
|
35
36
|
const DRIVER_NAME = 'local';
|
|
@@ -130,7 +131,11 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
130
131
|
const root = options.root.replace(/\/+$/, '');
|
|
131
132
|
const maxPutBytes = options.maxPutBytes ?? DEFAULT_MAX_UPLOAD_BYTES;
|
|
132
133
|
const clock = options.clock ?? systemClock;
|
|
133
|
-
|
|
134
|
+
// The segment is the disk's REGISTERED name, learned from `defineStorage` at boot — the driver
|
|
135
|
+
// kind is not a mount point, and minting under it made every disk not literally named `local`
|
|
136
|
+
// 404 its own URLs. An explicit `baseUrl` outranks the registration: that is the operator
|
|
137
|
+
// stating where the route is mounted, and inference must not overwrite a decision.
|
|
138
|
+
let baseUrl = options.baseUrl ?? signedUrlBaseFor(DRIVER_NAME);
|
|
134
139
|
// A dev disk must work with zero config. Outside development the fallback is refused rather
|
|
135
140
|
// than used: the literal is published, so signing with it hands every reader the power to mint
|
|
136
141
|
// a PUT for any key with any size and type limit — which `acceptSignedUpload` then trusts over
|
|
@@ -205,6 +210,15 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
205
210
|
return {
|
|
206
211
|
name: DRIVER_NAME,
|
|
207
212
|
|
|
213
|
+
/** A getter, not a captured string: `registerAs` runs after the driver is constructed. */
|
|
214
|
+
get signedUrlBase(): string {
|
|
215
|
+
return baseUrl;
|
|
216
|
+
},
|
|
217
|
+
|
|
218
|
+
registerAs(diskName: string): void {
|
|
219
|
+
if (options.baseUrl === undefined) baseUrl = signedUrlBaseFor(diskName);
|
|
220
|
+
},
|
|
221
|
+
|
|
208
222
|
async put(key: string, body: StorageBody, putOptions?: PutOptions): Promise<StorageObject> {
|
|
209
223
|
const safe = assertSafeKey(key);
|
|
210
224
|
refuseUnsupportedPut(putOptions);
|
|
@@ -297,13 +311,27 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
297
311
|
|
|
298
312
|
async list(listOptions?: ListOptions): Promise<ListPage> {
|
|
299
313
|
const prefix = listOptions?.prefix ?? '';
|
|
300
|
-
const limit = listOptions?.limit
|
|
314
|
+
const limit = resolveListLimit(listOptions?.limit);
|
|
301
315
|
const cursor = listOptions?.cursor;
|
|
302
316
|
const keys: string[] = [];
|
|
303
317
|
try {
|
|
304
|
-
|
|
318
|
+
// `dot: true`, and it is load-bearing: without it a glob matches no dot-prefixed entry, so
|
|
319
|
+
// every object whose key has one — `.hidden.txt`, `org/o1/pending/.x.png`, the
|
|
320
|
+
// `.metadata/a.json` `path.test.ts` pins as legal — was absent from the listing while
|
|
321
|
+
// `put`/`get`/`exists` handled it normally and the s3 listing returned it. `sweepOrphans`
|
|
322
|
+
// pages through `list()`, so those objects were swept as if they did not exist: a false
|
|
323
|
+
// erasure report by omission, which is what the classification below exists to prevent.
|
|
324
|
+
for await (const entry of new Bun.Glob('**/*').scan({
|
|
325
|
+
cwd: root,
|
|
326
|
+
onlyFiles: true,
|
|
327
|
+
dot: true,
|
|
328
|
+
})) {
|
|
305
329
|
const key = entry.replaceAll('\\', '/');
|
|
306
|
-
|
|
330
|
+
// The real filter now, not a second line of defence: the glob above yields the sidecar
|
|
331
|
+
// tree, and this is the one thing keeping `.meta/<key>.json` out of the object
|
|
332
|
+
// namespace. Folded like `assertSafeKey`'s reservation — `.META/` and `.meta/` are one
|
|
333
|
+
// directory on APFS and NTFS, and listing a key `get()` would refuse is the worse half.
|
|
334
|
+
if (key.toLowerCase().startsWith(`${META_DIR}/`)) continue;
|
|
307
335
|
if (!key.startsWith(prefix)) continue;
|
|
308
336
|
// The cursor IS the last key of the previous page — lexicographic order keeps it stable.
|
|
309
337
|
if (cursor !== undefined && key <= cursor) continue;
|
|
@@ -332,6 +360,9 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
332
360
|
if (object !== undefined) objects.push(object);
|
|
333
361
|
}
|
|
334
362
|
const truncated = keys.length > page.length;
|
|
363
|
+
// `limit` is a positive integer, so a truncated page always HAS a last key: the guard is the
|
|
364
|
+
// type's and not a second condition. It used to be one, and `limit: 0` fell through it —
|
|
365
|
+
// an empty page reported as complete over a disk that was not.
|
|
335
366
|
const last = page.at(-1);
|
|
336
367
|
return truncated && last !== undefined
|
|
337
368
|
? { objects, truncated, cursor: last }
|
|
@@ -350,5 +381,27 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
350
381
|
clock,
|
|
351
382
|
});
|
|
352
383
|
},
|
|
384
|
+
|
|
385
|
+
/**
|
|
386
|
+
* The mint above, run backwards, on the same three values from the same closure: `secret`,
|
|
387
|
+
* `baseUrl` and `clock`. That is the whole reason it belongs here rather than in a caller — a
|
|
388
|
+
* route holding this disk can now accept an upload without ever being handed the key, and
|
|
389
|
+
* before this member existed there was no way for one to verify at all, which is why the
|
|
390
|
+
* shipped `PUT` half of `/_storage` was never mounted.
|
|
391
|
+
*
|
|
392
|
+
* Never throws: `verifySignedUrl` returns a reason, and `accept.ts` owns which reason becomes
|
|
393
|
+
* which error. A driver that decided that here would be a second error taxonomy.
|
|
394
|
+
*/
|
|
395
|
+
async verifySigned(input: {
|
|
396
|
+
readonly url: string;
|
|
397
|
+
readonly clock?: Clock | undefined;
|
|
398
|
+
}): Promise<SignedUrlVerification> {
|
|
399
|
+
return verifySignedUrl({
|
|
400
|
+
url: input.url,
|
|
401
|
+
secret,
|
|
402
|
+
baseUrl,
|
|
403
|
+
clock: input.clock ?? clock,
|
|
404
|
+
});
|
|
405
|
+
},
|
|
353
406
|
};
|
|
354
407
|
}
|
package/src/driver-s3.ts
CHANGED
|
@@ -6,10 +6,10 @@
|
|
|
6
6
|
import { ConfigInvalidError, EnvMissingError, stringField } from '@ultimat3/core';
|
|
7
7
|
import {
|
|
8
8
|
DEFAULT_CONTENT_TYPE,
|
|
9
|
-
DEFAULT_LIST_LIMIT,
|
|
10
9
|
type ListOptions,
|
|
11
10
|
type ListPage,
|
|
12
11
|
type PutOptions,
|
|
12
|
+
resolveListLimit,
|
|
13
13
|
type SignedUrlOptions,
|
|
14
14
|
type StorageBody,
|
|
15
15
|
type StorageDriver,
|
|
@@ -306,13 +306,16 @@ export function s3Driver(options: S3DriverOptions): StorageDriver {
|
|
|
306
306
|
|
|
307
307
|
async list(listOptions?: ListOptions): Promise<ListPage> {
|
|
308
308
|
const prefix = listOptions?.prefix ?? '';
|
|
309
|
+
// Refused at the seam both drivers share, before the provider is asked: `maxKeys: 0` used to
|
|
310
|
+
// go straight through, while the local disk answered a complete-looking empty page.
|
|
311
|
+
const maxKeys = resolveListLimit(listOptions?.limit);
|
|
309
312
|
// `conn()` OUTSIDE the try: a missing credential or an absent `Bun.S3Client` is this disk
|
|
310
313
|
// misconfigured, and it already answers with its own code and its own fix.
|
|
311
314
|
const client = conn();
|
|
312
315
|
let result: S3ListResultLike;
|
|
313
316
|
try {
|
|
314
317
|
result = await client.list({
|
|
315
|
-
maxKeys
|
|
318
|
+
maxKeys,
|
|
316
319
|
...(listOptions?.prefix === undefined ? {} : { prefix: listOptions.prefix }),
|
|
317
320
|
...(listOptions?.cursor === undefined ? {} : { continuationToken: listOptions.cursor }),
|
|
318
321
|
});
|
package/src/driver.ts
CHANGED
|
@@ -3,7 +3,10 @@
|
|
|
3
3
|
// strings — a base64 round trip through JSON is how a "small" upload becomes a 33%-larger OOM —
|
|
4
4
|
// and never unbounded, which is the same failure with the sender choosing the size.
|
|
5
5
|
|
|
6
|
+
import type { Clock } from '@ultimat3/core';
|
|
7
|
+
import { assert } from '@ultimat3/core';
|
|
6
8
|
import { putTooLarge } from './errors';
|
|
9
|
+
import type { SignedUrlVerification } from './signed-url';
|
|
7
10
|
|
|
8
11
|
export type StorageBody = Uint8Array | ReadableStream<Uint8Array> | Blob;
|
|
9
12
|
|
|
@@ -97,6 +100,20 @@ export interface SignedUrlOptions {
|
|
|
97
100
|
export interface StorageDriver {
|
|
98
101
|
/** Disk-independent driver name (`local`, `s3`) — appears in every error cause. */
|
|
99
102
|
readonly name: string;
|
|
103
|
+
/**
|
|
104
|
+
* The path prefix THIS disk's own signed URLs hang off, read by `accept.ts` so the minting half
|
|
105
|
+
* and the verifying half cannot state it twice. Undefined on a driver whose URLs are the
|
|
106
|
+
* provider's (`s3`), which this package never verifies.
|
|
107
|
+
*/
|
|
108
|
+
readonly signedUrlBase?: string | undefined;
|
|
109
|
+
/**
|
|
110
|
+
* `defineStorage` tells the driver the key it was registered under, at boot, once. A driver that
|
|
111
|
+
* mints its own URLs hangs them off THAT, never off `name`: the mounted `/_storage/:disk/*key`
|
|
112
|
+
* route resolves the segment through the registry, so a disk registered as `uploads` whose URLs
|
|
113
|
+
* say `local` 404s every signature it just wrote. Optional — a driver with no URLs of its own
|
|
114
|
+
* (`s3`) needs none, and a third-party driver that ignores it keeps minting under `name`.
|
|
115
|
+
*/
|
|
116
|
+
registerAs?(diskName: string): void;
|
|
100
117
|
put(key: string, body: StorageBody, options?: PutOptions): Promise<StorageObject>;
|
|
101
118
|
get(key: string): Promise<StorageRead>;
|
|
102
119
|
/** Bytes without buffering — the only safe path for anything over a few MB. */
|
|
@@ -116,11 +133,54 @@ export interface StorageDriver {
|
|
|
116
133
|
exists(key: string): Promise<boolean>;
|
|
117
134
|
list(options?: ListOptions): Promise<ListPage>;
|
|
118
135
|
signedUrl(key: string, options?: SignedUrlOptions): Promise<string>;
|
|
136
|
+
/**
|
|
137
|
+
* Verify a URL THIS disk minted — without surrendering the key it minted with.
|
|
138
|
+
*
|
|
139
|
+
* The other half of `signedUrlBase`, and it was missing for the whole life of the package.
|
|
140
|
+
* `signedUrlBase` is on this interface so the minting half and the verifying half cannot state
|
|
141
|
+
* the base twice; the SECRET is the same pair and stayed closed over inside `localDriver`, so
|
|
142
|
+
* `acceptSignedUpload` — whose `secret:` was required — was reachable from no route that held
|
|
143
|
+
* only a `Storage` registry. That is why the shipped upload path has no mounted `PUT`.
|
|
144
|
+
*
|
|
145
|
+
* Verification and not the secret, deliberately. A member that handed the key back would let any
|
|
146
|
+
* holder of a driver MINT a URL for any key, and would put it in every `JSON.stringify(disk)` a
|
|
147
|
+
* log or an error payload performs. A driver can answer "did I sign this?" without that.
|
|
148
|
+
*
|
|
149
|
+
* Optional, like `registerAs`: a disk whose URLs are the provider's (`s3`) verifies at the
|
|
150
|
+
* provider and never reaches this path, and a third-party driver that implements neither is
|
|
151
|
+
* refused by name (`X_STORAGE_URL_UNVERIFIABLE`) rather than reported as a bad signature.
|
|
152
|
+
*/
|
|
153
|
+
verifySigned?(input: {
|
|
154
|
+
readonly url: string;
|
|
155
|
+
/** Overrides the driver's own, so a test's frozen clock still governs expiry. */
|
|
156
|
+
readonly clock?: Clock | undefined;
|
|
157
|
+
}): Promise<SignedUrlVerification>;
|
|
119
158
|
}
|
|
120
159
|
|
|
121
160
|
export const DEFAULT_CONTENT_TYPE = 'application/octet-stream';
|
|
122
161
|
export const DEFAULT_LIST_LIMIT = 1000;
|
|
123
162
|
|
|
163
|
+
/**
|
|
164
|
+
* The page size a `list()` honours — refused rather than clamped when it cannot be one.
|
|
165
|
+
*
|
|
166
|
+
* At the `ListOptions` seam and not inside a driver, because the two answered a non-positive
|
|
167
|
+
* limit differently: the local disk sliced `[0, 0)` and then dropped its own `truncated` flag, so
|
|
168
|
+
* `list({ limit: 0 })` over a full disk reported a COMPLETE empty listing, while the s3 disk
|
|
169
|
+
* handed `maxKeys: 0` straight to the provider. `sweepOrphans` pages through `list()`, so a page
|
|
170
|
+
* that is empty and claims to be complete is a false erasure report — the same lie a swallowed
|
|
171
|
+
* listing error tells, one call to the left. A fraction is refused for `chunk()`'s reason: a
|
|
172
|
+
* `slice` truncates it, so the pages silently stop being the size that was asked for.
|
|
173
|
+
*/
|
|
174
|
+
export function resolveListLimit(limit: number | undefined): number {
|
|
175
|
+
if (limit === undefined) return DEFAULT_LIST_LIMIT;
|
|
176
|
+
assert(
|
|
177
|
+
Number.isSafeInteger(limit) && limit > 0,
|
|
178
|
+
`a list limit must be a positive integer, got ${String(limit)}: a page of zero objects cannot be told apart from a disk that has none`,
|
|
179
|
+
`pass a positive limit — disk.list({ limit: ${DEFAULT_LIST_LIMIT} }), or omit it and take DEFAULT_LIST_LIMIT`,
|
|
180
|
+
);
|
|
181
|
+
return limit;
|
|
182
|
+
}
|
|
183
|
+
|
|
124
184
|
/** What `toBytes` needs to refuse a body: the ceiling, and the key to name in the refusal. */
|
|
125
185
|
export interface ByteLimit {
|
|
126
186
|
readonly driver: string;
|
package/src/errors.ts
CHANGED
|
@@ -14,6 +14,7 @@ export const STORAGE_OWNED_ERROR_CODES = [
|
|
|
14
14
|
'X_STORAGE_CHECKSUM_MISMATCH',
|
|
15
15
|
'X_STORAGE_URL_INVALID',
|
|
16
16
|
'X_STORAGE_URL_EXPIRED',
|
|
17
|
+
'X_STORAGE_URL_UNVERIFIABLE',
|
|
17
18
|
'X_STORAGE_ORG_MISMATCH',
|
|
18
19
|
'X_STORAGE_UPLOAD_FAILED',
|
|
19
20
|
'X_STORAGE_DELETE_FAILED',
|
|
@@ -48,6 +49,7 @@ export const STORAGE_ERROR_TITLES: Readonly<Record<StorageOwnedErrorCode, string
|
|
|
48
49
|
X_STORAGE_CHECKSUM_MISMATCH: 'bytes do not match the declared checksum',
|
|
49
50
|
X_STORAGE_URL_INVALID: 'signed URL does not match its signature',
|
|
50
51
|
X_STORAGE_URL_EXPIRED: 'signed URL is past its expiry',
|
|
52
|
+
X_STORAGE_URL_UNVERIFIABLE: 'no way to verify a signed URL for this disk',
|
|
51
53
|
X_STORAGE_ORG_MISMATCH: 'object key belongs to another org',
|
|
52
54
|
X_STORAGE_UPLOAD_FAILED: 'the signed upload was refused',
|
|
53
55
|
X_STORAGE_DELETE_FAILED: 'the object could not be deleted',
|
|
@@ -300,6 +302,21 @@ export const uploadFailed = (path: string, status: number, detail: string): Stor
|
|
|
300
302
|
* avatar grant becomes an unlimited upload of any type. Refused at construction, not at the
|
|
301
303
|
* first `signedUrl()`: a process that cannot sign safely must not finish booting.
|
|
302
304
|
*/
|
|
305
|
+
/**
|
|
306
|
+
* Neither half of the verification seam was available: no `secret:` in the call, and a disk that
|
|
307
|
+
* cannot verify what it signed. Its own condition rather than a signature failure, because the two
|
|
308
|
+
* need opposite edits — a mismatch is a forged or stale URL and this is a route that was never
|
|
309
|
+
* handed a way to check one, and reporting it as `X_STORAGE_URL_INVALID` sent an operator looking
|
|
310
|
+
* for an attacker.
|
|
311
|
+
*/
|
|
312
|
+
export const signedUrlUnverifiable = (diskName: string): StorageError =>
|
|
313
|
+
new StorageError({
|
|
314
|
+
code: 'X_STORAGE_URL_UNVERIFIABLE',
|
|
315
|
+
cause: `the "${diskName}" disk was asked to verify a signed URL with no secret in the call, and its driver implements no verifySigned()`,
|
|
316
|
+
fix: 'pass secret: to the call, or use a driver that signs its own URLs (localDriver) — a provider-signed disk (s3) verifies at the provider and never reaches this path',
|
|
317
|
+
meta: { disk: diskName },
|
|
318
|
+
});
|
|
319
|
+
|
|
303
320
|
export const signingSecretMissing = (environment: string): StorageError =>
|
|
304
321
|
new StorageError({
|
|
305
322
|
code: 'X_ENV_MISSING',
|
package/src/index.ts
CHANGED
|
@@ -46,6 +46,9 @@ export {
|
|
|
46
46
|
DEFAULT_CONTENT_TYPE,
|
|
47
47
|
DEFAULT_LIST_LIMIT,
|
|
48
48
|
etagOf,
|
|
49
|
+
// Exported for the same reason `toBytes` is: a driver written outside this package has to
|
|
50
|
+
// refuse a `limit` the same way both shipped ones do, or it is a third answer to one question.
|
|
51
|
+
resolveListLimit,
|
|
49
52
|
sha256Base64,
|
|
50
53
|
toBytes,
|
|
51
54
|
} from './driver';
|
|
@@ -84,6 +87,7 @@ export {
|
|
|
84
87
|
StorageError,
|
|
85
88
|
signedUrlExpired,
|
|
86
89
|
signedUrlRejected,
|
|
90
|
+
signedUrlUnverifiable,
|
|
87
91
|
signingSecretMissing,
|
|
88
92
|
storageNotImplemented,
|
|
89
93
|
tooLarge,
|
package/src/path.ts
CHANGED
|
@@ -40,7 +40,14 @@ function unsafeReason(key: string): string | undefined {
|
|
|
40
40
|
if (key.startsWith('/')) return 'is absolute (leading "/")';
|
|
41
41
|
if (ENCODED_SEPARATOR.test(key)) return 'contains a percent-encoded separator (%2e/%2f/%5c)';
|
|
42
42
|
const segments = key.split('/');
|
|
43
|
-
|
|
43
|
+
// Case-FOLDED, exactly as `isTenantScoped` folds and for the same filesystem: `.META/a.txt.json`
|
|
44
|
+
// and `.meta/a.txt.json` are one file on APFS and NTFS, so an exact-case reservation accepted a
|
|
45
|
+
// key that writes another object's sidecar — the whole attack this reservation closes, spelled
|
|
46
|
+
// with a shift key. The whole SEGMENT is compared, never a prefix of one: `.metadata/a.json` is
|
|
47
|
+
// an ordinary key and stays one.
|
|
48
|
+
if (segments[0]?.toLowerCase() === META_DIR) {
|
|
49
|
+
return `starts with the reserved "${META_DIR}" segment`;
|
|
50
|
+
}
|
|
44
51
|
for (const segment of segments) {
|
|
45
52
|
if (segment.length === 0) return 'contains an empty segment ("//" or a trailing "/")';
|
|
46
53
|
if (segment === '.' || segment === '..') return `contains a "${segment}" segment`;
|
package/src/storage.ts
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
// Single responsibility: named disks (Laravel's model) and the one module-level accessor.
|
|
2
2
|
// Call sites name a disk, never a driver — swapping `local` for `s3` in app.config.ts must
|
|
3
|
-
// not touch a single `storage.disk('uploads').put(...)` call.
|
|
3
|
+
// not touch a single `storage.disk('uploads').put(...)` call. This file is also where a driver
|
|
4
|
+
// LEARNS its disk name (`registerAs`), because the registry is the only holder of that fact and
|
|
5
|
+
// a signed URL's `:disk` segment has to be the name the mounted route resolves.
|
|
4
6
|
|
|
5
7
|
import { ConfigInvalidError } from '@ultimat3/core';
|
|
6
8
|
import type { StorageDriver } from './driver';
|
|
@@ -40,12 +42,33 @@ export function defineStorage(config: StorageConfig): Storage {
|
|
|
40
42
|
fix: `set storage.default to one of: ${names.join(', ')} in app.config.ts`,
|
|
41
43
|
});
|
|
42
44
|
}
|
|
45
|
+
// A Map, and the SAME own-keys read `names` above is built from. `config.disks[wanted]` walked
|
|
46
|
+
// the prototype chain, so `disk('constructor')` answered with the `Object` function and the next
|
|
47
|
+
// `.put()` was a bare `TypeError` inside app code — `diskUnknown` unreachable for `constructor`,
|
|
48
|
+
// `toString`, `valueOf`, `hasOwnProperty` and `__proto__`. One function was already answering
|
|
49
|
+
// one question two ways: `default: 'constructor'` is refused above, off `Object.keys`.
|
|
50
|
+
const disks = new Map(Object.entries(config.disks));
|
|
51
|
+
// One driver instance, one disk name. A driver told two names keeps the last, and every URL it
|
|
52
|
+
// minted under the first then resolves to a disk it is not — refused here rather than 404ing
|
|
53
|
+
// one alias at a time. Two disks over one root are two `localDriver()` calls.
|
|
54
|
+
const registered = new Map<StorageDriver, string>();
|
|
55
|
+
for (const [diskName, driver] of disks) {
|
|
56
|
+
const already = registered.get(driver);
|
|
57
|
+
if (already !== undefined) {
|
|
58
|
+
throw new ConfigInvalidError({
|
|
59
|
+
cause: `storage disks "${already}" and "${diskName}" are the same driver instance, and a driver can only mint URLs under one name`,
|
|
60
|
+
fix: `give "${diskName}" its own driver in app.config.ts: disks: { ${already}: localDriver({ root: '.storage/${already}' }), ${diskName}: localDriver({ root: '.storage/${diskName}' }) }`,
|
|
61
|
+
});
|
|
62
|
+
}
|
|
63
|
+
registered.set(driver, diskName);
|
|
64
|
+
driver.registerAs?.(diskName);
|
|
65
|
+
}
|
|
43
66
|
const storageInstance: Storage = {
|
|
44
67
|
defaultDisk,
|
|
45
68
|
diskNames: Object.freeze([...names]),
|
|
46
69
|
disk(name?: string): StorageDriver {
|
|
47
70
|
const wanted = name ?? defaultDisk;
|
|
48
|
-
const driver =
|
|
71
|
+
const driver = disks.get(wanted);
|
|
49
72
|
if (driver === undefined) throw diskUnknown(wanted, names);
|
|
50
73
|
return driver;
|
|
51
74
|
},
|
package/src/upload.ts
CHANGED
|
@@ -136,16 +136,19 @@ export function sniffContentType(bytes: Uint8Array): string | undefined {
|
|
|
136
136
|
return sniffText(bytes);
|
|
137
137
|
}
|
|
138
138
|
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
139
|
+
// A `Map`, not an object literal: `base` is the transport's own `Content-Type` header by the time
|
|
140
|
+
// `acceptSignedUpload` reaches here, and `ALIASES['constructor']` on an object answers the `Object`
|
|
141
|
+
// FUNCTION through a `: string` signature — which the refusal below then rendered as its `cause`.
|
|
142
|
+
const ALIASES: ReadonlyMap<string, string> = new Map([
|
|
143
|
+
['image/jpg', 'image/jpeg'],
|
|
144
|
+
['image/x-png', 'image/png'],
|
|
145
|
+
['application/x-pdf', 'application/pdf'],
|
|
146
|
+
]);
|
|
144
147
|
|
|
145
148
|
/** Strip parameters and case: `IMAGE/PNG; charset=binary` and `image/png` are one type. */
|
|
146
149
|
export function normalizeContentType(value: string): string {
|
|
147
150
|
const base = (value.split(';')[0] ?? '').trim().toLowerCase();
|
|
148
|
-
return ALIASES
|
|
151
|
+
return ALIASES.get(base) ?? base;
|
|
149
152
|
}
|
|
150
153
|
|
|
151
154
|
const ZIP_CONTAINERS = new Set<string>([...DOCUMENT_CONTENT_TYPES, 'application/epub+zip']);
|
|
@@ -167,7 +170,11 @@ export function contentTypeMatches(declared: string, sniffed: string): boolean {
|
|
|
167
170
|
return false;
|
|
168
171
|
}
|
|
169
172
|
|
|
170
|
-
/**
|
|
173
|
+
/**
|
|
174
|
+
* Throws the first violated constraint, in this order: key, size, type, checksum. The key comes
|
|
175
|
+
* before the size because a key nothing may store makes the other three moot, and which
|
|
176
|
+
* constraint a rejected upload reports is what the client retries on — `upload.test.ts` pins it.
|
|
177
|
+
*/
|
|
171
178
|
export function validateUpload(
|
|
172
179
|
candidate: UploadCandidate,
|
|
173
180
|
policy: UploadPolicy = uploadPolicy(),
|