@ultimat3/storage 29.3.1 → 29.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +2 -2
- package/README.md +28 -0
- package/package.json +2 -2
- package/src/driver-local-read.ts +13 -0
- package/src/driver-local.ts +21 -2
- package/src/driver-memory.ts +9 -1
- package/src/driver-s3-range.ts +99 -0
- package/src/driver-s3-signed.ts +1 -1
- package/src/driver-s3.ts +25 -1
- package/src/driver.ts +12 -2
- package/src/errors-range.ts +47 -0
- package/src/errors.ts +3 -1
- package/src/index.ts +7 -0
- package/src/range.ts +71 -0
package/CLAUDE.md
CHANGED
|
@@ -39,6 +39,7 @@ Tier 1. Object storage: named disks, safe keys, signed URLs, sniffed uploads.
|
|
|
39
39
|
| File | Owns |
|
|
40
40
|
|---|---|
|
|
41
41
|
| `driver.ts` | `StorageDriver` contract (9 methods; `registerAs`/`verifySigned`/`retentionOf` optional), `assertPutOptions`, + bounded `toBytes`/`sha256Base64`/`etagOf` |
|
|
42
|
+
| `range.ts` | `get(key, { range })`: screen + `servedRange` (`errors-range.ts`); s3's one-GET read is `driver-s3-range.ts` |
|
|
42
43
|
| `driver-local.ts` | dev default over `Bun.file`, `.meta/` sidecars, `Bun.Glob` listing |
|
|
43
44
|
| `driver-local-write.ts` | the local WRITE: staged under `.meta/.tmp/`, renamed marker → sidecar → bytes, `X_STORAGE_KEY_CONFLICT` / `X_STORAGE_PUT_FAILED`, the per-key queue |
|
|
44
45
|
| `driver-local-read.ts` | `headObject`: sidecar + pending marker → trusted, re-checked, or untyped |
|
|
@@ -274,8 +275,7 @@ Gotchas:
|
|
|
274
275
|
default header, and `decodeError` keeps a refusal `X_STORAGE_UPLOAD_FAILED` with the disk's
|
|
275
276
|
status. A request with no response at all is the transport's `X_CLIENT_TRANSPORT_FAILED` — it was a
|
|
276
277
|
bare `TypeError` before. This file calls no `fetch`; the XHR above is the one exemption.
|
|
277
|
-
- **Bundle** (`
|
|
278
|
-
**17,767 B** on `clientTransport`; the +5.9 kB over 20.2.1 is core's transport graph.
|
|
278
|
+
- **Bundle** (`uploadFile` only, browser, minified, `As of 2026-09-22`): **17,767 B**.
|
|
279
279
|
- `exactOptionalPropertyTypes` is on — declare optional fields as `x?: T | undefined`.
|
|
280
280
|
|
|
281
281
|
Why each rule above is shaped the way it is: [`docs/history/storage.md`](../../docs/history/storage.md).
|
package/README.md
CHANGED
|
@@ -194,6 +194,33 @@ the disk's own size (a HEAD on s3) before a byte is read — and is read with `s
|
|
|
194
194
|
because an object's size is not the server's to choose: a presigned PUT lands in a bucket
|
|
195
195
|
unmeasured, so a later `get()` of it was heap growth the uploader picked.
|
|
196
196
|
|
|
197
|
+
## Ranged reads — `get(key, { range: [first, last] })`
|
|
198
|
+
|
|
199
|
+
```ts
|
|
200
|
+
import { disk } from '@ultimat3/storage';
|
|
201
|
+
|
|
202
|
+
const page = await disk('debug').get('runs/r1/http.ndjson', { range: [0, 65_535] });
|
|
203
|
+
page.bytes; // at most 65 536 bytes
|
|
204
|
+
page.range; // [0, 65_535] — or [0, size - 1] when the object is shorter
|
|
205
|
+
page.object.size; // the WHOLE object's size, for `Content-Range: bytes 0-65535/<size>`
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
| Rule | |
|
|
209
|
+
|---|---|
|
|
210
|
+
| Offsets | `first` and `last` are BOTH inclusive — HTTP's `bytes=first-last`, S3's `Range:` |
|
|
211
|
+
| On the wire | s3: ONE signed GET with `Range:` and no HEAD — the bytes, the window and total size (`Content-Range`), the `ETag`, type and date all come from that one response, so a write racing the read (a shrink, a grow, an A → B → A of one length) can never pair one version's bytes with another's description. Asks at most `maxGetBytes`; the provider's 416 is `X_STORAGE_RANGE_UNSATISFIABLE` (its size from a HEAD). local: `Bun.file().slice()` in the key's queue; memory: a slice of its copy |
|
|
212
|
+
| Clamp | a `last` past the end is clamped to `size - 1`, as HTTP clamps it; `read.range` says what was served |
|
|
213
|
+
| Refusal | a `first` at or past the end — an empty object has none — is `X_STORAGE_RANGE_UNSATISFIABLE` (416); its `fix:` is the whole-object range, or the plain `get(key)` for an empty one |
|
|
214
|
+
| Screen | a backwards, negative, fractional or malformed tuple is the caller's defect: core's `X_INVARIANT` |
|
|
215
|
+
| Ceiling | `maxGetBytes` bounds the RANGE, not the object: a 40MB log is refused whole and readable 1MB at a time |
|
|
216
|
+
| Absence | `read.range` is present exactly when a range was asked — a third-party driver that ignored the option answers none |
|
|
217
|
+
|
|
218
|
+
**A SEALED object cannot be range-read meaningfully.** `seal()` (`@ultimat3/core`) writes one
|
|
219
|
+
`x1` envelope — one AES-256-GCM ciphertext, base64url, whose tag authenticates every byte — so a
|
|
220
|
+
slice of it neither opens nor means anything, and there is no framed or chunked form of the
|
|
221
|
+
envelope to slice along (`As of 2026-10`). Range-read plain objects. To page a sealed log, seal
|
|
222
|
+
each page as its OWN object (`runs/<id>/http/0000`, `…/0001`) and read the page you need whole.
|
|
223
|
+
|
|
197
224
|
## Write-once objects (S3 Object Lock)
|
|
198
225
|
|
|
199
226
|
```ts
|
|
@@ -425,6 +452,7 @@ Inside `pending/` deliberately: an upload nobody ever scanned is still an orphan
|
|
|
425
452
|
| `X_STORAGE_PUT_FAILED` | the disk REFUSED a `put()`/`copy()` — `EACCES`, `ENOSPC`, `EROFS`, a provider's refused PUT |
|
|
426
453
|
| `X_STORAGE_OBJECT_LOCKED` | local, memory: a delete, overwrite or copy onto a key under retention or a legal hold; the cause names the key, the mode and `retainUntil` or the hold, and the bytes are unchanged. Never raised by s3, where the provider keeps the locked version |
|
|
427
454
|
| `X_STORAGE_READ_FAILED` | the disk REFUSED a `get`/`stat`/`exists`/`stream` or the read half of a `copy` — a denied `s3:GetObject`, a throttle, an unreadable file. An **absent** object is still `X_STORAGE_NOT_FOUND`, including one deleted between the existence check and the read |
|
|
455
|
+
| `X_STORAGE_RANGE_UNSATISFIABLE` | `get(key, { range })` whose first byte is at or past the object's size — the 416 |
|
|
428
456
|
| `X_STORAGE_KEY_CONFLICT` | local and memory disks (never s3): the key's path is another key's directory, or the reverse (`a` and `a/b`) |
|
|
429
457
|
| `X_NOT_IMPLEMENTED` | `serverSideEncryption` on any driver |
|
|
430
458
|
| `X_ENV_MISSING` | core's: S3 credential env vars, or a `localDriver` built outside development where neither `signingSecret` nor `STORAGE_SIGNING_SECRET` holds a secret other than the published `DEV_SIGNING_SECRET` |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/storage",
|
|
3
|
-
"version": "29.
|
|
3
|
+
"version": "29.4.0",
|
|
4
4
|
"description": "Named disks over Bun.file and Bun.s3: safe keys, signed URLs, sniffed uploads",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -32,6 +32,6 @@
|
|
|
32
32
|
"test": "bun test"
|
|
33
33
|
},
|
|
34
34
|
"dependencies": {
|
|
35
|
-
"@ultimat3/core": "29.
|
|
35
|
+
"@ultimat3/core": "29.4.0"
|
|
36
36
|
}
|
|
37
37
|
}
|
package/src/driver-local-read.ts
CHANGED
|
@@ -61,6 +61,19 @@ export async function readObjectBytes(
|
|
|
61
61
|
.then((buffer) => new Uint8Array(buffer), refusedRead(root, key, disk));
|
|
62
62
|
}
|
|
63
63
|
|
|
64
|
+
/** `readObjectBytes` for the inclusive window `[first, last]` — `Bun.file().slice()`, one read. */
|
|
65
|
+
export async function readObjectRange(
|
|
66
|
+
root: string,
|
|
67
|
+
key: string,
|
|
68
|
+
disk: string,
|
|
69
|
+
[first, last]: readonly [number, number],
|
|
70
|
+
): Promise<Uint8Array> {
|
|
71
|
+
return Bun.file(`${root}/${key}`)
|
|
72
|
+
.slice(first, last + 1)
|
|
73
|
+
.arrayBuffer()
|
|
74
|
+
.then((buffer) => new Uint8Array(buffer), refusedRead(root, key, disk));
|
|
75
|
+
}
|
|
76
|
+
|
|
64
77
|
/** `etagOf`, streamed: a copy or a `stat()` must settle a doubt without buffering the object. */
|
|
65
78
|
export async function etagOfFile(root: string, key: string, disk = 'local'): Promise<string> {
|
|
66
79
|
const hasher = new Bun.CryptoHasher('sha256');
|
package/src/driver-local.ts
CHANGED
|
@@ -31,7 +31,13 @@ import {
|
|
|
31
31
|
sha256Base64,
|
|
32
32
|
toBytes,
|
|
33
33
|
} from './driver';
|
|
34
|
-
import {
|
|
34
|
+
import {
|
|
35
|
+
etagOfFile,
|
|
36
|
+
headObject,
|
|
37
|
+
readObjectBytes,
|
|
38
|
+
readObjectRange,
|
|
39
|
+
readSidecar,
|
|
40
|
+
} from './driver-local-read';
|
|
35
41
|
import { lockOfSidecar, type Sidecar, sidecarLock } from './driver-local-sidecar';
|
|
36
42
|
import {
|
|
37
43
|
commitObject,
|
|
@@ -50,6 +56,7 @@ import {
|
|
|
50
56
|
requestedLock,
|
|
51
57
|
} from './object-lock';
|
|
52
58
|
import { assertSafeKey, META_DIR } from './path';
|
|
59
|
+
import { assertGetOptions, type GetOptions, servedRange } from './range';
|
|
53
60
|
import type { SignedUrlVerification } from './signed-url';
|
|
54
61
|
import { buildSignedUrl, signedUrlBaseFor, verifySignedUrl } from './signed-url';
|
|
55
62
|
import { resolveSigningSecret } from './signing-secret';
|
|
@@ -256,13 +263,25 @@ export function localDriver(options: LocalDriverOptions): StorageDriver {
|
|
|
256
263
|
return entry && { ...entry, contentType: entry.contentType ?? DEFAULT_CONTENT_TYPE };
|
|
257
264
|
},
|
|
258
265
|
|
|
259
|
-
async get(key: string): Promise<StorageRead> {
|
|
266
|
+
async get(key: string, getOptions?: GetOptions): Promise<StorageRead> {
|
|
260
267
|
const safe = assertSafeKey(key);
|
|
268
|
+
assertGetOptions(DRIVER_NAME, getOptions);
|
|
269
|
+
const range = getOptions?.range;
|
|
261
270
|
// Queued behind this key's writers: an object is two files, and a read between a put()'s
|
|
262
271
|
// renames is a pair in doubt this process has no need to meet.
|
|
263
272
|
return oneAtATime(safe, async () => {
|
|
264
273
|
const file = Bun.file(filePath(safe));
|
|
265
274
|
if (!(await file.exists())) throw objectNotFound(registered, safe);
|
|
275
|
+
if (range !== undefined) {
|
|
276
|
+
const served = servedRange(DRIVER_NAME, safe, range, file.size, maxGetBytes);
|
|
277
|
+
const bytes = await readObjectRange(root, safe, registered, served);
|
|
278
|
+
// A doubt is settled by streaming the WHOLE file through the hash, never by buffering it:
|
|
279
|
+
// the range is the only part of this object the read may hold.
|
|
280
|
+
const entry = await headObject(root, safe, () => etagOfFile(root, safe, registered));
|
|
281
|
+
if (entry === undefined) throw objectNotFound(registered, safe);
|
|
282
|
+
const object = { ...entry, contentType: entry.contentType ?? DEFAULT_CONTENT_TYPE };
|
|
283
|
+
return { object, bytes, range: served };
|
|
284
|
+
}
|
|
266
285
|
if (file.size > maxGetBytes) throw getTooLarge(DRIVER_NAME, safe, file.size, maxGetBytes);
|
|
267
286
|
const bytes = await readObjectBytes(root, safe, registered);
|
|
268
287
|
// Settled against the bytes this read ALREADY holds: hashed only for a sidecar-less object
|
package/src/driver-memory.ts
CHANGED
|
@@ -42,6 +42,7 @@ import {
|
|
|
42
42
|
requestedLock,
|
|
43
43
|
} from './object-lock';
|
|
44
44
|
import { assertSafeKey } from './path';
|
|
45
|
+
import { assertGetOptions, type GetOptions, servedRange } from './range';
|
|
45
46
|
import type { SignedUrlVerification } from './signed-url';
|
|
46
47
|
import { buildSignedUrl, signedUrlBaseFor, verifySignedUrl } from './signed-url';
|
|
47
48
|
import { resolveSigningSecret } from './signing-secret';
|
|
@@ -190,9 +191,16 @@ export function memoryStorageDriver(options: MemoryStorageDriverOptions = {}): M
|
|
|
190
191
|
return hit && snapshot(hit.object);
|
|
191
192
|
},
|
|
192
193
|
|
|
193
|
-
async get(key: string): Promise<StorageRead> {
|
|
194
|
+
async get(key: string, getOptions?: GetOptions): Promise<StorageRead> {
|
|
194
195
|
const safe = assertSafeKey(key);
|
|
196
|
+
assertGetOptions(DRIVER_NAME, getOptions);
|
|
195
197
|
const hit = found(safe);
|
|
198
|
+
const range = getOptions?.range;
|
|
199
|
+
if (range !== undefined) {
|
|
200
|
+
const served = servedRange(DRIVER_NAME, safe, range, hit.bytes.byteLength, maxGetBytes);
|
|
201
|
+
const bytes = hit.bytes.slice(served[0], served[1] + 1);
|
|
202
|
+
return { object: snapshot(hit.object), bytes, range: served };
|
|
203
|
+
}
|
|
196
204
|
// The local disk's refusal, so a suite on this disk meets the ceiling production has.
|
|
197
205
|
if (hit.bytes.byteLength > maxGetBytes) {
|
|
198
206
|
throw getTooLarge(DRIVER_NAME, safe, hit.bytes.byteLength, maxGetBytes);
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
// Single responsibility: the s3 disk's ranged read — ONE signed GET whose `Range:` answer carries
|
|
2
|
+
// the bytes AND what they are (`Content-Range`'s window and total size, `ETag`, type, date), so the
|
|
3
|
+
// slice and its description come from one response about one version. A HEAD beside it could not
|
|
4
|
+
// promise that: a write between the two, or an A → B → A overwrite of one length, slips past both.
|
|
5
|
+
|
|
6
|
+
import type { StorageObject, StorageRead } from './driver';
|
|
7
|
+
import { DEFAULT_CONTENT_TYPE } from './driver';
|
|
8
|
+
import { type S3Refusal, type S3WireOptions, sendSigned } from './driver-s3-signed';
|
|
9
|
+
import { rangeTooLarge, rangeUnsatisfiable } from './errors-range';
|
|
10
|
+
import type { ByteRange } from './range';
|
|
11
|
+
|
|
12
|
+
/** What the driver lends this read: its coded refusals and its own HEAD for a 416's size. */
|
|
13
|
+
export interface S3RangeDeps {
|
|
14
|
+
readonly driver: string;
|
|
15
|
+
readonly maxGetBytes: number;
|
|
16
|
+
/** A refusal the provider gave (403, 503, a wrong region, a 404) as the disk's coded error. */
|
|
17
|
+
readonly refused: (key: string, refusal: S3Refusal) => never;
|
|
18
|
+
/** The disk's HEAD — asked only after a 416, for the size the refusal names. */
|
|
19
|
+
readonly sizeOf: (key: string) => Promise<number>;
|
|
20
|
+
/** A response the read cannot use: no `Content-Range`, or one that is not the window asked. */
|
|
21
|
+
readonly unreadable: (key: string, why: string) => never;
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
const CONTENT_RANGE = /^bytes (\d+)-(\d+)\/(\d+)$/;
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* The window asked, CAPPED at `maxGetBytes` before it is sent: the ceiling bounds the range, and the
|
|
28
|
+
* object's size is not known until the answer — so at most the ceiling crosses the wire, and an
|
|
29
|
+
* answer that shows the object went on past the cap is refused before its body is read.
|
|
30
|
+
*/
|
|
31
|
+
const errorMessage = (thrown: unknown): string =>
|
|
32
|
+
thrown instanceof Error ? thrown.message : String(thrown);
|
|
33
|
+
|
|
34
|
+
export async function readS3Range(
|
|
35
|
+
wire: S3WireOptions,
|
|
36
|
+
key: string,
|
|
37
|
+
range: ByteRange,
|
|
38
|
+
deps: S3RangeDeps,
|
|
39
|
+
): Promise<StorageRead> {
|
|
40
|
+
const [first, last] = range;
|
|
41
|
+
const capped = Math.min(last, first + deps.maxGetBytes - 1);
|
|
42
|
+
const result = await sendSigned(wire, {
|
|
43
|
+
method: 'GET',
|
|
44
|
+
key,
|
|
45
|
+
headers: { range: `bytes=${first}-${capped}` },
|
|
46
|
+
});
|
|
47
|
+
if (!result.ok) {
|
|
48
|
+
if (result.refusal.statusCode !== 416) return deps.refused(key, result.refusal);
|
|
49
|
+
// A first byte at or past the end — of the object as it is NOW, which the HEAD names.
|
|
50
|
+
throw rangeUnsatisfiable(deps.driver, key, range, await deps.sizeOf(key));
|
|
51
|
+
}
|
|
52
|
+
const { response } = result;
|
|
53
|
+
const window = CONTENT_RANGE.exec(response.headers.get('content-range') ?? '');
|
|
54
|
+
if (response.status !== 206 || window === null) {
|
|
55
|
+
await response.body?.cancel().catch(() => undefined);
|
|
56
|
+
return deps.unreadable(key, `the ranged GET answered ${response.status} with no byte range`);
|
|
57
|
+
}
|
|
58
|
+
const served: ByteRange = [Number(window[1]), Number(window[2])];
|
|
59
|
+
const size = Number(window[3]);
|
|
60
|
+
if (served[0] !== first || served[1] > capped) {
|
|
61
|
+
await response.body?.cancel().catch(() => undefined);
|
|
62
|
+
return deps.unreadable(
|
|
63
|
+
key,
|
|
64
|
+
`asked bytes ${first}-${capped}, answered ${served[0]}-${served[1]}`,
|
|
65
|
+
);
|
|
66
|
+
}
|
|
67
|
+
// Cut by the CAP, not by the end of the object: the range asked is over the ceiling.
|
|
68
|
+
if (last > capped && size - 1 > capped) {
|
|
69
|
+
await response.body?.cancel().catch(() => undefined);
|
|
70
|
+
throw rangeTooLarge(deps.driver, key, [first, Math.min(last, size - 1)], deps.maxGetBytes);
|
|
71
|
+
}
|
|
72
|
+
let bytes: Uint8Array;
|
|
73
|
+
try {
|
|
74
|
+
bytes = new Uint8Array(await response.arrayBuffer());
|
|
75
|
+
} catch (thrown) {
|
|
76
|
+
// The headers came, the body broke (a reset connection): the disk's coded read failure.
|
|
77
|
+
return deps.unreadable(key, `the ranged body broke mid-read: ${errorMessage(thrown)}`);
|
|
78
|
+
}
|
|
79
|
+
if (bytes.byteLength !== served[1] - served[0] + 1) {
|
|
80
|
+
return deps.unreadable(
|
|
81
|
+
key,
|
|
82
|
+
`Content-Range promised ${served[1] - served[0] + 1}B, the body held ${bytes.byteLength}B`,
|
|
83
|
+
);
|
|
84
|
+
}
|
|
85
|
+
return { object: objectOf(key, size, response.headers), bytes, range: served };
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** The WHOLE object, as the same response describes it — never a second request's view. */
|
|
89
|
+
function objectOf(key: string, size: number, headers: Headers): StorageObject {
|
|
90
|
+
const modified = new Date(headers.get('last-modified') ?? '');
|
|
91
|
+
return {
|
|
92
|
+
key,
|
|
93
|
+
size,
|
|
94
|
+
contentType: headers.get('content-type') ?? DEFAULT_CONTENT_TYPE,
|
|
95
|
+
etag: headers.get('etag') ?? '',
|
|
96
|
+
// Omitted when the provider sent none, never epoch 0 — `StorageListEntry.lastModified`.
|
|
97
|
+
...(Number.isFinite(modified.getTime()) ? { lastModified: modified } : {}),
|
|
98
|
+
};
|
|
99
|
+
}
|
package/src/driver-s3-signed.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
// Single responsibility: the s3 disk's SIGNED requests — the ones `Bun.S3Client` has no option for
|
|
2
|
-
// (`x-amz-meta-*`, `Cache-Control`,
|
|
2
|
+
// (`x-amz-meta-*`, `Cache-Control`, object lock, a ranged GET read off its own `Content-Range`) — built,
|
|
3
3
|
// signed through core's one SigV4 signer and sent with `fetch`. The endpoint, the addressing style,
|
|
4
4
|
// the region and the credentials are read from the SAME `S3DriverOptions` the Bun client is built
|
|
5
5
|
// from, so the two transports of one disk cannot point at two buckets.
|
package/src/driver-s3.ts
CHANGED
|
@@ -34,6 +34,7 @@ import {
|
|
|
34
34
|
import { isAbsentObject } from './driver-s3-absent';
|
|
35
35
|
import { buildClient, type S3ClientLike, type S3ListResultLike } from './driver-s3-client';
|
|
36
36
|
import { needsSignedPut, readObjectLock, signedPut } from './driver-s3-lock';
|
|
37
|
+
import { readS3Range } from './driver-s3-range';
|
|
37
38
|
import { regionMismatch } from './driver-s3-region';
|
|
38
39
|
import type { S3FetchLike } from './driver-s3-signed';
|
|
39
40
|
import {
|
|
@@ -48,6 +49,7 @@ import {
|
|
|
48
49
|
} from './errors';
|
|
49
50
|
import { assertObjectLockOptions, type ObjectLock } from './object-lock';
|
|
50
51
|
import { assertSafeKey } from './path';
|
|
52
|
+
import { assertGetOptions, type GetOptions } from './range';
|
|
51
53
|
import { DEFAULT_SIGNED_URL_TTL_MS } from './signed-url';
|
|
52
54
|
import { DEFAULT_MAX_UPLOAD_BYTES } from './upload';
|
|
53
55
|
|
|
@@ -261,8 +263,30 @@ export function s3Driver(options: S3DriverOptions): StorageDriver {
|
|
|
261
263
|
return (await present(safe)) ? statObject(safe) : undefined;
|
|
262
264
|
},
|
|
263
265
|
|
|
264
|
-
async get(key: string): Promise<StorageRead> {
|
|
266
|
+
async get(key: string, getOptions?: GetOptions): Promise<StorageRead> {
|
|
265
267
|
const safe = assertSafeKey(key);
|
|
268
|
+
assertGetOptions(DRIVER_NAME, getOptions);
|
|
269
|
+
const range = getOptions?.range;
|
|
270
|
+
// ONE signed GET, bytes and description from the same response (`driver-s3-range.ts`).
|
|
271
|
+
if (range !== undefined) {
|
|
272
|
+
return readS3Range(options, safe, range, {
|
|
273
|
+
driver: DRIVER_NAME,
|
|
274
|
+
maxGetBytes,
|
|
275
|
+
refused: (key, refusal) => refusedRead(key)(refusal),
|
|
276
|
+
sizeOf: async (key) => {
|
|
277
|
+
if (!(await present(key))) throw objectNotFound(diskName, key);
|
|
278
|
+
return (await statObject(key)).size;
|
|
279
|
+
},
|
|
280
|
+
unreadable: (key, why) => {
|
|
281
|
+
throw readFailed(
|
|
282
|
+
DRIVER_NAME,
|
|
283
|
+
key,
|
|
284
|
+
why,
|
|
285
|
+
`aws s3api get-object --bucket ${renderFixShellArg(options.bucket, "'<bucket>'")} --key ${renderFixShellArg(key, "'<key>'")} --range bytes=0-0 /dev/null # what the provider answers a ranged GET`,
|
|
286
|
+
);
|
|
287
|
+
},
|
|
288
|
+
});
|
|
289
|
+
}
|
|
266
290
|
const file = conn().file(safe);
|
|
267
291
|
if (!(await present(safe))) throw objectNotFound(diskName, safe);
|
|
268
292
|
// The HEAD comes FIRST, and it is the whole point: a client PUT straight into the bucket is
|
package/src/driver.ts
CHANGED
|
@@ -7,6 +7,7 @@ import type { Clock, Page } from '@ultimat3/core';
|
|
|
7
7
|
import { assert } from '@ultimat3/core';
|
|
8
8
|
import { putTooLarge } from './errors';
|
|
9
9
|
import type { ObjectLock, ObjectLockOptions } from './object-lock';
|
|
10
|
+
import type { ByteRange, GetOptions } from './range';
|
|
10
11
|
import type { SignedUrlVerification } from './signed-url';
|
|
11
12
|
|
|
12
13
|
export type StorageBody = Uint8Array | ReadableStream<Uint8Array> | Blob;
|
|
@@ -81,8 +82,15 @@ export interface ListOptions {
|
|
|
81
82
|
}
|
|
82
83
|
|
|
83
84
|
export interface StorageRead {
|
|
85
|
+
/** The WHOLE object's description — `size` is the object's, never the range's. */
|
|
84
86
|
readonly object: StorageObject;
|
|
85
87
|
readonly bytes: Uint8Array;
|
|
88
|
+
/**
|
|
89
|
+
* The bytes served, both inclusive, after the clamp — present exactly when `range` was asked.
|
|
90
|
+
* A third-party driver that ignored the option answers none, and that absence is how a caller
|
|
91
|
+
* tells a slice from the whole object rather than trusting `bytes.byteLength`.
|
|
92
|
+
*/
|
|
93
|
+
readonly range?: ByteRange | undefined;
|
|
86
94
|
}
|
|
87
95
|
|
|
88
96
|
export type SignedUrlMethod = 'GET' | 'PUT';
|
|
@@ -134,9 +142,11 @@ export interface StorageDriver {
|
|
|
134
142
|
stat(key: string): Promise<StorageObject | undefined>;
|
|
135
143
|
/**
|
|
136
144
|
* The whole object, BUFFERED — so bounded: an object over the driver's `maxGetBytes` is
|
|
137
|
-
* `X_STORAGE_TOO_LARGE`, exactly as `put()` refuses a body over `maxPutBytes`. Past it, `stream()
|
|
145
|
+
* `X_STORAGE_TOO_LARGE`, exactly as `put()` refuses a body over `maxPutBytes`. Past it, `stream()`
|
|
146
|
+
* — or `{ range: [first, last] }`, which reads only those bytes (inclusive) and is bounded by
|
|
147
|
+
* the range's length: s3 sends `Range:`, the local disk slices the file, memory slices its copy.
|
|
138
148
|
*/
|
|
139
|
-
get(key: string): Promise<StorageRead>;
|
|
149
|
+
get(key: string, options?: GetOptions): Promise<StorageRead>;
|
|
140
150
|
/** Bytes without buffering — the only safe path for anything over a few MB. */
|
|
141
151
|
stream(key: string): Promise<ReadableStream<Uint8Array>>;
|
|
142
152
|
/**
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
// The two refusals of a ranged `get(key, { range })`: a first byte past the end (HTTP's 416) and a
|
|
2
|
+
// window over the disk's get ceiling. Their codes are registered with the rest of the package's in
|
|
3
|
+
// `errors.ts`; each `fix:` is the call to make next, with the numbers this object allows.
|
|
4
|
+
|
|
5
|
+
import { renderFixLiteral } from '@ultimat3/core';
|
|
6
|
+
import { StorageError } from './errors';
|
|
7
|
+
|
|
8
|
+
const keyArg = (key: string): string => renderFixLiteral(key, "'<key>'");
|
|
9
|
+
|
|
10
|
+
/**
|
|
11
|
+
* A first byte at or past the end. The last byte is CLAMPED (`range.ts`), so only the first can
|
|
12
|
+
* miss; an empty object has no first byte at all, so its fix is the plain read.
|
|
13
|
+
*/
|
|
14
|
+
export const rangeUnsatisfiable = (
|
|
15
|
+
disk: string,
|
|
16
|
+
key: string,
|
|
17
|
+
range: readonly [number, number],
|
|
18
|
+
size: number,
|
|
19
|
+
): StorageError =>
|
|
20
|
+
new StorageError({
|
|
21
|
+
code: 'X_STORAGE_RANGE_UNSATISFIABLE',
|
|
22
|
+
cause: `get() asked "${key}" for bytes ${range[0]}-${range[1]}, and the object on the ${disk} disk is ${size}B — its first byte is past the end`,
|
|
23
|
+
fix:
|
|
24
|
+
size === 0
|
|
25
|
+
? `await disk.get(${keyArg(key)}) # the object is empty: no range has a first byte`
|
|
26
|
+
: `await disk.get(${keyArg(key)}, { range: [0, ${size - 1}] }) # the whole ${size}B object; any first byte below ${size}`,
|
|
27
|
+
meta: { disk, key, first: range[0], last: range[1], size },
|
|
28
|
+
});
|
|
29
|
+
|
|
30
|
+
/**
|
|
31
|
+
* A window over `maxGetBytes` — the ceiling bounds the RANGE, so the repair is a smaller window
|
|
32
|
+
* from the same first byte, or a higher ceiling on the disk.
|
|
33
|
+
*/
|
|
34
|
+
export const rangeTooLarge = (
|
|
35
|
+
disk: string,
|
|
36
|
+
key: string,
|
|
37
|
+
range: readonly [number, number],
|
|
38
|
+
maxBytes: number,
|
|
39
|
+
): StorageError => {
|
|
40
|
+
const bytes = range[1] - range[0] + 1;
|
|
41
|
+
return new StorageError({
|
|
42
|
+
code: 'X_STORAGE_TOO_LARGE',
|
|
43
|
+
cause: `the range asked of "${key}" is ${bytes}B against the ${disk} disk's get ceiling of ${maxBytes}B, and get() buffers the range`,
|
|
44
|
+
fix: `await disk.get(${keyArg(key)}, { range: [${range[0]}, ${range[0] + maxBytes - 1}] }) # one window within the ceiling; or raise maxGetBytes to ${bytes} on the disk`,
|
|
45
|
+
meta: { disk, key, bytes, maxBytes, first: range[0], last: range[1] },
|
|
46
|
+
});
|
|
47
|
+
};
|
package/src/errors.ts
CHANGED
|
@@ -32,6 +32,7 @@ export const STORAGE_OWNED_ERROR_CODES = [
|
|
|
32
32
|
'X_STORAGE_OBJECT_LOCKED',
|
|
33
33
|
'X_STORAGE_KEY_UNSHARED',
|
|
34
34
|
'X_STORAGE_ALREADY_PROMOTED',
|
|
35
|
+
'X_STORAGE_RANGE_UNSATISFIABLE',
|
|
35
36
|
] as const;
|
|
36
37
|
|
|
37
38
|
/**
|
|
@@ -74,6 +75,7 @@ export const STORAGE_ERROR_TITLES: Readonly<Record<StorageOwnedErrorCode, string
|
|
|
74
75
|
X_STORAGE_OBJECT_LOCKED: 'the object is under retention or a legal hold',
|
|
75
76
|
X_STORAGE_KEY_UNSHARED: 'object key is outside every tenant and every shared prefix',
|
|
76
77
|
X_STORAGE_ALREADY_PROMOTED: 'the upload was already promoted onto its row',
|
|
78
|
+
X_STORAGE_RANGE_UNSATISFIABLE: 'the requested byte range starts past the end of the object',
|
|
77
79
|
};
|
|
78
80
|
|
|
79
81
|
// One unconditional call, so a second package claiming one of storage's codes throws
|
|
@@ -193,7 +195,7 @@ export const getTooLarge = (
|
|
|
193
195
|
code: 'X_STORAGE_TOO_LARGE',
|
|
194
196
|
cause: `"${key}" is ${bytes}B against the ${disk} disk's get ceiling of ${maxBytes}B, and get() buffers the whole object`,
|
|
195
197
|
fix:
|
|
196
|
-
`
|
|
198
|
+
`await disk.stream(${renderFixLiteral(key, "'<key>'")}) # unbuffered; or a window with get(key, { range }), or raise the ceiling: ` +
|
|
197
199
|
(disk === 's3'
|
|
198
200
|
? `s3Driver({ bucket, maxGetBytes: ${bytes} })`
|
|
199
201
|
: disk === 'memory'
|
package/src/index.ts
CHANGED
|
@@ -97,6 +97,7 @@ export {
|
|
|
97
97
|
xmlBodyUnreadable,
|
|
98
98
|
} from './errors';
|
|
99
99
|
export { alreadyPromoted, quarantined } from './errors-promote';
|
|
100
|
+
export { rangeTooLarge, rangeUnsatisfiable } from './errors-range';
|
|
100
101
|
export type { GrantUploadInput, UploadGrant, UploadRequest } from './grant';
|
|
101
102
|
export { grantUpload } from './grant';
|
|
102
103
|
export type {
|
|
@@ -144,6 +145,12 @@ export {
|
|
|
144
145
|
orgPrefix,
|
|
145
146
|
scopedKey,
|
|
146
147
|
} from './path';
|
|
148
|
+
/**
|
|
149
|
+
* `get(key, { range })`. A third-party driver screens and clamps through these two, or it is a
|
|
150
|
+
* fourth answer to what a range means.
|
|
151
|
+
*/
|
|
152
|
+
export type { ByteRange, GetOptions } from './range';
|
|
153
|
+
export { assertGetOptions, servedRange } from './range';
|
|
147
154
|
export type {
|
|
148
155
|
SignedUrlConstraints,
|
|
149
156
|
SignedUrlFailure,
|
package/src/range.ts
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
// Single responsibility: what `get(key, { range })` may ask for and what it is answered with — the
|
|
2
|
+
// screen of the tuple for every disk, and for memory and local the clamp, the 416 and the ceiling
|
|
3
|
+
// (which bounds the BYTES ASKED FOR, not the object). s3 learns the size from its one GET's answer
|
|
4
|
+
// and applies the same three rules there (`driver-s3-range.ts`), with the same refusals.
|
|
5
|
+
|
|
6
|
+
import { assert } from '@ultimat3/core';
|
|
7
|
+
import { rangeTooLarge, rangeUnsatisfiable } from './errors-range';
|
|
8
|
+
|
|
9
|
+
/**
|
|
10
|
+
* The first and the last byte, BOTH INCLUSIVE — HTTP's `Range: bytes=first-last` and S3's, so a
|
|
11
|
+
* route copies a parsed header straight in and a `Content-Range` is read straight back out.
|
|
12
|
+
* `[0, 99]` is the first hundred bytes; `[0, 0]` is one.
|
|
13
|
+
*/
|
|
14
|
+
export type ByteRange = readonly [first: number, last: number];
|
|
15
|
+
|
|
16
|
+
export interface GetOptions {
|
|
17
|
+
/**
|
|
18
|
+
* Read only these bytes. The last byte is clamped to the object (`[0, 1e12]` on a 10-byte object
|
|
19
|
+
* answers `[0, 9]`), as HTTP clamps it; a first byte at or past the end is
|
|
20
|
+
* `X_STORAGE_RANGE_UNSATISFIABLE`, which is a 416. `maxGetBytes` bounds the RANGE'S length, so a
|
|
21
|
+
* page of a large object is readable where `get()` of the whole would be refused.
|
|
22
|
+
*
|
|
23
|
+
* Meaningless on a SEALED object: `seal()`'s `x1` envelope is one AES-GCM ciphertext whose tag
|
|
24
|
+
* authenticates every byte, so a slice of it neither opens nor means anything. Range-read plain
|
|
25
|
+
* objects; page a sealed one by sealing each page as its own object.
|
|
26
|
+
*/
|
|
27
|
+
readonly range?: ByteRange | undefined;
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
const isOffset = (value: unknown): value is number =>
|
|
31
|
+
typeof value === 'number' && Number.isSafeInteger(value) && value >= 0;
|
|
32
|
+
|
|
33
|
+
/**
|
|
34
|
+
* The tuple's SHAPE, before the disk is asked anything: two whole, non-negative offsets in order.
|
|
35
|
+
* Core's `assert` (`X_INVARIANT`), because a backwards or fractional range is the calling code's
|
|
36
|
+
* defect — no object makes it valid — while a first byte past the end depends on the object and
|
|
37
|
+
* is the coded, caller-facing refusal below.
|
|
38
|
+
*/
|
|
39
|
+
export function assertGetOptions(driver: string, options: GetOptions | undefined): void {
|
|
40
|
+
const range = options?.range;
|
|
41
|
+
if (range === undefined) return;
|
|
42
|
+
const [first, last] = Array.isArray(range) ? range : [];
|
|
43
|
+
assert(
|
|
44
|
+
Array.isArray(range) && range.length === 2 && isOffset(first) && isOffset(last),
|
|
45
|
+
`get() on the ${driver} disk was given a range that is not two whole, non-negative byte offsets`,
|
|
46
|
+
'get(key, { range: [first, last] }) — inclusive offsets: [0, 99] reads the first 100 bytes',
|
|
47
|
+
);
|
|
48
|
+
assert(
|
|
49
|
+
first <= last,
|
|
50
|
+
`get() on the ${driver} disk was given the range [${first}, ${last}], whose first byte is past its last`,
|
|
51
|
+
`get(key, { range: [${last}, ${first}] }) — first before last, both inclusive`,
|
|
52
|
+
);
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* The range a disk SERVES for an object of `size` bytes — clamped, refused, or over the ceiling.
|
|
57
|
+
* Called with the object's own measured size, before a byte is read, by every driver.
|
|
58
|
+
*/
|
|
59
|
+
export function servedRange(
|
|
60
|
+
disk: string,
|
|
61
|
+
key: string,
|
|
62
|
+
range: ByteRange,
|
|
63
|
+
size: number,
|
|
64
|
+
maxGetBytes: number,
|
|
65
|
+
): ByteRange {
|
|
66
|
+
const [first, wanted] = range;
|
|
67
|
+
if (first >= size) throw rangeUnsatisfiable(disk, key, range, size);
|
|
68
|
+
const last = Math.min(wanted, size - 1);
|
|
69
|
+
if (last - first + 1 > maxGetBytes) throw rangeTooLarge(disk, key, [first, last], maxGetBytes);
|
|
70
|
+
return [first, last];
|
|
71
|
+
}
|