@amalgm/live 0.2.58 → 0.2.60
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PURPOSE.md +29 -14
- package/dist/transfer/cargo.d.ts +11 -0
- package/dist/transfer/cargo.d.ts.map +1 -1
- package/dist/transfer/cargo.js +97 -23
- package/dist/transfer/cargo.js.map +1 -1
- package/dist/transfer/download.d.ts +65 -25
- package/dist/transfer/download.d.ts.map +1 -1
- package/dist/transfer/download.js +339 -84
- package/dist/transfer/download.js.map +1 -1
- package/dist/transfer/index.d.ts +1 -0
- package/dist/transfer/index.d.ts.map +1 -1
- package/dist/transfer/index.js +1 -0
- package/dist/transfer/index.js.map +1 -1
- package/dist/transfer/ranges.d.ts +20 -35
- package/dist/transfer/ranges.d.ts.map +1 -1
- package/dist/transfer/ranges.js +35 -121
- package/dist/transfer/ranges.js.map +1 -1
- package/dist/transfer/refs.d.ts +15 -14
- package/dist/transfer/refs.d.ts.map +1 -1
- package/dist/transfer/refs.js +33 -37
- package/dist/transfer/refs.js.map +1 -1
- package/dist/transfer/stream.d.ts +50 -0
- package/dist/transfer/stream.d.ts.map +1 -0
- package/dist/transfer/stream.js +78 -0
- package/dist/transfer/stream.js.map +1 -0
- package/dist-cjs/transfer/cargo.js +98 -23
- package/dist-cjs/transfer/cargo.js.map +1 -1
- package/dist-cjs/transfer/download.js +339 -85
- package/dist-cjs/transfer/download.js.map +1 -1
- package/dist-cjs/transfer/index.js +1 -0
- package/dist-cjs/transfer/index.js.map +1 -1
- package/dist-cjs/transfer/ranges.js +37 -124
- package/dist-cjs/transfer/ranges.js.map +1 -1
- package/dist-cjs/transfer/refs.js +34 -38
- package/dist-cjs/transfer/refs.js.map +1 -1
- package/dist-cjs/transfer/stream.js +81 -0
- package/dist-cjs/transfer/stream.js.map +1 -0
- package/package.json +1 -1
package/PURPOSE.md
CHANGED
|
@@ -186,26 +186,30 @@ its cutover.
|
|
|
186
186
|
neither act may weaken the other's result.
|
|
187
187
|
30. **Proof travels forward; disposable bytes need not.** Registration reads
|
|
188
188
|
every ordinary text or binary file through one bounded stream to classify
|
|
189
|
-
it
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
can recapture current ground; stale bytes can never be sealed or
|
|
197
|
-
published.
|
|
189
|
+
it and derive its content head. When cloud cargo is missing, that same
|
|
190
|
+
capture stream carries the verified bytes forward: bounded complete small
|
|
191
|
+
artifacts enter the cargo stream, while large-file blocks become durable
|
|
192
|
+
under their own digests as they are read and the authority seals the
|
|
193
|
+
manifest only after every named block exists. A graph-only Register
|
|
194
|
+
capture never copies or rereads ground merely to preserve bytes for a
|
|
195
|
+
later Upload stage. A typed source mismatch returns control to Register so
|
|
196
|
+
it can recapture current ground; stale bytes can never be sealed or
|
|
197
|
+
published. A final missing-cargo inventory remains the publication
|
|
198
|
+
guarantor and may reread ground only when fused transport was unavailable
|
|
199
|
+
or refused. During Add, ordinary text and binary bytes land directly in
|
|
198
200
|
their hidden staged destinations in manifest order while set-wide fetches
|
|
199
201
|
remain bounded and concurrent, so their whole-artifact proof advances from
|
|
200
202
|
arriving bytes without rereading staging; links and repository transports
|
|
201
203
|
keep their dedicated adapter/cache rails. No file type or size permits a
|
|
202
204
|
complete-file allocation in the bounded transfer path.
|
|
203
205
|
31. **A content block is not a round trip.** Text and binary retain independent
|
|
204
|
-
content-addressed block identities. Cold upload
|
|
205
|
-
artifacts in bounded cargo
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
206
|
+
content-addressed block identities. Cold upload moves complete small
|
|
207
|
+
artifacts in bounded cargo segments and adjacent missing large-file blocks
|
|
208
|
+
in bounded binary batches. Cold download plans whole waves against
|
|
209
|
+
placement: all missing manifests, then all missing blocks, each wave
|
|
210
|
+
grouped by pack so adjacent members merge into bounded range reads.
|
|
211
|
+
Packing may reduce transport work but can never weaken per-block
|
|
212
|
+
verification, durable sealing, or commit order.
|
|
209
213
|
32. **A content layout owns a cloud namespace.** A writer change that derives
|
|
210
214
|
different block identities for the same complete bytes advances the
|
|
211
215
|
content contract and storage namespace together. Readers remain additive;
|
|
@@ -294,6 +298,17 @@ its cutover.
|
|
|
294
298
|
current pointer atomically. Retries never reactivate an older id, prior
|
|
295
299
|
deployments remain addressable, and SQLite rows, WAL pages, locks, and
|
|
296
300
|
offsets remain machine-local materializations rather than sync payloads.
|
|
301
|
+
44. **Sealed answers correctness; packed answers placement.** `sealed` proves
|
|
302
|
+
an artifact's bytes are durably valid under the current content contract,
|
|
303
|
+
and record submission trusts it unconditionally, whatever the placement.
|
|
304
|
+
`packed` proves the artifact is stored efficiently inside an immutable
|
|
305
|
+
pack — and one manifest location row proves its complete packed closure,
|
|
306
|
+
because the authority records every block location and the manifest
|
|
307
|
+
location before sealing. Upload skips a small artifact only when it is
|
|
308
|
+
both sealed and packed; a large artifact requires only sealed. Repacking
|
|
309
|
+
is placement work, never correctness work: placement rows converge
|
|
310
|
+
first-wins, a source missing at repack time is skipped silently, and
|
|
311
|
+
what readers can open never changes.
|
|
297
312
|
|
|
298
313
|
## Five surfaces, one behavior
|
|
299
314
|
|
package/dist/transfer/cargo.d.ts
CHANGED
|
@@ -52,6 +52,17 @@ export interface CargoStreamReceipt {
|
|
|
52
52
|
}
|
|
53
53
|
/** Pack items in order into segments within the byte and artifact bounds. */
|
|
54
54
|
export declare function planCargoSegments<T extends CargoItem>(items: readonly T[]): readonly CargoSegment<T>[];
|
|
55
|
+
/** A cargo stream whose items arrive while a walk is still discovering
|
|
56
|
+
* them. Full segments ship as they close; `add` applies backpressure when
|
|
57
|
+
* the shared window is full, and `flush` ships the final partial segment
|
|
58
|
+
* and settles every acknowledgement. After any rejection, `flush` still
|
|
59
|
+
* settles in-flight segments before rethrowing the first failure, so their
|
|
60
|
+
* sealed work stays durable for the resuming rerun. */
|
|
61
|
+
export interface CargoStream<T extends CargoItem> {
|
|
62
|
+
add(item: T): Promise<void>;
|
|
63
|
+
flush(): Promise<CargoStreamReceipt>;
|
|
64
|
+
}
|
|
65
|
+
export declare function openCargoStream<T extends CargoItem>(ports: CargoStreamPorts<T>, options?: CargoStreamOptions): CargoStream<T>;
|
|
55
66
|
/** Stream one cold set of small artifacts as bounded pipelined segments.
|
|
56
67
|
* Every item must come back sealed or the stream fails; a failure stops new
|
|
57
68
|
* admission, lets in-flight segments settle, and leaves their sealed work
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cargo.d.ts","sourceRoot":"","sources":["../../src/transfer/cargo.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AAKH,eAAO,MAAM,mBAAmB,QAA4B,CAAC;AAC7D;;;uDAGuD;AACvD,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAC3C;;;2EAG2E;AAC3E,eAAO,MAAM,4BAA4B,QAA0B,CAAC;AAEpE,MAAM,WAAW,SAAS;IACxB,gEAAgE;IAChE,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,iDAAiD;IACjD,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,YAAY,CAAC,CAAC,SAAS,SAAS;IAC/C,QAAQ,CAAC,KAAK,EAAE,SAAS,CAAC,EAAE,CAAC;IAC7B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,gBAAgB,CAAC,CAAC,SAAS,SAAS;IACnD;;;uEAGmE;IACnE,IAAI,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,SAAS,MAAM,EAAE,CAAC,CAAC;CAC5D;AAED,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AAED,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;
|
|
1
|
+
{"version":3,"file":"cargo.d.ts","sourceRoot":"","sources":["../../src/transfer/cargo.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AAKH,eAAO,MAAM,mBAAmB,QAA4B,CAAC;AAC7D;;;uDAGuD;AACvD,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAC3C;;;2EAG2E;AAC3E,eAAO,MAAM,4BAA4B,QAA0B,CAAC;AAEpE,MAAM,WAAW,SAAS;IACxB,gEAAgE;IAChE,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,iDAAiD;IACjD,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,YAAY,CAAC,CAAC,SAAS,SAAS;IAC/C,QAAQ,CAAC,KAAK,EAAE,SAAS,CAAC,EAAE,CAAC;IAC7B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAED,MAAM,WAAW,gBAAgB,CAAC,CAAC,SAAS,SAAS;IACnD;;;uEAGmE;IACnE,IAAI,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,SAAS,MAAM,EAAE,CAAC,CAAC;CAC5D;AAED,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AAED,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;CACxB;AAcD,6EAA6E;AAC7E,wBAAgB,iBAAiB,CAAC,CAAC,SAAS,SAAS,EACnD,KAAK,EAAE,SAAS,CAAC,EAAE,GAClB,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,CAiB5B;AAED;;;;;uDAKuD;AACvD,MAAM,WAAW,WAAW,CAAC,CAAC,SAAS,SAAS;IAC9C,GAAG,CAAC,IAAI,EAAE,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5B,KAAK,IAAI,OAAO,CAAC,kBAAkB,CAAC,CAAC;CACtC;AAED,wBAAgB,eAAe,CAAC,CAAC,SAAS,SAAS,EACjD,KAAK,EAAE,gBAAgB,CAAC,CAAC,CAAC,EAC1B,OAAO,GAAE,kBAAuB,GAC/B,WAAW,CAAC,CAAC,CAAC,CAuEhB;AAED;;;qCAGqC;AACrC,wBAAsB,WAAW,CAAC,CAAC,SAAS,SAAS,EACnD,KAAK,EAAE,SAAS,CAAC,EAAE,EACnB,KAAK,EAAE,gBAAgB,CAAC,CAAC,CAAC,EAC1B,OAAO,GAAE,kBAAuB,GAC/B,OAAO,CAAC,kBAAkB,CAAC,CAS7B"}
|
package/dist/transfer/cargo.js
CHANGED
|
@@ -26,21 +26,24 @@ export const CARGO_SEGMENT_ARTIFACTS = 256;
|
|
|
26
26
|
* fan-out, and worst-case payload residency (plaintext plus one encoded
|
|
27
27
|
* copy per in-flight segment) stays comfortably inside a 1-GiB runtime. */
|
|
28
28
|
export const CARGO_STREAM_IN_FLIGHT_BYTES = 4 * CARGO_SEGMENT_BYTES;
|
|
29
|
+
const assertCargoItem = (item) => {
|
|
30
|
+
if (typeof item.contentHash !== 'string' || item.contentHash.length === 0) {
|
|
31
|
+
throw new Error('cargo item is missing its content identity');
|
|
32
|
+
}
|
|
33
|
+
if (!Number.isSafeInteger(item.bytes) || item.bytes < 0) {
|
|
34
|
+
throw new Error('cargo item byte count is invalid');
|
|
35
|
+
}
|
|
36
|
+
if (item.bytes > CARGO_SEGMENT_BYTES) {
|
|
37
|
+
throw new Error('cargo item exceeds the segment byte bound');
|
|
38
|
+
}
|
|
39
|
+
};
|
|
29
40
|
/** Pack items in order into segments within the byte and artifact bounds. */
|
|
30
41
|
export function planCargoSegments(items) {
|
|
31
42
|
const segments = [];
|
|
32
43
|
let current = [];
|
|
33
44
|
let bytes = 0;
|
|
34
45
|
for (const item of items) {
|
|
35
|
-
|
|
36
|
-
throw new Error('cargo item is missing its content identity');
|
|
37
|
-
}
|
|
38
|
-
if (!Number.isSafeInteger(item.bytes) || item.bytes < 0) {
|
|
39
|
-
throw new Error('cargo item byte count is invalid');
|
|
40
|
-
}
|
|
41
|
-
if (item.bytes > CARGO_SEGMENT_BYTES) {
|
|
42
|
-
throw new Error('cargo item exceeds the segment byte bound');
|
|
43
|
-
}
|
|
46
|
+
assertCargoItem(item);
|
|
44
47
|
if (current.length > 0
|
|
45
48
|
&& (bytes + item.bytes > CARGO_SEGMENT_BYTES || current.length >= CARGO_SEGMENT_ARTIFACTS)) {
|
|
46
49
|
segments.push({ items: current, bytes });
|
|
@@ -54,24 +57,95 @@ export function planCargoSegments(items) {
|
|
|
54
57
|
segments.push({ items: current, bytes });
|
|
55
58
|
return segments;
|
|
56
59
|
}
|
|
60
|
+
export function openCargoStream(ports, options = {}) {
|
|
61
|
+
const lanes = options.lanes ?? DEFAULT_TRANSFER_CONCURRENCY;
|
|
62
|
+
const maxInFlightBytes = options.maxInFlightBytes ?? CARGO_STREAM_IN_FLIGHT_BYTES;
|
|
63
|
+
const inFlight = new Set();
|
|
64
|
+
let inFlightBytes = 0;
|
|
65
|
+
let failure;
|
|
66
|
+
let current = [];
|
|
67
|
+
let currentBytes = 0;
|
|
68
|
+
let segments = 0;
|
|
69
|
+
let artifacts = 0;
|
|
70
|
+
let bytes = 0;
|
|
71
|
+
const dispatch = async (segment) => {
|
|
72
|
+
const weight = Math.max(segment.bytes, 1);
|
|
73
|
+
while (failure === undefined
|
|
74
|
+
&& (inFlight.size >= lanes
|
|
75
|
+
|| (inFlight.size > 0 && inFlightBytes + weight > maxInFlightBytes))) {
|
|
76
|
+
await Promise.race(inFlight);
|
|
77
|
+
}
|
|
78
|
+
if (failure !== undefined)
|
|
79
|
+
throw failure.error;
|
|
80
|
+
inFlightBytes += weight;
|
|
81
|
+
const settled = (async () => {
|
|
82
|
+
const sealed = new Set(await ports.send(segment));
|
|
83
|
+
for (const item of segment.items) {
|
|
84
|
+
if (!sealed.has(item.contentHash)) {
|
|
85
|
+
throw new Error(`cargo authority did not seal ${item.contentHash}`);
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
segments += 1;
|
|
89
|
+
artifacts += segment.items.length;
|
|
90
|
+
bytes += segment.bytes;
|
|
91
|
+
})().then(() => { inFlight.delete(settled); inFlightBytes -= weight; }, (error) => {
|
|
92
|
+
inFlight.delete(settled);
|
|
93
|
+
inFlightBytes -= weight;
|
|
94
|
+
failure ??= { error };
|
|
95
|
+
});
|
|
96
|
+
inFlight.add(settled);
|
|
97
|
+
};
|
|
98
|
+
const closeCurrent = () => {
|
|
99
|
+
const segment = { items: current, bytes: currentBytes };
|
|
100
|
+
current = [];
|
|
101
|
+
currentBytes = 0;
|
|
102
|
+
return segment;
|
|
103
|
+
};
|
|
104
|
+
return {
|
|
105
|
+
async add(item) {
|
|
106
|
+
assertCargoItem(item);
|
|
107
|
+
if (failure !== undefined)
|
|
108
|
+
throw failure.error;
|
|
109
|
+
if (current.length > 0
|
|
110
|
+
&& (currentBytes + item.bytes > CARGO_SEGMENT_BYTES
|
|
111
|
+
|| current.length >= CARGO_SEGMENT_ARTIFACTS)) {
|
|
112
|
+
await dispatch(closeCurrent());
|
|
113
|
+
}
|
|
114
|
+
current.push(item);
|
|
115
|
+
currentBytes += item.bytes;
|
|
116
|
+
},
|
|
117
|
+
async flush() {
|
|
118
|
+
if (failure === undefined && current.length > 0) {
|
|
119
|
+
try {
|
|
120
|
+
await dispatch(closeCurrent());
|
|
121
|
+
}
|
|
122
|
+
catch {
|
|
123
|
+
// The failure is recorded; settle the window before rethrowing.
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
while (inFlight.size > 0)
|
|
127
|
+
await Promise.race(inFlight);
|
|
128
|
+
if (failure !== undefined)
|
|
129
|
+
throw failure.error;
|
|
130
|
+
return { segments, artifacts, bytes };
|
|
131
|
+
},
|
|
132
|
+
};
|
|
133
|
+
}
|
|
57
134
|
/** Stream one cold set of small artifacts as bounded pipelined segments.
|
|
58
135
|
* Every item must come back sealed or the stream fails; a failure stops new
|
|
59
136
|
* admission, lets in-flight segments settle, and leaves their sealed work
|
|
60
137
|
* durable for the resuming rerun. */
|
|
61
138
|
export async function streamCargo(items, ports, options = {}) {
|
|
62
|
-
const
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
const
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
bytes += segment.bytes;
|
|
74
|
-
});
|
|
75
|
-
return { segments: segments.length, artifacts, bytes };
|
|
139
|
+
for (const item of items)
|
|
140
|
+
assertCargoItem(item);
|
|
141
|
+
const stream = openCargoStream(ports, options);
|
|
142
|
+
try {
|
|
143
|
+
for (const item of items)
|
|
144
|
+
await stream.add(item);
|
|
145
|
+
}
|
|
146
|
+
catch {
|
|
147
|
+
// add() already recorded the failure; flush settles and rethrows it.
|
|
148
|
+
}
|
|
149
|
+
return stream.flush();
|
|
76
150
|
}
|
|
77
151
|
//# sourceMappingURL=cargo.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cargo.js","sourceRoot":"","sources":["../../src/transfer/cargo.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AAEH,OAAO,EAAE,yBAAyB,EAAE,MAAM,cAAc,CAAC;AACzD,OAAO,EAAE,4BAA4B,EAAE,uBAAuB,EAAE,MAAM,eAAe,CAAC;AAEtF,MAAM,CAAC,MAAM,mBAAmB,GAAG,yBAAyB,CAAC;AAC7D;;;uDAGuD;AACvD,MAAM,CAAC,MAAM,uBAAuB,GAAG,GAAG,CAAC;AAC3C;;;2EAG2E;AAC3E,MAAM,CAAC,MAAM,4BAA4B,GAAG,CAAC,GAAG,mBAAmB,CAAC;AAiCpE,
|
|
1
|
+
{"version":3,"file":"cargo.js","sourceRoot":"","sources":["../../src/transfer/cargo.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;GAcG;AAEH,OAAO,EAAE,yBAAyB,EAAE,MAAM,cAAc,CAAC;AACzD,OAAO,EAAE,4BAA4B,EAAE,uBAAuB,EAAE,MAAM,eAAe,CAAC;AAEtF,MAAM,CAAC,MAAM,mBAAmB,GAAG,yBAAyB,CAAC;AAC7D;;;uDAGuD;AACvD,MAAM,CAAC,MAAM,uBAAuB,GAAG,GAAG,CAAC;AAC3C;;;2EAG2E;AAC3E,MAAM,CAAC,MAAM,4BAA4B,GAAG,CAAC,GAAG,mBAAmB,CAAC;AAiCpE,MAAM,eAAe,GAAG,CAAC,IAAe,EAAQ,EAAE;IAChD,IAAI,OAAO,IAAI,CAAC,WAAW,KAAK,QAAQ,IAAI,IAAI,CAAC,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QAC1E,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC,CAAC;IAChE,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,aAAa,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,IAAI,CAAC,KAAK,GAAG,CAAC,EAAE,CAAC;QACxD,MAAM,IAAI,KAAK,CAAC,kCAAkC,CAAC,CAAC;IACtD,CAAC;IACD,IAAI,IAAI,CAAC,KAAK,GAAG,mBAAmB,EAAE,CAAC;QACrC,MAAM,IAAI,KAAK,CAAC,2CAA2C,CAAC,CAAC;IAC/D,CAAC;AACH,CAAC,CAAC;AAEF,6EAA6E;AAC7E,MAAM,UAAU,iBAAiB,CAC/B,KAAmB;IAEnB,MAAM,QAAQ,GAAsB,EAAE,CAAC;IACvC,IAAI,OAAO,GAAQ,EAAE,CAAC;IACtB,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,eAAe,CAAC,IAAI,CAAC,CAAC;QACtB,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC;eACjB,CAAC,KAAK,GAAG,IAAI,CAAC,KAAK,GAAG,mBAAmB,IAAI,OAAO,CAAC,MAAM,IAAI,uBAAuB,CAAC,EAAE,CAAC;YAC7F,QAAQ,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC;YACzC,OAAO,GAAG,EAAE,CAAC;YACb,KAAK,GAAG,CAAC,CAAC;QACZ,CAAC;QACD,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACnB,KAAK,IAAI,IAAI,CAAC,KAAK,CAAC;IACtB,CAAC;IACD,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC;QAAE,QAAQ,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC;IACjE,OAAO,QAAQ,CAAC;AAClB,CAAC;AAaD,MAAM,UAAU,eAAe,CAC7B,KAA0B,EAC1B,UAA8B,EAAE;IAEhC,MAAM,KAAK,GAAG,OAAO,CAAC,KAAK,IAAI,4BAA4B,CAAC;IAC5D,MAAM,gBAAgB,GAAG,OAAO,CAAC,gBAAgB,IAAI,4BAA4B,CAAC;IAClF,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAiB,CAAC;IAC1C,IAAI,aAAa,GAAG,CAAC,CAAC;IACtB,IAAI,OAAuC,CAAC;IAC5C,IAAI,OAAO,GAAQ,EAAE,CAAC;IACtB,IAAI,YAAY,GAAG,CAAC,CAAC;IACrB,IAAI,QAAQ,GAAG,CAAC,CAAC;IACjB,IAAI,SAAS,GAAG,CAAC,CAAC;IAClB,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,MAAM,QAAQ,GAAG,KAAK,EAAE,OAAwB,EAAiB,EAAE;QACjE,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;QAC1C,OAAO,OAAO,KAAK,SAAS;eACvB,CAAC,QAAQ,CAAC,IAAI,IAAI,KAAK;mBACrB,CAAC,QAAQ,CAAC,IAAI,GAAG,CAAC,IAAI,aAAa,GAAG,MAAM,GAAG,gBAAgB,CAAC,CAAC,EAAE,CAAC;YACzE,MAAM,OAAO,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC/B,CAAC;QACD,IAAI,OAAO,KAAK,SAAS;YAAE,MAAM,OAAO,CAAC,KAAK,CAAC;QAC/C,aAAa,IAAI,MAAM,CAAC;QACxB,MAAM,OAAO,GAAkB,CAAC,KAAK,IAAI,EAAE;YACzC,MAAM,MAAM,GAAG,IAAI,GAAG,CAAC,MAAM,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC;YAClD,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;gBACjC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,WAAW,CAAC,EAAE,CAAC;oBAClC,MAAM,IAAI,KAAK,CAAC,gCAAgC,IAAI,CAAC,WAAW,EAAE,CAAC,CAAC;gBACtE,CAAC;YACH,CAAC;YACD,QAAQ,IAAI,CAAC,CAAC;YACd,SAAS,IAAI,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC;YAClC,KAAK,IAAI,OAAO,CAAC,KAAK,CAAC;QACzB,CAAC,CAAC,EAAE,CAAC,IAAI,CACP,GAAG,EAAE,GAAG,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,aAAa,IAAI,MAAM,CAAC,CAAC,CAAC,EAC5D,CAAC,KAAc,EAAE,EAAE;YACjB,QAAQ,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;YACzB,aAAa,IAAI,MAAM,CAAC;YACxB,OAAO,KAAK,EAAE,KAAK,EAAE,CAAC;QACxB,CAAC,CACF,CAAC;QACF,QAAQ,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;IACxB,CAAC,CAAC;IACF,MAAM,YAAY,GAAG,GAAoB,EAAE;QACzC,MAAM,OAAO,GAAG,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,CAAC;QACxD,OAAO,GAAG,EAAE,CAAC;QACb,YAAY,GAAG,CAAC,CAAC;QACjB,OAAO,OAAO,CAAC;IACjB,CAAC,CAAC;IACF,OAAO;QACL,KAAK,CAAC,GAAG,CAAC,IAAI;YACZ,eAAe,CAAC,IAAI,CAAC,CAAC;YACtB,IAAI,OAAO,KAAK,SAAS;gBAAE,MAAM,OAAO,CAAC,KAAK,CAAC;YAC/C,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC;mBACjB,CAAC,YAAY,GAAG,IAAI,CAAC,KAAK,GAAG,mBAAmB;uBAC9C,OAAO,CAAC,MAAM,IAAI,uBAAuB,CAAC,EAAE,CAAC;gBAClD,MAAM,QAAQ,CAAC,YAAY,EAAE,CAAC,CAAC;YACjC,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACnB,YAAY,IAAI,IAAI,CAAC,KAAK,CAAC;QAC7B,CAAC;QACD,KAAK,CAAC,KAAK;YACT,IAAI,OAAO,KAAK,SAAS,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAChD,IAAI,CAAC;oBACH,MAAM,QAAQ,CAAC,YAAY,EAAE,CAAC,CAAC;gBACjC,CAAC;gBAAC,MAAM,CAAC;oBACP,gEAAgE;gBAClE,CAAC;YACH,CAAC;YACD,OAAO,QAAQ,CAAC,IAAI,GAAG,CAAC;gBAAE,MAAM,OAAO,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;YACvD,IAAI,OAAO,KAAK,SAAS;gBAAE,MAAM,OAAO,CAAC,KAAK,CAAC;YAC/C,OAAO,EAAE,QAAQ,EAAE,SAAS,EAAE,KAAK,EAAE,CAAC;QACxC,CAAC;KACF,CAAC;AACJ,CAAC;AAED;;;qCAGqC;AACrC,MAAM,CAAC,KAAK,UAAU,WAAW,CAC/B,KAAmB,EACnB,KAA0B,EAC1B,UAA8B,EAAE;IAEhC,KAAK,MAAM,IAAI,IAAI,KAAK;QAAE,eAAe,CAAC,IAAI,CAAC,CAAC;IAChD,MAAM,MAAM,GAAG,eAAe,CAAC,KAAK,EAAE,OAAO,CAAC,CAAC;IAC/C,IAAI,CAAC;QACH,KAAK,MAAM,IAAI,IAAI,KAAK;YAAE,MAAM,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IACnD,CAAC;IAAC,MAAM,CAAC;QACP,qEAAqE;IACvE,CAAC;IACD,OAAO,MAAM,CAAC,KAAK,EAAE,CAAC;AACxB,CAAC"}
|
|
@@ -1,37 +1,59 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Download a set of immutable artifacts into a machine cache.
|
|
3
3
|
*
|
|
4
|
-
* The set is the primitive
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
*
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
4
|
+
* The set is the primitive and the plan is physical. Wave one proves every
|
|
5
|
+
* manifest: locally held ones are checked in place, missing ones are
|
|
6
|
+
* resolved as one want-list and fetched grouped by the pack that holds
|
|
7
|
+
* them. Wave two reads the proven manifests, asks the cache which chunks
|
|
8
|
+
* are gaps, resolves the whole gap list, and fetches it the same way: all
|
|
9
|
+
* members of one pack collapse into a few broad byte spans — approximately
|
|
10
|
+
* one read per pack — while standalone objects remain one read each. A
|
|
11
|
+
* lone artifact is a set of one; there is no second download path.
|
|
12
12
|
*
|
|
13
|
-
*
|
|
13
|
+
* Spans complete in whatever order storage answers, but each artifact
|
|
14
14
|
* stores its chunks in chunk order: a verified chunk waits until every
|
|
15
|
-
* earlier gap of the same artifact
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
15
|
+
* earlier gap of the same artifact is stored, so a host assembling a
|
|
16
|
+
* whole-file digest never re-reads what landed early. The scheduler admits
|
|
17
|
+
* fetch units strictly in fair plan order, and the plan orders units so a
|
|
18
|
+
* chunk's predecessor always lives in the same or an earlier unit — an
|
|
19
|
+
* artifact whose gaps do not line up ascending inside one pack falls back
|
|
20
|
+
* to per-object reads at its own positions — so ordered stores can never
|
|
21
|
+
* deadlock. Chains enter the plan round-robin by depth, so one multi-chunk
|
|
22
|
+
* artifact cannot monopolize the bounded window while unrelated artifacts
|
|
23
|
+
* wait. In-flight span bytes stay inside one explicit bound; raising request
|
|
24
|
+
* concurrency never silently raises memory.
|
|
19
25
|
*
|
|
20
|
-
* Every
|
|
21
|
-
* envelope lives alone or as a byte range inside an immutable pack
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
26
|
+
* Every object keeps one content-addressed identity whether its sealed
|
|
27
|
+
* envelope lives alone or as a byte range inside an immutable pack, and
|
|
28
|
+
* placement is immutable: a retried read may refresh its URL but never its
|
|
29
|
+
* geometry. Wrong or shifted bytes cannot leak — every dispensed slice is
|
|
30
|
+
* still an authenticated envelope the host opens and this engine
|
|
31
|
+
* hash-verifies before a single byte is stored.
|
|
25
32
|
*/
|
|
26
33
|
import { type ContentManifest } from '../contracts/content.js';
|
|
27
34
|
import type { Sha256Hex } from '../entities/types.js';
|
|
28
35
|
import { type TransferArtifact } from './artifacts.js';
|
|
36
|
+
import { type ByteSpan } from './ranges.js';
|
|
37
|
+
import { type ContentObjectName, type ContentRef } from './refs.js';
|
|
29
38
|
export interface DownloadPorts<LocalArtifact> {
|
|
30
39
|
readonly sha256Hex: Sha256Hex;
|
|
31
|
-
|
|
40
|
+
/** A manifest the host already holds for this artifact, or null to have
|
|
41
|
+
* the set fetch it. A hint only — the answer is still proven here. */
|
|
42
|
+
localManifest(artifact: TransferArtifact): unknown;
|
|
43
|
+
/** Resolve one batch of named objects to storage locations, aligned by
|
|
44
|
+
* index. Called with at most CONTENT_REFS_BATCH_OBJECTS names. */
|
|
45
|
+
requestRefs(objects: readonly ContentObjectName[]): Promise<readonly ContentRef[]>;
|
|
46
|
+
/** Perform one storage read: a byte range of a pack, or a whole
|
|
47
|
+
* standalone object when span is null. `refresh` re-resolves the read's
|
|
48
|
+
* location for a retry attempt and throws loudly if placement moved. */
|
|
49
|
+
fetchSpan(url: string, span: ByteSpan | null, refresh: () => Promise<string>): Promise<Uint8Array>;
|
|
50
|
+
/** Open one sealed envelope into its plaintext bytes. */
|
|
51
|
+
openEnvelope(object: ContentObjectName, sealed: Uint8Array): Promise<Uint8Array>;
|
|
52
|
+
/** A fetched manifest just proved against its artifact; the host persists
|
|
53
|
+
* it wherever this artifact is being assembled. */
|
|
54
|
+
manifestProven(artifact: TransferArtifact, manifest: ContentManifest): Promise<void>;
|
|
32
55
|
/** True only when this exact content-addressed chunk is locally verified. */
|
|
33
56
|
hasChunk(artifact: TransferArtifact, manifest: ContentManifest, index: number): Promise<boolean>;
|
|
34
|
-
getChunk(artifact: TransferArtifact, manifest: ContentManifest, index: number): Promise<Uint8Array>;
|
|
35
57
|
putChunk(artifact: TransferArtifact, manifest: ContentManifest, index: number, bytes: Uint8Array): Promise<void>;
|
|
36
58
|
/** Verify and expose the complete immutable artifact assembled from cache. */
|
|
37
59
|
sealArtifact(artifact: TransferArtifact, manifest: ContentManifest): Promise<SealedArtifact<LocalArtifact>>;
|
|
@@ -43,8 +65,10 @@ export interface SealedArtifact<LocalArtifact> {
|
|
|
43
65
|
}
|
|
44
66
|
export interface DownloadOptions {
|
|
45
67
|
readonly concurrency?: number;
|
|
46
|
-
/** Maximum
|
|
47
|
-
*
|
|
68
|
+
/** Maximum bytes held by in-flight fetch units. Raising request
|
|
69
|
+
* concurrency never silently raises this memory bound. A single sealed
|
|
70
|
+
* envelope larger than the bound still fetches whole; admission then
|
|
71
|
+
* serializes around it. */
|
|
48
72
|
readonly maxInFlightBytes?: number;
|
|
49
73
|
}
|
|
50
74
|
export interface DownloadReceipt<LocalArtifact> {
|
|
@@ -54,10 +78,26 @@ export interface DownloadReceipt<LocalArtifact> {
|
|
|
54
78
|
readonly downloadedChunks: number;
|
|
55
79
|
readonly reusedChunks: number;
|
|
56
80
|
}
|
|
81
|
+
/** The physical geometry one download performed — how knowledge arrived and
|
|
82
|
+
* how many storage reads carried it. */
|
|
83
|
+
export interface DownloadShape {
|
|
84
|
+
readonly localManifests: number;
|
|
85
|
+
readonly fetchedManifests: number;
|
|
86
|
+
readonly packedManifests: number;
|
|
87
|
+
readonly standaloneManifests: number;
|
|
88
|
+
/** Ranged pack reads that carried manifests. */
|
|
89
|
+
readonly manifestSpans: number;
|
|
90
|
+
readonly packedBlocks: number;
|
|
91
|
+
readonly standaloneBlocks: number;
|
|
92
|
+
/** Ranged pack reads that carried chunk blocks. */
|
|
93
|
+
readonly blockSpans: number;
|
|
94
|
+
}
|
|
95
|
+
export interface DownloadResult<LocalArtifact> {
|
|
96
|
+
readonly receipts: readonly DownloadReceipt<LocalArtifact>[];
|
|
97
|
+
readonly shape: DownloadShape;
|
|
98
|
+
}
|
|
57
99
|
/** Download a set of immutable artifacts, fetching only machine-cache gaps.
|
|
58
100
|
* Receipts return in input order; any failure fails the whole set after
|
|
59
101
|
* letting in-flight effects settle. */
|
|
60
|
-
export declare function downloadAll<LocalArtifact>(artifacts: readonly TransferArtifact[], ports: DownloadPorts<LocalArtifact>, options?: DownloadOptions): Promise<
|
|
61
|
-
/** Download one immutable artifact — a set of one. */
|
|
62
|
-
export declare function download<LocalArtifact>(artifact: TransferArtifact, ports: DownloadPorts<LocalArtifact>, options?: DownloadOptions): Promise<DownloadReceipt<LocalArtifact>>;
|
|
102
|
+
export declare function downloadAll<LocalArtifact>(artifacts: readonly TransferArtifact[], ports: DownloadPorts<LocalArtifact>, options?: DownloadOptions): Promise<DownloadResult<LocalArtifact>>;
|
|
63
103
|
//# sourceMappingURL=download.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"download.d.ts","sourceRoot":"","sources":["../../src/transfer/download.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"download.d.ts","sourceRoot":"","sources":["../../src/transfer/download.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA+BG;AAEH,OAAO,EAAqC,KAAK,eAAe,EAAE,MAAM,yBAAyB,CAAC;AAClG,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAA0B,KAAK,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AAM/E,OAAO,EAAwC,KAAK,QAAQ,EAAE,MAAM,aAAa,CAAC;AAClF,OAAO,EAAsB,KAAK,iBAAiB,EAAE,KAAK,UAAU,EAAE,MAAM,WAAW,CAAC;AAExF,MAAM,WAAW,aAAa,CAAC,aAAa;IAC1C,QAAQ,CAAC,SAAS,EAAE,SAAS,CAAC;IAC9B;0EACsE;IACtE,aAAa,CAAC,QAAQ,EAAE,gBAAgB,GAAG,OAAO,CAAC;IACnD;sEACkE;IAClE,WAAW,CAAC,OAAO,EAAE,SAAS,iBAAiB,EAAE,GAAG,OAAO,CAAC,SAAS,UAAU,EAAE,CAAC,CAAC;IACnF;;4EAEwE;IACxE,SAAS,CACP,GAAG,EAAE,MAAM,EACX,IAAI,EAAE,QAAQ,GAAG,IAAI,EACrB,OAAO,EAAE,MAAM,OAAO,CAAC,MAAM,CAAC,GAC7B,OAAO,CAAC,UAAU,CAAC,CAAC;IACvB,yDAAyD;IACzD,YAAY,CAAC,MAAM,EAAE,iBAAiB,EAAE,MAAM,EAAE,UAAU,GAAG,OAAO,CAAC,UAAU,CAAC,CAAC;IACjF;uDACmD;IACnD,cAAc,CAAC,QAAQ,EAAE,gBAAgB,EAAE,QAAQ,EAAE,eAAe,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IACrF,6EAA6E;IAC7E,QAAQ,CACN,QAAQ,EAAE,gBAAgB,EAC1B,QAAQ,EAAE,eAAe,EACzB,KAAK,EAAE,MAAM,GACZ,OAAO,CAAC,OAAO,CAAC,CAAC;IACpB,QAAQ,CACN,QAAQ,EAAE,gBAAgB,EAC1B,QAAQ,EAAE,eAAe,EACzB,KAAK,EAAE,MAAM,EACb,KAAK,EAAE,UAAU,GAChB,OAAO,CAAC,IAAI,CAAC,CAAC;IACjB,8EAA8E;IAC9E,YAAY,CACV,QAAQ,EAAE,gBAAgB,EAC1B,QAAQ,EAAE,eAAe,GACxB,OAAO,CAAC,cAAc,CAAC,aAAa,CAAC,CAAC,CAAC;CAC3C;AAED,MAAM,WAAW,cAAc,CAAC,aAAa;IAC3C,QAAQ,CAAC,KAAK,EAAE,aAAa,CAAC;IAC9B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;CAC9B;AAED,MAAM,WAAW,eAAe;IAC9B,QAAQ,CAAC,WAAW,CAAC,EAAE,MAAM,CAAC;IAC9B;;;+BAG2B;IAC3B,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AAED,MAAM,WAAW,eAAe,CAAC,aAAa;IAC5C,QAAQ,CAAC,QAAQ,EAAE,gBAAgB,CAAC;IACpC,QAAQ,CAAC,QAAQ,EAAE,eAAe,CAAC;IACnC,QAAQ,CAAC,KAAK,EAAE,aAAa,CAAC;IAC9B,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,YAAY,EAAE,MAAM,CAAC;CAC/B;AAED;wCACwC;AACxC,MAAM,WAAW,aAAa;IAC5B,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAC;IAChC,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;IACjC,QAAQ,CAAC,mBAAmB,EAAE,MAAM,CAAC;IACrC,gDAAgD;IAChD,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAC;IAC/B,QAAQ,CAAC,YAAY,EAAE,MAAM,CAAC;IAC9B,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC,mDAAmD;IACnD,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;CAC7B;AAED,MAAM,WAAW,cAAc,CAAC,aAAa;IAC3C,QAAQ,CAAC,QAAQ,EAAE,SAAS,eAAe,CAAC,aAAa,CAAC,EAAE,CAAC;IAC7D,QAAQ,CAAC,KAAK,EAAE,aAAa,CAAC;CAC/B;AA0PD;;uCAEuC;AACvC,wBAAsB,WAAW,CAAC,aAAa,EAC7C,SAAS,EAAE,SAAS,gBAAgB,EAAE,EACtC,KAAK,EAAE,aAAa,CAAC,aAAa,CAAC,EACnC,OAAO,GAAE,eAAoB,GAC5B,OAAO,CAAC,cAAc,CAAC,aAAa,CAAC,CAAC,CAkJxC"}
|