@poe-platform/safe-fs 0.1.803 → 0.1.804
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -4
- package/dist/safe-fs/bridge/filesystem.js +1 -1
- package/dist/safe-fs/contracts/errors.js +4 -3
- package/dist/safe-fs/contracts/filesystem.d.ts +3 -2
- package/dist/safe-fs/contracts/io.d.ts +2 -0
- package/dist/safe-fs/contracts/io.js +32 -7
- package/dist/safe-fs/contracts/portable-path.d.ts +8 -0
- package/dist/safe-fs/contracts/portable-path.js +22 -1
- package/dist/safe-fs/core.d.ts +1 -0
- package/dist/safe-fs/core.js +1 -0
- package/dist/safe-fs/fs/devices/index.js +2 -1
- package/dist/safe-fs/fs/directory-admission.js +2 -2
- package/dist/safe-fs/fs/memory/index.d.ts +1 -1
- package/dist/safe-fs/fs/memory/index.js +23 -53
- package/dist/safe-fs/fs/memory/ledger.js +3 -1
- package/dist/safe-fs/fs/object-publication/index.js +2 -1
- package/dist/safe-fs/fs/real/index.js +3 -2
- package/dist/safe-fs/fs/s3/http/request-fetch.d.ts +14 -3
- package/dist/safe-fs/fs/s3/http/request-node.d.ts +2 -2
- package/dist/safe-fs/fs/s3/http/types.d.ts +2 -2
- package/dist/safe-fs/fs/s3/manifest-admission.js +0 -4
- package/dist/safe-fs/fs/s3/namespace.js +2 -4
- package/dist/safe-fs/fs/webdav/webdav.js +4 -28
- package/dist/safe-fs/fs/webdav/xml.js +2 -2
- package/dist/safe-fs/python/emscripten.js +5 -2
- package/dist/safe-fs/python/filesystem.js +2 -2
- package/dist/safe-fs/python/native.js +1 -1
- package/dist/safe-fs/python/stat.js +5 -2
- package/package.json +12 -2
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# safe-fs
|
|
2
2
|
|
|
3
|
-
An asynchronous filesystem interface for memory, rooted host directories, S3, and WebDAV, with mounts, copy-on-write overlays, and Node-style bridges. Portable filesystem types and POSIX helpers compile without Node types.
|
|
3
|
+
An asynchronous filesystem interface for memory, rooted host directories, S3, and WebDAV, with mounts, copy-on-write overlays, and Node-style bridges. Portable filesystem types and POSIX helpers compile without Node types. The `posixPath` export from `@poe-code/safe-fs/core` provides `basename`, `dirname`, `extname`, `join`, `isAbsolute`, `normalize`, `resolve`, `relative`, `sep`, and `delimiter`; `resolve` and `relative` use `/` as their working directory in every runtime.
|
|
4
4
|
|
|
5
5
|
Under `workerd` and `browser` conditions, the root export includes portable filesystems, `S3FileSystem`, and `MockS3Client`. Supply an S3 transport to use object storage without Node globals.
|
|
6
6
|
|
|
@@ -86,14 +86,26 @@ cause. Preserve cancellation/control exceptions separately. Cross-family
|
|
|
86
86
|
|
|
87
87
|
All required methods must exist, but a backend may reject an operation with `FsError`, such as `EROFS` for a write or `ENOTSUP` for unsupported semantics. Check optional methods and `capabilities` rather than assuming every backend behaves like a local disk. Errors expose `code` and may include `syscall`, `path`, `dest`, and `cause`.
|
|
88
88
|
|
|
89
|
+
`readBytes` supports frozen host abort signals. The exported
|
|
90
|
+
`listenForAbort(signal, listener)` returns a cleanup function for custom
|
|
91
|
+
operations; if frozen host bookkeeping prevents registration, callers must
|
|
92
|
+
check cancellation at operation boundaries, as `readBytes` does.
|
|
93
|
+
|
|
89
94
|
`createDeviceFileSystem(fs)` adds portable `/dev/null` whole-file, stream, and descriptor I/O. Descriptor reads return EOF and writes discard bytes; stat remains a zero-size character device and descriptor position remains zero. Truncating opens are accepted, exclusive creation fails with `EEXIST`, and descriptor resizing and synchronization are unsupported. Access modes, cancellation, and closed handles use the normal descriptor checks. With an authoritative object store, use `createDeviceFileSystem(withObjectFileDescriptors(fs, store))` so null-device I/O never acquires or publishes an object version; ordinary files retain conditional publication. The device wrapper must be outermost for this composition.
|
|
90
95
|
|
|
96
|
+
`PythonStatTranslator` and `createPythonRuntimeStatMapper` allocate stable guest device
|
|
97
|
+
identities without a default device-count budget. Their optional `maxDevices` accepts
|
|
98
|
+
`Infinity` or a positive safe integer. Explicit budgets remain enforced; device numbers
|
|
99
|
+
must still fit the disjoint application/runtime ranges of the uint32 Python stat ABI.
|
|
100
|
+
|
|
91
101
|
`withObjectFileDescriptors(fs, store)` supports large shell and Python descriptor
|
|
92
102
|
writes when the host supplies `store.createStaging`: private externally backed
|
|
93
103
|
pages keep working memory bounded without publishing the growing file after
|
|
94
104
|
every write. Conditional publication still occurs at sync/close, and retained
|
|
95
105
|
readers keep their old versions. Without that optional backend primitive, the
|
|
96
|
-
optional dirty-page budget limits unflushed output only when configured.
|
|
106
|
+
optional dirty-page budget limits unflushed output only when configured. Descriptor
|
|
107
|
+
resource limits accept `Infinity` and are unlimited when omitted; `chunkBytes`
|
|
108
|
+
remains a positive finite integer. See the
|
|
97
109
|
[object descriptor and spill contract](src/contracts/object-publication.md) for
|
|
98
110
|
backend methods, failure semantics and qualification; no provider storage is
|
|
99
111
|
configured automatically.
|
|
@@ -270,7 +282,8 @@ Every raw filesystem operation accepts an optional `signal`. Additional fields a
|
|
|
270
282
|
|
|
271
283
|
| Operation | Options |
|
|
272
284
|
| --- | --- |
|
|
273
|
-
| `readFile` | `maxBytes`:
|
|
285
|
+
| `readFile` | `maxBytes`: nonnegative safe-integer byte cap; `Infinity` adds no per-read cap |
|
|
286
|
+
| `readdir` | `maxEntries`: nonnegative safe-integer entry cap; omission or `Infinity` preserves backend limits and ordering |
|
|
274
287
|
| `writeFile`, `writeStream` | `flag`: `w` (default, replace), `wx` (exclusive create), `a` (append), or `ax` (exclusive append/create); optional `mode` |
|
|
275
288
|
| `appendFile` | Optional `mode` |
|
|
276
289
|
| `mkdir` | `recursive` (default false); optional `mode` |
|
|
@@ -278,7 +291,7 @@ Every raw filesystem operation accepts an optional `signal`. Additional fields a
|
|
|
278
291
|
| `copyFile` | `exclusive` (default false) |
|
|
279
292
|
| `readStream` | `start` (default 0), `endExclusive` (default end of file), `chunkSize` (built-in default 64 KiB) |
|
|
280
293
|
|
|
281
|
-
`validatePath(path, { maxPathBytes, maxPathComponents })` accepts optional UTF-8 byte and component quotas; both default to `Infinity`. Memory accepts the same constructor options and counts cumulative symlink expansion. Device operations accept these quotas under `pathLimits`. WebDAV accepts `maxTimestampPropertyBytes` for timestamp metadata, also unlimited by default.
|
|
294
|
+
`validatePath(path, { maxPathBytes, maxPathComponents })` accepts optional UTF-8 byte and component quotas; both default to `Infinity`. Memory accepts the same constructor options and counts cumulative symlink expansion. Device operations and WebDAV stat, access, and writes accept these quotas under `pathLimits`. WebDAV accepts `maxTimestampPropertyBytes` for timestamp metadata, also unlimited by default.
|
|
282
295
|
|
|
283
296
|
`access` takes a separate mode bitmask from `ACCESS_MODES`. `chmod` takes a mode, `utimes` takes millisecond timestamps, and `truncate` takes a byte length (default 0). For conditional chmod, check `capabilitiesFor(path, { conditionalChmod: true })` (or `capabilities`) and require `conditionalChmod: true`; supply `parent`, `expected`, and complete root-to-parent `ancestors` together. An optional mutation-free `commitGuard` must return literal `true` synchronously. Memory validates at its metadata commit; mount and supported Memory overlays preserve wrapper ancestry. Real uses its existing externally isolated host-tree boundary and does not prevent races with other processes. Backend limits still apply. Node-shaped bridge methods translate their own options rather than accepting these raw option objects; see the [bridge signatures](src/bridge/filesystem.ts).
|
|
284
297
|
|
|
@@ -289,6 +302,11 @@ Every raw filesystem operation accepts an optional `signal`. Additional fields a
|
|
|
289
302
|
|
|
290
303
|
`S3FileSystem` requires `transport` and `bucket`. A transport supplies `headObject`, `getObject`, `putObject`, `deleteObject`, `copyObject`, and `listObjectsV2`; see the [transport interface](src/fs/s3/transport.ts).
|
|
291
304
|
|
|
305
|
+
`createS3NamespaceFileSystem` resource limits (`maxBytes`, `maxEntries`,
|
|
306
|
+
`maxManifestBytes`, `maxAttempts`, and inherited descriptor limits) are unlimited
|
|
307
|
+
when omitted or set to `Infinity`; positive safe integers enable finite limits.
|
|
308
|
+
Namespace paths have no implicit length cap; configured manifest budgets still apply.
|
|
309
|
+
|
|
292
310
|
`createS3Transport(client, capabilities?)` wraps a compatible client with explicit `streamingRead`, `streamingWrite`, `conditionalPut`, `conditionalCopy`, and `conditionalDelete` flags (all absent by default). Direct transport requests accept an optional `abortSignal`.
|
|
293
311
|
|
|
294
312
|
For in-memory S3 simulations, `new MockS3Client({ buckets, pageSize?, now?, authorize? })` requires bucket names. `pageSize` defaults to 1,000 (range 1–1,000), `now` defaults to the current date/time, and optional `authorize(request)` can reject requests. It does not verify a real provider.
|
|
@@ -70,7 +70,7 @@ export class FileSystemBridge {
|
|
|
70
70
|
assertBridgePath(this.#root, this.#cwd);
|
|
71
71
|
this.#signal = options.signal;
|
|
72
72
|
const maximum = options.readFileMaxBytes;
|
|
73
|
-
if (maximum !== undefined && (!Number.isSafeInteger(maximum) || maximum < 0))
|
|
73
|
+
if (maximum !== undefined && maximum !== Infinity && (!Number.isSafeInteger(maximum) || maximum < 0))
|
|
74
74
|
throw new TypeError("Invalid readFileMaxBytes");
|
|
75
75
|
this.#readFileMaxBytes = maximum;
|
|
76
76
|
if (options.reserveReadFile !== undefined && typeof options.reserveReadFile !== "function")
|
|
@@ -44,13 +44,14 @@ export class FsError extends Error {
|
|
|
44
44
|
const operation = options.syscall ? `, ${options.syscall}` : "";
|
|
45
45
|
const path = options.path === undefined ? "" : ` '${options.path}'`;
|
|
46
46
|
const destination = options.dest === undefined ? "" : ` -> '${options.dest}'`;
|
|
47
|
-
const
|
|
48
|
-
|
|
47
|
+
const errorConstructor = Error;
|
|
48
|
+
const prevStackLimit = errorConstructor.stackTraceLimit;
|
|
49
|
+
errorConstructor.stackTraceLimit = 0;
|
|
49
50
|
try {
|
|
50
51
|
super(`${code}: ${options.message ?? descriptions[code]}${operation}${path}${destination}`, options);
|
|
51
52
|
}
|
|
52
53
|
finally {
|
|
53
|
-
|
|
54
|
+
errorConstructor.stackTraceLimit = prevStackLimit;
|
|
54
55
|
}
|
|
55
56
|
this.name = "FsError";
|
|
56
57
|
this.code = code;
|
|
@@ -170,13 +170,14 @@ export interface RenameOptions extends FsOptions {
|
|
|
170
170
|
readonly noReplace?: boolean;
|
|
171
171
|
}
|
|
172
172
|
export interface ReadFileOptions extends FsOptions {
|
|
173
|
+
/** Nonnegative safe-integer byte cap; Infinity adds no per-read cap. */
|
|
173
174
|
readonly maxBytes?: number;
|
|
174
175
|
}
|
|
175
176
|
export interface ReadDirectoryOptions extends FsOptions {
|
|
176
177
|
/**
|
|
177
|
-
* Optional per-listing entry admission limit, a nonnegative safe integer.
|
|
178
|
+
* Optional per-listing entry admission limit, a nonnegative safe integer or Infinity.
|
|
178
179
|
* Zero permits an empty listing; overflow rejects with EFBIG, never truncates.
|
|
179
|
-
* Omission preserves the adapter's existing limits and ordering. Composed
|
|
180
|
+
* Omission or Infinity preserves the adapter's existing limits and ordering. Composed
|
|
180
181
|
* adapters may conservatively count distinct candidates before visibility
|
|
181
182
|
* filtering. This is not a global traversal, host-allocation or work quota.
|
|
182
183
|
*/
|
|
@@ -7,4 +7,6 @@ export interface CollectOptions {
|
|
|
7
7
|
}
|
|
8
8
|
export declare function toByteSource(input: string | Uint8Array): ByteSource;
|
|
9
9
|
export declare function collectBytes(source: ByteSource, options: CollectOptions): Promise<Uint8Array>;
|
|
10
|
+
/** Subscribe when supported; callers must still check cancellation at operation boundaries. */
|
|
11
|
+
export declare function listenForAbort(signal: AbortSignal, listener: () => void): () => void;
|
|
10
12
|
export declare function readBytes(source: ByteSource, signal?: AbortSignal): AsyncGenerator<Uint8Array>;
|
|
@@ -108,27 +108,52 @@ export async function collectBytes(source, options) {
|
|
|
108
108
|
options.signal?.throwIfAborted();
|
|
109
109
|
return buffer.subarray(0, size);
|
|
110
110
|
}
|
|
111
|
+
/** Subscribe when supported; callers must still check cancellation at operation boundaries. */
|
|
112
|
+
export function listenForAbort(signal, listener) {
|
|
113
|
+
let callback = listener;
|
|
114
|
+
const onAbort = () => callback?.();
|
|
115
|
+
const dispose = () => {
|
|
116
|
+
// Release operation state even when a frozen host refuses removal.
|
|
117
|
+
callback = undefined;
|
|
118
|
+
try {
|
|
119
|
+
signal.removeEventListener("abort", onAbort);
|
|
120
|
+
}
|
|
121
|
+
catch (error) {
|
|
122
|
+
if (!(error instanceof TypeError) || Object.isExtensible(signal))
|
|
123
|
+
throw error;
|
|
124
|
+
}
|
|
125
|
+
};
|
|
126
|
+
try {
|
|
127
|
+
signal.addEventListener("abort", onAbort, { once: true });
|
|
128
|
+
}
|
|
129
|
+
catch (error) {
|
|
130
|
+
// Frozen hosts may register before failing to update their bookkeeping.
|
|
131
|
+
dispose();
|
|
132
|
+
if (!(error instanceof TypeError) || Object.isExtensible(signal))
|
|
133
|
+
throw error;
|
|
134
|
+
}
|
|
135
|
+
return dispose;
|
|
136
|
+
}
|
|
111
137
|
async function abortable(operation, signal) {
|
|
112
138
|
signal?.throwIfAborted();
|
|
113
139
|
if (!signal)
|
|
114
140
|
return operation();
|
|
115
141
|
return new Promise((resolve, reject) => {
|
|
116
|
-
const
|
|
117
|
-
|
|
142
|
+
const dispose = listenForAbort(signal, () => {
|
|
143
|
+
dispose();
|
|
118
144
|
reject(signal.reason);
|
|
119
|
-
};
|
|
120
|
-
signal.addEventListener("abort", onAbort, { once: true });
|
|
145
|
+
});
|
|
121
146
|
try {
|
|
122
147
|
Promise.resolve(operation()).then((result) => {
|
|
123
|
-
|
|
148
|
+
dispose();
|
|
124
149
|
resolve(result);
|
|
125
150
|
}, (error) => {
|
|
126
|
-
|
|
151
|
+
dispose();
|
|
127
152
|
reject(error);
|
|
128
153
|
});
|
|
129
154
|
}
|
|
130
155
|
catch (error) {
|
|
131
|
-
|
|
156
|
+
dispose();
|
|
132
157
|
reject(error);
|
|
133
158
|
}
|
|
134
159
|
});
|
|
@@ -4,10 +4,18 @@ export declare const dirname: typeof virtualDirname;
|
|
|
4
4
|
export declare function extname(path: string): string;
|
|
5
5
|
export declare function isAbsolutePath(path: string): boolean;
|
|
6
6
|
export declare function joinPath(...paths: string[]): string;
|
|
7
|
+
declare function resolve(...paths: string[]): string;
|
|
8
|
+
declare function relative(from: string, to: string): string;
|
|
7
9
|
export declare const posixPath: Readonly<{
|
|
8
10
|
basename: typeof basename;
|
|
9
11
|
dirname: typeof virtualDirname;
|
|
10
12
|
extname: typeof extname;
|
|
11
13
|
join: typeof joinPath;
|
|
12
14
|
isAbsolute: typeof isAbsolutePath;
|
|
15
|
+
normalize: typeof joinPath;
|
|
16
|
+
resolve: typeof resolve;
|
|
17
|
+
relative: typeof relative;
|
|
18
|
+
sep: "/";
|
|
19
|
+
delimiter: ":";
|
|
13
20
|
}>;
|
|
21
|
+
export {};
|
|
@@ -54,4 +54,25 @@ export function joinPath(...paths) {
|
|
|
54
54
|
const result = `${absolute ? "/" : ""}${components.join("/")}` || ".";
|
|
55
55
|
return joined.endsWith("/") && result !== "/" ? `${result}/` : result;
|
|
56
56
|
}
|
|
57
|
-
|
|
57
|
+
// Portable paths have a stable root instead of a process working directory.
|
|
58
|
+
function resolve(...paths) {
|
|
59
|
+
let joined = "/";
|
|
60
|
+
for (const path of paths) {
|
|
61
|
+
assertString(path);
|
|
62
|
+
joined = path.startsWith("/") ? path : `${joined}/${path}`;
|
|
63
|
+
}
|
|
64
|
+
const normalized = joinPath(joined);
|
|
65
|
+
return normalized.length > 1 && normalized.endsWith("/") ? normalized.slice(0, -1) : normalized;
|
|
66
|
+
}
|
|
67
|
+
function relative(from, to) {
|
|
68
|
+
const source = resolve(from).split("/").filter(Boolean);
|
|
69
|
+
const target = resolve(to).split("/").filter(Boolean);
|
|
70
|
+
let common = 0;
|
|
71
|
+
while (common < source.length && common < target.length && source[common] === target[common])
|
|
72
|
+
common++;
|
|
73
|
+
return [...Array(source.length - common).fill(".."), ...target.slice(common)].join("/");
|
|
74
|
+
}
|
|
75
|
+
export const posixPath = Object.freeze({
|
|
76
|
+
basename, dirname, extname, join: joinPath, isAbsolute: isAbsolutePath,
|
|
77
|
+
normalize: joinPath, resolve, relative, sep: "/", delimiter: ":"
|
|
78
|
+
});
|
package/dist/safe-fs/core.d.ts
CHANGED
|
@@ -25,3 +25,4 @@ export * from "./contracts/object.js";
|
|
|
25
25
|
export { ObjectAuthority } from "./fs/object-authority.js";
|
|
26
26
|
export { bindConditionalMutation, type ConditionalMutationBinding } from "./fs/memory/index.js";
|
|
27
27
|
export * from "./fs/s3/index.js";
|
|
28
|
+
export * from "./fs/s3/http/index.js";
|
package/dist/safe-fs/core.js
CHANGED
|
@@ -309,7 +309,8 @@ export class DeviceFileSystem {
|
|
|
309
309
|
async #readFileSlow(path, options) {
|
|
310
310
|
const resolved = await this.#resolve(path, options);
|
|
311
311
|
if (resolved === nullPath) {
|
|
312
|
-
|
|
312
|
+
if (options.maxBytes !== Infinity)
|
|
313
|
+
nonnegative(options.maxBytes, path);
|
|
313
314
|
return new Uint8Array();
|
|
314
315
|
}
|
|
315
316
|
if (resolved === deviceDirectory)
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
import { FsError } from "../contracts/errors.js";
|
|
2
2
|
export function directoryEntryLimit(options, path) {
|
|
3
3
|
options.signal?.throwIfAborted();
|
|
4
|
-
const limit = options.maxEntries;
|
|
4
|
+
const limit = options.maxEntries === Infinity ? undefined : options.maxEntries;
|
|
5
5
|
if (limit !== undefined && (!Number.isSafeInteger(limit) || limit < 0)) {
|
|
6
|
-
throw new FsError("EINVAL", { syscall: "readdir", path, message: "directory entry limit must be a nonnegative safe integer" });
|
|
6
|
+
throw new FsError("EINVAL", { syscall: "readdir", path, message: "directory entry limit must be a nonnegative safe integer or Infinity" });
|
|
7
7
|
}
|
|
8
8
|
return limit;
|
|
9
9
|
}
|
|
@@ -173,7 +173,7 @@ export declare function utf8ByteLength(value: string): number;
|
|
|
173
173
|
export declare function isCleanAbsolutePath(path: string): boolean;
|
|
174
174
|
export declare function tryResolveMemoryDevicePath(filesystem: FileSystem, path: string, resizeCreate?: boolean): string | undefined;
|
|
175
175
|
export declare function getLastReadMemoryFileSourceRef(view: Uint8Array): Uint8Array | undefined;
|
|
176
|
-
export declare function tryReadMemoryFileViewSync(filesystem: FileSystem, path: string, maxBytes?: number, signal?: AbortSignal, captureSourceRef?: boolean): Uint8Array | undefined;
|
|
176
|
+
export declare function tryReadMemoryFileViewSync(filesystem: FileSystem, path: string, maxBytes?: number, signal?: AbortSignal, captureSourceRef?: boolean, beforeRead?: () => void): Uint8Array | undefined;
|
|
177
177
|
export declare function tryGetMemoryDirectoryEntryNamesSync(filesystem: FileSystem, path: string): ReadonlyMap<string, {
|
|
178
178
|
readonly type: "file" | "directory" | "symlink";
|
|
179
179
|
}> | undefined;
|
|
@@ -46,16 +46,10 @@ Object.assign(MemoryFileNode.prototype, {
|
|
|
46
46
|
sourceRef: undefined,
|
|
47
47
|
});
|
|
48
48
|
const EMPTY_ALLOC_BYTES = new Uint8Array(0);
|
|
49
|
-
const SMALL_ALLOC_SLAB_SIZE = 8192;
|
|
50
|
-
let smallAllocSlab = new Uint8Array(SMALL_ALLOC_SLAB_SIZE);
|
|
51
|
-
let smallAllocOffset = 0;
|
|
52
49
|
const DUMMY_POOL_LEDGER = new MemoryLedger(normalizeMemoryFileSystemLimits({}));
|
|
53
50
|
const DUMMY_POOL_ALLOCATION = new MemoryAllocation(EMPTY_ALLOC_BYTES, DUMMY_POOL_LEDGER);
|
|
54
51
|
const DUMMY_POOL_FILE_NODE = new MemoryFileNode(0, 0, null, 0, DUMMY_POOL_ALLOCATION, DUMMY_POOL_ALLOCATION.data);
|
|
55
52
|
DUMMY_POOL_FILE_NODE.atimeMs = DUMMY_POOL_FILE_NODE.mtimeMs = DUMMY_POOL_FILE_NODE.ctimeMs = DUMMY_POOL_FILE_NODE.birthtimeMs = 1700000000000;
|
|
56
|
-
const defaultDateNow = Date.now;
|
|
57
|
-
let fastWriteCachedNow = Date.now();
|
|
58
|
-
let fastWriteNowTick = 0;
|
|
59
53
|
class FastDirectoryEntriesMap {
|
|
60
54
|
// Keep speculative slots local so a failed write cannot pin a tenant globally.
|
|
61
55
|
_missKey = "";
|
|
@@ -350,19 +344,13 @@ function collectRemovedChild(child, name) {
|
|
|
350
344
|
}
|
|
351
345
|
function replenishSharedMemoryPools() {
|
|
352
346
|
while (sharedAllocationPoolLen < 112) {
|
|
353
|
-
if (smallAllocOffset + 64 > SMALL_ALLOC_SLAB_SIZE) {
|
|
354
|
-
smallAllocSlab = new Uint8Array(SMALL_ALLOC_SLAB_SIZE);
|
|
355
|
-
smallAllocOffset = 0;
|
|
356
|
-
}
|
|
357
|
-
const slice = smallAllocSlab.subarray(smallAllocOffset, smallAllocOffset + 64);
|
|
358
|
-
smallAllocOffset += 64;
|
|
359
347
|
DUMMY_POOL_LEDGER.reserve(64, 0, "init", "/");
|
|
360
|
-
const alloc = new MemoryAllocation(
|
|
348
|
+
const alloc = new MemoryAllocation(new Uint8Array(64), DUMMY_POOL_LEDGER);
|
|
361
349
|
alloc.release();
|
|
362
350
|
sharedAllocationPool[sharedAllocationPoolLen++] = alloc;
|
|
363
351
|
}
|
|
364
352
|
while (sharedFileNodePoolLen < 112) {
|
|
365
|
-
sharedFileNodePool[sharedFileNodePoolLen++] = new MemoryFileNode(0, 0,
|
|
353
|
+
sharedFileNodePool[sharedFileNodePoolLen++] = new MemoryFileNode(0, 0, 0, 0, DUMMY_POOL_ALLOCATION, undefined);
|
|
366
354
|
}
|
|
367
355
|
while (sharedLargeAllocationPoolLen < 4) {
|
|
368
356
|
DUMMY_POOL_LEDGER.reserve(65536, 0, "init", "/");
|
|
@@ -377,7 +365,7 @@ function replenishSharedMemoryPools() {
|
|
|
377
365
|
sharedMediumAllocationPool[sharedMediumAllocationPoolLen++] = alloc;
|
|
378
366
|
}
|
|
379
367
|
while (sharedDirectoryNodePoolLen < 16) {
|
|
380
|
-
sharedDirectoryNodePool[sharedDirectoryNodePoolLen++] = new MemoryDirectoryNode(0, 0,
|
|
368
|
+
sharedDirectoryNodePool[sharedDirectoryNodePoolLen++] = new MemoryDirectoryNode(0, 0, 0);
|
|
381
369
|
}
|
|
382
370
|
}
|
|
383
371
|
class MemoryDirectoryNode {
|
|
@@ -897,7 +885,6 @@ export class MemoryFileSystem {
|
|
|
897
885
|
if (alloc.isReleased64()) {
|
|
898
886
|
alloc.detachLedger(DUMMY_POOL_LEDGER);
|
|
899
887
|
if (this.ledger.hasInfiniteRetained && this.ledger.hasInfiniteFileBytes && sharedAllocationPoolLen < SHARED_POOL_CAPACITY) {
|
|
900
|
-
alloc.data.fill(0);
|
|
901
888
|
sharedAllocationPool[sharedAllocationPoolLen++] = alloc;
|
|
902
889
|
}
|
|
903
890
|
else if (cache.allocations.length < 128) {
|
|
@@ -907,7 +894,6 @@ export class MemoryFileSystem {
|
|
|
907
894
|
else if (alloc.isReleased65536()) {
|
|
908
895
|
alloc.detachLedger(DUMMY_POOL_LEDGER);
|
|
909
896
|
if (this.ledger.hasInfiniteRetained && this.ledger.hasInfiniteFileBytes && sharedLargeAllocationPoolLen < SHARED_LARGE_POOL_CAPACITY) {
|
|
910
|
-
alloc.data.fill(0);
|
|
911
897
|
sharedLargeAllocationPool[sharedLargeAllocationPoolLen++] = alloc;
|
|
912
898
|
}
|
|
913
899
|
}
|
|
@@ -953,14 +939,12 @@ export class MemoryFileSystem {
|
|
|
953
939
|
if (previous.isReleased64()) {
|
|
954
940
|
previous.detachLedger(DUMMY_POOL_LEDGER);
|
|
955
941
|
if (this.ledger.hasInfiniteRetained && this.ledger.hasInfiniteFileBytes && sharedAllocationPoolLen < SHARED_POOL_CAPACITY) {
|
|
956
|
-
previous.data.fill(0);
|
|
957
942
|
sharedAllocationPool[sharedAllocationPoolLen++] = previous;
|
|
958
943
|
}
|
|
959
944
|
}
|
|
960
945
|
else if (previous.isReleased65536()) {
|
|
961
946
|
previous.detachLedger(DUMMY_POOL_LEDGER);
|
|
962
947
|
if (this.ledger.hasInfiniteRetained && this.ledger.hasInfiniteFileBytes && sharedLargeAllocationPoolLen < SHARED_LARGE_POOL_CAPACITY) {
|
|
963
|
-
previous.data.fill(0);
|
|
964
948
|
sharedLargeAllocationPool[sharedLargeAllocationPoolLen++] = previous;
|
|
965
949
|
}
|
|
966
950
|
}
|
|
@@ -1372,17 +1356,6 @@ export class MemoryFileSystem {
|
|
|
1372
1356
|
}
|
|
1373
1357
|
}
|
|
1374
1358
|
try {
|
|
1375
|
-
// A shared slab's backing buffer exceeds the admitted slice capacity.
|
|
1376
|
-
// Finite file/storage ceilings require independently owned exact buffers.
|
|
1377
|
-
if (length <= 64 && this.ledger.hasInfiniteRetained && this.ledger.hasInfiniteFileBytes) {
|
|
1378
|
-
if (smallAllocOffset + length > SMALL_ALLOC_SLAB_SIZE) {
|
|
1379
|
-
smallAllocSlab = new Uint8Array(SMALL_ALLOC_SLAB_SIZE);
|
|
1380
|
-
smallAllocOffset = 0;
|
|
1381
|
-
}
|
|
1382
|
-
const slice = smallAllocSlab.subarray(smallAllocOffset, smallAllocOffset + length);
|
|
1383
|
-
smallAllocOffset += length;
|
|
1384
|
-
return new MemoryAllocation(slice, this.ledger);
|
|
1385
|
-
}
|
|
1386
1359
|
return new MemoryAllocation(new Uint8Array(length), this.ledger);
|
|
1387
1360
|
}
|
|
1388
1361
|
catch (cause) {
|
|
@@ -1568,11 +1541,12 @@ export class MemoryFileSystem {
|
|
|
1568
1541
|
}
|
|
1569
1542
|
if (position > curLen)
|
|
1570
1543
|
storage.fill(0, curLen, position);
|
|
1544
|
+
const source = data;
|
|
1571
1545
|
if (dataLen === data.byteLength) {
|
|
1572
1546
|
storage.set(data, position);
|
|
1573
1547
|
}
|
|
1574
|
-
else if (typeof
|
|
1575
|
-
|
|
1548
|
+
else if (typeof source.copy === "function") {
|
|
1549
|
+
source.copy(storage, position, 0, dataLen);
|
|
1576
1550
|
}
|
|
1577
1551
|
else {
|
|
1578
1552
|
for (let i = 0; i < dataLen; i++)
|
|
@@ -1771,7 +1745,7 @@ export class MemoryFileSystem {
|
|
|
1771
1745
|
1 + batchCount <= entries._keys.length) {
|
|
1772
1746
|
this.ledger.reserve(totalNameBytes, 2 * batchCount, "writeFile", fileNames[batchCount - 1]);
|
|
1773
1747
|
this.totalBytes += payloadBytesSum;
|
|
1774
|
-
const now =
|
|
1748
|
+
const now = Date.now();
|
|
1775
1749
|
const fileMode = typeModes.file | mode;
|
|
1776
1750
|
let ino = this.nextInode;
|
|
1777
1751
|
this.nextInode = ino + batchCount;
|
|
@@ -1844,7 +1818,7 @@ export class MemoryFileSystem {
|
|
|
1844
1818
|
const existing = nameHash !== undefined ? entries.getForWriteWithHash(name, nameHash) : entries.getForWrite(name);
|
|
1845
1819
|
if (existing === undefined && entries._next < entries._keys.length) {
|
|
1846
1820
|
this.ledger.reserve(name.length * 2, 2, "writeFile", name);
|
|
1847
|
-
const now =
|
|
1821
|
+
const now = Date.now();
|
|
1848
1822
|
const length = data.byteLength;
|
|
1849
1823
|
const node = sharedFileNodePool[--sharedFileNodePoolLen];
|
|
1850
1824
|
sharedFileNodePool[sharedFileNodePoolLen] = DUMMY_POOL_FILE_NODE;
|
|
@@ -1966,7 +1940,7 @@ export class MemoryFileSystem {
|
|
|
1966
1940
|
allocation.data.set(data);
|
|
1967
1941
|
this.ledger.reserve(nameBytes, 2, syscall, name);
|
|
1968
1942
|
try {
|
|
1969
|
-
const now =
|
|
1943
|
+
const now = Date.now();
|
|
1970
1944
|
const fileMode = typeModes.file | mode;
|
|
1971
1945
|
const view = zeroCopyConst ? new Uint8Array(data) : (capacity === length ? allocation.data : undefined);
|
|
1972
1946
|
let node;
|
|
@@ -2047,7 +2021,7 @@ export class MemoryFileSystem {
|
|
|
2047
2021
|
}
|
|
2048
2022
|
async readFile(path, options = {}) {
|
|
2049
2023
|
options.signal?.throwIfAborted();
|
|
2050
|
-
if (options.maxBytes !== undefined)
|
|
2024
|
+
if (options.maxBytes !== undefined && options.maxBytes !== Infinity)
|
|
2051
2025
|
this.integer(options.maxBytes, "readFile", path);
|
|
2052
2026
|
const node = this.file(path, "readFile");
|
|
2053
2027
|
this.permission(node, 4, "readFile", path);
|
|
@@ -3120,7 +3094,7 @@ export class MemoryRedirectHandle {
|
|
|
3120
3094
|
if (len === 0)
|
|
3121
3095
|
return;
|
|
3122
3096
|
const pos = this.append ? this.node.byteLength : this.position;
|
|
3123
|
-
const now = Date.now
|
|
3097
|
+
const now = Date.now();
|
|
3124
3098
|
this.fs.writeAtRange(this.node, src, len, pos, "write", this.path, now);
|
|
3125
3099
|
if (!this.append)
|
|
3126
3100
|
this.position = pos + len;
|
|
@@ -3140,7 +3114,7 @@ export class MemoryRedirectHandle {
|
|
|
3140
3114
|
mem.ledger.hasInfiniteRetained &&
|
|
3141
3115
|
mem.ledger.hasInfiniteFileBytes &&
|
|
3142
3116
|
mem.ledger.limits.maxBytes === undefined) {
|
|
3143
|
-
const now = Date.now
|
|
3117
|
+
const now = Date.now();
|
|
3144
3118
|
this.node.byteLength = len;
|
|
3145
3119
|
this.node.view = immutableData;
|
|
3146
3120
|
this.node.sourceRef = immutableData;
|
|
@@ -3266,7 +3240,7 @@ export function tryOpenMemoryRedirectHandleSync(filesystem, path, append, mode,
|
|
|
3266
3240
|
}
|
|
3267
3241
|
const cache = mem._cache;
|
|
3268
3242
|
cache.clearWrites();
|
|
3269
|
-
const now = Date.now
|
|
3243
|
+
const now = Date.now();
|
|
3270
3244
|
const fileMode = typeModes.file | validMode;
|
|
3271
3245
|
let newNode;
|
|
3272
3246
|
if (sharedFileNodePoolLen > 0) {
|
|
@@ -3467,7 +3441,7 @@ export function getLastReadMemoryFileSourceRef(view) {
|
|
|
3467
3441
|
lastReadViewSourceRef = undefined;
|
|
3468
3442
|
return ref;
|
|
3469
3443
|
}
|
|
3470
|
-
export function tryReadMemoryFileViewSync(filesystem, path, maxBytes, signal, captureSourceRef = false) {
|
|
3444
|
+
export function tryReadMemoryFileViewSync(filesystem, path, maxBytes, signal, captureSourceRef = false, beforeRead) {
|
|
3471
3445
|
// A capture belongs only to the most recent read, including failed/fallback reads.
|
|
3472
3446
|
lastReadViewData = undefined;
|
|
3473
3447
|
lastReadViewSourceRef = undefined;
|
|
@@ -3475,19 +3449,15 @@ export function tryReadMemoryFileViewSync(filesystem, path, maxBytes, signal, ca
|
|
|
3475
3449
|
if (mem._owner === undefined || !isStockMemoryMethods(mem, readFileFastMethodNames, false))
|
|
3476
3450
|
return undefined;
|
|
3477
3451
|
signal?.throwIfAborted();
|
|
3478
|
-
|
|
3452
|
+
beforeRead?.();
|
|
3453
|
+
if (maxBytes !== undefined && maxBytes !== Infinity)
|
|
3479
3454
|
mem.integer(maxBytes, "readFile", path);
|
|
3480
3455
|
const node = mem.file(path, "readFile");
|
|
3481
3456
|
mem.permission(node, 4, "readFile", path);
|
|
3482
3457
|
const data = node.data;
|
|
3483
3458
|
if (maxBytes !== undefined && data.byteLength > maxBytes)
|
|
3484
3459
|
mem.fail("EFBIG", "readFile", path);
|
|
3485
|
-
|
|
3486
|
-
node.atimeMs = Date.now();
|
|
3487
|
-
else if ((++fastWriteNowTick & 63) === 0)
|
|
3488
|
-
node.atimeMs = fastWriteCachedNow = Date.now();
|
|
3489
|
-
else
|
|
3490
|
-
node.atimeMs = fastWriteCachedNow;
|
|
3460
|
+
node.atimeMs = Date.now();
|
|
3491
3461
|
if (captureSourceRef && node.revision === 0) {
|
|
3492
3462
|
lastReadViewData = new WeakRef(data);
|
|
3493
3463
|
lastReadViewSourceRef = node.sourceRef === undefined ? undefined : new WeakRef(node.sourceRef);
|
|
@@ -3570,7 +3540,7 @@ export function tryGetMemoryDirectoryEntryNamesSync(filesystem, path) {
|
|
|
3570
3540
|
validatePath(path, owner.ledger.limits);
|
|
3571
3541
|
const root = mem.root;
|
|
3572
3542
|
if (path === "/") {
|
|
3573
|
-
if (((root.mode >> 6) &
|
|
3543
|
+
if (((root.mode >> 6) & 5) !== 5)
|
|
3574
3544
|
return undefined;
|
|
3575
3545
|
return root.entries;
|
|
3576
3546
|
}
|
|
@@ -3581,7 +3551,7 @@ export function tryGetMemoryDirectoryEntryNamesSync(filesystem, path) {
|
|
|
3581
3551
|
const fastPrefix = cache?.lastFastDirPrefix;
|
|
3582
3552
|
if (fastDir !== undefined &&
|
|
3583
3553
|
fastDir.nlink !== 0 &&
|
|
3584
|
-
((fastDir.mode >> 6) &
|
|
3554
|
+
((fastDir.mode >> 6) & 5) === 5 &&
|
|
3585
3555
|
fastPrefix !== undefined &&
|
|
3586
3556
|
(fastPrefix === path || (fastPrefix.length === path.length + 1 && fastPrefix.charCodeAt(path.length) === 47 && fastPrefix.startsWith(path)))) {
|
|
3587
3557
|
return fastDir.entries;
|
|
@@ -3594,7 +3564,7 @@ export function tryGetMemoryDirectoryEntryNamesSync(filesystem, path) {
|
|
|
3594
3564
|
const slash = path.indexOf("/", start);
|
|
3595
3565
|
if (slash === -1) {
|
|
3596
3566
|
const next = current.entries.getSubstr(path, start, path.length);
|
|
3597
|
-
if (!next || next.type !== "directory" || ((next.mode >> 6) &
|
|
3567
|
+
if (!next || next.type !== "directory" || ((next.mode >> 6) & 5) !== 5)
|
|
3598
3568
|
return undefined;
|
|
3599
3569
|
return next.entries;
|
|
3600
3570
|
}
|
|
@@ -3648,7 +3618,7 @@ export function tryMkdirMemorySync(filesystem, path, recursive, mode, signal, di
|
|
|
3648
3618
|
let next = current.entries.get(seg);
|
|
3649
3619
|
if (!next) {
|
|
3650
3620
|
mem.permission(current, 3, "mkdir", path);
|
|
3651
|
-
next = mem.addDirectoryNode(current, seg, validMode, "mkdir", path,
|
|
3621
|
+
next = mem.addDirectoryNode(current, seg, validMode, "mkdir", path, Date.now());
|
|
3652
3622
|
}
|
|
3653
3623
|
else if (next.type !== "directory") {
|
|
3654
3624
|
mem.fail(slash === -1 ? "EEXIST" : "ENOTDIR", "mkdir", path);
|
|
@@ -3669,7 +3639,7 @@ export function tryMkdirMemorySync(filesystem, path, recursive, mode, signal, di
|
|
|
3669
3639
|
if (location.node)
|
|
3670
3640
|
mem.fail("EEXIST", "mkdir", path);
|
|
3671
3641
|
mem.permission(location.parent, 3, "mkdir", path);
|
|
3672
|
-
mem.addDirectoryNode(location.parent, location.name, validMode, "mkdir", path,
|
|
3642
|
+
mem.addDirectoryNode(location.parent, location.name, validMode, "mkdir", path, Date.now());
|
|
3673
3643
|
return true;
|
|
3674
3644
|
}
|
|
3675
3645
|
export function tryRmRfMemorySync(filesystem, path, signal) {
|
|
@@ -3719,7 +3689,7 @@ export function tryRmRfMemorySync(filesystem, path, signal) {
|
|
|
3719
3689
|
sharedFastLocation.name = targetName;
|
|
3720
3690
|
sharedFastLocation.path = path;
|
|
3721
3691
|
try {
|
|
3722
|
-
mem.removeLocation(sharedFastLocation, path, "rm", true,
|
|
3692
|
+
mem.removeLocation(sharedFastLocation, path, "rm", true, Date.now());
|
|
3723
3693
|
}
|
|
3724
3694
|
finally {
|
|
3725
3695
|
sharedFastLocation.node = undefined;
|
|
@@ -84,8 +84,10 @@ export class MemoryAllocation {
|
|
|
84
84
|
return;
|
|
85
85
|
if (this.references <= 0)
|
|
86
86
|
throw new Error("Memory allocation already released");
|
|
87
|
-
if (--this.references === 0)
|
|
87
|
+
if (--this.references === 0) {
|
|
88
|
+
this.data.fill(0);
|
|
88
89
|
this.ledger.release(this.data.byteLength, 0);
|
|
90
|
+
}
|
|
89
91
|
}
|
|
90
92
|
isReleased64() {
|
|
91
93
|
return this.references === 0 && this.data.byteLength === 64;
|
|
@@ -9,7 +9,8 @@ export function withObjectFileDescriptors(filesystem, store, options = {}) {
|
|
|
9
9
|
const maxStagedPages = options.maxStagedPages ?? Infinity;
|
|
10
10
|
const maxFileBytes = options.maxFileBytes ?? Infinity;
|
|
11
11
|
const maxOpenFiles = options.maxOpenFiles ?? Infinity;
|
|
12
|
-
if (!
|
|
12
|
+
if (!Number.isSafeInteger(chunkBytes) || chunkBytes < 1
|
|
13
|
+
|| ![maxStagedBytes, maxStagedPages, maxFileBytes, maxOpenFiles].every(value => value === Infinity || Number.isSafeInteger(value) && value > 0)
|
|
13
14
|
|| typeof store.acquire !== "function"
|
|
14
15
|
|| store.publish !== undefined && typeof store.publish !== "function"
|
|
15
16
|
|| store.createStaging !== undefined && typeof store.createStaging !== "function")
|
|
@@ -608,8 +608,9 @@ export class RealFileSystem {
|
|
|
608
608
|
}
|
|
609
609
|
async readFile(path, options = {}) {
|
|
610
610
|
return this.operation("readFile", path, options, async () => {
|
|
611
|
-
const maxBytes = options.maxBytes ??
|
|
612
|
-
|
|
611
|
+
const maxBytes = options.maxBytes ?? Infinity;
|
|
612
|
+
if (maxBytes !== Infinity)
|
|
613
|
+
integer(maxBytes);
|
|
613
614
|
return collectBytes(this.readStream(path, options), { maxBytes, ...options });
|
|
614
615
|
});
|
|
615
616
|
}
|
|
@@ -1,4 +1,15 @@
|
|
|
1
|
-
import type { RequestOptions } from "node:http";
|
|
2
1
|
import type { RequestScope, WireResponse } from "./request.js";
|
|
3
|
-
|
|
4
|
-
|
|
2
|
+
export type S3HttpRequestFactory = never;
|
|
3
|
+
interface FetchRequestOptions {
|
|
4
|
+
readonly protocol?: string | undefined;
|
|
5
|
+
readonly hostname?: string | undefined;
|
|
6
|
+
readonly port?: string | number | undefined;
|
|
7
|
+
readonly path?: string | undefined;
|
|
8
|
+
readonly method?: string | undefined;
|
|
9
|
+
readonly headers?: Record<string, string> | undefined;
|
|
10
|
+
readonly agent?: false;
|
|
11
|
+
readonly maxHeaderSize?: number;
|
|
12
|
+
readonly rejectUnauthorized?: true;
|
|
13
|
+
}
|
|
14
|
+
export declare function sendRequest(options: FetchRequestOptions, body: Uint8Array, scope: RequestScope, factory?: S3HttpRequestFactory): Promise<WireResponse>;
|
|
15
|
+
export {};
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { RequestOptions } from "node:http";
|
|
2
|
-
import type { S3HttpRequestFactory } from "./types.js";
|
|
1
|
+
import type { ClientRequest, IncomingMessage, RequestOptions } from "node:http";
|
|
3
2
|
import type { RequestScope, WireResponse } from "./request.js";
|
|
3
|
+
export type S3HttpRequestFactory = (options: RequestOptions, onResponse: (response: IncomingMessage) => void) => ClientRequest;
|
|
4
4
|
export declare function sendRequest(options: RequestOptions, body: Uint8Array, scope: RequestScope, factory?: S3HttpRequestFactory): Promise<WireResponse>;
|
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import type {
|
|
1
|
+
import type { S3HttpRequestFactory } from "#safe-fs-s3-request";
|
|
2
|
+
export type { S3HttpRequestFactory } from "#safe-fs-s3-request";
|
|
2
3
|
export interface S3HttpCredentials {
|
|
3
4
|
readonly accessKeyId: string;
|
|
4
5
|
readonly secretAccessKey: string;
|
|
@@ -7,7 +8,6 @@ export interface S3HttpCredentials {
|
|
|
7
8
|
export type S3HttpCredentialProvider = (options: {
|
|
8
9
|
readonly signal: AbortSignal;
|
|
9
10
|
}) => Promise<S3HttpCredentials>;
|
|
10
|
-
export type S3HttpRequestFactory = (options: RequestOptions, onResponse: (response: IncomingMessage) => void) => ClientRequest;
|
|
11
11
|
export interface S3HttpTransportOptions {
|
|
12
12
|
readonly endpoint: string;
|
|
13
13
|
readonly region: string;
|
|
@@ -41,8 +41,6 @@ export async function* admitManifest(source, options) {
|
|
|
41
41
|
invalid();
|
|
42
42
|
const key = value;
|
|
43
43
|
if (frame.shape === 'nodes') {
|
|
44
|
-
if (key.length > 4096)
|
|
45
|
-
invalid();
|
|
46
44
|
if (++entries > options.maxEntries)
|
|
47
45
|
throw new FsError('ENOSPC');
|
|
48
46
|
}
|
|
@@ -83,8 +81,6 @@ export async function* admitManifest(source, options) {
|
|
|
83
81
|
if (token) {
|
|
84
82
|
if (quoted) {
|
|
85
83
|
token += character;
|
|
86
|
-
if (token.length > 4096 * 6 + 2)
|
|
87
|
-
invalid();
|
|
88
84
|
if (escaped)
|
|
89
85
|
escaped = false;
|
|
90
86
|
else if (character === '\\')
|
|
@@ -4,7 +4,7 @@ import { dirname, normalizePath, validatePath } from '../../contracts/virtual-pa
|
|
|
4
4
|
import { withObjectFileDescriptors } from '../object-publication/index.js';
|
|
5
5
|
import { admitManifest } from './manifest-admission.js';
|
|
6
6
|
function limit(value) {
|
|
7
|
-
if (value === undefined)
|
|
7
|
+
if (value === undefined || value === Infinity)
|
|
8
8
|
return Infinity;
|
|
9
9
|
if (!Number.isSafeInteger(value) || value < 1)
|
|
10
10
|
throw new RangeError('Invalid S3 namespace limit');
|
|
@@ -18,7 +18,7 @@ export async function createS3NamespaceFileSystem(options) {
|
|
|
18
18
|
throw new TypeError('An explicit S3 bucket and manifest key are required');
|
|
19
19
|
const maxBytes = limit(options.maxBytes);
|
|
20
20
|
const maxEntries = limit(options.maxEntries);
|
|
21
|
-
const maxManifestBytes =
|
|
21
|
+
const maxManifestBytes = limit(options.maxManifestBytes);
|
|
22
22
|
const maxAttempts = limit(options.maxAttempts);
|
|
23
23
|
const descriptorOptions = options;
|
|
24
24
|
for (const key of ['chunkBytes', 'maxStagedBytes', 'maxStagedPages', 'maxFileBytes', 'maxOpenFiles']) {
|
|
@@ -32,8 +32,6 @@ export async function createS3NamespaceFileSystem(options) {
|
|
|
32
32
|
validatePath(value);
|
|
33
33
|
if (!value)
|
|
34
34
|
throw new FsError('ENOENT');
|
|
35
|
-
if (value.length > 4096)
|
|
36
|
-
throw new FsError('ENAMETOOLONG');
|
|
37
35
|
return normalizePath(value);
|
|
38
36
|
};
|
|
39
37
|
const entryPath = (value) => {
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { FsError, isErrnoCode, isFsError } from "../../contracts/errors.js";
|
|
2
2
|
import { composeAbortSignals } from "../../contracts/abort.js";
|
|
3
3
|
import { readBytes } from "../../contracts/io.js";
|
|
4
|
+
import { validatePath } from "../../contracts/virtual-path.js";
|
|
4
5
|
import { davChild, davChildren, parseXmlSteps, scalar, XmlResponseLimitError, XmlResourceLimitError } from "./xml.js";
|
|
5
6
|
import { admitDirectoryEntries, directoryEntryLimit } from "../directory-admission.js";
|
|
6
7
|
import { assertCallbackAuthorityAllowed, compareEntries, registerEntryAuthority } from "../mount/comparison.js";
|
|
@@ -159,28 +160,6 @@ function normalize(path) {
|
|
|
159
160
|
}
|
|
160
161
|
return `/${segments.join("/")}`;
|
|
161
162
|
}
|
|
162
|
-
function validateDirectoryWalkPath(path, syscall = "access") {
|
|
163
|
-
if (typeof path !== "string")
|
|
164
|
-
fail("EINVAL", "resolve", String(path), "invalid WebDAV path");
|
|
165
|
-
let bytes = 0;
|
|
166
|
-
let components = 0;
|
|
167
|
-
let inComponent = false;
|
|
168
|
-
for (let offset = 0; offset < path.length; offset++) {
|
|
169
|
-
const point = path.codePointAt(offset);
|
|
170
|
-
if (point === 47)
|
|
171
|
-
inComponent = false;
|
|
172
|
-
else if (!inComponent) {
|
|
173
|
-
components++;
|
|
174
|
-
inComponent = true;
|
|
175
|
-
}
|
|
176
|
-
bytes += point <= 0x7f ? 1 : point <= 0x7ff ? 2 : point <= 0xffff ? 3 : 4;
|
|
177
|
-
if (point > 0xffff)
|
|
178
|
-
offset++;
|
|
179
|
-
if (bytes > 65_536 || components > 256) {
|
|
180
|
-
fail("ENAMETOOLONG", syscall, path, "directory access exceeds the 64KiB path or 256 component limit");
|
|
181
|
-
}
|
|
182
|
-
}
|
|
183
|
-
}
|
|
184
163
|
function requiresCollection(path) {
|
|
185
164
|
const last = path.split("/").at(-1);
|
|
186
165
|
return last === "" || last === "." || last === "..";
|
|
@@ -826,7 +805,7 @@ export class WebDavFileSystem {
|
|
|
826
805
|
}
|
|
827
806
|
}
|
|
828
807
|
async stat(path, options = {}) {
|
|
829
|
-
|
|
808
|
+
validatePath(path, options.pathLimits);
|
|
830
809
|
return this.statWithAncestors(path, options);
|
|
831
810
|
}
|
|
832
811
|
async statWithAncestors(path, options) {
|
|
@@ -909,7 +888,7 @@ export class WebDavFileSystem {
|
|
|
909
888
|
}.bind(this));
|
|
910
889
|
}
|
|
911
890
|
async prepareWrite(path, options) {
|
|
912
|
-
|
|
891
|
+
validatePath(path, options.pathLimits);
|
|
913
892
|
const normalized = normalize(path);
|
|
914
893
|
if (options.mode !== undefined)
|
|
915
894
|
this.unsupported("writeFile mode", path);
|
|
@@ -1329,10 +1308,7 @@ export class WebDavFileSystem {
|
|
|
1329
1308
|
fail("ECANCELED", "access", path);
|
|
1330
1309
|
if (mode & 2)
|
|
1331
1310
|
this.unsupported("access write/execute permission checks", path);
|
|
1332
|
-
|
|
1333
|
-
validateDirectoryWalkPath(path);
|
|
1334
|
-
// Non-execute probes do not inherit stat's raw-path limits.
|
|
1335
|
-
const stat = await (mode & 1 ? this.stat(path, options) : this.statWithAncestors(path, options));
|
|
1311
|
+
const stat = await this.stat(path, options);
|
|
1336
1312
|
if (options.signal?.aborted)
|
|
1337
1313
|
fail("ECANCELED", "access", path);
|
|
1338
1314
|
if ((mode & 1) && stat.type !== "directory")
|
|
@@ -20,8 +20,8 @@ export function parseXml(input, limits = {}) {
|
|
|
20
20
|
}
|
|
21
21
|
export function* parseXmlSteps(input, limits = {}) {
|
|
22
22
|
const { maxResponses, ...documentLimits } = limits;
|
|
23
|
-
if (maxResponses !== undefined && (!Number.isSafeInteger(maxResponses) || maxResponses < 1)) {
|
|
24
|
-
throw new RangeError("XML limits must be positive integers");
|
|
23
|
+
if (maxResponses !== undefined && maxResponses !== Infinity && (!Number.isSafeInteger(maxResponses) || maxResponses < 1)) {
|
|
24
|
+
throw new RangeError("XML limits must be positive integers or Infinity");
|
|
25
25
|
}
|
|
26
26
|
let responses = 0;
|
|
27
27
|
try {
|
|
@@ -332,8 +332,8 @@ export function writePythonStat(heap, pointer, stat) {
|
|
|
332
332
|
return 0;
|
|
333
333
|
}
|
|
334
334
|
/** Runtime storage and application storage occupy disjoint guest device ranges. */
|
|
335
|
-
export function createPythonRuntimeStatMapper(maxDevices =
|
|
336
|
-
if (!Number.isSafeInteger(maxDevices) || maxDevices < 1
|
|
335
|
+
export function createPythonRuntimeStatMapper(maxDevices = Infinity) {
|
|
336
|
+
if (maxDevices !== Infinity && (!Number.isSafeInteger(maxDevices) || maxDevices < 1))
|
|
337
337
|
throw new RangeError('Invalid runtime device limit');
|
|
338
338
|
const mapped = Symbol('mapped Python runtime device');
|
|
339
339
|
const devices = new Map();
|
|
@@ -346,6 +346,9 @@ export function createPythonRuntimeStatMapper(maxDevices = 1024) {
|
|
|
346
346
|
if (device === undefined) {
|
|
347
347
|
if (devices.size >= maxDevices)
|
|
348
348
|
throw new Error('EOVERFLOW');
|
|
349
|
+
// Preserve the uint32 guest ABI even when the resource budget is unlimited.
|
|
350
|
+
if (devices.size >= 0x80000000)
|
|
351
|
+
throw new Error('EOVERFLOW');
|
|
349
352
|
device = 0x80000000 + devices.size;
|
|
350
353
|
devices.set(stat.dev, device);
|
|
351
354
|
}
|
|
@@ -23,11 +23,11 @@ export class PythonFileSystem {
|
|
|
23
23
|
if (!options.cwd.startsWith("/"))
|
|
24
24
|
throw new FsError("EINVAL", { syscall: "python filesystem", path: options.cwd, message: "cwd must be absolute" });
|
|
25
25
|
this.#cwd = options.cwd;
|
|
26
|
-
this.#transfer = options.maxTransferBytes ??
|
|
26
|
+
this.#transfer = options.maxTransferBytes ?? Infinity;
|
|
27
27
|
this.#limit = options.maxOpenFiles ?? Infinity;
|
|
28
28
|
this.#directoryLimit = options.maxDirectoryEntries ?? Infinity;
|
|
29
29
|
for (const limit of [this.#transfer, options.maxOpenFiles, options.maxDirectoryEntries])
|
|
30
|
-
if (limit !== undefined && (!Number.isSafeInteger(limit) || limit < 1))
|
|
30
|
+
if (limit !== undefined && limit !== Infinity && (!Number.isSafeInteger(limit) || limit < 1))
|
|
31
31
|
throw new FsError("EINVAL", { syscall: "python filesystem" });
|
|
32
32
|
this.#open = options.open;
|
|
33
33
|
this.#scope = composeAbortSignals([...(options.signal === undefined ? [] : [options.signal]), this.#abort.signal]);
|
|
@@ -10,7 +10,7 @@ const errno = {
|
|
|
10
10
|
};
|
|
11
11
|
export function createPythonNativeSyscalls(options) {
|
|
12
12
|
const { runtime, signal, maxTransferBytes, runtimeMount } = options;
|
|
13
|
-
if (!Number.isSafeInteger(maxTransferBytes) || maxTransferBytes < 1)
|
|
13
|
+
if (maxTransferBytes !== Infinity && (!Number.isSafeInteger(maxTransferBytes) || maxTransferBytes < 1))
|
|
14
14
|
throw new RangeError('Invalid native transfer limit');
|
|
15
15
|
const descriptors = new Map();
|
|
16
16
|
const closedStdio = new Set();
|
|
@@ -4,8 +4,8 @@ export class PythonStatTranslator {
|
|
|
4
4
|
#devices = new Map();
|
|
5
5
|
#maxDevices;
|
|
6
6
|
#next = 1;
|
|
7
|
-
constructor(maxDevices =
|
|
8
|
-
if (!Number.isSafeInteger(maxDevices) || maxDevices < 1)
|
|
7
|
+
constructor(maxDevices = Infinity) {
|
|
8
|
+
if (maxDevices !== Infinity && (!Number.isSafeInteger(maxDevices) || maxDevices < 1))
|
|
9
9
|
throw new FsError("EINVAL");
|
|
10
10
|
this.#maxDevices = maxDevices;
|
|
11
11
|
}
|
|
@@ -19,6 +19,9 @@ export class PythonStatTranslator {
|
|
|
19
19
|
if (guestDevice === undefined) {
|
|
20
20
|
if (this.#next > this.#maxDevices)
|
|
21
21
|
throw new FsError("EFBIG", { syscall: "stat", message: "Python device identity limit exceeded" });
|
|
22
|
+
// The upper half of the uint32 guest ABI is reserved for runtime devices.
|
|
23
|
+
if (this.#next >= 0x80000000)
|
|
24
|
+
throw new FsError("EFBIG", { syscall: "stat", message: "Python device identity exceeds guest ABI range" });
|
|
22
25
|
if (!devices) {
|
|
23
26
|
devices = new Map();
|
|
24
27
|
this.#devices.set(identityScope, devices);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@poe-platform/safe-fs",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.804",
|
|
4
4
|
"description": "Composable filesystem with a portable core and explicit Node adapters",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -43,12 +43,14 @@
|
|
|
43
43
|
"types": "./dist/safe-fs/testing/atomic-filesystem.d.ts",
|
|
44
44
|
"workerd": "./dist/safe-fs/testing/atomic-filesystem.js",
|
|
45
45
|
"browser": "./dist/safe-fs/testing/atomic-filesystem.js",
|
|
46
|
+
"worker": "./dist/safe-fs/testing/atomic-filesystem.js",
|
|
46
47
|
"import": "./dist/safe-fs/testing/atomic-filesystem.js"
|
|
47
48
|
},
|
|
48
49
|
"./testing/object-publication": {
|
|
49
50
|
"types": "./dist/safe-fs/testing/object-publication.d.ts",
|
|
50
51
|
"workerd": "./dist/safe-fs/testing/object-publication.js",
|
|
51
52
|
"browser": "./dist/safe-fs/testing/object-publication.js",
|
|
53
|
+
"worker": "./dist/safe-fs/testing/object-publication.js",
|
|
52
54
|
"import": "./dist/safe-fs/testing/object-publication.js"
|
|
53
55
|
},
|
|
54
56
|
"./node": {
|
|
@@ -63,10 +65,12 @@
|
|
|
63
65
|
"types": {
|
|
64
66
|
"workerd": "./dist/safe-fs/node-unavailable.d.ts",
|
|
65
67
|
"browser": "./dist/safe-fs/node-unavailable.d.ts",
|
|
68
|
+
"worker": "./dist/safe-fs/node-unavailable.d.ts",
|
|
66
69
|
"default": "./dist/safe-fs/node/filesystem.d.ts"
|
|
67
70
|
},
|
|
68
71
|
"workerd": null,
|
|
69
72
|
"browser": null,
|
|
73
|
+
"worker": null,
|
|
70
74
|
"import": "./dist/safe-fs/node/filesystem.js"
|
|
71
75
|
},
|
|
72
76
|
"./fs/memory": {
|
|
@@ -138,9 +142,15 @@
|
|
|
138
142
|
},
|
|
139
143
|
"imports": {
|
|
140
144
|
"#safe-fs-s3-request": {
|
|
141
|
-
"types":
|
|
145
|
+
"types": {
|
|
146
|
+
"workerd": "./dist/safe-fs/fs/s3/http/request-fetch.d.ts",
|
|
147
|
+
"browser": "./dist/safe-fs/fs/s3/http/request-fetch.d.ts",
|
|
148
|
+
"worker": "./dist/safe-fs/fs/s3/http/request-fetch.d.ts",
|
|
149
|
+
"default": "./dist/safe-fs/fs/s3/http/request-node.d.ts"
|
|
150
|
+
},
|
|
142
151
|
"workerd": "./dist/safe-fs/fs/s3/http/request-fetch.js",
|
|
143
152
|
"browser": "./dist/safe-fs/fs/s3/http/request-fetch.js",
|
|
153
|
+
"worker": "./dist/safe-fs/fs/s3/http/request-fetch.js",
|
|
144
154
|
"default": "./dist/safe-fs/fs/s3/http/request-node.js"
|
|
145
155
|
},
|
|
146
156
|
"#safe-fs-platform": {
|