@owlmeans/file-streams 0.2.0-alpha.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/README.md +437 -21
  2. package/agent-meta/manifest.json +77 -8
  3. package/agent-meta/skills/file-streams/SKILL.md +84 -105
  4. package/agent-meta/skills/file-streams/parts/archives.md +80 -0
  5. package/agent-meta/skills/file-streams/parts/extending.md +117 -0
  6. package/agent-meta/skills/file-streams/parts/network.md +77 -22
  7. package/agent-meta/skills/file-streams/parts/workload.md +51 -0
  8. package/agent-meta/skills/file-streams/parts/workspaces.md +9 -22
  9. package/build/errors.d.ts +5 -0
  10. package/build/errors.d.ts.map +1 -1
  11. package/build/errors.js +5 -0
  12. package/build/errors.js.map +1 -1
  13. package/build/fetch.d.ts +12 -0
  14. package/build/fetch.d.ts.map +1 -0
  15. package/build/fetch.js +6 -0
  16. package/build/fetch.js.map +1 -0
  17. package/build/http/types.d.ts +19 -1
  18. package/build/http/types.d.ts.map +1 -1
  19. package/build/http.d.ts.map +1 -1
  20. package/build/http.js +7 -4
  21. package/build/http.js.map +1 -1
  22. package/build/index.d.ts +2 -0
  23. package/build/index.d.ts.map +1 -1
  24. package/build/index.js +1 -0
  25. package/build/index.js.map +1 -1
  26. package/build/pinned-fetch/consts.local.d.ts +9 -0
  27. package/build/pinned-fetch/consts.local.d.ts.map +1 -0
  28. package/build/pinned-fetch/consts.local.js +13 -0
  29. package/build/pinned-fetch/consts.local.js.map +1 -0
  30. package/build/pinned-fetch/types.d.ts +16 -0
  31. package/build/pinned-fetch/types.d.ts.map +1 -0
  32. package/build/pinned-fetch/types.js +2 -0
  33. package/build/pinned-fetch/types.js.map +1 -0
  34. package/build/pinned-fetch.d.ts +9 -0
  35. package/build/pinned-fetch.d.ts.map +1 -0
  36. package/build/pinned-fetch.js +82 -0
  37. package/build/pinned-fetch.js.map +1 -0
  38. package/package.json +18 -11
  39. package/src/errors.ts +6 -0
  40. package/src/fetch.ts +14 -0
  41. package/src/http/types.ts +21 -1
  42. package/src/http.ts +7 -4
  43. package/src/index.ts +2 -0
  44. package/src/pinned-fetch/consts.local.ts +15 -0
  45. package/src/pinned-fetch/types.ts +17 -0
  46. package/src/pinned-fetch.ts +93 -0
  47. package/tests/fetch.spec.ts +163 -0
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: file-streams
3
- description: "How to use @owlmeans/file-streams — online files as Node streams: URL downloads under a connect-time SSRF policy, HTTP and SOCKS proxy plugins (a list or a callback by key and URL), the Google Drive resolver, S3 buckets addressed exactly like storage-resource (with prefix listing and removal), folders, buffers and Viable ephemeral workloads (writes and reads); scoped temporary workspaces nested by organization/project/purpose and mirrored to a bucket for restore after a restart; zip/tar/tar.gz/gz with traversal and bomb guards; file-type verification (office, pdf, csv, media, images, source code) from a head or a few range reads. Auto-invoked when downloading a file, moving bytes between HTTP, S3, disk and a workload, keeping uploaded files in a service tmp folder backed by S3, extracting or packing an archive, checking that a file is what it claims, or configuring outbound proxies."
3
+ description: "How to use @owlmeans/file-streams — online files as Node streams: URL downloads under a connect-time SSRF policy (follow or return redirects, any status on request), a pinned WHATWG fetch on the aws-free ./fetch subpath, HTTP and SOCKS proxy plugins (a list or a callback by key and URL), the Google Drive resolver, S3 buckets addressed exactly like storage-resource (with prefix listing and removal), folders, buffers and Viable ephemeral workloads (writes and reads); scoped temporary workspaces nested by organization/project/purpose and mirrored to a bucket for restore after a restart; zip/tar/tar.gz/gz with traversal and bomb guards; file-type verification from a head or a few range reads. Auto-invoked when downloading a file, giving fetch-based code (llm-reader, a renderer) pinned egress, moving bytes between HTTP, S3, disk and a workload, keeping uploaded files in a service tmp folder backed by S3, extracting or packing an archive, checking that a file is what it claims, or configuring outbound proxies."
4
4
  user-invocable: false
5
5
  ---
6
6
  <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
@@ -8,42 +8,38 @@ user-invocable: false
8
8
  # @owlmeans/file-streams
9
9
 
10
10
  **Layer:** Infra
11
- **Install:** `"@owlmeans/file-streams": "^0.2.0-alpha.0"` in `dependencies`
11
+ **Install:** `"@owlmeans/file-streams": "^0.2.0"` in `dependencies`
12
12
 
13
13
  Server-side (Node and Bun). Every byte moves as a Node `Readable` — the shape storage-resource's
14
- `StoredRecord.stream`, the AWS SDK, zlib and the archive codecs share. A web stream, an async
15
- iterable or a `Uint8Array` is accepted where a source is built.
14
+ `StoredRecord.stream`, the AWS SDK, zlib and the archive codecs share. `createStreamSource` takes a
15
+ web stream or an async iterable, `createBufferSource` a `Uint8Array`.
16
16
 
17
17
  ## Key Exports
18
18
 
19
19
  | Export | What it is |
20
20
  |---|---|
21
- | `FileSource` / `FileSink` / `TreeSink` | Where bytes come from (`info`, `open`, `ranges`), where one file goes, where many go |
22
- | `httpClient` / `createHttpClient(opts)` | Requests under the network policy, through proxies, redirects and resolvers; `.source(url)` is a re-openable, Range-capable source |
23
- | `createFileSource`, `createBufferSource`, `createStreamSource` | A file on disk, bytes in memory (a screenshot), a stream opened elsewhere (one-shot) |
24
- | `createFileSink`, `createDirectorySink`, `createBufferSink` | Atomic file write, a confined directory tree, an in-memory collector |
25
- | `createS3Bucket(entry or model, opts)` | A `cfg.storageBuckets` entry as `source`, `sink`, `tree`, `head`, `remove`, `list`, `removePrefix`, `instanceOf` |
26
- | `makeWorkloadSink(writer, { root })` / `makeWorkloadSource(reader, { root })` | A Viable ephemeral workload as a tree, through its `writeBinaryFile` port; its files as sources, through `readBinaryFile` |
27
- | `createWorkspace({ root?, dirMode?, fileMode? })` | A service's temporary files, one confined folder per `FileScope` (`scope`, `scopes`, `prune`) |
28
- | `createBackedWorkspace({ workspace, bucket, prefix? })` | Scopes mirrored to a bucket under the same nesting: `put`, `restore`, `list`, `remove`, `evict`, `drop` |
29
- | `fileScopeHelper`, `FileScope`, `FileScopeSchema`, `FileScopeError` | Re-exported from `@owlmeans/storage-common`: the `<organization>/<project?>/<purpose>` nesting |
30
- | `transferHelper.copy(source, sink, opts)` / `.copyInto(source, tree, opts)` | One file moved under limits, its head verified before the sink sees a byte |
31
- | `fileInspector.probe(source)` / `.inspect(bytes, claims)` / `.enforce(verdict, policy)` | What a file really is, against what it claims — re-exported from `@owlmeans/file-inspector` |
32
- | `archiveHelper.extract / list / pack / gzip / gunzip / detect` | zip, tar, tar.gz and gz, detected from bytes |
33
- | `createProxyRouter`, `createHttpProxyPlugin`, `createSocksProxyPlugin` | Proxy selection and the built-in protocol plugins |
34
- | `createGoogleDriveResolver(opts)` / `googleDriveResolver` | Drive and Docs links to the bytes behind them (on by default in `createHttpClient`) |
35
- | `createNetworkPolicy(opts)` | What may be reached; every connection goes through it |
36
- | `appendFileStreamsService(ctx, alias?, opts?)` | The objects configured from `cfg.fileStreams` and `cfg.storageBuckets` |
37
- | `FileKind`, `ArchiveFormat`, `WORKLOAD_MAX_FILE_BYTES`, `S3_DELETE_BATCH_SIZE`, `WORKSPACE_SCOPE_MARKER` | Enums, ceilings, the reserved marker name |
38
- | `NetworkPolicyError`, `HttpError` (`.status`), `ProxyError`, `LimitError`, `SourceError`, `PathError`, `ArchiveError`, `ResolverError`, `WorkloadError` | Failures, by message code (`file-streams:<area>:<reason>`) |
39
- | `FileMimicryError`, `FileKindError` (storage `FileTypeError`, from file-inspector), `S3TransferError` (storage `StorageApiError`) | Type refusals and S3 failures a storage-resource catch site already handles |
21
+ | `FileSource` / `FileSink` / `TreeSink` | Where bytes come from (`info`, `open`, `ranges`), where one file goes (`write`), where many go (`entry`, `directory`) |
22
+ | `httpClient` / `createHttpClient(opts)` | `request(url, opts)` under the network policy, through proxies, redirects and resolvers; `.source(url)` is re-openable and Range-capable |
23
+ | `createPinnedFetch(opts)` (also `@owlmeans/file-streams/fetch`) | A WHATWG `fetch` over the client; the subpath loads no AWS SDK |
24
+ | `createFileSource`, `createBufferSource`, `createStreamSource` | A file on disk, bytes in memory, a stream opened elsewhere (one-shot) |
25
+ | `createFileSink`, `createDirectorySink`, `createBufferSink` | Atomic file write, a confined directory tree, an in-memory collector (`bytes()`) |
26
+ | `createS3Bucket(entry or model, { partSize?, concurrency? })` | A `cfg.storageBuckets` entry as `source`, `sink`, `tree`, `head`, `remove`, `list`, `removePrefix`, `keyOf`, `urlOf`, `instanceOf`, `close` |
27
+ | `transferHelper` / `createTransferHelper({ inspector?, limits? })` | `.copy(source, sink, opts)` / `.copyInto(source, tree, opts)` — one file under limits, its head verified first |
28
+ | `archiveHelper` / `createArchiveHelper({ limits?, inspector? })` | `extract`, `list`, `pack`, `gzip`, `gunzip`, `detect` — the [archives](parts/archives.md) part |
29
+ | `createWorkspace`, `createBackedWorkspace`, `fileScopeHelper`, `FileScope` | Scoped tmp folders, mirrored to a bucket — the [workspaces](parts/workspaces.md) part |
30
+ | `makeWorkloadSink`, `makeWorkloadSource` | A Viable ephemeral workload — the [workload](parts/workload.md) part |
31
+ | `createNetworkPolicy`, `createProxyRouter`, `createHttpProxyPlugin`, `createSocksProxyPlugin`, `parseProxyEndpoint`, `createGoogleDriveResolver` / `googleDriveResolver` | What may be reached, proxies, Drive links — the [network](parts/network.md) part |
32
+ | `fileInspector`, `createFileInspector`, `FileKind`, the inspector tables and verdict types | Re-exported from `@owlmeans/file-inspector` |
33
+ | `appendFileStreamsService(ctx, alias?, opts?)` / `createFileStreamsService`, `FILE_STREAMS_SERVICE` | The objects configured from `cfg.fileStreams` and `cfg.storageBuckets` |
34
+ | `FileStreamsError` → `NetworkPolicyError` (`.refused === true`), `HttpError` (`.status`), `ProxyError`, `LimitError`, `SourceError`, `PathError`, `ArchiveError`, `ResolverError`, `WorkloadError` | Failures, by message code (`file-streams:<area>:<reason>`) |
35
+ | `FileMimicryError`, `FileKindError` (storage `FileTypeError`), `S3TransferError` (storage `StorageApiError`, `s3:<op>`) | Refusals and S3 failures a storage-resource catch site already handles |
40
36
 
41
37
  ## Usage
42
38
 
43
39
  ```typescript
44
40
  import {
45
41
  archiveHelper, createBufferSource, createDirectorySink, createHttpClient, createS3Bucket, FileKind,
46
- fileInspector, httpClient, makeWorkloadSink, transferHelper,
42
+ fileInspector, httpClient, transferHelper,
47
43
  } from '@owlmeans/file-streams'
48
44
 
49
45
  // An image from the internet into a folder, refused unless it is an image
@@ -51,133 +47,116 @@ await transferHelper.copyInto(httpClient.source(url), createDirectorySink(workDi
51
47
 
52
48
  // A screenshot (a Buffer) into the bucket a storage entry names — same key and URL storage-resource uses
53
49
  const bucket = createS3Bucket(cfg.storageBuckets.media)
54
- const { sink } = await transferHelper.copy(createBufferSource(png, { name: 'shot.png' }), bucket.sink(`${entityId}/shots/${id}.png`))
50
+ const { sink } = await transferHelper.copy(createBufferSource(png, { name: 'shot.png' }), bucket.sink(`shots/${id}.png`))
55
51
  sink.url // === storage-resource's URL for that prefix
56
52
  bucket.instanceOf(sink, 'original') // a storage-common StoredFileInstance
57
53
 
58
- // A media archive into a folder or a workload; entries that cannot be written are skipped
59
- await archiveHelper.extract(httpClient.source(archiveUrl), createDirectorySink(dir), { onEntryError: 'skip' })
54
+ // An archive into a folder; entries are verified only when `verify` is given
55
+ await archiveHelper.extract(httpClient.source(archiveUrl), createDirectorySink(dir), { verify: {}, onEntryError: 'skip' })
60
56
 
61
57
  // What a remote file is, without downloading it (a few range reads, at most 1 MiB)
62
58
  const verdict = await fileInspector.probe(httpClient.source(url))
63
59
 
64
60
  // Proxies: a list (round-robin, random, failover, by-key) or a callback by key and URL
65
- const pooled = createHttpClient({ proxies: { proxies: ['socks5h://u:p@egress-1:1080', 'http://egress-2:3128'], strategy: 'by-key' } })
61
+ const pooled = createHttpClient({ proxies: { proxies: ['socks5://u:p@egress-1:1080', 'http://egress-2:3128'], strategy: 'by-key' } })
66
62
  const vaulted = createHttpClient({ proxies: { proxies: async ({ key, url }) => vault.proxyFor(key, url.hostname) } })
67
- await pooled.request(url, { proxyKey: entityId })
63
+ await pooled.request(url, { proxyKey: tenantKey })
68
64
  ```
69
65
 
70
66
  `copy` reads the head of the very stream it moves (no second download), verifies it, and replays
71
- it into the sink. Verification is on by default: a kind mismatch is refused, dangerous content (an
72
- executable under another claim, a script under a passive one, a PDF with active content in the head
73
- sample) is always refused, `verify.refuse` refuses kinds outright, an inconclusive verdict passes
74
- unless `refuseInconclusive`; `verify: false` turns it off.
67
+ it into the sink. Verification is on by default (`verify: {}`): a kind mismatch is refused,
68
+ dangerous content (an executable under another claim, a script under a passive one, a PDF with
69
+ active content in the head sample) is always refused, `verify.refuse` refuses kinds outright, an
70
+ inconclusive verdict passes unless `refuseInconclusive`; `verify: false` turns it off. A declared
71
+ size above `limits.maxBytes` is refused before the first byte, counted bytes after it. `info`
72
+ overrides the source's claims; `copyInto` writes under the source's sanitized name, or `download`.
75
73
 
76
- ## The Viable workload
74
+ ## The pinned fetch
77
75
 
78
- A workload accepts bytes only as the publisher's signed `writeBinaryFile { filePath, base64 }`
79
- command; the agent reaches it through viable-agent's `FileHelper`, which satisfies
80
- `WorkloadWriter` as is (pass the helper, never a destructured method):
76
+ `createPinnedFetch(opts)` from `@owlmeans/file-streams/fetch` (no AWS SDK, storage-resource or S3
77
+ module in its graph) is `fetch` over the client — any status a `Response`, `redirect: 'manual'` the
78
+ 3xx itself, GET/HEAD only, bodies streamed, policy refusals `NetworkPolicyError` (`refused === true`).
79
+ Rules: the [network](parts/network.md) part.
81
80
 
82
- ```typescript
83
- const media = makeWorkloadSink(fileHelper, { root: 'sources/web/public/media' })
84
- await transferHelper.copy(httpClient.source(imageUrl), media.entry('hero.png'), { verify: { allow: [FileKind.Image] } })
85
- await archiveHelper.extract(httpClient.source(zipUrl), media, { onEntryError: 'skip' })
86
- ```
87
-
88
- Each file is one atomic write of at most 12 MiB (`WORKLOAD_MAX_FILE_BYTES`, the publisher's 16 MiB
89
- base64 limit); `maxFileBytes` lowers it, and an oversized file is refused before any request
90
- (`WorkloadError('too-large')`). Larger files need a product route (chunked writes, or the workload
91
- pulling a URL itself — it has public egress and can run this package).
92
-
93
- Reads go the other way through `makeWorkloadSource(reader, { root })` (`readBinaryFile`, same
94
- ceiling): the [workspaces](parts/workspaces.md) part.
95
-
96
- ## Workspaces
81
+ ## Workspaces and workloads
97
82
 
98
- `createWorkspace().scope({ organization, project?, purpose })` is the folder
99
- `<root>/<organization>/<project?>/<purpose>` (segments by `fileScopeHelper`); a backed workspace
100
- mirrors every scope to `<basePrefix>/<prefix?>/<scope path>/<path>` in a bucket, and the bucket is the
101
- authority — a pod restart loses the tmp folder, `restore()` brings the scope back. Paths are confined
102
- (no `..`, no absolute paths, no existing symlink from the workspace root, the marker name reserved);
103
- a relative `root` lives under `os.tmpdir()`. Full rules, overlap and prune semantics, prefix
104
- operations: the [workspaces](parts/workspaces.md) part.
83
+ `createWorkspace().scope({ organization, project?, purpose })` is the confined folder
84
+ `<root>/<organization>/<project?>/<purpose>`; a backed workspace mirrors it to the same nesting in a
85
+ bucket, which `restore()` reads back after a pod restart — the [workspaces](parts/workspaces.md)
86
+ part. A Viable workload takes and gives files of at most 12 MiB — the [workload](parts/workload.md) part.
105
87
 
106
88
  ## Detection
107
89
 
108
- Verification is `@owlmeans/file-inspector` (its `file-inspector` skill: verdicts, policies, kinds,
109
- active PDF, probes). This package re-exports its inspector, kinds, tables, verdict types and the two
110
- refusals unchanged, and every source here (`FileSource`) is a file-inspector `ProbeSource` as is: a
111
- probe seeks through `ranges()` when the origin has them and reads the head of `open()` otherwise. The
112
- S3 sink takes the Content-Type from the verdict and writes `attachment` for active svg/html.
113
-
114
- ## Archives
115
-
116
- zip is read from its central directory (yauzl) — through range reads when the source has them, else
117
- from a private temporary spool — and written as a stream (yazl); tar and tar.gz stream through
118
- tar-stream (an entry is drained, never destroyed: destroying one destroys the archive); gz is zlib.
119
- The format comes from the bytes. Refused: absolute paths, `..`, drive letters, control and bidi
120
- characters, overlong or deep paths, duplicate and case-fold-twin entries, encrypted entries,
121
- overlapping zip entries, an expansion ratio above 200 (past 1 MiB), more entries or bytes than the
122
- limits. Links and devices are skipped (or refused with `links: 'refuse'`), never created. Nested
123
- archives stay files unless `limits.maxDepth` > 1. Each entry commits atomically; a refusal leaves the
124
- entries already written.
90
+ Verification is `@owlmeans/file-inspector` (its `file-inspector` skill), re-exported unchanged
91
+ except `fileNameHelper`, `TEXT_CARRIED_KINDS` and the `ProbeSource` types — import those from it.
92
+ Every `FileSource` is a `ProbeSource` as is: a probe seeks through `ranges()` when the origin has
93
+ them and reads the head of `open()` otherwise. The S3 sink takes the Content-Type from the verdict
94
+ and writes `attachment` for active svg/html (unless `allowActiveInline`).
125
95
 
126
96
  ## Defaults
127
97
 
128
98
  | Area | Default |
129
99
  |---|---|
130
100
  | Transfer | 512 MiB · 10 min total · 30 s idle |
131
- | HTTP | connect 10 s · headers 30 s · idle 30 s · 5 redirects · `Accept-Encoding: identity` |
101
+ | HTTP | connect 10 s · headers 30 s · idle 30 s · 5 redirects · `Accept-Encoding: identity` · User-Agent `owlmeans-file-streams` |
102
+ | Proxies | round-robin · 1 retry on another proxy · 30 s cooldown |
132
103
  | Probe | 1 MiB budget · 64 KiB head sample |
133
- | Archives | 10 000 entries · 512 MiB per entry · 2 GiB total · ratio 200 · depth 1 |
134
104
  | S3 | 8 MiB parts (≥ 5 MiB), 2 in flight; one PutObject when the body fits a part; aborted on failure |
135
- | Workload | 12 MiB per file (`WORKLOAD_MAX_FILE_BYTES`), one retry, writes and reads |
105
+ | Workload | 12 MiB per file (`WORKLOAD_MAX_FILE_BYTES`), one retry |
136
106
  | Workspace | `<os.tmpdir()>/owlmeans-workspace`, folders `0o700`, files `0o600` |
137
- | S3 prefixes | ListObjectsV2 all pages; DeleteObjects ≤ 1000 keys, checksummed by Content-MD5 |
138
107
 
139
108
  ## Service
140
109
 
141
110
  `appendFileStreamsService(ctx)` registers `file-streams`: `http()`, `transfer()`, `inspector()`,
142
111
  `archives()`, `bucket(configKey)`, `workspace()` and `backed(configKey, { prefix? })` (one object per
143
- key and prefix, over the configured workspace and transfer helper), built lazily from
144
- `cfg.fileStreams` (`limits`, `network`, `proxies: [{ url, username?, password? }]`, `proxyStrategy`,
145
- `userAgent`, `workspace: { root?, dirMode?, fileMode? }`) and `cfg.storageBuckets`. Proxy entries are
146
- objects so a server context resolves their secret-file leaves; callbacks and plugins come through the
147
- options argument. A server context reads EVERY string leaf starting with `/` as a file, so
148
- `workspace.root` is given relative to `os.tmpdir()` (or as a file holding an absolute path).
149
-
150
- ## External docs
151
-
152
- - https://github.com/thejoshwolfe/yauzl — central-directory zip reading, `fromRandomAccessReader` (override `read` and `_readStreamForRange`), `validateEntrySizes`, automatic `validateFileName` (v3)
153
- - https://github.com/mafintosh/tar-stream — v3 ships streamx typings; destroying an entry destroys the extract, so entries are drained
112
+ key and prefix), built lazily from `cfg.fileStreams` (`limits` — transfers only, `network`,
113
+ `proxies: [{ url, username?, password?, id? }]`, `proxyStrategy`, `userAgent`, `workspace`) and
114
+ `cfg.storageBuckets` (an unknown key → `FilePropertyError('storage-config:<key>')`). Proxy entries
115
+ are objects so a server context resolves their secret-file leaves; callbacks, plugins, resolvers, a
116
+ DNS `resolver`, `bucketOptions`, `inspector` and `archiveLimits` come through the options argument.
117
+ A server context reads EVERY string leaf starting with `/` as a file, so `workspace.root` is given
118
+ relative to `os.tmpdir()`. Timeouts, redirects, headers and CAs need your own `createHttpClient`.
119
+
120
+ ## Pitfalls
121
+
122
+ - Loopback and in-cluster origins are refused by default: give `network: { allowCidrs }` (tests:
123
+ `'127.0.0.1/32'`) or `allowPrivate`.
124
+ - `createStreamSource` is one-shot: a probe or `archiveHelper.detect` consumes it — verify through
125
+ `copy`'s `verify` instead.
126
+ - `archiveHelper.extract` verifies nothing unless `verify` is passed; `transferHelper` verifies by
127
+ default.
128
+ - Passing `resolvers` replaces the default `[googleDriveResolver]`; keep it when you add your own.
129
+ - A `socks5h`/`socks4a` endpoint resolves names at the proxy, so its connect is not pinned.
130
+ - storage-resource's default ACL is `public-read`: private files need a bucket entry with `acl: false`.
154
131
 
155
132
  ## Tests
156
133
 
157
- Category C. Offline specs run against local stand-ins (`tests/harness/`: an origin with Range,
158
- redirects and stalls, a CONNECT/forward proxy, a SOCKS5 server, a fake Drive, a workload
159
- `/command/` receiver answering `writeBinaryFile` and `readBinaryFile` with the real limits) and
160
- code-built fixtures; `workspace.spec.ts` runs offline over a temporary root; `s3.spec.ts` and
161
- `backed.spec.ts` need the S3 gate (path-style for a non-AWS endpoint, objects under a random base
162
- prefix, deleted after); `live.spec.ts` needs `FILE_STREAMS_LIVE=1` (and `FILE_STREAMS_LIVE_DRIVE_URL`).
134
+ `bun test ./tests` in `@owlmeans/file-streams` (category C; build first). Offline specs use local
135
+ stand-ins; S3 and live specs are gated — the [extending](parts/extending.md) part.
163
136
 
164
137
  ## Depends On
165
138
 
166
139
  - `@owlmeans/file-inspector` (detection, re-exported), `@owlmeans/storage-resource` (the bucket
167
- model), `@owlmeans/storage-common` (errors, instances), `@owlmeans/context`, `@owlmeans/error`,
168
- `@owlmeans/log`
140
+ model), `@owlmeans/storage-common` (errors, instances, scopes), `@owlmeans/context`,
141
+ `@owlmeans/error`, `@owlmeans/log`
169
142
  - `@aws-sdk/client-s3`, `socks`, `tar-stream`, `yauzl`, `yazl`
170
143
 
171
144
  ## Related
172
145
 
173
- - [[file-inspector]] — what a file is: verdicts, policies, kinds, active PDF, safe names
174
- - [[storage-resource]] — the upload resource and the bucket model
175
- - [[storage-common]] — stored-file types and errors, `FileScope` and `fileScopeHelper`
176
- - [[logging]] — scopes `file-streams:<area>`; URLs are logged redacted, contents never
146
+ - [[file-inspector]] (`@owlmeans/file-inspector`) — verdicts, policies, kinds, active PDF, safe names
147
+ - [[storage-resource]] (`@owlmeans/storage-resource`) — the upload resource and the bucket model
148
+ - [[storage-common]] (`@owlmeans/storage-common`) — stored-file types and errors, `FileScope`
149
+ - [[server-upload]] (`@owlmeans/server-upload`) — multi-file uploads over these objects
150
+ - [[upload]] (`@owlmeans/upload`) — the upload contract; its path rules match this package's
151
+ - [[llm-reader]] (`@owlmeans/llm-reader`) — takes the pinned fetch as its transport
152
+ - [[logging]] — scopes `file-streams:<area>`; URLs logged redacted, contents and proxy secrets never
177
153
 
178
154
  ## Parts
179
155
 
180
156
  | Part | Load it when |
181
157
  |---|---|
182
- | [network](parts/network.md) | changing what may be reached, routing through proxies, or downloading Google Drive and Docs links |
183
- | [workspaces](parts/workspaces.md) | keeping files in a service tmp folder by scope, mirroring them to a bucket, restoring after a restart, listing or deleting a bucket prefix, or reading files back from a workload |
158
+ | [network](parts/network.md) | changing what may be reached, routing through proxies, returning redirects, handing a pinned `fetch` to fetch-based code, or downloading Google Drive and Docs links |
159
+ | [archives](parts/archives.md) | extracting, listing, packing or (de)compressing zip, tar, tar.gz or gz, or tuning archive limits |
160
+ | [workspaces](parts/workspaces.md) | keeping files in a service tmp folder by scope, mirroring them to a bucket, restoring after a restart, or listing or deleting a bucket prefix |
161
+ | [workload](parts/workload.md) | writing files into a Viable ephemeral workload or reading them back |
162
+ | [extending](parts/extending.md) | writing your own source, sink, tree sink, proxy plugin or URL resolver, changing this package, or running and adding its tests |
@@ -0,0 +1,80 @@
1
+ ---
2
+ name: archives
3
+ description: Reading and writing zip, tar, tar.gz and gz with archiveHelper — format detection from bytes, listing, safe extraction into a tree (traversal, link, duplicate, bomb and overlap guards, per-entry verification, skip or fail), nested archives, packing and gzip/gunzip sources, and the archive limits.
4
+ metadata:
5
+ purposes: data
6
+ mentions: archiveHelper, createArchiveHelper, ArchiveFormat, ArchiveExtractOptions, ArchiveLimits, DEFAULT_ARCHIVE_LIMITS, ExtractResult, PackEntry, onEntryError, maxDepth, ArchiveError
7
+ ---
8
+ <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
9
+
10
+ # @owlmeans/file-streams — Archives
11
+
12
+ ```typescript
13
+ import { ArchiveFormat, archiveHelper, createDirectorySink, createFileSource, FileKind, httpClient } from '@owlmeans/file-streams'
14
+
15
+ const result = await archiveHelper.extract(httpClient.source(zipUrl), createDirectorySink(dir), {
16
+ verify: { refuse: [FileKind.Executable] }, onEntryError: 'skip', strip: 1, limits: { maxEntries: 500 },
17
+ })
18
+ result.extracted // [{ entry, path, result }, …]
19
+ result.skipped // [{ entry, reason }, …] — 'link', 'unsupported-type', 'stripped', 'filtered', or the error message
20
+
21
+ await archiveHelper.list(createFileSource(path)) // ArchiveEntryInfo[] — nothing written
22
+ await archiveHelper.detect(source) // ArchiveFormat | null
23
+ const zip = archiveHelper.pack(ArchiveFormat.Zip, [{ path: 'a.txt', source: bytes }, { path: 'b.pdf', source: fileSource }], { name: 'out.zip' })
24
+ const gz = archiveHelper.gzip(source) // a one-shot source of the compressed bytes
25
+ const plain = archiveHelper.gunzip(source) // the decompressed bytes, under the ratio and size limits
26
+ ```
27
+
28
+ ## Reading
29
+
30
+ zip is read from its central directory (yauzl) — through range reads when the source has them, else
31
+ from a private temporary spool (at most `maxTotalBytes`) — and written as a stream (yazl); tar and
32
+ tar.gz stream through tar-stream (an entry is drained, never destroyed: destroying one destroys the
33
+ archive); gz is zlib, its single entry named from the gzip header or the source's name without
34
+ `.gz`. The format comes from the bytes (`format` skips detection), never from names.
35
+
36
+ ## Extraction rules
37
+
38
+ - **Refused** (the extraction stops): absolute paths, `..`, drive letters, control and bidi
39
+ characters, overlong or deep paths (`maxPathBytes`, `maxPathSegments`), duplicate and
40
+ case-fold-twin entries, encrypted entries, compression methods other than store and deflate,
41
+ overlapping zip entries, sizes that lie (yauzl `validateEntrySizes`), a declared or measured
42
+ expansion ratio above `maxRatio` once past 1 MiB (`RATIO_FLOOR_BYTES`), more entries or bytes than
43
+ the limits.
44
+ - **Skipped** (reported in `skipped`): links (`links: 'refuse'` throws `ArchiveError('link')`
45
+ instead), devices and other special entries, entries `strip` empties, entries `filter` rejects.
46
+ Neither links nor devices are ever created.
47
+ - **Verification** runs only when `verify` is given: each entry's head is checked against the
48
+ entry's own name before it is written.
49
+ - **`onEntryError: 'skip'`** skips an entry whose write or verification fails (a too-large workload
50
+ file, a mimicry refusal); `LimitError`, `ArchiveError`, `PathError` and an abort always stop.
51
+ - **Nested archives** (`zip`, `tar`, `tar.gz`, `tgz`, `gz` names) stay files unless
52
+ `limits.maxDepth` > 1; a followed one is spooled and extracted under its name without the extension.
53
+ - Each entry commits atomically; a refusal leaves the entries already written. `onEntry(entry,
54
+ result)` reports each written file.
55
+
56
+ ## Packing
57
+
58
+ `pack(format, entries, { name?, level?, maxSpoolBytes? })` returns a one-shot `FileSource`. Entry
59
+ paths pass the same path rules; a `source` is a `FileSource` or a `Uint8Array`. A tar needs sizes up
60
+ front, so a source without one is spooled (up to 512 MiB by default). gz takes exactly one entry
61
+ (`ArchiveError('gzip-entries')`).
62
+
63
+ ## Limits
64
+
65
+ | Limit | Default |
66
+ |---|---|
67
+ | `maxEntries` | 10 000 |
68
+ | `maxEntryBytes` | 512 MiB |
69
+ | `maxTotalBytes` | 2 GiB |
70
+ | `maxRatio` | 200 |
71
+ | `maxDepth` | 1 |
72
+ | `maxPathBytes` / `maxPathSegments` | 1024 / 64 |
73
+
74
+ `createArchiveHelper({ limits, inspector })` sets the defaults of one helper; each call's `limits`
75
+ override them. The service takes them as `archiveLimits` in its options.
76
+
77
+ ## External docs
78
+
79
+ - https://github.com/thejoshwolfe/yauzl — central-directory zip reading, `fromRandomAccessReader` (override `read` and `_readStreamForRange`), `validateEntrySizes`, automatic `validateFileName` (v3)
80
+ - https://github.com/mafintosh/tar-stream — v3 ships streamx typings; destroying an entry destroys the extract, so entries are drained
@@ -0,0 +1,117 @@
1
+ ---
2
+ name: extending
3
+ description: Writing your own FileSource, FileSink, TreeSink, proxy plugin or URL resolver and the contract each must keep; the invariants of the package (pinned connects, the aws-free ./fetch graph, counted limits, drained tar entries, confined paths, redacted logs); and the package's specs, stand-ins and gates.
4
+ metadata:
5
+ purposes: data, test
6
+ mentions: FileSource, FileSink, TreeSink, RangeReader, SinkInput, SinkResult, ProxyPlugin, ProxyConnection, TunnelTarget, UrlResolver, ResolverStep, FILE_STREAMS_LIVE, S3_TEST_BUCKET
7
+ ---
8
+ <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
9
+
10
+ # @owlmeans/file-streams — Extending and testing the package
11
+
12
+ ## A source
13
+
14
+ A `FileSource` is `{ kind, info, open, ranges }`; it is also a file-inspector `ProbeSource`, so
15
+ probes, transfers and archive extraction take it as it is.
16
+
17
+ - `info({ signal? })` — metadata without the body, cached after the first success: `origin`
18
+ redacted (no credentials, no query), `name` a safe base name, `size` and `mime` as declared
19
+ (lower-cased, no parameters), `etag` a strong validator when the origin has one.
20
+ - `open({ range?, signal? })` — a fresh Node `Readable` from byte 0, or the honoured span with
21
+ `range: { start, end, total }`. A one-shot source refuses a second open with
22
+ `SourceError('consumed')`; a missing object is `SourceError('not-found')`.
23
+ - `ranges({ signal? })` — a `RangeReader` (`size`, `read(offset, length, signal?)`,
24
+ `stream(start, end, signal?)` with `end` exclusive, `close()`) over ONE immutable version: pin it
25
+ with the validator and throw `SourceError('changed')` when the origin moved. Answer `null`
26
+ without random access — probes then read the head of `open()` and a zip is spooled.
27
+ - An abort rejects with the signal's reason.
28
+
29
+ ## A sink
30
+
31
+ - `FileSink.write({ stream, info, verdict? }, { maxBytes?, signal? })` consumes the whole stream,
32
+ counts it and refuses past `maxBytes` (`LimitError('max-bytes')`), commits atomically, and on
33
+ failure destroys the input and leaves nothing behind. It answers `SinkResult { kind, location,
34
+ bytes, url?, etag?, mime? }`, with `mime` from `verdict?.detected?.mime` before `info.mime`.
35
+ - `TreeSink.entry(path)` normalizes and confines the path (`PathError`: `absolute`, `traversal`,
36
+ `invalid-char`, `too-long`, `too-deep`, `empty`), refuses case-fold twins where the store folds
37
+ case, and returns the entry's `FileSink`; `directory(path)` creates an empty directory or is a
38
+ no-op where the store has none.
39
+
40
+ ## A proxy plugin
41
+
42
+ ```typescript
43
+ import { createHttpClient, ProxyError } from '@owlmeans/file-streams'
44
+ import type { ProxyPlugin } from '@owlmeans/file-streams'
45
+
46
+ const bastionPlugin: ProxyPlugin = {
47
+ types: ['bastion'],
48
+ remoteDns: () => false,
49
+ connect: async (endpoint, target, { signal, timeoutMs }) => {
50
+ const socket = await openTunnel(endpoint, target.host, target.port, { signal, timeoutMs }) // your transport
51
+ if (socket == null) {
52
+ throw new ProxyError('connect')
53
+ }
54
+ return { socket, mode: 'tunnel' }
55
+ },
56
+ }
57
+ const client = createHttpClient({ proxies: { proxies: ['bastion://jump.example.com:2222'], plugins: [bastionPlugin] } })
58
+ ```
59
+
60
+ - Connect to `target.host` — the address the policy checked; `target.hostname` is for SNI and
61
+ `Host` only. `remoteDns(endpoint) === true` hands the name to the proxy instead and gives up the
62
+ pin (the name is still pre-checked).
63
+ - Never terminate TLS: return the raw tunnel, and the client layers TLS over it for https targets.
64
+ `mode: 'forward'` is for plain-http absolute-form proxies only, with their auth in `headers`.
65
+ - Honour `timeoutMs` and `signal`; fail with `ProxyError('connect' | 'handshake' | 'auth-failed' |
66
+ 'refused:<code>')` — the router cools the endpoint down and tries another. A custom type needs an
67
+ explicit port in its URL. Never log credentials.
68
+ - Register through `createProxyRouter({ plugins })`, `createHttpClient({ proxies: { plugins } })` or
69
+ the service option `proxyPlugins`; custom plugins are tried before the built-in ones.
70
+
71
+ ## A URL resolver
72
+
73
+ `UrlResolver { name, maxSteps, matches(url), rewrite(url), inspect(head, readBody), owns(url) }`:
74
+ `rewrite` turns a page link into the URL that serves the bytes (or throws
75
+ `ResolverError('<name>:<reason>')`); `inspect` sees each final 2xx GET answer before its body
76
+ reaches the caller, reads at most `readBody(maxBytes)` (replayed on accept) and answers
77
+ `{ action: 'accept' }`, `{ action: 'follow', url, headers?: { cookie } }` or
78
+ `{ action: 'fail', reason }`. A follow must stay on URLs the resolver `owns` (else
79
+ `foreign-follow`) and within `maxSteps`; its cookie is sent only there. Register with
80
+ `createHttpClient({ resolvers: [googleDriveResolver, mine] })` or the service option `resolvers`.
81
+
82
+ ## Invariants
83
+
84
+ - Every connection goes through the client's connector: the policy resolves and checks, the socket
85
+ goes to that address (or the address goes into the tunnel). Network code never calls `fetch`,
86
+ `http.get(url)` or a DNS lookup of its own.
87
+ - The `./fetch` graph imports only `@owlmeans/error`, `@owlmeans/file-inspector`, `@owlmeans/log`,
88
+ `@owlmeans/storage-common`, `socks` and `node:*`; `fetch.spec.ts` fails on any S3,
89
+ storage-resource, workspace, transfer or archive module in it.
90
+ - Limits count the bytes that flowed; a declared size only refuses early.
91
+ - Tar entries are drained, never destroyed.
92
+ - Every written path passes the path rules, and nothing writes through an existing symlink.
93
+ - Logs carry redacted URLs, proxy ids and sizes — never contents, credentials or query strings.
94
+ - Failures are the package's error classes with `file-streams:<area>:<reason>` codes; S3 failures
95
+ are `S3TransferError`, a storage `StorageApiError`.
96
+
97
+ ## Tests
98
+
99
+ `bun test ./tests` in `@owlmeans/file-streams`; suites are titled `@owlmeans/file-streams — <area>`.
100
+ Build first: `fetch.spec.ts` walks the built `./fetch` graph.
101
+
102
+ - Offline specs run against local stand-ins (`tests/harness/`: an origin with Range, redirects and
103
+ stalls, a CONNECT/forward proxy, a SOCKS5 server, a fake Drive, a workload `/command/` receiver
104
+ answering `writeBinaryFile` and `readBinaryFile` with the real limits) and code-built fixtures.
105
+ `tests/context.ts` holds `LOCAL_NETWORK` (`allowCidrs: ['127.0.0.1/32']`), `localClient()` and
106
+ `boot()` — a real server context with the service registered. `workspace.spec.ts` runs over a
107
+ temporary root.
108
+ - `s3.spec.ts` and `backed.spec.ts` need the S3 gate (`S3_ENDPOINT`, `S3_TEST_BUCKET`, `S3_KEY`,
109
+ `S3_SECRET`, `S3_REGION`; `gateHelper.s3Gate()` from `@owlmeans/test-integration`): path-style for
110
+ a non-AWS endpoint, `acl: false`, objects under a random base prefix, deleted after. Empty
111
+ variables print a skip.
112
+ - `live.spec.ts` needs `FILE_STREAMS_LIVE=1` (public HTTPS), and `FILE_STREAMS_LIVE_DRIVE_URL` for a
113
+ public Drive file.
114
+ - A new source is covered for info, whole and ranged opens, random access and a changed origin; a
115
+ sink for atomic commit, `maxBytes` and a failure that leaves nothing; a proxy plugin by a stand-in
116
+ that proves the checked address reaches the tunnel and no secret reaches the log; a resolver by a
117
+ fake origin.
@@ -1,44 +1,99 @@
1
1
  ---
2
2
  name: network
3
- description: What the HTTP client may reach — the connect-time network policy, redirects and sensitive headers, the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.
3
+ description: What the HTTP client may reach — the connect-time network policy, redirects (followed or returned) and sensitive headers, the pinned WHATWG fetch on the ./fetch subpath, the proxy router and the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.
4
4
  metadata:
5
5
  purposes: config, ops
6
- mentions: createNetworkPolicy, allowCidrs, allowPrivate, createProxyRouter, createHttpProxyPlugin, createSocksProxyPlugin, trustProxyDns, createGoogleDriveResolver, googleDriveResolver, ResolverError, NetworkPolicyError
6
+ mentions: createPinnedFetch, PinnedFetch, PinnedFetchOptions, HttpRedirectMode, anyStatus, createNetworkPolicy, allowCidrs, allowPrivate, createProxyRouter, createHttpProxyPlugin, createSocksProxyPlugin, parseProxyEndpoint, trustProxyDns, createGoogleDriveResolver, googleDriveResolver, ResolverError, NetworkPolicyError
7
7
  ---
8
8
  <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
9
9
 
10
10
  # @owlmeans/file-streams — Network policy, proxies and Google Drive
11
11
 
12
- ## Network policy and proxies
12
+ ## Network policy
13
13
 
14
- - Loopback, private, link-local, CGNAT, multicast, reserved and documentation ranges are refused,
15
- and the IPv4 inside a mapped, NAT64, 6to4 or Teredo address is checked too; internal names
16
- (`localhost`, `.local`, `.internal`, `.svc`, `.cluster.local`, `.home.arpa`) are refused before DNS.
17
- - The check is pinned to the connect: a direct socket goes to the address just resolved and
18
- checked, a proxy tunnel carries that address (TLS keeps the host name for SNI and certificates).
19
- Every redirect hop is checked again; https → http is refused; `Authorization`, `Cookie`,
20
- `Proxy-Authorization` and `sensitiveHeaders` never cross origins.
14
+ - Before any DNS, `assertUrl` checks the scheme (`schemes`, default `https:`/`http:`), credentials
15
+ in the URL (`allowUrlCredentials`), `allowHosts`/`denyHosts` (`x.com`, `*.x.com`), `ports` and an
16
+ https → http hop (`allowDowngrade`).
17
+ - Loopback, private, link-local, CGNAT, benchmarking, multicast, reserved and documentation ranges
18
+ are refused, and the IPv4 inside a mapped, NAT64, 6to4, Teredo or IPv4-compatible address is
19
+ checked too; internal names (`localhost`, `.localhost`, `.local`, `.internal`, `.svc`,
20
+ `.cluster.local`, `.home.arpa`, `.lan`, `.intranet`) are refused before DNS unless `allowPrivate`
21
+ or an `allowHosts` pattern names them. Every resolved address must pass — one refused address
22
+ refuses the name (`blocked-address`). `allowCidrs` wins over `denyCidrs`, which wins over
23
+ `allowPrivate`. A custom `resolver` (DoH, split horizon, a test seam) must answer every address.
24
+ - The check is pinned to the connect: a direct socket goes to an address just resolved and checked
25
+ (each in turn until one connects), a proxy tunnel carries that address (TLS keeps the host name
26
+ for SNI and certificates, TLS 1.2+, extra `ca` only — verification is never off). Every redirect
27
+ hop is checked again; https → http is refused; `Authorization`, `Cookie`, `Proxy-Authorization`
28
+ and `sensitiveHeaders` never cross origins.
29
+ - `redirect: 'manual'` returns a 3xx with its `Location` (and body) instead of following it — for a
30
+ caller that checks every hop itself; a resolver's own flow (Drive's virus-scan page and its hops)
31
+ still follows inside the client under the policy, because it carries its cookie across them.
32
+ `anyStatus: true` returns every status instead of `HttpError('status:<code>')`; a forward proxy's
33
+ 407 still throws `ProxyError('auth-failed')`. `HttpResponse.decoded` says the body was
34
+ transport-decoded and its `Content-Length` no longer applies; a gzip file is never decoded.
21
35
  - A local stand-in or an in-cluster origin needs `network: { allowCidrs: [...] }` (or
22
36
  `allowPrivate`). S3 traffic through the SDK is not policed — its endpoint is operator config.
23
- - Built-in plugins: `http`/`https` endpoints (CONNECT for https targets, absolute-form forward for
24
- plain http — which a proxy resolves itself, so pinning holds only through CONNECT:
25
- `createHttpProxyPlugin({ httpTargets: 'connect' })`), and `socks4`/`socks4a`/`socks5`/`socks5h`.
26
- Remote-DNS endpoints (`socks5h`, `socks4a`) still pre-check the name locally and are refused when
27
- it does not resolve unless `trustProxyDns`.
28
37
  - Bun's own `fetch({ proxy })` speaks only http/https proxies and undici dispatchers are unreliable
29
38
  under Bun, so the client is `node:http(s)` over a socket it connects itself — Bun 1.4.2 honours
30
39
  the `createConnection` option, an Agent's `createConnection`, the `lookup` option and TLS over an
31
40
  existing socket.
32
41
 
42
+ ## Proxies
43
+
44
+ - `createProxyRouter({ proxies, credentials?, strategy?, bypass?, plugins?, retries?, cooldownMs? })`
45
+ — or the same object as `createHttpClient({ proxies })`. `proxies` is a list of endpoints or URLs
46
+ (`parseProxyEndpoint` reads `socks5h://user:pass@host:1080`; default ports 8080/443/1080) or a
47
+ provider `({ key, url }) => endpoint | url | null` (null goes direct). `credentials(endpoint,
48
+ request)` fills an endpoint that carries none. A request's `proxyKey` is the key.
49
+ - Strategies: `round-robin` (default), `random`, `failover` (the first warm one), `by-key` (a key
50
+ hashes to one proxy; without a key, round-robin). A failed connect cools the proxy down
51
+ (`cooldownMs`, 30 s) and tries another (`retries`, 1); when every proxy is cooling, all are used.
52
+ `bypass` hosts go direct.
53
+ - Built-in plugins: `http`/`https` endpoints (CONNECT for https targets, absolute-form forward for
54
+ plain http — which a proxy resolves itself, so pinning holds only through CONNECT:
55
+ `createHttpProxyPlugin({ httpTargets: 'connect' })`), and `socks4`/`socks4a`/`socks5`/`socks5h`.
56
+ Custom `plugins` are tried first; an unknown type is `ProxyError('no-plugin')`.
57
+ - Remote-DNS endpoints (`socks5h`, `socks4a`, or a plugin built with `remoteDns: true`) receive the
58
+ name, not the checked address, so the proxy resolves it again: they still pre-check the name
59
+ locally and are refused when it does not resolve unless `trustProxyDns`.
60
+ - Proxy credentials never reach the log: an endpoint is logged as its `id` or `<type>://host:port`.
61
+
62
+ ## The pinned fetch
63
+
64
+ `createPinnedFetch` (on `@owlmeans/file-streams/fetch`) is `fetch` over this client: it asks with
65
+ `redirect: 'manual'` (unless the caller follows) and `anyStatus`, so the caller sees what a browser's
66
+ fetch would, while every connection — every redirect hop the caller takes, every resolver step —
67
+ goes to an address the policy resolved and checked a moment before. That closes the DNS-rebinding
68
+ window a guard-then-`fetch` pair leaves open: llm-reader injects it as its transport and hands it to
69
+ the Playwright renderer, which fulfils a page's sub-requests through it. Its module graph carries no
70
+ AWS SDK, so a consumer that only reads the web does not load one (`tests/fetch.spec.ts` walks the
71
+ built graph).
72
+
73
+ - `createPinnedFetch({ network, proxies, resolvers, timeouts, userAgent, headers, client? })` — the
74
+ client's options, or a ready `client`. Any status is a `Response` with `url` and `redirected` set;
75
+ `redirect: 'manual'` returns the 3xx, `'error'` rejects `HttpError('redirect')`; an abort ends a
76
+ request waiting for its head and a body in flight.
77
+ - Only GET and HEAD (another method, or a body, rejects `HttpError('method:…' | 'body')`);
78
+ `Accept-Encoding` is always identity, hop-by-hop, `host` and `content-length` request headers are
79
+ dropped; a transport-decoded body loses its `Content-Encoding`/`Content-Length`; a 204/205/304
80
+ answer has a null body.
81
+ - A policy refusal rejects with `NetworkPolicyError`, whose `refused === true` tells a consumer
82
+ (llm-reader maps it to `ReaderRefusedError('pinned:…')`) a refusal from a network failure.
83
+
33
84
  ## Google Drive
34
85
 
35
- Share links (`/file/d/<id>/…`, `open?id=`, `uc?id=`) become `drive.usercontent.google.com/download`;
36
- the "can't scan this file for viruses" page is followed through its own `download-form` (action must
37
- be a Google origin; hidden `confirm`/`uuid` carried), then the legacy `confirm=` link, then a
38
- `download_warning` cookie. Docs, Sheets, Slides and Drawings links become export URLs (docx, xlsx,
39
- pptx, png by default; csv/tsv keep the `gid`). Folders, quota pages, sign-in redirects and forms
40
- posting elsewhere fail as `ResolverError('google-drive:<reason>')`. The page flow is the one gdown
41
- documents; it is reverse-engineered, so `FILE_STREAMS_LIVE_DRIVE_URL` exercises it against Google.
86
+ Share links (`/file/d/<id>/…`, `open?id=`, `uc?id=`, `resourcekey` kept) become
87
+ `drive.usercontent.google.com/download`; the "can't scan this file for viruses" page is followed
88
+ through its own `download-form` (the action must be one of the resolver's Google origins; every
89
+ hidden input carried), then the legacy `confirm=` link, then a `download_warning` cookie — at most
90
+ 3 steps, 256 KiB of a page read. Docs, Sheets, Slides and Drawings links become export URLs (docx,
91
+ xlsx, pptx, png by default, `exports` changes them; csv/tsv keep the `gid`). Folders, quota pages,
92
+ sign-in redirects and forms posting elsewhere fail as `ResolverError('google-drive:<reason>')`
93
+ (`folder`, `quota`, `not-public`, `foreign-form`, `export-too-large`, `unrecognized`). A range
94
+ request resolves first, then asks the bytes' own URL for the range; `raw: true` skips resolvers. The
95
+ page flow is the one gdown documents; it is reverse-engineered, so `FILE_STREAMS_LIVE_DRIVE_URL`
96
+ exercises it against Google.
42
97
 
43
98
  ## External docs
44
99