@owlmeans/file-streams 0.1.22-rc.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +437 -21
- package/agent-meta/manifest.json +77 -8
- package/agent-meta/skills/file-streams/SKILL.md +84 -105
- package/agent-meta/skills/file-streams/parts/archives.md +80 -0
- package/agent-meta/skills/file-streams/parts/extending.md +117 -0
- package/agent-meta/skills/file-streams/parts/network.md +77 -22
- package/agent-meta/skills/file-streams/parts/workload.md +51 -0
- package/agent-meta/skills/file-streams/parts/workspaces.md +9 -22
- package/build/errors.d.ts +5 -0
- package/build/errors.d.ts.map +1 -1
- package/build/errors.js +5 -0
- package/build/errors.js.map +1 -1
- package/build/fetch.d.ts +12 -0
- package/build/fetch.d.ts.map +1 -0
- package/build/fetch.js +6 -0
- package/build/fetch.js.map +1 -0
- package/build/http/types.d.ts +19 -1
- package/build/http/types.d.ts.map +1 -1
- package/build/http.d.ts.map +1 -1
- package/build/http.js +7 -4
- package/build/http.js.map +1 -1
- package/build/index.d.ts +2 -0
- package/build/index.d.ts.map +1 -1
- package/build/index.js +1 -0
- package/build/index.js.map +1 -1
- package/build/pinned-fetch/consts.local.d.ts +9 -0
- package/build/pinned-fetch/consts.local.d.ts.map +1 -0
- package/build/pinned-fetch/consts.local.js +13 -0
- package/build/pinned-fetch/consts.local.js.map +1 -0
- package/build/pinned-fetch/types.d.ts +16 -0
- package/build/pinned-fetch/types.d.ts.map +1 -0
- package/build/pinned-fetch/types.js +2 -0
- package/build/pinned-fetch/types.js.map +1 -0
- package/build/pinned-fetch.d.ts +9 -0
- package/build/pinned-fetch.d.ts.map +1 -0
- package/build/pinned-fetch.js +82 -0
- package/build/pinned-fetch.js.map +1 -0
- package/package.json +18 -11
- package/src/errors.ts +6 -0
- package/src/fetch.ts +14 -0
- package/src/http/types.ts +21 -1
- package/src/http.ts +7 -4
- package/src/index.ts +2 -0
- package/src/pinned-fetch/consts.local.ts +15 -0
- package/src/pinned-fetch/types.ts +17 -0
- package/src/pinned-fetch.ts +93 -0
- package/src/sinks/workload/consts.ts +0 -1
- package/tests/fetch.spec.ts +163 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: file-streams
|
|
3
|
-
description: "How to use @owlmeans/file-streams — online files as Node streams: URL downloads under a connect-time SSRF policy, HTTP and SOCKS proxy plugins (a list or a callback by key and URL), the Google Drive resolver, S3 buckets addressed exactly like storage-resource (with prefix listing and removal), folders, buffers and Viable ephemeral workloads (writes and reads); scoped temporary workspaces nested by organization/project/purpose and mirrored to a bucket for restore after a restart; zip/tar/tar.gz/gz with traversal and bomb guards; file-type verification
|
|
3
|
+
description: "How to use @owlmeans/file-streams — online files as Node streams: URL downloads under a connect-time SSRF policy (follow or return redirects, any status on request), a pinned WHATWG fetch on the aws-free ./fetch subpath, HTTP and SOCKS proxy plugins (a list or a callback by key and URL), the Google Drive resolver, S3 buckets addressed exactly like storage-resource (with prefix listing and removal), folders, buffers and Viable ephemeral workloads (writes and reads); scoped temporary workspaces nested by organization/project/purpose and mirrored to a bucket for restore after a restart; zip/tar/tar.gz/gz with traversal and bomb guards; file-type verification from a head or a few range reads. Auto-invoked when downloading a file, giving fetch-based code (llm-reader, a renderer) pinned egress, moving bytes between HTTP, S3, disk and a workload, keeping uploaded files in a service tmp folder backed by S3, extracting or packing an archive, checking that a file is what it claims, or configuring outbound proxies."
|
|
4
4
|
user-invocable: false
|
|
5
5
|
---
|
|
6
6
|
<!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
|
|
@@ -8,42 +8,38 @@ user-invocable: false
|
|
|
8
8
|
# @owlmeans/file-streams
|
|
9
9
|
|
|
10
10
|
**Layer:** Infra
|
|
11
|
-
**Install:** `"@owlmeans/file-streams": "^0.
|
|
11
|
+
**Install:** `"@owlmeans/file-streams": "^0.2.0"` in `dependencies`
|
|
12
12
|
|
|
13
13
|
Server-side (Node and Bun). Every byte moves as a Node `Readable` — the shape storage-resource's
|
|
14
|
-
`StoredRecord.stream`, the AWS SDK, zlib and the archive codecs share.
|
|
15
|
-
|
|
14
|
+
`StoredRecord.stream`, the AWS SDK, zlib and the archive codecs share. `createStreamSource` takes a
|
|
15
|
+
web stream or an async iterable, `createBufferSource` a `Uint8Array`.
|
|
16
16
|
|
|
17
17
|
## Key Exports
|
|
18
18
|
|
|
19
19
|
| Export | What it is |
|
|
20
20
|
|---|---|
|
|
21
|
-
| `FileSource` / `FileSink` / `TreeSink` | Where bytes come from (`info`, `open`, `ranges`), where one file goes, where many go |
|
|
22
|
-
| `httpClient` / `createHttpClient(opts)` |
|
|
23
|
-
| `
|
|
24
|
-
| `
|
|
25
|
-
| `
|
|
26
|
-
| `
|
|
27
|
-
| `
|
|
28
|
-
| `
|
|
29
|
-
| `
|
|
30
|
-
| `
|
|
31
|
-
| `
|
|
32
|
-
| `
|
|
33
|
-
| `
|
|
34
|
-
| `
|
|
35
|
-
| `
|
|
36
|
-
| `appendFileStreamsService(ctx, alias?, opts?)` | The objects configured from `cfg.fileStreams` and `cfg.storageBuckets` |
|
|
37
|
-
| `FileKind`, `ArchiveFormat`, `WORKLOAD_MAX_FILE_BYTES`, `S3_DELETE_BATCH_SIZE`, `WORKSPACE_SCOPE_MARKER` | Enums, ceilings, the reserved marker name |
|
|
38
|
-
| `NetworkPolicyError`, `HttpError` (`.status`), `ProxyError`, `LimitError`, `SourceError`, `PathError`, `ArchiveError`, `ResolverError`, `WorkloadError` | Failures, by message code (`file-streams:<area>:<reason>`) |
|
|
39
|
-
| `FileMimicryError`, `FileKindError` (storage `FileTypeError`, from file-inspector), `S3TransferError` (storage `StorageApiError`) | Type refusals and S3 failures a storage-resource catch site already handles |
|
|
21
|
+
| `FileSource` / `FileSink` / `TreeSink` | Where bytes come from (`info`, `open`, `ranges`), where one file goes (`write`), where many go (`entry`, `directory`) |
|
|
22
|
+
| `httpClient` / `createHttpClient(opts)` | `request(url, opts)` under the network policy, through proxies, redirects and resolvers; `.source(url)` is re-openable and Range-capable |
|
|
23
|
+
| `createPinnedFetch(opts)` (also `@owlmeans/file-streams/fetch`) | A WHATWG `fetch` over the client; the subpath loads no AWS SDK |
|
|
24
|
+
| `createFileSource`, `createBufferSource`, `createStreamSource` | A file on disk, bytes in memory, a stream opened elsewhere (one-shot) |
|
|
25
|
+
| `createFileSink`, `createDirectorySink`, `createBufferSink` | Atomic file write, a confined directory tree, an in-memory collector (`bytes()`) |
|
|
26
|
+
| `createS3Bucket(entry or model, { partSize?, concurrency? })` | A `cfg.storageBuckets` entry as `source`, `sink`, `tree`, `head`, `remove`, `list`, `removePrefix`, `keyOf`, `urlOf`, `instanceOf`, `close` |
|
|
27
|
+
| `transferHelper` / `createTransferHelper({ inspector?, limits? })` | `.copy(source, sink, opts)` / `.copyInto(source, tree, opts)` — one file under limits, its head verified first |
|
|
28
|
+
| `archiveHelper` / `createArchiveHelper({ limits?, inspector? })` | `extract`, `list`, `pack`, `gzip`, `gunzip`, `detect` — the [archives](parts/archives.md) part |
|
|
29
|
+
| `createWorkspace`, `createBackedWorkspace`, `fileScopeHelper`, `FileScope` | Scoped tmp folders, mirrored to a bucket — the [workspaces](parts/workspaces.md) part |
|
|
30
|
+
| `makeWorkloadSink`, `makeWorkloadSource` | A Viable ephemeral workload — the [workload](parts/workload.md) part |
|
|
31
|
+
| `createNetworkPolicy`, `createProxyRouter`, `createHttpProxyPlugin`, `createSocksProxyPlugin`, `parseProxyEndpoint`, `createGoogleDriveResolver` / `googleDriveResolver` | What may be reached, proxies, Drive links — the [network](parts/network.md) part |
|
|
32
|
+
| `fileInspector`, `createFileInspector`, `FileKind`, the inspector tables and verdict types | Re-exported from `@owlmeans/file-inspector` |
|
|
33
|
+
| `appendFileStreamsService(ctx, alias?, opts?)` / `createFileStreamsService`, `FILE_STREAMS_SERVICE` | The objects configured from `cfg.fileStreams` and `cfg.storageBuckets` |
|
|
34
|
+
| `FileStreamsError` → `NetworkPolicyError` (`.refused === true`), `HttpError` (`.status`), `ProxyError`, `LimitError`, `SourceError`, `PathError`, `ArchiveError`, `ResolverError`, `WorkloadError` | Failures, by message code (`file-streams:<area>:<reason>`) |
|
|
35
|
+
| `FileMimicryError`, `FileKindError` (storage `FileTypeError`), `S3TransferError` (storage `StorageApiError`, `s3:<op>`) | Refusals and S3 failures a storage-resource catch site already handles |
|
|
40
36
|
|
|
41
37
|
## Usage
|
|
42
38
|
|
|
43
39
|
```typescript
|
|
44
40
|
import {
|
|
45
41
|
archiveHelper, createBufferSource, createDirectorySink, createHttpClient, createS3Bucket, FileKind,
|
|
46
|
-
fileInspector, httpClient,
|
|
42
|
+
fileInspector, httpClient, transferHelper,
|
|
47
43
|
} from '@owlmeans/file-streams'
|
|
48
44
|
|
|
49
45
|
// An image from the internet into a folder, refused unless it is an image
|
|
@@ -51,133 +47,116 @@ await transferHelper.copyInto(httpClient.source(url), createDirectorySink(workDi
|
|
|
51
47
|
|
|
52
48
|
// A screenshot (a Buffer) into the bucket a storage entry names — same key and URL storage-resource uses
|
|
53
49
|
const bucket = createS3Bucket(cfg.storageBuckets.media)
|
|
54
|
-
const { sink } = await transferHelper.copy(createBufferSource(png, { name: 'shot.png' }), bucket.sink(
|
|
50
|
+
const { sink } = await transferHelper.copy(createBufferSource(png, { name: 'shot.png' }), bucket.sink(`shots/${id}.png`))
|
|
55
51
|
sink.url // === storage-resource's URL for that prefix
|
|
56
52
|
bucket.instanceOf(sink, 'original') // a storage-common StoredFileInstance
|
|
57
53
|
|
|
58
|
-
//
|
|
59
|
-
await archiveHelper.extract(httpClient.source(archiveUrl), createDirectorySink(dir), { onEntryError: 'skip' })
|
|
54
|
+
// An archive into a folder; entries are verified only when `verify` is given
|
|
55
|
+
await archiveHelper.extract(httpClient.source(archiveUrl), createDirectorySink(dir), { verify: {}, onEntryError: 'skip' })
|
|
60
56
|
|
|
61
57
|
// What a remote file is, without downloading it (a few range reads, at most 1 MiB)
|
|
62
58
|
const verdict = await fileInspector.probe(httpClient.source(url))
|
|
63
59
|
|
|
64
60
|
// Proxies: a list (round-robin, random, failover, by-key) or a callback by key and URL
|
|
65
|
-
const pooled = createHttpClient({ proxies: { proxies: ['
|
|
61
|
+
const pooled = createHttpClient({ proxies: { proxies: ['socks5://u:p@egress-1:1080', 'http://egress-2:3128'], strategy: 'by-key' } })
|
|
66
62
|
const vaulted = createHttpClient({ proxies: { proxies: async ({ key, url }) => vault.proxyFor(key, url.hostname) } })
|
|
67
|
-
await pooled.request(url, { proxyKey:
|
|
63
|
+
await pooled.request(url, { proxyKey: tenantKey })
|
|
68
64
|
```
|
|
69
65
|
|
|
70
66
|
`copy` reads the head of the very stream it moves (no second download), verifies it, and replays
|
|
71
|
-
it into the sink. Verification is on by default: a kind mismatch is refused,
|
|
72
|
-
executable under another claim, a script under a passive one, a PDF with
|
|
73
|
-
sample) is always refused, `verify.refuse` refuses kinds outright, an
|
|
74
|
-
unless `refuseInconclusive`; `verify: false` turns it off.
|
|
67
|
+
it into the sink. Verification is on by default (`verify: {}`): a kind mismatch is refused,
|
|
68
|
+
dangerous content (an executable under another claim, a script under a passive one, a PDF with
|
|
69
|
+
active content in the head sample) is always refused, `verify.refuse` refuses kinds outright, an
|
|
70
|
+
inconclusive verdict passes unless `refuseInconclusive`; `verify: false` turns it off. A declared
|
|
71
|
+
size above `limits.maxBytes` is refused before the first byte, counted bytes after it. `info`
|
|
72
|
+
overrides the source's claims; `copyInto` writes under the source's sanitized name, or `download`.
|
|
75
73
|
|
|
76
|
-
## The
|
|
74
|
+
## The pinned fetch
|
|
77
75
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
76
|
+
`createPinnedFetch(opts)` from `@owlmeans/file-streams/fetch` (no AWS SDK, storage-resource or S3
|
|
77
|
+
module in its graph) is `fetch` over the client — any status a `Response`, `redirect: 'manual'` the
|
|
78
|
+
3xx itself, GET/HEAD only, bodies streamed, policy refusals `NetworkPolicyError` (`refused === true`).
|
|
79
|
+
Rules: the [network](parts/network.md) part.
|
|
81
80
|
|
|
82
|
-
|
|
83
|
-
const media = makeWorkloadSink(fileHelper, { root: 'sources/web/public/media' })
|
|
84
|
-
await transferHelper.copy(httpClient.source(imageUrl), media.entry('hero.png'), { verify: { allow: [FileKind.Image] } })
|
|
85
|
-
await archiveHelper.extract(httpClient.source(zipUrl), media, { onEntryError: 'skip' })
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
Each file is one atomic write of at most 12 MiB (`WORKLOAD_MAX_FILE_BYTES`, the publisher's 16 MiB
|
|
89
|
-
base64 limit); `maxFileBytes` lowers it, and an oversized file is refused before any request
|
|
90
|
-
(`WorkloadError('too-large')`). Larger files need a product route (chunked writes, or the workload
|
|
91
|
-
pulling a URL itself — it has public egress and can run this package).
|
|
92
|
-
|
|
93
|
-
Reads go the other way through `makeWorkloadSource(reader, { root })` (`readBinaryFile`, same
|
|
94
|
-
ceiling): the [workspaces](parts/workspaces.md) part.
|
|
95
|
-
|
|
96
|
-
## Workspaces
|
|
81
|
+
## Workspaces and workloads
|
|
97
82
|
|
|
98
|
-
`createWorkspace().scope({ organization, project?, purpose })` is the folder
|
|
99
|
-
`<root>/<organization>/<project?>/<purpose
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
(no `..`, no absolute paths, no existing symlink from the workspace root, the marker name reserved);
|
|
103
|
-
a relative `root` lives under `os.tmpdir()`. Full rules, overlap and prune semantics, prefix
|
|
104
|
-
operations: the [workspaces](parts/workspaces.md) part.
|
|
83
|
+
`createWorkspace().scope({ organization, project?, purpose })` is the confined folder
|
|
84
|
+
`<root>/<organization>/<project?>/<purpose>`; a backed workspace mirrors it to the same nesting in a
|
|
85
|
+
bucket, which `restore()` reads back after a pod restart — the [workspaces](parts/workspaces.md)
|
|
86
|
+
part. A Viable workload takes and gives files of at most 12 MiB — the [workload](parts/workload.md) part.
|
|
105
87
|
|
|
106
88
|
## Detection
|
|
107
89
|
|
|
108
|
-
Verification is `@owlmeans/file-inspector` (its `file-inspector` skill
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
## Archives
|
|
115
|
-
|
|
116
|
-
zip is read from its central directory (yauzl) — through range reads when the source has them, else
|
|
117
|
-
from a private temporary spool — and written as a stream (yazl); tar and tar.gz stream through
|
|
118
|
-
tar-stream (an entry is drained, never destroyed: destroying one destroys the archive); gz is zlib.
|
|
119
|
-
The format comes from the bytes. Refused: absolute paths, `..`, drive letters, control and bidi
|
|
120
|
-
characters, overlong or deep paths, duplicate and case-fold-twin entries, encrypted entries,
|
|
121
|
-
overlapping zip entries, an expansion ratio above 200 (past 1 MiB), more entries or bytes than the
|
|
122
|
-
limits. Links and devices are skipped (or refused with `links: 'refuse'`), never created. Nested
|
|
123
|
-
archives stay files unless `limits.maxDepth` > 1. Each entry commits atomically; a refusal leaves the
|
|
124
|
-
entries already written.
|
|
90
|
+
Verification is `@owlmeans/file-inspector` (its `file-inspector` skill), re-exported unchanged
|
|
91
|
+
except `fileNameHelper`, `TEXT_CARRIED_KINDS` and the `ProbeSource` types — import those from it.
|
|
92
|
+
Every `FileSource` is a `ProbeSource` as is: a probe seeks through `ranges()` when the origin has
|
|
93
|
+
them and reads the head of `open()` otherwise. The S3 sink takes the Content-Type from the verdict
|
|
94
|
+
and writes `attachment` for active svg/html (unless `allowActiveInline`).
|
|
125
95
|
|
|
126
96
|
## Defaults
|
|
127
97
|
|
|
128
98
|
| Area | Default |
|
|
129
99
|
|---|---|
|
|
130
100
|
| Transfer | 512 MiB · 10 min total · 30 s idle |
|
|
131
|
-
| HTTP | connect 10 s · headers 30 s · idle 30 s · 5 redirects · `Accept-Encoding: identity` |
|
|
101
|
+
| HTTP | connect 10 s · headers 30 s · idle 30 s · 5 redirects · `Accept-Encoding: identity` · User-Agent `owlmeans-file-streams` |
|
|
102
|
+
| Proxies | round-robin · 1 retry on another proxy · 30 s cooldown |
|
|
132
103
|
| Probe | 1 MiB budget · 64 KiB head sample |
|
|
133
|
-
| Archives | 10 000 entries · 512 MiB per entry · 2 GiB total · ratio 200 · depth 1 |
|
|
134
104
|
| S3 | 8 MiB parts (≥ 5 MiB), 2 in flight; one PutObject when the body fits a part; aborted on failure |
|
|
135
|
-
| Workload | 12 MiB per file (`WORKLOAD_MAX_FILE_BYTES`), one retry
|
|
105
|
+
| Workload | 12 MiB per file (`WORKLOAD_MAX_FILE_BYTES`), one retry |
|
|
136
106
|
| Workspace | `<os.tmpdir()>/owlmeans-workspace`, folders `0o700`, files `0o600` |
|
|
137
|
-
| S3 prefixes | ListObjectsV2 all pages; DeleteObjects ≤ 1000 keys, checksummed by Content-MD5 |
|
|
138
107
|
|
|
139
108
|
## Service
|
|
140
109
|
|
|
141
110
|
`appendFileStreamsService(ctx)` registers `file-streams`: `http()`, `transfer()`, `inspector()`,
|
|
142
111
|
`archives()`, `bucket(configKey)`, `workspace()` and `backed(configKey, { prefix? })` (one object per
|
|
143
|
-
key and prefix,
|
|
144
|
-
`
|
|
145
|
-
`
|
|
146
|
-
objects so a server context resolves their secret-file leaves; callbacks
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
-
|
|
112
|
+
key and prefix), built lazily from `cfg.fileStreams` (`limits` — transfers only, `network`,
|
|
113
|
+
`proxies: [{ url, username?, password?, id? }]`, `proxyStrategy`, `userAgent`, `workspace`) and
|
|
114
|
+
`cfg.storageBuckets` (an unknown key → `FilePropertyError('storage-config:<key>')`). Proxy entries
|
|
115
|
+
are objects so a server context resolves their secret-file leaves; callbacks, plugins, resolvers, a
|
|
116
|
+
DNS `resolver`, `bucketOptions`, `inspector` and `archiveLimits` come through the options argument.
|
|
117
|
+
A server context reads EVERY string leaf starting with `/` as a file, so `workspace.root` is given
|
|
118
|
+
relative to `os.tmpdir()`. Timeouts, redirects, headers and CAs need your own `createHttpClient`.
|
|
119
|
+
|
|
120
|
+
## Pitfalls
|
|
121
|
+
|
|
122
|
+
- Loopback and in-cluster origins are refused by default: give `network: { allowCidrs }` (tests:
|
|
123
|
+
`'127.0.0.1/32'`) or `allowPrivate`.
|
|
124
|
+
- `createStreamSource` is one-shot: a probe or `archiveHelper.detect` consumes it — verify through
|
|
125
|
+
`copy`'s `verify` instead.
|
|
126
|
+
- `archiveHelper.extract` verifies nothing unless `verify` is passed; `transferHelper` verifies by
|
|
127
|
+
default.
|
|
128
|
+
- Passing `resolvers` replaces the default `[googleDriveResolver]`; keep it when you add your own.
|
|
129
|
+
- A `socks5h`/`socks4a` endpoint resolves names at the proxy, so its connect is not pinned.
|
|
130
|
+
- storage-resource's default ACL is `public-read`: private files need a bucket entry with `acl: false`.
|
|
154
131
|
|
|
155
132
|
## Tests
|
|
156
133
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
`/command/` receiver answering `writeBinaryFile` and `readBinaryFile` with the real limits) and
|
|
160
|
-
code-built fixtures; `workspace.spec.ts` runs offline over a temporary root; `s3.spec.ts` and
|
|
161
|
-
`backed.spec.ts` need the S3 gate (path-style for a non-AWS endpoint, objects under a random base
|
|
162
|
-
prefix, deleted after); `live.spec.ts` needs `FILE_STREAMS_LIVE=1` (and `FILE_STREAMS_LIVE_DRIVE_URL`).
|
|
134
|
+
`bun test ./tests` in `@owlmeans/file-streams` (category C; build first). Offline specs use local
|
|
135
|
+
stand-ins; S3 and live specs are gated — the [extending](parts/extending.md) part.
|
|
163
136
|
|
|
164
137
|
## Depends On
|
|
165
138
|
|
|
166
139
|
- `@owlmeans/file-inspector` (detection, re-exported), `@owlmeans/storage-resource` (the bucket
|
|
167
|
-
model), `@owlmeans/storage-common` (errors, instances), `@owlmeans/context`,
|
|
168
|
-
`@owlmeans/log`
|
|
140
|
+
model), `@owlmeans/storage-common` (errors, instances, scopes), `@owlmeans/context`,
|
|
141
|
+
`@owlmeans/error`, `@owlmeans/log`
|
|
169
142
|
- `@aws-sdk/client-s3`, `socks`, `tar-stream`, `yauzl`, `yazl`
|
|
170
143
|
|
|
171
144
|
## Related
|
|
172
145
|
|
|
173
|
-
- [[file-inspector]]
|
|
174
|
-
- [[storage-resource]] — the upload resource and the bucket model
|
|
175
|
-
- [[storage-common]] — stored-file types and errors, `FileScope`
|
|
176
|
-
- [[
|
|
146
|
+
- [[file-inspector]] (`@owlmeans/file-inspector`) — verdicts, policies, kinds, active PDF, safe names
|
|
147
|
+
- [[storage-resource]] (`@owlmeans/storage-resource`) — the upload resource and the bucket model
|
|
148
|
+
- [[storage-common]] (`@owlmeans/storage-common`) — stored-file types and errors, `FileScope`
|
|
149
|
+
- [[server-upload]] (`@owlmeans/server-upload`) — multi-file uploads over these objects
|
|
150
|
+
- [[upload]] (`@owlmeans/upload`) — the upload contract; its path rules match this package's
|
|
151
|
+
- [[llm-reader]] (`@owlmeans/llm-reader`) — takes the pinned fetch as its transport
|
|
152
|
+
- [[logging]] — scopes `file-streams:<area>`; URLs logged redacted, contents and proxy secrets never
|
|
177
153
|
|
|
178
154
|
## Parts
|
|
179
155
|
|
|
180
156
|
| Part | Load it when |
|
|
181
157
|
|---|---|
|
|
182
|
-
| [network](parts/network.md) | changing what may be reached, routing through proxies, or downloading Google Drive and Docs links |
|
|
183
|
-
| [
|
|
158
|
+
| [network](parts/network.md) | changing what may be reached, routing through proxies, returning redirects, handing a pinned `fetch` to fetch-based code, or downloading Google Drive and Docs links |
|
|
159
|
+
| [archives](parts/archives.md) | extracting, listing, packing or (de)compressing zip, tar, tar.gz or gz, or tuning archive limits |
|
|
160
|
+
| [workspaces](parts/workspaces.md) | keeping files in a service tmp folder by scope, mirroring them to a bucket, restoring after a restart, or listing or deleting a bucket prefix |
|
|
161
|
+
| [workload](parts/workload.md) | writing files into a Viable ephemeral workload or reading them back |
|
|
162
|
+
| [extending](parts/extending.md) | writing your own source, sink, tree sink, proxy plugin or URL resolver, changing this package, or running and adding its tests |
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: archives
|
|
3
|
+
description: Reading and writing zip, tar, tar.gz and gz with archiveHelper — format detection from bytes, listing, safe extraction into a tree (traversal, link, duplicate, bomb and overlap guards, per-entry verification, skip or fail), nested archives, packing and gzip/gunzip sources, and the archive limits.
|
|
4
|
+
metadata:
|
|
5
|
+
purposes: data
|
|
6
|
+
mentions: archiveHelper, createArchiveHelper, ArchiveFormat, ArchiveExtractOptions, ArchiveLimits, DEFAULT_ARCHIVE_LIMITS, ExtractResult, PackEntry, onEntryError, maxDepth, ArchiveError
|
|
7
|
+
---
|
|
8
|
+
<!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
|
|
9
|
+
|
|
10
|
+
# @owlmeans/file-streams — Archives
|
|
11
|
+
|
|
12
|
+
```typescript
|
|
13
|
+
import { ArchiveFormat, archiveHelper, createDirectorySink, createFileSource, FileKind, httpClient } from '@owlmeans/file-streams'
|
|
14
|
+
|
|
15
|
+
const result = await archiveHelper.extract(httpClient.source(zipUrl), createDirectorySink(dir), {
|
|
16
|
+
verify: { refuse: [FileKind.Executable] }, onEntryError: 'skip', strip: 1, limits: { maxEntries: 500 },
|
|
17
|
+
})
|
|
18
|
+
result.extracted // [{ entry, path, result }, …]
|
|
19
|
+
result.skipped // [{ entry, reason }, …] — 'link', 'unsupported-type', 'stripped', 'filtered', or the error message
|
|
20
|
+
|
|
21
|
+
await archiveHelper.list(createFileSource(path)) // ArchiveEntryInfo[] — nothing written
|
|
22
|
+
await archiveHelper.detect(source) // ArchiveFormat | null
|
|
23
|
+
const zip = archiveHelper.pack(ArchiveFormat.Zip, [{ path: 'a.txt', source: bytes }, { path: 'b.pdf', source: fileSource }], { name: 'out.zip' })
|
|
24
|
+
const gz = archiveHelper.gzip(source) // a one-shot source of the compressed bytes
|
|
25
|
+
const plain = archiveHelper.gunzip(source) // the decompressed bytes, under the ratio and size limits
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Reading
|
|
29
|
+
|
|
30
|
+
zip is read from its central directory (yauzl) — through range reads when the source has them, else
|
|
31
|
+
from a private temporary spool (at most `maxTotalBytes`) — and written as a stream (yazl); tar and
|
|
32
|
+
tar.gz stream through tar-stream (an entry is drained, never destroyed: destroying one destroys the
|
|
33
|
+
archive); gz is zlib, its single entry named from the gzip header or the source's name without
|
|
34
|
+
`.gz`. The format comes from the bytes (`format` skips detection), never from names.
|
|
35
|
+
|
|
36
|
+
## Extraction rules
|
|
37
|
+
|
|
38
|
+
- **Refused** (the extraction stops): absolute paths, `..`, drive letters, control and bidi
|
|
39
|
+
characters, overlong or deep paths (`maxPathBytes`, `maxPathSegments`), duplicate and
|
|
40
|
+
case-fold-twin entries, encrypted entries, compression methods other than store and deflate,
|
|
41
|
+
overlapping zip entries, sizes that lie (yauzl `validateEntrySizes`), a declared or measured
|
|
42
|
+
expansion ratio above `maxRatio` once past 1 MiB (`RATIO_FLOOR_BYTES`), more entries or bytes than
|
|
43
|
+
the limits.
|
|
44
|
+
- **Skipped** (reported in `skipped`): links (`links: 'refuse'` throws `ArchiveError('link')`
|
|
45
|
+
instead), devices and other special entries, entries `strip` empties, entries `filter` rejects.
|
|
46
|
+
Neither links nor devices are ever created.
|
|
47
|
+
- **Verification** runs only when `verify` is given: each entry's head is checked against the
|
|
48
|
+
entry's own name before it is written.
|
|
49
|
+
- **`onEntryError: 'skip'`** skips an entry whose write or verification fails (a too-large workload
|
|
50
|
+
file, a mimicry refusal); `LimitError`, `ArchiveError`, `PathError` and an abort always stop.
|
|
51
|
+
- **Nested archives** (`zip`, `tar`, `tar.gz`, `tgz`, `gz` names) stay files unless
|
|
52
|
+
`limits.maxDepth` > 1; a followed one is spooled and extracted under its name without the extension.
|
|
53
|
+
- Each entry commits atomically; a refusal leaves the entries already written. `onEntry(entry,
|
|
54
|
+
result)` reports each written file.
|
|
55
|
+
|
|
56
|
+
## Packing
|
|
57
|
+
|
|
58
|
+
`pack(format, entries, { name?, level?, maxSpoolBytes? })` returns a one-shot `FileSource`. Entry
|
|
59
|
+
paths pass the same path rules; a `source` is a `FileSource` or a `Uint8Array`. A tar needs sizes up
|
|
60
|
+
front, so a source without one is spooled (up to 512 MiB by default). gz takes exactly one entry
|
|
61
|
+
(`ArchiveError('gzip-entries')`).
|
|
62
|
+
|
|
63
|
+
## Limits
|
|
64
|
+
|
|
65
|
+
| Limit | Default |
|
|
66
|
+
|---|---|
|
|
67
|
+
| `maxEntries` | 10 000 |
|
|
68
|
+
| `maxEntryBytes` | 512 MiB |
|
|
69
|
+
| `maxTotalBytes` | 2 GiB |
|
|
70
|
+
| `maxRatio` | 200 |
|
|
71
|
+
| `maxDepth` | 1 |
|
|
72
|
+
| `maxPathBytes` / `maxPathSegments` | 1024 / 64 |
|
|
73
|
+
|
|
74
|
+
`createArchiveHelper({ limits, inspector })` sets the defaults of one helper; each call's `limits`
|
|
75
|
+
override them. The service takes them as `archiveLimits` in its options.
|
|
76
|
+
|
|
77
|
+
## External docs
|
|
78
|
+
|
|
79
|
+
- https://github.com/thejoshwolfe/yauzl — central-directory zip reading, `fromRandomAccessReader` (override `read` and `_readStreamForRange`), `validateEntrySizes`, automatic `validateFileName` (v3)
|
|
80
|
+
- https://github.com/mafintosh/tar-stream — v3 ships streamx typings; destroying an entry destroys the extract, so entries are drained
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: extending
|
|
3
|
+
description: Writing your own FileSource, FileSink, TreeSink, proxy plugin or URL resolver and the contract each must keep; the invariants of the package (pinned connects, the aws-free ./fetch graph, counted limits, drained tar entries, confined paths, redacted logs); and the package's specs, stand-ins and gates.
|
|
4
|
+
metadata:
|
|
5
|
+
purposes: data, test
|
|
6
|
+
mentions: FileSource, FileSink, TreeSink, RangeReader, SinkInput, SinkResult, ProxyPlugin, ProxyConnection, TunnelTarget, UrlResolver, ResolverStep, FILE_STREAMS_LIVE, S3_TEST_BUCKET
|
|
7
|
+
---
|
|
8
|
+
<!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
|
|
9
|
+
|
|
10
|
+
# @owlmeans/file-streams — Extending and testing the package
|
|
11
|
+
|
|
12
|
+
## A source
|
|
13
|
+
|
|
14
|
+
A `FileSource` is `{ kind, info, open, ranges }`; it is also a file-inspector `ProbeSource`, so
|
|
15
|
+
probes, transfers and archive extraction take it as it is.
|
|
16
|
+
|
|
17
|
+
- `info({ signal? })` — metadata without the body, cached after the first success: `origin`
|
|
18
|
+
redacted (no credentials, no query), `name` a safe base name, `size` and `mime` as declared
|
|
19
|
+
(lower-cased, no parameters), `etag` a strong validator when the origin has one.
|
|
20
|
+
- `open({ range?, signal? })` — a fresh Node `Readable` from byte 0, or the honoured span with
|
|
21
|
+
`range: { start, end, total }`. A one-shot source refuses a second open with
|
|
22
|
+
`SourceError('consumed')`; a missing object is `SourceError('not-found')`.
|
|
23
|
+
- `ranges({ signal? })` — a `RangeReader` (`size`, `read(offset, length, signal?)`,
|
|
24
|
+
`stream(start, end, signal?)` with `end` exclusive, `close()`) over ONE immutable version: pin it
|
|
25
|
+
with the validator and throw `SourceError('changed')` when the origin moved. Answer `null`
|
|
26
|
+
without random access — probes then read the head of `open()` and a zip is spooled.
|
|
27
|
+
- An abort rejects with the signal's reason.
|
|
28
|
+
|
|
29
|
+
## A sink
|
|
30
|
+
|
|
31
|
+
- `FileSink.write({ stream, info, verdict? }, { maxBytes?, signal? })` consumes the whole stream,
|
|
32
|
+
counts it and refuses past `maxBytes` (`LimitError('max-bytes')`), commits atomically, and on
|
|
33
|
+
failure destroys the input and leaves nothing behind. It answers `SinkResult { kind, location,
|
|
34
|
+
bytes, url?, etag?, mime? }`, with `mime` from `verdict?.detected?.mime` before `info.mime`.
|
|
35
|
+
- `TreeSink.entry(path)` normalizes and confines the path (`PathError`: `absolute`, `traversal`,
|
|
36
|
+
`invalid-char`, `too-long`, `too-deep`, `empty`), refuses case-fold twins where the store folds
|
|
37
|
+
case, and returns the entry's `FileSink`; `directory(path)` creates an empty directory or is a
|
|
38
|
+
no-op where the store has none.
|
|
39
|
+
|
|
40
|
+
## A proxy plugin
|
|
41
|
+
|
|
42
|
+
```typescript
|
|
43
|
+
import { createHttpClient, ProxyError } from '@owlmeans/file-streams'
|
|
44
|
+
import type { ProxyPlugin } from '@owlmeans/file-streams'
|
|
45
|
+
|
|
46
|
+
const bastionPlugin: ProxyPlugin = {
|
|
47
|
+
types: ['bastion'],
|
|
48
|
+
remoteDns: () => false,
|
|
49
|
+
connect: async (endpoint, target, { signal, timeoutMs }) => {
|
|
50
|
+
const socket = await openTunnel(endpoint, target.host, target.port, { signal, timeoutMs }) // your transport
|
|
51
|
+
if (socket == null) {
|
|
52
|
+
throw new ProxyError('connect')
|
|
53
|
+
}
|
|
54
|
+
return { socket, mode: 'tunnel' }
|
|
55
|
+
},
|
|
56
|
+
}
|
|
57
|
+
const client = createHttpClient({ proxies: { proxies: ['bastion://jump.example.com:2222'], plugins: [bastionPlugin] } })
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
- Connect to `target.host` — the address the policy checked; `target.hostname` is for SNI and
|
|
61
|
+
`Host` only. `remoteDns(endpoint) === true` hands the name to the proxy instead and gives up the
|
|
62
|
+
pin (the name is still pre-checked).
|
|
63
|
+
- Never terminate TLS: return the raw tunnel, and the client layers TLS over it for https targets.
|
|
64
|
+
`mode: 'forward'` is for plain-http absolute-form proxies only, with their auth in `headers`.
|
|
65
|
+
- Honour `timeoutMs` and `signal`; fail with `ProxyError('connect' | 'handshake' | 'auth-failed' |
|
|
66
|
+
'refused:<code>')` — the router cools the endpoint down and tries another. A custom type needs an
|
|
67
|
+
explicit port in its URL. Never log credentials.
|
|
68
|
+
- Register through `createProxyRouter({ plugins })`, `createHttpClient({ proxies: { plugins } })` or
|
|
69
|
+
the service option `proxyPlugins`; custom plugins are tried before the built-in ones.
|
|
70
|
+
|
|
71
|
+
## A URL resolver
|
|
72
|
+
|
|
73
|
+
`UrlResolver { name, maxSteps, matches(url), rewrite(url), inspect(head, readBody), owns(url) }`:
|
|
74
|
+
`rewrite` turns a page link into the URL that serves the bytes (or throws
|
|
75
|
+
`ResolverError('<name>:<reason>')`); `inspect` sees each final 2xx GET answer before its body
|
|
76
|
+
reaches the caller, reads at most `readBody(maxBytes)` (replayed on accept) and answers
|
|
77
|
+
`{ action: 'accept' }`, `{ action: 'follow', url, headers?: { cookie } }` or
|
|
78
|
+
`{ action: 'fail', reason }`. A follow must stay on URLs the resolver `owns` (else
|
|
79
|
+
`foreign-follow`) and within `maxSteps`; its cookie is sent only there. Register with
|
|
80
|
+
`createHttpClient({ resolvers: [googleDriveResolver, mine] })` or the service option `resolvers`.
|
|
81
|
+
|
|
82
|
+
## Invariants
|
|
83
|
+
|
|
84
|
+
- Every connection goes through the client's connector: the policy resolves and checks, the socket
|
|
85
|
+
goes to that address (or the address goes into the tunnel). Network code never calls `fetch`,
|
|
86
|
+
`http.get(url)` or a DNS lookup of its own.
|
|
87
|
+
- The `./fetch` graph imports only `@owlmeans/error`, `@owlmeans/file-inspector`, `@owlmeans/log`,
|
|
88
|
+
`@owlmeans/storage-common`, `socks` and `node:*`; `fetch.spec.ts` fails on any S3,
|
|
89
|
+
storage-resource, workspace, transfer or archive module in it.
|
|
90
|
+
- Limits count the bytes that flowed; a declared size only refuses early.
|
|
91
|
+
- Tar entries are drained, never destroyed.
|
|
92
|
+
- Every written path passes the path rules, and nothing writes through an existing symlink.
|
|
93
|
+
- Logs carry redacted URLs, proxy ids and sizes — never contents, credentials or query strings.
|
|
94
|
+
- Failures are the package's error classes with `file-streams:<area>:<reason>` codes; S3 failures
|
|
95
|
+
are `S3TransferError`, a storage `StorageApiError`.
|
|
96
|
+
|
|
97
|
+
## Tests
|
|
98
|
+
|
|
99
|
+
`bun test ./tests` in `@owlmeans/file-streams`; suites are titled `@owlmeans/file-streams — <area>`.
|
|
100
|
+
Build first: `fetch.spec.ts` walks the built `./fetch` graph.
|
|
101
|
+
|
|
102
|
+
- Offline specs run against local stand-ins (`tests/harness/`: an origin with Range, redirects and
|
|
103
|
+
stalls, a CONNECT/forward proxy, a SOCKS5 server, a fake Drive, a workload `/command/` receiver
|
|
104
|
+
answering `writeBinaryFile` and `readBinaryFile` with the real limits) and code-built fixtures.
|
|
105
|
+
`tests/context.ts` holds `LOCAL_NETWORK` (`allowCidrs: ['127.0.0.1/32']`), `localClient()` and
|
|
106
|
+
`boot()` — a real server context with the service registered. `workspace.spec.ts` runs over a
|
|
107
|
+
temporary root.
|
|
108
|
+
- `s3.spec.ts` and `backed.spec.ts` need the S3 gate (`S3_ENDPOINT`, `S3_TEST_BUCKET`, `S3_KEY`,
|
|
109
|
+
`S3_SECRET`, `S3_REGION`; `gateHelper.s3Gate()` from `@owlmeans/test-integration`): path-style for
|
|
110
|
+
a non-AWS endpoint, `acl: false`, objects under a random base prefix, deleted after. Empty
|
|
111
|
+
variables print a skip.
|
|
112
|
+
- `live.spec.ts` needs `FILE_STREAMS_LIVE=1` (public HTTPS), and `FILE_STREAMS_LIVE_DRIVE_URL` for a
|
|
113
|
+
public Drive file.
|
|
114
|
+
- A new source is covered for info, whole and ranged opens, random access and a changed origin; a
|
|
115
|
+
sink for atomic commit, `maxBytes` and a failure that leaves nothing; a proxy plugin by a stand-in
|
|
116
|
+
that proves the checked address reaches the tunnel and no secret reaches the log; a resolver by a
|
|
117
|
+
fake origin.
|
|
@@ -1,44 +1,99 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: network
|
|
3
|
-
description: What the HTTP client may reach — the connect-time network policy, redirects and sensitive headers, the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.
|
|
3
|
+
description: What the HTTP client may reach — the connect-time network policy, redirects (followed or returned) and sensitive headers, the pinned WHATWG fetch on the ./fetch subpath, the proxy router and the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.
|
|
4
4
|
metadata:
|
|
5
5
|
purposes: config, ops
|
|
6
|
-
mentions: createNetworkPolicy, allowCidrs, allowPrivate, createProxyRouter, createHttpProxyPlugin, createSocksProxyPlugin, trustProxyDns, createGoogleDriveResolver, googleDriveResolver, ResolverError, NetworkPolicyError
|
|
6
|
+
mentions: createPinnedFetch, PinnedFetch, PinnedFetchOptions, HttpRedirectMode, anyStatus, createNetworkPolicy, allowCidrs, allowPrivate, createProxyRouter, createHttpProxyPlugin, createSocksProxyPlugin, parseProxyEndpoint, trustProxyDns, createGoogleDriveResolver, googleDriveResolver, ResolverError, NetworkPolicyError
|
|
7
7
|
---
|
|
8
8
|
<!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
|
|
9
9
|
|
|
10
10
|
# @owlmeans/file-streams — Network policy, proxies and Google Drive
|
|
11
11
|
|
|
12
|
-
## Network policy
|
|
12
|
+
## Network policy
|
|
13
13
|
|
|
14
|
-
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
14
|
+
- Before any DNS, `assertUrl` checks the scheme (`schemes`, default `https:`/`http:`), credentials
|
|
15
|
+
in the URL (`allowUrlCredentials`), `allowHosts`/`denyHosts` (`x.com`, `*.x.com`), `ports` and an
|
|
16
|
+
https → http hop (`allowDowngrade`).
|
|
17
|
+
- Loopback, private, link-local, CGNAT, benchmarking, multicast, reserved and documentation ranges
|
|
18
|
+
are refused, and the IPv4 inside a mapped, NAT64, 6to4, Teredo or IPv4-compatible address is
|
|
19
|
+
checked too; internal names (`localhost`, `.localhost`, `.local`, `.internal`, `.svc`,
|
|
20
|
+
`.cluster.local`, `.home.arpa`, `.lan`, `.intranet`) are refused before DNS unless `allowPrivate`
|
|
21
|
+
or an `allowHosts` pattern names them. Every resolved address must pass — one refused address
|
|
22
|
+
refuses the name (`blocked-address`). `allowCidrs` wins over `denyCidrs`, which wins over
|
|
23
|
+
`allowPrivate`. A custom `resolver` (DoH, split horizon, a test seam) must answer every address.
|
|
24
|
+
- The check is pinned to the connect: a direct socket goes to an address just resolved and checked
|
|
25
|
+
(each in turn until one connects), a proxy tunnel carries that address (TLS keeps the host name
|
|
26
|
+
for SNI and certificates, TLS 1.2+, extra `ca` only — verification is never off). Every redirect
|
|
27
|
+
hop is checked again; https → http is refused; `Authorization`, `Cookie`, `Proxy-Authorization`
|
|
28
|
+
and `sensitiveHeaders` never cross origins.
|
|
29
|
+
- `redirect: 'manual'` returns a 3xx with its `Location` (and body) instead of following it — for a
|
|
30
|
+
caller that checks every hop itself; a resolver's own flow (Drive's virus-scan page and its hops)
|
|
31
|
+
still follows inside the client under the policy, because it carries its cookie across them.
|
|
32
|
+
`anyStatus: true` returns every status instead of `HttpError('status:<code>')`; a forward proxy's
|
|
33
|
+
407 still throws `ProxyError('auth-failed')`. `HttpResponse.decoded` says the body was
|
|
34
|
+
transport-decoded and its `Content-Length` no longer applies; a gzip file is never decoded.
|
|
21
35
|
- A local stand-in or an in-cluster origin needs `network: { allowCidrs: [...] }` (or
|
|
22
36
|
`allowPrivate`). S3 traffic through the SDK is not policed — its endpoint is operator config.
|
|
23
|
-
- Built-in plugins: `http`/`https` endpoints (CONNECT for https targets, absolute-form forward for
|
|
24
|
-
plain http — which a proxy resolves itself, so pinning holds only through CONNECT:
|
|
25
|
-
`createHttpProxyPlugin({ httpTargets: 'connect' })`), and `socks4`/`socks4a`/`socks5`/`socks5h`.
|
|
26
|
-
Remote-DNS endpoints (`socks5h`, `socks4a`) still pre-check the name locally and are refused when
|
|
27
|
-
it does not resolve unless `trustProxyDns`.
|
|
28
37
|
- Bun's own `fetch({ proxy })` speaks only http/https proxies and undici dispatchers are unreliable
|
|
29
38
|
under Bun, so the client is `node:http(s)` over a socket it connects itself — Bun 1.4.2 honours
|
|
30
39
|
the `createConnection` option, an Agent's `createConnection`, the `lookup` option and TLS over an
|
|
31
40
|
existing socket.
|
|
32
41
|
|
|
42
|
+
## Proxies
|
|
43
|
+
|
|
44
|
+
- `createProxyRouter({ proxies, credentials?, strategy?, bypass?, plugins?, retries?, cooldownMs? })`
|
|
45
|
+
— or the same object as `createHttpClient({ proxies })`. `proxies` is a list of endpoints or URLs
|
|
46
|
+
(`parseProxyEndpoint` reads `socks5h://user:pass@host:1080`; default ports 8080/443/1080) or a
|
|
47
|
+
provider `({ key, url }) => endpoint | url | null` (null goes direct). `credentials(endpoint,
|
|
48
|
+
request)` fills an endpoint that carries none. A request's `proxyKey` is the key.
|
|
49
|
+
- Strategies: `round-robin` (default), `random`, `failover` (the first warm one), `by-key` (a key
|
|
50
|
+
hashes to one proxy; without a key, round-robin). A failed connect cools the proxy down
|
|
51
|
+
(`cooldownMs`, 30 s) and tries another (`retries`, 1); when every proxy is cooling, all are used.
|
|
52
|
+
`bypass` hosts go direct.
|
|
53
|
+
- Built-in plugins: `http`/`https` endpoints (CONNECT for https targets, absolute-form forward for
|
|
54
|
+
plain http — which a proxy resolves itself, so pinning holds only through CONNECT:
|
|
55
|
+
`createHttpProxyPlugin({ httpTargets: 'connect' })`), and `socks4`/`socks4a`/`socks5`/`socks5h`.
|
|
56
|
+
Custom `plugins` are tried first; an unknown type is `ProxyError('no-plugin')`.
|
|
57
|
+
- Remote-DNS endpoints (`socks5h`, `socks4a`, or a plugin built with `remoteDns: true`) receive the
|
|
58
|
+
name, not the checked address, so the proxy resolves it again: they still pre-check the name
|
|
59
|
+
locally and are refused when it does not resolve unless `trustProxyDns`.
|
|
60
|
+
- Proxy credentials never reach the log: an endpoint is logged as its `id` or `<type>://host:port`.
|
|
61
|
+
|
|
62
|
+
## The pinned fetch
|
|
63
|
+
|
|
64
|
+
`createPinnedFetch` (on `@owlmeans/file-streams/fetch`) is `fetch` over this client: it asks with
|
|
65
|
+
`redirect: 'manual'` (unless the caller follows) and `anyStatus`, so the caller sees what a browser's
|
|
66
|
+
fetch would, while every connection — every redirect hop the caller takes, every resolver step —
|
|
67
|
+
goes to an address the policy resolved and checked a moment before. That closes the DNS-rebinding
|
|
68
|
+
window a guard-then-`fetch` pair leaves open: llm-reader injects it as its transport and hands it to
|
|
69
|
+
the Playwright renderer, which fulfils a page's sub-requests through it. Its module graph carries no
|
|
70
|
+
AWS SDK, so a consumer that only reads the web does not load one (`tests/fetch.spec.ts` walks the
|
|
71
|
+
built graph).
|
|
72
|
+
|
|
73
|
+
- `createPinnedFetch({ network, proxies, resolvers, timeouts, userAgent, headers, client? })` — the
|
|
74
|
+
client's options, or a ready `client`. Any status is a `Response` with `url` and `redirected` set;
|
|
75
|
+
`redirect: 'manual'` returns the 3xx, `'error'` rejects `HttpError('redirect')`; an abort ends a
|
|
76
|
+
request waiting for its head and a body in flight.
|
|
77
|
+
- Only GET and HEAD (another method, or a body, rejects `HttpError('method:…' | 'body')`);
|
|
78
|
+
`Accept-Encoding` is always identity, hop-by-hop, `host` and `content-length` request headers are
|
|
79
|
+
dropped; a transport-decoded body loses its `Content-Encoding`/`Content-Length`; a 204/205/304
|
|
80
|
+
answer has a null body.
|
|
81
|
+
- A policy refusal rejects with `NetworkPolicyError`, whose `refused === true` tells a consumer
|
|
82
|
+
(llm-reader maps it to `ReaderRefusedError('pinned:…')`) a refusal from a network failure.
|
|
83
|
+
|
|
33
84
|
## Google Drive
|
|
34
85
|
|
|
35
|
-
Share links (`/file/d/<id>/…`, `open?id=`, `uc?id
|
|
36
|
-
the "can't scan this file for viruses" page is followed
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
86
|
+
Share links (`/file/d/<id>/…`, `open?id=`, `uc?id=`, `resourcekey` kept) become
|
|
87
|
+
`drive.usercontent.google.com/download`; the "can't scan this file for viruses" page is followed
|
|
88
|
+
through its own `download-form` (the action must be one of the resolver's Google origins; every
|
|
89
|
+
hidden input carried), then the legacy `confirm=` link, then a `download_warning` cookie — at most
|
|
90
|
+
3 steps, 256 KiB of a page read. Docs, Sheets, Slides and Drawings links become export URLs (docx,
|
|
91
|
+
xlsx, pptx, png by default, `exports` changes them; csv/tsv keep the `gid`). Folders, quota pages,
|
|
92
|
+
sign-in redirects and forms posting elsewhere fail as `ResolverError('google-drive:<reason>')`
|
|
93
|
+
(`folder`, `quota`, `not-public`, `foreign-form`, `export-too-large`, `unrecognized`). A range
|
|
94
|
+
request resolves first, then asks the bytes' own URL for the range; `raw: true` skips resolvers. The
|
|
95
|
+
page flow is the one gdown documents; it is reverse-engineered, so `FILE_STREAMS_LIVE_DRIVE_URL`
|
|
96
|
+
exercises it against Google.
|
|
42
97
|
|
|
43
98
|
## External docs
|
|
44
99
|
|