@owlmeans/file-streams 0.2.0-alpha.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/README.md +437 -21
  2. package/agent-meta/manifest.json +77 -8
  3. package/agent-meta/skills/file-streams/SKILL.md +84 -105
  4. package/agent-meta/skills/file-streams/parts/archives.md +80 -0
  5. package/agent-meta/skills/file-streams/parts/extending.md +117 -0
  6. package/agent-meta/skills/file-streams/parts/network.md +77 -22
  7. package/agent-meta/skills/file-streams/parts/workload.md +51 -0
  8. package/agent-meta/skills/file-streams/parts/workspaces.md +9 -22
  9. package/build/errors.d.ts +5 -0
  10. package/build/errors.d.ts.map +1 -1
  11. package/build/errors.js +5 -0
  12. package/build/errors.js.map +1 -1
  13. package/build/fetch.d.ts +12 -0
  14. package/build/fetch.d.ts.map +1 -0
  15. package/build/fetch.js +6 -0
  16. package/build/fetch.js.map +1 -0
  17. package/build/http/types.d.ts +19 -1
  18. package/build/http/types.d.ts.map +1 -1
  19. package/build/http.d.ts.map +1 -1
  20. package/build/http.js +7 -4
  21. package/build/http.js.map +1 -1
  22. package/build/index.d.ts +2 -0
  23. package/build/index.d.ts.map +1 -1
  24. package/build/index.js +1 -0
  25. package/build/index.js.map +1 -1
  26. package/build/pinned-fetch/consts.local.d.ts +9 -0
  27. package/build/pinned-fetch/consts.local.d.ts.map +1 -0
  28. package/build/pinned-fetch/consts.local.js +13 -0
  29. package/build/pinned-fetch/consts.local.js.map +1 -0
  30. package/build/pinned-fetch/types.d.ts +16 -0
  31. package/build/pinned-fetch/types.d.ts.map +1 -0
  32. package/build/pinned-fetch/types.js +2 -0
  33. package/build/pinned-fetch/types.js.map +1 -0
  34. package/build/pinned-fetch.d.ts +9 -0
  35. package/build/pinned-fetch.d.ts.map +1 -0
  36. package/build/pinned-fetch.js +82 -0
  37. package/build/pinned-fetch.js.map +1 -0
  38. package/package.json +18 -11
  39. package/src/errors.ts +6 -0
  40. package/src/fetch.ts +14 -0
  41. package/src/http/types.ts +21 -1
  42. package/src/http.ts +7 -4
  43. package/src/index.ts +2 -0
  44. package/src/pinned-fetch/consts.local.ts +15 -0
  45. package/src/pinned-fetch/types.ts +17 -0
  46. package/src/pinned-fetch.ts +93 -0
  47. package/tests/fetch.spec.ts +163 -0
package/README.md CHANGED
@@ -1,59 +1,475 @@
1
1
  # @owlmeans/file-streams
2
2
 
3
- Online files as Node streams, for OwlMeans servers and agents (Node and Bun):
3
+ Moves files as Node streams on OwlMeans servers and agents (Node and Bun). Bytes come from URLs, S3
4
+ objects, files, memory, foreign streams or a Viable workload, and go to files, folders, S3, memory
5
+ or a workload. Every outbound connection is pinned to an address the network policy checked a moment
6
+ before, every transfer runs under byte and time limits, and every file can be verified from its head
7
+ before the first byte lands.
4
8
 
5
9
  - **Sources** — a URL (under a connect-time SSRF policy, through HTTP/SOCKS proxy plugins, Google
6
- Drive links resolved), an S3 object, a file, bytes in memory, a foreign stream.
10
+ Drive and Docs links resolved), an S3 object, a file, bytes in memory, a foreign stream (Node, web
11
+ or async iterable), a Viable workload file.
7
12
  - **Sinks** — an atomic file, a confined directory tree, an S3 object or prefix (addressed exactly
8
- like `@owlmeans/storage-resource`), a buffer, a Viable ephemeral workload (which is also a source).
13
+ like `@owlmeans/storage-resource`), a buffer, a Viable ephemeral workload.
14
+ - **Transfers** — one file from a source to a sink under size, time and idle limits, its head
15
+ verified by `@owlmeans/file-inspector` before the sink sees a byte; progress and SHA-256 on request.
16
+ - **Archives** — zip, tar, tar.gz and gz: detect, list, extract, pack, with traversal, link, bomb,
17
+ overlap and duplicate guards.
9
18
  - **Workspaces** — a service's temporary files, one confined folder per organization / project /
10
- purpose, optionally mirrored to an S3 prefix under the same nesting so a restarted pod restores
11
- them; whole S3 prefixes listed and deleted.
12
- - **Archives** — zip, tar, tar.gz and gz: detect, list, extract, pack — with traversal, link, bomb and
13
- overlap guards.
14
- - **Verification** — what a file really is (images, audio, video, pdf, office documents including
15
- legacy doc/xls/ppt, csv, text, source code), compared with what its name and media type claim,
16
- from its head or a few range reads — never a full download.
19
+ purpose, optionally mirrored to an S3 prefix so a restarted pod restores them; whole S3 prefixes
20
+ listed and deleted.
21
+ - **Pinned fetch** — `@owlmeans/file-streams/fetch` (no AWS SDK in its graph): a WHATWG `fetch`
22
+ over the same client, for code written against `fetch` (`@owlmeans/llm-reader`, a renderer).
23
+ - **Verification** — the `@owlmeans/file-inspector` inspector, kinds, tables and refusals are
24
+ re-exported; every source here is a probe source as it is.
25
+ - **Context service** — `appendFileStreamsService` builds all of the above from `cfg.fileStreams`
26
+ and `cfg.storageBuckets`.
17
27
 
18
28
  ## Installation
19
29
 
20
30
  ```bash
21
- bun add @owlmeans/file-streams@^0.2.0-alpha.0
31
+ bun add @owlmeans/file-streams@^0.2.0
22
32
  ```
23
33
 
24
- ## Usage
34
+ No peer dependencies. `@aws-sdk/client-s3`, `socks`, `tar-stream`, `yauzl` and `yazl` are regular
35
+ dependencies. The `@owlmeans/file-streams/fetch` subpath never loads the AWS SDK,
36
+ `@owlmeans/storage-resource` or the S3 modules.
37
+
38
+ ## Quick start
25
39
 
26
40
  ```typescript
27
41
  import { createDirectorySink, createS3Bucket, FileKind, fileInspector, httpClient, transferHelper } from '@owlmeans/file-streams'
28
42
 
29
- // Download an image into a folder — refused before a byte is written unless it is an image
43
+ // Download an image into a folder: refused before a byte is written unless it is an image
30
44
  await transferHelper.copyInto(httpClient.source('https://example.com/hero.png'), createDirectorySink('/work/assets'), {
31
45
  verify: { allow: [FileKind.Image] },
32
46
  })
33
47
 
34
- // Check a remote file without downloading it
48
+ // Check a remote file without downloading it (a few range reads, at most 1 MiB)
35
49
  const verdict = await fileInspector.probe(httpClient.source('https://example.com/report.pdf'))
36
50
  verdict.status // 'match' | 'mismatch' | 'inconclusive'
37
51
  verdict.detected // { ext: 'pdf', mime: 'application/pdf', kind: 'pdf', … }
38
52
 
39
53
  // Store it in the bucket a cfg.storageBuckets entry names
40
54
  const bucket = createS3Bucket(cfg.storageBuckets.media)
41
- await transferHelper.copy(httpClient.source('https://example.com/report.pdf'), bucket.sink('reports/q3.pdf'))
55
+ const { sink } = await transferHelper.copy(httpClient.source('https://example.com/report.pdf'), bucket.sink('reports/q3.pdf'))
56
+ sink.url // the same public URL storage-resource answers for that prefix
57
+ ```
58
+
59
+ ## Usage
60
+
61
+ ### Sources and sinks
62
+
63
+ ```typescript
64
+ import {
65
+ createBufferSink, createBufferSource, createFileSink, createFileSource, createStreamSource, transferHelper,
66
+ } from '@owlmeans/file-streams'
67
+
68
+ const file = createFileSource('/spool/upload-17', { name: 'scan.pdf' }) // re-openable, random access
69
+ const shot = createBufferSource(png, { name: 'shot.png', mime: 'image/png' }) // re-openable, random access
70
+ const piped = createStreamSource(response.body, { name: 'feed.csv' }) // one-shot, no random access
71
+
72
+ await transferHelper.copy(file, createFileSink('/data/scan.pdf', { onConflict: 'overwrite' }))
73
+ const memory = createBufferSink({ maxBytes: 8 * 1024 * 1024 })
74
+ await transferHelper.copy(shot, memory)
75
+ memory.bytes()
76
+ ```
77
+
78
+ A `FileSource` answers `info()` (cached metadata), `open({ range? })` (a fresh Node `Readable`) and
79
+ `ranges()` (random access over one immutable version, or `null`). A `FileSink` writes one stream
80
+ and commits atomically. A `TreeSink` hands out the sink of each entry under one root.
81
+
82
+ ### Transfers
83
+
84
+ ```typescript
85
+ import { createFileSink, FileKind, httpClient, transferHelper } from '@owlmeans/file-streams'
86
+
87
+ const result = await transferHelper.copy(httpClient.source(url), createFileSink('/data/report.pdf'), {
88
+ verify: { allow: [FileKind.Pdf] }, // default {}: mismatches refused; false skips it
89
+ limits: { maxBytes: 50 * 1024 * 1024, timeoutMs: 120_000, idleTimeoutMs: 15_000 },
90
+ hash: 'sha256',
91
+ onProgress: ({ bytes, total, elapsedMs }) => report(bytes, total, elapsedMs),
92
+ info: { name: 'report.pdf' }, // overrides the source's claims
93
+ })
94
+ result.bytes // counted bytes
95
+ result.sha256 // hex digest
96
+ result.verdict // the head's verdict
97
+ result.sink // { kind, location, bytes, url?, etag?, mime? }
42
98
  ```
43
99
 
100
+ `copy` reads the head of the very stream it moves, verifies it and replays it into the sink: there
101
+ is no second download. `copyInto(source, tree)` writes under the source's sanitized name, or
102
+ `download` when nothing names it.
103
+
104
+ ### HTTP client
105
+
106
+ ```typescript
107
+ import { createHttpClient } from '@owlmeans/file-streams'
108
+
109
+ const client = createHttpClient({
110
+ network: { allowHosts: ['*.example.com'], ports: [443] },
111
+ timeouts: { connectMs: 5_000 },
112
+ maxRedirects: 3,
113
+ headers: { authorization: `Bearer ${token}` }, // dropped on a cross-origin redirect
114
+ sensitiveHeaders: ['x-api-key'],
115
+ })
116
+
117
+ const page = await client.request('https://api.example.com/missing', { anyStatus: true }) // a 404 is an answer
118
+ const hop = await client.request(url, { redirect: 'manual' }) // a 3xx comes back with its Location
119
+ const source = client.source(url, { name: 'export.csv' }) // Range-capable when the server answers 206
120
+ ```
121
+
122
+ Without `anyStatus`, a non-2xx answer throws `HttpError('status:<code>')`, and `error.status` reads
123
+ the code back.
124
+
125
+ ### Pinned fetch
126
+
127
+ ```typescript
128
+ import { createPinnedFetch } from '@owlmeans/file-streams/fetch'
129
+
130
+ const pinnedFetch = createPinnedFetch({ network: { denyHosts: ['*.corp.example'] }, userAgent: 'my-reader' })
131
+ const response = await pinnedFetch('https://example.com/article', { headers: { accept: 'text/html' } })
132
+ const html = await response.text()
133
+ const hop = await pinnedFetch(url, { redirect: 'manual' }) // the 3xx itself, Location included
134
+ ```
135
+
136
+ Any status is a `Response`, with `url` and `redirected` set. Only GET and HEAD are carried; another
137
+ method or a request body rejects with `HttpError`. A policy refusal rejects with
138
+ `NetworkPolicyError`, whose `refused === true` tells it apart from a network failure.
139
+
140
+ ### Proxies
141
+
142
+ ```typescript
143
+ import { createHttpClient } from '@owlmeans/file-streams'
144
+
145
+ // A list: round-robin (default), random, failover or by-key
146
+ const pooled = createHttpClient({
147
+ proxies: { proxies: ['socks5://u:p@egress-1:1080', 'http://egress-2:3128'], strategy: 'by-key', bypass: ['*.example.org'] },
148
+ })
149
+ await pooled.request(url, { proxyKey: tenantKey })
150
+
151
+ // A callback by key and URL; credentials looked up for endpoints that carry none
152
+ const provided = createHttpClient({
153
+ proxies: {
154
+ proxies: async ({ key, url }) => await egress.proxyFor(key, url.hostname),
155
+ credentials: async (endpoint, { key }) => await egress.credentialsFor(endpoint.host, key),
156
+ },
157
+ })
158
+ ```
159
+
160
+ Built-in plugins speak `http`/`https` (CONNECT tunnels; plain-http targets forwarded in absolute
161
+ form unless `createHttpProxyPlugin({ httpTargets: 'connect' })`) and `socks4`/`socks4a`/`socks5`/
162
+ `socks5h`. A failed proxy cools down for 30 s and the next one is tried once.
163
+
164
+ ### Google Drive and Docs
165
+
166
+ ```typescript
167
+ import { createGoogleDriveResolver, createHttpClient, httpClient } from '@owlmeans/file-streams'
168
+
169
+ // On by default: a share link downloads the file behind it, through the virus-scan page if needed
170
+ const response = await httpClient.request('https://drive.google.com/file/d/<file-id>/view?usp=sharing')
171
+
172
+ // Editor files export as docx, xlsx, pptx and png by default
173
+ const exporting = createHttpClient({ resolvers: [createGoogleDriveResolver({ exports: { document: 'pdf', spreadsheet: 'csv' } })] })
174
+ ```
175
+
176
+ `resolvers: []` turns resolving off; a list you pass replaces the default one.
177
+
178
+ ### S3 buckets
179
+
180
+ ```typescript
181
+ import { createBufferSource, createS3Bucket, transferHelper } from '@owlmeans/file-streams'
182
+
183
+ const bucket = createS3Bucket(cfg.storageBuckets.media, { partSize: 16 * 1024 * 1024, concurrency: 4 })
184
+ const { sink } = await transferHelper.copy(
185
+ createBufferSource(png, { name: 'shot.png' }), bucket.sink(`shots/${id}.png`, { cacheControl: 'public, max-age=86400' })
186
+ )
187
+ bucket.instanceOf(sink, 'original') // a storage-common StoredFileInstance
188
+ await bucket.head(`shots/${id}.png`) // { key, size, etag, mime, modifiedAt } or null
189
+ await bucket.list('shots') // [{ key, path, size, etag, modifiedAt }, …]
190
+ await bucket.removePrefix('shots') // how many objects went
191
+ bucket.close()
192
+ ```
193
+
194
+ A prefix lands at `<basePrefix>/<prefix>` and answers on the same public URL as
195
+ `@owlmeans/storage-resource`. A body that fits one part is one PutObject; a larger one is a
196
+ multipart upload, aborted on failure.
197
+
198
+ ### Archives
199
+
200
+ ```typescript
201
+ import {
202
+ ArchiveFormat, archiveHelper, createDirectorySink, createFileSource, FileKind, httpClient, transferHelper,
203
+ } from '@owlmeans/file-streams'
204
+
205
+ const result = await archiveHelper.extract(httpClient.source(zipUrl), createDirectorySink('/work/unpacked'), {
206
+ verify: { refuse: [FileKind.Executable] }, // entries are verified only when verify is given
207
+ onEntryError: 'skip', // a write or verification failure skips the entry
208
+ strip: 1,
209
+ limits: { maxEntries: 500, maxTotalBytes: 256 * 1024 * 1024 },
210
+ })
211
+ result.format // ArchiveFormat.Zip
212
+ result.extracted // [{ entry, path, result }, …]
213
+ result.skipped // [{ entry, reason }, …]
214
+
215
+ const entries = await archiveHelper.list(createFileSource('/spool/bundle.tar.gz'))
216
+ const packed = archiveHelper.pack(ArchiveFormat.Zip, [
217
+ { path: 'report.pdf', source: createFileSource('/data/report.pdf') },
218
+ { path: 'notes.txt', source: new TextEncoder().encode('hello') },
219
+ ], { name: 'bundle.zip' })
220
+ await transferHelper.copy(packed, bucket.sink('exports/bundle.zip'))
221
+ ```
222
+
223
+ The format comes from the bytes. A zip is read from its central directory, through range reads when
224
+ the source has them and from a private temporary spool otherwise.
225
+
226
+ ### Workspaces
227
+
44
228
  ```typescript
45
229
  import { createBackedWorkspace, createBufferSource, createS3Bucket, createWorkspace } from '@owlmeans/file-streams'
46
230
 
47
- // Keep an upload in the service's tmp folder and in the bucket, under <org>/<project>/<purpose>
48
- const backed = createBackedWorkspace({ workspace: createWorkspace(), bucket: createS3Bucket(cfg.storageBuckets.projects) })
49
- const files = backed.scope({ organization: 'acme', project: 'slot-12', purpose: 'new-project' })
231
+ const workspace = createWorkspace() // <os.tmpdir()>/owlmeans-workspace
232
+ const local = workspace.scope({ organization: entitySlug, project: 'slot-12', purpose: 'new-project' })
233
+ await local.list() // [{ path, size, modifiedAt }, …]
234
+ await workspace.prune({ olderThanMs: 24 * 3600_000 }) // whole scopes untouched for a day
235
+
236
+ // Mirrored to a bucket under the same nesting
237
+ const backed = createBackedWorkspace({ workspace, bucket: createS3Bucket(cfg.storageBuckets.projects), prefix: 'materials' })
238
+ const files = backed.scope({ organization: entitySlug, project: 'slot-12', purpose: 'new-project' })
50
239
  await files.put('brand/logo.png', createBufferSource(png, { name: 'logo.png' }))
51
240
  await files.restore() // after a restart: download what the tmp folder lost
52
241
  await files.drop() // the folder and every object under the scope's prefix
53
242
  ```
54
243
 
55
- Proxies, the Viable workload destination and source, workspaces, archive limits, the context service
56
- and every default are described in the package skill (`file-streams`).
244
+ ### Viable workloads
245
+
246
+ ```typescript
247
+ import { archiveHelper, FileKind, httpClient, makeWorkloadSink, makeWorkloadSource, transferHelper } from '@owlmeans/file-streams'
248
+
249
+ const media = makeWorkloadSink(fileHelper, { root: 'public/media' }) // a WorkloadWriter: writeBinaryFile(filePath, base64)
250
+ await transferHelper.copy(httpClient.source(imageUrl), media.entry('hero.png'), { verify: { allow: [FileKind.Image] } })
251
+ await archiveHelper.extract(httpClient.source(zipUrl), media, { onEntryError: 'skip' })
252
+
253
+ const materials = makeWorkloadSource(fileHelper, { root: 'docs/materials' }) // a WorkloadReader: readBinaryFile(filePath)
254
+ await transferHelper.copy(materials.entry('logo.png'), local.entry('logo.png'))
255
+ ```
256
+
257
+ Each file is one atomic write or read of at most 12 MiB (`WORKLOAD_MAX_FILE_BYTES`), retried once.
258
+
259
+ ### Context service
260
+
261
+ ```typescript
262
+ import { appendFileStreamsService, FILE_STREAMS_SERVICE } from '@owlmeans/file-streams'
263
+ import type { FileStreamsService } from '@owlmeans/file-streams'
264
+
265
+ appendFileStreamsService(context, FILE_STREAMS_SERVICE, { proxyCredentials: async endpoint => await egress.credentialsFor(endpoint.host) })
266
+ const files = context.service<FileStreamsService>(FILE_STREAMS_SERVICE)
267
+ await files.transfer().copy(files.http().source(url), files.bucket('media').sink('imports/a.png'))
268
+ const scope = files.backed('projects', { prefix: 'materials' }).scope({ organization: entitySlug, purpose: 'upload' })
269
+ ```
270
+
271
+ Objects are built lazily, once, from `cfg.fileStreams` and `cfg.storageBuckets`. An unknown storage
272
+ key throws `FilePropertyError('storage-config:<key>')`.
273
+
274
+ ## API
275
+
276
+ Main entry `@owlmeans/file-streams`:
277
+
278
+ | Export | What it is |
279
+ |---|---|
280
+ | `httpClient` / `createHttpClient(options?)` | `request(url, options?)` → `HttpResponse`; `source(url, options?)` → a re-openable, Range-capable `FileSource` |
281
+ | `createPinnedFetch(options?)` | A WHATWG `fetch` over the client (also on `./fetch`) |
282
+ | `networkPolicy` / `createNetworkPolicy(options?)` | `assertUrl(url, from?)`, `resolve(hostname)`, `allowsAddress(address)`, `trustProxyDns` |
283
+ | `createProxyRouter(options?)` | Picks a proxy per connection: `select(request)`, `report(endpoint, ok)`, `retries` |
284
+ | `createHttpProxyPlugin(options?)` / `httpProxyPlugin` | The `http`/`https` proxy plugin (CONNECT and forward mode) |
285
+ | `createSocksProxyPlugin(options?)` / `socksProxyPlugin` | The `socks4`/`socks4a`/`socks5`/`socks5h` proxy plugin |
286
+ | `parseProxyEndpoint(value)` | A proxy URL (`socks5h://user:pass@host:1080`) as a `ProxyEndpoint` |
287
+ | `googleDriveResolver` / `createGoogleDriveResolver(options?)` | Drive share links and Docs, Sheets, Slides, Drawings exports as a `UrlResolver` |
288
+ | `transferHelper` / `createTransferHelper(options?)` | `copy(source, sink, options?)`, `copyInto(source, tree, options?)` |
289
+ | `archiveHelper` / `createArchiveHelper(options?)` | `detect`, `list`, `extract`, `pack`, `gzip`, `gunzip` |
290
+ | `createS3Bucket(config, options?)` | A storage entry or bucket model as `source`, `sink`, `tree`, `head`, `remove`, `list`, `removePrefix`, `keyOf`, `urlOf`, `instanceOf`, `close`, `model` |
291
+ | `createFileSource(path, claims?)` | A regular file: re-openable, random access, `<mtimeMs>-<size>` validator |
292
+ | `createBufferSource(bytes, claims?)` | Bytes in memory: re-openable, random access |
293
+ | `createStreamSource(input, claims?)` | A Node, web or async-iterable stream: one-shot, no random access |
294
+ | `createFileSink(path, options?)` | One file written to a private temporary file and committed atomically |
295
+ | `createDirectorySink(root, options?)` | A `TreeSink` confined to `root`: no traversal, no symlinked parents, no duplicates |
296
+ | `createBufferSink(options?)` | Collects one file in memory (64 MiB default ceiling); `bytes()` |
297
+ | `makeWorkloadSink(writer, options?)` | A Viable workload folder as a `TreeSink` through `writeBinaryFile` |
298
+ | `makeWorkloadSource(reader, options?)` | Viable workload files as sources through `readBinaryFile`: `entry(path, claims?)` |
299
+ | `createWorkspace(options?)` | Scoped temporary folders: `scope`, `scopes`, `prune`, `root` |
300
+ | `createBackedWorkspace(options)` | Scopes mirrored to a bucket: `scope(value)` → `put`, `restore`, `list`, `remove`, `evict`, `drop`, `key`, `prefixOf`, `local` |
301
+ | `createFileStreamsService(alias?, options?)` | The context service object |
302
+ | `appendFileStreamsService(ctx, alias?, options?)` | Registers the service (alias `file-streams`) |
303
+ | `FILE_STREAMS_SERVICE` | `'file-streams'` |
304
+ | `FileStreamsError` | Base `ResilientError`, `file-streams:<area>:<reason>` |
305
+ | `NetworkPolicyError` | `policy:<reason>` — `scheme`, `credentials`, `downgrade`, `blocked-host`, `port`, `unresolvable`, `blocked-address`; `refused === true` |
306
+ | `HttpError` | `http:<reason>` — `status:<code>` (`.status`), `redirects`, `location`, `connect`, `connect-timeout`, `tls:<code>`, `tls-timeout`, `headers-timeout`, `truncated`, `encoding`, `request`, `method:<m>`, `body`, `redirect` |
307
+ | `ProxyError` | `proxy:<reason>` — `no-plugin`, `endpoint`, `connect`, `handshake`, `auth-failed`, `refused:<code>` |
308
+ | `LimitError` | `limit:<reason>` — `max-bytes`, `timeout`, `idle-timeout`, `entries`, `entry-bytes`, `total-bytes`, `ratio`, `s3-parts` |
309
+ | `SourceError` | `source:<reason>` — `consumed`, `changed`, `not-found`, `not-a-file`, `range-unsupported` |
310
+ | `PathError` | `path:<reason>` — `absolute`, `traversal`, `invalid-char`, `too-long`, `too-deep`, `empty`, `symlink`, `exists`, `duplicate`, `reserved`, `scope-overlap` |
311
+ | `ArchiveError` | `archive:<reason>` — `format`, `corrupt`, `encrypted`, `link`, `overlap`, `method:<n>`, `gzip-entries` |
312
+ | `ResolverError` | `resolver:<name>:<reason>` — `folder`, `not-public`, `quota`, `export-too-large`, `foreign-form`, `foreign-follow`, `steps`, `unrecognized` |
313
+ | `WorkloadError` | `workload:<reason>` — `too-large`, `write-failed`, `read-failed` |
314
+ | `S3TransferError` | A storage `StorageApiError`: `s3:<op>` — `head`, `get`, `put`, `multipart`, `delete`, `list`, `delete-objects:<code>` |
315
+ | `ArchiveFormat` | `zip`, `tar`, `tar.gz`, `gz` |
316
+ | `DEFAULT_ARCHIVE_LIMITS` / `RATIO_FLOOR_BYTES` / `ZIP_LOCAL_HEADER_BYTES` / `NESTED_ARCHIVE_EXTENSIONS` | Archive limits; ratios judged past 1 MiB; zip local header size; names that mark a nested archive |
317
+ | `DENIED_IPV4_CIDRS` / `DENIED_IPV6_CIDRS` / `DENIED_HOST_SUFFIXES` / `DEFAULT_SCHEMES` | What the default policy refuses; `https:`, `http:` |
318
+ | `GOOGLE_DRIVE_RESOLVER` / `DEFAULT_GOOGLE_ORIGINS` / `DEFAULT_EXPORT_FORMATS` / `DRIVE_ID_PATTERN` / `MAX_DRIVE_STEPS` / `DEFAULT_DRIVE_PAGE_BYTES` | The Drive resolver's name, origins, export formats, id shape, 3 steps, 256 KiB page read |
319
+ | `DEFAULT_PART_SIZE` / `MIN_PART_SIZE` / `MAX_PARTS` / `DEFAULT_UPLOAD_CONCURRENCY` / `S3_DELETE_BATCH_SIZE` | 8 MiB, 5 MiB, 10 000, 2, 1000 |
320
+ | `WORKLOAD_MAX_BASE64_CHARS` / `WORKLOAD_MAX_FILE_BYTES` | 16 MiB of base64; 12 MiB per file |
321
+ | `DEFAULT_TRANSFER_LIMITS` / `FALLBACK_FILE_NAME` | 512 MiB, 10 min, 30 s idle; `download` |
322
+ | `DEFAULT_WORKSPACE_DIR` / `WORKSPACE_SCOPE_MARKER` / `DEFAULT_WORKSPACE_DIR_MODE` / `DEFAULT_WORKSPACE_FILE_MODE` | `owlmeans-workspace`, `.owlmeans-scope`, `0o700`, `0o600` |
323
+ | `fileInspector` / `createFileInspector` / `FileKind` / `FileMimicryError` / `FileKindError` | Re-exported from `@owlmeans/file-inspector` |
324
+ | `EXTENSION_MIME`, `KIND_BY_MIME`, `KIND_BY_MIME_PREFIX`, `MIME_ALIASES`, `NON_INFORMATIVE_MIMES`, `CODE_LANGUAGE_BY_EXT`, `SHEBANG_LANGUAGES`, `TEXT_FAMILY`, `PASSIVE_KINDS`, `OFFICE_KINDS`, `EXTENSION_GROUPS`, `CONTAINER_EXTENSIONS`, `DEFAULT_PROBE_BYTES`, `DEFAULT_SAMPLE_BYTES`, `PDF_TRAILER_BYTES` | Re-exported `@owlmeans/file-inspector` tables and defaults |
325
+ | `fileScopeHelper` / `FileScopeSchema` / `FileScopeError` | Re-exported from `@owlmeans/storage-common`: the `<organization>/<project?>/<purpose>` nesting |
326
+ | Types: sources and sinks | `FileSource`, `FileSink`, `TreeSink`, `FileInfo`, `ByteRange`, `SignalOptions`, `OpenOptions`, `OpenedFile`, `RangeReader`, `SinkInput`, `SinkWriteOptions`, `SinkResult`, `SourceClaims`, `StreamInput`, `ConflictPolicy`, `FileSinkOptions`, `DirectorySinkOptions`, `BufferSinkOptions`, `BufferSink` |
327
+ | Types: HTTP and fetch | `HttpClient`, `HttpClientOptions`, `HttpTimeouts`, `HttpMethod`, `HttpRedirectMode`, `HttpRequestOptions`, `HttpResponse`, `HttpSourceOptions`, `PinnedFetch`, `PinnedFetchOptions` |
328
+ | Types: network and proxies | `NetworkPolicy`, `NetworkPolicyOptions`, `AddressResolver`, `ResolvedAddress`, `ProxyEndpoint`, `ProxyCredentials`, `ProxyRequest`, `ProxyProvider`, `ProxyCredentialsProvider`, `ProxyStrategy`, `ProxyRouterOptions`, `ProxySelection`, `ProxyRouter`, `TunnelTarget`, `ProxyConnectContext`, `ProxyConnection`, `ProxyPlugin`, `HttpProxyPluginOptions`, `SocksProxyPluginOptions` |
329
+ | Types: resolvers | `UrlResolver`, `ResolverStep`, `ResolverAccept`, `ResolverFollow`, `ResolverFail`, `ResolverResponseHead`, `ResolverBodyReader`, `GoogleDriveResolverOptions`, `GoogleExportFormats`, `GoogleDriveOrigins` |
330
+ | Types: transfers and archives | `TransferHelper`, `TransferHelperOptions`, `TransferOptions`, `TransferLimits`, `TransferProgress`, `TransferProgressListener`, `TransferResult`, `ArchiveHelper`, `ArchiveHelperOptions`, `ArchiveLimits`, `ArchiveEntryType`, `ArchiveEntryInfo`, `ArchiveListOptions`, `ArchiveExtractOptions`, `ArchivePackOptions`, `ExtractedEntry`, `SkippedEntry`, `ExtractResult`, `PackEntry` |
331
+ | Types: S3 and workloads | `S3Bucket`, `S3BucketOptions`, `S3SinkOptions`, `S3ObjectInfo`, `S3ListedObject`, `WorkloadWriter`, `WorkloadSinkOptions`, `WorkloadReader`, `WorkloadSourceOptions`, `WorkloadSource` |
332
+ | Types: workspaces | `Workspace`, `WorkspaceOptions`, `WorkspaceScope`, `WorkspaceFile`, `WorkspaceWriteOptions`, `WorkspacePruneOptions`, `BackedWorkspace`, `BackedWorkspaceOptions`, `BackedScope`, `BackedPutOptions`, `BackedPutResult`, `BackedRestoreResult`, `FileScope`, `FileScopeHelper` |
333
+ | Types: service | `FileStreamsService`, `FileStreamsServiceOptions`, `FileStreamsConfig`, `FileStreamsConfigAppend`, `FileStreamsProxyConfig`, `FileStreamsBackedOptions` |
334
+ | Types: verification | `FileVerdict`, `VerdictStatus`, `DeclaredType`, `DetectedType`, `FileDetails`, `FileClaims`, `VerifyPolicy`, `ProbeOptions`, `InspectOptions`, `FileInspector`, `FileInspectorOptions` (from `@owlmeans/file-inspector`) |
335
+
336
+ Subpath `@owlmeans/file-streams/fetch`:
337
+
338
+ | Export | What it is |
339
+ |---|---|
340
+ | `createPinnedFetch(options?)` | The pinned WHATWG `fetch` |
341
+ | `createHttpClient(options?)` | The HTTP client it runs on |
342
+ | `createNetworkPolicy(options?)` | The policy |
343
+ | `createGoogleDriveResolver(options?)` / `googleDriveResolver` | The Drive resolver |
344
+ | `HttpError` / `NetworkPolicyError` / `ProxyError` / `ResolverError` | The errors a fetch rejects with |
345
+ | Types | `PinnedFetch`, `PinnedFetchOptions`, `HttpClient`, `HttpClientOptions`, `HttpTimeouts`, `NetworkPolicy`, `NetworkPolicyOptions`, `AddressResolver`, `ResolvedAddress`, `ProxyRouter`, `ProxyRouterOptions`, `UrlResolver`, `GoogleDriveResolverOptions` |
346
+
347
+ ## Configuration
348
+
349
+ ### `cfg.fileStreams`
350
+
351
+ ```typescript
352
+ import type { FileStreamsConfig } from '@owlmeans/file-streams'
353
+
354
+ const fileStreams: FileStreamsConfig = {
355
+ limits: { maxBytes: 128 * 1024 * 1024 }, // transfers
356
+ network: { allowCidrs: ['10.20.0.0/16'] }, // NetworkPolicyOptions without resolver
357
+ proxies: [{ url: 'socks5://egress:1080', username: 'svc', password: '/run/secrets/egress-password' }],
358
+ proxyStrategy: 'failover',
359
+ userAgent: 'my-service',
360
+ workspace: { root: 'my-service-workspace' }, // relative to os.tmpdir()
361
+ }
362
+ ```
363
+
364
+ A server context reads every string leaf that starts with `/` as a file, which is why proxies are
365
+ objects (their secrets can be mounted files) and why `workspace.root` is given relative to
366
+ `os.tmpdir()`. Callbacks, plugins and other non-config objects go through the service options:
367
+ `proxies` (replaces the configured list), `proxyCredentials`, `proxyPlugins`, `resolvers`,
368
+ `resolver` (DNS), `bucketOptions`, `inspector`, `archiveLimits`. Buckets come from
369
+ `cfg.storageBuckets` entries, read exactly as `@owlmeans/storage-resource` reads them.
370
+
371
+ ### Network policy (`NetworkPolicyOptions`)
372
+
373
+ | Option | Default | Effect |
374
+ |---|---|---|
375
+ | `schemes` | `['https:', 'http:']` | URL schemes allowed |
376
+ | `allowPrivate` | `false` | Allow loopback, private, link-local, CGNAT, multicast, reserved ranges and internal names |
377
+ | `allowCidrs` | none | Allowed even inside the denied ranges (`'127.0.0.1/32'` for a local stand-in) |
378
+ | `denyCidrs` | none | Extra ranges to deny |
379
+ | `allowHosts` | any | Only these hosts (`x.com`, `*.x.com`); an internal name listed here is allowed |
380
+ | `denyHosts` | none | Hosts never reached |
381
+ | `ports` | any | Ports allowed |
382
+ | `allowDowngrade` | `false` | Follow an https → http redirect |
383
+ | `allowUrlCredentials` | `false` | Accept `user:pass@` in URLs |
384
+ | `trustProxyDns` | `false` | Accept a name the local resolver cannot resolve when a remote-DNS proxy will |
385
+ | `resolver` | `dns.lookup` (all addresses) | Your own resolver; it must answer every address of the name |
386
+
387
+ `allowCidrs` wins over `denyCidrs`, which wins over `allowPrivate`.
388
+
389
+ ### HTTP client (`HttpClientOptions`)
390
+
391
+ | Option | Default |
392
+ |---|---|
393
+ | `network` | The default policy (a `NetworkPolicy` or its options) |
394
+ | `proxies` | None: direct connections (a `ProxyRouter` or its options) |
395
+ | `resolvers` | `[googleDriveResolver]`; `[]` turns resolving off |
396
+ | `timeouts` | `connectMs` 10 000 · `headersMs` 30 000 · `idleMs` 30 000 |
397
+ | `maxRedirects` | 5 |
398
+ | `userAgent` | `owlmeans-file-streams` |
399
+ | `headers` | Sent with every request; `Accept-Encoding` defaults to `identity` |
400
+ | `sensitiveHeaders` | Dropped on a cross-origin redirect, beside `authorization`, `cookie`, `proxy-authorization` |
401
+ | `ca` | Extra CAs; certificate verification is never turned off |
402
+
403
+ ### Proxies (`ProxyRouterOptions`)
404
+
405
+ `proxies` (endpoints, proxy URLs or a provider callback), `credentials`, `strategy`
406
+ (`round-robin` default, `random`, `failover`, `by-key`), `bypass` (hosts never proxied), `plugins`
407
+ (tried before the built-in ones), `retries` (1), `cooldownMs` (30 000).
408
+
409
+ ### Limits and defaults
410
+
411
+ | Area | Default |
412
+ |---|---|
413
+ | Transfer | 512 MiB · 10 min total · 30 s idle · progress every 250 ms |
414
+ | Probe | 1 MiB budget · 64 KiB head sample |
415
+ | Archives | 10 000 entries · 512 MiB per entry · 2 GiB total · ratio 200 (judged past 1 MiB) · depth 1 · paths ≤ 1024 bytes and 64 segments |
416
+ | Tar packing | Sources without a size spooled up to 512 MiB (`maxSpoolBytes`) |
417
+ | S3 | 8 MiB parts (at least 5 MiB), 2 in flight, at most 10 000 parts; DeleteObjects batches of 1000 |
418
+ | Buffer sink | 64 MiB |
419
+ | Workload | 12 MiB per file · one retry |
420
+ | Workspace | `<os.tmpdir()>/owlmeans-workspace` · folders `0o700` · files `0o600` |
421
+ | File and directory sinks | Files `0o644` · folders `0o755` · `onConflict: 'error'` |
422
+ | Google Drive | 3 follow-up steps · 256 KiB of an interstitial page read |
423
+
424
+ ## Security and limits
425
+
426
+ - **SSRF pinning.** A URL is checked before any DNS: scheme, credentials in the URL, host patterns,
427
+ port and https → http downgrade. Its name is then resolved, and every address must pass: one
428
+ refused address refuses the name. The socket connects to an address from that check, so a second
429
+ lookup cannot rebind the name. A proxied connection carries the checked address into the CONNECT
430
+ or SOCKS request. TLS keeps the host name for SNI and certificate checks.
431
+ - **Denied by default.** Loopback, private, link-local, CGNAT, benchmarking, documentation,
432
+ multicast and reserved IPv4 and IPv6 ranges. The IPv4 inside a mapped, NAT64, 6to4, Teredo or
433
+ IPv4-compatible IPv6 address is checked too. Internal names (`localhost`, `.localhost`, `.local`,
434
+ `.internal`, `.svc`, `.cluster.local`, `.home.arpa`, `.lan`, `.intranet`) are refused before DNS.
435
+ - **Where pinning stops.** A remote-DNS proxy (`socks5h`, `socks4a`, or a plugin with
436
+ `remoteDns: true`) receives the name and resolves it itself; the name is still pre-checked
437
+ locally. A plain-http target through an HTTP proxy in forward mode is resolved by the proxy;
438
+ `httpTargets: 'connect'` keeps it pinned. S3 traffic through the AWS SDK is not policed: its
439
+ endpoint is operator configuration.
440
+ - **Redirects.** Every hop is checked again, at most 5 by default. `Authorization`, `Cookie`,
441
+ `Proxy-Authorization` and `sensitiveHeaders` never cross origins. A resolver may follow only URLs
442
+ it owns, and its cookie goes only there.
443
+ - **Bodies.** `identity` encoding is requested; a transport-encoded body is decoded, but a gzip file
444
+ never is. A body shorter than its `Content-Length` fails as `truncated`, and a silent body fails on
445
+ the idle timeout. Limits count received bytes; declared sizes only refuse early.
446
+ - **Verification.** A transfer verifies the head before the sink sees a byte (on by default).
447
+ Dangerous content is always refused. The S3 sink takes the Content-Type from the verdict and writes
448
+ active svg/html as `attachment`. Archive entries are verified only when `verify` is passed.
449
+ - **Archives.** Refused: absolute paths, `..`, drive letters, control and bidi characters, overlong
450
+ or deep paths, duplicate and case-fold-twin entries, encrypted entries, unsupported compression
451
+ methods, overlapping zip entries, declared sizes that lie, an expansion ratio above the limit, and
452
+ more entries or bytes than the limits. Links are skipped (or refused with `links: 'refuse'`) and
453
+ devices skipped; neither is ever created. Nested archives stay files unless `maxDepth` is raised.
454
+ - **Writes.** A file sink writes a private temporary file (`0o600`, exclusive create) and commits it
455
+ by link (no clobbering) or rename (`overwrite`); nothing is left on failure. A directory sink
456
+ confines every entry to its root and refuses any existing symlink on the way.
457
+ - **Names.** A name from `Content-Disposition`, a URL or a key is reduced to a base name without
458
+ control or bidi characters, leading dots or trailing dots and spaces, at most 255 bytes.
459
+ - **Logs.** URLs are logged without userinfo, query or fragment, with long opaque path segments
460
+ masked. Proxy credentials and file contents are never logged.
461
+
462
+ ## Related packages
463
+
464
+ - `@owlmeans/file-inspector` — detection, verdicts, policies and safe names (`fileNameHelper`,
465
+ which this package does not re-export).
466
+ - `@owlmeans/storage-resource` — the upload resource and the bucket model `createS3Bucket` reads.
467
+ - `@owlmeans/storage-common` — stored-file types and errors, `FileScope` and `fileScopeHelper`.
468
+ - `@owlmeans/server-upload` — uses the workspace, transfer, archive and inspector objects for
469
+ multi-file uploads.
470
+ - `@owlmeans/upload` — the upload contract; its path rules match the ones this package writes with.
471
+ - `@owlmeans/llm-reader` — takes the pinned `fetch` from `@owlmeans/file-streams/fetch` as its
472
+ transport.
57
473
 
58
474
  <!-- owlmeans:agent-guidance:start -->
59
475
  ## Agent guidance
@@ -63,7 +479,7 @@ This package ships embedded agent skills under `agent-meta/`. After installing y
63
479
  your project's skill store (`.agents/skills/`):
64
480
 
65
481
  ```sh
66
- npx @owlmeans/agent-skills@^0.2.0-alpha.2
482
+ npx @owlmeans/agent-skills@^0.2.0
67
483
  ```
68
484
 
69
485
  The embedded files are version-matched to this package release. Do not edit them
@@ -1,8 +1,8 @@
1
1
  {
2
2
  "schemaVersion": 3,
3
3
  "package": "@owlmeans/file-streams",
4
- "version": "0.2.0-alpha.0",
5
- "generatedAt": "2026-11-11T15:44:02.969Z",
4
+ "version": "0.2.0",
5
+ "generatedAt": "2026-11-15T13:38:38.704Z",
6
6
  "canonicalRepo": "https://github.com/owlmeans/common",
7
7
  "entries": [
8
8
  {
@@ -12,21 +12,72 @@
12
12
  "file": "skills/file-streams/SKILL.md",
13
13
  "canonicalPath": ".agents/skills/file-streams/SKILL.md",
14
14
  "parts": [
15
+ {
16
+ "name": "archives",
17
+ "file": "skills/file-streams/parts/archives.md",
18
+ "description": "Reading and writing zip, tar, tar.gz and gz with archiveHelper — format detection from bytes, listing, safe extraction into a tree (traversal, link, duplicate, bomb and overlap guards, per-entry verification, skip or fail), nested archives, packing and gzip/gunzip sources, and the archive limits.",
19
+ "purposes": [
20
+ "data"
21
+ ],
22
+ "mentions": [
23
+ "archiveHelper",
24
+ "createArchiveHelper",
25
+ "ArchiveFormat",
26
+ "ArchiveExtractOptions",
27
+ "ArchiveLimits",
28
+ "DEFAULT_ARCHIVE_LIMITS",
29
+ "ExtractResult",
30
+ "PackEntry",
31
+ "onEntryError",
32
+ "maxDepth",
33
+ "ArchiveError"
34
+ ]
35
+ },
36
+ {
37
+ "name": "extending",
38
+ "file": "skills/file-streams/parts/extending.md",
39
+ "description": "Writing your own FileSource, FileSink, TreeSink, proxy plugin or URL resolver and the contract each must keep; the invariants of the package (pinned connects, the aws-free ./fetch graph, counted limits, drained tar entries, confined paths, redacted logs); and the package's specs, stand-ins and gates.",
40
+ "purposes": [
41
+ "data",
42
+ "test"
43
+ ],
44
+ "mentions": [
45
+ "FileSource",
46
+ "FileSink",
47
+ "TreeSink",
48
+ "RangeReader",
49
+ "SinkInput",
50
+ "SinkResult",
51
+ "ProxyPlugin",
52
+ "ProxyConnection",
53
+ "TunnelTarget",
54
+ "UrlResolver",
55
+ "ResolverStep",
56
+ "FILE_STREAMS_LIVE",
57
+ "S3_TEST_BUCKET"
58
+ ]
59
+ },
15
60
  {
16
61
  "name": "network",
17
62
  "file": "skills/file-streams/parts/network.md",
18
- "description": "What the HTTP client may reach — the connect-time network policy, redirects and sensitive headers, the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.",
63
+ "description": "What the HTTP client may reach — the connect-time network policy, redirects (followed or returned) and sensitive headers, the pinned WHATWG fetch on the ./fetch subpath, the proxy router and the HTTP and SOCKS proxy plugins and their pinning, why the client is node:http over its own socket under Bun, and the Google Drive and Docs resolver.",
19
64
  "purposes": [
20
65
  "config",
21
66
  "ops"
22
67
  ],
23
68
  "mentions": [
69
+ "createPinnedFetch",
70
+ "PinnedFetch",
71
+ "PinnedFetchOptions",
72
+ "HttpRedirectMode",
73
+ "anyStatus",
24
74
  "createNetworkPolicy",
25
75
  "allowCidrs",
26
76
  "allowPrivate",
27
77
  "createProxyRouter",
28
78
  "createHttpProxyPlugin",
29
79
  "createSocksProxyPlugin",
80
+ "parseProxyEndpoint",
30
81
  "trustProxyDns",
31
82
  "createGoogleDriveResolver",
32
83
  "googleDriveResolver",
@@ -34,10 +85,30 @@
34
85
  "NetworkPolicyError"
35
86
  ]
36
87
  },
88
+ {
89
+ "name": "workload",
90
+ "file": "skills/file-streams/parts/workload.md",
91
+ "description": "Writing files into a Viable ephemeral workload (makeWorkloadSink over writeBinaryFile) and reading them back (makeWorkloadSource over readBinaryFile) — the 12 MiB per-file ceiling, atomic writes, retries, random access over one read, and archives fanned out into a workload.",
92
+ "purposes": [
93
+ "agent",
94
+ "data"
95
+ ],
96
+ "mentions": [
97
+ "makeWorkloadSink",
98
+ "makeWorkloadSource",
99
+ "WorkloadWriter",
100
+ "WorkloadReader",
101
+ "WorkloadSource",
102
+ "writeBinaryFile",
103
+ "readBinaryFile",
104
+ "WORKLOAD_MAX_FILE_BYTES",
105
+ "WorkloadError"
106
+ ]
107
+ },
37
108
  {
38
109
  "name": "workspaces",
39
110
  "file": "skills/file-streams/parts/workspaces.md",
40
- "description": "Scoped temporary workspaces nested by organization/project/purpose, workspaces mirrored to a bucket (put, restore, evict, drop), bucket prefix listing and removal, reading files back from a Viable workload, and the service's workspace config.",
111
+ "description": "Scoped temporary workspaces nested by organization/project/purpose, workspaces mirrored to a bucket (put, restore, evict, drop), bucket prefix listing and removal, and the service's workspace config.",
41
112
  "purposes": [
42
113
  "data",
43
114
  "config",
@@ -46,16 +117,14 @@
46
117
  "mentions": [
47
118
  "createWorkspace",
48
119
  "createBackedWorkspace",
49
- "makeWorkloadSource",
50
- "WorkloadReader",
51
- "readBinaryFile",
52
120
  "removePrefix",
53
121
  "fileScopeHelper",
54
122
  "FileScope",
55
123
  "WorkspaceScope",
56
124
  "BackedScope",
57
125
  "BackedWorkspace",
58
- "WORKSPACE_SCOPE_MARKER"
126
+ "WORKSPACE_SCOPE_MARKER",
127
+ "S3ListedObject"
59
128
  ]
60
129
  }
61
130
  ]