@t4h.framework/zip 0.2.10 → 0.2.11

package/.ai/skills/framework-zip/SKILL.md

@@ -1,67 +1,43 @@
 ---
 name: framework-zip
 description: >-
-  Guides correct use of @t4h.framework/zip in workflows: opening a ZIP archive
-  that arrives from SFTP, HTTP or storage, listing what is inside it without
-  downloading it, and extracting the entries that matter into `Binary` files
-  that `read`, `split` and `sftp.put` accept. Use whenever a workflow receives
-  a `.zip` (a compressed batch file, a bundle of invoices, a partner's return
-  package), whenever the user mentions unzipping, decompressing, inflate,
-  deflate, zip entries, or the symbols `unzip`, `list`, `ZipEntry`,
-  `ZipExtractActivity`, and above all when someone is about to hand-roll a
-  lazy `Binary` that inflates on `stream()` in App code. Prefer this skill
-  over `node:zlib` in workflow code and over any hand-written ZIP parsing.
+  Guides the use of @t4h.framework/zip to read a ZIP archive in a T4H
+  Framework workflow: list() to read the entries without their content, and
+  unzip() with include/exclude globs to extract entries into Binary files that
+  read, split and sftp.put accept (ZipEntry, ZipEntryInfo, ZipExtractActivity,
+  ZipListActivity). Use when a workflow receives a .zip (a compressed batch
+  file, a bundle of invoices, a partner return package), when someone asks to
+  unzip, decompress or inflate, and when someone is about to use node:zlib or
+  write a lazy Binary that inflates on stream() in workflow code.
 ---
 
 # @t4h.framework/zip
 
-Reading ZIP archives inside a workflow. Package path: `framework/packages/zip`.
-Peer dependencies: `@t4h.framework/core`, `@t4h.framework/fs`.
+## Extract the entry you need
 
-## Import surface
-
-```typescript
-import { list, unzip } from '@t4h.framework/zip'
-```
-
-Two durable activities, nothing else to wire:
-
-- `unzip(archive, options?)` — extracts entries into files, resolves with
-  `ZipEntry[]` (`{ path, binary, contentLength, compressedLength,
-compressionMethod, crc32, lastModifiedAt, directory }`), directories
-  excluded, in central directory order.
-- `list(archive, options?)` — the same metadata **without** `binary` and
-  without transferring content, directories included.
-
-`archive` is any `Binary`: `sftp.get`, an HTTP response body, `write`, a slice.
-
-## Workflow usage
-
-The canonical pipeline — fetch a compressed batch, extract the one file that
-carries the records, fan it out:
+1. Get the archive as a `Binary` (`sftp.get`, an HTTP body, `write`).
+2. Call `unzip` once, high in the workflow, with an `include` glob.
+3. Check the result: no match gives an empty array, not an error.
+4. Pass `entry.binary` down to `read`, `split` or `sftp.put`.
 
 ```typescript
 import { split, Split } from '@t4h.framework/fs'
 import { unzip } from '@t4h.framework/zip'
 
-const main = new Workflow({ id: 'main', schema: inputSchema }, async input => {
-  const archive = await sftp.get(input.path)
+const archive = await sftp.get(input.path)
 
-  const [file] = await unzip(archive, { include: 'MOVIMENTO.TXT' })
+const [file] = await unzip(archive, { include: 'MOVIMENTO.TXT' })
 
-  if (!file) throw new Error(`${input.path} has no movement file`)
+if (!file) throw new Error(`${input.path} has no movement file`)
 
-  return await split(file.binary, processRecord, {
-    delimiter: Split.line(),
-  })
-})
+return await split(file.binary, processRecord, { delimiter: Split.line() })
 ```
 
-Deciding from metadata first, when most runs do not need the content:
+When the decision needs only names, sizes or dates, call `list` first. It does
+not extract content:
 
 ```typescript
 const entries = await list(archive)
-
 const invoice = entries.find(entry => entry.path.endsWith(`/${id}.txt`))
 
 if (!invoice) return { status: 'missing' }
@@ -69,80 +45,39 @@ if (!invoice) return { status: 'missing' }
 const [extracted] = await unzip(archive, { include: invoice.path })
 ```
 
-## Options
-
-| Option        | Applies to      | Default | Behavior                                                               |
-| ------------- | --------------- | ------- | ---------------------------------------------------------------------- |
-| `include`     | `unzip`, `list` | —       | Glob or array of globs an entry must match.                            |
-| `exclude`     | `unzip`, `list` | —       | Globs that drop entries, applied after `include`.                      |
-| `encoding`    | `unzip`, `list` | `utf8`  | Decoding of entry names not flagged UTF-8 (legacy archives: `latin1`). |
-| `contentType` | `unzip`         | —       | Content type of the extracted files; omitted, the claim infers it.     |
-
-Patterns match the whole entry path, case sensitively: `*` and `?` never cross
-a `/`, `**` does (`'**.csv'` reaches any depth, `'invoices/*'` one level). An
-`include` matching nothing resolves with an empty array — always check.
-
-## Behaviors you can rely on
-
-- **Content never enters the workflow.** Only file references do; the bytes go
-  from the archive to storage, inflated in a stream with backpressure.
-- **You pay for what you extract.** A ZIP's index sits at the end of the file,
-  so the listing is a ranged read of the tail and each entry is fetched by
-  offset — pulling one file out of a 6 MB archive does not download 6 MB. A
-  binary the claim cannot slice (SFTP, HTTP) is streamed **once** into storage
-  first, then read by range.
-- **Corruption fails the activity.** CRC-32 and size are verified as the entry
-  flows to storage.
-- **Format coverage.** Stored and deflate, ZIP64, data descriptors, archive
-  comments, UTF-8 and legacy names, empty entries.
-- **What `unzip` refuses.** It throws a `TypeError` for an encrypted entry and
-  for a compression method other than stored or deflate (bzip2, LZMA, zstd).
-  It also throws for a hostile path (`../`, absolute, drive qualified), before
-  it writes any entry.
-
-## Limits
-
-- **No writing.** The package reads archives; it does not create them. A file
-  to send compressed has to be produced another way.
-- **Extraction is one activity per call.** The result is recorded whole, so a
-  `unzip` with no `include` over a 3 000-entry archive puts 3 000 references in
-  the history and 3 000 files in storage. Filter.
-- **What is bounded and what is not.** Entry content, the reads over the
-  archive and the history are constant whatever the archive weighs. Memory
-  grows with the _number_ of entries — the central directory is read whole,
-  50 to 100 bytes per entry — not with their size. A zip bomb costs storage,
-  not memory: the read stops at the declared compressed size and the inflate is
-  aborted the moment the output goes past the declared uncompressed size, so a
-  lying entry never writes more than it declares — one that honestly declares
-  40 GB still writes 40 GB, and `list` shows `contentLength` beforehand. A
-  non-sliceable source adds one durable copy of the archive, and whether that
-  copy streams is the `FileSystemClaim`'s contract, as it is for `split`.
-
-## Runtime requirements
-
-- Claim provider: `FileSystemClaim` (from `@t4h.framework/fs`).
-- Production allowlist: register `ZipExtractActivity` and `ZipListActivity`
-  via `runtime.addActivity`.
-
-## Anti-patterns
-
-- `@t4h.framework/core/no-binary-subclass` rejects a lazy `Binary` that
-  inflates on `stream()` (`class ZipEntryBinary extends Binary`). `unzip` does
-  the same work inside an activity.
-- `@t4h.framework/core/no-io-in-workflow` rejects `node:zlib` calls, such as
-  `createInflateRaw`, in a workflow. Extraction belongs to an activity.
-- **Calling `unzip` per record.** Extract once, high in the workflow, and pass
-  the entry `Binary` down to children. `unzip` inside a loop or inside a helper
-  every child calls turns one extraction into thousands of activities.
-- **Extracting to read metadata.** If the run only needs a name, a size or a
-  date, `list` answers without transferring content.
-- **Trusting `include` to find something.** It is a filter: no match is an
-  empty array, not an error.
-
-## Related skills
-
-- `framework-fs-split` — `split`/`join`/`read`: what to do with the entry once
-  it is a `Binary`.
-- `framework-sftp` — `sftp.get` is where most archives come from.
-- `framework-transforms` — JSON inside the archive: normalize to NDJSON before
-  fanning out.
+## Traps
+
+- Globs match the whole entry path, case sensitive. `*` and `?` do not cross a
+  `/`; `**` does. `'*.csv'` matches only the root; use `'**.csv'` for any
+  depth.
+- `unzip` skips directory entries; `list` returns them with
+  `directory: true`.
+- `unzip` without `include` extracts every entry: one file in storage and one
+  reference in the history for each. Filter.
+- Do not call `unzip` in a loop or in a helper that each child workflow runs.
+  Extract once and pass the `Binary` down.
+- `lastModifiedAt` is a string in local time with no time zone
+  (`'2024-03-07T14:25:38'`). `contentLength` is the uncompressed size in
+  bytes.
+- Only stored and deflate entries. An encrypted entry, another compression
+  method or a path with `..`, `/` or a drive letter throws a `TypeError`. A
+  bad or corrupted archive throws a `SyntaxError`. The package cannot create a
+  ZIP.
+- `@t4h.framework/core/no-io-in-workflow` rejects `node:zlib` in a workflow,
+  and `@t4h.framework/core/no-binary-subclass` rejects a `Binary` subclass. Do
+  not inflate by hand; use `unzip`. See the `framework-fs` skill for `Binary`.
+- The runtime must provide `FileSystemClaim` and allow `ZipExtractActivity`
+  and `ZipListActivity`.
+
+## Test
+
+`unzip` and `list` run in the test. Commit a real `.zip` fixture and mock
+`FileSystemClaim` with `write` (keep the bytes) and `slice` (return a window).
+Make `slice` throw for a binary the mock did not write to test the fallback
+copy. The cookbook recipe below has a complete test.
+
+## Reference
+
+- Guide: https://tech4humans-brasil.github.io/framework/zip/
+- Recipe with a test: https://tech4humans-brasil.github.io/framework/cookbook/zip-batch
+- Types: `node_modules/@t4h.framework/zip/dist/zip.d.ts`

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @t4h.framework/zip
 
+## 0.2.11
+
+### Patch Changes
+
+- [#257](https://github.com/tech4humans-brasil/framework/pull/257) [`586bfb4`](https://github.com/tech4humans-brasil/framework/commit/586bfb4663f4a478527ee0c8db360603f034d615) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The zip README is now short and links to the site. The docs page now has a testing section and all the errors, and it no longer names the internals of sftp, the dev server or the runtime.
+
 ## 0.2.10
 
 ### Patch Changes

package/README.md

@@ -1,208 +1,34 @@
 # @t4h.framework/zip
 
-Reads ZIP archives inside a workflow. `list` tells you what is in an archive
-without transferring its content, and `unzip` unpacks the entries you ask for
-into files — each one a `Binary`, the same reference every other package
-consumes.
+Reads ZIP archives in a T4H Framework workflow. `list` tells what an archive
+holds without its content, and `unzip` extracts the entries you select into
+files. Each entry is a `Binary`.
 
-Both are durable activities: the work happens once, the result is recorded in
-the workflow history, and a replay never re-reads the archive.
-
-## Installation
+## Install
 
 ```bash
 npm install @t4h.framework/zip
-# or
-yarn add @t4h.framework/zip
-```
-
-Peer dependencies: `@t4h.framework/core` and `@t4h.framework/fs` — entries are
-written through the `FileSystemClaim`, so a file system provider must be
-registered in the runtime.
-
-## Basic usage
-
-```typescript
-import { unzip } from '@t4h.framework/zip'
-
-const [entry] = await unzip(archive)
-
-// entry.binary is a regular Binary: read a window, split it, upload it
-const header = await read(entry.binary, { range: { start: 0, end: 240 } })
-```
-
-`archive` is any `Binary`: an `sftp.get`, an HTTP response body, a file written
-by `write`. `unzip` resolves with one `ZipEntry` per file in the archive, in
-central directory order, directories excluded.
-
-## Extract only what you need
-
-An archive with three thousand invoices in it does not have to become three
-thousand files:
-
-```typescript
-const [invoice] = await unzip(archive, { include: 'invoices/0001.txt' })
-
-const csvs = await unzip(archive, {
-  include: '**.csv',
-  exclude: '**/draft-*.csv',
-})
 ```
 
-`include` and `exclude` are glob patterns (or arrays of them) matched against
-the whole entry path:
-
-| Pattern        | Matches                                 |
-| -------------- | --------------------------------------- |
-| `notes.txt`    | exactly that path                       |
-| `*.txt`        | `.txt` files at the root of the archive |
-| `invoices/*`   | anything directly inside `invoices/`    |
-| `**.csv`       | `.csv` files at any depth               |
-| `report-?.txt` | `report-1.txt`, not `report-42.txt`     |
+Peer dependencies: `@t4h.framework/core` and `@t4h.framework/fs`. The runtime
+must provide the `FileSystemClaim`.
 
-Matching is case sensitive, `include` runs before `exclude`, and an `include`
-that matches nothing resolves with an empty array rather than throwing.
-
-## Listing without unpacking
+## Usage
 
 ```typescript
-import { list } from '@t4h.framework/zip'
+import { read } from '@t4h.framework/fs'
+import { list, unzip } from '@t4h.framework/zip'
 
 const entries = await list(archive)
-// [{ path, contentLength, compressedLength, compressionMethod,
-//    crc32, lastModifiedAt, directory }, ...]
-```
-
-`list` reads the archive's index (its central directory), which lives in the
-last few kilobytes of the file — so on a sliceable binary a listing costs about
-a kilobyte, whatever the archive weighs. Use it whenever the decision (does
-this archive contain the file I need? how big is it?) can be made from metadata
-alone, and unzip afterwards only if the content is really needed.
-
-Unlike `unzip`, `list` reports directory entries too, flagged with
-`directory: true`.
-
-## How the bytes move
 
-The package never holds an archive, or an entry, in memory.
+const [entry] = await unzip(archive, { include: 'invoices/0001.txt' })
 
-1. **Ranged reads.** A ZIP is read back to front — the central directory sits
-   at the end of the file and points backwards at each entry — so every read
-   goes through `FileSystemClaim.slice`: the tail for the index, one local
-   header per entry, then the entry's compressed range.
-2. **Streaming inflate.** That range is streamed through `inflateRaw` and
-   straight into `FileSystemClaim.write`, with backpressure all the way.
-   Extracting a thousand entries costs the same memory as extracting one.
-3. **One entry at a time.** Entries are extracted sequentially, on purpose:
-   parallel extraction would multiply the memory and the storage pressure for
-   no gain on a single source stream.
-4. **Verified content.** Every entry is checked against the CRC-32 and the size
-   the archive declares while it flows to storage, so a truncated download or a
-   corrupted archive fails the activity instead of quietly producing a shorter
-   file.
-
-### Binaries that cannot be sliced
-
-Ranged reads need a binary the file system claim owns. An `sftp.get` reference
-or an HTTP body is not one: for those, the archive is streamed **once** into
-storage (`zip/<activity-id>.zip`) and the ranges are read from that copy. It
-costs one pass and one durable copy of the archive, and everything downstream
-behaves identically.
-
-Nothing has to be configured for this; it is the fallback, not a mode.
-
-## Memory: what scales with what
-
-Bounded, whatever the archive weighs: entry content (source range →
-`inflateRaw` → CRC-32 → `FileSystemClaim.write`, one entry at a time), the
-reads over the archive (a ranged read of the tail plus one 30-byte local header
-per entry) and the workflow history (references only).
-
-What scales, and with what:
-
-- **The central directory is read whole**, so memory grows with the _number_ of
-  entries — 50 to 100 bytes each. Three thousand entries is 200 KB; a million
-  is around 100 MB.
-- **`unzip` without `include` extracts everything**: one file in storage and
-  one reference in the history per entry.
-- **A zip bomb costs storage, not memory.** The read stops at the declared
-  compressed size, and the inflate is aborted the moment the output goes past
-  the declared uncompressed size — an entry that lies never writes more than it
-  declares. One that honestly declares 40 GB still writes 40 GB, so check
-  `contentLength` with `list` beforehand.
-- **A non-sliceable source** is streamed once into a durable copy of the
-  archive; whether that copy streams is the `FileSystemClaim`'s contract.
-- **Downstream is yours.** `buffer(await entry.binary.stream())` pulls the file
-  into workflow memory; `read` is capped at 1 MiB and `split` streams.
-
-## Never build a lazy `Binary` by hand
-
-The temptation, when a workflow needs one entry out of a zip, is to write a
-`Binary` subclass that inflates on `stream()` and pass it around. It works in
-the dev server and breaks in production:
-
-- A `Binary` handed to a primitive becomes the **input of a durable activity**,
-  and that input is serialized to storage before the activity runs. A class
-  holding a closure over another stream has nothing to serialize into.
-- After a replay the activity may run in **another process**, where the object
-  no longer exists.
-- `contentLength` has to lie (the uncompressed size is only in the central
-  directory), and every consumer that trusts it — a ranged `read`, a
-  `Content-Length` header — is then wrong.
-
-`unzip` is the supported answer: it does the same work inside an activity, and
-what travels is a file reference the runtime knows how to rehydrate.
-
-## Cost model
-
-`unzip` and `list` are activities, so each call is one history entry with the
-work behind it done exactly once. The reference to watch is **how many times
-you call them**, not how big the archive is:
-
-```typescript
-// once per run, however many records follow
-const [entry] = await unzip(archive, { include: 'MOVIMENTO.TXT' })
-
-const batch = await split(entry.binary, processRecord, {
-  delimiter: Split.line(),
-})
+const header = await read(entry.binary, { range: { start: 0, end: 240 } })
 ```
 
-Calling `unzip` once per record instead — inside a loop, or inside a helper
-called by every child — turns one extraction into thousands of activities, each
-with its own history entry and its own copy of the entry in storage. Extract
-high, pass the `Binary` down.
-
-## Limits
-
-- **Deflate and stored only** (methods `0` and `8`) — the two an archiver
-  produces without being asked. Anything else (bzip2, LZMA, zstd) is refused
-  with a `TypeError` naming the method.
-- **No encryption.** An encrypted entry is refused; the ZIP crypto formats are
-  either broken (ZipCrypto) or vendor-specific (AES extensions).
-- **No writing.** This package reads archives; it does not create them.
-- **Reading, not extracting to disk.** Entries land wherever the
-  `FileSystemClaim` puts them, under `zip/<activity-id>/<entry path>`.
-- **Hostile paths are refused.** An entry whose path is absolute, drive
-  qualified or contains `..` fails the whole activity before anything is
-  written (the "zip slip" class of archive).
-
-ZIP64 archives, entries written with a data descriptor (the streaming writers'
-flavour, where the local header has no sizes), UTF-8 and legacy entry names,
-archive comments and empty entries are all handled.
-
-## Errors
-
-| Situation                                       | Error         |
-| ----------------------------------------------- | ------------- |
-| Not a ZIP, or truncated beyond repair           | `SyntaxError` |
-| CRC-32 or size mismatch on an entry             | `SyntaxError` |
-| Encrypted entry, unsupported compression method | `TypeError`   |
-| Entry path escaping the extraction directory    | `TypeError`   |
-| Multi-disk archive                              | `TypeError`   |
+`archive` is any `Binary`, for example from `sftp.get`, an HTTP response body
+or `write`.
 
-## Runtime requirements
+## Documentation
 
-- Claim provider: `FileSystemClaim` (from `@t4h.framework/fs`).
-- Production allowlist: register `ZipExtractActivity` and `ZipListActivity`
-  through `runtime.addActivity`.
+https://tech4humans-brasil.github.io/framework/zip/

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t4h.framework/zip",
-  "version": "0.2.10",
+  "version": "0.2.11",
   "description": "ZIP extraction module for the T4H Framework",
   "homepage": "https://github.com/tech4humans-brasil/framework/tree/main/packages/zip",
   "bugs": "https://github.com/tech4humans-brasil/framework/issues",
@@ -33,8 +33,8 @@
     "test:watch": "vitest"
   },
   "devDependencies": {
-    "@t4h.framework/core": "^0.20.0",
-    "@t4h.framework/fs": "^0.13.1",
+    "@t4h.framework/core": "^0.21.0",
+    "@t4h.framework/fs": "^0.14.0",
     "typescript": "^5.9.3",
     "vitest": "^4.0.18"
   },