@t4h.framework/fs 0.13.0 → 0.14.0

package/.ai/skills/framework-fs-write/SKILL.md

@@ -1,242 +1,82 @@
 ---
 name: framework-fs-write
 description: >-
-  Guides the use of `write` from @t4h.framework/fs — the only supported way to
-  turn content produced inside a workflow (a JSON string, a CSV, a positional
-  return file, a Buffer) into a `Binary` that other primitives accept. Use
-  whenever a workflow must upload, send or save content it built itself:
-  `sftp.put`/`sftp.append` with generated content, an HTTP request carrying a
-  file, persisting a report, or when the user mentions creating/building a
-  Binary, converting a string or Buffer to a Binary, "stringToBinary",
-  "BufferBinary", or hits `Unsupported binary instance`. A workflow must NEVER
-  implement its own `Binary` subclass — this skill explains why and what to do
-  instead.
+  Shows how to turn content that a T4H Framework workflow built (a JSON
+  string, a CSV, a positional return file, a Buffer) into a Binary with
+  write() from @t4h.framework/fs, and how to test it. Use when a workflow must
+  upload, send or save content it made itself (sftp.put or sftp.append with
+  generated content, an HTTP request with a file body, a report), or when
+  someone asks to create a Binary, convert a string or a Buffer to a Binary,
+  writes a stringToBinary or BufferBinary helper, or extends Binary. A
+  workflow never implements its own Binary.
 ---
 
-# `write` — turn workflow content into a `Binary`
+# write — workflow content into a Binary
 
-Every file primitive in the framework speaks `Binary`: `sftp.put`,
-`sftp.append`, `split`, `join`, `pdf`, HTTP bodies. A `Binary` is a
-**reference** to stored content, not the bytes themselves.
+Read the skill `framework-fs` first for the `Binary` rules.
 
-`write` is what produces one from content your workflow built:
+## Write and upload
 
 ```typescript
 import { write } from '@t4h.framework/fs'
 
 const file = await write(JSON.stringify(state, null, 2), {
   contentType: 'application/json',
+  filename: 'state.json',
 })
 
 await sftp.put('/out/state.json', file)
 ```
 
-Package path: `framework/packages/fs`. Import only from the package root.
-
----
-
-## Never implement `Binary` yourself
-
-This is the rule this skill exists to enforce:
+1. Build the content as a `string` or a `Buffer` in the workflow.
+2. Call `write(content, { contentType })`. Set `contentType` when the
+   receiver cares. Without it, the claim infers one from `filename`, which
+   defaults to the activity id.
+3. Pass the `Binary` to the next primitive. One `Binary` can go to many
+   primitives (for example `sftp.put` to a `.tmp` path and then to the final
+   path); nothing is copied.
+
+## Traps
+
+- Do not write a `Binary` subclass or a `stringToBinary` helper that wraps
+  bytes. Replace its body with `write`; keep the helper name if it reads well.
+- The option is `filename`, not `path` or `name`. It is the storage name, not
+  the upload path.
+- `encoding` applies only to a string `content` (default `'utf8'`). A
+  `Buffer` is written as it is.
+- `write` takes only `string | Buffer`. A value that already is a `Binary`
+  goes straight to the next primitive.
+- `write` is for content that fits in workflow memory: a state file, a
+  report, a return file. To build a large file from many records, use
+  `split` and `join` (skill `framework-fs-split`); never concatenate records in
+  the workflow.
+- `write` is an activity. On replay it returns the recorded `Binary` and does
+  not write again.
+
+## Test
+
+Mock `FileSystemClaim.write`. `write` passes the content to the claim as a
+`Buffer`. Use the `TestBinary` double from the skill `framework-fs`:
 
 ```typescript
-// ❌ NEVER — this class cannot be persisted
-class BufferBinary extends Binary {
-  constructor(
-    private readonly content: Buffer,
-    readonly contentType = '...',
-  ) {
-    super()
-  }
-
-  get contentLength() {
-    return this.content.length
-  }
-
-  stream() {
-    return Readable.from(this.content)
-  }
-}
-
-await sftp.put('/out/state.json', new BufferBinary(Buffer.from(json)))
-```
-
-```typescript
-// ✅ ALWAYS
-await sftp.put(
-  '/out/state.json',
-  await write(json, {
-    contentType: 'application/json',
-  }),
+const writeFile = vi.fn(
+  async (_path: string, content: Buffer, contentType?: string) =>
+    new TestBinary(content, contentType),
 )
-```
-
-### Why a custom `Binary` cannot work
-
-A workflow is **durable**: the body replays, and each activity is persisted
-before it runs. When you pass a value to a primitive, that value becomes the
-activity **input**, and the input is serialized to storage.
-
-A `Binary` holding bytes in memory has nothing to serialize into — the buffer
-lives in the process that built it, and the activity may run in another
-process, minutes later, after a replay. The runtime therefore has to spool the
-content somewhere before the activity can be persisted at all.
-
-Symptoms of doing it by hand:
-
-- `TypeError: Unsupported binary instance` when the activity input is
-  serialized, on older runtimes;
-- the activity is never persisted, the transaction rolls back, and the message
-  ends up in the dead letter queue after the retries — the run stops mid-way
-  with no obvious error in the workflow itself;
-- it may still "work" locally, because the dev server keeps values in memory
-  and never crosses a process boundary. **Passing locally proves nothing here.**
-
-Current runtimes spool an unknown `Binary` to storage instead of failing, so a
-hand-rolled class may survive — but it pays a hidden copy, loses the content
-type you meant to set, and depends on runtime-version behavior. `write` is the
-contract.
-
----
-
-## API
-
-```typescript
-function write(
-  content: string | Buffer,
-  options?: FileWriteOptions,
-): Promise<Binary>
-```
-
-| Option        | Type             | Required | Purpose                                                                                                                                                     |
-| ------------- | ---------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `contentType` | `string`         | No       | Content type stored with the file — `'application/json'`, `'text/csv'`, … Defaults to what the filename extension implies, then `application/octet-stream`. |
-| `filename`    | `string`         | No       | Name the file is stored under. Defaults to the activity id.                                                                                                 |
-| `encoding`    | `BufferEncoding` | No       | How a **string** `content` is encoded. Defaults to `'utf8'`. Ignored for a `Buffer`, which always travels as base64.                                        |
-
-`write` is a durable activity: on replay the content is **not** written again;
-the recorded `Binary` is returned.
-
----
-
-## When to use it
-
-| Situation                                                                     | Use                                             |
-| ----------------------------------------------------------------------------- | ----------------------------------------------- |
-| Send content the workflow built (JSON, CSV, positional file) over SFTP/HTTP   | `write`                                         |
-| Read a small window of an existing file into the workflow                     | [`read`](./read) — see **framework-fs-split**   |
-| Fan a large file out into one workflow per record                             | `split` — see **framework-fs-split**            |
-| Fold a finished batch back into a file                                        | `join` — see **framework-fs-split**             |
-| Content already is a `Binary` (HTTP response body, `sftp.get`, `join` result) | pass it straight through — do not re-`write` it |
-
----
-
-## Size matters: keep content bounded
-
-`write` takes content **in workflow memory**, so it is bounded by what the
-workflow can reasonably hold — a state file, a report, a return file. It is not
-the tool for gigabyte outputs.
-
-To build a large file out of many records, use `join` over a batch: it streams
-results one at a time into storage, and nothing but the resulting `Binary`
-reaches the workflow.
 
----
-
-## Patterns
-
-### A) Write then upload
-
-```typescript
-const csv = await write(rows.map(toCsvLine).join('\n'), {
-  contentType: 'text/csv',
-  filename: `report-${input.date}.csv`,
+await TestWorkflowEnvironment.create().execute(workflow, input, {
+  claims: [mockClaim(FileSystemClaim, { write: writeFile })],
 })
 
-await sftp.put(`/out/report-${input.date}.csv`, csv)
-```
-
-### B) Write once, use twice
-
-The same `Binary` can be handed to more than one primitive — it is a reference,
-so nothing is copied and nothing is re-uploaded:
-
-```typescript
-const file = await write(json, { contentType: 'application/json' })
-
-await sftp.put(`${path}.tmp`, file)
-await sftp.put(path, file)
-```
-
-Note that a hand-rolled `Binary` backed by a single-use stream would break on
-the second call — another reason not to build one.
-
-### C) Binary content
-
-```typescript
-const pdfBytes: Buffer = await buildPdf()
-
-await sftp.put(
-  '/out/invoice.pdf',
-  await write(pdfBytes, {
-    contentType: 'application/pdf',
-  }),
+expect(writeFile).toHaveBeenCalledWith(
+  'report.csv',
+  expect.any(Buffer),
+  'text/csv',
 )
 ```
 
----
-
-## Testing
-
-`write` goes through `FileSystemClaim.write`, so mock that claim like any
-other. See the **framework-workflow-testing** skill for the harness API.
-
-```typescript
-import { TestWorkflowEnvironment, mockClaim } from '@t4h.framework/core/testing'
-import { FileSystemClaim } from '@t4h.framework/fs'
-
-// replay runs the workflow twice — return a fresh Binary per call
-const write = vi
-  .fn()
-  .mockImplementation(
-    async (path, content, contentType) =>
-      new MockBinary(
-        Buffer.isBuffer(content) ? content : await toBuffer(content),
-        contentType,
-      ),
-  )
-
-await env.execute(workflow, input, {
-  claims: [
-    mockClaim(FileSystemClaim, { write, read: vi.fn(), slice: vi.fn() }),
-  ],
-})
-
-expect(write).toHaveBeenCalledWith('report.csv', expect.any(Buffer), 'text/csv')
-```
-
----
-
-## Anti-patterns
-
-- **Declaring a class that extends `Binary`** in an App. There is no valid
-  reason to; use `write`.
-- Helper functions like `stringToBinary(value)` that wrap content in a custom
-  class — replace the implementation with `write`, keeping the helper name if
-  it reads well.
-- `write`-ing a value that already is a `Binary` (an HTTP response body, a
-  `sftp.get` result, a `join` output) — that copies a file to obtain a
-  reference you already had.
-- Building a large file by concatenating records in memory and calling `write`
-  once — use `split`/`join` instead.
-- Relying on the filename extension to set the content type when the consumer
-  cares about it — pass `contentType` explicitly.
-
----
-
-## Related skills
+## Reference
 
-- `framework-fs` — the `FileSystemClaim` contract and runtime providers.
-- `framework-fs-split` — `read`/`split`/`join`: windows, fan-out and fold.
-- `framework-sftp` — `sftp.put`/`append`, the most common consumer of `write`.
-- `framework-workflow-creation` — the rules a workflow body must follow.
+- `write`: https://tech4humans-brasil.github.io/framework/fs/write
+- Types: `node_modules/@t4h.framework/fs/dist/activities/FileWriteActivity.d.ts`
+- Related skills: `framework-fs`, `framework-fs-split`, `framework-sftp`.

package/CHANGELOG.md

@@ -1,5 +1,61 @@
 # @t4h.framework/fs
 
+## 0.14.0
+
+### Minor Changes
+
+- [#260](https://github.com/tech4humans-brasil/framework/pull/260) [`e9e8036`](https://github.com/tech4humans-brasil/framework/commit/e9e80366440b8da611489267dd02680592ca32e9) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - `split` gets two options. `groupBy` starts one child per key, with all the records of the key. `with` gives each child the rows of other files that have the same key. The key is a `match` `RegExp` with optional `replace` steps. The `input` template gets two more markers: `Split.key` and `Split.rows(name)`. `split` reads each file one time into a temporary SQLite file on the local disk, keeps only the rows of `with` whose key is in the main file, and removes the file at the end. The files can have 1 GiB in total. The package now needs Node 22.13 or later, the first version with `node:sqlite` and `iterate()` without a flag.
+
+  The types of `split` now have the `Binary` prefix: `BinarySplitOptions`, `BinarySourceOptions`, `BinaryRecordKeyOptions`, `BinaryReplace`, `BinaryDelimiter` and `BinaryWhere`, with a `JSON` form for the stored options. `FileSplitOptions` and `FileSplitActivityOptions` extend them and keep their shape. `FileSplitDelimiter` and `FileSplitWhere` are deprecated: use `BinaryDelimiter` and `BinaryWhere`.
+
+- [#262](https://github.com/tech4humans-brasil/framework/pull/262) [`e46ebbb`](https://github.com/tech4humans-brasil/framework/commit/e46ebbbe1e711febd21ddcb920e890125fbb9f0a) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - `split` gets the `input` option: a template that each child gets, with its slice in place of the `Split.slice` marker. The other values of the template stay as they are. `tsc` checks the template against the schema of the child workflow, so `split` and `FileSplitOptions` take the child workflow as a type parameter. `FileSplitActivityInput` stores the template in `input`, and its `options` use the new `FileSplitActivityOptions` type, which has no `input`.
+
+### Patch Changes
+
+- [#248](https://github.com/tech4humans-brasil/framework/pull/248) [`ae6d224`](https://github.com/tech4humans-brasil/framework/commit/ae6d22475ee07307ac59582581a6077788b902ca) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - Adds the `no-batch-query-in-loop` rule to `@t4h.framework/core/eslint` and turns it on in `configs.recommended`, at the `error` level, for TypeScript files that are not tests. After the upgrade, a project that uses the recommended config can get new lint errors. The rule reports these calls in a loop or in the callback of an array method:
+  - `join`, `split` and `index` from `@t4h.framework/fs`, and `size()`, `take()` and `toArray()` on a `WorkflowBatchCollectionRef` (also after `.filter(...)`). Each call queries or scans the whole batch or file, so a loop does N scans.
+  - `read` from `@t4h.framework/fs`. Each call is a separate activity and a separate file read, and each result goes into the workflow history.
+
+  The rule checks all the functions of the file. In an activity class, it checks only the `constructor`, `toInput` and `toOutput`. It checks only the call site, not a function that the loop calls. The core docs and skills, and the `framework-fs-split` skill of `@t4h.framework/fs`, tell you to build an index one time and call `lookup` in the loop.
+
+- [#253](https://github.com/tech4humans-brasil/framework/pull/253) [`d9f5640`](https://github.com/tech4humans-brasil/framework/commit/d9f56403f90ffa1e04640b244f8c9247a119c084) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The fs docs and skills no longer describe how the dev server or a runtime stores a hand-rolled `Binary`. They say that such a class can pass in one environment and fail in another.
+
+- [#257](https://github.com/tech4humans-brasil/framework/pull/257) [`586bfb4`](https://github.com/tech4humans-brasil/framework/commit/586bfb4663f4a478527ee0c8db360603f034d615) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The fs site pages now hold the full contract, the README is short, and each skill gives only the steps and traps for a consumer app. The docs now name the `split` output `WorkflowBatchCollectionRef` and no longer name a runtime limit.
+
+- [#258](https://github.com/tech4humans-brasil/framework/pull/258) [`421f163`](https://github.com/tech4humans-brasil/framework/commit/421f16390834a26b87da5936dfa7c30c669785ed) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - `join` now fails the file stream when the batch fails before the end. Before, the stream stayed open and the write of the host did not end. `read` now rejects a range with an `end` that is not an integer. The other changes are internal and the public API does not change.
+
+- [#246](https://github.com/tech4humans-brasil/framework/pull/246) [`b99b41f`](https://github.com/tech4humans-brasil/framework/commit/b99b41f60d69b3037187e9fcdc27ab0d47ff18a2) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - Adds byte windows to `SftpBinary`. `binary.slice(range)` returns a view of a byte window of the remote file that reuses the stream of `binary`. The new `range` field is absolute in the file, with an exclusive `end`, and `contentLength` is `range.end - range.start`. A slice of a slice composes. `toJSON()` includes `range` on a slice only, so the JSON of a whole file does not change. `slice` throws a `RangeError` when the window is not inside the binary. In `@t4h.framework/fs`, the docs no longer give an SFTP reference as an example of a binary that cannot be sliced.
+
+## 0.13.1
+
+### Patch Changes
+
+- [#229](https://github.com/tech4humans-brasil/framework/pull/229) [`d4d7d97`](https://github.com/tech4humans-brasil/framework/commit/d4d7d973233ec1f4d78628a69283701b9a55fbf9) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - Fixes the published packages:
+  - harness: the published types import `typings/PublicProperties.js`, which was not in `dist`. `uuidv7` is now a dependency.
+  - dev: `tar` is now a dependency.
+  - fs: the published types of `FileSplitActivity` no longer import `typebox`, which `fs` does not depend on.
+
+- [#241](https://github.com/tech4humans-brasil/framework/pull/241) [`f6b8454`](https://github.com/tech4humans-brasil/framework/commit/f6b845484d22e7c8e7240fe2e113f1cdea7c8ce3) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - Adds seven rules to `@t4h.framework/core/eslint` and turns them on in `configs.recommended`:
+  - `no-io-in-workflow` reports `fetch` and calls to network, file, database and mail modules in a workflow and in the `constructor`, `toInput` and `toOutput` of an activity. It also reports `zlib` and `stream` calls there, because they work on the content in memory on each replay.
+  - `no-binary-subclass` reports a class that extends `Binary`.
+  - `no-framework-activity-in-app` reports `new` and `extends` on an activity of a framework package other than the core.
+  - `no-testing-import-in-source` reports an import of a `@t4h.framework/<package>/testing` entry out of a test file.
+  - `require-workflow-id` reports a `Workflow` with no `id`.
+  - `no-secret-to-string` reports a `Secret` that becomes a string.
+  - `no-literal-secret-in-authorization` reports a string literal in the `secret` or `clientSecret` field of an `Authorization`.
+
+  The package skills now name these rules in place of the text that the rules enforce. The docs and skills give each `Workflow` an `id`, read the HMAC secret from `EnvDesign`, and do not document a custom HTTP activity.
+
+- [#238](https://github.com/tech4humans-brasil/framework/pull/238) [`8567806`](https://github.com/tech4humans-brasil/framework/commit/8567806ec537288076c3a18a4816f26d58a67905) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The `framework-fs` skill says to call `Binary.stream()` only in a method of an activity, not in the workflow.
+
+- [#240](https://github.com/tech4humans-brasil/framework/pull/240) [`0606115`](https://github.com/tech4humans-brasil/framework/commit/0606115e2e0baa994e9f0ed3559ee54ecac88e44) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - The skills do not repeat a rule that a lint rule, `tsc` or a runtime error already enforces. Each such rule is now one line that names the gate. The examples that broke a gate (`process.env`, `as any`, an inline secret, an `EnvDesign` in a workflow file, `ref.output` before `wait()`) are correct now.
+
+- [#228](https://github.com/tech4humans-brasil/framework/pull/228) [`d29e1ab`](https://github.com/tech4humans-brasil/framework/commit/d29e1ab29a039b13c3e4b075472bd615566ae75e) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - Uses exact types and `unknown` in place of `any`, and checks external data with guards.
+  - harness: the fix-issues prompt lists each sensor as `- checkTypes: PASS`. Before, it printed `- 0: [object Object]`.
+  - hmac: an algorithm name such as `constructor` throws `Unsupported algorithm`.
+  - oidc: a discovery response with no `introspection_endpoint`, or a null introspection body, throws a clear error.
+  - `HttpRequestActivity.toOutput` takes `HttpRequestActivityOutput` in place of `any`.
+
 ## 0.13.0
 
 ### Minor Changes