@t4h.framework/fs 0.4.0 → 0.6.0

package/.ai/skills/framework-fs/SKILL.md

@@ -2,11 +2,11 @@
 name: framework-fs
 description: >-
   Guides correct use of @t4h.framework/fs in T4H Framework workflows: the
-  FileSystemClaim abstraction for reading/writing files inside activities,
-  Claim wiring, Binary results, and runtime providers. Use when implementing
-  activities that persist or load file payloads, mocking FileSystemClaim in tests,
-  registering fs claim providers, debugging missing fs provider errors, or working
-  in packages/fs, packages/http, or packages/jsx.
+  FileSystemClaim abstraction that the package activities (http, jsx) use to
+  read/write files, the Binary result workflows consume, and runtime providers.
+  Use when a workflow orchestrates file-backed package activities, consuming
+  Binary results, mocking FileSystemClaim in workflow tests, registering fs
+  claim providers, or debugging missing fs provider errors.
 ---
 
 # @t4h.framework/fs
@@ -19,10 +19,16 @@ Package path: `framework/packages/fs`. Peer dependency: `@t4h.framework/core` (`
 
 ## Import surface
 
-The package exports a single public symbol from `src/fs.ts`:
+Public exports live in `src/fs.ts`:
 
 ```typescript
-import { FileSystemClaim } from '@t4h.framework/fs'
+import {
+  FileSystemClaim,
+  // file fan-out / fold — see the framework-fs-split skill
+  split, Split, FileSplitActivity, type FileSplitOptions,
+  join, FileJoinActivity, type FileJoinOptions,
+  ByteDelimiterScanner, type FileSplitDelimiter, type ByteRange,
+} from '@t4h.framework/fs'
 ```
 
 Do not import from internal paths such as `src/FileSystemClaim.ts` in application code.
@@ -42,6 +48,12 @@ export abstract class FileSystemClaim {
   ): Binary | Promise<Binary>
 
   public abstract read(path: string): Binary | Promise<Binary>
+
+  public abstract slice(
+    binary: Binary,
+    range: { start: number; end: number },
+    encoding?: BufferEncoding,
+  ): Binary | Promise<Binary>
 }
 ```
 
@@ -49,8 +61,9 @@ export abstract class FileSystemClaim {
 |--------|---------|
 | `write(path, readable)` | Persist stream or buffer at `path`; return a `Binary` handle (metadata + `stream()`). |
 | `read(path)` | Load an existing file at `path`; return a `Binary` handle. |
+| `slice(binary, range, encoding?)` | A view over a byte range of an existing binary, **without copying data**. When `encoding` is set, the runtime hydrates the slice as a decoded string instead of a binary — this is how `split` children receive plain strings. |
 
-Both methods may return synchronously or as a `Promise`. Implementations choose storage layout; callers should only rely on the returned `Binary` API.
+All methods may return synchronously or as a `Promise`. Implementations choose storage layout; callers should only rely on the returned `Binary` API.
 
 ### Binary (from `@t4h.framework/core`)
 
@@ -62,45 +75,25 @@ export abstract class Binary {
 }
 ```
 
-Typical activity flow after `write` or `read`:
+The package activities return this `Binary` to your workflow — e.g. an `http`
+response `Body` is backed by it. Consume it with `stream()` (or the higher-level
+`body.json()` / `text()` that packages expose):
 
 ```typescript
-const binary = await this.claims.fs.write('reports/output.pdf', buffer)
 const stream = await binary.stream()
 // consume stream (e.g. stream.toArray() then Buffer.concat)
 ```
 
 ---
 
-## Declaring the claim in an activity
-
-Activities that touch files declare `FileSystemClaim` via `Claim` from `@t4h.framework/core`. Access `this.claims.fs` only while execution is inside `Claim.run` (same as any claim).
-
-```typescript
-import { SyncActivity, Claim } from '@t4h.framework/core'
-import { FileSystemClaim } from '@t4h.framework/fs'
-
-class SaveReportActivity extends SyncActivity<[data: Buffer], { path: string }, any, any> {
-  private readonly claims = new Claim({
-    fs: FileSystemClaim,
-  })
-
-  public async toInput(data: Buffer) {
-    return { path: 'reports/output.pdf' }
-  }
-
-  public async run(input: { path: string }) {
-    const binary = await this.claims.fs.write(input.path, Buffer.from('...'))
-    return { contentType: binary.contentType }
-  }
-
-  public toOutput(output: any) {
-    return output
-  }
-}
-```
+## How workflows use it
 
-Claim key name (`fs` above) is arbitrary but must match how you access `this.claims.fs`. Downstream packages use `fs` consistently (`AbstractHttpRequestActivity`, jsx render activities).
+Workflows never call `FileSystemClaim` directly. File I/O happens inside the
+**ready-made package activities** that already own the claim — the `http` client
+persists response bodies, jsx render activities write their output — and you
+read the resulting `Binary` (e.g. `await res.body.json()` / `text()`). Your only
+responsibility is that the runtime **registers a provider** (below). See
+**Known consumers** for which package primitives to orchestrate.
 
 ---
 
@@ -122,94 +115,59 @@ These packages depend on `@t4h.framework/fs` and declare `FileSystemClaim`; read
 
 | Package | Usage |
 |---------|--------|
-| `@t4h.framework/http` | `AbstractHttpRequestActivity` writes HTTP response bodies via `fs.write(\`${uuid}-http-request-activity-response-body.${ext}\`, response.body)`; wraps result in `Body`. |
-| `@t4h.framework/jsx` | `RenderWorkflowActivity` / `RenderFormWorkflowActivity` use `fs.write('rsc.txt', ...)` in `toInput` and consume `Binary` in `run`. |
-
-### HTTP response persistence (reference)
-
-From `AbstractHttpRequestActivity.run`:
-
-```typescript
-const ext =
-  typeof contentType === 'string' ? mime.getExtension(contentType) : 'bin'
-
-const binary = await this[CLAIMS_KEY].fs.write(
-  `${uuidv7()}-http-request-activity-response-body.${ext}`,
-  response.body,
-)
-
-return this.toHttpOutput({
-  status: response.status,
-  headers: response.headers,
-  body: new Body(binary),
-})
-```
+| `@t4h.framework/http` | The HTTP activity persists response bodies to `fs` and returns them wrapped in a `Body` — you read them with `res.body.json()` / `text()`. |
+| `@t4h.framework/jsx` | Render activities write their output to `fs` and hand back a `Binary` you consume in the workflow. |
+| `@t4h.framework/fs` itself | `FileSplitActivity` persists the durable copy and slices it by ranges; `FileJoinActivity` streams batch results into a new file. See the `framework-fs-split` skill. |
 
-Extension comes from response `content-type` when present; otherwise `'bin'`. Path is unique per request (uuid prefix).
-
-### JSX render activities (reference)
-
-```typescript
-public readonly claims = new Claim({
-  fs: FileSystemClaim,
-})
-
-public async toInput() {
-  return await this.claims.fs.write('rsc.txt', Buffer.from('Hello World!'))
-}
-
-public async run(binary: Binary) {
-  const stream = await binary.stream()
-  const chunks = await stream.toArray()
-  return Buffer.concat(chunks).toString()
-}
-```
-
-Current `toInput` stubs are placeholders; real implementations should still use `FileSystemClaim.write(path, Buffer | Readable)` per README.
+Orchestrate these package primitives from your workflow; the `fs` write/read happens inside them.
 
 ---
 
 ## Testing
 
-There are **no unit tests inside `packages/fs`**. Use patterns from `packages/http` activity tests.
-
-### Mocking FileSystemClaim in activity tests
-
-From `AbstractHttpRequestActivity.spec.ts` — wrap activity execution in `Claim.run` and provide partial mocks:
+Test the **workflow** with `TestWorkflowEnvironment` from
+`@t4h.framework/core/testing`, mocking `FileSystemClaim` (plus whatever other
+claims the triggered activities need, e.g. `HttpClientRequestClaim` for an HTTP
+call). See the **framework-workflow-testing** skill for the harness API.
 
 ```typescript
-import { Claim } from '@t4h.framework/core'
+import { Readable } from 'node:stream'
+import { Binary } from '@t4h.framework/core'
+import { TestWorkflowEnvironment, mockClaim } from '@t4h.framework/core/testing'
 import { FileSystemClaim } from '@t4h.framework/fs'
-import { HttpClientRequestClaim } from '@t4h.framework/http'
 
-mockWrite.mockImplementation(async (_path: string, readable: Readable) => {
+class MockBinary extends Binary {
+  public readonly contentType: string
+  public readonly contentLength: number
+
+  constructor(private readonly data: Buffer, contentType: string) {
+    super()
+    this.contentType = contentType
+    this.contentLength = data.byteLength
+  }
+
+  public stream() {
+    return Readable.from(this.data)
+  }
+}
+
+// mockImplementation: replay runs the workflow twice, so return a fresh Binary each call.
+const mockWrite = vi.fn().mockImplementation(async (_path: string, readable: Readable) => {
   const chunks = await readable.toArray()
-  return new MockBinary(Buffer.concat(chunks), contentType)
+  return new MockBinary(Buffer.concat(chunks), 'application/json')
 })
 
-await Claim.run(
-  [
-    {
-      provide: HttpClientRequestClaim,
-      value: { request: mockRequest } as HttpClientRequestClaim,
-    },
-    {
-      provide: FileSystemClaim,
-      value: {
-        write: mockWrite,
-        read: vi.fn(),
-      } as unknown as FileSystemClaim,
-    },
+const env = TestWorkflowEnvironment.create({ now: 0 })
+
+await env.execute(workflow, input, {
+  claims: [
+    mockClaim(FileSystemClaim, { write: mockWrite, read: vi.fn() }),
+    // ...other claims the workflow's activities need
   ],
-  () => new TestableActivity().run(input),
-)
+})
 ```
 
-Mock `write` to return a `Binary` compatible with what the activity expects (e.g. `Body` for HTTP). Assert `mockWrite` was called with expected path and body when relevant.
-
-### Workflow-level tests
-
-`WorkflowTesting.run` from `@t4h.framework/core/testing` runs the full claim stack when you pass explicit `FileSystemClaim` providers. For isolated activity unit tests, prefer `Claim.run` with partial mocks.
+Assert `mockWrite` was called with the expected path and body when relevant.
 
 ---
 
@@ -218,7 +176,7 @@ Mock `write` to return a `Binary` compatible with what the activity expects (e.g
 1. **Extend** `FileSystemClaim` from `@t4h.framework/fs`.
 2. **Return** a `Binary` subclass (or core-compatible `Binary`) from `write` and `read` with correct `contentType`, `contentLength`, and `stream()`.
 3. **Accept** `Readable | Buffer` in `write` (match the abstract signature).
-4. **Register** `{ provide: FileSystemClaim, value: new YourImpl() }` in runtime or test `Claim.run` providers.
+4. **Register** `{ provide: FileSystemClaim, value: new YourImpl() }` in the runtime (tests use `mockClaim(FileSystemClaim, ...)` instead).
 5. **Keep paths deterministic** when activities replay — unpredictable temp paths are fine if only `Binary` metadata is serialized and replay re-executes `write`; know your serializer/replay story if `path` is persisted.
 
 ---
@@ -228,15 +186,13 @@ Mock `write` to return a `Binary` compatible with what the activity expects (e.g
 1. Add `"@t4h.framework/fs": "workspace:^"` (or published version) to app `dependencies` when activities use `FileSystemClaim`.
 2. Ensure **`@t4h.framework/core`** is available (peer of `fs`).
 3. Register **`FileSystemClaim`** in the runtime or test `claims` array.
-4. Declare `new Claim({ fs: FileSystemClaim })` (or equivalent key) on every activity that calls `this.claims.fs`.
-5. Build before publish: `yarn build` in `packages/fs` (`tsc --project tsconfig.build.json`).
+4. Orchestrate file I/O through the package activities that own `FileSystemClaim` (http, jsx) — workflow code never declares an activity or accesses `this.claims.fs`.
 
 ---
 
 ## Anti-patterns
 
 - Importing from `@t4h.framework/fs` paths other than the package export (`FileSystemClaim` only).
-- Calling `this.claims.fs` **outside** `Claim.run` / workflow execution context → `ClaimStorageNotActiveException`.
 - Omitting `FileSystemClaim` from providers while using `@t4h.framework/http` or `@t4h.framework/jsx` activities that require `fs`.
 - Assuming `packages/fs` ships a concrete disk/S3 implementation — it does not; implement and register `FileSystemClaim` in your runtime or test harness.
 
@@ -244,6 +200,7 @@ Mock `write` to return a `Binary` compatible with what the activity expects (e.g
 
 ## Related skills
 
-- `framework-core` — `Claim`, `SyncActivity`, `Binary`, `Claim.run`, `WorkflowTesting`.
+- `framework-fs-split` — `split`/`join`: fan a file out into a batch of workflows and fold the results back into a file.
+- `framework-core` — `Claim`, `Binary`, and the `TestWorkflowEnvironment` testing harness.
 - `framework-http` — HTTP activities that persist bodies via `FileSystemClaim`.
 - `framework-jsx` — render activities that write/read via `fs`.

package/.ai/skills/framework-fs-split/SKILL.md

@@ -0,0 +1,255 @@
+---
+name: framework-fs-split
+description: >-
+  Guides correct use of `split` and `join` from @t4h.framework/fs: fanning one
+  large file out into a batch of workflows (one per byte-range slice) and
+  folding the batch results back into a single file — with O(1) parent history
+  and bounded memory regardless of file size. Use whenever a workflow must
+  process a large file record by record — positional files, plain-text lines,
+  NDJSON, batch imports, per-record fan-out with an aggregated output (e.g. a
+  CSV report) — or when the user mentions splitting/chunking a file, running a
+  workflow per line/record, waiting for a batch, collecting batch results, or
+  the symbols `split`, `join`, `Split`, `FileSplitActivity`,
+  `FileJoinActivity`, `WorkflowBatchRef`, `WorkflowBatchClaim`. Prefer this
+  skill over hand-rolling stream loops + `startWorkflow` calls.
+---
+
+# `split` / `join` — file fan-out and fold for workflows
+
+`split` turns **one file into a batch of workflows**: it streams the file once,
+cuts it into **byte-range slices** (never copying slice content), starts one
+child workflow per slice, and **suspends the parent until every child is
+terminal**. `join` is the inverse: it folds the batch results back into a
+single file, streaming one result at a time.
+
+```typescript
+import { split, join, Split } from '@t4h.framework/fs'
+
+const processRecord = new Workflow(
+  { id: 'process-record', schema: Type.String() },
+  async (record: string) => {
+    // record is the raw slice content — no Binary handling needed
+    return toCsvRow(parsePositional(record))
+  },
+)
+
+const main = new Workflow(fileSchema, async (file: Binary) => {
+  const batch = await split(file, processRecord, {
+    delimiter: Split.char('\n10', { next: 2 }),
+    timeout: 3_600_000,
+  })
+  // this line only runs once the WHOLE batch is terminal;
+  // batch: WorkflowBatchRef — { id: string, count: number }
+
+  return await join(batch, { format: 'lines', header: 'id;value;date' })
+  // Binary: the aggregated file — no slice ever crossed workflow memory
+})
+```
+
+Package path: `framework/packages/fs`. Import only from the package root.
+
+---
+
+## The memory model (the golden rule)
+
+Everything here exists to keep the parent history **O(1)** whatever the file
+size:
+
+1. `split` reads the source stream **exactly once, worker side**: the same
+   pass persists a durable copy through `FileSystemClaim.write` while a byte
+   scanner records slice offsets. No slice content is materialized.
+2. Children receive **range views** over the durable copy
+   (`FileSystemClaim.slice`); the runtime hydrates each slice only inside that
+   child's own run — as a plain decoded string when the child's schema is
+   `Type.String()`.
+3. The parent resumes with only `WorkflowBatchRef { id, count }`. Per-child
+   results are **never aggregated onto the ref**: they live with each child
+   run.
+4. `join` streams results one at a time through `WorkflowBatchClaim.outputs`
+   into `FileSystemClaim.write` with backpressure; only the resulting `Binary`
+   reaches history.
+
+**Golden rule:** volume proportional to the batch size only ever flows inside
+an activity, through a claim. Into workflow history goes a fold result or a
+bounded slice — never the N items.
+
+---
+
+## `split`
+
+```typescript
+async function split(
+  binary: Binary,
+  workflow: Workflow<any, any>,
+  options: FileSplitOptions,
+): Promise<WorkflowBatchRef>
+```
+
+The child workflow declares its **real input type** (e.g. `Type.String()`) —
+it never sees `Binary`, ranges, or storage. `await split(...)` suspends the
+parent (one pending activity, one resume, one replay) until every child is
+terminal — **regardless of each child succeeding or failing**: a rejected
+child never rejects the batch (allSettled semantics). An empty batch resolves
+immediately.
+
+### Delimiters (`options.delimiter`)
+
+The cut is a serializable spec. When the delimiter is found at
+`[p, p + length)`, the previous slice ends at `p + keep.previous` and the next
+slice starts at `p + length - keep.next`; the bytes in between are discarded
+(defaults discard the whole delimiter — classic line semantics).
+
+```typescript
+Split.line()                     // cut at every '\n', delimiter discarded
+Split.char(';')                  // cut at any byte sequence
+Split.char(';', { previous: 1 }) // keep the ';' with the previous slice
+Split.char('\n10', { next: 2 })  // positional files, see below
+```
+
+Positional files where a logical record spans multiple physical lines and a
+new record only starts when a line begins with a marker (the record type
+code):
+
+```
+10JOHN;123\n20detail-a\n20detail-b\n10MARY;456\n20detail-c
+└──────────── record 1 ───────────┘└─────── record 2 ──────┘
+```
+
+`Split.char('\n10', { next: 2 })`: the newline between records is discarded
+and the `10` marker — which is **data**, not separator — stays at the start of
+the next slice. Forgetting `{ next: 2 }` silently strips the record type code
+from every slice.
+
+Offsets are always computed over **raw bytes** (never decoded characters), so
+multi-byte encodings cannot skew the ranges a ranged download will use.
+
+### Other options
+
+| Option | Default | Behavior |
+|--------|---------|----------|
+| `every` | `1` | Cut only every N delimiter occurrences — groups N records per child. One million lines with `every: 10_000` start 100 children, each receiving a raw block (inner delimiters untouched) to iterate locally. `keep` arithmetic applies only at actual cut points. |
+| `skipEmpty` | `true` | Drops zero-length slices (consecutive delimiters, file starting/ending with one) so they never become child workflows. Turn off when slice index must align 1:1 with record position. "Empty" means zero bytes — a whitespace-only line is a regular slice. |
+| `encoding` | `'utf8'` | Converts the delimiter to bytes and tells the runtime how to hydrate each slice for the child (string when set on the slice). |
+| `timeout` | none | Max milliseconds to await the batch; on expiry the activity **rejects** (reason `timeout`) instead of suspending forever — the safety net for a child that never reaches a terminal state. |
+
+---
+
+## `join`
+
+```typescript
+async function join(
+  batch: WorkflowBatchRef,
+  options: FileJoinOptions,
+): Promise<Binary>
+```
+
+Formats (content is the **child's** job — the child returns the final line;
+`join` only owns structure):
+
+| Format | Behavior |
+|--------|----------|
+| `lines` | Concatenates raw child outputs, one per line. Options: `header`, `separator` (default `'\n'`). Non-string outputs are JSON-stringified. |
+| `entries` | NDJSON dump of `{ index, status, output }`, one per line — the format-agnostic view of the batch, rejected children included by default. |
+| `csv` | Projects object outputs into RFC 4180 escaped columns. Options: `columns` (required; entries are `{ key, header? }` — `key` selects the value on the child output, `header` labels the header row, plain string is shorthand for `{ key }`), `delimiter` (default `','`), `header` (default `true`). Missing keys become empty cells; nested values are JSON inside the cell. Value *formatting* belongs in the child workflow (sandboxed user code); the join only owns layout. |
+
+**Status filter** (`status?: ('fulfilled' | 'rejected')[]`) selects which
+results make it into the file — there is no error policy. Defaults:
+`['fulfilled']` for `lines`/`csv`, both for `entries`. Filtering is silent by
+design; a reprocessing file of failures is one call away:
+
+```typescript
+const failures = await join(batch, { format: 'entries', status: ['rejected'] })
+```
+
+---
+
+## Runtime requirements
+
+- Claim providers: `FileSystemClaim` (with `slice`), `WorkflowClaim` (with
+  `startMany`), and `WorkflowBatchClaim` (from `@t4h.framework/core`).
+- Production whitelist: `FileSplitActivity`, `FileJoinActivity` and core's
+  `StartManyWorkflowsActivity` must be registered via `runtime.addActivity`.
+- `FileSplitActivity` is an **AsyncActivity**: the heavy work happens in
+  `bootstrap`, executed worker side atomically with the pending suspension —
+  children cannot finish before the parent is awaiting them.
+- `FileJoinActivity` is a **SyncActivity**: batches are only consumed after
+  the completion barrier resolved, so there is nothing to await.
+
+---
+
+## Testing
+
+Drive `bootstrap`/`run` directly with fake claims inside `Claim.run` (patterns
+from `src/activities/__tests__/`):
+
+```typescript
+import { AsyncActivityResultPending, Claim, WorkflowBatchClaim, WorkflowBatchRef, WorkflowClaim } from '@t4h.framework/core'
+import { FileSplitActivity, FileSystemClaim, Split } from '@t4h.framework/fs'
+
+const write = vi.fn(async (_path, readable) => durableBinary) // must consume the stream
+const slice = vi.fn((binary, range, encoding) => ({ binary, range, encoding }))
+const startMany = vi.fn(async (_wf, inputs) => new WorkflowBatchRef('batch-1', inputs.length))
+
+const result = await Claim.run(
+  [
+    { provide: FileSystemClaim, value: { write, slice } as never },
+    { provide: WorkflowClaim, value: { startMany } as never },
+  ],
+  () => {
+    const activity = new FileSplitActivity()
+    activity.id = 'activity-1'
+    return activity.bootstrap({ binary, workflow: 'child', options: { delimiter: Split.line() } })
+  },
+)
+
+expect(result).toBeInstanceOf(AsyncActivityResultPending)
+expect(slice).toHaveBeenCalledWith(durableBinary, { start: 0, end: 3 }, 'utf8')
+```
+
+For `join`, provide `WorkflowBatchClaim` as an async generator:
+
+```typescript
+const outputs = vi.fn(async function* () {
+  yield { index: 0, status: 'fulfilled', output: 'line-1' }
+})
+```
+
+Behaviors worth asserting because they regress easily: the durable copy keeps
+the original bytes; slices are views over the durable copy (not the source);
+`keep` arithmetic on positional cuts; the empty batch resolving immediately;
+CSV escaping (quotes doubled, delimiter/newline wrapped); the status filter.
+
+---
+
+## Anti-patterns
+
+- **Paging batch results into the workflow.** A `readBatch(ref, cursor)` loop
+  in workflow code turns every page into a replayed history entry — O(N) is
+  back. Aggregation belongs inside `join` (or a framework activity consuming
+  `WorkflowBatchClaim`).
+- **Expecting per-child outputs on the ref.** `WorkflowBatchRef` is
+  deliberately `{ id, count }` only. Results are read through `join` — e.g.
+  `format: 'entries'` for the faithful dump.
+- **Treating a rejected child as a batch failure.** The batch always resolves
+  (allSettled). Decide what failures mean: compare against `batch.count`,
+  generate the `status: ['rejected']` file, or tolerate.
+- **Baking record format into `join`.** The child returns the final line
+  (`lines`) or a projectable object (`csv`); custom formats live in the child
+  workflow — sandboxed user code — not in new join options.
+- **Splitting when you need the file whole.** If downstream wants the blob
+  intact (hash it, upload it), pass the `Binary` through.
+- **Forgetting `keep` on positional delimiters.** `Split.char('\n10')` without
+  `{ next: 2 }` silently discards the record type code from every slice.
+- **Hand-rolling the loop.** A stream-and-`startWorkflow` loop inside an
+  activity re-parses on replay, misses the barrier, and pushes N entries into
+  history.
+
+---
+
+## Related skills
+
+- `framework-fs` — `FileSystemClaim` contract (including `slice`), `Binary`,
+  provider wiring.
+- `framework-core` — `startWorkflowMany` (the array-input sibling of `split`),
+  `WorkflowBatchRef`, `WorkflowBatchClaim`, `Claim.run`, activity model,
+  testing harness.