@t4h.framework/fs 0.5.0 → 0.7.0

package/.ai/skills/framework-fs/SKILL.md

@@ -19,10 +19,17 @@ Package path: `framework/packages/fs`. Peer dependency: `@t4h.framework/core` (`
 
 ## Import surface
 
-The package exports a single public symbol from `src/fs.ts`:
+Public exports live in `src/fs.ts`:
 
 ```typescript
-import { FileSystemClaim } from '@t4h.framework/fs'
+import {
+  FileSystemClaim,
+  // file fan-out / fold / windows — see the framework-fs-split skill
+  split, Split, FileSplitActivity, type FileSplitOptions,
+  join, FileJoinActivity, type FileJoinOptions, type FileJoinWhere,
+  read, FileReadActivity, type FileReadOptions,
+  ByteDelimiterScanner, type FileSplitDelimiter, type ByteRange, type ByteTrim,
+} from '@t4h.framework/fs'
 ```
 
 Do not import from internal paths such as `src/FileSystemClaim.ts` in application code.
@@ -42,6 +49,12 @@ export abstract class FileSystemClaim {
   ): Binary | Promise<Binary>
 
   public abstract read(path: string): Binary | Promise<Binary>
+
+  public abstract slice(
+    binary: Binary,
+    range: { start: number; end: number },
+    encoding?: BufferEncoding,
+  ): Binary | Promise<Binary>
 }
 ```
 
@@ -49,8 +62,9 @@ export abstract class FileSystemClaim {
 |--------|---------|
 | `write(path, readable)` | Persist stream or buffer at `path`; return a `Binary` handle (metadata + `stream()`). |
 | `read(path)` | Load an existing file at `path`; return a `Binary` handle. |
+| `slice(binary, range, encoding?)` | A view over a byte range of an existing binary, **without copying data**. When `encoding` is set, the runtime hydrates the slice as a decoded string instead of a binary — this is how `split` children receive plain strings. Implementations must **compose ranges**: slicing a slice is relative to it, and the stored range is absolute against the underlying file. |
 
-Both methods may return synchronously or as a `Promise`. Implementations choose storage layout; callers should only rely on the returned `Binary` API.
+All methods may return synchronously or as a `Promise`. Implementations choose storage layout; callers should only rely on the returned `Binary` API.
 
 ### Binary (from `@t4h.framework/core`)
 
@@ -82,6 +96,10 @@ read the resulting `Binary` (e.g. `await res.body.json()` / `text()`). Your only
 responsibility is that the runtime **registers a provider** (below). See
 **Known consumers** for which package primitives to orchestrate.
 
+This package's own workflow-facing primitives are `split`, `join` and `read`
+(small byte windows — headers, trailers — hydrated into the workflow as a
+string, capped at 1 MiB). See the `framework-fs-split` skill.
+
 ---
 
 ## Runtime provider
@@ -104,6 +122,7 @@ These packages depend on `@t4h.framework/fs` and declare `FileSystemClaim`; read
 |---------|--------|
 | `@t4h.framework/http` | The HTTP activity persists response bodies to `fs` and returns them wrapped in a `Body` — you read them with `res.body.json()` / `text()`. |
 | `@t4h.framework/jsx` | Render activities write their output to `fs` and hand back a `Binary` you consume in the workflow. |
+| `@t4h.framework/fs` itself | `FileSplitActivity` persists the durable copy and slices it by ranges; `FileJoinActivity` streams batch results into a new file; `FileReadActivity` slices (or streams, when the binary is not file-backed) a small window into a string. See the `framework-fs-split` skill. |
 
 Orchestrate these package primitives from your workflow; the `fs` write/read happens inside them.
 
@@ -186,6 +205,7 @@ Assert `mockWrite` was called with the expected path and body when relevant.
 
 ## Related skills
 
+- `framework-fs-split` — `split`/`join`: fan a file out into a batch of workflows and fold the results back into a file.
 - `framework-core` — `Claim`, `Binary`, and the `TestWorkflowEnvironment` testing harness.
 - `framework-http` — HTTP activities that persist bodies via `FileSystemClaim`.
 - `framework-jsx` — render activities that write/read via `fs`.

package/.ai/skills/framework-fs-split/SKILL.md

@@ -0,0 +1,350 @@
+---
+name: framework-fs-split
+description: >-
+  Guides correct use of `split`, `join` and `read` from @t4h.framework/fs:
+  fanning one large file out into a batch of workflows (one per byte-range
+  slice), folding the batch results back into a single file, and reading small
+  byte windows (headers, trailers) into the workflow — with O(1) parent
+  history and bounded memory regardless of file size. Use whenever a workflow
+  must process a large file record by record — positional files with
+  header/trailer, plain-text lines, NDJSON, batch imports, per-record fan-out
+  with an aggregated output (e.g. a CSV report) — or when the user mentions
+  splitting/chunking a file, trimming a header/trailer, reading a file header,
+  filtering batch results, running a workflow per line/record, waiting for a
+  batch, collecting batch results, or the symbols `split`, `join`, `read`,
+  `Split`, `FileSplitActivity`, `FileJoinActivity`, `FileReadActivity`,
+  `WorkflowBatchRef`, `WorkflowBatchClaim`. Prefer this skill over
+  hand-rolling stream loops + `startWorkflow` calls.
+---
+
+# `split` / `join` / `read` — file fan-out, fold and windows for workflows
+
+`split` turns **one file into a batch of workflows**: it streams the file once,
+cuts it into **byte-range slices** (never copying slice content), starts one
+child workflow per slice, and **suspends the parent until every child is
+terminal**. `join` is the inverse: it folds the batch results back into a
+single file, streaming one result at a time.
+
+```typescript
+import { read, split, join, Split } from '@t4h.framework/fs'
+
+const processRecord = new Workflow(
+  { id: 'process-record', schema: Type.String() },
+  async (record: string) => {
+    // record is the raw slice content — no Binary handling needed
+    return toCsvRow(parsePositional(record))
+  },
+)
+
+const main = new Workflow(fileSchema, async (file: Binary) => {
+  const header = await read(file, { range: { start: 0, end: 240 } })
+  // header: string — recorded in history, never re-read on replay
+
+  const batch = await split(file, processRecord, {
+    delimiter: Split.char('\n10', { next: 2 }),
+    trim: { start: 241, end: 241 }, // header/trailer stay out of the batch
+    timeoutInMs: 3_600_000,
+  })
+  // this line only runs once the WHOLE batch is terminal;
+  // batch: WorkflowBatchRef — { id: string, count: number }
+
+  return await join(batch, { format: 'lines', header: 'id;value;date' })
+  // Binary: the aggregated file — no slice ever crossed workflow memory
+})
+```
+
+Package path: `framework/packages/fs`. Import only from the package root.
+
+---
+
+## The memory model (the golden rule)
+
+Everything here exists to keep the parent history **O(1)** whatever the file
+size:
+
+1. `split` reads the source stream **exactly once, worker side**: the same
+   pass persists a durable copy through `FileSystemClaim.write` while a byte
+   scanner records slice offsets. No slice content is materialized.
+2. Children receive **range views** over the durable copy
+   (`FileSystemClaim.slice`); the runtime hydrates each slice only inside that
+   child's own run — as a plain decoded string when the child's schema is
+   `Type.String()`.
+3. The parent resumes with only `WorkflowBatchRef { id, count }`. Per-child
+   results are **never aggregated onto the ref**: they live with each child
+   run.
+4. `join` streams results one at a time through `WorkflowBatchClaim.outputs`
+   into `FileSystemClaim.write` with backpressure; only the resulting `Binary`
+   reaches history.
+
+**Golden rule:** volume proportional to the batch size only ever flows inside
+an activity, through a claim. Into workflow history goes a fold result or a
+bounded slice — never the N items.
+
+---
+
+## `split`
+
+```typescript
+async function split(
+  binary: Binary,
+  workflow: Workflow<any, any>,
+  options: FileSplitOptions,
+): Promise<WorkflowBatchRef>
+```
+
+The child workflow declares its **real input type** (e.g. `Type.String()`) —
+it never sees `Binary`, ranges, or storage. `await split(...)` suspends the
+parent (one pending activity, one resume, one replay) until every child is
+terminal — **regardless of each child succeeding or failing**: a rejected
+child never rejects the batch (allSettled semantics). An empty batch resolves
+immediately.
+
+### Delimiters (`options.delimiter`)
+
+The cut is a serializable spec. When the delimiter is found at
+`[p, p + length)`, the previous slice ends at `p + keep.previous` and the next
+slice starts at `p + length - keep.next`; the bytes in between are discarded
+(defaults discard the whole delimiter — classic line semantics).
+
+```typescript
+Split.line()                     // cut at every '\n', delimiter discarded
+Split.char(';')                  // cut at any byte sequence
+Split.char(';', { previous: 1 }) // keep the ';' with the previous slice
+Split.char('\n10', { next: 2 })  // positional files, see below
+```
+
+Positional files where a logical record spans multiple physical lines and a
+new record only starts when a line begins with a marker (the record type
+code):
+
+```
+10JOHN;123\n20detail-a\n20detail-b\n10MARY;456\n20detail-c
+└──────────── record 1 ───────────┘└─────── record 2 ──────┘
+```
+
+`Split.char('\n10', { next: 2 })`: the newline between records is discarded
+and the `10` marker — which is **data**, not separator — stays at the start of
+the next slice. Forgetting `{ next: 2 }` silently strips the record type code
+from every slice.
+
+Offsets are always computed over **raw bytes** (never decoded characters), so
+multi-byte encodings cannot skew the ranges a ranged download will use.
+
+### Other options
+
+| Option | Default | Behavior |
+|--------|---------|----------|
+| `every` | `1` | Cut only every N delimiter occurrences — groups N records per child. One million lines with `every: 10_000` start 100 children, each receiving a raw block (inner delimiters untouched) to iterate locally. `keep` arithmetic applies only at actual cut points. |
+| `trim` | none | `{ start?, end? }` **bytes** cut off each end of the file before scanning — a positional header/trailer. Byte-based on purpose: a record-start delimiter (`'\n10'`) fuses the trailer into the last record's slice, so a record-count skip would drop a valid record along with it. Trimmed content never becomes a slice nor counts toward `every`; fetch it with `read`. Positional layouts make the byte counts constants — getting them right is the caller's business. |
+| `skipEmpty` | `true` | Drops zero-length slices (consecutive delimiters, file starting/ending with one) so they never become child workflows. Turn off when slice index must align 1:1 with record position. "Empty" means zero bytes — a whitespace-only line is a regular slice. |
+| `encoding` | `'utf8'` | Converts the delimiter to bytes and tells the runtime how to hydrate each slice for the child (string when set on the slice). |
+| `timeoutInMs` | none | Max milliseconds to await the batch; on expiry the activity **rejects** (reason `timeout`) instead of suspending forever — the safety net for a child that never reaches a terminal state. |
+
+---
+
+## `join`
+
+```typescript
+async function join(
+  batch: WorkflowBatchRef,
+  options: FileJoinOptions,
+): Promise<Binary>
+```
+
+Formats (content is the **child's** job — the child returns the final line;
+`join` only owns structure):
+
+| Format | Behavior |
+|--------|----------|
+| `lines` | Concatenates raw child outputs, one per line. Options: `header`, `separator` (default `'\n'`). Non-string outputs are JSON-stringified. |
+| `entries` | NDJSON dump of `{ index, status, output }`, one per line — the format-agnostic view of the batch, rejected children included by default. |
+| `csv` | Projects object outputs into RFC 4180 escaped columns. Options: `columns` (required; entries are `{ key, header? }` — `key` selects the value on the child output (a dot-path like `'meta.total'` reaches nested fields, matching the `where` semantics; a literal top-level key with dots wins over the nested path), `header` labels the header row, plain string is shorthand for `{ key }`), `delimiter` (default `','`), `header` (default `true`). Missing keys become empty cells; nested values selected whole are JSON inside the cell. Value *formatting* belongs in the child workflow (sandboxed user code); the join only owns layout. |
+
+**Status filter** (`status?: ('fulfilled' | 'rejected')[]`) selects which
+results make it into the file — there is no error policy. Defaults:
+`['fulfilled']` for `lines`/`csv`, both for `entries`. Filtering is silent by
+design; a reprocessing file of failures is one call away:
+
+```typescript
+const failures = await join(batch, { format: 'entries', status: ['rejected'] })
+```
+
+**Where filter** (`where?`) narrows results with a **sift (MongoDB-style)
+query** over each entry's output, applied after `status`. Whitelisted subset
+only — declarative, JSON-only operators: `$eq`, `$ne`, `$lt`, `$lte`, `$gt`,
+`$gte`, `$in`, `$nin`, `$exists`, `$not`, `$and`, `$or`, `$nor`, plus plain
+equality and dot-paths for nested fields. Pass the child output type as a
+generic to narrow top-level keys/values (dot-paths accepted, not narrowed):
+
+```typescript
+const relevant = await join<InvoiceResult>(batch, {
+  format: 'csv',
+  columns: ['id', 'total'],
+  where: { valid: true, total: { $gt: 100 }, 'meta.situacao': 'aprovada' },
+})
+```
+
+Non-serializable filters are rejected eagerly at `toInput` (`TypeError`):
+functions (incl. `$where`), `RegExp` (incl. `$regex`), `undefined`, operators
+outside the subset — a dropped closure would make the replay filter
+differently than the first execution. Arbitrary predicates belong in a custom
+activity in project code (the function then lives in versioned code).
+MongoDB semantics apply: array fields match by containment (`{ tags: 'a' }`
+matches `['a', 'b']`), object values compare deeply, no type coercion. The
+query compiles once and runs inside the existing streaming loop over
+already-hydrated outputs — same memory profile; app-side (decides what enters
+the file, not what leaves the database).
+
+---
+
+## `read`
+
+```typescript
+async function read(
+  binary: Binary,
+  options?: { range?: { start: number; end: number }; encoding?: BufferEncoding },
+): Promise<string>
+```
+
+Reads a byte window of a `Binary` into the workflow as a decoded string. The
+output is recorded in history (replay never re-reads the file), so reads are
+**capped at 1 MiB** — this is the primitive for headers, trailers and single
+records; a data file flows through `split`, never through `read`. `range`
+defaults to the whole binary (same cap applies); `encoding` defaults to
+`'utf8'`.
+
+Two paths under the hood:
+
+1. **Fast path** — `FileSystemClaim.slice`: on a file-backed binary a trailer
+   read is a ranged download, whatever the file size.
+2. **Streaming fallback** — binaries the claim cannot slice (e.g. an SFTP
+   reference): the stream is consumed up to the window and destroyed as soon
+   as it is filled. A header read pulls the first chunks only, whatever the
+   file size; a **trailing window over a non-sliceable binary** is the one
+   expensive case — the stream traverses the whole file to reach it.
+
+Reading a slice is **relative to the slice** (`range: { start: 0, end: 10 }`
+returns its first 10 bytes, not the underlying file's) — slice ranges compose.
+
+The positional-file recipe — header/trailer in the parent, records in the
+batch:
+
+```typescript
+const header = await read(file, { range: { start: 0, end: HEADER } })
+const trailer = await read(file, {
+  range: { start: file.contentLength - TRAILER, end: file.contentLength },
+})
+
+const batch = await split(file, processInvoice, {
+  delimiter: Split.char('\n10', { next: 2 }),
+  trim: { start: HEADER + 1, end: TRAILER + 1 }, // +1: boundary newlines
+})
+
+// e.g. validate the trailer's record count against batch.count
+```
+
+---
+
+## Runtime requirements
+
+- Claim providers: `FileSystemClaim` (with `slice`), `WorkflowClaim` (with
+  `startMany`), and `WorkflowBatchClaim` (from `@t4h.framework/core`).
+- Production whitelist: `FileSplitActivity`, `FileJoinActivity`,
+  `FileReadActivity` and core's `StartManyWorkflowsActivity` must be
+  registered via `runtime.addActivity`.
+- `FileSystemClaim.slice` implementations must **compose ranges** when slicing
+  an already-sliced binary (relative in, absolute against the underlying file
+  out) — `read` and slice consumers rely on that semantics.
+- `FileSplitActivity` is an **AsyncActivity**: the heavy work happens in
+  `bootstrap`, executed worker side atomically with the pending suspension —
+  children cannot finish before the parent is awaiting them.
+- `FileJoinActivity` is a **SyncActivity**: batches are only consumed after
+  the completion barrier resolved, so there is nothing to await.
+
+---
+
+## Testing
+
+Drive `bootstrap`/`run` directly with fake claims inside `Claim.run` (patterns
+from `src/activities/__tests__/`):
+
+```typescript
+import { AsyncActivityResultPending, Claim, WorkflowBatchClaim, WorkflowBatchRef, WorkflowClaim } from '@t4h.framework/core'
+import { FileSplitActivity, FileSystemClaim, Split } from '@t4h.framework/fs'
+
+const write = vi.fn(async (_path, readable) => durableBinary) // must consume the stream
+const slice = vi.fn((binary, range, encoding) => ({ binary, range, encoding }))
+const startMany = vi.fn(async (_wf, inputs) => new WorkflowBatchRef('batch-1', inputs.length))
+
+const result = await Claim.run(
+  [
+    { provide: FileSystemClaim, value: { write, slice } as never },
+    { provide: WorkflowClaim, value: { startMany } as never },
+  ],
+  () => {
+    const activity = new FileSplitActivity()
+    activity.id = 'activity-1'
+    return activity.bootstrap({ binary, workflow: 'child', options: { delimiter: Split.line() } })
+  },
+)
+
+expect(result).toBeInstanceOf(AsyncActivityResultPending)
+expect(slice).toHaveBeenCalledWith(durableBinary, { start: 0, end: 3 }, 'utf8')
+```
+
+For `join`, provide `WorkflowBatchClaim` as an async generator:
+
+```typescript
+const outputs = vi.fn(async function* () {
+  yield { index: 0, status: 'fulfilled', output: 'line-1' }
+})
+```
+
+Behaviors worth asserting because they regress easily: the durable copy keeps
+the original bytes; slices are views over the durable copy (not the source);
+`keep` arithmetic on positional cuts; the empty batch resolving immediately;
+CSV escaping (quotes doubled, delimiter/newline wrapped); the status filter.
+
+---
+
+## Anti-patterns
+
+- **Paging batch results into the workflow.** A `readBatch(ref, cursor)` loop
+  in workflow code turns every page into a replayed history entry — O(N) is
+  back. Aggregation belongs inside `join` (or a framework activity consuming
+  `WorkflowBatchClaim`).
+- **Expecting per-child outputs on the ref.** `WorkflowBatchRef` is
+  deliberately `{ id, count }` only. Results are read through `join` — e.g.
+  `format: 'entries'` for the faithful dump.
+- **Treating a rejected child as a batch failure.** The batch always resolves
+  (allSettled). Decide what failures mean: compare against `batch.count`,
+  generate the `status: ['rejected']` file, or tolerate.
+- **Baking record format into `join`.** The child returns the final line
+  (`lines`) or a projectable object (`csv`); custom formats live in the child
+  workflow — sandboxed user code — not in new join options.
+- **Splitting when you need the file whole.** If downstream wants the blob
+  intact (hash it, upload it), pass the `Binary` through.
+- **Forgetting `keep` on positional delimiters.** `Split.char('\n10')` without
+  `{ next: 2 }` silently discards the record type code from every slice.
+- **Splitting a positional file without `trim`.** The header rides inside the
+  first slice (with `every: N` it silently consumes a record slot) and the
+  trailer fuses into the last one. Trim both ends; fetch them with `read`.
+- **Reading a data file with `read`.** The output lands in workflow history;
+  the 1 MiB cap is a guard, not a target. Records flow through `split`.
+- **Passing a function as a join filter.** Options serialize into history — a
+  closure cannot survive it, so `toInput` rejects functions, `$where`,
+  `RegExp` and out-of-subset operators with a `TypeError`. Use the `where`
+  sift subset or a custom activity whose predicate lives in versioned code.
+- **Hand-rolling the loop.** A stream-and-`startWorkflow` loop inside an
+  activity re-parses on replay, misses the barrier, and pushes N entries into
+  history.
+
+---
+
+## Related skills
+
+- `framework-fs` — `FileSystemClaim` contract (including `slice`), `Binary`,
+  provider wiring.
+- `framework-core` — `startWorkflowMany` (the array-input sibling of `split`),
+  `WorkflowBatchRef`, `WorkflowBatchClaim`, `Claim.run`, activity model,
+  testing harness.