tranfi 0.2.0 → 0.2.1

package/README.md

@@ -1,14 +1,11 @@
 # tranfi (Node.js / WASM)
 
-Streaming-first ETL in JavaScript, powered by a native C11 core via N-API
-(Node.js) or WASM (browsers). Tranfi processes CSV, JSONL, and text byte streams;
-it is not an in-memory DataFrame API. Row-local operations stream, bounded
-operators declare their limits, and full-input operators require an explicit
-blocking or spill policy.
-
-> **Unreleased main:** The prepared-transform API documented below targets
-> Tranfi 0.2. Current npm 0.1.x installs do not include it; build this branch
-> from source until 0.2 is published.
+Tranfi transforms CSV, JSON Lines and plain text as streams. In Node.js it runs
+on a native C core; in the browser it runs on WebAssembly. Data flows through in
+chunks, so most pipelines use a small, fixed amount of memory however large the
+input is. Operations that need the whole input, such as `sort`, must be allowed
+explicitly. Tranfi works on byte streams; it is not an in-memory DataFrame
+library.
 
 Save this example as `quickstart.mjs`:
 
@@ -35,6 +32,7 @@ console.log(result.outputText)
 
 Or use the pipe DSL for one-liners:
 
+<!-- readme-test: js-api -->
 ```js
 const result = await pipeline('csv | filter "col(age) > 25" | top-k 100 age | csv')
   .run({ inputFile: 'data.csv' })
@@ -46,17 +44,19 @@ const result = await pipeline('csv | filter "col(age) > 25" | top-k 100 age | cs
 npm install tranfi
 ```
 
-The default install compiles the N-API addon and reports a nonzero failure if
-the native toolchain or synchronized C sources are unavailable. For an
-intentional WASM-only/browser installation, set
-`TRANFI_SKIP_NATIVE_BUILD=1` and import `tranfi/wasm` explicitly. The ordinary
-pipeline can execute with WASM, but root-entry prepared transforms require the
-native addon.
+Installation compiles Tranfi's C core as a native Node.js addon, so you need a C
+compiler. If that compile fails, the install fails.
+
+In the browser, import `tranfi/wasm`: it provides pipelines and prepared
+transforms without the native addon. To install only the WebAssembly build, run
+`TRANFI_SKIP_NATIVE_BUILD=1 npm install tranfi`. The root `tranfi` entry then
+runs pipelines through WebAssembly, but its prepared-transform classes need the
+native addon, so use `tranfi/wasm` for those.
 
-The native addon is currently built and tested on Linux. Windows is supported
-through the packed package's `tranfi/wasm` entry with
-`TRANFI_SKIP_NATIVE_BUILD=1`; the release CI runs streaming and prepared-transform
-smokes in that configuration. This is not a Windows native-addon support claim.
+| | Linux | Windows | macOS |
+|---|---|---|---|
+| Native addon | Tested | Not tested | Not tested |
+| WebAssembly (`tranfi/wasm`) | Tested in Node.js and Chromium | Tested in Node.js | Not tested |
 
 Use `pipeline(...)` for byte-stream ETL. The separate
 `TransformRecipe -> TransformAnalyzer -> TransformPlan -> TransformApply`
@@ -70,7 +70,7 @@ Installing the package also provides the `tranfi` command:
 
 ```bash
 # Via npx (no install)
-echo 'name,age\nAlice,30\nBob,25' | npx tranfi -q 'csv | filter "age > 25" | csv'
+printf 'name,age\nAlice,30\nBob,25\n' | npx tranfi -q 'csv | filter "age > 25" | csv'
 
 # Or install globally
 npm i -g tranfi
@@ -79,7 +79,11 @@ tranfi profile < data.csv
 tranfi -R  # list recipes
 ```
 
-Run `tranfi -h` for all options.
+Run `tranfi -h` for the npm CLI options. Use `--allow-blocking` for known-small
+full-input operations, `--memory max:64MB` for a memory policy, and
+`--spill-dir DIR` for an existing private spill directory. `--stats-json FILE`
+writes the stats channel; `--target json|sql` compiles without executing.
+The detailed `--explain` report requires the standalone C CLI built from source.
 
 ## Quick start
 
@@ -87,6 +91,7 @@ Run `tranfi -h` for all options.
 
 **Builder API** -- composable, structured, and IDE-friendly:
 
+<!-- readme-test: js-api -->
 ```js
 const p = pipeline([
   codec.csv(),
@@ -100,6 +105,7 @@ const result = await p.run({ inputFile: 'students.csv' })
 
 **DSL strings** -- compact, suitable for CLI-like use:
 
+<!-- readme-test: js-api -->
 ```js
 const p = pipeline('csv | filter "col(score) >= 80" | top-k 10 score | csv')
 const result = await p.run({ inputFile: 'students.csv' })
@@ -111,12 +117,13 @@ Dataframe-style DSL aliases are accepted and normalize to canonical ops: `mutate
 
 ### Running pipelines
 
+<!-- readme-test: js-pipeline -->
 ```js
 // From string or Buffer
 const result = await p.run({ input: 'name,age\nAlice,30\n' })
 
 // From file (streamed in 64 KB chunks)
-const result = await p.run({ inputFile: 'data.csv' })
+const fileResult = await p.run({ inputFile: 'data.csv' })
 
 // Access results
 result.output         // Buffer
@@ -129,8 +136,9 @@ result.samples        // Buffer (sample channel)
 
 For large outputs, drain chunks instead of collecting `result.output`:
 
+<!-- readme-test: js-pipeline -->
 ```js
-const { createReadStream, createWriteStream } = require("fs")
+import { createReadStream, createWriteStream } from 'node:fs'
 
 // Write to a Node Writable and wait for `drain` when the sink applies backpressure.
 const out = createWriteStream("out.csv")
@@ -159,12 +167,13 @@ for await (const chunk of p.iterChunks({ inputFiles, sourceColumn: "src" })) {
 }
 ```
 
-`sourceColumn` appends the path for each input row without preloading file contents. Tranfi flushes decoder input at each file boundary so an unterminated final record belongs to the correct source file. It does not remove repeated CSV headers from later files; use shards without repeated headers, `header: false`, or a pre-cleaning step when every file has its own header.
+`sourceColumn` appends the path for each input row without preloading file contents. Tranfi flushes decoder input at each file boundary so an unterminated final record belongs to the correct source file. Repeated headers are kept by default. Use `codec.csv({ skipRepeatedHeader: true })` to skip a later file's first non-comment record when it exactly matches the original header; matching rows inside a file remain data.
 
 Standalone WASM exposes the same non-collecting shape for in-memory data: `tf.run(dsl, data, { onOutput, collectOutput: false })` and `tf.iterChunks(dsl, data)`.
 
 For browser UI work, keep Worker placement outside the core pipeline and use the optional WASM Worker adapter:
 
+<!-- readme-test: browser -->
 ```js
 // main thread
 import { createWorkerClient } from 'tranfi/wasm/worker'
@@ -189,21 +198,16 @@ The bundled Tranfi app runner is also preview-bounded by default: file chunks ar
 
 ### Prepared reusable transforms
 
-WASM cancellation tokens created with `createTransformCancelToken()` expose a read-only `requested` boolean for host-side conversion loops. Reading a closed token fails.
-
-Prepared transforms are a separate typed-table API for operations whose parameters must be learned from reference data. `analyze` accumulates bounded statistics over one or more batches, `finalize` freezes an immutable plan and output schema, and `apply` runs that plan either over a second pass of the original data (`fit_transform`-style) or over later compatible batches. It does not replace the byte-stream pipeline API.
-
-The current slice accepts declared `float32`/`float64` columns. It supports numeric none/zero/constant/mean/exact-median imputation and none/standard/min-max normalization. Declared categorical columns support mode imputation with `allMissing: 'error' | 'zero'` or `impute.op: 'none'`; encoders discover finite typed categories, while the no-imputation/no-encoding combination needs no learned dictionary. A column may instead provide both branches with `kind.op: 'infer'`, `kind.rule: 'finite-integer-cardinality-v1'`, and `kind.maxCategories >= 2`: missing/NaN values are ignored, `2..maxCategories` distinct finite integers resolve categorical, while zero/one distinct value, any noninteger, or the next distinct value resolves numeric. `impute.op: 'none'` with `encode.op: 'none'` passes every finite value through and emits canonical qNaN for missing input. Mode with `encode.op: 'none'` retains its learned dictionary and rejects unseen finite values with code `108`. `encode.op: 'label'` freezes zero-based sorted ordinals and supports `unknown: 'error' | 'sentinel' | 'other'`; `encode.op: 'onehot'` emits source-ordered category blocks and supports `unknown: 'error' | 'all_zero' | 'other'`. Without categorical imputation, missing label/one-hot input follows that encoder's unknown policy. The label sentinel must be a safe integer outside the learned ordinal range; label/one-hot `other` appends the reserved ordinal/field after known categories. Generated output IDs/names and one-hot category metadata are deterministic, and collisions fail with code `102`. Exact median obeys the configured allocation and resident-state limits; inference, categorical discovery, and output expansion obey category, output-width, allocation, and resident-state limits. Limit failures use resource code `104`. Prepared-transform host-policy and spill fields are reserved but not implemented; nonempty use fails with unsupported-runtime code `113` instead of being silently ignored.
-
-Declared categorical columns also accept a fixed, nonempty `encode.categories` array of finite, sorted, unique tags (`{ "t": "f64", "v": "4000000000000000" }` represents 2). Tags must match the input dtype; negative zero is represented as positive zero. Fixed dictionaries retain unobserved categories. Unknown training values follow the encoder policy and never vote for mode; an all-missing zero fallback must exist in the dictionary. Fixed encoding without imputation can finalize without training rows. Fixed dictionaries with kind inference, string categories, and categorical constant imputation remain unsupported.
+Learn imputation, scaling or category mappings from reference data, then apply
+the same plan to new batches. Analyze the reference batches, finalize the plan,
+and apply it. Replaying the reference data through that plan gives a second-pass
+`fit_transform` workflow.
 
-Prepared runtime option objects reject unknown fields. Native Node accepts
-`limits`, `cancelFlag`, `hostPolicy`, and `spillDir`; standalone WASM accepts
-`limits`, `cancelToken`, `hostPolicy`, and `spillDir`. The host/spill fields are
-reserved as described above, and cancellation spellings are not interchangeable.
+Inputs are typed `float32`/`float64` columns. This example learns mean imputation
+and standard scaling, applies them to the reference batch, and exports the plan:
 
 ```js
-const tf = require('tranfi')
+import tf from 'tranfi'
 
 const recipeSpec = {
   format: 'tranfi.transform-recipe',
@@ -245,8 +249,85 @@ analyzer.close()
 recipe.close()
 ```
 
+<details markdown="1">
+<summary>Available methods, missing values and kind inference</summary>
+
+Recipe fields use the same names in all bindings.
+
+| Recipe field | Choices / behavior |
+|--------------|--------------------|
+| Numeric `impute.op` | `none`, `zero`, `constant`, `mean`, exact `median` |
+| Numeric `normalize.op` | `none`, `standard`, `minmax` |
+| Categorical `impute.op` | `none` or mode; mode supports `allMissing` of `error` or `zero` |
+| Categorical `encode.op` | `none`, `label`, `onehot` |
+| Label unknown policy | `error`, `sentinel`, `other` |
+| One-hot unknown policy | `error`, `all_zero`, `other` |
+
+**No imputation or encoding:** finite values pass through; missing input becomes
+canonical qNaN. This combination needs no learned category dictionary.
+Mode imputation with no encoding retains a dictionary and rejects unseen finite
+values with error `108`.
+
+**Encoding:** label ordinals are zero-based and sorted; one-hot blocks follow
+source order. A label sentinel must be a safe integer outside the known ordinal
+range. `other` appends an ordinal or field after known categories. Without
+imputation, missing input follows the encoder's unknown policy.
+
+Generated IDs, names and category metadata are deterministic. Collisions fail
+with error `102`.
+
+**Kind inference:** configure both numeric and categorical branches, set
+`kind.op` to `infer`, `kind.rule` to `finite-integer-cardinality-v1`, and
+`kind.maxCategories` to at least `2`. Missing/NaN values are ignored.
+
+| Observed reference values | Selected branch |
+|---------------------------|-----------------|
+| `2..maxCategories` distinct finite integers | Categorical |
+| Zero or one distinct value, any noninteger, or more than `maxCategories` | Numeric |
+
+</details>
+
+<details markdown="1">
+<summary>Fixed category dictionaries</summary>
+
+Set `encode.categories` to a nonempty, sorted, unique array of finite typed tags.
+For example, `{ "t": "f64", "v": "4000000000000000" }` represents `2`.
+Tags must match the input dtype; represent negative zero as positive zero.
+
+- Unobserved categories stay in the dictionary.
+- Unknown training values follow the encoder policy and never vote for mode.
+- An all-missing zero fallback must exist in the dictionary.
+- Fixed encoding without imputation can finalize without training rows.
+
+String categories, categorical constant imputation and fixed dictionaries
+combined with kind inference are not supported.
+
+</details>
+
+<details markdown="1">
+<summary>Limits, errors and cancellation</summary>
+
+Exact median obeys allocation and resident-state limits. Inference, category
+discovery and output expansion also enforce category/output-width limits.
+Resource-limit failures use error `104`.
+
+Host-policy and spill settings are reserved. A non-null host policy or nonempty
+spill path fails with unsupported-runtime error `113`.
+
+Runtime options reject unknown fields:
+
+| Runtime | Options |
+|---------|---------|
+| Native Node | `limits`, `cancelFlag`, `hostPolicy`, `spillDir` |
+| Standalone WASM | `limits`, `cancelToken`, `hostPolicy`, `spillDir` |
+
+Cancellation spellings are not interchangeable. WASM tokens created with
+`createTransformCancelToken()` expose a read-only `requested` boolean for host
+conversion loops; reading a closed token fails.
+
 `tranfi/wasm` exposes the same classes on the initialized module. The Worker adapter adds `analyzeTransform()` and `applyTransform()`. With `SharedArrayBuffer`, an `AbortSignal` interrupts a synchronous C call through an atomic poll cell. Without it, cancellation terminates the whole worker and reclaims its WASM heap. Pass a worker URL directly so the client can recreate it, or supply an owned worker plus `workerFactory`:
 
+<!-- readme-test: browser -->
 ```js
 const workerUrl = new URL('./tranfi-worker.js', import.meta.url)
 const makeWorker = () => new Worker(workerUrl, { type: 'module' })
@@ -258,29 +339,30 @@ const client = createWorkerClient(makeWorker(), {
 
 All prepared-transform failures use `TranfiTransformError`; its numeric `code` is stable across native Node and WASM. Native Node accepts a SharedArrayBuffer-backed `Int32Array` as `cancelFlag`: cell 0 is the cancellation request, and an optional cell 1 is incremented modulo 2^32 at every native poll so another realm can observe operation progress without a timer.
 
-## Codecs
-
-Codecs convert between raw bytes and columnar batches. Every pipeline starts with a decoder and ends with an encoder.
+</details>
 
-| Method | Description |
-|--------|-------------|
-| `codec.csv({ delimiter, header, batchSize, repair, mode, strict, maxErrorBytes, maxRecordBytes, maxColumns, nulls, quotedNulls, skip, nMax, maxRows, comment, trimWs, skipEmptyRows, audit, auditLimit, auditIncludeRow, auditColumns, auditRedact, auditHashColumns, auditMaxBytes, auditMaxCellBytes })` | CSV decoder. `mode: 'strict'` fails on field-count mismatches; `repair: true` / `mode: 'repair'` emits repair diagnostics; `maxRecordBytes` bounds buffered records; `nulls: ['NA']` adds null sentinels; `skip: 2`, `nMax: 100`, `comment: '#'`, `trimWs: false`, and `skipEmptyRows: true` control row-local parsing; repair audit/raw diagnostics support privacy controls with pseudo-column `raw` |
-| `codec.csvEncode({ delimiter })` | CSV encoder |
-| `codec.jsonl({ batchSize, onError, maxErrorBytes, maxRecordBytes })` | JSON Lines decoder. `onError` is `skip`, `fail`, `warn`, or `quarantine` |
-| `codec.jsonlEncode()` | JSON Lines encoder |
-| `codec.text({ batchSize, maxErrorBytes, maxRecordBytes })` | Line-oriented text decoder (single `_line` column) |
-| `codec.textEncode()` | Text encoder |
-| `codec.tableEncode({ maxWidth, maxRows })` | Pretty-print Markdown table |
+## Codecs
 
-CSV treats unquoted empty fields as null by default. Pass `nulls: ['NA', 'NULL']` to add sentinel strings; pass `quotedNulls: false` when quoted sentinels like `"NA"` or `""` should remain strings. Default field-count handling is permissive for compatibility. Use `mode: 'strict'` or `strict: true` to fail on row/header width mismatches. Use `repair: true` or `mode: 'repair'` to pad/truncate and collect JSONL diagnostics in `result.errors`; `maxErrorBytes` bounds raw previews, and `auditIncludeRow`, `auditColumns`, `auditRedact`, `auditHashColumns`, `auditMaxBytes`, and `auditMaxCellBytes` govern repair audit/raw payloads with the raw preview exposed as pseudo-column `raw`. `maxRecordBytes` defaults to `67108864` bytes, caps the current record buffer before a newline is seen, and rejects with a `csv_record_too_large` diagnostic when exceeded; pass `0` to disable the guard. `maxColumns` defaults to `8192`; records above the cap reject with a bounded `csv_too_many_columns` diagnostic instead of silently dropping columns. Decoder size options are checked before execution: `batchSize` must be `1..65536`, `maxErrorBytes` must be `0..67108864`, `maxRecordBytes` must be `0..1073741824`, and `maxColumns` must be `1..65536`. `skip: 2` discards preamble records before header/schema discovery; comments are applied after skipped rows. `nMax: 100` / `maxRows: 100` keeps at most that many decoded data rows after skip/comment/header handling and uses only one counter; `nMax: 0` preserves a header-only schema batch. `comment: '#'` removes text after an unquoted marker and skips comment-only rows; quoted markers are preserved. Unquoted spaces/tabs are trimmed by default; pass `trimWs: false` to preserve them. Blank physical rows after the header are preserved as all-null rows by default; pass `skipEmptyRows: true` to drop them.
+Start with a decoder and finish with an encoder. Input and output formats can differ.
 
-Malformed JSONL records are skipped by default. Set `onError: 'warn'` or `onError: 'quarantine'` to keep valid rows and collect JSONL diagnostics in `result.errors`; set `onError: 'fail'` to reject on the first malformed line. `maxErrorBytes` bounds the raw preview stored in diagnostics, and `maxRecordBytes` applies the same current-line guard as CSV/text.
+| Format | Decoder | Encoder |
+|--------|---------|---------|
+| CSV | `codec.csv(options)` | `codec.csvEncode({ delimiter })` |
+| JSON Lines | `codec.jsonl({ batchSize, onError, maxErrorBytes, maxRecordBytes })` | `codec.jsonlEncode()` |
+| Text (one `_line` column) | `codec.text({ batchSize, maxErrorBytes, maxRecordBytes })` | `codec.textEncode()` |
+| Markdown table | — | `codec.tableEncode({ maxWidth, maxRows })` |
 
-Each JSONL record must be one complete UTF-8 JSON object. Invalid number syntax, trailing content, embedded NULs, and numbers outside the finite float64 range follow the same malformed-record policy. The encoder rejects nonfinite values. Schema order comes from the first valid record; repeated keys use their first value.
+**CSV defaults to permissive field-count handling.** Choose `mode: 'strict'`
+(or `strict: true`) to reject width mismatches. Choose `repair: true`
+(or `mode: 'repair'`) to pad/truncate rows and collect diagnostics in `result.errors`.
 
+**Malformed JSONL is skipped by default.** Set `onError` to `fail` to stop,
+`warn` to keep valid rows with diagnostics, or `quarantine` to route bad records
+to error diagnostics. Read those diagnostics from `result.errors`.
 
 Cross-codec pipelines work naturally:
 
+<!-- readme-test: js-api -->
 ```js
 // CSV in, JSONL out
 pipeline([codec.csv(), ops.head(5), codec.jsonlEncode()])
@@ -289,6 +371,57 @@ pipeline([codec.csv(), ops.head(5), codec.jsonlEncode()])
 pipeline([codec.jsonl(), ops.sort(['name']), codec.csvEncode()])
 ```
 
+<details markdown="1">
+<summary>CSV options for nulls, row limits and whitespace</summary>
+
+| Need | Option | Behavior |
+|------|--------|----------|
+| Field separator / header | `delimiter`, `header` | Configure CSV decoding; encoder accepts `delimiter` |
+| Add null markers | `nulls: ['NA', 'NULL']` | Adds to default unquoted-empty-field null handling |
+| Preserve quoted markers | `quotedNulls: false` | Keeps quoted `"NA"` and `""` as strings |
+| Skip a preamble | `skip: 2` | Before schema/header discovery; comments are handled afterward |
+| Limit rows | `nMax: 100` or `maxRows: 100` | One shared counter after skip/comment/header handling |
+| Keep only the schema | `nMax: 0` | Preserves a header-only batch |
+| Comments | `comment: '#'` | Removes text after unquoted markers and skips comment-only rows; quoted markers remain |
+| Preserve spaces/tabs | `trimWs: false` | Disables default trimming of unquoted values |
+| Drop blank rows | `skipEmptyRows: true` | Otherwise blank physical rows after the header become all-null rows |
+| Read CSV shards | `skipRepeatedHeader: true` | Skips matching headers only at explicit file boundaries |
+
+</details>
+
+<details markdown="1">
+<summary>Decoder limits and repair diagnostics</summary>
+
+All decoder size options are checked integers:
+
+| Option | Range | Behavior |
+|--------|-------|----------|
+| `batchSize` | `1..65536` | Rows per batch |
+| `maxErrorBytes` | `0..67108864` | Bounds raw diagnostic previews |
+| `maxRecordBytes` | `0..1073741824` | Default `67108864` bytes (64 MiB); `0` disables the record guard |
+| `maxColumns` (CSV) | `1..65536` | Default `8192`; overflow fails with `csv_too_many_columns` |
+
+Record limits apply before a newline is seen, including CSV, JSONL and text.
+CSV record overflow fails with `csv_record_too_large`.
+
+Repair audit/raw payloads accept `auditIncludeRow, auditColumns, auditRedact, auditHashColumns, auditMaxBytes, auditMaxCellBytes`. The raw preview is exposed to
+those controls as pseudo-column `raw`. CSV repair also accepts
+`audit, auditLimit`.
+
+</details>
+
+<details markdown="1">
+<summary>JSONL record and schema rules</summary>
+
+Each record must be one complete UTF-8 JSON object. Invalid number syntax,
+trailing content, embedded NULs and numbers outside finite float64 range follow
+the configured malformed-record policy. The encoder rejects nonfinite values.
+
+The first valid record determines schema order. Repeated keys use their first
+value. `maxErrorBytes` bounds diagnostic previews, and `maxRecordBytes` caps the current line.
+
+</details>
+
 ## Operators
 
 ### Row filtering
@@ -351,6 +484,7 @@ Example: `ops.across(['starts_with(score_)'], { fn: 'round' })` replaces selecte
 | `ops.frequency(columns?, { maxValues, maxStateBytes, overflow, other, audit, auditLimit, auditIncludeRow, auditColumns, auditRedact, auditHashColumns, auditMaxBytes, auditMaxCellBytes }?)` | Value counts; `overflow: "other"` can emit bounded category-overflow audit records with privacy controls |
 | `ops.groupAgg(groupBy, aggs)` | Group by + aggregate. `count` on a column counts non-null values; `column: '*'` counts rows. |
 
+<!-- readme-test: js-api -->
 ```js
 // Group aggregation
 ops.groupAgg(['city'], [
@@ -409,6 +543,7 @@ ops.groupAgg(['city'], [
 
 Used in `filter`, `derive`, `validate`, and `assert`. Reference columns with `col('name')`.
 
+<!-- readme-test: js-api -->
 ```js
 ops.filter(expr("col('age') > 25 and contains(col('name'), 'A')"))
 ops.derive({
@@ -432,6 +567,7 @@ ops.derive({
 
 Aliases: `substr`=`slice`, `length`=`len`, `lpad`=`pad_left`, `rpad`=`pad_right`, `min`=`least`, `max`=`greatest`. Date/time functions are row-local and accept date/timestamp values plus parseable date/timestamp strings; `weekday()` returns `0=Sunday` through `6=Saturday`.
 
+<!-- readme-test: js-api -->
 ```js
 ops.derive({
   year: expr("year(col('date'))"),
@@ -443,9 +579,10 @@ ops.derive({
 
 Built-in named pipelines for common tasks. Use by name:
 
+<!-- readme-test: js-api -->
 ```js
 const result = await pipeline('preview').run({ inputFile: 'data.csv' })
-const result = await pipeline('freq').run({ inputFile: 'data.csv' })
+const frequencies = await pipeline('freq').run({ inputFile: 'data.csv' })
 ```
 
 | Recipe | Pipeline | Description |
@@ -488,12 +625,14 @@ for (const r of await recipes()) {
 
 Native Node/WASM execution rejects full-input blocking steps such as `sort`, `pivot`, `normalize`, `acf`, and table encoding unless you opt in for known-small data:
 
+<!-- readme-test: js-api -->
 ```js
 await pipeline('csv | sort age | csv').run({ inputFile: 'small.csv', allowBlocking: true })
 ```
 
 For capped key-state operators, pass `memory` to validate the conservative native state estimate before execution:
 
+<!-- readme-test: js-api -->
 ```js
 await pipeline('csv | unique city max_keys=10000 | csv').run({ inputFile: 'data.csv', memory: '64MB' })
 ```
@@ -510,6 +649,7 @@ Run pipelines on DuckDB instead of the native C streaming core. The DSL is trans
 npm install duckdb
 ```
 
+<!-- readme-test: js-data -->
 ```js
 import { pipeline, compileToSql } from 'tranfi'
 
@@ -526,6 +666,7 @@ const result2 = await pipeline('csv | head 10 | csv', { engine: 'duckdb' })
 
 Generate SQL directly from DSL strings:
 
+<!-- readme-test: js-sql -->
 ```js
 const sql = await compileToSql(
   'csv | filter "col(age) > 25" | sort -age | head 10 | csv',
@@ -546,6 +687,7 @@ their own compatibility tests and SQL-generation rules.
 
 In the browser, use `@duckdb/duckdb-wasm` with the tranfi WASM module:
 
+<!-- readme-test: browser -->
 ```js
 import createTranfi from 'tranfi/wasm'
 import * as duckdb from '@duckdb/duckdb-wasm'
@@ -571,7 +713,7 @@ console.log(result.rows)        // Array of row objects
 ### DSL compilation
 
 ```js
-import { compileDsl, saveRecipe, loadRecipe } from 'tranfi'
+import { compileDsl, saveRecipe, loadRecipe, codec, ops } from 'tranfi'
 
 // Compile DSL to JSON plan
 const json = await compileDsl('csv | filter "col(age) > 25" | sort -age | csv')
@@ -591,6 +733,7 @@ Every pipeline produces four output channels:
 - **stats** -- newline-delimited execution statistics: run summary plus per-step counters, state estimates, and warnings
 - **samples** -- reserved for sampling operators
 
+<!-- readme-test: js-pipeline -->
 ```js
 const result = await p.run({ inputFile: 'data.csv' })
 console.log(result.statsText)   // newline-delimited JSON: summary plus step_stats/state_bytes_estimate/warnings
@@ -599,6 +742,8 @@ console.log(result.statsText)   // newline-delimited JSON: summary plus step_sta
 ### Pipeline from JSON
 
 ```js
+import { loadRecipe } from 'tranfi'
+
 const p = await loadRecipe({
   steps: [
     { op: 'codec.csv.decode', args: {} },
@@ -613,7 +758,7 @@ const p = await loadRecipe({
 The package automatically selects the best backend:
 
 1. **N-API** (Node.js) -- native C addon, fastest, used when available
-2. **WASM** (browsers/fallback) -- same C core compiled to WebAssembly, ~500 KB single-file
+2. **WASM** (browsers/fallback) -- same C core compiled to WebAssembly, single-file module
 3. **DuckDB** (opt-in) -- SQL execution via `{ engine: 'duckdb' }`, requires `npm install duckdb`
 
 ```js
@@ -624,4 +769,19 @@ const tf = await createTranfi()
 
 ## Architecture
 
-The Node.js package wraps the same C11 core used by the CLI, Python, and WASM targets. Data flows through columnar batches with typed columns (`bool`, `int64`, `float64`, `string`, `date`, `timestamp`) and per-cell null bitmaps. `run({ inputFile })` streams files with `createReadStream()` and drains native/WASM main output after each push and each incremental finish boundary; by default it still collects the final output for convenience. `.gz` input files are decompressed through a `zlib.createGunzip()` source transform; use `compression: "none"` to force raw bytes or `compression: "gzip"` to force gzip for `inputFile`/`inputStream`. Use `inputStream`, `toReadable()`, `writeTo()`, `onOutput` with `collectOutput: false`, or `iterChunks()` for large inputs/outputs and backpressure-aware sinks. Native execution is strict by default: row-local and bounded-state operators stream, blocking operators require `allowBlocking: true` or supported `spillDir`, capped key-state plans can be checked with `memory: "64MB"`, and core plan file reads require explicit host-policy options.
+Node, Python, CLI and WASM use the same C11 engine. It processes columnar batches
+with typed columns (`bool`, `int64`, `float64`, `string`, `date`, `timestamp`)
+and per-cell null bitmaps.
+
+| Need | Node API behavior |
+|------|-------------------|
+| Stream input | `run({ inputFile })` uses `createReadStream()` and drains output after each push and incremental finish boundary |
+| Avoid collecting all output | Use `inputStream`, `toReadable()`, `writeTo()`, `iterChunks()`, or `onOutput` with `collectOutput: false` |
+| Read gzip | `.gz` files use `zlib.createGunzip()`; override with `compression: "none"` or `"gzip"` for `inputFile`/`inputStream` |
+| Permit a blocking operation | Set `allowBlocking: true` or use a supported `spillDir` path |
+| Cap retained key state | Supply caps and check the plan with `memory: "64MB"` |
+| Allow plan file reads | Supply explicit host-policy options |
+
+Input streaming still collects the final output by default. Choose a streaming
+sink for large outputs; row-local and bounded-state operators stream under the
+default strict native memory policy.

package/csrc/pipeline.c

@@ -16,7 +16,7 @@
 #include <unistd.h>
 #endif
 
-#define TRANFI_VERSION "0.2.0"
+#define TRANFI_VERSION "0.2.1"
 
 enum {
     TF_FINISH_PHASE_DECODER = 0,

package/csrc/size_utils.c

@@ -104,6 +104,9 @@ char *tf_string_append_suffix_checked(const char *prefix, const char *suffix) {
     return copy;
 }
 
+/* Platforms with qsort_r do not compile the portable introsort fallback. */
+#if !defined(__GLIBC__) && !defined(__APPLE__) && !defined(__FreeBSD__) && \
+    !defined(__OpenBSD__) && !defined(__NetBSD__)
 static void tf_index_swap(size_t *a, size_t *b) {
     size_t tmp = *a;
     *a = *b;
@@ -225,6 +228,7 @@ static void tf_sort_indices_intro_loop(size_t *indices, size_t lo, size_t hi, si
     }
     tf_sort_indices_insertion_range(indices, lo, hi, compare, ctx);
 }
+#endif
 
 typedef struct {
     tf_index_compare_fn compare;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tranfi",
-  "version": "0.2.0",
+  "version": "0.2.1",
   "description": "Streaming ETL language + runtime",
   "type": "commonjs",
   "main": "src/index.js",
@@ -39,7 +39,7 @@
     "prepack": "node scripts/prepack.js",
     "install": "node scripts/install-native.js",
     "build:native": "node scripts/sync-csrc.js && node-gyp rebuild",
-    "test": "node --test ../test/test_install_native.js && node ../test/test_node.js && node ../test/test_transform_node.js && node ../test/test_transform_wasm_node.js && node ../test/test_transform_worker_node.js"
+    "test": "node --test ../test/test_install_native.js ../test/test_cli_node.js && node ../test/test_node.js && node ../test/test_transform_node.js && node ../test/test_transform_wasm_node.js && node ../test/test_transform_worker_node.js"
   },
   "peerDependencies": {
     "duckdb": ">=1.0.0",

package/src/cli.js

@@ -9,8 +9,8 @@
  *   tranfi profile < data.csv
  */
 
-const { pipeline, compileDsl, recipes } = require('./index.js')
-const { readFileSync, writeFileSync, existsSync } = require('fs')
+const { pipeline, compileDsl, compileToSql, recipes } = require('./index.js')
+const { readFileSync, writeFileSync, existsSync, openSync, closeSync } = require('fs')
 const { resolve, dirname } = require('path')
 const nativeBinding = require('./native.js')
 
@@ -34,6 +34,13 @@ Options:
   -i FILE   Read input from file instead of stdin
   -o FILE   Write output to file instead of stdout
   -j        Compile only, output JSON plan
+  --target native|json|sql  Execute or compile a pipeline
+  --dialect NAME           SQL dialect (duckdb)
+  --allow-blocking         Permit full-input operations on known-small data
+  --fail-on-blocking       Reject blocking operations (default)
+  --memory SIZE            Native memory policy, e.g. max:64MB
+  --spill-dir DIR          Existing private spill directory
+  --stats-json FILE        Write stats NDJSON; - means stderr
   -q        Quiet mode (suppress stats)
   -v        Show version
   -R        List built-in recipes
@@ -102,10 +109,21 @@ async function main() {
   let outputFile = null
   let jsonMode = false
   let quiet = false
+  let allowBlocking = false
+  let failOnBlocking = false
+  let memory, spillDir, statsFile
+  let target = 'native'
+  let dialect = 'duckdb'
+  function valueAt(i, inline, flag) {
+    const value = inline === undefined ? argv[i + 1] : inline
+    if (!value || value.startsWith('--')) throw new Error(`${flag} requires a value`)
+    return value
+  }
 
   // Parse args
   for (let i = 0; i < argv.length; i++) {
-    const arg = argv[i]
+    const [arg, ...assigned] = argv[i].split('=')
+    const inline = assigned.length ? assigned.join('=') : undefined
     if (arg === '-h' || arg === '--help') { usage(); process.exit(0) }
     else if (arg === '-v' || arg === '--version') {
       const v = nativeBinding ? nativeBinding.version() : 'unknown'
@@ -121,18 +139,35 @@ async function main() {
       }
       process.exit(0)
     }
-    else if (arg === '-j') jsonMode = true
-    else if (arg === '-q') quiet = true
-    else if (arg === '-f') { pipelineFile = argv[++i] }
-    else if (arg === '-i') { inputFile = argv[++i] }
-    else if (arg === '-o') { outputFile = argv[++i] }
-    else if (!arg.startsWith('-')) { pipelineText = arg }
+    else if (arg === '-j' || arg === '--json') jsonMode = true
+    else if (arg === '-q' || arg === '--quiet') quiet = true
+    else if (arg === '--allow-blocking') allowBlocking = true
+    else if (arg === '--fail-on-blocking') failOnBlocking = true
+    else if (['-f', '--file', '-i', '--input', '-o', '--output', '--memory', '--spill-dir', '--stats-json', '--target', '--dialect'].includes(arg)) {
+      const value = valueAt(i, inline, arg)
+      if (inline === undefined) i++
+      if (arg === '-f' || arg === '--file') pipelineFile = value
+      else if (arg === '-i' || arg === '--input') inputFile = value
+      else if (arg === '-o' || arg === '--output') outputFile = value
+      else if (arg === '--memory') memory = value
+      else if (arg === '--spill-dir') spillDir = value
+      else if (arg === '--stats-json') statsFile = value
+      else if (arg === '--target') target = value
+      else if (arg === '--dialect') dialect = value
+    }
+    else if (!argv[i].startsWith('-')) {
+      if (pipelineText !== null) throw new Error('only one pipeline argument is allowed')
+      pipelineText = argv[i]
+    }
     else {
       process.stderr.write(`error: unknown option '${arg}'\n`)
       process.exit(1)
     }
   }
 
+  if (allowBlocking && failOnBlocking) throw new Error('--allow-blocking conflicts with --fail-on-blocking')
+  if (!['native', 'json', 'sql'].includes(target)) throw new Error('unknown --target; expected native, json or sql')
+
   // Get pipeline text
   if (pipelineFile) {
     pipelineText = readFileSync(pipelineFile, 'utf-8')
@@ -143,26 +178,44 @@ async function main() {
     process.exit(1)
   }
 
-  // JSON mode
-  if (jsonMode) {
+  const named = (await recipes()).find(recipe => recipe.name === pipelineText.trim())
+  if (named) pipelineText = named.dsl
+  if (target === 'sql') {
+    process.stdout.write(await compileToSql(pipelineText, { dialect }) + '\n')
+    return
+  }
+
+  if (jsonMode || target === 'json') {
     const json = await compileDsl(pipelineText)
     process.stdout.write(json + '\n')
     process.exit(0)
   }
 
-  // Create and run pipeline
   const p = pipeline(pipelineText)
-  const input = inputFile ? undefined : await readStdin()
-  const result = await p.run({
+  const options = {
     inputFile: inputFile || undefined,
-    input
-  })
-
-  // Write output
+    inputStream: inputFile ? undefined : process.stdin,
+    allowBlocking, memory, spillDir, allowFs: true, allowRulesFile: true
+  }
+  let result
   if (outputFile) {
-    writeFileSync(outputFile, result.output)
+    let fd
+    try {
+      result = await p.run({
+        ...options,
+        collectOutput: false,
+        onOutput(chunk) {
+          // Open only after plan validation; a rejected plan must not truncate output.
+          if (fd === undefined) fd = openSync(outputFile, 'w')
+          writeFileSync(fd, chunk)
+        }
+      })
+      if (fd === undefined) fd = openSync(outputFile, 'w')
+    } finally {
+      if (fd !== undefined) closeSync(fd)
+    }
   } else {
-    process.stdout.write(result.output)
+    result = await p.writeTo(process.stdout, { ...options, end: false })
   }
 
   // Errors to stderr
@@ -170,18 +223,8 @@ async function main() {
     process.stderr.write(result.errors)
   }
 
-  // Stats to stderr (unless quiet)
-  if (!quiet && result.stats.length > 0) {
-    process.stderr.write(result.stats)
-  }
-}
-
-function readStdin() {
-  return new Promise((resolve) => {
-    const chunks = []
-    process.stdin.on('data', chunk => chunks.push(chunk))
-    process.stdin.on('end', () => resolve(Buffer.concat(chunks)))
-  })
+  if (statsFile && statsFile !== '-') writeFileSync(statsFile, result.stats)
+  else if ((statsFile || !quiet) && result.stats.length > 0) process.stderr.write(result.stats)
 }
 
 main().catch(err => {