@eigenpal/cli 0.4.6 → 0.4.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -227,6 +227,70 @@ eigenpal workflow experiment status <workflow-id> <batch-id>
227
227
  eigenpal workflow experiment results <workflow-id> <batch-id> --format csv --out r.csv
228
228
  ```
229
229
 
230
+ ### Long documents — split → chunk → extract
231
+
232
+ The canonical chain when a document is too long for one `ai.extract` call (loan
233
+ contracts, SaaS agreements, RFPs). Each step preserves source `pageIndex` so
234
+ the final extracted fields trace back to the originating page.
235
+
236
+ ```yaml
237
+ - name: parse
238
+ type: ai.parse
239
+ with: { input: "{{ input.document }}" }
240
+
241
+ # LLM-driven section split — each split carries page_range + confidence
242
+ - name: sections
243
+ type: ai.split
244
+ with:
245
+ input: "{{ steps.parse.output }}"
246
+ sections:
247
+ - { name: header, description: "Title page + parties block" }
248
+ - { name: priloha2, description: "Príloha 2 odkladacie podmienky", required: true }
249
+
250
+ # Optional: chunk a long section before extraction (regex-anchored, keeps page refs)
251
+ - name: chunks
252
+ type: transform.text-chunker
253
+ with:
254
+ input: "{{ steps.sections.output.splits[1] }}" # priloha2 split
255
+ maxChars: 8000
256
+ overlap: 500
257
+ splitOn: ['(?:^|\\n)\\s*\\d+\\.\\d+\\s+', '\\n\\n+', '\\n']
258
+
259
+ # Deterministic field extraction — counterpart to ai.extract; matches carry _evidence.pageIndex
260
+ - name: header_fields
261
+ type: transform.regex-extract
262
+ with:
263
+ input: "{{ steps.parse.output }}"
264
+ fields:
265
+ contractNumber:
266
+ pattern: 'č\\.\\s*(\\d{1,4}/\\w+/\\d{4})'
267
+ normalize: strip-spaces
268
+ signatureDate:
269
+ pattern: 'uzavretá\\s+dňa\\s+(\\d{1,2})\\.(\\d{1,2})\\.(\\d{4})'
270
+ format: iso-date
271
+ currency:
272
+ pattern: '\\b(EUR|USD|CZK)\\b'
273
+ default: EUR
274
+ flags: i
275
+ ```
276
+
277
+ Discovery from the CLI:
278
+
279
+ ```bash
280
+ eigenpal workflow step-type get ai.split | jq '.configSchema.properties'
281
+ eigenpal workflow step-type get transform.text-chunker | jq '.configSchema.properties'
282
+ eigenpal workflow step-type get transform.regex-extract | jq '.configSchema.properties'
283
+ ```
284
+
285
+ Pick the right operator for the job:
286
+
287
+ | Goal | Operator |
288
+ | ---------------------------------------- | ------------------------- |
289
+ | Extract structured data with an LLM | `ai.extract` |
290
+ | Pull fields by regex (no LLM) | `transform.regex-extract` |
291
+ | Split a doc into named sections (LLM) | `ai.split` |
292
+ | Cut text by regex boundary + overlap | `transform.text-chunker` |
293
+
230
294
  ### Debug a failing workflow
231
295
 
232
296
  ```bash
@@ -35,6 +35,7 @@ Manage workflows: push, pull, run, evaluate.
35
35
  - [`eigenpal workflow experiment list [options] <workflow-id>`](#eigenpal-workflow-experiment-list-options-workflow-id)
36
36
  - [`eigenpal workflow experiment run [options] <workflow-id>`](#eigenpal-workflow-experiment-run-options-workflow-id)
37
37
  - [`eigenpal workflow experiment status [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-status-options-workflow-id-batchid)
38
+ - [`eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`](#eigenpal-workflow-experiment-cancel-options-workflow-id-batchid)
38
39
  - [`eigenpal workflow experiment results [options] <workflow-id> [batchId]`](#eigenpal-workflow-experiment-results-options-workflow-id-batchid)
39
40
  - [`eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>`](#eigenpal-workflow-experiment-compare-options-batchida-batchidb)
40
41
  - [`eigenpal workflow execution run [options] <workflow-id> [examples...]`](#eigenpal-workflow-execution-run-options-workflow-id-examples)
@@ -76,6 +77,7 @@ workflow
76
77
  │ ├── list <workflow-id>
77
78
  │ ├── run <workflow-id>
78
79
  │ ├── status <workflow-id> <batchId>
80
+ │ ├── cancel <workflow-id> <batchId>
79
81
  │ ├── results <workflow-id> [batchId]
80
82
  │ └── compare <batchIdA> <batchIdB>
81
83
  ├── execution
@@ -140,6 +142,7 @@ workflow
140
142
  | `eigenpal workflow experiment list [options] <workflow-id>` | List executions for the workflow, newest first. |
141
143
  | `eigenpal workflow experiment run [options] <workflow-id>` | Start a batch eval against the workflow's dataset. |
142
144
  | `eigenpal workflow experiment status [options] <workflow-id> <batchId>` | Aggregate progress for a batch by `batchId`. |
145
+ | `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>` | Cancel every execution in a batch. Idempotent. |
143
146
  | `eigenpal workflow experiment results [options] <workflow-id> [batchId]` | Download eval results in CSV or JSON. |
144
147
  | `eigenpal workflow experiment compare [options] <batchIdA> <batchIdB>` | Diff eval scores between two experiment batches. |
145
148
 
@@ -177,9 +180,9 @@ workflow
177
180
 
178
181
  ### Step
179
182
 
180
- | Command | Description |
181
- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
182
- | `eigenpal workflow step exec [options] <type>` | Run one step locally with sample inputs and print its output as JSON. <type> is any value from `workflow step-type list`; only types with a local runner actually execute (today: transform.script, ai.extract). |
183
+ | Command | Description |
184
+ | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
185
+ | `eigenpal workflow step exec [options] <type>` | DISABLED — local mimic runners removed pending server-side redesign (EIG-104). Use `workflow execution run` or `workflow experiment run` instead. |
183
186
 
184
187
  ## Details
185
188
 
@@ -505,13 +508,33 @@ Aggregate progress for a batch by `batchId`.
505
508
 
506
509
  ### Options
507
510
 
508
- | Flag | Required | Default | Description |
509
- | ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------- |
510
- | `--watch` | no | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit |
511
- | `--interval <seconds>` | no | `5` | Poll interval in seconds when --watch is set (default 5) |
512
- | `--max-wait <seconds>` | no | `1800` | Hard ceiling for --watch in seconds (default 1800 = 30 min) |
513
- | `--base-url <url>` | no | | Server base URL |
514
- | `--json` | no | | Output the raw server response as JSON |
511
+ | Flag | Required | Default | Description |
512
+ | ---------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
513
+ | `--watch` | no | `false` | Poll until every execution reaches a terminal state (completed/failed/cancelled/rejected), then exit |
514
+ | `--interval <seconds>` | no | `5` | Poll interval in seconds when --watch is set (default 5) |
515
+ | `--max-wait <seconds>` | no | `1800` | Hard ceiling for --watch in seconds (default 1800 = 30 min) |
516
+ | `--include <kinds>` | no | `""` | Comma-separated extras to attach when --watch terminates: payload (full per-execution snapshot, can be hundreds of KB) |
517
+ | `--base-url <url>` | no | | Server base URL |
518
+ | `--json` | no | | Output the raw server response as JSON |
519
+
520
+ ### `eigenpal workflow experiment cancel [options] <workflow-id> <batchId>`
521
+
522
+ Cancel every execution in a batch. Idempotent.
523
+
524
+ ### Arguments
525
+
526
+ | Name | Required | Variadic | Description |
527
+ | ------------- | -------- | -------- | ----------- |
528
+ | `workflow-id` | yes | no | |
529
+ | `batchId` | yes | no | |
530
+
531
+ ### Options
532
+
533
+ | Flag | Required | Default | Description |
534
+ | ------------------ | -------- | ------- | --------------------------------------------------------------------- |
535
+ | `--yes` | no | | Required for non-TTY shells (CI, pipes). Acts immediately, no prompt. |
536
+ | `--base-url <url>` | no | | Server base URL |
537
+ | `--json` | no | | Output the raw server response as JSON |
515
538
 
516
539
  ### `eigenpal workflow experiment results [options] <workflow-id> [batchId]`
517
540
 
@@ -584,12 +607,12 @@ Fetch a single execution payload. Optionally narrow to one step.
584
607
 
585
608
  ### Options
586
609
 
587
- | Flag | Required | Default | Description |
588
- | ------------------- | -------- | ------------------------------- | ----------------------------------------------------- |
589
- | `--step <name>` | no | | Show only this step (or comma-separated list) |
590
- | `--include <kinds>` | no | `"input,output,error,duration"` | Comma-separated subset of input,output,error,duration |
591
- | `--base-url <url>` | no | | Server base URL |
592
- | `--json` | no | | Output the raw server response as JSON |
610
+ | Flag | Required | Default | Description |
611
+ | ------------------- | -------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
612
+ | `--step <name>` | no | | Show only this step (or comma-separated list) |
613
+ | `--include <kinds>` | no | `"input,output,error,duration"` | Comma-separated subset of input,output,error,duration. `input` projects the resolved templated config (= what the processor actually received). `inputRef` returns the minimal predecessor-id reference instead. |
614
+ | `--base-url <url>` | no | | Server base URL |
615
+ | `--json` | no | | Output the raw server response as JSON |
593
616
 
594
617
  ### `eigenpal workflow execution list [options] <workflow-id>`
595
618
 
@@ -763,7 +786,7 @@ Fetch the JSON Schema for one evaluator type. Pipe through `jq` to inspect speci
763
786
 
764
787
  ### `eigenpal workflow step exec [options] <type>`
765
788
 
766
- Run one step locally with sample inputs and print its output as JSON. <type> is any value from `workflow step-type list`; only types with a local runner actually execute (today: transform.script, ai.extract).
789
+ DISABLED — local mimic runners removed pending server-side redesign (EIG-104). Use `workflow execution run` or `workflow experiment run` instead.
767
790
 
768
791
  ### Arguments
769
792
 
@@ -773,11 +796,9 @@ Run one step locally with sample inputs and print its output as JSON. <type> is
773
796
 
774
797
  ### Options
775
798
 
776
- | Flag | Required | Default | Description |
777
- | ------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
778
- | `--config-json <json>` | no | | Full step config as a JSON literal. Mutually exclusive with --config-file. |
779
- | `--config-file <path\|->` | no | | Full step config from a JSON file (or `-` for stdin). Mutually exclusive with --config-json. |
780
- | `--inputs <k=v...>` | no | `[]` | Repeatable. Value can be `@path` (file contents, JSON-parsed when applicable) or a literal string. Merges into the config `inputs` map for transform.script; provides the `input` for ai.extract when set as `input=…`. |
781
- | `--output-schema <path>` | no | | Optional JSON Schema. Validates the step output; defaults to the step type’s built-in `outputSchema` from STEP_SCHEMAS when omitted. |
782
- | `--timeout-ms <n>` | no | | Override transform.script wall-clock cap (default 5000) |
783
- | `--memory-mb <n>` | no | | Override transform.script heap cap in MB (default 10) |
799
+ | Flag | Required | Default | Description |
800
+ | ------------------------ | -------- | ------- | -------------------------------------- |
801
+ | `--config-json <json>` | no | | (unused; kept for back-compat parsing) |
802
+ | `--config-file <path>` | no | | (unused) |
803
+ | `--inputs <pairs...>` | no | | (unused) |
804
+ | `--output-schema <path>` | no | | (unused) |
@@ -1,164 +1,62 @@
1
- # Run one step locally — `workflow step exec`
1
+ # `workflow step exec` — currently disabled (EIG-104)
2
2
 
3
- Tightest iteration loop in the CLI. One step, no server, no queue, no
4
- DAG. Milliseconds per run. Use this when you're tweaking a single
5
- step's logic; use `workflow execution run` when you need the whole
6
- workflow.
3
+ > **Status: disabled.** The local-runner implementation was removed in
4
+ > the ai.split branch (EIG-102). It mimicked production via a CLI-side
5
+ > QuickJS sandbox (`transform.script`) and direct OpenAI SDK calls
6
+ > (`ai.extract`), which diverged from the worker's behavior (no
7
+ > retries, no tracing, no credit metering, no tenant-default LLM, no
8
+ > strict-mode response_format) and assumed shell env state
9
+ > (`WORKER_LLM_*`) that real CLI users — who authenticate via
10
+ > `EIGENPAL_API_KEY` — don't have.
11
+ >
12
+ > Calling `eigenpal workflow step exec <type>` today exits 2 with a
13
+ > redirect message. The server-side replacement is tracked in EIG-104:
14
+ > a thin `POST /api/v1/workflows/step-exec` endpoint that runs the same
15
+ > code path executions take.
7
16
 
8
- ```bash
9
- eigenpal workflow step exec <type> [flags]
10
- ```
11
-
12
- `<type>` is any value from `eigenpal workflow step-type list`. Only
13
- some types have a local runner today (table below). The rest exit 2
14
- with a "not yet supported locally" message — they still need the
15
- server.
16
-
17
- ## What runs locally today
18
-
19
- | Step type | Runner | Notes |
20
- | --- | --- | --- |
21
- | `transform.script` | QuickJS sandbox in-process | Same sandbox as prod. TDZ traps and timeout / memory limits reproduce here. |
22
- | `ai.extract` | OpenAI SDK | Reads `WORKER_LLM_API_KEY` (or `OPENAI_API_KEY` for openai/openai-compatible providers) from env. Real network call. |
23
- | anything else | — | Exits 2 with "isn't yet supported by 'workflow step exec' locally". |
24
-
25
- If the runner you need isn't here yet, fall back to `workflow execution run`
26
- on the server (slower loop, but covers every step type).
17
+ ## What to use instead
27
18
 
28
- ## Inputs — `--inputs k=v`
29
-
30
- Repeatable. Three value forms:
19
+ Until EIG-104 lands, single-step iteration goes through the queue:
31
20
 
32
21
  ```bash
33
- --inputs limit=10 # inline literal (string; numeric strings stay strings)
34
- --inputs items=@items.json # file contents (JSON-parsed if valid; raw text otherwise)
35
- --inputs payload=@- # stdin (the `@` prefix is required; bare `-` is a literal)
36
- ```
37
-
38
- For `transform.script`, every `--inputs k=v` becomes a top-level
39
- binding in the sandbox (so `k` is referenced as `k` inside the code,
40
- mirroring runtime semantics — including TDZ traps for `const x = x || []`).
22
+ # Run the whole workflow once with a specific example.
23
+ eigenpal workflow execution run <workflow-id>
41
24
 
42
- For `ai.extract`, `--inputs input=…` is the document text.
25
+ # Or iterate on one example via the experiment surface.
26
+ eigenpal workflow experiment run <workflow-id> --example-id <name>
43
27
 
44
- ## Config — `--config-json` or `--config-file`
45
-
46
- Mutually exclusive. The shape is whatever the step type's `with:` block
47
- accepts (see `eigenpal workflow step-type get <type>`).
48
-
49
- ```bash
50
- --config-json '{"code":"return items.reduce((s,i)=>s+i.v,0)"}'
51
- --config-file ./step-config.json
52
- --config-file - # config from stdin (e.g. piped from `workflow pull`)
28
+ # Watch progress without dumping the full per-execution payload.
29
+ eigenpal workflow experiment status <workflow-id> <batchId> --watch
53
30
  ```
54
31
 
55
- ## Output
32
+ For `--example-id`, the human slug works too (`ex-01-koifer-97zf`),
33
+ not just the `evx_…` id — added as part of the same iteration.
56
34
 
57
- Goes to **stdout as JSON**. Status messages go to stderr — pipe stdout
58
- into `jq` cleanly:
35
+ To inspect what a step actually produced (input / output / resolved
36
+ config), use:
59
37
 
60
38
  ```bash
61
- eigenpal workflow step exec transform.script --config-file cfg.json --inputs items=@arr.json | jq '.'
39
+ eigenpal workflow execution get <executionId> --json --include input,output,config
62
40
  ```
63
41
 
64
- ## Output schema validation — `--output-schema`
42
+ The server returns `inputData` / `outputData` / `resolvedConfig` per
43
+ step on `?includeSteps=true`; the CLI projects them under shorter
44
+ names (`input` / `output` / `config`).
65
45
 
66
- Optional. Pass a JSON Schema file:
67
-
68
- ```bash
69
- --output-schema ./expected-schema.json
70
- ```
71
-
72
- When omitted, the CLI falls back to the step type's built-in
73
- `outputSchema` from `STEP_SCHEMAS` — so even without the flag, output
74
- shape gets checked.
75
-
76
- A violation **exits 2** with a machine-readable envelope on stderr:
77
-
78
- ```json
79
- { "code": "output_schema_violation", "issues": [
80
- { "field": "/total", "message": "must be number" }
81
- ]}
82
- ```
83
-
84
- `field` is the JSON Pointer to the offending value (Ajv's `instancePath`,
85
- remapped to `field` for stability) or `<root>` if the whole output is
86
- wrong. Agents can parse this directly — no text scraping.
87
-
88
- ## Resource limits (transform.script only)
89
-
90
- ```bash
91
- --timeout-ms 10000 # default 5000 (5 s wall clock)
92
- --memory-mb 25 # default 10 MB heap
93
- ```
94
-
95
- Use these when iterating on a memory-heavy reducer. They override the
96
- config's `timeout` / `memoryLimit` fields just for this run.
97
-
98
- ## Worked examples
99
-
100
- ### Tweak a JS reducer until tests pass
101
-
102
- Put the config in a file so the JS doesn't need escaping:
103
-
104
- ```bash
105
- cat > cfg.json <<'EOF'
106
- { "code": "return items.reduce((sum, i) => sum + i.value, 0)" }
107
- EOF
108
- echo '[{"value":1},{"value":2},{"value":3}]' > items.json
109
-
110
- eigenpal workflow step exec transform.script --config-file cfg.json --inputs items=@items.json
111
- # 6
112
- ```
113
-
114
- For a one-liner without a config file, `--config-json '{"code":"..."}'`
115
- works too — but mind the shell escaping.
116
-
117
- ### Run an extract once with sample text
118
-
119
- ```bash
120
- cat > extract.json <<'EOF'
121
- {
122
- "prompt": "Extract the invoice total as a number.",
123
- "schema": { "type": "object", "properties": { "total": { "type": "number" } } }
124
- }
125
- EOF
126
-
127
- eigenpal workflow step exec ai.extract \
128
- --config-file extract.json \
129
- --inputs input=@invoice.txt
130
- # {"total": 1234.56}
131
- ```
132
-
133
- ### Pipe config straight from a workflow YAML
134
-
135
- When you've already pushed a workflow and want to re-run one of its
136
- steps locally:
137
-
138
- ```bash
139
- eigenpal workflow pull wf_abc > workflow.yaml
140
- jq '.steps[] | select(.name == "extract") | .with' workflow.yaml \
141
- | eigenpal workflow step exec ai.extract --config-file - --inputs input=@invoice.txt
142
- ```
143
-
144
- ## Exit codes
46
+ ## Exit codes (current placeholder)
145
47
 
146
48
  | Code | When |
147
49
  | --- | --- |
148
- | 0 | Step ran; output validated. |
149
- | 1 | Runtime crash (sandbox blew up; `ai.extract` network failure; unexpected exception). |
150
- | 2 | Misuse OR `output_schema_violation` (envelope on stderr) OR unknown / unsupported step type. |
50
+ | 2 | Always — the command is disabled and redirects to `workflow execution run` / `experiment run`. |
151
51
 
152
- Exit 2 is always recoverable — fix your input / config / schema and
153
- re-run. Exit 1 means something is broken upstream (network, runtime,
154
- your code).
52
+ ## When this comes back
155
53
 
156
- ## Discovery
54
+ When EIG-104 lands, this page will document the server-routed shape:
157
55
 
158
- ```bash
159
- eigenpal workflow step-type list # all registered types
160
- eigenpal workflow step-type get <type> # config + output schema
161
- ```
56
+ - `--inputs k=v` for binding values
57
+ - `--config-json` / `--config-file` for the `with:` block
58
+ - `--output-schema` for caller-supplied output validation
59
+ - Streaming output JSON from the same engine that runs production
60
+ - Same `output_schema_violation` envelope on stderr, exit 2
162
61
 
163
- The catalog is generated from `STEP_SCHEMAS` in `@eigenpal/types` and
164
- is always in sync with what the deployment supports.
62
+ The CLI will be a thin client; the engine stays in one place.
@@ -160,6 +160,26 @@ Extract structured data from text using AI with a JSON schema
160
160
 
161
161
  > Extracted structured data matching the provided schema
162
162
 
163
+ #### `ai.split` — Split Document
164
+
165
+ Split a parsed document into named sections using an LLM. Consumes ai.parse output; emits per-section page ranges and text ready for downstream ai.extract via control.parallel_map.
166
+
167
+ **Config** (in `step.with`):
168
+
169
+ | Field | Type | Required | Default | Description |
170
+ | --- | --- | --- | --- | --- |
171
+ | `input` | string | yes | | Template expression resolving to ai.parse output, e.g. "{{ steps.parse.output }}" |
172
+ | `sections` | array<object> | yes | | Named sections to find in the document |
173
+ | `rules` | string | no | | Optional natural-language rules appended to the system prompt. E.g. "End-of-section markers like *Koniec prílohy 2* close the current section." |
174
+ | `provider` | string | no | | Provider ID from eigenpal.config.yaml (e.g. "openai-gpt5.4-mini"). Falls back to the tenant default LLM provider when omitted. |
175
+ | `windowTokenBudget` | integer | no | | Override the per-window token ceiling for this step. Defaults to env SPLIT_WINDOW_TOKEN_BUDGET or 20000. Smaller windows give sharper anchors on contract-style documents (less competing context for the LLM to mis-anchor on); bump to 50k–100k when sections routinely exceed per-window page count. |
176
+
177
+ **Output:** `object`
178
+
179
+ | Field | Type | Required | Default | Description |
180
+ | --- | --- | --- | --- | --- |
181
+ | `splits` | array<object> | yes | | Sections found in the document, in page order. Absent sections are omitted. |
182
+
163
183
  ### Transform steps — deterministic data transforms
164
184
 
165
185
  #### `transform.set` — Set Value
@@ -326,6 +346,46 @@ Execute JavaScript code in a secure sandbox. Input keys become TOP-LEVEL variabl
326
346
 
327
347
  > Value returned from script (validated against outputSchema if provided)
328
348
 
349
+ #### `transform.text-chunker` — Text Chunker
350
+
351
+ Split long text into chunks with regex-anchored boundaries, overlap, and header preservation. Accepts raw text or a parsed-document object; chunks carry source page indexes when pages are provided.
352
+
353
+ **Config** (in `step.with`):
354
+
355
+ | Field | Type | Required | Default | Description |
356
+ | --- | --- | --- | --- | --- |
357
+ | `input` | string \| record<string, unknown> | yes | | Either raw text or a parsed-document object `{ pages: [{ pageIndex, text }] }` (e.g. `{{ steps.parse.output }}`). Pages preserve per-chunk page provenance. |
358
+ | `maxChars` | integer | yes | | Target chunk size in characters. Hard ceiling per chunk is 1.5×. |
359
+ | `overlap` | integer | no | `0` | Characters duplicated at chunk boundaries (default 0). Must be < maxChars / 2. |
360
+ | `splitOn` | array<string> | no | | Ordered list of regexes; the first that matches near the chunk boundary wins. Falls back to char-cut when none match. Tip: list narrowest first (e.g. /\d+\.\d+\s+/ before /\n\n+/). |
361
+ | `maxChunks` | integer | no | `64` | Safety cap; later chunks are dropped and `summary.truncated` flips to true. |
362
+ | `preserveHeader` | integer | no | `0` | Prepend the first N characters of the input to every chunk (good for "always include the contract title"). |
363
+ | `minChunkChars` | integer | no | `0` | Trailing chunks shorter than this are merged into the previous chunk. |
364
+
365
+ **Output:** `object`
366
+
367
+ | Field | Type | Required | Default | Description |
368
+ | --- | --- | --- | --- | --- |
369
+ | `chunks` | array<object> | yes | | |
370
+ | `summary` | object | yes | | |
371
+
372
+ #### `transform.regex-extract` — Regex Extract
373
+
374
+ Pull named fields from text via regex patterns (deterministic counterpart to ai.extract). Accepts raw text or a parsed-document object; matches carry `_evidence.pageIndex` when pages are provided.
375
+
376
+ **Config** (in `step.with`):
377
+
378
+ | Field | Type | Required | Default | Description |
379
+ | --- | --- | --- | --- | --- |
380
+ | `input` | string \| record<string, unknown> | yes | | Either raw text or a parsed-document object `{ pages: [{ pageIndex, text }] }`. Pages enable per-match `_evidence.pageIndex`. |
381
+ | `fields` | record<string, object> | yes | | Named field → pattern mapping. |
382
+ | `flags` | string | no | | Default regex flags applied when a field omits its own `flags`. Subset of "gimsuy". |
383
+ | `searchWindow` | integer | no | | Only search the first N characters of input (perf). Omit for full search. |
384
+
385
+ **Output:** `record<string, unknown>`
386
+
387
+ > Field name → extracted value (or default), plus `_evidence: { [field]: { pageIndex, matchOffset, raw } }` and `_unmatched: string[]`.
388
+
329
389
  ### Action steps — external side effects
330
390
 
331
391
  #### `action.http` — HTTP Request
@@ -226,6 +226,104 @@ on the step execution.
226
226
  schema: { type: object, properties: { value: { type: string } } }
227
227
  ```
228
228
 
229
+ ### Split a long doc into named sections, then extract each
230
+
231
+ For documents that bundle multiple sub-documents (e.g. a contract pack with
232
+ several annexes), use `ai.split` to find section boundaries with an LLM,
233
+ then `control.parallel_map` to run section-specific `ai.extract`. The
234
+ splitter operates on parsed pages, so DOCX / RTF / scans all reduce to the
235
+ same per-page representation.
236
+
237
+ Description-writing tips (mirrors Reducto's guidance):
238
+
239
+ - Use the document's own terminology — a Slovak contract calls it `Príloha`,
240
+ not "annex". List the multilingual variants you've seen.
241
+ - Mention stable visual cues if any — a centered title-page header, a
242
+ signature line, the start of a specific table.
243
+ - Be specific. "The collateral schedule table at the start of Príloha 2"
244
+ beats "annex".
245
+
246
+ ```yaml
247
+ - name: parse
248
+ type: ai.parse
249
+ with:
250
+ input: '{{ input.document }}'
251
+
252
+ - name: split
253
+ type: ai.split
254
+ with:
255
+ input: '{{ steps.parse.output }}'
256
+ sections:
257
+ - name: intro
258
+ description: >-
259
+ The introductory cover/preamble of the loan pack — borrower details,
260
+ loan purpose, signature block. Always before any "Príloha".
261
+ - name: priloha_2
262
+ description: >-
263
+ "Príloha č. 2 / Anlage 2 / Załącznik nr 2" — the collateral
264
+ schedule. Starts with a section header on its own page; contains
265
+ a table of pledged assets.
266
+ required: true # warn if not found
267
+ endHints: # LLM-judged end cues — phrases the model treats semantically
268
+ - "PRÍLOHA Č. 3"
269
+ - "ANLAGE 3"
270
+ - name: priloha_3
271
+ description: >-
272
+ "Príloha č. 3 / Anlage 3" — the repayment schedule. Tabular
273
+ amortization plan with monthly rows.
274
+ rules: |
275
+ End-of-section markers like *Koniec prílohy 2* close the current section.
276
+ Repeated headers/footers are continuation, not new sections.
277
+
278
+ - name: extract-each-section
279
+ type: control.parallel_map
280
+ items: '{{ steps.split.output.splits }}'
281
+ as: section
282
+ steps:
283
+ - name: extract-section-fields
284
+ type: ai.extract
285
+ with:
286
+ input: '{{ section.text }}'
287
+ schema:
288
+ type: object
289
+ properties:
290
+ section_name: { type: string }
291
+ key_facts: { type: array, items: { type: string } }
292
+ ```
293
+
294
+ Each `splits[i]` carries `{ name, page_range, confidence, notes, evidence,
295
+ end_evidence, text, pages, pages_anchored, pages_inferred }`.
296
+
297
+ - `confidence` is `'low' | 'medium' | 'high'` — coarse enum, not a 0..1
298
+ number. LLMs cluster numeric scores meaninglessly; the enum is reliable.
299
+ - `evidence.start_heading_text` is the verbatim heading the LLM cited (parse
300
+ this instead of regexing over `notes`).
301
+ - `end_evidence` is set when the LLM detected a section close (matched
302
+ against the section's `endHints` or an explicit closing marker like
303
+ `*Koniec prílohy*`). Carries `{ end_page, confidence, notes }`. Absent
304
+ when continuity-fill alone closed the section. Pair `evidence.notes` +
305
+ `end_evidence.notes` for a reconciliation pass that judges whether the
306
+ section is correctly bounded.
307
+ - `pages[i].source` is `'anchored'` (direct LLM evidence) or `'inferred'`
308
+ (filled by continuity) — use it to deprioritize inferred content.
309
+
310
+ Pages that match no section are silently dropped (Reducto-style); add an
311
+ explicit "everything else" section if you need a catch-all.
312
+
313
+ Per-section knobs:
314
+
315
+ - `required: true` — log a warn if missing and lean the LLM toward expecting
316
+ it. Default: false.
317
+ - `endHints: [string]` — natural-language cues the LLM uses to find the
318
+ section's END. Treated semantically (casing variants, missing diacritics,
319
+ multilingual phrasings all qualify) — this is NOT a regex match. The LLM
320
+ emits a structured `end_anchor`; deterministic merge applies it. Replaces
321
+ the older `endAnchors` line-prefix matcher.
322
+
323
+ `extract-each-section.output` is `{ items, count, totalIterations }`, not a
324
+ flat array — `items[i]` holds the per-iteration result keyed by the last
325
+ inner step's name (here `{ "extract-section-fields": {...} }`).
326
+
229
327
  ## Validate before pushing
230
328
 
231
329
  ```bash
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@eigenpal/cli",
3
- "version": "0.4.6",
3
+ "version": "0.4.8",
4
4
  "description": "Eigenpal CLI — author and run document workflows from your terminal. Agent-ready.",
5
5
  "keywords": [
6
6
  "eigenpal",