@forzalabs/remora 2.0.1 → 2.0.3

package/CHANGELOG.md

@@ -6,6 +6,29 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.0.3 - 2026-09-03
+
+### Added
+
+- `remora ui-preview` now has **search**, at both levels the project has: a box in the header finds a **node**, and a box above the panel's tables filters the **fields inside the selected one**. The node search matches on name, kind, description and badges, all terms in any order (`cons canary` finds `c_canary_extract`), and its answer lands **in the picture** rather than only beside it — everything that does not match fades on the canvas, so a search over fifty nodes leaves the matching ones legible in the graph they belong to instead of handing back a list to go and find them in. `/` from anywhere focuses it, ↑/↓ walk the results, Enter selects and pans to it, Escape clears. The field filter keeps only the rows that match — a dimension, a consumer field, a resolved lineage path, an output, a run — and the **tab strip carries the count per tab**, which is what makes it usable on a resource whose fields are spread over four tables: `addr0Street` says which tab holds it without each one having to be opened to find out. Matched text is marked in place, since a filtered row of near-identical generated column names is not self-explanatory once found. It filters a `Data` sample the same way, is dropped when the selection moves to another resource, and survives a live reload of the same one — like the open tab and the scroll position. All of it runs **in the page**, on the documents it already has: no route, no new capability and no contract change, so a snapshot opened with `remora-preview <file>` searches exactly the same way
+
+### Changed
+
+- **Sampling now runs the pipeline.** `remora sample <consumer>`, and the **Data** tab of `remora ui-preview`, used to answer a question nobody asked: the rows shown for a consumer were its *first producer's* rows, projected through the consumer's field list. No filters, no field transformations, no unions, no `distinct`, no `pivot`, no measures, no validation. Under a heading that says "Data", beside a consumer's name, that read as what the consumer produces — and it was not: a consumer whose only job is to uppercase a column showed it lowercase, and one that keeps one row in a hundred showed the ninety-nine. Sampling a consumer now **executes it** — every filter, transformation, mask, union, `distinct`, `pivot`, measure and dataset validation, exactly as `remora run` applies them — so the rows are the consumer's own output, always, rather than an approximation of it. It **touches nothing**: no write to the destination, no `onSuccess`/`onError` action, and although the execution is recorded like any other it is marked as a sample, so it stays out of the run overlay, out of a consumer's run list and out of its duration averages — a node's dot and its "how long does this take" keep meaning a real run over real volume. The bound on the rows is the same `limit` `remora run --limit` uses, so the cost follows the same rule: where nothing between the source and the output can *drop* a record the limit reaches the source itself and the read stops early, and where something can — any filter, `distinct`, `distinctOn`, `pivot`, or a validation set to `"onFail": "skip"` — there is no knowing how many rows must go in for ten to come out, so the whole source is read. On a large source that is slow; it is still only a read. Sampling a consumer needs a source the pipeline can stage — `local`, `aws-s3` or `delta-share` — so one whose producer reads `aws-redshift` or `http-api` (whose drivers have no `ready()`) is refused up front with a message naming the producer and pointing at it, instead of a worker pool starting and then surfacing `Not implemented yet` from three layers down. In the preview it takes the same one-at-a-time lock as `▶ Run`, since it is the same worker pool. Sampling a **producer** is unchanged in shape and still costs one bounded read; what it now does is the producer stage in full (see the fix below). None of this added anything to the execution path: a sample is `launch` with three flags the pipeline already had for the materialization runs it already did — `options.limit`, `skipExport` and `captureMainDatasetTo` — so the executor cannot tell it is being sampled, and one code path still serves both callers
+- A sampled consumer's columns are now ordered by the **fields the consumer declares**, rather than by the order the producer's dimensions happened to be read in. Invisible in an exported JSON object; the whole shape of a table you are reading
+- `remora ui-preview`'s **Data** tab is quieter. The button keeps one label — `Sample 10 rows` — rather than re-wording itself to `Sample again 10 rows` once rows are on screen: it does the same thing on the tenth press as on the first, and moving the text under the pointer said nothing new. The paragraph beside it, explaining what sampling costs, is gone; what actually ran is stated over the table by the host that ran it, which is the sentence worth reading and is there every time. The button still says what it is doing while it waits
+
+### Fixed
+
+- **A masked producer dimension was shown in the clear when sampled.** A dimension declared `"mask": "hash"` (or `crypt`, `random`, `seeded-random`) is masked in `ProducerExecutor.processLine`, on the run path — and the sampling path never went through it. So `remora sample <producer>` and the preview's **Data** tab printed the raw value of every column the project had asked to be masked, as did the projected consumer rows built from them. That is a disclosure rather than a display bug: the values reach a terminal and a browser page, and the one place someone checks that masking works is the place that was not applying it. `sourceFilename` dimensions were blank there for the same reason. The producer stage now lives in one place — `ProducerManager.compileFills`/`applyFills` — which both the worker and the sample read call, so a mask cannot be applied on one path and skipped on the other
+- **A sample in `remora ui-preview` showed no rows at all.** The Data tab ran the sample, the host answered with the rows, and the panel stayed empty with nothing in the log to explain it. Two things in the run overlay caused it, both from a sample being *recorded* like any other execution and then filtered out of the picture: `RunOverlay.signature()` — the token the preview polls to notice a finished run — signed the usage **store** rather than the runs it paints, so writing a sample's own record moved the token while the picture stayed identical; and the sample marker was dropped *after* the 500-row read limit, so a sample also pushed a real run off the end of the history. Either way the preview concluded the project had changed and told the page so, and a page told its resource changed **drops the sample it is showing** — deliberately, since rows taken against the previous definition must never be shown under the new one. So the rows were fetched and then discarded on arrival, every time. The signature now signs the overlay it describes (gated on the store's mtime, so polling still reads metadata and not records) and the marker is applied as the store reads, so a sample cannot displace a run. A real run notifies and repaints exactly as before
+- **`remora ui-preview`'s minimap and zoom controls were white in dark mode**, which on a dark canvas left them as a bright rectangle and three invisible buttons. xyflow paints its own chrome from its own palette and ships a `.dark` variant of it that has to be switched on and then kept in step with greys that are not this product's; it now reads the theme's tokens through the `--xy-*` slots the library exposes, so one declaration themes the canvas in both modes and the canvas can never be in a mode the rest of the page is not
+- **The minimap was empty**, whatever colour it was. It draws a node from the size the node DECLARES, and the layout — which fixes every card at the same box and reserves exactly that box in dagre — had never said so on the node itself, so there was nothing for it to draw and the panel was a blank rectangle. The box is now declared as `initialWidth`/`initialHeight`, the size xyflow assumes until it measures the rendered card: the minimap gets its nodes and the canvas keeps measuring, so the picture itself is unchanged
+- `remora ui-preview`'s sample route is now a `POST` carrying the session token, like `▶ Run` already was. It **executes**: a producer's sample spends a query against your warehouse and a consumer's spends this machine's threads, and loopback is not privacy — every site you visit can address 127.0.0.1, so a `GET` let any page you happened to have open start one
+- A consumer's `onError` actions no longer fire for a run that exports nothing. A `code_lookup` table or an upstream consumer-input that failed while being materialized ran the dependent consumer's `onError` — moving its source file, starting other consumers — for an execution nobody asked for; the success path had always skipped them, and the failure path now does too
+- `remora sample <producer> -s 10` on a file with a header row returned **nine** rows. The header was one of the lines read and then dropped, so the count asked for was always short by one
+- `remora ui-preview` no longer describes itself as **read-only**. The panel's footer promised "this viewer is read-only and never reads your data", and the command's own `--help` said the same, while the page it was printed on samples real rows, runs consumers and saves configuration files. What is still true is narrower and is where it is now said: the `@forzalabs/remora-preview` **package** has no drivers, no credentials and no configuration reader, so everything past rendering is a capability of the host and reaches the page as a route only when the host advertises one — and `remora-preview <snapshot.json>`, which has no host behind it, advertises none and is read-only in fact rather than by claim
+
 ## V 2.0.1 - 2026-09-01
 
 ### Changed
@@ -26,7 +49,7 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 - A consumer's filters are now **pushed down to a Delta Share source**, so the share server skips the files that cannot hold a matching row instead of handing over the whole table for remora to read, stage and throw away. The eligible filters go out as `jsonPredicateHints` on the table query — the protocol's structured predicate form, which prunes on partition columns and, where the share keeps them, on file statistics. Filters supplied by an invocation (an API or queue request) are pushed on the same terms as the ones written into the consumer, which is where it pays most: an API caller asking for one region no longer drags the other twenty across the wire first. **The result is identical either way.** The hint is best-effort by protocol — a server may ignore it, may fail to parse it and return everything, and may return files that don't match — so every filter is still evaluated against every record exactly as before; pushing one down only ever saves download time and staging space. Which is also why it is deliberately conservative about what it pushes. A filter goes to the source only when it can mean the same thing there: the field has to read a producer dimension **untouched** (no `transform`, no `validate`, no `default`, not `fixed`/`copyFrom`/`lineCount`, not a masked or `sourceFilename` dimension, and not a field belonging to a different producer), the operator has to have a counterpart in the Delta Sharing grammar (`equals`, `in`, `between` and the four `greaterThan`/`lessThan` comparisons — the substring and regex operators have none), and — the subtle one — the filter must not also keep rows whose value is **missing**. A value the source doesn't have reads back on the record as `""` or `0` rather than as null, so `notEquals`, `isNotNull`, `notBetween` and any range covering `0` keep those rows; a source pruning on them would drop data the run is supposed to return, so none of them is ever pushed. Date bounds are only translated when they are ISO-8601, and one carrying milliseconds is left local rather than rounded to a second, since rounding a bound is exactly how a row that belongs in the result stops being fetched. Boolean columns are never pushed, because remora reads a boolean cell through `Boolean(value)` and the source's own `false` reads back as `true`. Nothing needs configuring: it applies to any consumer with filters over a Delta Share producer. A producer whose share mishandles the hint can opt out with `"disablePushdown": true` in its settings and lose nothing but bandwidth. What the run sends is written to the log, predicate and all
 - The **record limit** (`remora run -l`, or `limit` on an API/queue request) is pushed down to a Delta Share source too, as the query's `limitHint`, so a share server can stop listing files once it has covered the rows being asked for. Where the filter pushdown saves reading rows that will be discarded, this saves reading a table that was never going to be looked at: `remora run c_big -l 100` against a table of millions no longer downloads all of it to write a hundred records. The rule for pushing it is stricter than for a filter, because a bounded read is only sound when every row read reaches the output — so the limit is dropped whenever anything between the source and the export can *drop* a record: any filter at all (a bounded read could have fetched exactly the rows the filter goes on to reject), `distinct`, `distinctOn`, `pivot`, a field validation set to `"onFail": "skip"`, a producer `recordSelector`, or a group fanning a line out to rows. An `offset` or an `order` drops it as well, since both need rows a bounded read would stop before. A transformation's own `"onError": "skip"` does not, since it abandons that one transformation rather than the record. Like the predicate it is a hint — the server may return files holding far more rows than asked for — and the run's own limit is unchanged and still enforced. `"disablePushdown": true` on a producer turns both hints off
-- Added `remora ui-preview`, a read-only visual preview of the project in the browser: the lineage graph on a pan/zoom canvas, and a click-through inspector for every resource. Where `remora graph` prints the picture, this one is explorable — select a producer and its dimensions, measures and settings are **tables**, not JSON; the source it reads links to the source; a consumer's fields link to the producer dimension each one comes from, its producers to the producers, its outputs to where the data lands and to the consumers they chain into, and the panel keeps a back stack so you can walk the pipeline by clicking through it. Nodes are coloured by what is wrong with them and every problem the project has is listed in one panel, each one a click from the resource it is about. **It updates while you edit.** The configuration is watched, and a save re-reads the project and refreshes the page in place — keeping your selection, your scroll and your position on the canvas — so the picture is never the one from two saves ago. A file caught mid-write briefly shows as a red node and goes green again a second later, because the preview reads a project the lenient way (`Environment.inspect()`): a broken configuration renders rather than failing, which is when the picture is most wanted. `-p, --port <n>` picks the port (default 5070, falling forward to the next free one), `--no-open` leaves the browser alone. The page is served on loopback only, and **the viewer cannot reach your data**: it is a separate package, `@forzalabs/remora-preview`, whose only runtime dependency is `express` — no drivers, no credentials, no configuration reader. It receives one versioned JSON document and renders it. **It can also show you the data.** A producer or a consumer gets a **Data** tab that reads a handful of real rows and puts them beside the columns the resource declares, so "is this column actually populated?" is answered where the question comes up — a consumer's rows arriving projected through its own field list, which is the shape it will export. It is the same read `remora sample` does, through the same code path, so the two can never disagree. It reads **only when you click it** — never on selecting a node and never on a save — because a sample is a query against the real source with your credentials, and the rows are dropped whenever the project is re-read, so you are never looking at old data under a new definition. That is a capability of the **CLI**, not of the viewer: the page offers the tab only when the host it is talking to advertises one, so a snapshot opened with `remora-preview <file>` has no such tab and no route behind it, and a source that cannot be reached comes back as the reason rather than as a blank panel. **It shows you where every column comes from.** A consumer gets a **Lineage** tab that resolves each of its output columns back to the producer column it started as — through however many consumers sit in between — with every transformation applied along the way and a link to the origin, so `country_upper` reads back as `p_orders.country → c_stage.country → c_final.country_upper · to_uppercase` instead of being a walk through three files with the references held in your head. `fixed`, `lineCount` and `copyFrom` fields resolve to what they actually are, and a column whose chain cannot be completed says why — a `from` naming no producer of the consumer, a key the producer does not declare, a name ambiguous between two non-union producers — rather than going quietly blank. A producer gets the same thing from the other end, a **Used by** tab: which consumers read each of its columns, and, the half that is hard to get any other way, which columns **nothing** reads at all. It is resolved the way the run path resolves it, by the same `from`/union/ambiguity rules, because a lineage that disagreed with the executor would be worse than no lineage at all. **And it shows you what happened the last time each consumer ran.** Every node carries a dot coloured by the outcome of its most recent run, with the age, the row count and the failure reason in its tooltip, and that consumer's recent runs as a **Runs** tab — outcome, when, how long, how many rows and which Remora version ran it, since a timing is only comparable within one release. Those come from the CLI's own local run store, the one `remora run` already writes to `remora/.temp/usage`, so a run that finishes while the preview is open repaints the graph within a second even though no configuration file changed. A deployed worker's history is not read here — it has its own channel by design, and colouring a graph is not a reason to poll a database once a second. `remora graph -f json` carries the field lineage, which is configuration and travels with the document, and deliberately does not carry the runs: “failed 3 minutes ago” frozen into a file someone opens next week reads as permanent. **And it does things, not only shows them.** A consumer has a **Run** button: it starts the same execution `remora run` does, with the same licence, the same records and the same output, and hands back control immediately — the node's own dot turns blue while it runs and green or red when it finishes, so the picture you are already looking at is the progress display. One run at a time, a producer cannot be run (only consumers execute), and a consumer whose node is already red is refused with the diagnostic as the reason, because running a configuration the preview has just told you is broken fails somewhere less legible. Beside the run history there is now a **Metrics** table — runs recorded, last and average duration over the successful ones, rows in and out, workers, output cycles, and the per-stage timings of the last run, which is where a slow run says *which* stage was slow. A **Logs** drawer across the bottom tails the log file the CLI writes, refreshing while it is open. And a **File** tab holds the resource **as authored**, editable, with Save and Revert: it is the file on disk rather than the loaded object, because a loaded producer has already had its groups and `occurs` expanded and writing that back would persist the expansion as if you had typed it. Saving is refused, with the reason, for JSON that will not parse, for a rename (a resource's name is its identity in every reference to it and in the name of its own file), and for anything its JSON schema rejects — checked with the same code `compile` runs, so the panel can never save a project the CLI would then refuse to load, and says so in the same words. A save lands atomically and the page refreshes itself a moment later, and an unsaved draft survives that refresh rather than being thrown away, with a warning when the file moved underneath it. **All of that is guarded.** Binding to loopback keeps the page off the network but does not stop a *browser* — every site you visit can address 127.0.0.1, and a hostname that resolves there defeats a naive check — which was a curiosity while every route returned a document and is not one now that a route can run your pipeline with your warehouse credentials. So the server refuses any request whose `Host` header is not one of its own (closing DNS rebinding), refuses anything carrying a cross-origin `Origin`, and requires a per-session token, injected into the page itself, on every request that changes something. The reads are guarded the same way. Running, editing and log-tailing are all **capabilities of the CLI**, like sampling: a snapshot opened with `remora-preview <file>` has no such routes at all, so there is nothing there to refuse. **The panel resizes.** The divider between the graph and the inspector drags, so a wide table — a resolved lineage path, a 1,300-column dimension list, a JSON file — gets the room it needs without the graph being stuck at whatever is left; double-click the divider to go back to the default, and the width you chose is remembered. That package also installs a `remora-preview <snapshot.json>` command of its own, which serves a snapshot written by `remora graph -f json` for someone with no CLI and no licence
+- Added `remora ui-preview`, a visual preview of the project in the browser: the lineage graph on a pan/zoom canvas, and a click-through inspector for every resource. Where `remora graph` prints the picture, this one is explorable — select a producer and its dimensions, measures and settings are **tables**, not JSON; the source it reads links to the source; a consumer's fields link to the producer dimension each one comes from, its producers to the producers, its outputs to where the data lands and to the consumers they chain into, and the panel keeps a back stack so you can walk the pipeline by clicking through it. Nodes are coloured by what is wrong with them and every problem the project has is listed in one panel, each one a click from the resource it is about. **It updates while you edit.** The configuration is watched, and a save re-reads the project and refreshes the page in place — keeping your selection, your scroll and your position on the canvas — so the picture is never the one from two saves ago. A file caught mid-write briefly shows as a red node and goes green again a second later, because the preview reads a project the lenient way (`Environment.inspect()`): a broken configuration renders rather than failing, which is when the picture is most wanted. `-p, --port <n>` picks the port (default 5070, falling forward to the next free one), `--no-open` leaves the browser alone. The page is served on loopback only, and **the viewer cannot reach your data**: it is a separate package, `@forzalabs/remora-preview`, whose only runtime dependency is `express` — no drivers, no credentials, no configuration reader. It receives one versioned JSON document and renders it. **It can also show you the data.** A producer or a consumer gets a **Data** tab that reads a handful of real rows and puts them beside the columns the resource declares, so "is this column actually populated?" is answered where the question comes up — a consumer's rows arriving projected through its own field list, which is the shape it will export. It is the same read `remora sample` does, through the same code path, so the two can never disagree. It reads **only when you click it** — never on selecting a node and never on a save — because a sample is a query against the real source with your credentials, and the rows are dropped whenever the project is re-read, so you are never looking at old data under a new definition. That is a capability of the **CLI**, not of the viewer: the page offers the tab only when the host it is talking to advertises one, so a snapshot opened with `remora-preview <file>` has no such tab and no route behind it, and a source that cannot be reached comes back as the reason rather than as a blank panel. **It shows you where every column comes from.** A consumer gets a **Lineage** tab that resolves each of its output columns back to the producer column it started as — through however many consumers sit in between — with every transformation applied along the way and a link to the origin, so `country_upper` reads back as `p_orders.country → c_stage.country → c_final.country_upper · to_uppercase` instead of being a walk through three files with the references held in your head. `fixed`, `lineCount` and `copyFrom` fields resolve to what they actually are, and a column whose chain cannot be completed says why — a `from` naming no producer of the consumer, a key the producer does not declare, a name ambiguous between two non-union producers — rather than going quietly blank. A producer gets the same thing from the other end, a **Used by** tab: which consumers read each of its columns, and, the half that is hard to get any other way, which columns **nothing** reads at all. It is resolved the way the run path resolves it, by the same `from`/union/ambiguity rules, because a lineage that disagreed with the executor would be worse than no lineage at all. **And it shows you what happened the last time each consumer ran.** Every node carries a dot coloured by the outcome of its most recent run, with the age, the row count and the failure reason in its tooltip, and that consumer's recent runs as a **Runs** tab — outcome, when, how long, how many rows and which Remora version ran it, since a timing is only comparable within one release. Those come from the CLI's own local run store, the one `remora run` already writes to `remora/.temp/usage`, so a run that finishes while the preview is open repaints the graph within a second even though no configuration file changed. A deployed worker's history is not read here — it has its own channel by design, and colouring a graph is not a reason to poll a database once a second. `remora graph -f json` carries the field lineage, which is configuration and travels with the document, and deliberately does not carry the runs: “failed 3 minutes ago” frozen into a file someone opens next week reads as permanent. **And it does things, not only shows them.** A consumer has a **Run** button: it starts the same execution `remora run` does, with the same licence, the same records and the same output, and hands back control immediately — the node's own dot turns blue while it runs and green or red when it finishes, so the picture you are already looking at is the progress display. One run at a time, a producer cannot be run (only consumers execute), and a consumer whose node is already red is refused with the diagnostic as the reason, because running a configuration the preview has just told you is broken fails somewhere less legible. Beside the run history there is now a **Metrics** table — runs recorded, last and average duration over the successful ones, rows in and out, workers, output cycles, and the per-stage timings of the last run, which is where a slow run says *which* stage was slow. A **Logs** drawer across the bottom tails the log file the CLI writes, refreshing while it is open. And a **File** tab holds the resource **as authored**, editable, with Save and Revert: it is the file on disk rather than the loaded object, because a loaded producer has already had its groups and `occurs` expanded and writing that back would persist the expansion as if you had typed it. Saving is refused, with the reason, for JSON that will not parse, for a rename (a resource's name is its identity in every reference to it and in the name of its own file), and for anything its JSON schema rejects — checked with the same code `compile` runs, so the panel can never save a project the CLI would then refuse to load, and says so in the same words. A save lands atomically and the page refreshes itself a moment later, and an unsaved draft survives that refresh rather than being thrown away, with a warning when the file moved underneath it. **All of that is guarded.** Binding to loopback keeps the page off the network but does not stop a *browser* — every site you visit can address 127.0.0.1, and a hostname that resolves there defeats a naive check — which was a curiosity while every route returned a document and is not one now that a route can run your pipeline with your warehouse credentials. So the server refuses any request whose `Host` header is not one of its own (closing DNS rebinding), refuses anything carrying a cross-origin `Origin`, and requires a per-session token, injected into the page itself, on every request that changes something. The reads are guarded the same way. Running, editing and log-tailing are all **capabilities of the CLI**, like sampling: a snapshot opened with `remora-preview <file>` has no such routes at all, so there is nothing there to refuse. **The panel resizes.** The divider between the graph and the inspector drags, so a wide table — a resolved lineage path, a 1,300-column dimension list, a JSON file — gets the room it needs without the graph being stuck at whatever is left; double-click the divider to go back to the default, and the width you chose is remembered. That package also installs a `remora-preview <snapshot.json>` command of its own, which serves a snapshot written by `remora graph -f json` for someone with no CLI and no licence
 - Added **repeating groups** to a `FIXED` producer's dimensions, for flat files transcribed from a COBOL copybook (`OCCURS`). A dimension that carries its own nested `dimensions` is a *group*: it declares no `type`, produces no column of its own, and instead declares the fields that repeat — with each child `position` **relative to one occurrence**, and the group's own `position` describing **one occurrence** (`start` of the first, `length` per occurrence). `"occurs": 3` on a 30-character address segment starting at column 32 therefore reads occurrences at 32, 62 and 92, generating `addr0Street`, `addr0State`, … `addr2Zip`. Names follow a `naming` template (`{group}`, `{index}`, `{field}`, `{Field}`; default `{group}{index}{Field}`) with `indexFrom` choosing whether indexes start at 0 or 1, so generated names can match the project's own style. Groups expand **in place** when the project loads, at the group's own index, so the columns keep the physical order of the record and a dimension declared after a group stays after it — and because expansion happens before anything reads the dimensions, consumers, transforms, validations, exports, `sample`, `mock` and `synth` all work on a grouped producer with no changes at all. `occurs` defaults to `1`, which makes a group also a way to simply structure a segment of a layout. Mistakes are rejected at `compile` with a message naming the dimension: a child position that doesn't fit inside one occurrence (an absolute offset pasted where a relative one belongs), a group carrying properties that only mean anything on a column (`type`, `alias`, `pk`, `mask`, `synth`, `format`, `sourceFilename`), a `naming` template with no `{index}`, a generated name colliding with another dimension, a group on a non-`FIXED` producer, and — for now — a nested group
 - Added `flags` on a `FIXED` producer dimension, decoding a per-character bitmap into one dimension per character: `"flags": ["mailing", "home", "billing"]` on a 3-character field yields `addressUseMailing`, `addressUseHome` and `addressUseBilling`, each a 1-character slice from `position.start`. `flags` must list exactly one name per character the dimension spans. Each character is *sliced, never interpreted*, so a value that isn't strictly binary survives as itself (a `"3"` stays `"3"`) and can be decoded downstream with `conditional` or `code_lookup`. Names follow the same `naming` template mechanism (`{field}`, `{Field}`, `{flag}`, `{Flag}`; default `{field}{Flag}`), and `flags` composes with a repeating group, where the group's naming applies first
 - Added **pattern field keys** on a consumer: a `key` holding a `*` (`"addr*Street"`) selects every field whose name matches, and each match keeps the rest of that field's configuration — so one entry can alias, transform and validate a whole family of columns, which is what makes a repeating group's generated columns usable without writing one field per occurrence. Matching is literal apart from the wildcards, never a regular expression, and a pattern expands where it is declared, in the producer's own column order. Because several columns come out of one entry, `alias` becomes a template that must vary per match: `*` is replaced by what the pattern captured (positionally) and `{key}` by the matched field name, so `{ "key": "addr*Street", "alias": "street_*" }` yields `street_0`, `street_1`, `street_2`. A pattern that matches nothing, an `alias` that would give two matches the same output name, and a pattern combined with `fixed`/`copyFrom`/`lineCount` (fields that read no producer column) are all rejected at `compile`. A consumer reading another consumer's output sees the expanded names, not the pattern. The bare `"key": "*"` keeps its existing meaning