@forzalabs/remora 1.8.0 → 2.0.0

package/CHANGELOG.md

@@ -6,6 +6,66 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.0.0 - 2026-09-01
+
+**Upgrading from 1.x.** Four changes can alter what an existing project does. JSON schema validation now actually runs at load, so a configuration that was silently schema-invalid fails at `compile` — run `remora compile` (or `remora graph`, which lists every problem at once without failing) before upgrading. `filters[].sql` and the `"sql"` filter operator are gone; rewrite them as `rule` filters. Filters on `number`, `datetime` and `boolean` fields now match, so a consumer whose filters were silently matching nothing will start returning rows. And `remora run -l` is now the whole run's limit rather than each worker's, so a limited run returns the number of rows asked for instead of that number multiplied by the chunk count. The `mock` command is removed — use `synth`.
+
+### Added
+
+- A consumer's filters are now **pushed down to a Delta Share source**, so the share server skips the files that cannot hold a matching row instead of handing over the whole table for remora to read, stage and throw away. The eligible filters go out as `jsonPredicateHints` on the table query — the protocol's structured predicate form, which prunes on partition columns and, where the share keeps them, on file statistics. Filters supplied by an invocation (an API or queue request) are pushed on the same terms as the ones written into the consumer, which is where it pays most: an API caller asking for one region no longer drags the other twenty across the wire first. **The result is identical either way.** The hint is best-effort by protocol — a server may ignore it, may fail to parse it and return everything, and may return files that don't match — so every filter is still evaluated against every record exactly as before; pushing one down only ever saves download time and staging space. Which is also why it is deliberately conservative about what it pushes. A filter goes to the source only when it can mean the same thing there: the field has to read a producer dimension **untouched** (no `transform`, no `validate`, no `default`, not `fixed`/`copyFrom`/`lineCount`, not a masked or `sourceFilename` dimension, and not a field belonging to a different producer), the operator has to have a counterpart in the Delta Sharing grammar (`equals`, `in`, `between` and the four `greaterThan`/`lessThan` comparisons — the substring and regex operators have none), and — the subtle one — the filter must not also keep rows whose value is **missing**. A value the source doesn't have reads back on the record as `""` or `0` rather than as null, so `notEquals`, `isNotNull`, `notBetween` and any range covering `0` keep those rows; a source pruning on them would drop data the run is supposed to return, so none of them is ever pushed. Date bounds are only translated when they are ISO-8601, and one carrying milliseconds is left local rather than rounded to a second, since rounding a bound is exactly how a row that belongs in the result stops being fetched. Boolean columns are never pushed, because remora reads a boolean cell through `Boolean(value)` and the source's own `false` reads back as `true`. Nothing needs configuring: it applies to any consumer with filters over a Delta Share producer. A producer whose share mishandles the hint can opt out with `"disablePushdown": true` in its settings and lose nothing but bandwidth. What the run sends is written to the log, predicate and all
+- The **record limit** (`remora run -l`, or `limit` on an API/queue request) is pushed down to a Delta Share source too, as the query's `limitHint`, so a share server can stop listing files once it has covered the rows being asked for. Where the filter pushdown saves reading rows that will be discarded, this saves reading a table that was never going to be looked at: `remora run c_big -l 100` against a table of millions no longer downloads all of it to write a hundred records. The rule for pushing it is stricter than for a filter, because a bounded read is only sound when every row read reaches the output — so the limit is dropped whenever anything between the source and the export can *drop* a record: any filter at all (a bounded read could have fetched exactly the rows the filter goes on to reject), `distinct`, `distinctOn`, `pivot`, a field validation set to `"onFail": "skip"`, a producer `recordSelector`, or a group fanning a line out to rows. An `offset` or an `order` drops it as well, since both need rows a bounded read would stop before. A transformation's own `"onError": "skip"` does not, since it abandons that one transformation rather than the record. Like the predicate it is a hint — the server may return files holding far more rows than asked for — and the run's own limit is unchanged and still enforced. `"disablePushdown": true` on a producer turns both hints off
+- Added `remora ui-preview`, a read-only visual preview of the project in the browser: the lineage graph on a pan/zoom canvas, and a click-through inspector for every resource. Where `remora graph` prints the picture, this one is explorable — select a producer and its dimensions, measures and settings are **tables**, not JSON; the source it reads links to the source; a consumer's fields link to the producer dimension each one comes from, its producers to the producers, its outputs to where the data lands and to the consumers they chain into, and the panel keeps a back stack so you can walk the pipeline by clicking through it. Nodes are coloured by what is wrong with them and every problem the project has is listed in one panel, each one a click from the resource it is about. **It updates while you edit.** The configuration is watched, and a save re-reads the project and refreshes the page in place — keeping your selection, your scroll and your position on the canvas — so the picture is never the one from two saves ago. A file caught mid-write briefly shows as a red node and goes green again a second later, because the preview reads a project the lenient way (`Environment.inspect()`): a broken configuration renders rather than failing, which is when the picture is most wanted. `-p, --port <n>` picks the port (default 5070, falling forward to the next free one), `--no-open` leaves the browser alone. The page is served on loopback only, and **the viewer cannot reach your data**: it is a separate package, `@forzalabs/remora-preview`, whose only runtime dependency is `express` — no drivers, no credentials, no configuration reader. It receives one versioned JSON document and renders it. **It can also show you the data.** A producer or a consumer gets a **Data** tab that reads a handful of real rows and puts them beside the columns the resource declares, so "is this column actually populated?" is answered where the question comes up — a consumer's rows arriving projected through its own field list, which is the shape it will export. It is the same read `remora sample` does, through the same code path, so the two can never disagree. It reads **only when you click it** — never on selecting a node and never on a save — because a sample is a query against the real source with your credentials, and the rows are dropped whenever the project is re-read, so you are never looking at old data under a new definition. That is a capability of the **CLI**, not of the viewer: the page offers the tab only when the host it is talking to advertises one, so a snapshot opened with `remora-preview <file>` has no such tab and no route behind it, and a source that cannot be reached comes back as the reason rather than as a blank panel. **It shows you where every column comes from.** A consumer gets a **Lineage** tab that resolves each of its output columns back to the producer column it started as — through however many consumers sit in between — with every transformation applied along the way and a link to the origin, so `country_upper` reads back as `p_orders.country → c_stage.country → c_final.country_upper · to_uppercase` instead of being a walk through three files with the references held in your head. `fixed`, `lineCount` and `copyFrom` fields resolve to what they actually are, and a column whose chain cannot be completed says why — a `from` naming no producer of the consumer, a key the producer does not declare, a name ambiguous between two non-union producers — rather than going quietly blank. A producer gets the same thing from the other end, a **Used by** tab: which consumers read each of its columns, and, the half that is hard to get any other way, which columns **nothing** reads at all. It is resolved the way the run path resolves it, by the same `from`/union/ambiguity rules, because a lineage that disagreed with the executor would be worse than no lineage at all. **And it shows you what happened the last time each consumer ran.** Every node carries a dot coloured by the outcome of its most recent run, with the age, the row count and the failure reason in its tooltip, and that consumer's recent runs as a **Runs** tab — outcome, when, how long, how many rows and which Remora version ran it, since a timing is only comparable within one release. Those come from the CLI's own local run store, the one `remora run` already writes to `remora/.temp/usage`, so a run that finishes while the preview is open repaints the graph within a second even though no configuration file changed. A deployed worker's history is not read here — it has its own channel by design, and colouring a graph is not a reason to poll a database once a second. `remora graph -f json` carries the field lineage, which is configuration and travels with the document, and deliberately does not carry the runs: “failed 3 minutes ago” frozen into a file someone opens next week reads as permanent. **And it does things, not only shows them.** A consumer has a **Run** button: it starts the same execution `remora run` does, with the same licence, the same records and the same output, and hands back control immediately — the node's own dot turns blue while it runs and green or red when it finishes, so the picture you are already looking at is the progress display. One run at a time, a producer cannot be run (only consumers execute), and a consumer whose node is already red is refused with the diagnostic as the reason, because running a configuration the preview has just told you is broken fails somewhere less legible. Beside the run history there is now a **Metrics** table — runs recorded, last and average duration over the successful ones, rows in and out, workers, output cycles, and the per-stage timings of the last run, which is where a slow run says *which* stage was slow. A **Logs** drawer across the bottom tails the log file the CLI writes, refreshing while it is open. And a **File** tab holds the resource **as authored**, editable, with Save and Revert: it is the file on disk rather than the loaded object, because a loaded producer has already had its groups and `occurs` expanded and writing that back would persist the expansion as if you had typed it. Saving is refused, with the reason, for JSON that will not parse, for a rename (a resource's name is its identity in every reference to it and in the name of its own file), and for anything its JSON schema rejects — checked with the same code `compile` runs, so the panel can never save a project the CLI would then refuse to load, and says so in the same words. A save lands atomically and the page refreshes itself a moment later, and an unsaved draft survives that refresh rather than being thrown away, with a warning when the file moved underneath it. **All of that is guarded.** Binding to loopback keeps the page off the network but does not stop a *browser* — every site you visit can address 127.0.0.1, and a hostname that resolves there defeats a naive check — which was a curiosity while every route returned a document and is not one now that a route can run your pipeline with your warehouse credentials. So the server refuses any request whose `Host` header is not one of its own (closing DNS rebinding), refuses anything carrying a cross-origin `Origin`, and requires a per-session token, injected into the page itself, on every request that changes something. The reads are guarded the same way. Running, editing and log-tailing are all **capabilities of the CLI**, like sampling: a snapshot opened with `remora-preview <file>` has no such routes at all, so there is nothing there to refuse. **The panel resizes.** The divider between the graph and the inspector drags, so a wide table — a resolved lineage path, a 1,300-column dimension list, a JSON file — gets the room it needs without the graph being stuck at whatever is left; double-click the divider to go back to the default, and the width you chose is remembered. That package also installs a `remora-preview <snapshot.json>` command of its own, which serves a snapshot written by `remora graph -f json` for someone with no CLI and no licence
+- Added **repeating groups** to a `FIXED` producer's dimensions, for flat files transcribed from a COBOL copybook (`OCCURS`). A dimension that carries its own nested `dimensions` is a *group*: it declares no `type`, produces no column of its own, and instead declares the fields that repeat — with each child `position` **relative to one occurrence**, and the group's own `position` describing **one occurrence** (`start` of the first, `length` per occurrence). `"occurs": 3` on a 30-character address segment starting at column 32 therefore reads occurrences at 32, 62 and 92, generating `addr0Street`, `addr0State`, … `addr2Zip`. Names follow a `naming` template (`{group}`, `{index}`, `{field}`, `{Field}`; default `{group}{index}{Field}`) with `indexFrom` choosing whether indexes start at 0 or 1, so generated names can match the project's own style. Groups expand **in place** when the project loads, at the group's own index, so the columns keep the physical order of the record and a dimension declared after a group stays after it — and because expansion happens before anything reads the dimensions, consumers, transforms, validations, exports, `sample`, `mock` and `synth` all work on a grouped producer with no changes at all. `occurs` defaults to `1`, which makes a group also a way to simply structure a segment of a layout. Mistakes are rejected at `compile` with a message naming the dimension: a child position that doesn't fit inside one occurrence (an absolute offset pasted where a relative one belongs), a group carrying properties that only mean anything on a column (`type`, `alias`, `pk`, `mask`, `synth`, `format`, `sourceFilename`), a `naming` template with no `{index}`, a generated name colliding with another dimension, a group on a non-`FIXED` producer, and — for now — a nested group
+- Added `flags` on a `FIXED` producer dimension, decoding a per-character bitmap into one dimension per character: `"flags": ["mailing", "home", "billing"]` on a 3-character field yields `addressUseMailing`, `addressUseHome` and `addressUseBilling`, each a 1-character slice from `position.start`. `flags` must list exactly one name per character the dimension spans. Each character is *sliced, never interpreted*, so a value that isn't strictly binary survives as itself (a `"3"` stays `"3"`) and can be decoded downstream with `conditional` or `code_lookup`. Names follow the same `naming` template mechanism (`{field}`, `{Field}`, `{flag}`, `{Flag}`; default `{field}{Flag}`), and `flags` composes with a repeating group, where the group's naming applies first
+- Added **pattern field keys** on a consumer: a `key` holding a `*` (`"addr*Street"`) selects every field whose name matches, and each match keeps the rest of that field's configuration — so one entry can alias, transform and validate a whole family of columns, which is what makes a repeating group's generated columns usable without writing one field per occurrence. Matching is literal apart from the wildcards, never a regular expression, and a pattern expands where it is declared, in the producer's own column order. Because several columns come out of one entry, `alias` becomes a template that must vary per match: `*` is replaced by what the pattern captured (positionally) and `{key}` by the matched field name, so `{ "key": "addr*Street", "alias": "street_*" }` yields `street_0`, `street_1`, `street_2`. A pattern that matches nothing, an `alias` that would give two matches the same output name, and a pattern combined with `fixed`/`copyFrom`/`lineCount` (fields that read no producer column) are all rejected at `compile`. A consumer reading another consumer's output sees the expanded names, not the pattern. The bare `"key": "*"` keeps its existing meaning
+- Added `emit: "rows"` on a repeating group, turning its occurrences into **records instead of columns**: one output record per occurrence, where the children keep their own names, every dimension outside the group repeats on each record, and a generated field (`occurrenceField`, default `{group}Occurrence`) carries the index. That is what makes a slot list usable as data — three addresses become three rows rather than `addr0Street`/`addr1Street`/`addr2Street` — and the records stay flat, so CSV/TXT export, `pivot`, `distinctOn`, transforms and validations all keep working. Occurrences whose fields are all empty are skipped by default (`skipEmptyOccurrences`), since a layout reserving a hundred slots typically populates a handful. At most one group per producer may fan out; a `lineCount` field keeps meaning the **source** line, so the records fanned out of one line share it, while dataset row counts (`min_rows`/`max_rows`) count output records. `sample` shows the same fan-out a run produces
+- Added **variable-length repeating groups** to a `FIXED` producer: `occurs` accepts an object when the number of occurrences changes from line to line, which is the one thing about such a layout that fails silently rather than loudly. `{ "max": 36, "min": 34, "while": { "position": { "start": 1, "length": 1 }, "equals": "R" } }` takes occurrences while a marker inside the occurrence matches; `{ "max": 61, "from": "$slotCount" }` reads the count from an earlier dimension. `max` is mandatory in both forms — it is how many columns the group generates, so the output shape is still known before a line is read — and a line carrying fewer occurrences reads the missing ones as `null` rather than as the characters of whatever follows the block
+- Added `position.after` on a `FIXED` producer dimension, anchoring a position to the end of an earlier group's **actual** occurrences on that line instead of to the start of the line: `{ "after": "reimb", "length": 18 }`. Everything past a group whose span varies needs it — with an absolute `start`, those fields read plausible, wrong values on every line but the longest. It replaces `start`, which becomes an optional offset inside the anchored region (`{ "after": "reimb", "start": 11, "length": 2 }`), and works on both groups and plain dimensions. Positions are still resolved once per producer: only a layout that actually has a variable group or an anchor is resolved per line, and only its dynamic part. Dangling, self- and forward-references, an anchor to something that isn't a group, a variable group without `max` or with neither/both count rules, and an `occurs.from` naming a group, an anchored field or a dimension declared after the group are all rejected at `compile`; a dimension left at an absolute column after a variable group is warned about instead, since a layout may legitimately end with one
+- Added `omitNull` on a `JSON`/`API` consumer output: fields whose value is `null` are left out of each exported record instead of being written as `"field": null`. On a wide, sparse record — a 1,300-field positional layout with a few dozen populated fields — that is most of the payload. Only `null` is dropped; an empty string, `0` and `false` are values and are exported. Records then legitimately differ in shape, so `omitNull` is **rejected** rather than ignored on a positional format (`CSV`/`TXT`), where a column can't be missing from a row, and on a `JSON` output whose consumer also writes one, since every output is serialized from the same record
+- Consumers now reject two fields that would write the same output field name at `compile`. A record is keyed by each field's final name (`alias ?? key`), so a collision never failed at run time — one field silently overwrote the other
+- Added `recordSelector.minLength` and `recordSelector.onLengthMismatch` for `FIXED` producers. A matching record-type marker only says a line *claims* to be that record type, not that all of it is there, so a truncated record was parsed into a plausible half-record whose missing tail read as `null`. A matching line shorter than `minLength` is now dropped and logged at debug level (`"skip"`, the default) or fails the run (`"fail"`). The check runs only on lines the marker already matched, so a short header or trailer record is unaffected
+- Added `remora package`, which builds the deployable `.zip` of the local configuration and writes it to disk instead of uploading it — `./remora-config.zip` by default, or wherever `-o, --output` points. It is byte-for-byte the archive `deploy` sends, so the same file can be handed to a worker as the `remora_config` upload, mounted into a container, or hosted where a worker reads `REMORA_CONFIG_URI` from — which is what a deployment needs when the machine holding the configuration cannot reach the worker directly (an air-gapped or CI-mediated environment), or when the package has to be reviewed or archived before it goes anywhere. The configuration is validated first, exactly as `compile` does, so a package is never built from a project that would fail to load. Entries are stored relative to the project root, and the run scratch directories (`temp`, `.temp`), log output and OS noise are left out — `deploy` now excludes them too, having previously shipped `remora/.temp` (debug logs and temp datasets) inside every upload. `deploy --build-only`, which until now was accepted and silently ignored, does the same thing as `package`
+- Added `synth.unique` on a producer dimension: the column emits a distinct value in every generated row. Until now only a `pk` did that, so a natural key, an external reference or an account number — a column with a uniqueness constraint that is not *the* primary key — came out with duplicates, and a consumer's `distinct`, a unique dataset validation or a join against it was tested on data that could never have come from the real source. A unique column is served from the coordinated key pool the way a primary key is, and walked by row: `external_ref_1`, `external_ref_2`, … That is also its cost, and it is deliberate — uniqueness is not something a name-based generator can promise, so `synth.unique` on `email` yields `email_1` rather than a realistic address. It applies to `string` and `number` columns, combines with `synth.min`/`synth.max` the way any key does (so a unique column can also be pinned to a width), and is rejected on a column whose value comes from `allowedValues`/`codeSet` (a closed set repeats by definition) or from `references` (a unique foreign key is a one-to-one relation, which `synth` does not generate yet) — each with an error naming the dimension
+- A `synth` run that asks for more records than the key pool can hold (100,000) now fails when something in the project has to stay distinct — a primary key, a `synth.unique` column, or a field another producer references. Past that point the pool started over, so a 150,000-record run silently emitted its primary keys twice and every join over them fanned out; the error says which field, and that the run has to stay at or under the cap. A project with nothing to keep distinct is unaffected and still generates any number of records
+- `synth.min` / `synth.max` now also apply to a **key/ID** column — a primary key, or a field whose name reads like an identifier — which was the one place a declared width still could not reach, and the columns most likely to have one (`CHAR(10)` member IDs, fixed-width claim numbers). A key's values come from a shared pool rather than a generator, so the bounds **size the pool** instead of clamping its values: `{ "min": 10, "max": 10 }` on `member_id` generates `memberid01`, `memberid02`, … — a name-derived prefix plus a sequence number, together exactly 10 characters, with the prefix giving way to the digits when the width is tight. Clamping was not an option: cutting `member_id_10` and `member_id_100` to a width would collide them and silently break the unions and foreign keys the pool exists to keep. On a `number` key the range works the same way, taken from its start (`{ "min": 500000 }` → `500000`, `500001`, …). Because the pool is shared, two things are now rejected with an error instead of quietly producing broken data: a width too narrow to hold one distinct value per generated record (it names the width needed), and two producers declaring different bounds for the same key name. A foreign key still cannot be bounded directly — its values *are* the parent's, so bounding the parent field it references is what sets its width, and the error says so
+- `synth.min` / `synth.max` now also apply to a **`string`** dimension, where they are the value's length in characters — the one part of a generated value that was impossible to control, and the reason a synthetic run could produce data that would not load into the column it was generated for. Both ends are inclusive and either can be given alone: `{ "max": 20 }` keeps a name inside a `VARCHAR(20)`, `{ "min": 8, "max": 8 }` pins an identifier to a fixed width. On a string the bounds **clamp** the generated value instead of replacing it the way they do on a number — a value over `max` is cut to it (and trimmed, so it never ends on a space), one under `min` is filled out with letters — so a bounded `city` column keeps generating cities rather than turning into random characters. Lengths must be whole numbers (`min` at least 0, `max` at least 1), and the existing rules still hold: bounds are rejected on a dimension whose value already comes from an `allowedValues`/`codeSet`/`references` binding or a shared key pool, and now on a `boolean` dimension, rather than being silently ignored
+- A deployed worker now writes its **operational state to stdout as single-line JSON**, which is the only channel available where the worker sits inside the customer's VPC and no developer machine can reach it. Every deployed container already ships stdout to the customer's log pipeline, so this needs no new infrastructure and no inbound network access: a `{"remora":"heartbeat",...}` record every 60 seconds carrying worker version, uptime, whether the configuration actually loaded (and the error if it did not), the config signature, resource counts, CRON jobs with their next fire times, queue mappings, in-flight executions with elapsed time, and heap/RSS — plus one `{"remora":"run",...}` record at the end of every execution with its outcome, row counts, duration and per-stage timings. The discriminator makes both greppable and queryable (`filter remora = "run" and status = "failed"`) without parsing surrounding text. Two rules are enforced in the emitter rather than left to callers: **no PHI** (keys naming row content or credentials are redacted, never printed, and anything URL-shaped loses its query string, where a presigned URL carries its credentials) and **bounded payloads** (depth, array length, string length and total line size are all capped, with an oversize record replaced by an explicit marker rather than dropped silently). `REMORA_STATE_INTERVAL_MS` sets the interval (default 60000, clamped to 5s–1h) and `REMORA_STATE_ENABLED` turns emission off or on; it is off in the CLI, where JSON lines would corrupt command output. Until now a worker that failed to load its configuration kept serving and passed its health check with zero resources loaded, and `/health` — which returns the bare string `OK` — could not express that
+- A queue message that keeps failing now **stops being processed**, instead of triggering a consumer run on every redelivery until the queue quietly discards it. A message is only deleted once a consumer has handled it successfully — which is what keeps a shared queue usable — but that also meant a message that can never succeed (a malformed payload, a consumer whose source is gone) was redelivered every five minutes for the queue's whole retention period: roughly a thousand failed runs before SQS silently dropped it. `maxReceiveCount` now bounds that for a consumer's `QUEUE` trigger, and defaults to 5 failed deliveries. What happens to the exhausted message depends on the new `deadLetterQueue` setting — a queue URL, or a name resolved against the queue's own `region`/`accountId`. **With one**, the message is copied there and then deleted from the source queue, in that order, so a dead-letter send that itself fails leaves the message where it is rather than losing it; the copy carries `remoraSourceQueue`, `remoraSourceMessageId` and `remoraReceiveCount` as message attributes, so it can be diagnosed and replayed. **Without one**, the message is skipped on every later delivery without running a consumer, and nothing is deleted — deleting a message Remora did not produce is not its call — so it stays in the queue for inspection and expires on the queue's own retention period. The delivery count is SQS's own `ApproximateReceiveCount` rather than an in-memory tally, so it survives a worker restart and is shared across every worker polling the queue; the skip list is per-process and *is* cleared on restart, which gives every message another chance once a fix is deployed. A message no consumer **claimed** is not a failure and is never dead-lettered or skipped: a `messageType` matching nobody still leaves it on the queue for other systems. The worker's state heartbeat reports `quarantinedCount`, since a skipped message is otherwise invisible
+- Every execution record now carries the **Remora version that ran it** (`engineVersion`) — in the usage collection, in the CLI's local `remora/.temp/usage/remora_usage.csv`, in the worker's `{"remora":"run",...}` stdout record and in an `ops history` entry alike. Run timings were only comparable to each other while the engine stayed still: a run that got faster or slower between two releases was indistinguishable from one whose input changed, so the version has to sit on the record itself rather than be inferred from when it ran. The column is appended last in the local file, so rows written before it existed keep reading correctly and simply have it empty
+- Added `remora graph`, which prints the project's lineage — what feeds what — to stdout: `-f mermaid` (the default) for a diagram that pastes straight into a README or a PR description, `-f dot` for graphviz, `-f json` for the machine-readable form. Until now the only way to see the shape of a project was to open four JSON files side by side and hold the references in your head; the graph draws every one of them, including the three that were previously invisible even in the frontend's canvas — where a consumer's data **lands** (`outputs[].exportDestination`, and a local file or API endpoint when there is no destination source), the **chains** between consumers (`onSuccess: run-consumer`), and the **triggers** that fire them (a CRON expression, an API path, a queue). A reference to something that does not exist is drawn rather than dropped, dashed and marked `not declared`, because a dangling reference is exactly what you opened the picture to find. Alongside it the command reports, on stderr so the diagram itself stays pipeable, every problem it found across resources: a producer reading a source nobody declared, a consumer reading something that is neither a producer nor a consumer, a field reading a producer its consumer does not list, an unloaded schema, an export destination or chain target that is missing, a cycle in the consumer chain (reported on every consumer on the loop), and — as warnings — a source nothing reads or writes, a producer no consumer uses, a schema nobody validates against, a consumer with no output at all. A project with errors still renders: the picture is wanted most when the configuration is wrong
+- The CLI progress bar now moves during **staging** — the download or table read that happens before a single record is processed, and on a large source the longest part of the run. A source driver reports its own progress as a fraction of that producer's work (`onProgress` on the driver's ready request), so a multi-gigabyte S3 object or a Delta Share table read advances the bar instead of parking it while the run looks hung; a driver that cannot measure itself never reports one and behaves exactly as before. The renderer also always draws a phase boundary and a completed phase rather than throttling it away, so the bar can no longer be left showing a stale fraction of a phase that has already ended
+
+### Changed
+
+- A **broken configuration no longer stops the project from being read.** `Environment.load()` is now one policy over a new `Environment.inspect()`, which loads everything loadable and reports the rest instead of throwing: `load()` takes the first problem and throws it, so the CLI and the worker still fail fast with the same message they always did, while a read-only consumer like `remora graph` renders everything that did load. The practical difference is a file that will not parse. That used to abandon the whole batch — one consumer caught mid-save and the entire project was unreadable, with a message naming the *directory* rather than the file — so now each unparseable file is one problem naming that file, and every other file still loads. A resource that fails its schema is **kept**, not dropped, so a consumer with one malformed field still shows where its data comes from; a producer whose fixed-width layout cannot be resolved is kept unexpanded rather than unloaded. That is what makes a picture of a project available exactly when the project is wrong, which is when it is most wanted
+- **A run now shares one worker pool across every producer, file and chunk**, instead of building and tearing one down per file — which is what made a project of ordinary-sized files run single-threaded no matter how many cores the machine had. A file under 10MB is never split, so it becomes exactly one chunk and one worker, and because files were processed one at a time that meant one worker at a time for the whole run: 30 files of 6MB overlapped **0.35x**, i.e. the wall clock was every thread spawn, every configuration load and every worker's work laid end to end. Each of those spawns also loaded the 2.4MB worker bundle again, so a 30-file run paid 30 spawns serially where 9 threads would have covered it. The whole work list — every (producer, file, chunk) — is now resolved before any thread exists and submitted at once, and the pool schedules across it: **that 30-file run went from 18.7s to 2.7s (7x), overlapping 4.7x on 9 threads instead of 30 spawned one after another**, and it now finishes within 1.2x of the same data arriving as a single large file, where it used to take 8.5x longer. Nothing about how a file is split changed, and nothing about how many workers may run at once changed either: the pool is sized by the same memory-aware worker count as before (CPUs × 0.7, capped by available memory against each worker's 2000MB heap allowance), and work beyond that **queues instead of spawning threads** — so peak concurrency on a small machine is identical, 1 worker before and 1 worker after on a 4GB box. What is bounded for the first time is how much a run may queue: 100,000 chunks, with an error naming the consumer rather than a failure part-way through dispatch. Row numbering is unaffected — a `lineCount` field still numbers each source file from its own first line, even though chunks from different files now finish interleaved
+- A worker thread now **receives the loaded configuration when it starts**, instead of reading and validating the whole project from disk again for every chunk it runs. That load — walk the project directory, parse every source, producer, consumer and code set, validate each against its schema, expand every repeating group — used to be charged once per chunk, so it was the one startup cost that did not shrink when a run started sharing one pool of threads: a 30-file run paid it 30 times on 9 threads, and a machine constrained to a single thread paid all 30 of them one after another. The orchestrator now snapshots the configuration it has already loaded and hands it to each thread once, which takes the configuration load of a 30-chunk run from **1.51s of worker time down to 28ms** — and on a single-threaded machine, where none of it could be overlapped, the same run went from **7.02s to 6.40s**. The saving grows with the number of chunks, because what it removes is the part that grew with them. Independently of speed, this also means a run can no longer straddle two versions of the configuration: re-reading the project mid-run meant a configuration edited while a run was in flight was silently picked up by whichever chunks had not started yet, so different chunks of one output could be produced by different configurations
+- `REMORA_MAX_WORKERS` caps how many worker threads a run may use. It can only ever **lower** the computed count, never raise it: the count is derived from available memory against each worker's heap allowance, and raising it past that bound is precisely the oversubscription the bound exists to prevent — an 8-core/4GB machine given 7 workers each allowed 2000MB is oversubscribed by construction, which is the fatal error the memory-aware count was introduced to fix. Its two uses are capping Remora where it shares hardware, or runs in a container with less memory than its host, and making a constrained machine reproducible without owning one. A value that is not a whole number of 1 or more is rejected with an error rather than ignored
+- A **failed `API_QUEUE` command now leaves the queue** instead of being redelivered. A message on the project's API queue is a command — it asks for one consumer run — not an event to be replayed, so it is removed once it has run, whether the run succeeded or failed: it had already run by the time it failed, and a redelivery would be a second execution rather than a resumption of the first. The failures that actually happen here are not ones another delivery fixes either (a `consumer` naming something that no longer exists, a malformed payload, a run that broke halfway), and the retry that used to follow meant a single bad command ran the pipeline again every five minutes until its deliveries were exhausted. `API_QUEUE.deadLetterQueue` decides where the failed command goes: **with one**, it is copied there — carrying `remoraSourceQueue`, `remoraSourceMessageId` and `remoraReceiveCount`, so it can be diagnosed and replayed — and then deleted from the API queue, in that order, so a dead-letter send that itself fails leaves the command in place rather than losing it; **without one**, it is deleted outright and the worker's log is the only trace, which is why configuring one is worth it. `API_QUEUE.maxReceiveCount` goes with the retry it used to bound, since a command is never delivered a second time. A consumer's `QUEUE` trigger is unaffected and still retries, dead-letters and quarantines as before — the difference is exactly that a trigger message is an event several systems may react to, while an API queue message is an instruction addressed to Remora
+- `create-consumer -p <producer>` now carries the producer's documentation over to the consumer it generates: the producer's own `description` becomes the consumer's (instead of a generic `"Consumer for <producer> data"`), and each dimension's and measure's `description` becomes the corresponding field's. What a column means is written once, on the producer, and no longer has to be retyped on every consumer built from it. A producer or field with no description is left without one, as before
+- `schemas` is now **optional** in `project.json`, and a declared schema path that does not exist on disk is skipped rather than failing the load. Custom JSON schemas are only needed by a consumer that validates against one, so a project that has none no longer has to carry an empty `remora/schemas` directory to start — `compile` stops requiring the directory too
+- **A consumer's per-record work is now compiled once per worker** instead of being re-derived for every record. Mapping a record used to make five sequential passes over the field list and, inside them, a `dimensions.find` per field and a `fields.find` per dimension — O(fields × dimensions) twice, per row — plus two `delete`s that push the record object into V8's dictionary mode and cost every later property access on it, and `alias ?? key` recomputed three times per field. `RecordPlan.compile` now resolves all of that once into a flat op list with the dimensions already looked up, a plain passthrough field compiling to no op at all, and the transform and validation passes walking only the fields that declare one. `process-record` fell **3.69s → 2.62s (-29.1%)** on a single large file and **3.53s → 2.49s (-29.5%)** across many small ones, summed across workers, with row counts and output bytes unchanged. The plan reproduces the old bookkeeping symbolically rather than just listing the final keys, because the deletes were observable twice over: they set the record's key insertion order, which *is* the key order of JSON output, and they decided which keys came out null. One behaviour is deliberately not preserved — a transformation referencing a renamed-away dimension **dynamically** (`$name`, `combine_fields`, `multiplyBy`) used to throw "field absent from the record" and now resolves to the value, which can only affect a consumer that fails on every record today
+- **Output serialization is compiled once per run** as well. `outputRecord` re-derived the internal record format per record, `toJSON` recomputed whether any output sets `omitNull`, and `toCSV` allocated a filtered visible-fields array — all three run-constant — while `stringifyRow` built each line with a `map`, a template literal, an **unconditional** `replaceAll('"', '""')` and a `join`. The format, the `omitNull` decision and the visible/hidden key lists are now resolved once, and `CSVParser.escapeField` carries the escaping rule behind an `indexOf('"') === -1` fast path, with `stringifyRow` a manual loop over it. `output-record` fell **1.65s → 813ms (-50.6%)** summed across workers, byte-identical output pinned against the expression it replaces over a fixture of awkward fields (bare quote, doubled quote, quote at either edge, embedded delimiter, newline, null). JSON still deletes its hidden keys per record rather than listing the visible ones out, because a JSON record serializes in the record's own key order and an aliased field is appended — writing the field order instead would change the bytes
+- **A worker now writes its output in ~256KB batches** rather than calling `write` once per record: about **1,000,000 calls down to ~800** on a million-row run, and the throwaway `line + '\n'` string per record gone with them. The drain contract is unchanged — `write` returns false exactly when the caller must await, which is what keeps this from costing a microtask per record and giving back what the batching bought. Stated plainly: **the wall clock did not measurably move** (+0.8% and -4.9% on the two scenarios, both inside this harness's noise floor), because ~140ms of summed worker time per million rows sits below what these fixtures can resolve. What it buys is fewer syscalls and one fewer allocation per row, which scales with row count rather than with file count. The stream keeps Node's default high-water mark: raising it takes the drain count to zero, which is cosmetic — it moves the threshold rather than queueing less work — and measured slightly slower for a megabyte more buffered per worker
+- **The per-line work is compiled once per chunk too**, and where a producer needs none of it the pass is gone entirely. The list of masked and `sourceFilename` dimensions was rebuilt **on every line** — two array allocations plus a `getMask()` per dimension, each allocating its own list of allowed values to `.includes` against — and `path.basename(fileUri)` was recomputed per record on top of it, none of which changes across a whole chunk. `ProducerExecutor.compileLine` resolves one fill op per dimension that actually needs a file name or a mask, and an empty plan returns the parsed records straight out, so a producer with neither — most producers — now pays neither the loop nor the timing pair around it. `process-line` fell **-11.5%** on a single large file and **-8.2%** across many small ones, and the mask/filename operation no longer appears in the report at all. Mask validation moved with it: a masking mode Remora does not implement is now rejected at load and by `verify`, naming the dimension and the resolved value, instead of failing part-way through a run — the producer schema cannot do this because the value may be a `{VAR}` reference that only resolves at runtime. `mask: "none"` still compiles to a fill even though it hashes nothing, since it casts the value to a string and dropping it would silently unquote numbers in JSON output
+
+### Removed
+
+- Removed `filters[].sql` from a consumer. It was never read: nothing in the pipeline compiled it, pushed it to a source, or evaluated it, so a consumer whose filters were all `sql` exported its full result set while the configuration said it was filtered. Only `filters[].rule` ever did anything, and it still does — every filter must now declare one, and a filter with neither is rejected at `compile` rather than crashing mid-run. The `"sql"` filter operator goes with it, for the same reason: it was accepted by the schema and then thrown on as an unsupported operator by the only code that reads operators. Rewrite a `{ "sql": "${C.amount} > 1000" }` filter as `{ "rule": { "member": "amount", "operator": "greaterThan", "values": ["1000"] } }`
+- Removed the `mock` command. `synth` covers what it did and does it better: `mock` wrote a producer's source file from an unseeded `Math.random()`, so no two runs produced the same data and nothing generated could be replayed, while `synth` generates seeded, deterministic input with realistic values, code sets and referential integrity between producers, and runs the real pipeline on it. Generating a producer's source file on disk is the one thing that goes with it — `synth` feeds the pipeline directly, so there is no fixed-width writer any more either. Use `remora synth [consumer] -r <records> -s <seed>` instead
+
+### Fixed
+
+- **JSON schema validation now actually runs when a project loads.** `SchemaValidator.validate()` returns `{ isValid, errors, data }`, and all five call sites in the loader tested that object for truthiness — `if (!SchemaValidator.validate('producer-schema', producer))` — which is never true. Every one of the resulting throws was unreachable, so **no configuration has ever been rejected by its own JSON schema at load time**: the schemas documented a contract that nothing enforced, and a producer with no dimensions, a source missing its `engine`, or a consumer with a misspelled property loaded silently and failed later somewhere less obvious, or not at all. Validation now runs on every source, producer, consumer, code set and on `project.json` itself, at `compile`, `run`, `deploy` and on a worker's config load, and the message names what is wrong rather than only what is invalid — `Invalid producer configuration: p_orders (/dimensions must NOT have fewer than 1 items)`. Resources are checked **as authored**, before repeating groups and flag bitmaps are expanded, since expansion generates the properties the schema forbids; and where a fixed-width layout is wrong in a way both checks would catch, the expander's message wins, because "a variable group must declare a whole-number maximum" says what to do where "must have required property 'max'" sends you looking. **This can reject a project that used to load.** A configuration that is schema-invalid now fails at `compile` — which is where it should have been failing all along — and a deployed worker that reloads such a configuration keeps serving with none, exactly as it does for any other bad config. Run `remora compile` before upgrading; `remora graph` reports the same problems without failing, one per resource, if you would rather see all of them at once
+- **A filter on a `number`, `datetime` or `boolean` field now actually matches.** A filter's values are always text — they come out of a JSON config, a URL query or a queue message — while a record holds what the producer cast the column to: a `number` dimension holds a number, a `boolean` one a boolean, a `datetime` one a UTC ISO string. The record evaluator compared the two as they arrived, so `5 === "5"` and `true === "true"` were false for every row: `equals`, `notEquals`, `in` and `notIn` on any non-string field **returned nothing at all** (or, for the two negations, everything), and the four `greaterThan`/`lessThan` comparisons were hard-coded to numbers, so they dropped every row of a date or text column. Each side is now read as the type the record holds before being compared. `"150"` matches a `monthly_spend` of `150`; `"true"`, `"false"`, `"1"` and `"0"` match a boolean; an ISO date matches a `datetime` **as an instant**, so `"2024-03-01"` matches the record's `2024-03-01T00:00:00.000Z` and a bound carrying an offset (`2024-03-01T11:30+02:00`) means the moment it names rather than the characters it is written with — read as UTC where it carries no offset, so a filter means the same thing on every host. A date is only read as a date where **both** sides spell it ISO-8601, so a column in a producer's own format (`"05/03/24"`) still compares as the text it is rather than being handed to `Date` to guess at. Ordering comparisons now work on text and dates as well as numbers, which is also what the source-side predicate has always assumed. The string operators (`contains`, `startsWith`, `endsWith`, `matches`) read a non-string value's text — a number's digits, a date's ISO form — instead of failing on the type, and their negations now keep a row whose value is null, consistently with `notEquals` and `notIn`. Two things that follow, worth knowing: a date bound naming a day means that day's **first instant**, exactly as a SQL date literal does, so `between ["2024-03-01", "2024-03-31"]` stops at the start of the 31st — give a time (`"2024-03-31T23:59:59"`) to include it; and a value that cannot be read as the record's type (`equals` `"abc"` on a number) matches nothing rather than throwing. **This can change what a consumer returns**: one whose filters were silently matching no rows will now return the rows it was written to return
+- A consumer on a **SQL source can now filter at all**. `compile` rejected rule-based filters on a SQL engine ("Filters based on rules are only valid for non-SQL based sources"), leaving `filters[].sql` as the only accepted form — and that one was never evaluated, so the two rules together meant a Redshift-backed consumer either failed to compile or ran unfiltered, with no way to express a working filter. Filters are applied per record to the result set after the producers are read, which is engine-agnostic, so the engine restriction described a pushdown that does not exist. Rule filters are now valid on every source
+- Fixed day-of-year ("Julian") dates parsing to a wrong date with no error. dayjs has no parse token for a day of year at all — `DDD` exists only for formatting — so `"YYYYDDD"` either failed as invalid (strict, the default) or, with `"strict": false`, consumed the `YYYY`, ignored the rest and read `2007305` as **2007-01-05** instead of 2007-11-01: silent corruption of every ordinal date in a mainframe feed. `DDD`/`DDDD` are now parsed as a 3-digit day of year, in both a producer dimension's `format` and a consumer's `date_format` transform, and combine with `YYYY`/`YY`, `HH`, `mm`, `ss`, `SSS` and literal separators. A day of year outside its year (366 in a non-leap year) and a format mixing an ordinal day with a calendar `MM`/`DD` are both rejected with an explicit error rather than resolved to something plausible. `synth` renders the same format correctly too, zero-padded to 3 digits so what they write can be read back by the layout that declared it
+- A date format built on a token dayjs can only *write* and never read (`Do`, `Q`, `w`, `W`, `k`, `gggg`, `z`) is now rejected with an error naming the token, instead of being parsed loosely into a wrong date. Bracket it (`"[Q]YYYY"`) if it is meant as literal text
+- Fixed a CRON job's reported status always reading as *not running*. The scheduler compared the task state against `scheduled`, a state name from node-cron 2.x; the installed node-cron reports `idle`, `running`, `stopped` or `destroyed`, so `isRunning` was false for every job including the ones firing normally — in the worker's own CRON status API and anything built on it. It is now derived from the real states, and each job also reports its raw `state`, the consumer and expression it came from, and its next fire time, so a job that is scheduled but will never fire is distinguishable from one waiting to run
+- **A run's record limit is now the whole run's, not each worker's.** `remora run -l 10` (and a request's `limit`) was applied inside every worker against its own chunk, so a consumer whose producer split into nine chunks returned **ninety** rows — nine separate blocks of ten, one from each chunk's start — and the count scaled with the machine: the same command returned a different number of rows on a box with more cores, and `REMORA_MAX_WORKERS=1` was the only way to get the limit asked for. Multiple producers multiplied it the same way. Each worker still stops reading at the limit within its own chunk, which is what keeps a limited run from reading a whole table, but the run-wide cut is now made once on the merged dataset — after `distinct`/`distinctOn`/`pivot`, so a limit is never met by rows those passes were about to remove, and before dataset validation and export, so both see exactly the rows returned. The chunks are merged in source order, so the rows kept are the first `limit` rows of the result, the same ones a single-threaded run has always returned
+- **A numeric filter pushed to a Delta Share now actually prunes anything.** Every `number` dimension's literals went out typed as the protocol's `double`, but a share server only prunes when the type it is given is the column's own — hand an `integer` column a `double` and Databricks quietly ignores the hint and returns every file. So the predicate was well-formed, accepted, logged, and worth nothing: a filter on a numeric column downloaded the whole table exactly as it did before pushdown existed, while string filters (whose one type is unambiguous) pruned correctly. Each column's literals are now typed from the share's own schema, read once per table from the table `metadata` endpoint and only when there is a filter to push, so `integer` goes over as `int`, `long` as `long` and a genuine `double` as `double`. Columns the protocol's value types cannot express (`decimal`, and the nested types) and columns whose Delta type disagrees with the producer's own dimension type are left out of the tree rather than sent under a type that would make the server and remora prune different rows, and a schema that cannot be read costs the predicate and nothing else. A filter that now prunes **every** file also no longer fails the run: an empty read is still the misconfiguration error it has always been, except where the run itself asked the source to filter, in which case no rows is the answer and the consumer exports an empty result — the same thing it produces when remora applies that filter per record
+- **A fixed-width producer whose first field is space-padded no longer reads every column shifted.** Each line was `trim()`ed before parsing, but a `FIXED` dimension is a character range into the line, so leading padding — which is exactly what a right-aligned numeric in column 1 looks like — moved every position after it and each dimension read the wrong bytes. The record selector was already matched against the **untrimmed** line, so the two had disagreed all along. The trim is gone, and it was redundant everywhere else: `CSVParser.parseRow` trims each unquoted field and breaks on `\r`/`\n`, `FixedWidthParser.readField` trims each field it slices, and `JSON.parse` ignores surrounding whitespace. One line changes outcome as a result — a **whitespace-only** line in a CSV file, which used to trim to `''` and fail the entire run, now parses to a record of nulls, which is what a `FIXED` producer has always done with it
+
 ## V 1.8.0 - 2026-08-05
 
 ### Added

package/documentation/README.md

@@ -29,6 +29,7 @@ The main project configuration file.
 | `settings.STRING_MAX_CHARACTERS_LENGTH` | Maximum length for string fields | Positive integer |
 | `settings.MAX_ITEMS_IN_MEMORY` | Maximum number of items to keep in memory | Positive integer |
 | `settings.API_QUEUE.source` | Name of an `aws-sqs` source used as an alternative execute-consumer API | Must match a source `name` |
+| `settings.API_QUEUE.deadLetterQueue` | Queue a failed command is moved to instead of being deleted | Queue URL, or queue name resolved against the source's region/account |
 
 ## Source Configuration
 
@@ -99,7 +100,6 @@ Consumers transform and combine data from producers for specific use cases.
 | `fields[].from` | Producer to get the field from | Must match a producer `name` |
 | `fields[].grouping.groupingKey` | Field to group by | Any string without spaces |
 | `fields[].grouping.subFields` | Fields to include in each group | Array of field objects |
-| `filters[].sql` | SQL condition for filtering | SQL WHERE condition |
 | `filters[].rule.member` | Field to filter on | Field name |
 | `filters[].rule.operator` | Comparison operator | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `contains`, `not_contains` |
 | `filters[].rule.values` | Values to compare against | Array of strings |