@forzalabs/remora 2.6.0 → 2.7.0

package/CHANGELOG.md

@@ -1,557 +0,0 @@
-# Changelog
-
-All notable changes to this project will be documented in this file.
-
-The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
-
-## Unreleased
-
-## V 2.6.0 - 2026-10-26
-
-**Upgrading from 2.5.** Nothing in an existing project changes, and nothing that compiled under 2.5 is refused. One thing moves, and it is not opt-out: a `CSV` or `TXT` export now **quotes a field only where the quotes carry meaning**, where it used to quote every field. The values and the nulls are the same values and the same nulls — `"USR-2","free"` is now `USR-2,free` — but the bytes are not, so a destination that matches the file against a fixture, or parses it by splitting on `","` rather than with a CSV reader, needs checking before this version runs. This is the second consecutive release to move those bytes, after 2.4.0's null marker; a proper CSV reader has been unaffected by both.
-
-### Changed
-
-- **A positional field is quoted only when it needs to be.** Every field of a `CSV`/`TXT` export and of the internal staged dataset used to be wrapped in quotes unconditionally, which cost two bytes per field per row — written by each worker, concatenated by the merge, re-read by every post-processing pass, and shipped to the destination. A value is now written bare unless it holds the delimiter, a `"`, a line break, or a space at either end, and an **empty string stays quoted** because that is what keeps `""` (the empty string) apart from nothing at all (a null). The header line follows the same rule, so it still parses with the reader the data does.
-
-  ```
-  id,note,amount
-  USR-1,"",12.5        <- note is the empty string
-  USR-2,,              <- note and amount are null
-  USR-3,"a, b",12.5    <- note holds the delimiter, so it is quoted
-  USR-4,"  padded  ",0 <- note has edge spaces, so it is quoted
-  ```
-
-  **Measured -14.7% on the bytes** of a real 192MB export (201,795,101 → 172,095,071 over 990,000 records of 15 columns), which comes off every stage that reads or writes a staged file and off what the destination stores. The encode itself is 1.17x faster per row, and end to end the wall clock moves inside the harness's own ±6% noise floor — the bytes are the result here, not the throughput. A staged dataset written by an older version, where everything is quoted, still reads back identically, so nothing in a half-finished run is stranded.
-
-- **Edge spaces are quoted deliberately**, which is a rule the original specification did not have. Most CSV readers trim an unquoted field — Remora's own does — so a value of `  padded  ` written bare would come back shorter than it went out. Quoting it is what keeps the round trip exact, and it costs nothing measurable: the check rides in the same single pass that looks for the delimiter and the quote.
-
-## V 2.5.0 - 2026-10-26
-
-**Upgrading from 2.4.** Nothing in an existing project changes. `FIXED` is a new output format and every rule this release adds fires only on a consumer that declares one, so no configuration needs touching, no exported bytes move, and nothing that compiled under 2.4 is refused. The one thing to know before writing a FIXED output is what the receiver has to do with the file: it leaves Remora as UTF-8 text with `lf` or `crlf` line terminators, so a z/OS target still needs the ASCII→EBCDIC and `RECFM=FB` conversion on its own side (see Added).
-
-### Added
-
-- **`"format": "FIXED"`, a positional flat-file output.** Remora has read fixed-width files since 1.8 and could not write one — every output format it had quotes and delimits — so a customer who sends a positional feed and wants one back had no way out of the pipeline. A `FIXED` output declares a `width` per field and writes the record at those columns: no delimiter, no quotes, no header. It reaches the same destinations the other file formats do (local, S3, a pre-signed URL) with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour, because it is a per-line transform at export time and not a new write path — the dataset is staged exactly as a `TXT` one is and re-emitted positionally on its way out, so nothing in the run pipeline changed to carry it.
-
-- **Each visible field gains a `position` block** — the write-side counterpart of the producer dimension's `position`, named the same for symmetry. `width` (required) is how many characters the field occupies; `align` (`left`/`right`), `pad` and `nullFill` are the single characters it is filled out with; `overflow` is what a value too long for its field does. **Column order is field order and the offsets are computed from the widths**, so no offset is ever transcribed by hand — a `start` may be declared, but it is an assertion `compile` recomputes rather than a second source of truth, and a gap or a COBOL FILLER needs no new concept because a `fixed` field with a `width` already is one.
-
-  ```json
-  {
-      "fields": [
-          { "key": "key_serv_plan", "position": { "width": 3, "start": 1 } },
-          { "key": "key_tran_id", "position": { "width": 4, "start": 4 } },
-          { "key": "balance", "position": { "width": 11, "align": "right", "pad": "0" } }
-      ],
-      "outputs": [{
-          "format": "FIXED",
-          "exportDestination": "s_output",
-          "recordLength": 44,
-          "lineEnding": "crlf",
-          "defaults": { "number": { "align": "right", "pad": "0" } }
-      }]
-  }
-  ```
-
-- **Five fill rules, each chosen so a wrong file fails loudly instead of shipping.** A value shorter than its `width` is padded on the side `align` names. **A value longer than its width fails the run**, naming the field, the value and both widths — a truncated account number is a plausible-looking account number, and an `overflow: "truncate"` escape exists per field for the layouts that genuinely specify a cut (the end that survives follows `align`, which is what a COBOL `MOVE` does). A null is the full width in `nullFill`, **which defaults to spaces rather than to `pad`**: zero-filling an empty numeric column writes `00`, and the receiver reads that as the number zero where there was no value at all. An empty string is written as a null, because the staged encoding keeps the two apart and a positional line cannot. And every record is exactly `recordLength`, a record whose last field is null included — the line is never cut short to the last character that carries something.
-
-- **`recordLength`, `defaults` and `lineEnding` on the output.** `recordLength` is the length of the record as written, which is not always the length in the copybook — a NASCO POD 1300 is 44 characters of a 256-column layout, cut after the last field that carries anything — and `compile` fails if the widths do not sum to it. `defaults` sets `align`/`pad`/`nullFill`/`overflow` once per dimension type (`string`, `number`, `datetime`, `boolean`) so a 90-field layout states "numbers are right-aligned and zero-padded" once and names only the exceptions, with a field's own `position` winning where both set the same rule. `lineEnding` is `lf` (the default) or `crlf`, because a real positional feed picks one terminator and holds it on every line including the last.
-
-- **`layouts` — several record layouts in one file.** A positional feed is one file even when it carries three kinds of record, and shipping one file per record type is not the same file. The output names a `discriminator` field, lists its values in `order`, and gives each one a field list and a `recordLength` of its own; the consumer declares the union of every layout's fields and each layout names the ones its records carry. This is the write-side mirror of the producer's `recordSelector`. **The records come out grouped, one contiguous block per value in `order`, keeping the order they were read in within each block** — which is how a real feed is written. The grouping is a two-pass reorder over the staged file (route each line to its block, then append the blocks in order), so it costs one extra copy of the dataset on disk rather than one per block, and `CONSTANTS.defaults.MAX_FIXED_LAYOUTS` (64) bounds how many blocks one output may open at once.
-
-- **Four kinds of layout error are caught before a producer is read, and `remora compile` prints the layout it resolved.** A missing `width`, a `start` that disagrees with the computed offset, widths that do not sum to `recordLength`, a layout field or discriminator that is not one of the consumer's visible fields — all of them are configuration, so all of them are resolved by the same code the export runs and reported together, every problem in the output named in one `compile` rather than one per run. The same check runs again at the start of a run, so a width that does not add up fails before the first producer is read instead of after the whole dataset has been built. `compile` then prints each layout as a table — field, start, end, width, in the order the record is written — which is the whole point of declaring widths instead of columns: the author reads the start columns back and checks them against the spec once, rather than counting characters across 90 fields to find the filler they forgot. `run` and `verify` compile too and stay quiet, since a 90-field table is noise in front of a run.
-
-- **A FIXED output refuses what it has no room for, rather than ignoring it.** `header` is rejected — a field name does not sit on the columns it describes, and the receiver reads by position anyway. `delimiter` is rejected, because a positional line has none. The wildcard field key `"*"` is rejected, since every field of a positional line declares its own width and a wildcard expands to columns nobody has written one for. `omitNull` stays rejected as on every positional format. And the four FIXED-only output properties plus a field `position` are all rejected when the format is something else, the way a `position` on a non-FIXED producer is: silently dropping a rule the author wrote is how a file ships looking almost right.
-
-- **The bar is a byte-identical round trip, and it is a canary stage.** A three-layout positional file is read by three producers, exported as JSON, read back and written out again as one `FIXED` file, and the stage fails on any difference at all — a width the writer gets wrong is a width the reader also got wrong, and only a byte comparison catches the pair. It runs over a committed fixture and over any real batch `POD_SAMPLE_DIR` names.
-
-- **What the receiver has to do is documented rather than assumed.** The file leaves Remora as UTF-8 text, one record per line, one character per column. A z/OS target needs ASCII→EBCDIC and reframing to `RECFM=FB` on the customer's side — the mirror of the conversion job a mainframe feed already requires inbound, and the same shops run it in that direction — and a non-ASCII character leaves the record `recordLength` characters but the line no longer `recordLength` bytes. Implied decimals and overpunch signs are not written: POD-class layouts are `PIC X` and unsigned `9` throughout, and a feed that needs scale or sign needs the read-side work first. [Remora - Consumers.md](docs/public_docs/Remora%20-%20Consumers.md) carries all of it under **FIXED Output**.
-
-- **The full prose reference now ships with the CLI.** `docs/public_docs` is copied into the published package, so the four reference documents (Sources, Producers, Consumers, Code Sets) sit on disk beside the installed version — a user, or an AI assistant working in their project, reads the reference for the release they actually have rather than fetching a copy that may describe a different one.
-
-### Changed
-
-- **The README `remora init` drops into a new project was rewritten.** It used to mirror the JSON schemas key by key, which rotted: the schemas moved and the tables did not. It is now an orientation document — how the three resource types chain together, what `init` just created, the commands, the environment variables, and where the full reference lives — with enum lists but no key-by-key tables, so the inline contract has exactly one home. It also has a section written for an AI assistant working in the project folder.
-
-## V 2.4.0 - 2026-10-19
-
-**Upgrading from 2.3.** Two configurations that compiled under 2.3 are now refused, and both fail at `compile` with the offending resource named rather than at run time. A producer carrying `measures` must have them deleted — the key is gone from the schema and a producer rejects unknown keys (nothing is lost: no stage of the run path ever evaluated one, see Removed). A consumer declaring **more than one** `outputs` entry must keep the one it actually used — a second entry was never exported, so the one to keep is the first (see Changed). Two further changes alter the bytes an existing project produces, neither of them opt-out: a `CSV`/`TXT` export now writes a null as an unquoted empty field (`"USR-2",,`) where it used to write `""`, and a value containing a newline is no longer split across two records. A destination that reads `''` where a null is now written, or that relied on the old split rows, needs checking before this version runs.
-
-### Added
-
-- **`"format": "PARQUET"`, a fourth file output.** It writes an Apache Parquet file — one typed column per visible field, nulls preserved, SNAPPY compressed — to any file destination the other formats reach (local, S3, a pre-signed URL), with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour as every other file output. Parquet is what a lake expects to read, and it gives a reader two things a `CSV` export cannot: the file **declares its types**, so nothing downstream guesses them back out of the text, and it is a fraction of the size — a four-column, 2-million-row dataset measures 28 MB as parquet against 115 MB as quoted CSV. The schema is taken from what the consumer already declares, nothing new to configure: `string` -> `STRING`, `number` -> `DOUBLE` (every value remora holds is a double, and an `INT64` would promise a reader precision the record never had), `boolean` -> `BOOLEAN`, `datetime` -> `TIMESTAMP`, and a field with no dimension behind it — a `fixed`, `lineCount` or `copyFrom` field, or a column a pattern key matched — is a string, because there is no declared type to read. Every column is nullable, so **a null stays a null** instead of becoming `0`, `false` or `""`: in a column store those are four different things. A `hidden` field is absent from the file entirely, not an empty column in it. An unparseable value under a `datetime` dimension **fails the run**, naming the field and the value, rather than writing a timestamp column quietly full of 1970.
-
-- Parquet is the one output that is **not** a line-for-line pass. It is a columnar container whose footer can only be written once the last row group is encoded, so there is no per-line mapping to hand a destination driver: the file is built in the run's scope folder and then handed over as bytes with no header, through the `move` that is already byte-faithful in all three destination drivers. **No driver method was added and nothing in the driver layer changed.** The writer is `hyparquet-writer`, the companion of the `hyparquet` reader the Delta Share driver already depends on — same author, MIT, one dependency, and reached by dynamic `import()` exactly as that driver reaches the reader. Rows are pulled from an async generator one row group at a time, so **peak memory is bounded by a row group, not by the dataset**, which is the property the whole pipeline is built on; row groups are sized from a budget of *cells* rather than a row count, since 100,000 rows is 400k values on a four-column consumer and 130 million on a 1,300-field one.
-
-- Two flags behave differently on it, both rejected at compile rather than ignored. `header` is **rejected**: a parquet file carries its column names in its own schema, and a header line would be a first row of data claiming to be one. `omitNull` stays rejected as on every positional format — a parquet row group declares every column up front, and a null is written as a null in it. `delimiter` means nothing to it and is ignored, as it is for every format but `TXT`. One caveat is documented rather than worked around: on a split parquet output, **`split.maxFileSizeGb` caps the internal staged bytes, not the parquet bytes**, so the exported files land comfortably under the cap rather than at it — the cap exists to bound what a destination is asked to take in one object, and rotating a file after it was compressed would mean rewriting it. A `PARQUET` output is declared alone on its consumer, as every output now is (see Changed).
-
-### Changed
-
-- **A `CSV` or `TXT` export now distinguishes a null from an empty string.** Every value in those files is quoted, which left the unquoted empty field free to mean something — and it now means **null**, while `""` means the empty string. The two have been the identical two bytes for as long as the format existed, so a column that was genuinely empty and one that held no value at all arrived at the destination indistinguishable, and nothing downstream could get the difference back. **This changes the bytes of every existing CSV/TXT export that carries a null**: where a row used to read `"USR-2","",""` it now reads `"USR-2",,`. It is the shape most readers already assume — `COPY`, `LOAD DATA` and a pandas `read_csv` all treat an unquoted empty field as a null and a quoted one as the empty string — but a destination that reads the two the same way will now see a null where it saw `''`. A field a dimension declares `nullable: false` is unaffected: it carries its type's own empty value (`"0"`, `"false"`, `""`) as it always did, which is a value and so is quoted
-
-- **A value holding a newline no longer becomes two records.** A line of a remora dataset *is* a record — every pass reads it with a line reader and the record limit counts rows by counting newline bytes — so a value carrying a `\n` used to split the row in two, silently, and every stage after it was reading garbage: the wrong column count, a `distinct` that compared halves of rows, a split that bucketed on whatever landed under the key. Such a value is now written as the two characters `\n` (and a carriage return as `\r`) and the record stays whole. The escape is deliberately **one-way** — nothing turns it back into a newline — because a reversible escape would have to escape the backslash too, and that would rewrite every ordinary `C:\path` in the file. A consumer that must carry an embedded newline verbatim has `JSON` and `PARQUET`, whose encodings hold one natively
-
-- **A consumer declares exactly one output.** `outputs` stays an array — nothing in a config file moves, and lineage, the bug report and `ui-preview` keep iterating it — but a second entry is now refused by the schema (`/outputs must NOT have more than 1 items`) and named by the validator. This is a **cap on something that never worked**, not a feature withdrawn: a run stages its records in ONE encoding, chosen from `outputs[0]`, and then *moves* that staged file to the destination, which unlinks it; the export loop also carried its `return` inside the loop body. So a consumer declaring two outputs has always written the first and silently dropped the rest, whatever the second one said. Failing at `compile` tells its author something they need to know rather than taking a working feature away, and so it rejects from this release with no deprecation window. **Nothing in this repository is affected** — of the 140 consumer configs across the canary, the stress and parallel projects, the demos and the apps' own projects, every single one declares exactly one output. To write the same rows twice, declare a second consumer over the same producers. What the cap removed with it: the duplicate-format check, the half of the `omitNull` validation that existed only because two outputs shared one serialized record (`omitNull` on a `JSON` output is no longer rejected for having a `CSV` sibling — there are no siblings), and the "which of the outputs decides the internal format" question, which is now correct by construction
-
-### Fixed
-
-- **A `hidden` field shifted every column after it, for four passes that read the dataset back.** A staged dataset holds the VISIBLE fields only, and it carries no header — but `distinctOn`, `split.by`, the `unique` dataset validation and the `pivot` discovery each worked the column list out for themselves from the consumer's full field list, hidden fields included. So on any consumer with a hidden field, each of them read every column after it off by one: `distinctOn` deduplicated on the wrong column, `split.by` bucketed files by it, and `unique` reported duplicates in a field it had never actually looked at. Nothing failed, and the output looked plausible. The run now compiles **one** description of what is on a line — the columns, in line order, with the type behind each one — hands it to every writer and every reader, and `LineParser` takes it as a required parameter with no overload that accepts a field list, so the derivation that was wrong is not a check someone has to remember but a parameter that no longer exists. The same description travels to a `code_lookup` table and to a consumer-as-input capture, both of which re-derived it too
-
-- **A `pivot` changed the shape of the dataset and nothing downstream was told.** A pivot replaces the record's columns with its row keys plus one column per pivot value — and the CSV header, the split pass, the dataset-level validations and the parquet schema all still described the record as it was BEFORE the pivot. A pivoted `CSV` export therefore carried a header naming columns that were no longer in the file, and a split or a validation over a pivoted consumer addressed columns by the pre-pivot positions. The pivot now returns the shape it wrote, and everything after it reads that
-
-- **A value re-read by a post-pass came back as text.** Once a record had been written to the dataset, any pass that read it again (`distinctOn`, `pivot`, the `unique` validation, a `code_lookup` table, a split key, the consumer-as-input capture) got a string for a column whose dimension declared a `number`, a `boolean` or a `datetime` unless the field happened to resolve its dimension — so a numeric `distinctOn` with a `max` resolution compared `"9"` against `"10"` and a lookup table keyed on a number held string keys. Each column of the dataset now carries the three properties a cast reads — `type`, `format` and `nullable` — so a value is cast back to what it was declared as, without anything going near the consumer config
-
-- **Every boolean read from a text source was `true`.** The cast was `Boolean(value)`, and `Boolean('false')` is `true` — as is `Boolean('0')`, `Boolean('no')` and `Boolean('N')`. Every non-empty string is. So a `boolean` dimension over a CSV, a Delta share, a SQL extract or a re-read of the staged dataset reported `true` on every row that held anything at all, and the column was worthless without anyone being told. A boolean arriving as text is now matched against the spellings a source actually writes (`1`, `true`, `yes`, `y`, `t`, case-insensitively) and is `false` otherwise; a real boolean and a missing value are unchanged. **This changes what such a consumer produces** — a column that was uniformly `true` will now hold the falses it always had
-
-- **`pivot` never ran on a source under 10 MB.** `distinct` and `distinctOn` have a worker-side counterpart and are redone over the merged file only when there was more than one chunk, so the whole post-pass block sat behind `executorResults.length > 1` — and `pivot`, which has no worker-side counterpart and is the only pass that MUST run there, inherited that gate. A consumer with a `pivot` over a source below `MIN_FILE_SIZE_FOR_PARALLEL` (10 MB) therefore exported its rows unpivoted, with no warning and no error: the configuration asked for a shape and got a different one. `pivot` now runs whenever it is configured. **This changes what such a consumer produces** — the output gains the pivoted columns it always asked for and loses the raw ones — so anything downstream that adapted to the unpivoted shape needs looking at. Nothing changes for a consumer whose source was already over the threshold
-
-- **A `lineCount` field was renumbered against a file that no longer matched it.** The pass attributes line *i* of the merged dataset to the worker that wrote it by walking each chunk's output count in order, and adds that chunk's row offset — which only holds while the merged file still carries exactly `sum(outputCount)` lines in worker order. It ran LAST, after `distinct`, `distinctOn` and `pivot` had rewritten the file and changed both the count and the order, so on any consumer combining one of those with a `lineCount` field the offsets landed on the wrong rows. It now runs first, which is also the right meaning: a row that survives `distinct` keeps the line number it had in its source file. (Recounting afterwards would not have fixed it — that renumbers the survivors 1..N, which is a different thing, and a pivoted row has no source line at all)
-
-- **Two producers feeding one consumer serialized the same record into two different byte strings.** A `JSON`/`API` record was written out in the order its keys happened to be inserted, which is the order the PRODUCER's dimensions were read — the source's layout, not the consumer's. So the same values under the same names came out as `{"id":7,"country":"IT"}` from one producer and `{"country":"IT","id":7}` from the other, and both `distinct` and the `no_duplicates` dataset validation, which compare the serialized line, saw two different rows. Records are now written in the consumer's own field order, which is one order for every writer in the run — and is the order the fields are declared in, so a `JSON` export reads top to bottom like the consumer does. `remora sample` loses the re-ordering it did at the surface to work around this, because there is nothing left to work around
-
-- **A clean exit deleted a concurrent run's temp folder.** The shutdown handler cleared "open" executor scopes by listing the directories under the temp root — which cannot tell this process's leftovers from another process's live run — and it ran on every exit, not only an unexpected one. Two remora processes sharing one `REMORA_DATA_PATH` (a docker worker beside a CLI run, two scheduled runs, the backend beside anything) were enough: whichever finished first deleted the other's dataset mid-run, and the survivor then failed on a file that had vanished under it. `ExecutorScope` now registers the scopes **this process** opened and the cleanup clears only those, which also makes it a true no-op on a clean exit, since a run that finished already cleared its own scope. The directory listing stays for what it is actually good for — watching folders come and go, which the parallel canary does — and deletes nothing
-
-- An export to a destination that is not a source of the project now says so, naming the consumer and the destination, instead of failing as a `TypeError` from inside the driver factory. The two unreachable branches of the export switch are gone with it (`CSV` and `JSON` both match the staged format and are byte-`move`d, so neither could ever reach the re-encoding path), and an `API` output no longer falls into a branch that parsed a JSON line as CSV — it stages as JSON and ships as JSON
-
-- Bumped `proxy-addr` to 2.0.8 (with its `ipaddr.js` dependency) through the lockfile. It is a transitive dependency of `express`, reached by the backend and worker servers and by the `ui-preview` host, and is not a package remora depends on directly.
-
-### Removed
-
-- **`measures` on a producer.** A producer measure was a `{ name, description, sql }` triple meant to define a calculated metric, and nothing ever calculated one. `ResourcesUtils.getAvailableColumns` exposed each measure as a column a consumer could select, `ConsumerManager` carried it through field expansion as `ICompiledColumn.measure`, and there the trail ended: no stage of the run path ever read `measure.sql`, so the expression was never sent to a source, never evaluated here, and never produced a value. What it did produce was a column that passed `compile` and then exported nothing — a consumer naming a measure validated cleanly and wrote an empty field for it, which is worse than the field not existing, because the configuration says it should be there. Not one producer in this repository declared a measure across all 88 config files that mentioned the property: every one carried the inert `"measures": []` the old templates emitted. The property is now gone from `IProducer`, from `IProducerMeasure`, from `ICompiledColumn`/`ICompiledConsumerField`, from the JSON schema, from the field-lineage chain and the `ui-preview` Measures table, from `create-consumer` (which used to map measures into the generated consumer's fields), from the frontend's producer view, and from the project README and the public Producers doc. Aggregation was never a producer's job — a producer declares what it reads, and a consumer is where values are combined. **Because producer configs are `additionalProperties: false`, this moves the refusal to load time and makes it total**: a project still carrying `measures` — including the empty `"measures": []` almost every generated producer has — now fails `remora compile`/`verify` with `Invalid producer configuration: p_x (remora/producers/p_x.json) - / has unknown property "measures"` instead of loading and ignoring it. Delete the property; it was doing nothing. If the metric it named is one you want, it belongs on the consumer that reads the producer
-
-## V 2.3.0 - 2026-10-15
-
-**Upgrading from 2.2.** One change alters what an existing project produces: a `CSV` or `TXT` output now writes a header line, so every exported file of that kind gains a first line naming its columns. It is opt-out — `"header": false` on the output restores the old headerless shape — but it is on by default, so a destination that reads the first line as data needs that flag set before this version runs. Nothing else here touches an existing configuration: `remora report` is a new command, and the `.env` refusal only fires on an archive that should never have contained one.
-
-### Added
-
-- **`remora report [run-id]`, which packages a run into a `.zip` someone else can read.** A bug report is worth exactly what the reproduction is worth, so the bundle is designed backwards from what a reproduction needs rather than forwards from what is easy to collect: the usage row for the run (counts, per-stage timings, the error), that run's slice of the log file, the project configuration with its secrets removed, the environment it ran in, a one-page `report.md` that states the remora version, the node and platform, the consumer and its producers with the engine behind each source, the failure and its cause chain — and the commands that run the failing pipeline again on generated input. It defaults to the last recorded run and takes a run id for any other. **The bundle contains no executable**: the reproduction is a fenced block inside `report.md`, to be pasted or saved, rather than a `repro.sh` — an archive carrying a shell script is what a mail gateway or an endpoint scanner quarantines, and a bundle that never reaches the person who needs it reproduces nothing. Those commands bootstrap the two things a config carries from the machine it came from: a placeholder `.env` for every `{ENV_VAR}` the configuration names (`compile` refuses to run while a reference is unset, and placeholders are enough because `synth` contacts no source), and the local directories the sources point at (a missing destination fails the export AFTER the run, which reads as a second bug and is not one). Unpack, paste, and the reported failure happens again on synthetic rows. **It never compiles the project first**, unlike every other command: a bug report is wanted most when the configuration does NOT load, so the project is read leniently with `Environment.inspect()` and what will not load is collected as a finding rather than as a reason to refuse. For the same reason **no section can fail the command** — no run history, no log file, no `remora` directory at all — each one is recorded absent in the manifest with the reason, and the rest is still written; a reporter that dies on a broken project is useless precisely where it is needed. The log is sliced by run id AND by time window, because only the orchestrator's own lines carry `[usageId]` and a driver or parser line can be placed by its timestamp alone; a run whose process died before recording a finish leaves the window open to the end of the file, bounded by a record cap. Where `MONGO_URI` is set there is no local history to read at all (`UsageManager.recordsLocally()` is false), so the command degrades to the configuration, the environment and the tail of the log, and says so in the bundle
-
-- **The configuration travels as authored; the log is scrubbed of credential shapes.** The asymmetry is deliberate. A remora config is not supposed to HOLD a credential — it names one (`{PG_PASSWORD}`, resolved by `SecretManager` at the point of use) and every authentication field already accepts a reference — so `config/` is shipped byte for byte: what a dev reads there is what runs, which is the whole reason it is in the bundle, and the reproduction copies it straight into a project directory. A literal password in a source file is an authoring mistake, and rewriting it on the way out would only hide it from the one person positioned to fix it. A log is the opposite case and gets no such assumption: by the time a value reaches a log line it has been resolved, so `SecretRedactor` removes the shapes a credential has in free text — a JWT, an AWS access key id, the query string of any URL (a presigned URL carries its signature there), and `password=`/`"apiKey":`/`Authorization: Bearer` pairs — while keeping `{ENV_VAR}` placeholders verbatim, since which variable a source reads is part of the question being asked. Nothing else travels either way: only `.json` configuration files are carried, and no row of customer data is collected at any level this version can produce
-
-### Changed
-
-- **A `CSV` or `TXT` export now writes a header line.** A positional file carries values and nothing that names them, so every exported `CSV`/`TXT` file now opens with its field names — the exported ones, a field's `alias` where it has one and its `key` otherwise, in column order, quoted and delimited exactly as the rows under them so the header parses with the same reader the data does. A `hidden` field is not written and so is not named, and with a `split` **every** file of the output carries the header rather than only the first: a file that cannot be read on its own is not a file anyone can be handed. **This changes the bytes every existing CSV/TXT consumer produces**, and a destination that reads the first line as data will read the header as a row — set `"header": false` on that output to keep the old shape. The flag is `CSV`/`TXT` only: on a `JSON` or `API` output, where each record already carries its own field names, it is rejected at compile rather than ignored, so a config that asks for a header always gets one. Nothing internal changed with it — the header is composed and written at export time, not held in the dataset, so the split pass, the consumer-as-input capture, lookups and `sample` all still read a dataset of records and only records. It costs no pass over the data either: the destination drivers write the line ahead of the file they were already copying or streaming
-
-- **`remora package` and `remora deploy` now refuse an archive containing a `.env` file.** A project keeps its `.env` above `./remora` and so out of reach of the packager's walk — but that is a property of where someone put a file, and this archive is uploaded to a worker and is now also packaged into bug reports. The check makes it a property of the packager instead, with a message saying to move the file. `ConfigPackager` moved from the CLI into `packages/common` on the way, so the deployment archive and the bug report share one set of exclusions rather than two that can drift
-
-- `EnvironmentVariables` is now derived from a runtime `ENVIRONMENT_VARIABLES` list rather than written beside one, so a reader that has to enumerate the variables — the bug report's presence map, which records which are SET and never what they hold — gets exactly the names the type allows. `REMORA_MASK_IN_DEV` joins the catalogue, which the docs already named and the type did not
-
-## V 2.2.1 - 2026-10-15
-
-### Fixed
-
-- **A compressed source file that could not be read failed the run with `unexpected end of file` and nothing else.** zlib reports WHAT went wrong and never WHICH file it was reading, and nothing between it and the CLI added the name back, so a producer staging hundreds of files through a `%` fileKey died with a six-word string that named no file, no producer and no consumer stage — the one thing needed to go and look at it. Worse, that one string covers three different mistakes: a truncated archive, a **zero-byte** file, and (as `incorrect header check`) a file that is not compressed at all. The first two are what a drop folder produces constantly, since a file being written into one exists at zero bytes before it holds anything. Every decompression failure is now re-thrown naming the file and the declared format — `Failed to decompress "/data/claims/claimline_2026_03.csv.gz" as GZ: unexpected end of file. The archive ends mid-stream, so it is truncated...` — and the two cases that are decidable before any decompressor runs are refused ahead of it by their own name: an empty file says it is empty and says that a writer should stage under a temporary name and rename into place, and a file whose first bytes are not the declared format's signature (`1f 8b` for GZ, `50 4b` for ZIP) says it is not that archive whatever its name says. TAR has no signature at offset 0, so it is left to the extractor. ZIP and TAR failures gain the same naming, since yauzl and tar are no more forthcoming than zlib. An unsupported `compressionType` is also now refused as a precondition, naming the three that exist, instead of from inside the switch
-
-- **The file log sink could be ended while the records it had accepted were still in flight.** A winston logger is a Transform piped into its transports, and `Logger._final` ends every transport the moment the logger's writable side closes — while records already accepted sit in its readable buffer waiting for the transport to have room. The transport then drains, the pipe resumes on that drain, and the next record is written into a stream that has ended: `ERR_STREAM_WRITE_AFTER_END`, raised from a stream with no line of remora's in its stack. Re-pointing the sink and closing it now drain the in-flight records into the transports first, and a re-point releases the old logger only after the new one is serving, so nothing can be routed to a logger on its way out. `flush()` also writes ONE sentinel record rather than one per transport — every transport sees every record, so a sentinel each only wrote N blank lines into the same file
-
-## V 2.2.0 - 2026-10-15
-
-**Upgrading from 2.1.** Three changes can alter what an existing project does, despite the minor version. `nullable` on a producer dimension is no longer documentation: it defaults to `true` and keeps a missing value as `null` as before, but a dimension that declares `"nullable": false` now gets the type's own empty value (`""`, `0`, `false`) where it used to get `null`. A `NULL` in a `number` column now stages as `null` rather than as `0`, so a consumer that was summing or filtering over those zeros sees different figures — the fix is in the staging of every CSV-staged source (SQL, Delta Share, local and S3 CSV). And `settings.sql` on a producer is gone from the schema, so a producer still carrying it fails `compile` — the property never did anything; delete it. Also worth a glance: `remora package` writes `./remora.zip` instead of `./remora-config.zip` when no `-o` is given, and the resource folder lists in `project.json` (`sources`, `producers`, `consumers`, `schemas`, `codeSets`) are ignored — the folders are fixed. They still validate, so nothing breaks, but a project pointing them somewhere else was already not being read that way.
-
-### Added
-
-- **`split` on a consumer: the output broken into several files by a field value, by a size cap, or by both.** `"split": { "by": ["country"], "maxFileSizeGb": 2 }` writes one file per country and rotates each of them at 2 Gb. Either property stands on its own — `by` alone never rotates, `maxFileSizeGb` alone is the rotation that has always existed — because they are one operation and not two: every line of the merged dataset is routed to a bucket and each bucket owns a file, so the old `splitBySize` is the degenerate case of the new pass with a single bucket, and it is deleted rather than extended. The order is forced, key first and then size within each key, since rotating across mixed keys would only produce files that need splitting again. The file name carries a label — the sanitized key value, the rotation index, or both joined with an underscore: `sales_Italy.csv`, `sales_0.csv`, `sales_Italy_1.csv` — and an unsplit output keeps the `_0` it has always had, so no existing export is renamed. **Null and empty each get their own bucket** (`_null`, `_empty`) rather than a shared catch-all, which would merge two different things invisibly; a value that cannot go in a file name is sanitized, and two values that sanitize alike are kept apart by a numeric suffix rather than written into one file. Two bounds keep a mis-chosen key from taking the host down: at most 1000 files per consumer (`MAX_SPLIT_FILES` — "split by customer_id" fails the run with a named error instead of exhausting the descriptor table), and at most 64 streams open at once (`MAX_OPEN_SPLIT_STREAMS`), the least recently written bucket being closed and reopened in append mode when its next line arrives — which works because each bucket's byte counter lives in the bucket and not in its stream. Rejected at compile rather than at run time: a `split.by` key the consumer does not have, a hidden key on a positional (`CSV`/`TXT`) output whose file never carries that column, a cap that is not greater than zero, a split on an `API` output (a response is not a file), and a `split` block that sets neither property. `maximumFileSize` keeps working as the shorthand for `split.maxFileSizeGb`; setting both spellings on one consumer is rejected, naming the migration
-
-- **`remora discover <producer> --replace`**, which rewrites a producer's dimensions from the source instead of merging into them: a dimension the source has no field for is dropped, and a declared type the source disagrees with is overwritten. It is the deliberate way back to what discovery used to do to every producer by default (see Fixed), and it exists for one situation — bootstrapping a producer created by `remora create-producer`, whose template ships five placeholder dimensions that nothing else removes. It is also the only way to delete a dimension whose column is gone upstream. Dimensions no discovery could speak for are kept even here: a `sourceFilename` dimension holds the name of the file a row was read from, not a field of the data
-
-- **A consumer's filters and the run's record limit now reach a SQL source as a `WHERE` and a `LIMIT`.** Redshift and Postgres both build their staging statement through the same `SqlQueryBuilder`, so both now stage `SELECT "id", "country" FROM "public"."accounts" WHERE "country" IN ('Italy', 'Spain') LIMIT 100` where they used to read the table whole and discard the excess here — which on a wide table is the difference between a few hundred rows over the wire and all of them. The filters are the consumer's own and the ones an API or queue request carries; the limit is `remora run -l` or `limit` on a request, under the rule the Delta Share hint already used (`PushdownEngine.resolveLimit`: no filters, `distinct`, `distinctOn`, `pivot`, `"onFail": "skip"` validation, `recordSelector` or row-fanning group, since each one can drop a record a bounded read would have to make up for). **Unlike the Delta Share hints this is exact** — whatever the predicate prunes is gone, where a share server may ignore a hint and hand back everything — so the whole design is one-directional: the clause may be WIDER than the per-record filter and must never be narrower, and anything that cannot be guaranteed that way is left out of the statement and applied per record exactly as before. What that rules out, deliberately: a `datetime` column, where an ISO bound names one instant to the client and another to the server depending on how the value was parsed and what the session timezone is; a `boolean` one, which `TypeCaster` reads back through `Boolean(value)` and so reads a staged "false" as TRUE; every negation and null check, which a missing value satisfies; the regex operators, since a JS regular expression is not a POSIX one; a filter value spelled like an ISO date on a `string` column, which the record evaluator compares as an instant and the server as text; and a value carrying a backslash or a control character. What is pushed: `equals`, `in`, `contains`, `startsWith` and `endsWith` on text, and `equals`, `in`, `between` and the four comparisons on numbers — the `%` and `_` a LIKE pattern may pick up from a value only ever match MORE than the literal they stand for, which is the safe direction. One check has no counterpart in the Delta Share path: a filter is pushed only once the SOURCE's own catalog agrees with the dimension's declared type, read from `information_schema` once per producer and only when there is a filter to type, because a producer may declare a `timestamptz` as a `string` and the statement comparing it as one either prunes the wrong rows or fails the staging read outright. A catalog that cannot be read — a Redshift external (Spectrum) table, a late-binding view — costs the pushdown and nothing else. As before, a filter on a field whose value is touched between the source and the filter (a transform, a default, a mask, a validation) is never pushed at all, and `"disablePushdown": true` on the producer turns the lot off. One visible consequence of the limit being exact: `remora run c_big -l 100` against a SQL source now stages 100 rows, so it exports at most 100 — where the run's own limit is enforced per worker and could emit up to `100 x workers`
-
-- **A `presigned-url` engine, usable as a source and as a destination.** It covers the case where the bucket is reachable only through a URL something else signs — typically a Lambda behind API Gateway — so remora needs no AWS credentials for it at all: a signed URL authenticates itself. Every transfer is two calls, mint then transfer. The mint call is an ordinary authenticated HTTP request (`oidc` for an M2M-protected gateway, or `bearer-token`, `api-key`, `username-password`, `none`; `oidc` reuses the same cached, auto-refreshing `OidcTokenProvider` delta-share uses, so the token is fetched per process rather than per file), and the signed URL that comes back is transferred with no credentials whatsoever. Where the URL sits in the response is configuration, not convention: `signedUrlField` takes a dot path, `fileNameParam` names the field the file name travels in — body on a `POST`/`PUT`/`PATCH`, query parameter otherwise — and `signedHeadersField` carries headers the signature covers, which must be echoed verbatim or S3 answers `403 SignatureDoesNotMatch`. As a source the file is staged locally by `ready()` and parsed by the producer's `fileType` like any other file; as a destination `move`/`transformAndMove` PUT it. **The URL is minted per file, immediately before its transfer** — the reason the endpoint is configured rather than the URL: a URL in a config expires, and addresses one object, whereas a consumer rotating by `maximumFileSize` writes several and each now gets its own. Four limits are deliberate and refused or documented rather than worked around: a `%` pattern in `fileKey` is refused outright (the endpoint signs one named object and there is nothing on this side to list the rest); a single object cannot exceed S3's 5 GB limit for one `PUT`, which `maximumFileSize` rotation sidesteps by shipping several objects; there is no resume, so a dropped connection restarts the file; and `init()` deliberately does not probe the way the `http-api` driver does, because minting is not a free health check — it would leave a live signed URL behind every time `verify` ran. The transfers carry no timeout either, since a clock does not measure whether a multi-gigabyte download is healthy; the mint call, a request/response, keeps one. Signed URLs are never logged whole — only origin and path, the same rule `StateEmitter` applies, since the signature in the query string is a working credential
-
-### Changed
-
-- **Output rotation now counts the bytes it WRITES, not the bytes it has read.** `splitBySize` compared `readStream.bytesRead` against the cap, which is the reader's position in the source and includes the 64 KB the stream reads ahead — so the first rotation fired as soon as the read-ahead crossed the limit rather than when the file it was writing reached it, and on a small cap that was effectively the first line. The split pass counts each bucket's own written bytes, which it has to: with a key split there is one reader and many files, and the reader's position says nothing about any of them. A consumer setting a cap therefore gets fewer, fuller files than before — the same data, divided at different boundaries. The export also stops rediscovering those files with a `readdirSync` of the scope folder and taking their order from the directory: the split pass returns the files it wrote, each with the label that names it at the destination
-
-- **`nullable` on a producer dimension now decides what a MISSING value becomes**, where it used to be documentation and nothing else. It defaults to `true`, so a missing value — `null`, `undefined`, or the empty field a SQL `NULL` is staged as — is kept as `null`; a dimension declaring `nullable: false` asks for the empty value of its own type instead (`''` for `string`, `0` for `number`, `false` for `boolean`). A `datetime` has no empty value to give — any instant invented here would be a real date no reader could tell from one the source sent — so it stays `null` whatever is declared. An empty string IS a value for a `string` dimension and is never read as missing, which is also why nothing changes for the string columns of an existing project. The flag is read in exactly one place, `TypeCaster.castDimension`, which is now the single entry point for giving a declared dimension's value its type: the file parsers (CSV, JSON/JSONL, fixed-width), the in-memory read a SQL query returns, and the re-read of remora's own CSV working dataset all go through it. To reject a missing value rather than fill it, use a consumer field `required` validation, as before
-
-- The three stream-to-lines readers (delimited, Excel, XML) that lived on `S3SourceDriver` are now a shared `StreamReader`, which `S3SourceDriver._get` and the new presigned source both call. They never touched S3 state; keeping the file-type switch in one place is what lets a new transport driver be the fetch and nothing else
-
-- **The configuration archive is called `remora.zip`.** `remora package` wrote `./remora-config.zip` when no `-o, --output` was given and the worker's `downloaddeployment` offered the same name, so the file a project deploys carried a name that matches neither the directory it holds (`remora`) nor the command that builds it. Both now write `remora.zip`, which is also the name the worker's own fetch errors have always used for it. Nothing reads the archive by name — `deploy` uploads it as the `remora_config` form field regardless, and `REMORA_CONFIG_URI` is whatever URI it points at — so this only changes what lands on disk: a pipeline that hardcodes `remora-config.zip` either passes `-o` or expects the new name
-
-### Removed
-
-- **The resource folder names in `project.json`.** `sources`, `producers`, `consumers`, `schemas` and `codeSets` each took a list of paths, so a project could put its consumers anywhere and nothing outside the loader could find a resource from its path alone: `compile`, `create-producer`, `create-consumer` and `init` had already hardcoded `remora/producers` and friends, and only `Environment` read the lists — which meant a project that moved them passed `init` and then failed on the first command that wrote a file. The folders are now fixed by `CONSTANTS.RESOURCE_FOLDERS` and spelled in one place: `/sources`, `/producers` and `/consumers` must exist, and `/schemas` and `/code_sets` are read when they are there and ignored when they are not — so a code set no longer needs declaring at all, it just needs the folder. The five keys are still accepted by the schema and are ignored: a project that declares them loads exactly as before and logs one warning naming what to delete, which is also the first warning the loader has ever raised, so `load()` now prints warnings instead of dropping them
-
-- **`settings.sql` on a producer.** It was documented as the name of a file holding a hand-written statement to run instead of the generated one, and nothing in remora ever resolved that file — on a SQL source it was refused by name, and on every other kind of source it was read by nothing at all, so a producer could carry it and quietly not use it. `sqlTable` plus the producer's dimensions say the same thing and say it in one place: remora builds the `SELECT` itself, from the table and the columns the producer declares, which is what lets every identifier be checked against an identifier pattern before anything reaches the server. The option is now gone from the type, the JSON schema, the preview's summary of what a producer reads, and the docs. Because producer `settings` is `additionalProperties: false`, this moves the refusal from staging time to load time and widens it to every source: a project that still declares `settings.sql` fails `remora compile`/`verify` naming the property, instead of validating and then throwing on the first run that touches the source. Remove the property; if it named a query you actually want, the table goes in `sqlTable` and the columns go in `dimensions`. Two unrelated fields keep the name and are untouched: `measures[].sql` (the expression behind a metric) and a consumer join's `sql`
-
-### Fixed
-
-- **`remora discover` overwrote hand-authored dimension properties.** It rewrote the producer's `dimensions` wholesale from what it had just found, so every property discovery knows nothing about was dropped and replaced with the inferred default: `pk` went, every hand-written description became `Auto-mapped field: <name>`, and `alias`, `format`, `mask`, `classification` and `synth` went with them — as did the producer's `measures`, which were rewritten to `[]`. `pk` is not cosmetic, it is what `synth` and the reference checks read, so re-running discovery on a producer anyone had curated since the last run silently un-curated it. That made the command one you could only safely run once, which is exactly backwards now that discovery on a SQL source finally finds the columns a producer has not declared yet: "add a column upstream, re-discover" was the workflow it broke. Discovery now **merges** into the producer as it is written on disk. An existing dimension keeps everything it declares and gains only what it declared nothing for — in practice `nullable: false`, when the source states the column is `NOT NULL` — and a dimension is matched by its `alias` where it has one, including an `alias` that reaches into a field (`address.city` matches the field `address`), so a renamed or decomposed column is recognised rather than re-added as a duplicate. A **declared type is never changed**, even where the source disagrees: a producer is free to read a `timestamptz` as a `string` or a padded `char(n)` as one, so the disagreement is printed and the decision left to the author. A dimension the source has no field for is kept and printed rather than deleted. Everything outside `dimensions` — the producer's description, settings and measures — is now written back exactly as it was, because the merge starts from the file on disk rather than from the loaded (expanded) resource. The command prints what it did: what it added, what it filled in, what it left alone and why
-
-- **`remora discover` on a SQL source reported success and discovered nothing.** It sampled the statement `SqlQueryBuilder.buildProducerSelect` emits — which is built FROM the producer's dimensions — and then derived the dimensions from what came back, so it could only ever rediscover the columns that were already declared. Pointed at a fourteen-column table, a producer naming one column discovered exactly that one column and printed `✔ Producer has been successfully discovered`: the same "wrong answer, no error" shape as the `NextToken` truncation, and it made the command useless for the one thing it exists for, since a table's unmapped columns are precisely what you run it to find. Discovery on a SQL source now reads the table's own columns out of `information_schema.columns` — the table named by `settings.sqlTable`, never the producer's dimensions — through one small statement that reads no data at all. Because the catalog states the types, nothing is inferred either: a column's SQL type decides the dimension type, so a `numeric` is a `number` even when every value sampled from it happened to be whole and a `timestamptz` is a `datetime` even when the sample was empty, where value inference would have called both a string. A column the table declares `NOT NULL` is written back with `"nullable": false`, which no sample could establish; a nullable one carries nothing, since `nullable` already defaults to true. `information_schema` rather than a dialect catalog keeps Redshift and Postgres answering identically, at one known cost: a Redshift external (Spectrum) or late-binding-view column is not in it and so is not discovered. A producer with no `sqlTable` is now refused before anything is sent, naming the setting, and a table name that matches nothing fails saying so — including that names are matched with the case they were created with, since both engines fold an unquoted identifier to lower case
-
-- **A NULL in a `number` column was silently staged as `0`.** Every source that stages through CSV — SQL (Redshift, Postgres), Delta Share, and local or S3 CSV files — writes a NULL as an empty field, and the cast read that empty field as a value: `algo.hasVal` only rejects `null` and `undefined`, so `''` went through to `Number('')`, which is **`0`**. A missing value therefore became a REAL one, and 0 joins, aggregates, sorts and filters like a 0 the source actually holds — a null contract id quietly became contract id 0. It was found by reconciling a 4.3-million-row Redshift table against the cluster's own `count(distinct)`, which came back exactly one too high on each affected column because SQL skips NULLs and remora had turned them into a value; a local Postgres table with a NULL `numeric` and a NULL `integer` reproduced it independently, as did a boolean (`Boolean('')` is `false`) and a datetime (which threw outright). A missing value is now resolved against the dimension's `nullable` declaration — null by default, the type's empty value when the column declares `nullable: false` — so what remora writes for it is a decision the config makes rather than an artefact of the cast. **This changes what existing projects export**: a number column that holds NULLs and has been emitting `0` for them will now emit `null`, and an output declaring `omitNull` will drop the key entirely. Declare `nullable: false` on the dimension to keep the old value. String columns are unaffected: `''` is a value for a string, not a missing one
-
-- **An export could silently lose its last row.** `readline.Interface` is an EventEmitter whose own async iterator pauses and resumes it, so a loop that awaits — `for await (const line of lineReader)` with an `await` in the body — lets the iterator buffer; if the input ends while it is paused, the interface closes and the next pull resumes a closed interface, throwing `ERR_USE_AFTER_CLOSE`. The throw escapes the loop, so the final line is never written. Whether it happened was decided by where the last read landed against the stream's high-water mark, which makes it deterministic per output size and invisible on small inputs: the same consumer lost a row every run at one data volume and none at another, and a project could pass every test it had and still ship short files. A TXT export of 85,928 rows wrote 85,927 of them. Every loop that awaited per line was exposed — the local and S3 `TXT`/`CSV`/`API` exports, the `distinct` and `lineCount` post-processing passes, the dataset captured for `code_lookup` and consumer-as-input, output splitting by `maximumFileSize`, and the worker's own per-chunk read. `JSON`/`JSONL` exports were never affected: they are copied, not streamed line by line. Line reading now goes through `readLines` (`packages/core/src/LineReader.ts`), which reads the stream directly — a plain Readable has no close to be caught out by, so a caller may await freely between lines. Its line splitting is the same as readline's with `crlfDelay: Infinity` (CRLF however the chunks fall across it, LF, a lone CR, and a final line with no terminator), verified against readline over several thousand randomized inputs and chunk boundaries
-
-- **Two runs of one consumer exporting at the same time could produce a file holding neither of them.** A configured `exportName` is a fixed name, which is the point of it — so a run started by CRON while an API call was still exporting targeted the same path, and `transformAndMove` opened it with a truncating write stream. Both streams then wrote into one file at their own offsets, and the result was a file with the two runs' lines interleaved and a torn line where they met: not one run's output, not the other's, and not valid for its format. It reproduced in roughly one of three attempts with four concurrent runs. An export is now built under a unique name beside its destination and renamed into place, which is atomic within a filesystem, so concurrent runs of one consumer degrade to a clean last-writer-wins — one run's output, whole — instead of corruption. This does not make the losing runs' output survive: a fixed `exportName` still means the last run to finish owns the file, which is what it has always meant. `move` stages the same way, and a staging file is removed when an export fails
-
-- **A failed export finished the run as a success.** The destination drivers catch their own errors, log them and answer `res: false` rather than throwing, and nothing upstream read the flag: the empty key it carries became the run's `resultUri`, so a write that never landed was reported to the caller — API response, usage record, CLI output — as a completed run, with the only trace in the logs. `_unifingResult` made it worse for a consumer writing several files, folding the per-file results with `some`, so one successful file reported the whole export as done. The fold is now `every`, and `launch` refuses the run on a false result, naming the consumer and its destinations and pointing at the logged driver error for the cause. A run that matches no rows is unaffected: writing an empty file is a successful export, not a failed one
-
-- **A remote configuration that failed to extract wiped the one already running.** `Environment._saveConfigFile` deleted every file under the local `remora` directory and only then called `extractAllTo`, on the assumption that an archive which had been constructed was an archive which could be read. It is not: adm-zip parses the entry table lazily, so `new AdmZip(buffer)` succeeds on a malformed archive and the first real read happens inside the extract — after the deletion. A configuration re-uploaded to S3 while a worker was polling it therefore took the worker from a working project to an empty directory, and the process kept serving with no configuration at all. The archive is now read in full before anything live is touched, and extracted to a staging directory beside `remora` that is swapped in only once every entry is on disk, so a refresh either replaces the configuration or leaves the previous one exactly as it was. The commonest way to produce such an archive is to add a file to the existing one rather than rebuild it — an append leaves two entries under the same name, which adm-zip 0.6 rejects as ambiguous because the two can hold different content — so the failure now says that, naming the archive, instead of reporting `Error at bucket ..., while obtainign remora.zip` as though the bucket were at fault; the S3 fetch and the extract are no longer wrapped in the same `catch`
-
-- **`region` and `bucket` could not be `{ENV_VAR}` references on an S3 or Redshift source.** Every other authentication field goes through `SecretManager.replaceSecret`, and these two did not: `RedshiftDriver`, `S3SourceDriver` and `S3DestinationDriver` each read `authentication['region']` (and the S3 pair `authentication['bucket']`) raw and handed it to the AWS SDK. So a source declaring `"region": "{AWS_REGION}"` — the notation the rest of the file uses, and one the project validator already checks and resolves at compile time — failed at run time with `Region not accepted: region="{AWS_REGION}" is not a valid hostname component`, an SDK error naming neither the source nor the variable that went unresolved. The worst of it was that `remora compile` PASSED: the validator confirms the variable is set, so a project could be validated, packaged and deployed, and only fail when a run first touched the source. `QueueManager` had always resolved its region correctly, which is why this survived — the bug was in three drivers and not in the fourth. All four now do the same thing
-
-- **A Postgres source that could not be reached said so without saying why.** A refused or timed-out connection is the commonest thing to go wrong with a SQL source, and it was the one case that reported nothing: node resolves `localhost` to both `::1` and `127.0.0.1`, and when every address fails `pg` rejects with an `AggregateError` whose `message` is the **empty string** and whose actual reasons sit in `errors`. Passing that through produced `Unable to connect to postgres://localhost:5432/analytics for source "s_warehouse": ` — a message naming the host, the database and the source, and then stopping exactly where the reason belongs. The reason is now taken from whichever of `message`, `code` and `errors` carries one, so the same failure reads `...: ECONNREFUSED: connect ECONNREFUSED ::1:5432; connect ECONNREFUSED 127.0.0.1:5432`, and duplicate addresses are collapsed because a host refusing on both of its addresses is one fact, not two. A failure that does report a message — a wrong password, a database that does not exist — is unchanged
-
-- **`synth` no longer fills a column with nulls when a built-in code set is auto-bound to it by name but cannot produce its declared type.** Built-in sets are matched on the field name alone, so a numeric surrogate key like `heb_quality_gaps_measure_code_sk` matched the `measure_code` binding and was written as a HEDIS code (`"CBP"`); read back as its declared `number` it cast to `NaN`, which serializes as `null`, and every row of the column was empty. A code set now has to be able to produce the dimension's type: an automatic binding that cannot is only a name-based guess and is dropped, so the column is generated from its own type (that surrogate key is a dense numeric key again), while an explicit `synth.codeSet` or `synth.allowedValues` that cannot is a configuration error and is refused by name
-
-## V 2.1.1 - 2026-09-23
-
-### Added
-
-- **SQL sources can be run.** `aws-redshift` has been a valid engine since the beginning, but `RedshiftDriver.ready` — the one method every run goes through to stage a producer — threw `Not implemented yet`, so a Redshift producer could be configured, validated and sampled, and never executed. It runs now, and a new **`postgres`** engine runs beside it: Amazon RDS, Aurora PostgreSQL, or a self-hosted server, over the native wire protocol. A producer on either names its table with `sqlTable` (`table` or `schema.table`, falling back to the source's `schema`) and lists the columns it wants as dimensions, with `alias` naming the column where it differs from the dimension name; remora builds the `SELECT` from exactly those dimensions and stages the result as a local CSV dataset, which is the same thing a Delta share does and the reason the rest of the pipeline needed no change. The statement is built in one place from the producer's own config, and every identifier in it is checked against an identifier pattern rather than escaped, so a table or alias that is not a name is refused where it is written instead of reaching the server as syntax. Both engines are now in `STAGEABLE_ENGINES`, so `remora run`, `remora sample` on a consumer and the preview's **Data** tab all reach them
-- **Postgres authenticates with an RDS IAM auth token**, with `method: "iam"` or `"arn"`, so there is no database password in the configuration at all: a short-lived token is signed per connection, by the SDK default credential chain or by an assumed role, through the same `resolveAwsCredentials` every other AWS driver uses. `method: "username-password"` keeps the ordinary path, where the password is an `{ENV_VAR}` reference like any other secret. TLS is on by default for a remote host (`sslMode`: `require`, or `verify-full` to verify the chain) and off for `localhost`, where a local database usually speaks none
-
-### Changed
-
-- A Redshift statement is **polled for up to 30 minutes** with exponential backoff, instead of 20 cycles of a 1 ms sleep. The Data API is submit-and-poll, and the old loop was sized for the one-row warm-up query in `init`: a table scan legitimately sits in `STARTED` for minutes and would have been abandoned as a timeout. Provisioned clusters also work now — a statement carries `ClusterIdentifier`/`DbUser` when the source declares `clusterId` rather than a `workgroup`, where before only Serverless workgroups were ever sent
-
-- **A live run now says what it read, without `REMORA_DEBUG_MODE`.** Everything that reported what a run actually *did* was `logger.log` — DEBUG, which returns on its first line unless the variable was `true` when the process started — so a deployed worker logged a launch line, a completion line and nothing in between. Restarting it with debug on is not an option when the question is about a run that already happened, and turning it on permanently means every diagnostic line of every driver, forever. Three things are now `info`, chosen because each one answers a question that cannot be answered after the fact from anywhere else. **The options the run was invoked with**: the launch line named the consumer, the invoker, the user and the producer count but never the request, and `IUsageStat` does not keep it either, so there was no way to tell whether a run had been handed filters and a limit or had run on the consumer's own config — it now reads `..., producer(s): 2, options: 2 filter(s), limit 100)`, and is unchanged on a run that carried no options. **What a Delta Share query returned**: `Delta Share table "x" returned N file(s) in Yms` is the number that says whether the pushdown pruned anything, and it now names the hints that were sent with it (`(pushdown: predicate, limit 100)`) so the count can be read against what was actually asked of the share; the predicate tree itself stays on the debug line, since it is a JSON document and not a thing to put in every run's log. **What the read staged**: records, dataset files and bytes per producer, which is the run's own account of the volume it pulled. Nothing new is computed and nothing else moved level — the remaining delta-share diagnostics, the per-file skips and the metadata dump are still debug
-
-- **The options an invocation asks a run for are now checked at the door.** A consumer run can be handed `filters` and a `limit` by an API call (`POST` execute) or a queue message, and both entry points took whatever arrived: the queue path checked that `options` was a non-array object and CAST it, the HTTP route read `req.body.options` and passed it straight to the orchestrator. So a `limit` could arrive as `"100"`, as `0` or as `-5` and bound nothing; a `filters` could be an object; an `operator` could be a word `RequestExecutor` has no case for; and a `member` could name no field of the consumer — which is the worst of them, because an unknown member reads `undefined` off every record and so *silently filters nothing*, returning a full dataset that looks like an answer. The rest surfaced as a throw from the middle of a run that had already queried the source and paid for the read. Both entry points now go through one check, `RequestValidator.validate`, before anything is read: `filters` an array of objects, `member` a field the consumer actually outputs (the expanded `alias ?? key` names, which is what a record holds when filters run), `operator` one of `FilterOperator`, `values` an array of strings of the arity the operator needs (`between` exactly two, `in` at least one, `isNull` none), `limit` a whole number between 1 and `MAX_REQUEST_LIMIT`, and `and`/`or` children validated the same way down to `MAX_REQUEST_FILTER_DEPTH`. Every rejection names the offending option and where it sits — `Invalid filter at filters[1].or[0]: consumer "sales" has no field named "regionn"` — and the API answers **400** instead of running and returning 500 or, worse, 200. Two things are refused outright: **`offset` and `order`**, which the request type carries but no stage of a run implements (the only code reading them is `PushdownEngine.resolveLimit`, where their presence merely suppresses the limit pushdown), and the **`sql`** operator, which `RequestExecutor` throws on rather than evaluating — accepting either would mean answering as if the request had been honoured. An unknown option key (`limits`, `filter`) is refused too, since the alternative is ignoring it silently. Bounds on what may be asked for live with the rest: `MAX_REQUEST_LIMIT` (10,000,000), `MAX_REQUEST_FILTERS` (100) and `MAX_REQUEST_FILTER_DEPTH` (5)
-
-### Fixed
-
-- **A ZIP producer read the whole archive into memory — and had in fact never run at all.** `ParseCompression` extracted a zip through AdmZip, which reads the entire archive into one buffer and then decompresses each member into a second one. On a 71 MB archive holding a 441 MB CSV that is **1.06 GB of peak RSS, 2.4x the member**: a file a 2 GB worker stages comfortably as plain CSV exhausts it the moment it arrives zipped. `.gz` and `.tar` beside it already streamed; only ZIP was left behind, the same shape of bug as the delta-share crash. A zip is now read entry by entry with `yauzl`, each member piped from the archive straight to disk, and the same file costs **136 MB peak — about 44 MB above an empty process** — bounded by the pipeline's own buffers rather than by what the archive holds. Two things fell out of the rewrite. The extraction **never ran**: `ExecutorScope.ensurePath` touches the dataset path as an empty *file* before a driver is called, and extracting an archive into that path went straight to `EEXIST: mkdir`, so ZIP and TAR were both dead on arrival on the local source driver. And an archive with more than one member is now read **in full** — `decompressToFile` returns every file it extracted rather than the directory it put them in, and `LocalSourceDriver.ready` fans those out exactly the way it already fans out a wildcard `fileKey`, where before a multi-member archive handed the run a directory path to read as though it were a data file. TAR reports its members the same way. Entry paths are still resolved and checked against the target directory before anything is opened (yauzl refuses a traversal name of its own accord; the `affirm` is the second line), an encrypted member is refused by name instead of writing its ciphertext, and `MAX_ARCHIVE_ENTRIES` (10 000) bounds what a single fileKey may fan out into. `c_canary_zip` in the canary reads a two-member archive and checks the row count against the unsplit source, because reading only the first member is a silent half-answer rather than a failure
-- **Redshift silently returned only the first page of every query.** `getStatementResult` never read `result.NextToken`, and the Data API pages results at roughly 1 MB — so any statement returning more than that came back **short, with no error**: `SQL_MAX_QUERY_ROWS` could not be honoured, a sample of a wide table was a fraction of what was asked for, and nothing anywhere said so. Not a crash but a wrong answer, which is worse, and it had to be fixed before a staging read could be built on the same call. Every read now follows the chain to the end. The in-memory paths (`query`, `execute`) stop at `SQL_MAX_QUERY_ROWS` and **log that they truncated**, rather than doing it quietly; staging accumulates nothing at all — each page is written to the dataset file before the next is fetched — and then **checks what it staged against the row count the cluster reports**, so a lost page fails the run by name instead of producing a short dataset that looks complete
-- Staging a table costs disk rather than memory, on both engines: Postgres reads through a server-side cursor inside a `READ ONLY` transaction, `FETCH`ing 10 000 rows per round trip and writing each batch before asking for the next, and the cursor is closed and the transaction ended even when the read is abandoned half way. Redshift does the same with result pages. Neither ever holds the table
-- **A run that read no rows crashed on the S3 upload.** `S3DestinationDriver` uploads every output through a multipart upload, and an empty dataset — a Delta Share (or any other source) that legitimately returned nothing for the period asked of it — produces zero parts, so the run ended on a `CompleteMultipartUpload` carrying an empty part list, which S3 rejects as `MalformedXML`. The error named neither the consumer nor the reason, so an empty read looked like a broken destination. An upload with no parts is now aborted and the object written directly as a zero-byte `PutObject`, which is what the local destination driver has always done: a no-data run lands an empty file and succeeds. A failed abort is also no longer able to mask the error that triggered it — it is logged with the key and upload id, and the original exception is the one that propagates
-
-## V 2.1.0 - 2026-09-17
-
-### Added
-
-- **`remora ui-preview` now shows a field's `description`.** A consumer field, a producer dimension or measure, and a JSON-schema property can all carry one, and the viewer was the one place a project's own documentation did not reach: the Fields and Dimensions tables printed the name, the type and the transformations, and dropped the sentence saying what the column is *for* — which on a layout of a thousand generated column names is the only thing that tells two of them apart. A description now reads as a **second line under its row**, across the full width of the panel, muted and small. Not as a column: free prose would have set the width of a table whose other columns are a word wide, and printed a dash on every row nobody documented, where this costs a line on the rows that carry one and nothing at all on the rest. Long prose is clamped to two lines with the whole of it as the row's tooltip (and verbatim in the **File** tab), and it wraps inside the panel however wide the table is, so nothing ends up hidden behind a horizontal scroll. The **field filter matches descriptions too**, so a field can now be found by what it does and not only by what it is called — and the match is marked in the description the same way it is in a name. In the contract this is one optional `note` on a cell, so the page still renders tables it knows nothing about and the format version is unchanged; a snapshot written by an older CLI simply has no notes to draw
-- Tests for the delta-share driver — the package's first — covering the batched read that replaced it: that the batches cover a file exactly once and never span a row group whatever the shape of its footer, that the streamed CSV is byte-for-byte what the old buffered write produced, that a range read stops as soon as it is filled, and that `ready` stages a real parquet file over a real socket and leaves nothing behind. The row-group arithmetic is pinned against a stubbed reader, which can describe the million-row group that caused the crash; what only a real file can settle — that hyparquet's `rowStart`/`rowEnd` are absolute row indexes and slice correctly *inside* a group — is pinned against three small fixtures from hyparquet's own corpus. `npm run test:unit` runs them
-- A **Delta Share consumer in the canary** (`c_canary_deltashare`). The share already configured there carries `mock_data` — the same 120 accounts as the canary's own `mock_data.jsonl` — so the check is an equivalence one: every row that came over the Delta Sharing protocol, through a share query, a pre-signed URL and a parquet file staged in row batches, must equal the row the local file already holds. That is what tells a *wrong* read from a merely successful one, which matters for a staging path that reads a part file in pieces: a batch that lost, repeated or misaligned rows would still produce a plausible-looking output. The consumer's country filter doubles as pushdown coverage — it reaches the share as a json predicate hint — and since a hint is only ever an optimization the expected rows are derived locally either way. Delta Share was the one source engine with no canary coverage at all, which is how a four-copies-of-the-file bug survived in it
-
-### Changed
-
-- The **state heartbeat now emits every five minutes** instead of every minute. The payload is a full status object — worker version, config signature, resource counts, every CRON job with its next fire time, queue mappings, in-flight runs, heap — and at the old cadence an idle worker wrote 1440 of them a day, forever, each one an ingested and retained CloudWatch event the customer pays for. The evidence it exists to provide survives the longer interval: a wedged run is the same `executionId` with a growing `elapsedMS` across a handful of records, and a task replaced at 03:00 still announces itself at once, because the first heartbeat goes out immediately on start rather than after a full interval. `REMORA_STATE_INTERVAL_MS` overrides it as before, still clamped to 5s–1h — and a value that is unusable or out of that range is now **logged as a warning** naming the interval actually used, since an operator who sets an interval and silently gets a different one has no other way to find out
-
-### Fixed
-
-- **A delta-share producer crashed the worker with `JavaScript heap out of memory`.** A part file is small on the wire and large in the heap — one 2.4 MB zstd file of a real share decodes to 572k rows — and the driver held that file four times over at once: hyparquet's row objects, a second array of them from `toJson`, an array of the CSV lines, and the single string joining those lines before the write. Four copies of half a million rows do not fit in a worker's 2 GB heap, so the run died on the file rather than on the table's size, with no error of its own — just V8's `FATAL ERROR` and exit code 134. The staging path now reads a part file **in batches of 50k rows** and writes each batch out as it arrives, so only one batch is ever in memory and a file's row count stops being a memory limit. To make the batched reads cheap the file is first copied to the run's temp folder, streamed, and read from there: reading it in passes over its pre-signed URL would have refetched the same bytes for each batch, and the download itself never holds more than a socket buffer. The staged dataset file is byte-for-byte what it was — same header, same order, same escaping — and the download is dropped once converted
-- The same unbounded read is gone from the **preview and sample** path. `readLinesInRange` used to decode every row of every part file of a table and then keep the ten it was asked for; it now stops at the first batch that fills the range, so asking a 572k-row table for ten rows reads 50k of them instead of all of them, and a part file past the range is never opened at all. A range of **zero** rows now reads nothing rather than decoding a batch to return an empty list
-- **A logged error arrived in CloudWatch as thirty unrelated log events.** The awslogs driver ends a log event at every newline on stdout, so a stack trace — the one thing worth reading when a run fails — was split line by line, each fragment its own event, none of them carrying the message it belonged to, and all of them interleaved with whatever the other threads were writing at the same moment. Deployed processes (`REMORA_RUNTIME_CONTEXT` other than `cli`) now render every console record as **one line of JSON**: `{"remora":"log","v":1,"ts":…,"level":…,"message":…,"stack":…}`, with the stack a field of the record rather than a hundred events after it. One record is one event, searchable by level and by message in Logs Insights, the same shape the heartbeat and run records already use. The CLI is untouched — a terminal keeps the human-readable format, colours and the stack on its own lines — and `REMORA_LOG_FORMAT=text|json` overrides the default either way. `console.log`/`console.error` are routed through the logger in the deployed entrypoints too, since a rule enforced only at our own call sites is one the next `console.error` — ours, express's or a driver's — quietly breaks. An oversized field is clamped, with the truncation stated in the record: a log event is capped at 256 KB and what exceeds it is split or dropped, which is the problem this format exists to solve
-
-## V 2.0.3 - 2026-09-03
-
-### Added
-
-- `remora ui-preview` now has **search**, at both levels the project has: a box in the header finds a **node**, and a box above the panel's tables filters the **fields inside the selected one**. The node search matches on name, kind, description and badges, all terms in any order (`cons canary` finds `c_canary_extract`), and its answer lands **in the picture** rather than only beside it — everything that does not match fades on the canvas, so a search over fifty nodes leaves the matching ones legible in the graph they belong to instead of handing back a list to go and find them in. `/` from anywhere focuses it, ↑/↓ walk the results, Enter selects and pans to it, Escape clears. The field filter keeps only the rows that match — a dimension, a consumer field, a resolved lineage path, an output, a run — and the **tab strip carries the count per tab**, which is what makes it usable on a resource whose fields are spread over four tables: `addr0Street` says which tab holds it without each one having to be opened to find out. Matched text is marked in place, since a filtered row of near-identical generated column names is not self-explanatory once found. It filters a `Data` sample the same way, is dropped when the selection moves to another resource, and survives a live reload of the same one — like the open tab and the scroll position. All of it runs **in the page**, on the documents it already has: no route, no new capability and no contract change, so a snapshot opened with `remora-preview <file>` searches exactly the same way
-
-### Changed
-
-- **Sampling now runs the pipeline.** `remora sample <consumer>`, and the **Data** tab of `remora ui-preview`, used to answer a question nobody asked: the rows shown for a consumer were its *first producer's* rows, projected through the consumer's field list. No filters, no field transformations, no unions, no `distinct`, no `pivot`, no measures, no validation. Under a heading that says "Data", beside a consumer's name, that read as what the consumer produces — and it was not: a consumer whose only job is to uppercase a column showed it lowercase, and one that keeps one row in a hundred showed the ninety-nine. Sampling a consumer now **executes it** — every filter, transformation, mask, union, `distinct`, `pivot`, measure and dataset validation, exactly as `remora run` applies them — so the rows are the consumer's own output, always, rather than an approximation of it. It **touches nothing**: no write to the destination, no `onSuccess`/`onError` action, and although the execution is recorded like any other it is marked as a sample, so it stays out of the run overlay, out of a consumer's run list and out of its duration averages — a node's dot and its "how long does this take" keep meaning a real run over real volume. The bound on the rows is the same `limit` `remora run --limit` uses, so the cost follows the same rule: where nothing between the source and the output can *drop* a record the limit reaches the source itself and the read stops early, and where something can — any filter, `distinct`, `distinctOn`, `pivot`, or a validation set to `"onFail": "skip"` — there is no knowing how many rows must go in for ten to come out, so the whole source is read. On a large source that is slow; it is still only a read. Sampling a consumer needs a source the pipeline can stage — `local`, `aws-s3` or `delta-share` — so one whose producer reads `aws-redshift` or `http-api` (whose drivers have no `ready()`) is refused up front with a message naming the producer and pointing at it, instead of a worker pool starting and then surfacing `Not implemented yet` from three layers down. In the preview it takes the same one-at-a-time lock as `▶ Run`, since it is the same worker pool. Sampling a **producer** is unchanged in shape and still costs one bounded read; what it now does is the producer stage in full (see the fix below). None of this added anything to the execution path: a sample is `launch` with three flags the pipeline already had for the materialization runs it already did — `options.limit`, `skipExport` and `captureMainDatasetTo` — so the executor cannot tell it is being sampled, and one code path still serves both callers
-- A sampled consumer's columns are now ordered by the **fields the consumer declares**, rather than by the order the producer's dimensions happened to be read in. Invisible in an exported JSON object; the whole shape of a table you are reading
-- `remora ui-preview`'s **Data** tab is quieter. The button keeps one label — `Sample 10 rows` — rather than re-wording itself to `Sample again 10 rows` once rows are on screen: it does the same thing on the tenth press as on the first, and moving the text under the pointer said nothing new. The paragraph beside it, explaining what sampling costs, is gone; what actually ran is stated over the table by the host that ran it, which is the sentence worth reading and is there every time. The button still says what it is doing while it waits
-
-### Fixed
-
-- **A masked producer dimension was shown in the clear when sampled.** A dimension declared `"mask": "hash"` (or `crypt`, `random`, `seeded-random`) is masked in `ProducerExecutor.processLine`, on the run path — and the sampling path never went through it. So `remora sample <producer>` and the preview's **Data** tab printed the raw value of every column the project had asked to be masked, as did the projected consumer rows built from them. That is a disclosure rather than a display bug: the values reach a terminal and a browser page, and the one place someone checks that masking works is the place that was not applying it. `sourceFilename` dimensions were blank there for the same reason. The producer stage now lives in one place — `ProducerManager.compileFills`/`applyFills` — which both the worker and the sample read call, so a mask cannot be applied on one path and skipped on the other
-- **A sample in `remora ui-preview` showed no rows at all.** The Data tab ran the sample, the host answered with the rows, and the panel stayed empty with nothing in the log to explain it. Two things in the run overlay caused it, both from a sample being *recorded* like any other execution and then filtered out of the picture: `RunOverlay.signature()` — the token the preview polls to notice a finished run — signed the usage **store** rather than the runs it paints, so writing a sample's own record moved the token while the picture stayed identical; and the sample marker was dropped *after* the 500-row read limit, so a sample also pushed a real run off the end of the history. Either way the preview concluded the project had changed and told the page so, and a page told its resource changed **drops the sample it is showing** — deliberately, since rows taken against the previous definition must never be shown under the new one. So the rows were fetched and then discarded on arrival, every time. The signature now signs the overlay it describes (gated on the store's mtime, so polling still reads metadata and not records) and the marker is applied as the store reads, so a sample cannot displace a run. A real run notifies and repaints exactly as before
-- **`remora ui-preview`'s minimap and zoom controls were white in dark mode**, which on a dark canvas left them as a bright rectangle and three invisible buttons. xyflow paints its own chrome from its own palette and ships a `.dark` variant of it that has to be switched on and then kept in step with greys that are not this product's; it now reads the theme's tokens through the `--xy-*` slots the library exposes, so one declaration themes the canvas in both modes and the canvas can never be in a mode the rest of the page is not
-- **The minimap was empty**, whatever colour it was. It draws a node from the size the node DECLARES, and the layout — which fixes every card at the same box and reserves exactly that box in dagre — had never said so on the node itself, so there was nothing for it to draw and the panel was a blank rectangle. The box is now declared as `initialWidth`/`initialHeight`, the size xyflow assumes until it measures the rendered card: the minimap gets its nodes and the canvas keeps measuring, so the picture itself is unchanged
-- `remora ui-preview`'s sample route is now a `POST` carrying the session token, like `▶ Run` already was. It **executes**: a producer's sample spends a query against your warehouse and a consumer's spends this machine's threads, and loopback is not privacy — every site you visit can address 127.0.0.1, so a `GET` let any page you happened to have open start one
-- A consumer's `onError` actions no longer fire for a run that exports nothing. A `code_lookup` table or an upstream consumer-input that failed while being materialized ran the dependent consumer's `onError` — moving its source file, starting other consumers — for an execution nobody asked for; the success path had always skipped them, and the failure path now does too
-- `remora sample <producer> -s 10` on a file with a header row returned **nine** rows. The header was one of the lines read and then dropped, so the count asked for was always short by one
-- `remora ui-preview` no longer describes itself as **read-only**. The panel's footer promised "this viewer is read-only and never reads your data", and the command's own `--help` said the same, while the page it was printed on samples real rows, runs consumers and saves configuration files. What is still true is narrower and is where it is now said: the `@forzalabs/remora-preview` **package** has no drivers, no credentials and no configuration reader, so everything past rendering is a capability of the host and reaches the page as a route only when the host advertises one — and `remora-preview <snapshot.json>`, which has no host behind it, advertises none and is read-only in fact rather than by claim
-
-## V 2.0.1 - 2026-09-01
-
-### Changed
-
-- A configuration that fails its JSON schema now reports **the mistake** rather than every reason it wasn't something else. A `oneOf` — a consumer `transform`, one alternative per transformation — makes Ajv explain why all 27 alternatives were rejected, so a `"cast": "date"` typo came out as 82 complaints whose first three were `must be equal to one of the allowed values; must have required property 'multiply'; must NOT have additional properties`, naming a transformation the field never mentioned and burying the one line that mattered. The alternatives' rejections are now collapsed: what is reported is whatever an alternative said about a value *inside* the branch — the one that got far enough in to find the real mistake — and when none did, the branch itself, listing what was actually allowed there and what the field has instead. Messages also carry the offending **value** (`must be one of: string, number, datetime, boolean (got "date")`), the **file** the resource was read from, since two resources can carry the same `name` and the one to go and edit is a path, and the **name** of the array item the pointer lands in, so `/fields/2` reads as `/fields/2 (field "created_at")` instead of sending you to count array elements
-- `remora compile` no longer prints a configuration error twice, once under `❌ Unexpected error during validation`. A resource that fails its schema is an expected outcome of `compile`, and is now reported in the same list as every other validation error
-
-### Fixed
-
-- A debug-mode run (`REMORA_DEBUG_MODE=true`) could fail every chunk with `write after end`, reported as `Affirm failed: The worker pool ran 7 thread(s) at once but was sized 2`. The executor worker **closed** the file log sink at the end of every task, but a thread outlives its task under the run-long pool, and a closed winston logger still accepts lines and writes them into an ended stream — `ERR_STREAM_WRITE_AFTER_END`, thrown from a stream nothing listens to, which inside a worker thread kills the thread and fails the chunk it was running. The sink is now flushed rather than closed; closing it resets it, so a line logged after a close re-opens a fresh sink instead of ending the process; and pointing the sink at a file it already writes no longer builds a second transport for it, which a reused thread was doing once per task. A failing sink can no longer take the process with it either — winston re-emits transport failures, and an unhandled `error` on a stream is an uncaughtException
-- A run that lost a worker thread now reports the **chunk error that killed it** instead of `The worker pool ran N thread(s) at once but was sized M`. The pool never oversubscribed: the thread tally counted spawns as concurrency, and workerpool drops a dead worker from its list synchronously while only firing its termination callback from a handler an already-exited thread never triggers — so every crashed-and-replaced thread inflated the count by one and the assertion fired ahead of the real error. Concurrency is now measured on the pool itself, and a run with dead threads says how many chunks failed and how many threads were spawned to serve them
-
-## V 2.0.0 - 2026-09-01
-
-**Upgrading from 1.x.** Four changes can alter what an existing project does. JSON schema validation now actually runs at load, so a configuration that was silently schema-invalid fails at `compile` — run `remora compile` (or `remora graph`, which lists every problem at once without failing) before upgrading. `filters[].sql` and the `"sql"` filter operator are gone; rewrite them as `rule` filters. Filters on `number`, `datetime` and `boolean` fields now match, so a consumer whose filters were silently matching nothing will start returning rows. And `remora run -l` is now the whole run's limit rather than each worker's, so a limited run returns the number of rows asked for instead of that number multiplied by the chunk count. The `mock` command is removed — use `synth`.
-
-### Added
-
-- A consumer's filters are now **pushed down to a Delta Share source**, so the share server skips the files that cannot hold a matching row instead of handing over the whole table for remora to read, stage and throw away. The eligible filters go out as `jsonPredicateHints` on the table query — the protocol's structured predicate form, which prunes on partition columns and, where the share keeps them, on file statistics. Filters supplied by an invocation (an API or queue request) are pushed on the same terms as the ones written into the consumer, which is where it pays most: an API caller asking for one region no longer drags the other twenty across the wire first. **The result is identical either way.** The hint is best-effort by protocol — a server may ignore it, may fail to parse it and return everything, and may return files that don't match — so every filter is still evaluated against every record exactly as before; pushing one down only ever saves download time and staging space. Which is also why it is deliberately conservative about what it pushes. A filter goes to the source only when it can mean the same thing there: the field has to read a producer dimension **untouched** (no `transform`, no `validate`, no `default`, not `fixed`/`copyFrom`/`lineCount`, not a masked or `sourceFilename` dimension, and not a field belonging to a different producer), the operator has to have a counterpart in the Delta Sharing grammar (`equals`, `in`, `between` and the four `greaterThan`/`lessThan` comparisons — the substring and regex operators have none), and — the subtle one — the filter must not also keep rows whose value is **missing**. A value the source doesn't have reads back on the record as `""` or `0` rather than as null, so `notEquals`, `isNotNull`, `notBetween` and any range covering `0` keep those rows; a source pruning on them would drop data the run is supposed to return, so none of them is ever pushed. Date bounds are only translated when they are ISO-8601, and one carrying milliseconds is left local rather than rounded to a second, since rounding a bound is exactly how a row that belongs in the result stops being fetched. Boolean columns are never pushed, because remora reads a boolean cell through `Boolean(value)` and the source's own `false` reads back as `true`. Nothing needs configuring: it applies to any consumer with filters over a Delta Share producer. A producer whose share mishandles the hint can opt out with `"disablePushdown": true` in its settings and lose nothing but bandwidth. What the run sends is written to the log, predicate and all
-- The **record limit** (`remora run -l`, or `limit` on an API/queue request) is pushed down to a Delta Share source too, as the query's `limitHint`, so a share server can stop listing files once it has covered the rows being asked for. Where the filter pushdown saves reading rows that will be discarded, this saves reading a table that was never going to be looked at: `remora run c_big -l 100` against a table of millions no longer downloads all of it to write a hundred records. The rule for pushing it is stricter than for a filter, because a bounded read is only sound when every row read reaches the output — so the limit is dropped whenever anything between the source and the export can *drop* a record: any filter at all (a bounded read could have fetched exactly the rows the filter goes on to reject), `distinct`, `distinctOn`, `pivot`, a field validation set to `"onFail": "skip"`, a producer `recordSelector`, or a group fanning a line out to rows. An `offset` or an `order` drops it as well, since both need rows a bounded read would stop before. A transformation's own `"onError": "skip"` does not, since it abandons that one transformation rather than the record. Like the predicate it is a hint — the server may return files holding far more rows than asked for — and the run's own limit is unchanged and still enforced. `"disablePushdown": true` on a producer turns both hints off
-- Added `remora ui-preview`, a visual preview of the project in the browser: the lineage graph on a pan/zoom canvas, and a click-through inspector for every resource. Where `remora graph` prints the picture, this one is explorable — select a producer and its dimensions, measures and settings are **tables**, not JSON; the source it reads links to the source; a consumer's fields link to the producer dimension each one comes from, its producers to the producers, its outputs to where the data lands and to the consumers they chain into, and the panel keeps a back stack so you can walk the pipeline by clicking through it. Nodes are coloured by what is wrong with them and every problem the project has is listed in one panel, each one a click from the resource it is about. **It updates while you edit.** The configuration is watched, and a save re-reads the project and refreshes the page in place — keeping your selection, your scroll and your position on the canvas — so the picture is never the one from two saves ago. A file caught mid-write briefly shows as a red node and goes green again a second later, because the preview reads a project the lenient way (`Environment.inspect()`): a broken configuration renders rather than failing, which is when the picture is most wanted. `-p, --port <n>` picks the port (default 5070, falling forward to the next free one), `--no-open` leaves the browser alone. The page is served on loopback only, and **the viewer cannot reach your data**: it is a separate package, `@forzalabs/remora-preview`, whose only runtime dependency is `express` — no drivers, no credentials, no configuration reader. It receives one versioned JSON document and renders it. **It can also show you the data.** A producer or a consumer gets a **Data** tab that reads a handful of real rows and puts them beside the columns the resource declares, so "is this column actually populated?" is answered where the question comes up — a consumer's rows arriving projected through its own field list, which is the shape it will export. It is the same read `remora sample` does, through the same code path, so the two can never disagree. It reads **only when you click it** — never on selecting a node and never on a save — because a sample is a query against the real source with your credentials, and the rows are dropped whenever the project is re-read, so you are never looking at old data under a new definition. That is a capability of the **CLI**, not of the viewer: the page offers the tab only when the host it is talking to advertises one, so a snapshot opened with `remora-preview <file>` has no such tab and no route behind it, and a source that cannot be reached comes back as the reason rather than as a blank panel. **It shows you where every column comes from.** A consumer gets a **Lineage** tab that resolves each of its output columns back to the producer column it started as — through however many consumers sit in between — with every transformation applied along the way and a link to the origin, so `country_upper` reads back as `p_orders.country → c_stage.country → c_final.country_upper · to_uppercase` instead of being a walk through three files with the references held in your head. `fixed`, `lineCount` and `copyFrom` fields resolve to what they actually are, and a column whose chain cannot be completed says why — a `from` naming no producer of the consumer, a key the producer does not declare, a name ambiguous between two non-union producers — rather than going quietly blank. A producer gets the same thing from the other end, a **Used by** tab: which consumers read each of its columns, and, the half that is hard to get any other way, which columns **nothing** reads at all. It is resolved the way the run path resolves it, by the same `from`/union/ambiguity rules, because a lineage that disagreed with the executor would be worse than no lineage at all. **And it shows you what happened the last time each consumer ran.** Every node carries a dot coloured by the outcome of its most recent run, with the age, the row count and the failure reason in its tooltip, and that consumer's recent runs as a **Runs** tab — outcome, when, how long, how many rows and which Remora version ran it, since a timing is only comparable within one release. Those come from the CLI's own local run store, the one `remora run` already writes to `remora/.temp/usage`, so a run that finishes while the preview is open repaints the graph within a second even though no configuration file changed. A deployed worker's history is not read here — it has its own channel by design, and colouring a graph is not a reason to poll a database once a second. `remora graph -f json` carries the field lineage, which is configuration and travels with the document, and deliberately does not carry the runs: “failed 3 minutes ago” frozen into a file someone opens next week reads as permanent. **And it does things, not only shows them.** A consumer has a **Run** button: it starts the same execution `remora run` does, with the same licence, the same records and the same output, and hands back control immediately — the node's own dot turns blue while it runs and green or red when it finishes, so the picture you are already looking at is the progress display. One run at a time, a producer cannot be run (only consumers execute), and a consumer whose node is already red is refused with the diagnostic as the reason, because running a configuration the preview has just told you is broken fails somewhere less legible. Beside the run history there is now a **Metrics** table — runs recorded, last and average duration over the successful ones, rows in and out, workers, output cycles, and the per-stage timings of the last run, which is where a slow run says *which* stage was slow. A **Logs** drawer across the bottom tails the log file the CLI writes, refreshing while it is open. And a **File** tab holds the resource **as authored**, editable, with Save and Revert: it is the file on disk rather than the loaded object, because a loaded producer has already had its groups and `occurs` expanded and writing that back would persist the expansion as if you had typed it. Saving is refused, with the reason, for JSON that will not parse, for a rename (a resource's name is its identity in every reference to it and in the name of its own file), and for anything its JSON schema rejects — checked with the same code `compile` runs, so the panel can never save a project the CLI would then refuse to load, and says so in the same words. A save lands atomically and the page refreshes itself a moment later, and an unsaved draft survives that refresh rather than being thrown away, with a warning when the file moved underneath it. **All of that is guarded.** Binding to loopback keeps the page off the network but does not stop a *browser* — every site you visit can address 127.0.0.1, and a hostname that resolves there defeats a naive check — which was a curiosity while every route returned a document and is not one now that a route can run your pipeline with your warehouse credentials. So the server refuses any request whose `Host` header is not one of its own (closing DNS rebinding), refuses anything carrying a cross-origin `Origin`, and requires a per-session token, injected into the page itself, on every request that changes something. The reads are guarded the same way. Running, editing and log-tailing are all **capabilities of the CLI**, like sampling: a snapshot opened with `remora-preview <file>` has no such routes at all, so there is nothing there to refuse. **The panel resizes.** The divider between the graph and the inspector drags, so a wide table — a resolved lineage path, a 1,300-column dimension list, a JSON file — gets the room it needs without the graph being stuck at whatever is left; double-click the divider to go back to the default, and the width you chose is remembered. That package also installs a `remora-preview <snapshot.json>` command of its own, which serves a snapshot written by `remora graph -f json` for someone with no CLI and no licence
-- Added **repeating groups** to a `FIXED` producer's dimensions, for flat files transcribed from a COBOL copybook (`OCCURS`). A dimension that carries its own nested `dimensions` is a *group*: it declares no `type`, produces no column of its own, and instead declares the fields that repeat — with each child `position` **relative to one occurrence**, and the group's own `position` describing **one occurrence** (`start` of the first, `length` per occurrence). `"occurs": 3` on a 30-character address segment starting at column 32 therefore reads occurrences at 32, 62 and 92, generating `addr0Street`, `addr0State`, … `addr2Zip`. Names follow a `naming` template (`{group}`, `{index}`, `{field}`, `{Field}`; default `{group}{index}{Field}`) with `indexFrom` choosing whether indexes start at 0 or 1, so generated names can match the project's own style. Groups expand **in place** when the project loads, at the group's own index, so the columns keep the physical order of the record and a dimension declared after a group stays after it — and because expansion happens before anything reads the dimensions, consumers, transforms, validations, exports, `sample`, `mock` and `synth` all work on a grouped producer with no changes at all. `occurs` defaults to `1`, which makes a group also a way to simply structure a segment of a layout. Mistakes are rejected at `compile` with a message naming the dimension: a child position that doesn't fit inside one occurrence (an absolute offset pasted where a relative one belongs), a group carrying properties that only mean anything on a column (`type`, `alias`, `pk`, `mask`, `synth`, `format`, `sourceFilename`), a `naming` template with no `{index}`, a generated name colliding with another dimension, a group on a non-`FIXED` producer, and — for now — a nested group
-- Added `flags` on a `FIXED` producer dimension, decoding a per-character bitmap into one dimension per character: `"flags": ["mailing", "home", "billing"]` on a 3-character field yields `addressUseMailing`, `addressUseHome` and `addressUseBilling`, each a 1-character slice from `position.start`. `flags` must list exactly one name per character the dimension spans. Each character is *sliced, never interpreted*, so a value that isn't strictly binary survives as itself (a `"3"` stays `"3"`) and can be decoded downstream with `conditional` or `code_lookup`. Names follow the same `naming` template mechanism (`{field}`, `{Field}`, `{flag}`, `{Flag}`; default `{field}{Flag}`), and `flags` composes with a repeating group, where the group's naming applies first
-- Added **pattern field keys** on a consumer: a `key` holding a `*` (`"addr*Street"`) selects every field whose name matches, and each match keeps the rest of that field's configuration — so one entry can alias, transform and validate a whole family of columns, which is what makes a repeating group's generated columns usable without writing one field per occurrence. Matching is literal apart from the wildcards, never a regular expression, and a pattern expands where it is declared, in the producer's own column order. Because several columns come out of one entry, `alias` becomes a template that must vary per match: `*` is replaced by what the pattern captured (positionally) and `{key}` by the matched field name, so `{ "key": "addr*Street", "alias": "street_*" }` yields `street_0`, `street_1`, `street_2`. A pattern that matches nothing, an `alias` that would give two matches the same output name, and a pattern combined with `fixed`/`copyFrom`/`lineCount` (fields that read no producer column) are all rejected at `compile`. A consumer reading another consumer's output sees the expanded names, not the pattern. The bare `"key": "*"` keeps its existing meaning
-- Added `emit: "rows"` on a repeating group, turning its occurrences into **records instead of columns**: one output record per occurrence, where the children keep their own names, every dimension outside the group repeats on each record, and a generated field (`occurrenceField`, default `{group}Occurrence`) carries the index. That is what makes a slot list usable as data — three addresses become three rows rather than `addr0Street`/`addr1Street`/`addr2Street` — and the records stay flat, so CSV/TXT export, `pivot`, `distinctOn`, transforms and validations all keep working. Occurrences whose fields are all empty are skipped by default (`skipEmptyOccurrences`), since a layout reserving a hundred slots typically populates a handful. At most one group per producer may fan out; a `lineCount` field keeps meaning the **source** line, so the records fanned out of one line share it, while dataset row counts (`min_rows`/`max_rows`) count output records. `sample` shows the same fan-out a run produces
-- Added **variable-length repeating groups** to a `FIXED` producer: `occurs` accepts an object when the number of occurrences changes from line to line, which is the one thing about such a layout that fails silently rather than loudly. `{ "max": 36, "min": 34, "while": { "position": { "start": 1, "length": 1 }, "equals": "R" } }` takes occurrences while a marker inside the occurrence matches; `{ "max": 61, "from": "$slotCount" }` reads the count from an earlier dimension. `max` is mandatory in both forms — it is how many columns the group generates, so the output shape is still known before a line is read — and a line carrying fewer occurrences reads the missing ones as `null` rather than as the characters of whatever follows the block
-- Added `position.after` on a `FIXED` producer dimension, anchoring a position to the end of an earlier group's **actual** occurrences on that line instead of to the start of the line: `{ "after": "reimb", "length": 18 }`. Everything past a group whose span varies needs it — with an absolute `start`, those fields read plausible, wrong values on every line but the longest. It replaces `start`, which becomes an optional offset inside the anchored region (`{ "after": "reimb", "start": 11, "length": 2 }`), and works on both groups and plain dimensions. Positions are still resolved once per producer: only a layout that actually has a variable group or an anchor is resolved per line, and only its dynamic part. Dangling, self- and forward-references, an anchor to something that isn't a group, a variable group without `max` or with neither/both count rules, and an `occurs.from` naming a group, an anchored field or a dimension declared after the group are all rejected at `compile`; a dimension left at an absolute column after a variable group is warned about instead, since a layout may legitimately end with one
-- Added `omitNull` on a `JSON`/`API` consumer output: fields whose value is `null` are left out of each exported record instead of being written as `"field": null`. On a wide, sparse record — a 1,300-field positional layout with a few dozen populated fields — that is most of the payload. Only `null` is dropped; an empty string, `0` and `false` are values and are exported. Records then legitimately differ in shape, so `omitNull` is **rejected** rather than ignored on a positional format (`CSV`/`TXT`), where a column can't be missing from a row, and on a `JSON` output whose consumer also writes one, since every output is serialized from the same record
-- Consumers now reject two fields that would write the same output field name at `compile`. A record is keyed by each field's final name (`alias ?? key`), so a collision never failed at run time — one field silently overwrote the other
-- Added `recordSelector.minLength` and `recordSelector.onLengthMismatch` for `FIXED` producers. A matching record-type marker only says a line *claims* to be that record type, not that all of it is there, so a truncated record was parsed into a plausible half-record whose missing tail read as `null`. A matching line shorter than `minLength` is now dropped and logged at debug level (`"skip"`, the default) or fails the run (`"fail"`). The check runs only on lines the marker already matched, so a short header or trailer record is unaffected
-- Added `remora package`, which builds the deployable `.zip` of the local configuration and writes it to disk instead of uploading it — `./remora-config.zip` by default, or wherever `-o, --output` points. It is byte-for-byte the archive `deploy` sends, so the same file can be handed to a worker as the `remora_config` upload, mounted into a container, or hosted where a worker reads `REMORA_CONFIG_URI` from — which is what a deployment needs when the machine holding the configuration cannot reach the worker directly (an air-gapped or CI-mediated environment), or when the package has to be reviewed or archived before it goes anywhere. The configuration is validated first, exactly as `compile` does, so a package is never built from a project that would fail to load. Entries are stored relative to the project root, and the run scratch directories (`temp`, `.temp`), log output and OS noise are left out — `deploy` now excludes them too, having previously shipped `remora/.temp` (debug logs and temp datasets) inside every upload. `deploy --build-only`, which until now was accepted and silently ignored, does the same thing as `package`
-- Added `synth.unique` on a producer dimension: the column emits a distinct value in every generated row. Until now only a `pk` did that, so a natural key, an external reference or an account number — a column with a uniqueness constraint that is not *the* primary key — came out with duplicates, and a consumer's `distinct`, a unique dataset validation or a join against it was tested on data that could never have come from the real source. A unique column is served from the coordinated key pool the way a primary key is, and walked by row: `external_ref_1`, `external_ref_2`, … That is also its cost, and it is deliberate — uniqueness is not something a name-based generator can promise, so `synth.unique` on `email` yields `email_1` rather than a realistic address. It applies to `string` and `number` columns, combines with `synth.min`/`synth.max` the way any key does (so a unique column can also be pinned to a width), and is rejected on a column whose value comes from `allowedValues`/`codeSet` (a closed set repeats by definition) or from `references` (a unique foreign key is a one-to-one relation, which `synth` does not generate yet) — each with an error naming the dimension
-- A `synth` run that asks for more records than the key pool can hold (100,000) now fails when something in the project has to stay distinct — a primary key, a `synth.unique` column, or a field another producer references. Past that point the pool started over, so a 150,000-record run silently emitted its primary keys twice and every join over them fanned out; the error says which field, and that the run has to stay at or under the cap. A project with nothing to keep distinct is unaffected and still generates any number of records
-- `synth.min` / `synth.max` now also apply to a **key/ID** column — a primary key, or a field whose name reads like an identifier — which was the one place a declared width still could not reach, and the columns most likely to have one (`CHAR(10)` member IDs, fixed-width claim numbers). A key's values come from a shared pool rather than a generator, so the bounds **size the pool** instead of clamping its values: `{ "min": 10, "max": 10 }` on `member_id` generates `memberid01`, `memberid02`, … — a name-derived prefix plus a sequence number, together exactly 10 characters, with the prefix giving way to the digits when the width is tight. Clamping was not an option: cutting `member_id_10` and `member_id_100` to a width would collide them and silently break the unions and foreign keys the pool exists to keep. On a `number` key the range works the same way, taken from its start (`{ "min": 500000 }` → `500000`, `500001`, …). Because the pool is shared, two things are now rejected with an error instead of quietly producing broken data: a width too narrow to hold one distinct value per generated record (it names the width needed), and two producers declaring different bounds for the same key name. A foreign key still cannot be bounded directly — its values *are* the parent's, so bounding the parent field it references is what sets its width, and the error says so
-- `synth.min` / `synth.max` now also apply to a **`string`** dimension, where they are the value's length in characters — the one part of a generated value that was impossible to control, and the reason a synthetic run could produce data that would not load into the column it was generated for. Both ends are inclusive and either can be given alone: `{ "max": 20 }` keeps a name inside a `VARCHAR(20)`, `{ "min": 8, "max": 8 }` pins an identifier to a fixed width. On a string the bounds **clamp** the generated value instead of replacing it the way they do on a number — a value over `max` is cut to it (and trimmed, so it never ends on a space), one under `min` is filled out with letters — so a bounded `city` column keeps generating cities rather than turning into random characters. Lengths must be whole numbers (`min` at least 0, `max` at least 1), and the existing rules still hold: bounds are rejected on a dimension whose value already comes from an `allowedValues`/`codeSet`/`references` binding or a shared key pool, and now on a `boolean` dimension, rather than being silently ignored
-- A deployed worker now writes its **operational state to stdout as single-line JSON**, which is the only channel available where the worker sits inside the customer's VPC and no developer machine can reach it. Every deployed container already ships stdout to the customer's log pipeline, so this needs no new infrastructure and no inbound network access: a `{"remora":"heartbeat",...}` record every 60 seconds carrying worker version, uptime, whether the configuration actually loaded (and the error if it did not), the config signature, resource counts, CRON jobs with their next fire times, queue mappings, in-flight executions with elapsed time, and heap/RSS — plus one `{"remora":"run",...}` record at the end of every execution with its outcome, row counts, duration and per-stage timings. The discriminator makes both greppable and queryable (`filter remora = "run" and status = "failed"`) without parsing surrounding text. Two rules are enforced in the emitter rather than left to callers: **no PHI** (keys naming row content or credentials are redacted, never printed, and anything URL-shaped loses its query string, where a presigned URL carries its credentials) and **bounded payloads** (depth, array length, string length and total line size are all capped, with an oversize record replaced by an explicit marker rather than dropped silently). `REMORA_STATE_INTERVAL_MS` sets the interval (default 60000, clamped to 5s–1h) and `REMORA_STATE_ENABLED` turns emission off or on; it is off in the CLI, where JSON lines would corrupt command output. Until now a worker that failed to load its configuration kept serving and passed its health check with zero resources loaded, and `/health` — which returns the bare string `OK` — could not express that
-- A queue message that keeps failing now **stops being processed**, instead of triggering a consumer run on every redelivery until the queue quietly discards it. A message is only deleted once a consumer has handled it successfully — which is what keeps a shared queue usable — but that also meant a message that can never succeed (a malformed payload, a consumer whose source is gone) was redelivered every five minutes for the queue's whole retention period: roughly a thousand failed runs before SQS silently dropped it. `maxReceiveCount` now bounds that for a consumer's `QUEUE` trigger, and defaults to 5 failed deliveries. What happens to the exhausted message depends on the new `deadLetterQueue` setting — a queue URL, or a name resolved against the queue's own `region`/`accountId`. **With one**, the message is copied there and then deleted from the source queue, in that order, so a dead-letter send that itself fails leaves the message where it is rather than losing it; the copy carries `remoraSourceQueue`, `remoraSourceMessageId` and `remoraReceiveCount` as message attributes, so it can be diagnosed and replayed. **Without one**, the message is skipped on every later delivery without running a consumer, and nothing is deleted — deleting a message Remora did not produce is not its call — so it stays in the queue for inspection and expires on the queue's own retention period. The delivery count is SQS's own `ApproximateReceiveCount` rather than an in-memory tally, so it survives a worker restart and is shared across every worker polling the queue; the skip list is per-process and *is* cleared on restart, which gives every message another chance once a fix is deployed. A message no consumer **claimed** is not a failure and is never dead-lettered or skipped: a `messageType` matching nobody still leaves it on the queue for other systems. The worker's state heartbeat reports `quarantinedCount`, since a skipped message is otherwise invisible
-- Every execution record now carries the **Remora version that ran it** (`engineVersion`) — in the usage collection, in the CLI's local `remora/.temp/usage/remora_usage.csv`, in the worker's `{"remora":"run",...}` stdout record and in an `ops history` entry alike. Run timings were only comparable to each other while the engine stayed still: a run that got faster or slower between two releases was indistinguishable from one whose input changed, so the version has to sit on the record itself rather than be inferred from when it ran. The column is appended last in the local file, so rows written before it existed keep reading correctly and simply have it empty
-- Added `remora graph`, which prints the project's lineage — what feeds what — to stdout: `-f mermaid` (the default) for a diagram that pastes straight into a README or a PR description, `-f dot` for graphviz, `-f json` for the machine-readable form. Until now the only way to see the shape of a project was to open four JSON files side by side and hold the references in your head; the graph draws every one of them, including the three that were previously invisible even in the frontend's canvas — where a consumer's data **lands** (`outputs[].exportDestination`, and a local file or API endpoint when there is no destination source), the **chains** between consumers (`onSuccess: run-consumer`), and the **triggers** that fire them (a CRON expression, an API path, a queue). A reference to something that does not exist is drawn rather than dropped, dashed and marked `not declared`, because a dangling reference is exactly what you opened the picture to find. Alongside it the command reports, on stderr so the diagram itself stays pipeable, every problem it found across resources: a producer reading a source nobody declared, a consumer reading something that is neither a producer nor a consumer, a field reading a producer its consumer does not list, an unloaded schema, an export destination or chain target that is missing, a cycle in the consumer chain (reported on every consumer on the loop), and — as warnings — a source nothing reads or writes, a producer no consumer uses, a schema nobody validates against, a consumer with no output at all. A project with errors still renders: the picture is wanted most when the configuration is wrong
-- The CLI progress bar now moves during **staging** — the download or table read that happens before a single record is processed, and on a large source the longest part of the run. A source driver reports its own progress as a fraction of that producer's work (`onProgress` on the driver's ready request), so a multi-gigabyte S3 object or a Delta Share table read advances the bar instead of parking it while the run looks hung; a driver that cannot measure itself never reports one and behaves exactly as before. The renderer also always draws a phase boundary and a completed phase rather than throttling it away, so the bar can no longer be left showing a stale fraction of a phase that has already ended
-
-### Changed
-
-- A **broken configuration no longer stops the project from being read.** `Environment.load()` is now one policy over a new `Environment.inspect()`, which loads everything loadable and reports the rest instead of throwing: `load()` takes the first problem and throws it, so the CLI and the worker still fail fast with the same message they always did, while a read-only consumer like `remora graph` renders everything that did load. The practical difference is a file that will not parse. That used to abandon the whole batch — one consumer caught mid-save and the entire project was unreadable, with a message naming the *directory* rather than the file — so now each unparseable file is one problem naming that file, and every other file still loads. A resource that fails its schema is **kept**, not dropped, so a consumer with one malformed field still shows where its data comes from; a producer whose fixed-width layout cannot be resolved is kept unexpanded rather than unloaded. That is what makes a picture of a project available exactly when the project is wrong, which is when it is most wanted
-- **A run now shares one worker pool across every producer, file and chunk**, instead of building and tearing one down per file — which is what made a project of ordinary-sized files run single-threaded no matter how many cores the machine had. A file under 10MB is never split, so it becomes exactly one chunk and one worker, and because files were processed one at a time that meant one worker at a time for the whole run: 30 files of 6MB overlapped **0.35x**, i.e. the wall clock was every thread spawn, every configuration load and every worker's work laid end to end. Each of those spawns also loaded the 2.4MB worker bundle again, so a 30-file run paid 30 spawns serially where 9 threads would have covered it. The whole work list — every (producer, file, chunk) — is now resolved before any thread exists and submitted at once, and the pool schedules across it: **that 30-file run went from 18.7s to 2.7s (7x), overlapping 4.7x on 9 threads instead of 30 spawned one after another**, and it now finishes within 1.2x of the same data arriving as a single large file, where it used to take 8.5x longer. Nothing about how a file is split changed, and nothing about how many workers may run at once changed either: the pool is sized by the same memory-aware worker count as before (CPUs × 0.7, capped by available memory against each worker's 2000MB heap allowance), and work beyond that **queues instead of spawning threads** — so peak concurrency on a small machine is identical, 1 worker before and 1 worker after on a 4GB box. What is bounded for the first time is how much a run may queue: 100,000 chunks, with an error naming the consumer rather than a failure part-way through dispatch. Row numbering is unaffected — a `lineCount` field still numbers each source file from its own first line, even though chunks from different files now finish interleaved
-- A worker thread now **receives the loaded configuration when it starts**, instead of reading and validating the whole project from disk again for every chunk it runs. That load — walk the project directory, parse every source, producer, consumer and code set, validate each against its schema, expand every repeating group — used to be charged once per chunk, so it was the one startup cost that did not shrink when a run started sharing one pool of threads: a 30-file run paid it 30 times on 9 threads, and a machine constrained to a single thread paid all 30 of them one after another. The orchestrator now snapshots the configuration it has already loaded and hands it to each thread once, which takes the configuration load of a 30-chunk run from **1.51s of worker time down to 28ms** — and on a single-threaded machine, where none of it could be overlapped, the same run went from **7.02s to 6.40s**. The saving grows with the number of chunks, because what it removes is the part that grew with them. Independently of speed, this also means a run can no longer straddle two versions of the configuration: re-reading the project mid-run meant a configuration edited while a run was in flight was silently picked up by whichever chunks had not started yet, so different chunks of one output could be produced by different configurations
-- `REMORA_MAX_WORKERS` caps how many worker threads a run may use. It can only ever **lower** the computed count, never raise it: the count is derived from available memory against each worker's heap allowance, and raising it past that bound is precisely the oversubscription the bound exists to prevent — an 8-core/4GB machine given 7 workers each allowed 2000MB is oversubscribed by construction, which is the fatal error the memory-aware count was introduced to fix. Its two uses are capping Remora where it shares hardware, or runs in a container with less memory than its host, and making a constrained machine reproducible without owning one. A value that is not a whole number of 1 or more is rejected with an error rather than ignored
-- A **failed `API_QUEUE` command now leaves the queue** instead of being redelivered. A message on the project's API queue is a command — it asks for one consumer run — not an event to be replayed, so it is removed once it has run, whether the run succeeded or failed: it had already run by the time it failed, and a redelivery would be a second execution rather than a resumption of the first. The failures that actually happen here are not ones another delivery fixes either (a `consumer` naming something that no longer exists, a malformed payload, a run that broke halfway), and the retry that used to follow meant a single bad command ran the pipeline again every five minutes until its deliveries were exhausted. `API_QUEUE.deadLetterQueue` decides where the failed command goes: **with one**, it is copied there — carrying `remoraSourceQueue`, `remoraSourceMessageId` and `remoraReceiveCount`, so it can be diagnosed and replayed — and then deleted from the API queue, in that order, so a dead-letter send that itself fails leaves the command in place rather than losing it; **without one**, it is deleted outright and the worker's log is the only trace, which is why configuring one is worth it. `API_QUEUE.maxReceiveCount` goes with the retry it used to bound, since a command is never delivered a second time. A consumer's `QUEUE` trigger is unaffected and still retries, dead-letters and quarantines as before — the difference is exactly that a trigger message is an event several systems may react to, while an API queue message is an instruction addressed to Remora
-- `create-consumer -p <producer>` now carries the producer's documentation over to the consumer it generates: the producer's own `description` becomes the consumer's (instead of a generic `"Consumer for <producer> data"`), and each dimension's and measure's `description` becomes the corresponding field's. What a column means is written once, on the producer, and no longer has to be retyped on every consumer built from it. A producer or field with no description is left without one, as before
-- `schemas` is now **optional** in `project.json`, and a declared schema path that does not exist on disk is skipped rather than failing the load. Custom JSON schemas are only needed by a consumer that validates against one, so a project that has none no longer has to carry an empty `remora/schemas` directory to start — `compile` stops requiring the directory too
-- **A consumer's per-record work is now compiled once per worker** instead of being re-derived for every record. Mapping a record used to make five sequential passes over the field list and, inside them, a `dimensions.find` per field and a `fields.find` per dimension — O(fields × dimensions) twice, per row — plus two `delete`s that push the record object into V8's dictionary mode and cost every later property access on it, and `alias ?? key` recomputed three times per field. `RecordPlan.compile` now resolves all of that once into a flat op list with the dimensions already looked up, a plain passthrough field compiling to no op at all, and the transform and validation passes walking only the fields that declare one. `process-record` fell **3.69s → 2.62s (-29.1%)** on a single large file and **3.53s → 2.49s (-29.5%)** across many small ones, summed across workers, with row counts and output bytes unchanged. The plan reproduces the old bookkeeping symbolically rather than just listing the final keys, because the deletes were observable twice over: they set the record's key insertion order, which *is* the key order of JSON output, and they decided which keys came out null. One behaviour is deliberately not preserved — a transformation referencing a renamed-away dimension **dynamically** (`$name`, `combine_fields`, `multiplyBy`) used to throw "field absent from the record" and now resolves to the value, which can only affect a consumer that fails on every record today
-- **Output serialization is compiled once per run** as well. `outputRecord` re-derived the internal record format per record, `toJSON` recomputed whether any output sets `omitNull`, and `toCSV` allocated a filtered visible-fields array — all three run-constant — while `stringifyRow` built each line with a `map`, a template literal, an **unconditional** `replaceAll('"', '""')` and a `join`. The format, the `omitNull` decision and the visible/hidden key lists are now resolved once, and `CSVParser.escapeField` carries the escaping rule behind an `indexOf('"') === -1` fast path, with `stringifyRow` a manual loop over it. `output-record` fell **1.65s → 813ms (-50.6%)** summed across workers, byte-identical output pinned against the expression it replaces over a fixture of awkward fields (bare quote, doubled quote, quote at either edge, embedded delimiter, newline, null). JSON still deletes its hidden keys per record rather than listing the visible ones out, because a JSON record serializes in the record's own key order and an aliased field is appended — writing the field order instead would change the bytes
-- **A worker now writes its output in ~256KB batches** rather than calling `write` once per record: about **1,000,000 calls down to ~800** on a million-row run, and the throwaway `line + '\n'` string per record gone with them. The drain contract is unchanged — `write` returns false exactly when the caller must await, which is what keeps this from costing a microtask per record and giving back what the batching bought. Stated plainly: **the wall clock did not measurably move** (+0.8% and -4.9% on the two scenarios, both inside this harness's noise floor), because ~140ms of summed worker time per million rows sits below what these fixtures can resolve. What it buys is fewer syscalls and one fewer allocation per row, which scales with row count rather than with file count. The stream keeps Node's default high-water mark: raising it takes the drain count to zero, which is cosmetic — it moves the threshold rather than queueing less work — and measured slightly slower for a megabyte more buffered per worker
-- **The per-line work is compiled once per chunk too**, and where a producer needs none of it the pass is gone entirely. The list of masked and `sourceFilename` dimensions was rebuilt **on every line** — two array allocations plus a `getMask()` per dimension, each allocating its own list of allowed values to `.includes` against — and `path.basename(fileUri)` was recomputed per record on top of it, none of which changes across a whole chunk. `ProducerExecutor.compileLine` resolves one fill op per dimension that actually needs a file name or a mask, and an empty plan returns the parsed records straight out, so a producer with neither — most producers — now pays neither the loop nor the timing pair around it. `process-line` fell **-11.5%** on a single large file and **-8.2%** across many small ones, and the mask/filename operation no longer appears in the report at all. Mask validation moved with it: a masking mode Remora does not implement is now rejected at load and by `verify`, naming the dimension and the resolved value, instead of failing part-way through a run — the producer schema cannot do this because the value may be a `{VAR}` reference that only resolves at runtime. `mask: "none"` still compiles to a fill even though it hashes nothing, since it casts the value to a string and dropping it would silently unquote numbers in JSON output
-
-### Removed
-
-- Removed `filters[].sql` from a consumer. It was never read: nothing in the pipeline compiled it, pushed it to a source, or evaluated it, so a consumer whose filters were all `sql` exported its full result set while the configuration said it was filtered. Only `filters[].rule` ever did anything, and it still does — every filter must now declare one, and a filter with neither is rejected at `compile` rather than crashing mid-run. The `"sql"` filter operator goes with it, for the same reason: it was accepted by the schema and then thrown on as an unsupported operator by the only code that reads operators. Rewrite a `{ "sql": "${C.amount} > 1000" }` filter as `{ "rule": { "member": "amount", "operator": "greaterThan", "values": ["1000"] } }`
-- Removed the `mock` command. `synth` covers what it did and does it better: `mock` wrote a producer's source file from an unseeded `Math.random()`, so no two runs produced the same data and nothing generated could be replayed, while `synth` generates seeded, deterministic input with realistic values, code sets and referential integrity between producers, and runs the real pipeline on it. Generating a producer's source file on disk is the one thing that goes with it — `synth` feeds the pipeline directly, so there is no fixed-width writer any more either. Use `remora synth [consumer] -r <records> -s <seed>` instead
-
-### Fixed
-
-- **JSON schema validation now actually runs when a project loads.** `SchemaValidator.validate()` returns `{ isValid, errors, data }`, and all five call sites in the loader tested that object for truthiness — `if (!SchemaValidator.validate('producer-schema', producer))` — which is never true. Every one of the resulting throws was unreachable, so **no configuration has ever been rejected by its own JSON schema at load time**: the schemas documented a contract that nothing enforced, and a producer with no dimensions, a source missing its `engine`, or a consumer with a misspelled property loaded silently and failed later somewhere less obvious, or not at all. Validation now runs on every source, producer, consumer, code set and on `project.json` itself, at `compile`, `run`, `deploy` and on a worker's config load, and the message names what is wrong rather than only what is invalid — `Invalid producer configuration: p_orders (/dimensions must NOT have fewer than 1 items)`. Resources are checked **as authored**, before repeating groups and flag bitmaps are expanded, since expansion generates the properties the schema forbids; and where a fixed-width layout is wrong in a way both checks would catch, the expander's message wins, because "a variable group must declare a whole-number maximum" says what to do where "must have required property 'max'" sends you looking. **This can reject a project that used to load.** A configuration that is schema-invalid now fails at `compile` — which is where it should have been failing all along — and a deployed worker that reloads such a configuration keeps serving with none, exactly as it does for any other bad config. Run `remora compile` before upgrading; `remora graph` reports the same problems without failing, one per resource, if you would rather see all of them at once
-- **A filter on a `number`, `datetime` or `boolean` field now actually matches.** A filter's values are always text — they come out of a JSON config, a URL query or a queue message — while a record holds what the producer cast the column to: a `number` dimension holds a number, a `boolean` one a boolean, a `datetime` one a UTC ISO string. The record evaluator compared the two as they arrived, so `5 === "5"` and `true === "true"` were false for every row: `equals`, `notEquals`, `in` and `notIn` on any non-string field **returned nothing at all** (or, for the two negations, everything), and the four `greaterThan`/`lessThan` comparisons were hard-coded to numbers, so they dropped every row of a date or text column. Each side is now read as the type the record holds before being compared. `"150"` matches a `monthly_spend` of `150`; `"true"`, `"false"`, `"1"` and `"0"` match a boolean; an ISO date matches a `datetime` **as an instant**, so `"2024-03-01"` matches the record's `2024-03-01T00:00:00.000Z` and a bound carrying an offset (`2024-03-01T11:30+02:00`) means the moment it names rather than the characters it is written with — read as UTC where it carries no offset, so a filter means the same thing on every host. A date is only read as a date where **both** sides spell it ISO-8601, so a column in a producer's own format (`"05/03/24"`) still compares as the text it is rather than being handed to `Date` to guess at. Ordering comparisons now work on text and dates as well as numbers, which is also what the source-side predicate has always assumed. The string operators (`contains`, `startsWith`, `endsWith`, `matches`) read a non-string value's text — a number's digits, a date's ISO form — instead of failing on the type, and their negations now keep a row whose value is null, consistently with `notEquals` and `notIn`. Two things that follow, worth knowing: a date bound naming a day means that day's **first instant**, exactly as a SQL date literal does, so `between ["2024-03-01", "2024-03-31"]` stops at the start of the 31st — give a time (`"2024-03-31T23:59:59"`) to include it; and a value that cannot be read as the record's type (`equals` `"abc"` on a number) matches nothing rather than throwing. **This can change what a consumer returns**: one whose filters were silently matching no rows will now return the rows it was written to return
-- A consumer on a **SQL source can now filter at all**. `compile` rejected rule-based filters on a SQL engine ("Filters based on rules are only valid for non-SQL based sources"), leaving `filters[].sql` as the only accepted form — and that one was never evaluated, so the two rules together meant a Redshift-backed consumer either failed to compile or ran unfiltered, with no way to express a working filter. Filters are applied per record to the result set after the producers are read, which is engine-agnostic, so the engine restriction described a pushdown that does not exist. Rule filters are now valid on every source
-- Fixed day-of-year ("Julian") dates parsing to a wrong date with no error. dayjs has no parse token for a day of year at all — `DDD` exists only for formatting — so `"YYYYDDD"` either failed as invalid (strict, the default) or, with `"strict": false`, consumed the `YYYY`, ignored the rest and read `2007305` as **2007-01-05** instead of 2007-11-01: silent corruption of every ordinal date in a mainframe feed. `DDD`/`DDDD` are now parsed as a 3-digit day of year, in both a producer dimension's `format` and a consumer's `date_format` transform, and combine with `YYYY`/`YY`, `HH`, `mm`, `ss`, `SSS` and literal separators. A day of year outside its year (366 in a non-leap year) and a format mixing an ordinal day with a calendar `MM`/`DD` are both rejected with an explicit error rather than resolved to something plausible. `synth` renders the same format correctly too, zero-padded to 3 digits so what they write can be read back by the layout that declared it
-- A date format built on a token dayjs can only *write* and never read (`Do`, `Q`, `w`, `W`, `k`, `gggg`, `z`) is now rejected with an error naming the token, instead of being parsed loosely into a wrong date. Bracket it (`"[Q]YYYY"`) if it is meant as literal text
-- Fixed a CRON job's reported status always reading as *not running*. The scheduler compared the task state against `scheduled`, a state name from node-cron 2.x; the installed node-cron reports `idle`, `running`, `stopped` or `destroyed`, so `isRunning` was false for every job including the ones firing normally — in the worker's own CRON status API and anything built on it. It is now derived from the real states, and each job also reports its raw `state`, the consumer and expression it came from, and its next fire time, so a job that is scheduled but will never fire is distinguishable from one waiting to run
-- **A run's record limit is now the whole run's, not each worker's.** `remora run -l 10` (and a request's `limit`) was applied inside every worker against its own chunk, so a consumer whose producer split into nine chunks returned **ninety** rows — nine separate blocks of ten, one from each chunk's start — and the count scaled with the machine: the same command returned a different number of rows on a box with more cores, and `REMORA_MAX_WORKERS=1` was the only way to get the limit asked for. Multiple producers multiplied it the same way. Each worker still stops reading at the limit within its own chunk, which is what keeps a limited run from reading a whole table, but the run-wide cut is now made once on the merged dataset — after `distinct`/`distinctOn`/`pivot`, so a limit is never met by rows those passes were about to remove, and before dataset validation and export, so both see exactly the rows returned. The chunks are merged in source order, so the rows kept are the first `limit` rows of the result, the same ones a single-threaded run has always returned
-- **A numeric filter pushed to a Delta Share now actually prunes anything.** Every `number` dimension's literals went out typed as the protocol's `double`, but a share server only prunes when the type it is given is the column's own — hand an `integer` column a `double` and Databricks quietly ignores the hint and returns every file. So the predicate was well-formed, accepted, logged, and worth nothing: a filter on a numeric column downloaded the whole table exactly as it did before pushdown existed, while string filters (whose one type is unambiguous) pruned correctly. Each column's literals are now typed from the share's own schema, read once per table from the table `metadata` endpoint and only when there is a filter to push, so `integer` goes over as `int`, `long` as `long` and a genuine `double` as `double`. Columns the protocol's value types cannot express (`decimal`, and the nested types) and columns whose Delta type disagrees with the producer's own dimension type are left out of the tree rather than sent under a type that would make the server and remora prune different rows, and a schema that cannot be read costs the predicate and nothing else. A filter that now prunes **every** file also no longer fails the run: an empty read is still the misconfiguration error it has always been, except where the run itself asked the source to filter, in which case no rows is the answer and the consumer exports an empty result — the same thing it produces when remora applies that filter per record
-- **A fixed-width producer whose first field is space-padded no longer reads every column shifted.** Each line was `trim()`ed before parsing, but a `FIXED` dimension is a character range into the line, so leading padding — which is exactly what a right-aligned numeric in column 1 looks like — moved every position after it and each dimension read the wrong bytes. The record selector was already matched against the **untrimmed** line, so the two had disagreed all along. The trim is gone, and it was redundant everywhere else: `CSVParser.parseRow` trims each unquoted field and breaks on `\r`/`\n`, `FixedWidthParser.readField` trims each field it slices, and `JSON.parse` ignores surrounding whitespace. One line changes outcome as a result — a **whitespace-only** line in a CSV file, which used to trim to `''` and fail the entire run, now parses to a record of nulls, which is what a `FIXED` producer has always done with it
-
-## V 1.8.0 - 2026-08-05
-
-### Added
-- Added `synth.min` / `synth.max` on a producer dimension to bound the values `synth` generates. Both ends are inclusive and either can be given alone, in which case the other spans the default range's width from it. On a `number` dimension they replace the name-based generator, so a column whose real-world range the field name can't imply (`"synth": { "min": 18, "max": 90, "decimals": 0 }`) stays inside it instead of getting a generic 0–1000 draw; values stay whole unless `decimals` asks for more, except where the range spans less than one whole number (e.g. `0.4`–`0.6`), which is generated fractionally so it doesn't collapse to a single value. On a `datetime` dimension they are date strings that replace the default 2020–2024 window — written the way the column's own values are, so with the dimension's `format` when it declares one (`"min": "20240101"` for `"YYYYMMDD"`) and as a plain/ISO date otherwise. A date bound that names a day covers that whole day, so `20240101`–`20240131` includes the 31st rather than stopping at its first instant; give a time to end the window at a precise one. Bounds apply only to `number`/`datetime` dimensions and only where the value is actually generated: setting them alongside `allowedValues`, `codeSet` or `references`, or on a key field served from a shared key pool, is rejected with an error naming the dimension rather than being silently ignored, as are an inverted range and a bound of the wrong kind for the dimension's type. `synth` only; no effect on a real producer read
-- Synthetic `datetime` values are now rendered in UTC, the timezone their bounds are parsed in, instead of the host's local timezone. A formatted date (e.g. `"format": "YYYYMMDD"`) generated west of UTC could otherwise render as the previous day — which would also put it outside a declared `synth` window — and the same seed produced different output on machines in different timezones
-- Added a new producer input type for **fixed-width (positional) flat files**: `"fileType": "FIXED"`. Such a file has no delimiter and no header, so each dimension declares where its value sits in the line with `position` — `{ "start": 21, "length": 10 }` or `{ "start": 21, "end": 30 }`, 1-based and inclusive, the way a layout spec is written ("columns 21-30"). Values are trimmed, so space- and zero-padded fields read as their real value (`"0000123.45"` → `123.45`) and an all-padding field reads as `null`; gaps (unmapped filler columns) and overlaps (a composite field plus its subfields) are both allowed; `format` still applies, so a date written as bare digits parses with `"format": "YYYYMMDD"`; and a `sourceFilename` dimension needs no position. Works everywhere a flat file already does — local and S3 sources, wildcard `fileKey` patterns, compression, chunked parallel reads, `sample`, `mock` and `synth`
-- Added `settings.recordSelector` for flat files that interleave record types in one file (a header, many detail records, a trailer). It reads only the lines whose marker at a given position matches — `{ "position": { "start": 1, "length": 1 }, "equals": "D" }`, or a list of values — so one producer describes exactly one record layout and every other line is dropped before parsing. Applied per line, so it is unaffected by how a file is chunked across workers
-
-### Fixed
-- Fixed a deployed worker producing no logs. The whole run pipeline logged on the debug channel, which is dropped at `prod` level, so a deployment that did not set `REMORA_DEBUG_MODE=true` emitted nothing at all beyond startup lines and errors. The operational lifecycle events of a run are now logged at `INFO`, so they appear without turning debug on: worker-pool startup, consumer launch/completion, producers skipped as optional or empty, export start/finish per destination, downstream (`run-consumer`) triggers and cycle skips, and CRON/queue scheduler setup, per-trigger execution and stats. Per-line, per-chunk and per-worker detail stays on the debug channel, so `REMORA_DEBUG_MODE=true` is still what you set to see it. These lines follow the log file's existing exclusivity, so a deployment that enables file logging still gets a quiet stdout: `INFO` now goes to the file *instead of* the console (previously it went to both), while `warn`/`error` always stay on the console so a failure can never be silent
-- Fixed the CLI progress bar erasing log lines. The bar redraws in place by moving the cursor up one line, which overwrote whatever had been printed since its last frame — now that the run pipeline logs at `INFO`, that was the log messages themselves. The logger counts what it writes to the console and the renderer only reclaims the previous line when it is still its own bar, so a log line printed mid-run survives and the bar continues below it
-- Fixed the periodic configuration refresh ignoring IAM-role authentication for S3. `Environment.refreshIfNeeded` still fetched the config archive over plain HTTP even for an `s3` URI, so a deployment loading its project from S3 with the ambient IAM role (added in 1.7.0) succeeded at startup and then failed on every refresh. Refresh now takes the same S3 path as the initial load; non-S3 remote URIs keep using the HTTP download with ETag caching
-- API queue execution failures now include the underlying error text in the log message, not just the queue URL, matching the poll-failure logging fixed in 1.7.0
-- Fixed JSON/JSONL producer dimensions ignoring their `format` when casting. The flat-file read path passed the dimension's `format` to the type cast but the JSON path did not, so a date written as a formatted string (e.g. `"20211213"` with `"format": "YYYYMMDD"`) failed to parse with `Invalid time value`. Both paths now cast identically
-- Fixed the `sample` command emitting the header row of a CSV/Excel producer as if it were a data row when `hasHeaderRow` was not set explicitly. Deciding where the data starts is now one shared rule for the run path and the sample path, so the two can no longer disagree
-- Fixed a multi-file (wildcard `fileKey`) read of a headerless flat file failing with a "different header line than the other files" error: the first line of each file was compared as a header even for file types that don't have one, where it is data and legitimately differs
-
-## V 1.7.0 - 2026-07-29
-
-### Added
-- Added IAM-role authentication for loading the project configuration from S3. When `REMORA_CONFIG_URI` (or the URI passed to `Environment.loadFromUri`) points at an S3 URL, the config zip is now fetched with the AWS SDK using the ambient IAM role — the ECS task role or instance profile — instead of a plain HTTP GET, so a deployment no longer needs a presigned URL or static access keys. Bucket, key and region are derived from the URL, and the archive is extracted into the local cache path exactly as the HTTP path does (both share `_saveConfigFile`). Non-S3 remote URIs keep using the existing HTTP download with ETag caching
-
-### Changed
-- **Breaking (config)**: grouped every synthetic-data property of a producer dimension under a single `synth` object. `decimals`, `allowedValues`, `codeSet` and `references` move from the dimension's top level into `synth`, e.g. `{ "name": "gap_status", "type": "string", "synth": { "codeSet": "gap_statuses" } }`. These properties only ever affected the `synth` command, so nesting them makes that structural instead of a caveat on each one, keeps the dimension's real-read properties uncluttered, and gives future synth-only options one obvious home. Behaviour is unchanged. A property left at the dimension's top level is now rejected with an error naming the dimension and the property to move (both at `compile` and at the start of `synth`), rather than being silently ignored — and the producer schema rejects unknown keys inside `synth`, so a typo like `codeSets` fails in the editor
-- Replaced the `decompress` package on the archive-reading path. ZIP extraction now uses `adm-zip` and TAR extraction uses `tar`, both behind explicit path-traversal guards: every zip entry is resolved and asserted to stay inside the target directory before anything is written (zip-slip), and `tar.x` strips leading `/` and `..` segments. `decompress` had no patched release for CVE-2026-53486 (arbitrary file read via a crafted archive), so replacement was the only fix. `GZ` handling is unchanged
-- Upgraded the file-parsing and HTTP dependency surface off known-vulnerable versions: `xlsx` → `@e965/xlsx` 0.20.3 (prototype pollution + ReDoS on untrusted Excel), `multer` 2.0.0 (8 DoS CVEs on the Express upload path), `fast-xml-parser` 5.3.5, `adm-zip` 0.6.0, plus root `overrides` pinning `undici`, `form-data` and `path-to-regexp`. Unused packages were dropped from the published CLI and root dependency sets (`next`, `bcrypt`, `geist`, `react-ui`) so they no longer ship or get scanned. See [CVE_REMEDIATION.md](docs/CVE_REMEDIATION.md) for the scan, per-image triage and remaining items
-- Hardened the three deployed Docker images. Backend and worker now install with a workspace-scoped `npm install` followed by `npm prune --omit=dev` in the same layer, so build tooling (tsc, eslint, vite, tsup, nodemon) no longer ships to production; the base image's bundled global `npm` is patched; both drop to the non-root `node` user; and all three declare a `HEALTHCHECK`. `.dockerignore` now excludes `apps/**/testdata` and `apps/**/test`, which alone removed roughly 1 GB of image weight
-- Queue polling failures now include the underlying error text in the log message instead of only the queue URL, so a poll failure is diagnosable from the log line alone
-
-### Fixed
-- Fixed consumer field transformations rejecting a legitimately `null` referenced field. A `"$fieldName"` operand only requires the field to be *present* in the record: a `null` value is real data and is now passed through to the operation, while a field genuinely absent from the record still throws so the field's `onError` policy applies. Previously both cases threw, so a null in a referenced column failed the row
-- Fixed the worker throwing `new error(...)` — a lowercase import from `node:console` — when the licence key is missing or invalid, which raised a `TypeError` instead of the intended "Invalid Remora licence key" message. It now throws a proper `Error`
-
-## V 1.6.0 - 2026-07-21
-
-### Added
-- Added an optional `decimals` property on producer dimensions to control how many decimal places generated `number` values get for synthetic-data generation (`synth`). `decimals: 0` forces whole integers (so values load into an `integer` column), and a positive value rounds to that many decimals — so generated numbers load without decimal-into-integer failures. Metadata only; no effect on normal producer reads
-
-### Fixed
-- Fixed synthetic-data generation (`synth`) emitting `YYYYMMDD` integer dates into rate/proportion/percentage columns. A field name such as `..._year_to_date_percentage` contains the substring `_date` (inside `to_date`) and was matched by the date generator, overflowing e.g. a `numeric(10,4)` column. Rate/adherence metric topics (`percentage`, `proportion`, `days_covered`, `pdc`, `probability`) are now matched ahead of the date topic, so these columns get a proportion/percentage value instead of a date
-
-## V 1.5.0 - 2026-07-15
-
-### Added
-- Added the `synth` command, which runs the full pipeline on generated fake data: `remora synth [consumer] [-r <records>] [-s <seed>] [-p <project>]`. It generates synthetic INPUT for every producer (one JSONL file per producer, keyed by each dimension), then executes one consumer, all consumers of a project (`--project`), or all consumers through the REAL execution path — filters, field transformations, unions, distinct, pivot, measures, dataset validation and destination export — exporting to their real destinations. Generation is fully deterministic: the same `--seed` reproduces the same input and therefore the same output. Only producer-side source parsing and masking are not exercised, since records are injected already-parsed at the producer boundary. `--records` defaults to `100`, `--seed` to `remora`
-- Added code sets: a new top-level, optional project resource that constrains a producer dimension's synthetic values (used only by `synth`). Point `project.json` at a folder with `"codeSets": ["/code_sets"]` and place one JSON file per set. A set is either a **flat** list (`{ "name": "gap_statuses", "values": ["OPEN", "CLOSED", "COMPLIANT"] }`) or a **structured** table of correlated rows (`{ "name": "hedis_measures", "records": [{ "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" }] }`). A dimension binds via `codeSet` — a string names a flat set (one value picked per row), an object `{ "name", "field" }` binds to a field of a structured set so correlated columns land on the same row. A dimension can also inline a one-off list with `allowedValues` (mutually exclusive with `codeSet`). Validated against a new `code-set-schema.json`
-- Added a library of built-in code sets and healthcare data generators for realistic synthetic data. Built-in sets (HEDIS measures, code systems, gap statuses, product types, specialties, genders, indicators, and more) are applied automatically by field name with no configuration, and a user set of the same name overrides its built-in counterpart. Open-ended generators produce format-correct identifiers and clinical values by field name — Medicare Beneficiary Identifier (MBI), NPI, ICD-10, CPT/HCPCS, NDC, HL7 OIDs, and clinical ranges (HbA1c, systolic/diastolic, etc.)
-- Integrated `@faker-js/faker` to back synthetic generation: each field picks a realistic generator from its declared type and name (email, first/last name, city, phone, price, dates, …), seeded per-cell so output stays deterministic and reproducible regardless of worker order
-- Added the `mock` command, `remora mock <producer> <records>`, which generates a mock source data file for a producer directly from its dimensions (distinct from `synth`, which runs the whole pipeline on synthetic input rather than writing a producer source file)
-- Added a `references` foreign-key declaration on producer dimensions for synthetic data generation (`synth`): `{ "name": "patient_id", "type": "string", "references": { "producer": "p_members", "field": "member_id" } }`. Generated values are drawn from the referenced parent field's key domain, so a consumer's joins/unions over the two producers always line up (no orphan foreign keys) — even when the foreign key and parent key have different names. The referenced parent field is generated as a unique key (a warning is logged if it isn't a `pk`). Single-column relationships only; mutually exclusive with `allowedValues`/`codeSet`. Deterministic under a fixed `--seed`. Canary coverage added (`p_canary_ref_parent`/`p_canary_ref_child` + `c_canary_ref_child`, asserting 0 orphan keys through the real pipeline)
-- Added a descriptive `nullable` flag on producer dimensions (`{ "name": "note", "type": "string", "nullable": true }`). It documents whether a field may hold null/missing values for consumers, schema generation and AI mappers, but is metadata only — it does not change runtime behaviour (null/missing source values are already passed through untouched). Enforce presence with a consumer field `required` validation when a hard check is needed
-- Added an optional `description` property on consumer fields (`{ "key": "total", "description": "Order total in cents" }`) for documenting field intent. Descriptive only, no runtime effect
-- Added a `run-consumer` post-processing action for consumer output `onSuccess` / `onError`: `{ "onSuccess": [{ "action": "run-consumer", "consumers": ["c_downstream", "c_notify"] }] }`. Listed consumers run sequentially in declared order as fresh full runs once the output finishes (on the success or failure path). A downstream failure is logged and swallowed, so it never changes the parent consumer's result or stops the rest of the chain. Runtime cycle detection skips a consumer already on the chain, and a depth cap (`MAX_ON_FINISH_DEPTH`) bounds nesting; `Validator` also rejects unknown/empty targets and static cycles at compile time. Downstream runs are attributed to a new `ON_FINISH` invocation origin
-- Added canary coverage for the `run-consumer` action (`c_onsuccess_run` and `c_onerror_run` triggering `c_onfinish_downstream` on the success and failure paths)
-
-### Changed
-- `remora init` is now interactive: it prompts (via `inquirer`) for the project name and description, seeds a starter project, and prints guidance on the required `REMORA_LICENCE_KEY` environment variable
-- Extended the consumer JSON schema `consumerOutputOnFinish` definition with the `run-consumer` action and its `consumers` list
-- The CLI progress renderer now no-ops instead of crashing when stdout is not a TTY (piped output, CI, Docker logs), where cursor-control methods are unavailable
-
-## V 1.4.0 - 2026-07-08
-
-### Added
-- Added support for using another consumer's full output as a consumer input: name a consumer in a consumer's `producers` array and its output is streamed in as a source, e.g. `{ "producers": [{ "name": "c_upstream" }] }`. The upstream consumer is executed through the normal pipeline, its output captured to disk as JSONL, and read back through a synthetic producer, so arbitrarily large outputs are chunked across workers rather than held in memory (unlike `code_lookup`, which stays a bounded in-memory table). Producers are unioned; nested chains work, with depth/cycle detection (`MAX_CONSUMER_INPUT_DEPTH`) and per-run caching so a referenced consumer is materialized at most once per run
-- Added `ConsumerInputResolver` (executors): before a consumer's workers spawn, it detects `producers` entries that reference a consumer, executes each referenced consumer once (capturing its merged output as JSONL under the run's scope), and exposes it as a synthetic JSONL producer. `ExecutorOrchestrator` gained a `captureFormat` capture path (CSV internal output is reprojected to JSONL; JSON output is copied verbatim) and a synthetic-producer branch in `readySourceFiles`
-- Added the `multiplyBy` and `addBy` field transformations, which multiply/sum a field by the values of several other fields: `{ "multiplyBy": { "fields": ["qty", "price"] } }` and `{ "addBy": { "fields": ["base", "bonus"] } }` — distinct from `multiply`/`add`, which take a single literal or `$fieldName` operand
-- Added `TXT` as a consumer output format with a configurable single-character `delimiter` (defaults to a tab), mirroring the producer-side TXT delimiter: `{ "outputs": [{ "format": "TXT", "delimiter": "|" }] }`
-- Added `onSuccess` and `onError` post-processing actions on consumer outputs. The `move-file` action relocates the consumer's static source input file to another Remora source once the run finishes, on both the success and failure paths: `{ "onSuccess": [{ "action": "move-file", "moveToDestination": "s_archive" }] }`
-- Added the `oidc` authentication method for Delta Share sources (OAuth2 client-credentials via `clientId` / `clientSecret` / `tokenUrl`), extending `ISourceAuthentication` and the source JSON schema
-- Added canary coverage for consumer-as-input (`c_ci_base` → `c_ci_derived` → `c_ci_chain`) with a value-level correctness check (`verifyConsumerInputValues`) that re-derives the expected rows from the source, plus a `c_canary_extract` consumer and a `verifyExtractValues` check for the extract transform
-
-### Changed
-- Extended the consumer JSON schema with the `multiplyBy` / `addBy` transforms, the `TXT` output format and its `delimiter`, the `onSuccess` / `onError` output actions, and consumer names as valid `producers[].name` entries
-- Enhanced HTTP API source configuration in the source JSON schema (widened auth/request-method enums and added API source detail/examples)
-- Consumer producer/source validation now accepts consumer inputs: an entry that resolves to a consumer rather than a producer is treated as a sourceless, local (file) input, and source/engine-uniqueness checks apply only to real producers
-- Internal cleanup: removed a swath of unused code to reduce surface (`EventsEngine`, `JWTManager.verifyCLI`, `LocalSourceDriver.moveFile`, `DatabaseEngine.disconnect`, unused `UserManager` methods, and dead SQL-library / usage-stat interfaces)
-
-### Removed
-- Removed the deprecated `aws-dynamodb` and `postgres` source engines from `DataEngineType` and the source JSON schema
-
-### Fixed
-- Fixed the `extract` transformation (`year` / `month` / `day` / `hour` / `minute`) throwing `date.getTime is not a function` for every input: `TypeCaster.cast(value, 'date')` returns a normalized ISO string rather than a `Date`, so component getters failed. It now rebuilds a `Date` from that ISO string and reads UTC components, making extraction deterministic regardless of host timezone
-- Fixed a consumer-as-input cache-staleness bug: a capture path cached from a nested run's scope (since cleared) could be reused by a later top-level run, causing a "data file is missing" failure. `ConsumerInputResolver` now resets its cache at each top-level run and re-materializes if a cached capture no longer exists on disk
-
-## V 1.3.0 - 2026-07-07
-
-### Added
-- Added the `bit_extract` field transformation to extract a single bit from an integer bitmask (e.g. decoding a NAPS LOB byte): `{ "bit_extract": { "position": 0, "as_boolean": false } }`. `position` is a 0-based index from the least-significant bit (bounded to `[0, 52]`); returns `1`/`0`, or `true`/`false` when `as_boolean` is set. Uses arithmetic rather than JS bitwise operators so masks wider than 31 bits stay correct
-- Added the `date_format` field transformation to reformat a date string from an input format to an output format (dayjs tokens), e.g. `{ "date_format": { "input_fmt": "DDMMYYYY", "output_fmt": "YYYY-MM-DD" } }` — distinct from `cast`, which only parses or formats with a single token set. Backed by a new `TypeCaster.reformatDate` helper
-- Added the `code_lookup` field transformation that maps a code to a value using the output of another consumer as the lookup table: `{ "code_lookup": { "consumer": "c_lookup_plans", "key": "plan_code", "value": "plan_tier", "default": "Unknown Tier" } }`. When the code is not found and no `default` is set, the field's `onError` policy applies
-- Added `LookupResolver` (executors): before a consumer's workers spawn, it detects `code_lookup` references, executes each referenced consumer through the normal pipeline, captures its output, projects it to a bounded `code -> value` table written under the run's scope, and hydrates it into each worker via a new in-memory `LookupRegistry`. Includes cycle/depth detection (`MAX_LOOKUP_DEPTH`), an in-memory row bound (`MAX_LOOKUP_ROWS`), and per-run caching so a referenced consumer is executed at most once
-- Added canary coverage for the new transformations: `c_canary_new_transforms` (exercising `bit_extract`, `date_format`, and `code_lookup`) driven off a `c_lookup_plans` lookup consumer and a `p_lookup_plans` producer
-
-### Changed
-- Extended the consumer JSON schema `singleTransformation` definition with `bit_extract`, `date_format`, and `code_lookup` entries
-
-## V 1.2.12 - 2026-07-02
-
-### Added
-- Added IAM-role based authentication for AWS sources (S3, Redshift, SQS) via a shared `resolveAwsCredentials` resolver: `method: "implicit"` (or `"iam"`) uses the AWS SDK default credential chain — including EC2/ECS/EKS instance/task roles — and `method: "arn"` assumes an IAM role via STS with auto-refreshing temporary credentials, in addition to the existing static access keys
-- Added `roleArn`, `roleSessionName`, and `externalId` authentication fields to `ISourceAuthentication` and the source JSON schema for STS AssumeRole configuration
-- Added `method`, `roleArn`, `roleSessionName`, and `externalId` metadata fields to consumer QUEUE trigger configuration so SQS queue polling can also authenticate via IAM roles / STS
-- Added source validation for AWS credential methods: `arn` requires `roleArn`, and `access-secret-key` requires `accessKey` and `secretKey`
-- Added a cross-account `arn` S3 example source to the source JSON schema
-
-### Changed
-- Changed the AWS drivers (S3 source/destination, Redshift) and the SQS `QueueManager` to resolve credentials through the shared resolver, so all AWS services handle credentials identically
-
-### Fixed
-- Fixed the AWS drivers silently ignoring IAM instance/task roles: S3 and Redshift always built a static-key credentials object (even when keys were empty), which disabled the AWS SDK default credential chain. Credentials are now omitted when no static keys are configured, letting the SDK discover instance/task roles
-
-## V 1.2.11 - 2026-06-30
-
-### Added
-- Added `lineCount` as a consumer field property, allowing consumers to emit a consistent 1-based source line number even when files are processed across multiple workers
-- Added a dedicated `ConsumerExecutor` post-processing pass to normalize `lineCount` values after worker merge, with tracked execution timing similar to other dataset-level steps
-- Added `lineCount` test coverage to canary and CLI consumer configurations for easier verification of the feature
-
-## V 1.2.2 - 2026-04-10
-
-### Added
-- Added field-level consumer validations with support for multiple rules per field and per-rule failure actions: `fail`, `skip`, `warn`, and `set_default`
-- Added dataset-level consumer validations for `unique_fields`, `min_rows`, `max_rows`, `no_duplicates`, and `not_empty`
-- Added `DataValidationEngine` to centralize field and dataset validation logic
-- Added validation result type definitions to the definitions package for shared use across engines and executors
-- Added `warn()` logging support for non-fatal validation outcomes
-- Added canary consumer coverage for field-level and dataset-level validations with passing, warning, skipped, defaulted, and failing scenarios
-- Added `verify:local` to the canary package to build the local CLI and run the canary suite against it instead of the published package
-- Added canary coverage for `sample` and `discover` against gzipped local producer inputs
-- Added compressed CSV and JSONL canary fixtures plus matching producer definitions for gzip-based CLI verification
-
-### Changed
-- Updated consumer field validation configuration from a single flat validation object to an ordered array of validation rules with explicit `onFail` behavior
-
-### Fixed
-- Fixed the consumer JSON schema to support the new field-level and dataset-level validation configuration
-- Fixed AJV strict-mode compatibility for validation `in` and `not_in` rule arrays by replacing union `type` declarations with `oneOf`
-- Fixed CLI `sample` and `discover` commands so they can read producers configured with compressed local files by reusing the existing decompression logic during sampling
-- Fixed conflict in export with unique values and split by max size
-
-## V 1.1.15 - 2026-03-26
-
-### Added
-- Added validation that consumer field keys exist in the referenced producer's dimensions/measures
-- Added validation that every consumer field defines the required `key` property
-- Added validation that `copyFrom` references a field that appears earlier in the consumer's field list
-- Added validation that `distinctOn` keys and `orderBy` reference fields present in the consumer
-- Added validation that join SQL `${P.field}` and `${producer.field}` references point to valid fields
-
-### Fixed 
-- Fixed environment variable not exposed to front-end
-- Fixed database endpoint selector
-- Fixed worker image volume usage in a cloud environment
-- Fixed worker-thread execution errors being logged only to the terminal by propagating them back to the orchestrator file logger
-- Fixed `ConsumerExecutor.processRecord` error reporting to log step-specific failures for field resolution, aliasing, transformations, and filter evaluation
-
-## V 1.1.11 - 2026-02-05
-
-### Added
-- Added `startRow` and `startColumn` settings for Excel producers (.xls/.xlsx), allowing users to specify the 1-indexed row and column from which to begin reading data
-
-### Fixed
-- Fixed `MaxListenersExceededWarning` on `WriteStream` during consumer execution by replacing shared stream merge with per-file append pipelines
-- Fixed CLI `run` command always exiting with code 1 even on successful runs
-- Fixed incomplete file logging caused by `process.exit()` terminating before winston could flush buffered writes; added `logger.flush()` to worker threads, orchestrator, and CLI exit paths
-- Added `logger.flush()` before `process.exit()` in all data-processing CLI actions (sample, mock, automap, discover, debug) and worker startup
-- Fixed CLI `discover` command exiting with code 1 on success instead of code 0
-- Fixed per-worker `WriteStream` in `Executor.ts` never being closed, risking data loss before distinct/distinctOn post-processing passes
-- Fixed `Dataset.ts` stream await pattern where `resolve` was never called in the `finish` handler (5 sites: transformStream, sort batches, k-way merge, append), causing promises to hang indefinitely
-- Fixed `ExecutorWriter.ts` not awaiting intermediate stream flush during file-size-based rotation
-- Fixed `DriverHelper.appendObjectsToUnifiedFile` and `LocalDestinationDriver.transformAndMove` not awaiting stream flush before returning
-
-## V 1.1.9 - 2026-02-04
-
-### Added
-- Added `switch_case` transformation for mapping specific values to other values (similar to a switch/case statement)
-- Added validation to detect multiple consumer fields reading from the same producer dimension (suggests using `copyFrom` instead)
-- Added detailed logging to the executor orchestrator with usage ID tracing throughout the execution lifecycle
-
-### Changed
-- Cleaned up CLI execution error output to show concise messages in console while preserving full stack traces in internal logs
-
-## V 1.1.8 - 2026-02-03
-
-### Added
-- Added `pivot` option to consumers, enabling row-to-column transformation with aggregation (sum, count, avg, min, max)
-- Added `copyFrom` property to consumer fields, allowing a field to be a value copy of another field in the dataset
-
-## V 1.1.7 - 2026-02-02
-
-### Changed
-- Improved the mock engine
-- Improved logging
-
-## V 1.1.6 - 2026-02-02
-
-### Added
-- Added `--limit` option to `remora run` command to process only the first N records
-- Added descriptive error messages for failed field transformations with full stack trace preservation
-- Added file logging with rotation (enabled via `REMORA_DEBUG_MODE=true` in production)
-- Added structured logging across key application areas
-
-### Changed
-- Moved `DEBUG_MODE` from project.json settings to `REMORA_DEBUG_MODE` environment variable
-
-## V 1.1.5 - 2026-02-01
-
-### Added
-- Refactored for monorepository
-- Added output maximum file size definable from consumer
-- Added support for nested subfolders inside remora configuration directories (sources, producers, consumers, schemas)
-
-### Fixed
-- Bug in parsing via GZ file
-- Issues with concurrent requests
-
-### Changed
-- Dockerfile for apps in a monorepo build
-- Package.json to workspaces compliance 
-- Refactored internal module structure
-- Removed the _file annotations for environment variables
-
-## V 1.0.18