@forzalabs/remora 2.2.1 → 2.4.0

package/CHANGELOG.md

@@ -6,6 +6,70 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.4.0 - 2026-10-19
+
+**Upgrading from 2.3.** Two configurations that compiled under 2.3 are now refused, and both fail at `compile` with the offending resource named rather than at run time. A producer carrying `measures` must have them deleted — the key is gone from the schema and a producer rejects unknown keys (nothing is lost: no stage of the run path ever evaluated one, see Removed). A consumer declaring **more than one** `outputs` entry must keep the one it actually used — a second entry was never exported, so the one to keep is the first (see Changed). Two further changes alter the bytes an existing project produces, neither of them opt-out: a `CSV`/`TXT` export now writes a null as an unquoted empty field (`"USR-2",,`) where it used to write `""`, and a value containing a newline is no longer split across two records. A destination that reads `''` where a null is now written, or that relied on the old split rows, needs checking before this version runs.
+
+### Added
+
+- **`"format": "PARQUET"`, a fourth file output.** It writes an Apache Parquet file — one typed column per visible field, nulls preserved, SNAPPY compressed — to any file destination the other formats reach (local, S3, a pre-signed URL), with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour as every other file output. Parquet is what a lake expects to read, and it gives a reader two things a `CSV` export cannot: the file **declares its types**, so nothing downstream guesses them back out of the text, and it is a fraction of the size — a four-column, 2-million-row dataset measures 28 MB as parquet against 115 MB as quoted CSV. The schema is taken from what the consumer already declares, nothing new to configure: `string` -> `STRING`, `number` -> `DOUBLE` (every value remora holds is a double, and an `INT64` would promise a reader precision the record never had), `boolean` -> `BOOLEAN`, `datetime` -> `TIMESTAMP`, and a field with no dimension behind it — a `fixed`, `lineCount` or `copyFrom` field, or a column a pattern key matched — is a string, because there is no declared type to read. Every column is nullable, so **a null stays a null** instead of becoming `0`, `false` or `""`: in a column store those are four different things. A `hidden` field is absent from the file entirely, not an empty column in it. An unparseable value under a `datetime` dimension **fails the run**, naming the field and the value, rather than writing a timestamp column quietly full of 1970.
+
+- Parquet is the one output that is **not** a line-for-line pass. It is a columnar container whose footer can only be written once the last row group is encoded, so there is no per-line mapping to hand a destination driver: the file is built in the run's scope folder and then handed over as bytes with no header, through the `move` that is already byte-faithful in all three destination drivers. **No driver method was added and nothing in the driver layer changed.** The writer is `hyparquet-writer`, the companion of the `hyparquet` reader the Delta Share driver already depends on — same author, MIT, one dependency, and reached by dynamic `import()` exactly as that driver reaches the reader. Rows are pulled from an async generator one row group at a time, so **peak memory is bounded by a row group, not by the dataset**, which is the property the whole pipeline is built on; row groups are sized from a budget of *cells* rather than a row count, since 100,000 rows is 400k values on a four-column consumer and 130 million on a 1,300-field one.
+
+- Two flags behave differently on it, both rejected at compile rather than ignored. `header` is **rejected**: a parquet file carries its column names in its own schema, and a header line would be a first row of data claiming to be one. `omitNull` stays rejected as on every positional format — a parquet row group declares every column up front, and a null is written as a null in it. `delimiter` means nothing to it and is ignored, as it is for every format but `TXT`. One caveat is documented rather than worked around: on a split parquet output, **`split.maxFileSizeGb` caps the internal staged bytes, not the parquet bytes**, so the exported files land comfortably under the cap rather than at it — the cap exists to bound what a destination is asked to take in one object, and rotating a file after it was compressed would mean rewriting it. A `PARQUET` output is declared alone on its consumer, as every output now is (see Changed).
+
+### Changed
+
+- **A `CSV` or `TXT` export now distinguishes a null from an empty string.** Every value in those files is quoted, which left the unquoted empty field free to mean something — and it now means **null**, while `""` means the empty string. The two have been the identical two bytes for as long as the format existed, so a column that was genuinely empty and one that held no value at all arrived at the destination indistinguishable, and nothing downstream could get the difference back. **This changes the bytes of every existing CSV/TXT export that carries a null**: where a row used to read `"USR-2","",""` it now reads `"USR-2",,`. It is the shape most readers already assume — `COPY`, `LOAD DATA` and a pandas `read_csv` all treat an unquoted empty field as a null and a quoted one as the empty string — but a destination that reads the two the same way will now see a null where it saw `''`. A field a dimension declares `nullable: false` is unaffected: it carries its type's own empty value (`"0"`, `"false"`, `""`) as it always did, which is a value and so is quoted
+
+- **A value holding a newline no longer becomes two records.** A line of a remora dataset *is* a record — every pass reads it with a line reader and the record limit counts rows by counting newline bytes — so a value carrying a `\n` used to split the row in two, silently, and every stage after it was reading garbage: the wrong column count, a `distinct` that compared halves of rows, a split that bucketed on whatever landed under the key. Such a value is now written as the two characters `\n` (and a carriage return as `\r`) and the record stays whole. The escape is deliberately **one-way** — nothing turns it back into a newline — because a reversible escape would have to escape the backslash too, and that would rewrite every ordinary `C:\path` in the file. A consumer that must carry an embedded newline verbatim has `JSON` and `PARQUET`, whose encodings hold one natively
+
+- **A consumer declares exactly one output.** `outputs` stays an array — nothing in a config file moves, and lineage, the bug report and `ui-preview` keep iterating it — but a second entry is now refused by the schema (`/outputs must NOT have more than 1 items`) and named by the validator. This is a **cap on something that never worked**, not a feature withdrawn: a run stages its records in ONE encoding, chosen from `outputs[0]`, and then *moves* that staged file to the destination, which unlinks it; the export loop also carried its `return` inside the loop body. So a consumer declaring two outputs has always written the first and silently dropped the rest, whatever the second one said. Failing at `compile` tells its author something they need to know rather than taking a working feature away, and so it rejects from this release with no deprecation window. **Nothing in this repository is affected** — of the 140 consumer configs across the canary, the stress and parallel projects, the demos and the apps' own projects, every single one declares exactly one output. To write the same rows twice, declare a second consumer over the same producers. What the cap removed with it: the duplicate-format check, the half of the `omitNull` validation that existed only because two outputs shared one serialized record (`omitNull` on a `JSON` output is no longer rejected for having a `CSV` sibling — there are no siblings), and the "which of the outputs decides the internal format" question, which is now correct by construction
+
+### Fixed
+
+- **A `hidden` field shifted every column after it, for four passes that read the dataset back.** A staged dataset holds the VISIBLE fields only, and it carries no header — but `distinctOn`, `split.by`, the `unique` dataset validation and the `pivot` discovery each worked the column list out for themselves from the consumer's full field list, hidden fields included. So on any consumer with a hidden field, each of them read every column after it off by one: `distinctOn` deduplicated on the wrong column, `split.by` bucketed files by it, and `unique` reported duplicates in a field it had never actually looked at. Nothing failed, and the output looked plausible. The run now compiles **one** description of what is on a line — the columns, in line order, with the type behind each one — hands it to every writer and every reader, and `LineParser` takes it as a required parameter with no overload that accepts a field list, so the derivation that was wrong is not a check someone has to remember but a parameter that no longer exists. The same description travels to a `code_lookup` table and to a consumer-as-input capture, both of which re-derived it too
+
+- **A `pivot` changed the shape of the dataset and nothing downstream was told.** A pivot replaces the record's columns with its row keys plus one column per pivot value — and the CSV header, the split pass, the dataset-level validations and the parquet schema all still described the record as it was BEFORE the pivot. A pivoted `CSV` export therefore carried a header naming columns that were no longer in the file, and a split or a validation over a pivoted consumer addressed columns by the pre-pivot positions. The pivot now returns the shape it wrote, and everything after it reads that
+
+- **A value re-read by a post-pass came back as text.** Once a record had been written to the dataset, any pass that read it again (`distinctOn`, `pivot`, the `unique` validation, a `code_lookup` table, a split key, the consumer-as-input capture) got a string for a column whose dimension declared a `number`, a `boolean` or a `datetime` unless the field happened to resolve its dimension — so a numeric `distinctOn` with a `max` resolution compared `"9"` against `"10"` and a lookup table keyed on a number held string keys. Each column of the dataset now carries the three properties a cast reads — `type`, `format` and `nullable` — so a value is cast back to what it was declared as, without anything going near the consumer config
+
+- **Every boolean read from a text source was `true`.** The cast was `Boolean(value)`, and `Boolean('false')` is `true` — as is `Boolean('0')`, `Boolean('no')` and `Boolean('N')`. Every non-empty string is. So a `boolean` dimension over a CSV, a Delta share, a SQL extract or a re-read of the staged dataset reported `true` on every row that held anything at all, and the column was worthless without anyone being told. A boolean arriving as text is now matched against the spellings a source actually writes (`1`, `true`, `yes`, `y`, `t`, case-insensitively) and is `false` otherwise; a real boolean and a missing value are unchanged. **This changes what such a consumer produces** — a column that was uniformly `true` will now hold the falses it always had
+
+- **`pivot` never ran on a source under 10 MB.** `distinct` and `distinctOn` have a worker-side counterpart and are redone over the merged file only when there was more than one chunk, so the whole post-pass block sat behind `executorResults.length > 1` — and `pivot`, which has no worker-side counterpart and is the only pass that MUST run there, inherited that gate. A consumer with a `pivot` over a source below `MIN_FILE_SIZE_FOR_PARALLEL` (10 MB) therefore exported its rows unpivoted, with no warning and no error: the configuration asked for a shape and got a different one. `pivot` now runs whenever it is configured. **This changes what such a consumer produces** — the output gains the pivoted columns it always asked for and loses the raw ones — so anything downstream that adapted to the unpivoted shape needs looking at. Nothing changes for a consumer whose source was already over the threshold
+
+- **A `lineCount` field was renumbered against a file that no longer matched it.** The pass attributes line *i* of the merged dataset to the worker that wrote it by walking each chunk's output count in order, and adds that chunk's row offset — which only holds while the merged file still carries exactly `sum(outputCount)` lines in worker order. It ran LAST, after `distinct`, `distinctOn` and `pivot` had rewritten the file and changed both the count and the order, so on any consumer combining one of those with a `lineCount` field the offsets landed on the wrong rows. It now runs first, which is also the right meaning: a row that survives `distinct` keeps the line number it had in its source file. (Recounting afterwards would not have fixed it — that renumbers the survivors 1..N, which is a different thing, and a pivoted row has no source line at all)
+
+- **Two producers feeding one consumer serialized the same record into two different byte strings.** A `JSON`/`API` record was written out in the order its keys happened to be inserted, which is the order the PRODUCER's dimensions were read — the source's layout, not the consumer's. So the same values under the same names came out as `{"id":7,"country":"IT"}` from one producer and `{"country":"IT","id":7}` from the other, and both `distinct` and the `no_duplicates` dataset validation, which compare the serialized line, saw two different rows. Records are now written in the consumer's own field order, which is one order for every writer in the run — and is the order the fields are declared in, so a `JSON` export reads top to bottom like the consumer does. `remora sample` loses the re-ordering it did at the surface to work around this, because there is nothing left to work around
+
+- **A clean exit deleted a concurrent run's temp folder.** The shutdown handler cleared "open" executor scopes by listing the directories under the temp root — which cannot tell this process's leftovers from another process's live run — and it ran on every exit, not only an unexpected one. Two remora processes sharing one `REMORA_DATA_PATH` (a docker worker beside a CLI run, two scheduled runs, the backend beside anything) were enough: whichever finished first deleted the other's dataset mid-run, and the survivor then failed on a file that had vanished under it. `ExecutorScope` now registers the scopes **this process** opened and the cleanup clears only those, which also makes it a true no-op on a clean exit, since a run that finished already cleared its own scope. The directory listing stays for what it is actually good for — watching folders come and go, which the parallel canary does — and deletes nothing
+
+- An export to a destination that is not a source of the project now says so, naming the consumer and the destination, instead of failing as a `TypeError` from inside the driver factory. The two unreachable branches of the export switch are gone with it (`CSV` and `JSON` both match the staged format and are byte-`move`d, so neither could ever reach the re-encoding path), and an `API` output no longer falls into a branch that parsed a JSON line as CSV — it stages as JSON and ships as JSON
+
+- Bumped `proxy-addr` to 2.0.8 (with its `ipaddr.js` dependency) through the lockfile. It is a transitive dependency of `express`, reached by the backend and worker servers and by the `ui-preview` host, and is not a package remora depends on directly.
+
+### Removed
+
+- **`measures` on a producer.** A producer measure was a `{ name, description, sql }` triple meant to define a calculated metric, and nothing ever calculated one. `ResourcesUtils.getAvailableColumns` exposed each measure as a column a consumer could select, `ConsumerManager` carried it through field expansion as `ICompiledColumn.measure`, and there the trail ended: no stage of the run path ever read `measure.sql`, so the expression was never sent to a source, never evaluated here, and never produced a value. What it did produce was a column that passed `compile` and then exported nothing — a consumer naming a measure validated cleanly and wrote an empty field for it, which is worse than the field not existing, because the configuration says it should be there. Not one producer in this repository declared a measure across all 88 config files that mentioned the property: every one carried the inert `"measures": []` the old templates emitted. The property is now gone from `IProducer`, from `IProducerMeasure`, from `ICompiledColumn`/`ICompiledConsumerField`, from the JSON schema, from the field-lineage chain and the `ui-preview` Measures table, from `create-consumer` (which used to map measures into the generated consumer's fields), from the frontend's producer view, and from the project README and the public Producers doc. Aggregation was never a producer's job — a producer declares what it reads, and a consumer is where values are combined. **Because producer configs are `additionalProperties: false`, this moves the refusal to load time and makes it total**: a project still carrying `measures` — including the empty `"measures": []` almost every generated producer has — now fails `remora compile`/`verify` with `Invalid producer configuration: p_x (remora/producers/p_x.json) - / has unknown property "measures"` instead of loading and ignoring it. Delete the property; it was doing nothing. If the metric it named is one you want, it belongs on the consumer that reads the producer
+
+## V 2.3.0 - 2026-10-15
+
+**Upgrading from 2.2.** One change alters what an existing project produces: a `CSV` or `TXT` output now writes a header line, so every exported file of that kind gains a first line naming its columns. It is opt-out — `"header": false` on the output restores the old headerless shape — but it is on by default, so a destination that reads the first line as data needs that flag set before this version runs. Nothing else here touches an existing configuration: `remora report` is a new command, and the `.env` refusal only fires on an archive that should never have contained one.
+
+### Added
+
+- **`remora report [run-id]`, which packages a run into a `.zip` someone else can read.** A bug report is worth exactly what the reproduction is worth, so the bundle is designed backwards from what a reproduction needs rather than forwards from what is easy to collect: the usage row for the run (counts, per-stage timings, the error), that run's slice of the log file, the project configuration with its secrets removed, the environment it ran in, a one-page `report.md` that states the remora version, the node and platform, the consumer and its producers with the engine behind each source, the failure and its cause chain — and the commands that run the failing pipeline again on generated input. It defaults to the last recorded run and takes a run id for any other. **The bundle contains no executable**: the reproduction is a fenced block inside `report.md`, to be pasted or saved, rather than a `repro.sh` — an archive carrying a shell script is what a mail gateway or an endpoint scanner quarantines, and a bundle that never reaches the person who needs it reproduces nothing. Those commands bootstrap the two things a config carries from the machine it came from: a placeholder `.env` for every `{ENV_VAR}` the configuration names (`compile` refuses to run while a reference is unset, and placeholders are enough because `synth` contacts no source), and the local directories the sources point at (a missing destination fails the export AFTER the run, which reads as a second bug and is not one). Unpack, paste, and the reported failure happens again on synthetic rows. **It never compiles the project first**, unlike every other command: a bug report is wanted most when the configuration does NOT load, so the project is read leniently with `Environment.inspect()` and what will not load is collected as a finding rather than as a reason to refuse. For the same reason **no section can fail the command** — no run history, no log file, no `remora` directory at all — each one is recorded absent in the manifest with the reason, and the rest is still written; a reporter that dies on a broken project is useless precisely where it is needed. The log is sliced by run id AND by time window, because only the orchestrator's own lines carry `[usageId]` and a driver or parser line can be placed by its timestamp alone; a run whose process died before recording a finish leaves the window open to the end of the file, bounded by a record cap. Where `MONGO_URI` is set there is no local history to read at all (`UsageManager.recordsLocally()` is false), so the command degrades to the configuration, the environment and the tail of the log, and says so in the bundle
+
+- **The configuration travels as authored; the log is scrubbed of credential shapes.** The asymmetry is deliberate. A remora config is not supposed to HOLD a credential — it names one (`{PG_PASSWORD}`, resolved by `SecretManager` at the point of use) and every authentication field already accepts a reference — so `config/` is shipped byte for byte: what a dev reads there is what runs, which is the whole reason it is in the bundle, and the reproduction copies it straight into a project directory. A literal password in a source file is an authoring mistake, and rewriting it on the way out would only hide it from the one person positioned to fix it. A log is the opposite case and gets no such assumption: by the time a value reaches a log line it has been resolved, so `SecretRedactor` removes the shapes a credential has in free text — a JWT, an AWS access key id, the query string of any URL (a presigned URL carries its signature there), and `password=`/`"apiKey":`/`Authorization: Bearer` pairs — while keeping `{ENV_VAR}` placeholders verbatim, since which variable a source reads is part of the question being asked. Nothing else travels either way: only `.json` configuration files are carried, and no row of customer data is collected at any level this version can produce
+
+### Changed
+
+- **A `CSV` or `TXT` export now writes a header line.** A positional file carries values and nothing that names them, so every exported `CSV`/`TXT` file now opens with its field names — the exported ones, a field's `alias` where it has one and its `key` otherwise, in column order, quoted and delimited exactly as the rows under them so the header parses with the same reader the data does. A `hidden` field is not written and so is not named, and with a `split` **every** file of the output carries the header rather than only the first: a file that cannot be read on its own is not a file anyone can be handed. **This changes the bytes every existing CSV/TXT consumer produces**, and a destination that reads the first line as data will read the header as a row — set `"header": false` on that output to keep the old shape. The flag is `CSV`/`TXT` only: on a `JSON` or `API` output, where each record already carries its own field names, it is rejected at compile rather than ignored, so a config that asks for a header always gets one. Nothing internal changed with it — the header is composed and written at export time, not held in the dataset, so the split pass, the consumer-as-input capture, lookups and `sample` all still read a dataset of records and only records. It costs no pass over the data either: the destination drivers write the line ahead of the file they were already copying or streaming
+
+- **`remora package` and `remora deploy` now refuse an archive containing a `.env` file.** A project keeps its `.env` above `./remora` and so out of reach of the packager's walk — but that is a property of where someone put a file, and this archive is uploaded to a worker and is now also packaged into bug reports. The check makes it a property of the packager instead, with a message saying to move the file. `ConfigPackager` moved from the CLI into `packages/common` on the way, so the deployment archive and the bug report share one set of exclusions rather than two that can drift
+
+- `EnvironmentVariables` is now derived from a runtime `ENVIRONMENT_VARIABLES` list rather than written beside one, so a reader that has to enumerate the variables — the bug report's presence map, which records which are SET and never what they hold — gets exactly the names the type allows. `REMORA_MASK_IN_DEV` joins the catalogue, which the docs already named and the type did not
+
 ## V 2.2.1 - 2026-10-15
 
 ### Fixed

package/documentation/README.md

@@ -71,9 +71,6 @@ Producers extract data from sources and define the structure of the data.
 | `dimensions[].alias` | Alternative name for the column in the source | String |
 | `dimensions[].classification` | Data privacy classification | Array containing: `PII`, `PHI`, `GDPR` |
 | `dimensions[].mask` | How to protect sensitive data | `hash`, `mask`, `crypt` |
-| `measures[].name` | Name of calculated measure | Any string without spaces |
-| `measures[].description` | Description of the measure | Any descriptive text |
-| `measures[].sql` | SQL expression to calculate the measure | Valid SQL expression (e.g., `SUM(field_name)`) |
 | `settings.sqlTable` | Source table name, `table` or `schema.table` | Table name as string |
 | `settings.direct` | Whether to query directly without creating views | `true`, `false` |
 | `settings.fileKey` | File key for S3 sources | S3 file path |
@@ -102,7 +99,7 @@ Consumers transform and combine data from producers for specific use cases.
 | `filters[].rule.member` | Field to filter on | Field name |
 | `filters[].rule.operator` | Comparison operator | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `contains`, `not_contains` |
 | `filters[].rule.values` | Values to compare against | Array of strings |
-| `outputs[].format` | Output format | `SQL`, `API`, `CSV`, `PARQUET`, `JSON` |
+| `outputs[].format` | Output format | `API`, `CSV`, `JSON`, `PARQUET`, `TXT` |
 | `outputs[].accellerated` | Whether to materialize for performance | `true`, `false` |
 | `outputs[].direct` | Whether to query directly without creating views | `true`, `false` |
 | `outputs[].exportDestination` | Where to export data | Must match a source `name` |