@forzalabs/remora 2.0.2 → 2.1.0

package/CHANGELOG.md

@@ -6,12 +6,45 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.1.0 - 2026-09-17
+
+### Added
+
+- **`remora ui-preview` now shows a field's `description`.** A consumer field, a producer dimension or measure, and a JSON-schema property can all carry one, and the viewer was the one place a project's own documentation did not reach: the Fields and Dimensions tables printed the name, the type and the transformations, and dropped the sentence saying what the column is *for* — which on a layout of a thousand generated column names is the only thing that tells two of them apart. A description now reads as a **second line under its row**, across the full width of the panel, muted and small. Not as a column: free prose would have set the width of a table whose other columns are a word wide, and printed a dash on every row nobody documented, where this costs a line on the rows that carry one and nothing at all on the rest. Long prose is clamped to two lines with the whole of it as the row's tooltip (and verbatim in the **File** tab), and it wraps inside the panel however wide the table is, so nothing ends up hidden behind a horizontal scroll. The **field filter matches descriptions too**, so a field can now be found by what it does and not only by what it is called — and the match is marked in the description the same way it is in a name. In the contract this is one optional `note` on a cell, so the page still renders tables it knows nothing about and the format version is unchanged; a snapshot written by an older CLI simply has no notes to draw
+- Tests for the delta-share driver — the package's first — covering the batched read that replaced it: that the batches cover a file exactly once and never span a row group whatever the shape of its footer, that the streamed CSV is byte-for-byte what the old buffered write produced, that a range read stops as soon as it is filled, and that `ready` stages a real parquet file over a real socket and leaves nothing behind. The row-group arithmetic is pinned against a stubbed reader, which can describe the million-row group that caused the crash; what only a real file can settle — that hyparquet's `rowStart`/`rowEnd` are absolute row indexes and slice correctly *inside* a group — is pinned against three small fixtures from hyparquet's own corpus. `npm run test:unit` runs them
+- A **Delta Share consumer in the canary** (`c_canary_deltashare`). The share already configured there carries `mock_data` — the same 120 accounts as the canary's own `mock_data.jsonl` — so the check is an equivalence one: every row that came over the Delta Sharing protocol, through a share query, a pre-signed URL and a parquet file staged in row batches, must equal the row the local file already holds. That is what tells a *wrong* read from a merely successful one, which matters for a staging path that reads a part file in pieces: a batch that lost, repeated or misaligned rows would still produce a plausible-looking output. The consumer's country filter doubles as pushdown coverage — it reaches the share as a json predicate hint — and since a hint is only ever an optimization the expected rows are derived locally either way. Delta Share was the one source engine with no canary coverage at all, which is how a four-copies-of-the-file bug survived in it
+
+### Changed
+
+- The **state heartbeat now emits every five minutes** instead of every minute. The payload is a full status object — worker version, config signature, resource counts, every CRON job with its next fire time, queue mappings, in-flight runs, heap — and at the old cadence an idle worker wrote 1440 of them a day, forever, each one an ingested and retained CloudWatch event the customer pays for. The evidence it exists to provide survives the longer interval: a wedged run is the same `executionId` with a growing `elapsedMS` across a handful of records, and a task replaced at 03:00 still announces itself at once, because the first heartbeat goes out immediately on start rather than after a full interval. `REMORA_STATE_INTERVAL_MS` overrides it as before, still clamped to 5s–1h — and a value that is unusable or out of that range is now **logged as a warning** naming the interval actually used, since an operator who sets an interval and silently gets a different one has no other way to find out
+
+### Fixed
+
+- **A delta-share producer crashed the worker with `JavaScript heap out of memory`.** A part file is small on the wire and large in the heap — one 2.4 MB zstd file of a real share decodes to 572k rows — and the driver held that file four times over at once: hyparquet's row objects, a second array of them from `toJson`, an array of the CSV lines, and the single string joining those lines before the write. Four copies of half a million rows do not fit in a worker's 2 GB heap, so the run died on the file rather than on the table's size, with no error of its own — just V8's `FATAL ERROR` and exit code 134. The staging path now reads a part file **in batches of 50k rows** and writes each batch out as it arrives, so only one batch is ever in memory and a file's row count stops being a memory limit. To make the batched reads cheap the file is first copied to the run's temp folder, streamed, and read from there: reading it in passes over its pre-signed URL would have refetched the same bytes for each batch, and the download itself never holds more than a socket buffer. The staged dataset file is byte-for-byte what it was — same header, same order, same escaping — and the download is dropped once converted
+- The same unbounded read is gone from the **preview and sample** path. `readLinesInRange` used to decode every row of every part file of a table and then keep the ten it was asked for; it now stops at the first batch that fills the range, so asking a 572k-row table for ten rows reads 50k of them instead of all of them, and a part file past the range is never opened at all. A range of **zero** rows now reads nothing rather than decoding a batch to return an empty list
+- **A logged error arrived in CloudWatch as thirty unrelated log events.** The awslogs driver ends a log event at every newline on stdout, so a stack trace — the one thing worth reading when a run fails — was split line by line, each fragment its own event, none of them carrying the message it belonged to, and all of them interleaved with whatever the other threads were writing at the same moment. Deployed processes (`REMORA_RUNTIME_CONTEXT` other than `cli`) now render every console record as **one line of JSON**: `{"remora":"log","v":1,"ts":…,"level":…,"message":…,"stack":…}`, with the stack a field of the record rather than a hundred events after it. One record is one event, searchable by level and by message in Logs Insights, the same shape the heartbeat and run records already use. The CLI is untouched — a terminal keeps the human-readable format, colours and the stack on its own lines — and `REMORA_LOG_FORMAT=text|json` overrides the default either way. `console.log`/`console.error` are routed through the logger in the deployed entrypoints too, since a rule enforced only at our own call sites is one the next `console.error` — ours, express's or a driver's — quietly breaks. An oversized field is clamped, with the truncation stated in the record: a log event is capped at 256 KB and what exceeds it is split or dropped, which is the problem this format exists to solve
+
+## V 2.0.3 - 2026-09-03
+
 ### Added
 
 - `remora ui-preview` now has **search**, at both levels the project has: a box in the header finds a **node**, and a box above the panel's tables filters the **fields inside the selected one**. The node search matches on name, kind, description and badges, all terms in any order (`cons canary` finds `c_canary_extract`), and its answer lands **in the picture** rather than only beside it — everything that does not match fades on the canvas, so a search over fifty nodes leaves the matching ones legible in the graph they belong to instead of handing back a list to go and find them in. `/` from anywhere focuses it, ↑/↓ walk the results, Enter selects and pans to it, Escape clears. The field filter keeps only the rows that match — a dimension, a consumer field, a resolved lineage path, an output, a run — and the **tab strip carries the count per tab**, which is what makes it usable on a resource whose fields are spread over four tables: `addr0Street` says which tab holds it without each one having to be opened to find out. Matched text is marked in place, since a filtered row of near-identical generated column names is not self-explanatory once found. It filters a `Data` sample the same way, is dropped when the selection moves to another resource, and survives a live reload of the same one — like the open tab and the scroll position. All of it runs **in the page**, on the documents it already has: no route, no new capability and no contract change, so a snapshot opened with `remora-preview <file>` searches exactly the same way
 
+### Changed
+
+- **Sampling now runs the pipeline.** `remora sample <consumer>`, and the **Data** tab of `remora ui-preview`, used to answer a question nobody asked: the rows shown for a consumer were its *first producer's* rows, projected through the consumer's field list. No filters, no field transformations, no unions, no `distinct`, no `pivot`, no measures, no validation. Under a heading that says "Data", beside a consumer's name, that read as what the consumer produces — and it was not: a consumer whose only job is to uppercase a column showed it lowercase, and one that keeps one row in a hundred showed the ninety-nine. Sampling a consumer now **executes it** — every filter, transformation, mask, union, `distinct`, `pivot`, measure and dataset validation, exactly as `remora run` applies them — so the rows are the consumer's own output, always, rather than an approximation of it. It **touches nothing**: no write to the destination, no `onSuccess`/`onError` action, and although the execution is recorded like any other it is marked as a sample, so it stays out of the run overlay, out of a consumer's run list and out of its duration averages — a node's dot and its "how long does this take" keep meaning a real run over real volume. The bound on the rows is the same `limit` `remora run --limit` uses, so the cost follows the same rule: where nothing between the source and the output can *drop* a record the limit reaches the source itself and the read stops early, and where something can — any filter, `distinct`, `distinctOn`, `pivot`, or a validation set to `"onFail": "skip"` — there is no knowing how many rows must go in for ten to come out, so the whole source is read. On a large source that is slow; it is still only a read. Sampling a consumer needs a source the pipeline can stage — `local`, `aws-s3` or `delta-share` — so one whose producer reads `aws-redshift` or `http-api` (whose drivers have no `ready()`) is refused up front with a message naming the producer and pointing at it, instead of a worker pool starting and then surfacing `Not implemented yet` from three layers down. In the preview it takes the same one-at-a-time lock as `▶ Run`, since it is the same worker pool. Sampling a **producer** is unchanged in shape and still costs one bounded read; what it now does is the producer stage in full (see the fix below). None of this added anything to the execution path: a sample is `launch` with three flags the pipeline already had for the materialization runs it already did — `options.limit`, `skipExport` and `captureMainDatasetTo` — so the executor cannot tell it is being sampled, and one code path still serves both callers
+- A sampled consumer's columns are now ordered by the **fields the consumer declares**, rather than by the order the producer's dimensions happened to be read in. Invisible in an exported JSON object; the whole shape of a table you are reading
+- `remora ui-preview`'s **Data** tab is quieter. The button keeps one label — `Sample 10 rows` — rather than re-wording itself to `Sample again 10 rows` once rows are on screen: it does the same thing on the tenth press as on the first, and moving the text under the pointer said nothing new. The paragraph beside it, explaining what sampling costs, is gone; what actually ran is stated over the table by the host that ran it, which is the sentence worth reading and is there every time. The button still says what it is doing while it waits
+
 ### Fixed
 
+- **A masked producer dimension was shown in the clear when sampled.** A dimension declared `"mask": "hash"` (or `crypt`, `random`, `seeded-random`) is masked in `ProducerExecutor.processLine`, on the run path — and the sampling path never went through it. So `remora sample <producer>` and the preview's **Data** tab printed the raw value of every column the project had asked to be masked, as did the projected consumer rows built from them. That is a disclosure rather than a display bug: the values reach a terminal and a browser page, and the one place someone checks that masking works is the place that was not applying it. `sourceFilename` dimensions were blank there for the same reason. The producer stage now lives in one place — `ProducerManager.compileFills`/`applyFills` — which both the worker and the sample read call, so a mask cannot be applied on one path and skipped on the other
+- **A sample in `remora ui-preview` showed no rows at all.** The Data tab ran the sample, the host answered with the rows, and the panel stayed empty with nothing in the log to explain it. Two things in the run overlay caused it, both from a sample being *recorded* like any other execution and then filtered out of the picture: `RunOverlay.signature()` — the token the preview polls to notice a finished run — signed the usage **store** rather than the runs it paints, so writing a sample's own record moved the token while the picture stayed identical; and the sample marker was dropped *after* the 500-row read limit, so a sample also pushed a real run off the end of the history. Either way the preview concluded the project had changed and told the page so, and a page told its resource changed **drops the sample it is showing** — deliberately, since rows taken against the previous definition must never be shown under the new one. So the rows were fetched and then discarded on arrival, every time. The signature now signs the overlay it describes (gated on the store's mtime, so polling still reads metadata and not records) and the marker is applied as the store reads, so a sample cannot displace a run. A real run notifies and repaints exactly as before
+- **`remora ui-preview`'s minimap and zoom controls were white in dark mode**, which on a dark canvas left them as a bright rectangle and three invisible buttons. xyflow paints its own chrome from its own palette and ships a `.dark` variant of it that has to be switched on and then kept in step with greys that are not this product's; it now reads the theme's tokens through the `--xy-*` slots the library exposes, so one declaration themes the canvas in both modes and the canvas can never be in a mode the rest of the page is not
+- **The minimap was empty**, whatever colour it was. It draws a node from the size the node DECLARES, and the layout — which fixes every card at the same box and reserves exactly that box in dagre — had never said so on the node itself, so there was nothing for it to draw and the panel was a blank rectangle. The box is now declared as `initialWidth`/`initialHeight`, the size xyflow assumes until it measures the rendered card: the minimap gets its nodes and the canvas keeps measuring, so the picture itself is unchanged
+- `remora ui-preview`'s sample route is now a `POST` carrying the session token, like `▶ Run` already was. It **executes**: a producer's sample spends a query against your warehouse and a consumer's spends this machine's threads, and loopback is not privacy — every site you visit can address 127.0.0.1, so a `GET` let any page you happened to have open start one
+- A consumer's `onError` actions no longer fire for a run that exports nothing. A `code_lookup` table or an upstream consumer-input that failed while being materialized ran the dependent consumer's `onError` — moving its source file, starting other consumers — for an execution nobody asked for; the success path had always skipped them, and the failure path now does too
+- `remora sample <producer> -s 10` on a file with a header row returned **nine** rows. The header was one of the lines read and then dropped, so the count asked for was always short by one
 - `remora ui-preview` no longer describes itself as **read-only**. The panel's footer promised "this viewer is read-only and never reads your data", and the command's own `--help` said the same, while the page it was printed on samples real rows, runs consumers and saves configuration files. What is still true is narrower and is where it is now said: the `@forzalabs/remora-preview` **package** has no drivers, no credentials and no configuration reader, so everything past rendering is a capability of the host and reaches the page as a route only when the host advertises one — and `remora-preview <snapshot.json>`, which has no host behind it, advertises none and is read-only in fact rather than by claim
 
 ## V 2.0.1 - 2026-09-01