@forzalabs/remora 2.0.3 → 2.1.0

package/CHANGELOG.md

@@ -6,6 +6,24 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.1.0 - 2026-09-17
+
+### Added
+
+- **`remora ui-preview` now shows a field's `description`.** A consumer field, a producer dimension or measure, and a JSON-schema property can all carry one, and the viewer was the one place a project's own documentation did not reach: the Fields and Dimensions tables printed the name, the type and the transformations, and dropped the sentence saying what the column is *for* — which on a layout of a thousand generated column names is the only thing that tells two of them apart. A description now reads as a **second line under its row**, across the full width of the panel, muted and small. Not as a column: free prose would have set the width of a table whose other columns are a word wide, and printed a dash on every row nobody documented, where this costs a line on the rows that carry one and nothing at all on the rest. Long prose is clamped to two lines with the whole of it as the row's tooltip (and verbatim in the **File** tab), and it wraps inside the panel however wide the table is, so nothing ends up hidden behind a horizontal scroll. The **field filter matches descriptions too**, so a field can now be found by what it does and not only by what it is called — and the match is marked in the description the same way it is in a name. In the contract this is one optional `note` on a cell, so the page still renders tables it knows nothing about and the format version is unchanged; a snapshot written by an older CLI simply has no notes to draw
+- Tests for the delta-share driver — the package's first — covering the batched read that replaced it: that the batches cover a file exactly once and never span a row group whatever the shape of its footer, that the streamed CSV is byte-for-byte what the old buffered write produced, that a range read stops as soon as it is filled, and that `ready` stages a real parquet file over a real socket and leaves nothing behind. The row-group arithmetic is pinned against a stubbed reader, which can describe the million-row group that caused the crash; what only a real file can settle — that hyparquet's `rowStart`/`rowEnd` are absolute row indexes and slice correctly *inside* a group — is pinned against three small fixtures from hyparquet's own corpus. `npm run test:unit` runs them
+- A **Delta Share consumer in the canary** (`c_canary_deltashare`). The share already configured there carries `mock_data` — the same 120 accounts as the canary's own `mock_data.jsonl` — so the check is an equivalence one: every row that came over the Delta Sharing protocol, through a share query, a pre-signed URL and a parquet file staged in row batches, must equal the row the local file already holds. That is what tells a *wrong* read from a merely successful one, which matters for a staging path that reads a part file in pieces: a batch that lost, repeated or misaligned rows would still produce a plausible-looking output. The consumer's country filter doubles as pushdown coverage — it reaches the share as a json predicate hint — and since a hint is only ever an optimization the expected rows are derived locally either way. Delta Share was the one source engine with no canary coverage at all, which is how a four-copies-of-the-file bug survived in it
+
+### Changed
+
+- The **state heartbeat now emits every five minutes** instead of every minute. The payload is a full status object — worker version, config signature, resource counts, every CRON job with its next fire time, queue mappings, in-flight runs, heap — and at the old cadence an idle worker wrote 1440 of them a day, forever, each one an ingested and retained CloudWatch event the customer pays for. The evidence it exists to provide survives the longer interval: a wedged run is the same `executionId` with a growing `elapsedMS` across a handful of records, and a task replaced at 03:00 still announces itself at once, because the first heartbeat goes out immediately on start rather than after a full interval. `REMORA_STATE_INTERVAL_MS` overrides it as before, still clamped to 5s–1h — and a value that is unusable or out of that range is now **logged as a warning** naming the interval actually used, since an operator who sets an interval and silently gets a different one has no other way to find out
+
+### Fixed
+
+- **A delta-share producer crashed the worker with `JavaScript heap out of memory`.** A part file is small on the wire and large in the heap — one 2.4 MB zstd file of a real share decodes to 572k rows — and the driver held that file four times over at once: hyparquet's row objects, a second array of them from `toJson`, an array of the CSV lines, and the single string joining those lines before the write. Four copies of half a million rows do not fit in a worker's 2 GB heap, so the run died on the file rather than on the table's size, with no error of its own — just V8's `FATAL ERROR` and exit code 134. The staging path now reads a part file **in batches of 50k rows** and writes each batch out as it arrives, so only one batch is ever in memory and a file's row count stops being a memory limit. To make the batched reads cheap the file is first copied to the run's temp folder, streamed, and read from there: reading it in passes over its pre-signed URL would have refetched the same bytes for each batch, and the download itself never holds more than a socket buffer. The staged dataset file is byte-for-byte what it was — same header, same order, same escaping — and the download is dropped once converted
+- The same unbounded read is gone from the **preview and sample** path. `readLinesInRange` used to decode every row of every part file of a table and then keep the ten it was asked for; it now stops at the first batch that fills the range, so asking a 572k-row table for ten rows reads 50k of them instead of all of them, and a part file past the range is never opened at all. A range of **zero** rows now reads nothing rather than decoding a batch to return an empty list
+- **A logged error arrived in CloudWatch as thirty unrelated log events.** The awslogs driver ends a log event at every newline on stdout, so a stack trace — the one thing worth reading when a run fails — was split line by line, each fragment its own event, none of them carrying the message it belonged to, and all of them interleaved with whatever the other threads were writing at the same moment. Deployed processes (`REMORA_RUNTIME_CONTEXT` other than `cli`) now render every console record as **one line of JSON**: `{"remora":"log","v":1,"ts":…,"level":…,"message":…,"stack":…}`, with the stack a field of the record rather than a hundred events after it. One record is one event, searchable by level and by message in Logs Insights, the same shape the heartbeat and run records already use. The CLI is untouched — a terminal keeps the human-readable format, colours and the stack on its own lines — and `REMORA_LOG_FORMAT=text|json` overrides the default either way. `console.log`/`console.error` are routed through the logger in the deployed entrypoints too, since a rule enforced only at our own call sites is one the next `console.error` — ours, express's or a driver's — quietly breaks. An oversized field is clamped, with the truncation stated in the record: a log event is capped at 256 KB and what exceeds it is split or dropped, which is the problem this format exists to solve
+
 ## V 2.0.3 - 2026-09-03
 
 ### Added