@forzalabs/remora 2.0.3 → 2.1.1

package/CHANGELOG.md

@@ -6,6 +6,46 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.1.1 - 2026-09-23
+
+### Added
+
+- **SQL sources can be run.** `aws-redshift` has been a valid engine since the beginning, but `RedshiftDriver.ready` — the one method every run goes through to stage a producer — threw `Not implemented yet`, so a Redshift producer could be configured, validated and sampled, and never executed. It runs now, and a new **`postgres`** engine runs beside it: Amazon RDS, Aurora PostgreSQL, or a self-hosted server, over the native wire protocol. A producer on either names its table with `sqlTable` (`table` or `schema.table`, falling back to the source's `schema`) and lists the columns it wants as dimensions, with `alias` naming the column where it differs from the dimension name; remora builds the `SELECT` from exactly those dimensions and stages the result as a local CSV dataset, which is the same thing a Delta share does and the reason the rest of the pipeline needed no change. The statement is built in one place from the producer's own config, and every identifier in it is checked against an identifier pattern rather than escaped, so a table or alias that is not a name is refused where it is written instead of reaching the server as syntax. Both engines are now in `STAGEABLE_ENGINES`, so `remora run`, `remora sample` on a consumer and the preview's **Data** tab all reach them
+- **Postgres authenticates with an RDS IAM auth token**, with `method: "iam"` or `"arn"`, so there is no database password in the configuration at all: a short-lived token is signed per connection, by the SDK default credential chain or by an assumed role, through the same `resolveAwsCredentials` every other AWS driver uses. `method: "username-password"` keeps the ordinary path, where the password is an `{ENV_VAR}` reference like any other secret. TLS is on by default for a remote host (`sslMode`: `require`, or `verify-full` to verify the chain) and off for `localhost`, where a local database usually speaks none
+
+### Changed
+
+- A Redshift statement is **polled for up to 30 minutes** with exponential backoff, instead of 20 cycles of a 1 ms sleep. The Data API is submit-and-poll, and the old loop was sized for the one-row warm-up query in `init`: a table scan legitimately sits in `STARTED` for minutes and would have been abandoned as a timeout. Provisioned clusters also work now — a statement carries `ClusterIdentifier`/`DbUser` when the source declares `clusterId` rather than a `workgroup`, where before only Serverless workgroups were ever sent
+
+- **A live run now says what it read, without `REMORA_DEBUG_MODE`.** Everything that reported what a run actually *did* was `logger.log` — DEBUG, which returns on its first line unless the variable was `true` when the process started — so a deployed worker logged a launch line, a completion line and nothing in between. Restarting it with debug on is not an option when the question is about a run that already happened, and turning it on permanently means every diagnostic line of every driver, forever. Three things are now `info`, chosen because each one answers a question that cannot be answered after the fact from anywhere else. **The options the run was invoked with**: the launch line named the consumer, the invoker, the user and the producer count but never the request, and `IUsageStat` does not keep it either, so there was no way to tell whether a run had been handed filters and a limit or had run on the consumer's own config — it now reads `..., producer(s): 2, options: 2 filter(s), limit 100)`, and is unchanged on a run that carried no options. **What a Delta Share query returned**: `Delta Share table "x" returned N file(s) in Yms` is the number that says whether the pushdown pruned anything, and it now names the hints that were sent with it (`(pushdown: predicate, limit 100)`) so the count can be read against what was actually asked of the share; the predicate tree itself stays on the debug line, since it is a JSON document and not a thing to put in every run's log. **What the read staged**: records, dataset files and bytes per producer, which is the run's own account of the volume it pulled. Nothing new is computed and nothing else moved level — the remaining delta-share diagnostics, the per-file skips and the metadata dump are still debug
+
+- **The options an invocation asks a run for are now checked at the door.** A consumer run can be handed `filters` and a `limit` by an API call (`POST` execute) or a queue message, and both entry points took whatever arrived: the queue path checked that `options` was a non-array object and CAST it, the HTTP route read `req.body.options` and passed it straight to the orchestrator. So a `limit` could arrive as `"100"`, as `0` or as `-5` and bound nothing; a `filters` could be an object; an `operator` could be a word `RequestExecutor` has no case for; and a `member` could name no field of the consumer — which is the worst of them, because an unknown member reads `undefined` off every record and so *silently filters nothing*, returning a full dataset that looks like an answer. The rest surfaced as a throw from the middle of a run that had already queried the source and paid for the read. Both entry points now go through one check, `RequestValidator.validate`, before anything is read: `filters` an array of objects, `member` a field the consumer actually outputs (the expanded `alias ?? key` names, which is what a record holds when filters run), `operator` one of `FilterOperator`, `values` an array of strings of the arity the operator needs (`between` exactly two, `in` at least one, `isNull` none), `limit` a whole number between 1 and `MAX_REQUEST_LIMIT`, and `and`/`or` children validated the same way down to `MAX_REQUEST_FILTER_DEPTH`. Every rejection names the offending option and where it sits — `Invalid filter at filters[1].or[0]: consumer "sales" has no field named "regionn"` — and the API answers **400** instead of running and returning 500 or, worse, 200. Two things are refused outright: **`offset` and `order`**, which the request type carries but no stage of a run implements (the only code reading them is `PushdownEngine.resolveLimit`, where their presence merely suppresses the limit pushdown), and the **`sql`** operator, which `RequestExecutor` throws on rather than evaluating — accepting either would mean answering as if the request had been honoured. An unknown option key (`limits`, `filter`) is refused too, since the alternative is ignoring it silently. Bounds on what may be asked for live with the rest: `MAX_REQUEST_LIMIT` (10,000,000), `MAX_REQUEST_FILTERS` (100) and `MAX_REQUEST_FILTER_DEPTH` (5)
+
+### Fixed
+
+- **A ZIP producer read the whole archive into memory — and had in fact never run at all.** `ParseCompression` extracted a zip through AdmZip, which reads the entire archive into one buffer and then decompresses each member into a second one. On a 71 MB archive holding a 441 MB CSV that is **1.06 GB of peak RSS, 2.4x the member**: a file a 2 GB worker stages comfortably as plain CSV exhausts it the moment it arrives zipped. `.gz` and `.tar` beside it already streamed; only ZIP was left behind, the same shape of bug as the delta-share crash. A zip is now read entry by entry with `yauzl`, each member piped from the archive straight to disk, and the same file costs **136 MB peak — about 44 MB above an empty process** — bounded by the pipeline's own buffers rather than by what the archive holds. Two things fell out of the rewrite. The extraction **never ran**: `ExecutorScope.ensurePath` touches the dataset path as an empty *file* before a driver is called, and extracting an archive into that path went straight to `EEXIST: mkdir`, so ZIP and TAR were both dead on arrival on the local source driver. And an archive with more than one member is now read **in full** — `decompressToFile` returns every file it extracted rather than the directory it put them in, and `LocalSourceDriver.ready` fans those out exactly the way it already fans out a wildcard `fileKey`, where before a multi-member archive handed the run a directory path to read as though it were a data file. TAR reports its members the same way. Entry paths are still resolved and checked against the target directory before anything is opened (yauzl refuses a traversal name of its own accord; the `affirm` is the second line), an encrypted member is refused by name instead of writing its ciphertext, and `MAX_ARCHIVE_ENTRIES` (10 000) bounds what a single fileKey may fan out into. `c_canary_zip` in the canary reads a two-member archive and checks the row count against the unsplit source, because reading only the first member is a silent half-answer rather than a failure
+- **Redshift silently returned only the first page of every query.** `getStatementResult` never read `result.NextToken`, and the Data API pages results at roughly 1 MB — so any statement returning more than that came back **short, with no error**: `SQL_MAX_QUERY_ROWS` could not be honoured, a sample of a wide table was a fraction of what was asked for, and nothing anywhere said so. Not a crash but a wrong answer, which is worse, and it had to be fixed before a staging read could be built on the same call. Every read now follows the chain to the end. The in-memory paths (`query`, `execute`) stop at `SQL_MAX_QUERY_ROWS` and **log that they truncated**, rather than doing it quietly; staging accumulates nothing at all — each page is written to the dataset file before the next is fetched — and then **checks what it staged against the row count the cluster reports**, so a lost page fails the run by name instead of producing a short dataset that looks complete
+- Staging a table costs disk rather than memory, on both engines: Postgres reads through a server-side cursor inside a `READ ONLY` transaction, `FETCH`ing 10 000 rows per round trip and writing each batch before asking for the next, and the cursor is closed and the transaction ended even when the read is abandoned half way. Redshift does the same with result pages. Neither ever holds the table
+- **A run that read no rows crashed on the S3 upload.** `S3DestinationDriver` uploads every output through a multipart upload, and an empty dataset — a Delta Share (or any other source) that legitimately returned nothing for the period asked of it — produces zero parts, so the run ended on a `CompleteMultipartUpload` carrying an empty part list, which S3 rejects as `MalformedXML`. The error named neither the consumer nor the reason, so an empty read looked like a broken destination. An upload with no parts is now aborted and the object written directly as a zero-byte `PutObject`, which is what the local destination driver has always done: a no-data run lands an empty file and succeeds. A failed abort is also no longer able to mask the error that triggered it — it is logged with the key and upload id, and the original exception is the one that propagates
+
+## V 2.1.0 - 2026-09-17
+
+### Added
+
+- **`remora ui-preview` now shows a field's `description`.** A consumer field, a producer dimension or measure, and a JSON-schema property can all carry one, and the viewer was the one place a project's own documentation did not reach: the Fields and Dimensions tables printed the name, the type and the transformations, and dropped the sentence saying what the column is *for* — which on a layout of a thousand generated column names is the only thing that tells two of them apart. A description now reads as a **second line under its row**, across the full width of the panel, muted and small. Not as a column: free prose would have set the width of a table whose other columns are a word wide, and printed a dash on every row nobody documented, where this costs a line on the rows that carry one and nothing at all on the rest. Long prose is clamped to two lines with the whole of it as the row's tooltip (and verbatim in the **File** tab), and it wraps inside the panel however wide the table is, so nothing ends up hidden behind a horizontal scroll. The **field filter matches descriptions too**, so a field can now be found by what it does and not only by what it is called — and the match is marked in the description the same way it is in a name. In the contract this is one optional `note` on a cell, so the page still renders tables it knows nothing about and the format version is unchanged; a snapshot written by an older CLI simply has no notes to draw
+- Tests for the delta-share driver — the package's first — covering the batched read that replaced it: that the batches cover a file exactly once and never span a row group whatever the shape of its footer, that the streamed CSV is byte-for-byte what the old buffered write produced, that a range read stops as soon as it is filled, and that `ready` stages a real parquet file over a real socket and leaves nothing behind. The row-group arithmetic is pinned against a stubbed reader, which can describe the million-row group that caused the crash; what only a real file can settle — that hyparquet's `rowStart`/`rowEnd` are absolute row indexes and slice correctly *inside* a group — is pinned against three small fixtures from hyparquet's own corpus. `npm run test:unit` runs them
+- A **Delta Share consumer in the canary** (`c_canary_deltashare`). The share already configured there carries `mock_data` — the same 120 accounts as the canary's own `mock_data.jsonl` — so the check is an equivalence one: every row that came over the Delta Sharing protocol, through a share query, a pre-signed URL and a parquet file staged in row batches, must equal the row the local file already holds. That is what tells a *wrong* read from a merely successful one, which matters for a staging path that reads a part file in pieces: a batch that lost, repeated or misaligned rows would still produce a plausible-looking output. The consumer's country filter doubles as pushdown coverage — it reaches the share as a json predicate hint — and since a hint is only ever an optimization the expected rows are derived locally either way. Delta Share was the one source engine with no canary coverage at all, which is how a four-copies-of-the-file bug survived in it
+
+### Changed
+
+- The **state heartbeat now emits every five minutes** instead of every minute. The payload is a full status object — worker version, config signature, resource counts, every CRON job with its next fire time, queue mappings, in-flight runs, heap — and at the old cadence an idle worker wrote 1440 of them a day, forever, each one an ingested and retained CloudWatch event the customer pays for. The evidence it exists to provide survives the longer interval: a wedged run is the same `executionId` with a growing `elapsedMS` across a handful of records, and a task replaced at 03:00 still announces itself at once, because the first heartbeat goes out immediately on start rather than after a full interval. `REMORA_STATE_INTERVAL_MS` overrides it as before, still clamped to 5s–1h — and a value that is unusable or out of that range is now **logged as a warning** naming the interval actually used, since an operator who sets an interval and silently gets a different one has no other way to find out
+
+### Fixed
+
+- **A delta-share producer crashed the worker with `JavaScript heap out of memory`.** A part file is small on the wire and large in the heap — one 2.4 MB zstd file of a real share decodes to 572k rows — and the driver held that file four times over at once: hyparquet's row objects, a second array of them from `toJson`, an array of the CSV lines, and the single string joining those lines before the write. Four copies of half a million rows do not fit in a worker's 2 GB heap, so the run died on the file rather than on the table's size, with no error of its own — just V8's `FATAL ERROR` and exit code 134. The staging path now reads a part file **in batches of 50k rows** and writes each batch out as it arrives, so only one batch is ever in memory and a file's row count stops being a memory limit. To make the batched reads cheap the file is first copied to the run's temp folder, streamed, and read from there: reading it in passes over its pre-signed URL would have refetched the same bytes for each batch, and the download itself never holds more than a socket buffer. The staged dataset file is byte-for-byte what it was — same header, same order, same escaping — and the download is dropped once converted
+- The same unbounded read is gone from the **preview and sample** path. `readLinesInRange` used to decode every row of every part file of a table and then keep the ten it was asked for; it now stops at the first batch that fills the range, so asking a 572k-row table for ten rows reads 50k of them instead of all of them, and a part file past the range is never opened at all. A range of **zero** rows now reads nothing rather than decoding a batch to return an empty list
+- **A logged error arrived in CloudWatch as thirty unrelated log events.** The awslogs driver ends a log event at every newline on stdout, so a stack trace — the one thing worth reading when a run fails — was split line by line, each fragment its own event, none of them carrying the message it belonged to, and all of them interleaved with whatever the other threads were writing at the same moment. Deployed processes (`REMORA_RUNTIME_CONTEXT` other than `cli`) now render every console record as **one line of JSON**: `{"remora":"log","v":1,"ts":…,"level":…,"message":…,"stack":…}`, with the stack a field of the record rather than a hundred events after it. One record is one event, searchable by level and by message in Logs Insights, the same shape the heartbeat and run records already use. The CLI is untouched — a terminal keeps the human-readable format, colours and the stack on its own lines — and `REMORA_LOG_FORMAT=text|json` overrides the default either way. `console.log`/`console.error` are routed through the logger in the deployed entrypoints too, since a rule enforced only at our own call sites is one the next `console.error` — ours, express's or a driver's — quietly breaks. An oversized field is clamped, with the truncation stated in the record: a log event is capped at 256 KB and what exceeds it is split or dropped, which is the problem this format exists to solve
+
 ## V 2.0.3 - 2026-09-03
 
 ### Added