@forzalabs/remora 2.1.0 → 2.1.1

package/CHANGELOG.md

@@ -6,6 +6,28 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.1.1 - 2026-09-23
+
+### Added
+
+- **SQL sources can be run.** `aws-redshift` has been a valid engine since the beginning, but `RedshiftDriver.ready` — the one method every run goes through to stage a producer — threw `Not implemented yet`, so a Redshift producer could be configured, validated and sampled, and never executed. It runs now, and a new **`postgres`** engine runs beside it: Amazon RDS, Aurora PostgreSQL, or a self-hosted server, over the native wire protocol. A producer on either names its table with `sqlTable` (`table` or `schema.table`, falling back to the source's `schema`) and lists the columns it wants as dimensions, with `alias` naming the column where it differs from the dimension name; remora builds the `SELECT` from exactly those dimensions and stages the result as a local CSV dataset, which is the same thing a Delta share does and the reason the rest of the pipeline needed no change. The statement is built in one place from the producer's own config, and every identifier in it is checked against an identifier pattern rather than escaped, so a table or alias that is not a name is refused where it is written instead of reaching the server as syntax. Both engines are now in `STAGEABLE_ENGINES`, so `remora run`, `remora sample` on a consumer and the preview's **Data** tab all reach them
+- **Postgres authenticates with an RDS IAM auth token**, with `method: "iam"` or `"arn"`, so there is no database password in the configuration at all: a short-lived token is signed per connection, by the SDK default credential chain or by an assumed role, through the same `resolveAwsCredentials` every other AWS driver uses. `method: "username-password"` keeps the ordinary path, where the password is an `{ENV_VAR}` reference like any other secret. TLS is on by default for a remote host (`sslMode`: `require`, or `verify-full` to verify the chain) and off for `localhost`, where a local database usually speaks none
+
+### Changed
+
+- A Redshift statement is **polled for up to 30 minutes** with exponential backoff, instead of 20 cycles of a 1 ms sleep. The Data API is submit-and-poll, and the old loop was sized for the one-row warm-up query in `init`: a table scan legitimately sits in `STARTED` for minutes and would have been abandoned as a timeout. Provisioned clusters also work now — a statement carries `ClusterIdentifier`/`DbUser` when the source declares `clusterId` rather than a `workgroup`, where before only Serverless workgroups were ever sent
+
+- **A live run now says what it read, without `REMORA_DEBUG_MODE`.** Everything that reported what a run actually *did* was `logger.log` — DEBUG, which returns on its first line unless the variable was `true` when the process started — so a deployed worker logged a launch line, a completion line and nothing in between. Restarting it with debug on is not an option when the question is about a run that already happened, and turning it on permanently means every diagnostic line of every driver, forever. Three things are now `info`, chosen because each one answers a question that cannot be answered after the fact from anywhere else. **The options the run was invoked with**: the launch line named the consumer, the invoker, the user and the producer count but never the request, and `IUsageStat` does not keep it either, so there was no way to tell whether a run had been handed filters and a limit or had run on the consumer's own config — it now reads `..., producer(s): 2, options: 2 filter(s), limit 100)`, and is unchanged on a run that carried no options. **What a Delta Share query returned**: `Delta Share table "x" returned N file(s) in Yms` is the number that says whether the pushdown pruned anything, and it now names the hints that were sent with it (`(pushdown: predicate, limit 100)`) so the count can be read against what was actually asked of the share; the predicate tree itself stays on the debug line, since it is a JSON document and not a thing to put in every run's log. **What the read staged**: records, dataset files and bytes per producer, which is the run's own account of the volume it pulled. Nothing new is computed and nothing else moved level — the remaining delta-share diagnostics, the per-file skips and the metadata dump are still debug
+
+- **The options an invocation asks a run for are now checked at the door.** A consumer run can be handed `filters` and a `limit` by an API call (`POST` execute) or a queue message, and both entry points took whatever arrived: the queue path checked that `options` was a non-array object and CAST it, the HTTP route read `req.body.options` and passed it straight to the orchestrator. So a `limit` could arrive as `"100"`, as `0` or as `-5` and bound nothing; a `filters` could be an object; an `operator` could be a word `RequestExecutor` has no case for; and a `member` could name no field of the consumer — which is the worst of them, because an unknown member reads `undefined` off every record and so *silently filters nothing*, returning a full dataset that looks like an answer. The rest surfaced as a throw from the middle of a run that had already queried the source and paid for the read. Both entry points now go through one check, `RequestValidator.validate`, before anything is read: `filters` an array of objects, `member` a field the consumer actually outputs (the expanded `alias ?? key` names, which is what a record holds when filters run), `operator` one of `FilterOperator`, `values` an array of strings of the arity the operator needs (`between` exactly two, `in` at least one, `isNull` none), `limit` a whole number between 1 and `MAX_REQUEST_LIMIT`, and `and`/`or` children validated the same way down to `MAX_REQUEST_FILTER_DEPTH`. Every rejection names the offending option and where it sits — `Invalid filter at filters[1].or[0]: consumer "sales" has no field named "regionn"` — and the API answers **400** instead of running and returning 500 or, worse, 200. Two things are refused outright: **`offset` and `order`**, which the request type carries but no stage of a run implements (the only code reading them is `PushdownEngine.resolveLimit`, where their presence merely suppresses the limit pushdown), and the **`sql`** operator, which `RequestExecutor` throws on rather than evaluating — accepting either would mean answering as if the request had been honoured. An unknown option key (`limits`, `filter`) is refused too, since the alternative is ignoring it silently. Bounds on what may be asked for live with the rest: `MAX_REQUEST_LIMIT` (10,000,000), `MAX_REQUEST_FILTERS` (100) and `MAX_REQUEST_FILTER_DEPTH` (5)
+
+### Fixed
+
+- **A ZIP producer read the whole archive into memory — and had in fact never run at all.** `ParseCompression` extracted a zip through AdmZip, which reads the entire archive into one buffer and then decompresses each member into a second one. On a 71 MB archive holding a 441 MB CSV that is **1.06 GB of peak RSS, 2.4x the member**: a file a 2 GB worker stages comfortably as plain CSV exhausts it the moment it arrives zipped. `.gz` and `.tar` beside it already streamed; only ZIP was left behind, the same shape of bug as the delta-share crash. A zip is now read entry by entry with `yauzl`, each member piped from the archive straight to disk, and the same file costs **136 MB peak — about 44 MB above an empty process** — bounded by the pipeline's own buffers rather than by what the archive holds. Two things fell out of the rewrite. The extraction **never ran**: `ExecutorScope.ensurePath` touches the dataset path as an empty *file* before a driver is called, and extracting an archive into that path went straight to `EEXIST: mkdir`, so ZIP and TAR were both dead on arrival on the local source driver. And an archive with more than one member is now read **in full** — `decompressToFile` returns every file it extracted rather than the directory it put them in, and `LocalSourceDriver.ready` fans those out exactly the way it already fans out a wildcard `fileKey`, where before a multi-member archive handed the run a directory path to read as though it were a data file. TAR reports its members the same way. Entry paths are still resolved and checked against the target directory before anything is opened (yauzl refuses a traversal name of its own accord; the `affirm` is the second line), an encrypted member is refused by name instead of writing its ciphertext, and `MAX_ARCHIVE_ENTRIES` (10 000) bounds what a single fileKey may fan out into. `c_canary_zip` in the canary reads a two-member archive and checks the row count against the unsplit source, because reading only the first member is a silent half-answer rather than a failure
+- **Redshift silently returned only the first page of every query.** `getStatementResult` never read `result.NextToken`, and the Data API pages results at roughly 1 MB — so any statement returning more than that came back **short, with no error**: `SQL_MAX_QUERY_ROWS` could not be honoured, a sample of a wide table was a fraction of what was asked for, and nothing anywhere said so. Not a crash but a wrong answer, which is worse, and it had to be fixed before a staging read could be built on the same call. Every read now follows the chain to the end. The in-memory paths (`query`, `execute`) stop at `SQL_MAX_QUERY_ROWS` and **log that they truncated**, rather than doing it quietly; staging accumulates nothing at all — each page is written to the dataset file before the next is fetched — and then **checks what it staged against the row count the cluster reports**, so a lost page fails the run by name instead of producing a short dataset that looks complete
+- Staging a table costs disk rather than memory, on both engines: Postgres reads through a server-side cursor inside a `READ ONLY` transaction, `FETCH`ing 10 000 rows per round trip and writing each batch before asking for the next, and the cursor is closed and the transaction ended even when the read is abandoned half way. Redshift does the same with result pages. Neither ever holds the table
+- **A run that read no rows crashed on the S3 upload.** `S3DestinationDriver` uploads every output through a multipart upload, and an empty dataset — a Delta Share (or any other source) that legitimately returned nothing for the period asked of it — produces zero parts, so the run ended on a `CompleteMultipartUpload` carrying an empty part list, which S3 rejects as `MalformedXML`. The error named neither the consumer nor the reason, so an empty read looked like a broken destination. An upload with no parts is now aborted and the object written directly as a zero-byte `PutObject`, which is what the local destination driver has always done: a no-data run lands an empty file and succeeds. A failed abort is also no longer able to mask the error that triggered it — it is logged with the key and upload id, and the original exception is the one that propagates
+
 ## V 2.1.0 - 2026-09-17
 
 ### Added