@forzalabs/remora 2.1.0 → 2.2.0

package/CHANGELOG.md

@@ -6,6 +6,80 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.2.0 - 2026-10-15
+
+**Upgrading from 2.1.** Three changes can alter what an existing project does, despite the minor version. `nullable` on a producer dimension is no longer documentation: it defaults to `true` and keeps a missing value as `null` as before, but a dimension that declares `"nullable": false` now gets the type's own empty value (`""`, `0`, `false`) where it used to get `null`. A `NULL` in a `number` column now stages as `null` rather than as `0`, so a consumer that was summing or filtering over those zeros sees different figures — the fix is in the staging of every CSV-staged source (SQL, Delta Share, local and S3 CSV). And `settings.sql` on a producer is gone from the schema, so a producer still carrying it fails `compile` — the property never did anything; delete it. Also worth a glance: `remora package` writes `./remora.zip` instead of `./remora-config.zip` when no `-o` is given, and the resource folder lists in `project.json` (`sources`, `producers`, `consumers`, `schemas`, `codeSets`) are ignored — the folders are fixed. They still validate, so nothing breaks, but a project pointing them somewhere else was already not being read that way.
+
+### Added
+
+- **`split` on a consumer: the output broken into several files by a field value, by a size cap, or by both.** `"split": { "by": ["country"], "maxFileSizeGb": 2 }` writes one file per country and rotates each of them at 2 Gb. Either property stands on its own — `by` alone never rotates, `maxFileSizeGb` alone is the rotation that has always existed — because they are one operation and not two: every line of the merged dataset is routed to a bucket and each bucket owns a file, so the old `splitBySize` is the degenerate case of the new pass with a single bucket, and it is deleted rather than extended. The order is forced, key first and then size within each key, since rotating across mixed keys would only produce files that need splitting again. The file name carries a label — the sanitized key value, the rotation index, or both joined with an underscore: `sales_Italy.csv`, `sales_0.csv`, `sales_Italy_1.csv` — and an unsplit output keeps the `_0` it has always had, so no existing export is renamed. **Null and empty each get their own bucket** (`_null`, `_empty`) rather than a shared catch-all, which would merge two different things invisibly; a value that cannot go in a file name is sanitized, and two values that sanitize alike are kept apart by a numeric suffix rather than written into one file. Two bounds keep a mis-chosen key from taking the host down: at most 1000 files per consumer (`MAX_SPLIT_FILES` — "split by customer_id" fails the run with a named error instead of exhausting the descriptor table), and at most 64 streams open at once (`MAX_OPEN_SPLIT_STREAMS`), the least recently written bucket being closed and reopened in append mode when its next line arrives — which works because each bucket's byte counter lives in the bucket and not in its stream. Rejected at compile rather than at run time: a `split.by` key the consumer does not have, a hidden key on a positional (`CSV`/`TXT`) output whose file never carries that column, a cap that is not greater than zero, a split on an `API` output (a response is not a file), and a `split` block that sets neither property. `maximumFileSize` keeps working as the shorthand for `split.maxFileSizeGb`; setting both spellings on one consumer is rejected, naming the migration
+
+- **`remora discover <producer> --replace`**, which rewrites a producer's dimensions from the source instead of merging into them: a dimension the source has no field for is dropped, and a declared type the source disagrees with is overwritten. It is the deliberate way back to what discovery used to do to every producer by default (see Fixed), and it exists for one situation — bootstrapping a producer created by `remora create-producer`, whose template ships five placeholder dimensions that nothing else removes. It is also the only way to delete a dimension whose column is gone upstream. Dimensions no discovery could speak for are kept even here: a `sourceFilename` dimension holds the name of the file a row was read from, not a field of the data
+
+- **A consumer's filters and the run's record limit now reach a SQL source as a `WHERE` and a `LIMIT`.** Redshift and Postgres both build their staging statement through the same `SqlQueryBuilder`, so both now stage `SELECT "id", "country" FROM "public"."accounts" WHERE "country" IN ('Italy', 'Spain') LIMIT 100` where they used to read the table whole and discard the excess here — which on a wide table is the difference between a few hundred rows over the wire and all of them. The filters are the consumer's own and the ones an API or queue request carries; the limit is `remora run -l` or `limit` on a request, under the rule the Delta Share hint already used (`PushdownEngine.resolveLimit`: no filters, `distinct`, `distinctOn`, `pivot`, `"onFail": "skip"` validation, `recordSelector` or row-fanning group, since each one can drop a record a bounded read would have to make up for). **Unlike the Delta Share hints this is exact** — whatever the predicate prunes is gone, where a share server may ignore a hint and hand back everything — so the whole design is one-directional: the clause may be WIDER than the per-record filter and must never be narrower, and anything that cannot be guaranteed that way is left out of the statement and applied per record exactly as before. What that rules out, deliberately: a `datetime` column, where an ISO bound names one instant to the client and another to the server depending on how the value was parsed and what the session timezone is; a `boolean` one, which `TypeCaster` reads back through `Boolean(value)` and so reads a staged "false" as TRUE; every negation and null check, which a missing value satisfies; the regex operators, since a JS regular expression is not a POSIX one; a filter value spelled like an ISO date on a `string` column, which the record evaluator compares as an instant and the server as text; and a value carrying a backslash or a control character. What is pushed: `equals`, `in`, `contains`, `startsWith` and `endsWith` on text, and `equals`, `in`, `between` and the four comparisons on numbers — the `%` and `_` a LIKE pattern may pick up from a value only ever match MORE than the literal they stand for, which is the safe direction. One check has no counterpart in the Delta Share path: a filter is pushed only once the SOURCE's own catalog agrees with the dimension's declared type, read from `information_schema` once per producer and only when there is a filter to type, because a producer may declare a `timestamptz` as a `string` and the statement comparing it as one either prunes the wrong rows or fails the staging read outright. A catalog that cannot be read — a Redshift external (Spectrum) table, a late-binding view — costs the pushdown and nothing else. As before, a filter on a field whose value is touched between the source and the filter (a transform, a default, a mask, a validation) is never pushed at all, and `"disablePushdown": true` on the producer turns the lot off. One visible consequence of the limit being exact: `remora run c_big -l 100` against a SQL source now stages 100 rows, so it exports at most 100 — where the run's own limit is enforced per worker and could emit up to `100 x workers`
+
+- **A `presigned-url` engine, usable as a source and as a destination.** It covers the case where the bucket is reachable only through a URL something else signs — typically a Lambda behind API Gateway — so remora needs no AWS credentials for it at all: a signed URL authenticates itself. Every transfer is two calls, mint then transfer. The mint call is an ordinary authenticated HTTP request (`oidc` for an M2M-protected gateway, or `bearer-token`, `api-key`, `username-password`, `none`; `oidc` reuses the same cached, auto-refreshing `OidcTokenProvider` delta-share uses, so the token is fetched per process rather than per file), and the signed URL that comes back is transferred with no credentials whatsoever. Where the URL sits in the response is configuration, not convention: `signedUrlField` takes a dot path, `fileNameParam` names the field the file name travels in — body on a `POST`/`PUT`/`PATCH`, query parameter otherwise — and `signedHeadersField` carries headers the signature covers, which must be echoed verbatim or S3 answers `403 SignatureDoesNotMatch`. As a source the file is staged locally by `ready()` and parsed by the producer's `fileType` like any other file; as a destination `move`/`transformAndMove` PUT it. **The URL is minted per file, immediately before its transfer** — the reason the endpoint is configured rather than the URL: a URL in a config expires, and addresses one object, whereas a consumer rotating by `maximumFileSize` writes several and each now gets its own. Four limits are deliberate and refused or documented rather than worked around: a `%` pattern in `fileKey` is refused outright (the endpoint signs one named object and there is nothing on this side to list the rest); a single object cannot exceed S3's 5 GB limit for one `PUT`, which `maximumFileSize` rotation sidesteps by shipping several objects; there is no resume, so a dropped connection restarts the file; and `init()` deliberately does not probe the way the `http-api` driver does, because minting is not a free health check — it would leave a live signed URL behind every time `verify` ran. The transfers carry no timeout either, since a clock does not measure whether a multi-gigabyte download is healthy; the mint call, a request/response, keeps one. Signed URLs are never logged whole — only origin and path, the same rule `StateEmitter` applies, since the signature in the query string is a working credential
+
+### Changed
+
+- **Output rotation now counts the bytes it WRITES, not the bytes it has read.** `splitBySize` compared `readStream.bytesRead` against the cap, which is the reader's position in the source and includes the 64 KB the stream reads ahead — so the first rotation fired as soon as the read-ahead crossed the limit rather than when the file it was writing reached it, and on a small cap that was effectively the first line. The split pass counts each bucket's own written bytes, which it has to: with a key split there is one reader and many files, and the reader's position says nothing about any of them. A consumer setting a cap therefore gets fewer, fuller files than before — the same data, divided at different boundaries. The export also stops rediscovering those files with a `readdirSync` of the scope folder and taking their order from the directory: the split pass returns the files it wrote, each with the label that names it at the destination
+
+- **`nullable` on a producer dimension now decides what a MISSING value becomes**, where it used to be documentation and nothing else. It defaults to `true`, so a missing value — `null`, `undefined`, or the empty field a SQL `NULL` is staged as — is kept as `null`; a dimension declaring `nullable: false` asks for the empty value of its own type instead (`''` for `string`, `0` for `number`, `false` for `boolean`). A `datetime` has no empty value to give — any instant invented here would be a real date no reader could tell from one the source sent — so it stays `null` whatever is declared. An empty string IS a value for a `string` dimension and is never read as missing, which is also why nothing changes for the string columns of an existing project. The flag is read in exactly one place, `TypeCaster.castDimension`, which is now the single entry point for giving a declared dimension's value its type: the file parsers (CSV, JSON/JSONL, fixed-width), the in-memory read a SQL query returns, and the re-read of remora's own CSV working dataset all go through it. To reject a missing value rather than fill it, use a consumer field `required` validation, as before
+
+- The three stream-to-lines readers (delimited, Excel, XML) that lived on `S3SourceDriver` are now a shared `StreamReader`, which `S3SourceDriver._get` and the new presigned source both call. They never touched S3 state; keeping the file-type switch in one place is what lets a new transport driver be the fetch and nothing else
+
+- **The configuration archive is called `remora.zip`.** `remora package` wrote `./remora-config.zip` when no `-o, --output` was given and the worker's `downloaddeployment` offered the same name, so the file a project deploys carried a name that matches neither the directory it holds (`remora`) nor the command that builds it. Both now write `remora.zip`, which is also the name the worker's own fetch errors have always used for it. Nothing reads the archive by name — `deploy` uploads it as the `remora_config` form field regardless, and `REMORA_CONFIG_URI` is whatever URI it points at — so this only changes what lands on disk: a pipeline that hardcodes `remora-config.zip` either passes `-o` or expects the new name
+
+### Removed
+
+- **The resource folder names in `project.json`.** `sources`, `producers`, `consumers`, `schemas` and `codeSets` each took a list of paths, so a project could put its consumers anywhere and nothing outside the loader could find a resource from its path alone: `compile`, `create-producer`, `create-consumer` and `init` had already hardcoded `remora/producers` and friends, and only `Environment` read the lists — which meant a project that moved them passed `init` and then failed on the first command that wrote a file. The folders are now fixed by `CONSTANTS.RESOURCE_FOLDERS` and spelled in one place: `/sources`, `/producers` and `/consumers` must exist, and `/schemas` and `/code_sets` are read when they are there and ignored when they are not — so a code set no longer needs declaring at all, it just needs the folder. The five keys are still accepted by the schema and are ignored: a project that declares them loads exactly as before and logs one warning naming what to delete, which is also the first warning the loader has ever raised, so `load()` now prints warnings instead of dropping them
+
+- **`settings.sql` on a producer.** It was documented as the name of a file holding a hand-written statement to run instead of the generated one, and nothing in remora ever resolved that file — on a SQL source it was refused by name, and on every other kind of source it was read by nothing at all, so a producer could carry it and quietly not use it. `sqlTable` plus the producer's dimensions say the same thing and say it in one place: remora builds the `SELECT` itself, from the table and the columns the producer declares, which is what lets every identifier be checked against an identifier pattern before anything reaches the server. The option is now gone from the type, the JSON schema, the preview's summary of what a producer reads, and the docs. Because producer `settings` is `additionalProperties: false`, this moves the refusal from staging time to load time and widens it to every source: a project that still declares `settings.sql` fails `remora compile`/`verify` naming the property, instead of validating and then throwing on the first run that touches the source. Remove the property; if it named a query you actually want, the table goes in `sqlTable` and the columns go in `dimensions`. Two unrelated fields keep the name and are untouched: `measures[].sql` (the expression behind a metric) and a consumer join's `sql`
+
+### Fixed
+
+- **`remora discover` overwrote hand-authored dimension properties.** It rewrote the producer's `dimensions` wholesale from what it had just found, so every property discovery knows nothing about was dropped and replaced with the inferred default: `pk` went, every hand-written description became `Auto-mapped field: <name>`, and `alias`, `format`, `mask`, `classification` and `synth` went with them — as did the producer's `measures`, which were rewritten to `[]`. `pk` is not cosmetic, it is what `synth` and the reference checks read, so re-running discovery on a producer anyone had curated since the last run silently un-curated it. That made the command one you could only safely run once, which is exactly backwards now that discovery on a SQL source finally finds the columns a producer has not declared yet: "add a column upstream, re-discover" was the workflow it broke. Discovery now **merges** into the producer as it is written on disk. An existing dimension keeps everything it declares and gains only what it declared nothing for — in practice `nullable: false`, when the source states the column is `NOT NULL` — and a dimension is matched by its `alias` where it has one, including an `alias` that reaches into a field (`address.city` matches the field `address`), so a renamed or decomposed column is recognised rather than re-added as a duplicate. A **declared type is never changed**, even where the source disagrees: a producer is free to read a `timestamptz` as a `string` or a padded `char(n)` as one, so the disagreement is printed and the decision left to the author. A dimension the source has no field for is kept and printed rather than deleted. Everything outside `dimensions` — the producer's description, settings and measures — is now written back exactly as it was, because the merge starts from the file on disk rather than from the loaded (expanded) resource. The command prints what it did: what it added, what it filled in, what it left alone and why
+
+- **`remora discover` on a SQL source reported success and discovered nothing.** It sampled the statement `SqlQueryBuilder.buildProducerSelect` emits — which is built FROM the producer's dimensions — and then derived the dimensions from what came back, so it could only ever rediscover the columns that were already declared. Pointed at a fourteen-column table, a producer naming one column discovered exactly that one column and printed `✔ Producer has been successfully discovered`: the same "wrong answer, no error" shape as the `NextToken` truncation, and it made the command useless for the one thing it exists for, since a table's unmapped columns are precisely what you run it to find. Discovery on a SQL source now reads the table's own columns out of `information_schema.columns` — the table named by `settings.sqlTable`, never the producer's dimensions — through one small statement that reads no data at all. Because the catalog states the types, nothing is inferred either: a column's SQL type decides the dimension type, so a `numeric` is a `number` even when every value sampled from it happened to be whole and a `timestamptz` is a `datetime` even when the sample was empty, where value inference would have called both a string. A column the table declares `NOT NULL` is written back with `"nullable": false`, which no sample could establish; a nullable one carries nothing, since `nullable` already defaults to true. `information_schema` rather than a dialect catalog keeps Redshift and Postgres answering identically, at one known cost: a Redshift external (Spectrum) or late-binding-view column is not in it and so is not discovered. A producer with no `sqlTable` is now refused before anything is sent, naming the setting, and a table name that matches nothing fails saying so — including that names are matched with the case they were created with, since both engines fold an unquoted identifier to lower case
+
+- **A NULL in a `number` column was silently staged as `0`.** Every source that stages through CSV — SQL (Redshift, Postgres), Delta Share, and local or S3 CSV files — writes a NULL as an empty field, and the cast read that empty field as a value: `algo.hasVal` only rejects `null` and `undefined`, so `''` went through to `Number('')`, which is **`0`**. A missing value therefore became a REAL one, and 0 joins, aggregates, sorts and filters like a 0 the source actually holds — a null contract id quietly became contract id 0. It was found by reconciling a 4.3-million-row Redshift table against the cluster's own `count(distinct)`, which came back exactly one too high on each affected column because SQL skips NULLs and remora had turned them into a value; a local Postgres table with a NULL `numeric` and a NULL `integer` reproduced it independently, as did a boolean (`Boolean('')` is `false`) and a datetime (which threw outright). A missing value is now resolved against the dimension's `nullable` declaration — null by default, the type's empty value when the column declares `nullable: false` — so what remora writes for it is a decision the config makes rather than an artefact of the cast. **This changes what existing projects export**: a number column that holds NULLs and has been emitting `0` for them will now emit `null`, and an output declaring `omitNull` will drop the key entirely. Declare `nullable: false` on the dimension to keep the old value. String columns are unaffected: `''` is a value for a string, not a missing one
+
+- **An export could silently lose its last row.** `readline.Interface` is an EventEmitter whose own async iterator pauses and resumes it, so a loop that awaits — `for await (const line of lineReader)` with an `await` in the body — lets the iterator buffer; if the input ends while it is paused, the interface closes and the next pull resumes a closed interface, throwing `ERR_USE_AFTER_CLOSE`. The throw escapes the loop, so the final line is never written. Whether it happened was decided by where the last read landed against the stream's high-water mark, which makes it deterministic per output size and invisible on small inputs: the same consumer lost a row every run at one data volume and none at another, and a project could pass every test it had and still ship short files. A TXT export of 85,928 rows wrote 85,927 of them. Every loop that awaited per line was exposed — the local and S3 `TXT`/`CSV`/`API` exports, the `distinct` and `lineCount` post-processing passes, the dataset captured for `code_lookup` and consumer-as-input, output splitting by `maximumFileSize`, and the worker's own per-chunk read. `JSON`/`JSONL` exports were never affected: they are copied, not streamed line by line. Line reading now goes through `readLines` (`packages/core/src/LineReader.ts`), which reads the stream directly — a plain Readable has no close to be caught out by, so a caller may await freely between lines. Its line splitting is the same as readline's with `crlfDelay: Infinity` (CRLF however the chunks fall across it, LF, a lone CR, and a final line with no terminator), verified against readline over several thousand randomized inputs and chunk boundaries
+
+- **Two runs of one consumer exporting at the same time could produce a file holding neither of them.** A configured `exportName` is a fixed name, which is the point of it — so a run started by CRON while an API call was still exporting targeted the same path, and `transformAndMove` opened it with a truncating write stream. Both streams then wrote into one file at their own offsets, and the result was a file with the two runs' lines interleaved and a torn line where they met: not one run's output, not the other's, and not valid for its format. It reproduced in roughly one of three attempts with four concurrent runs. An export is now built under a unique name beside its destination and renamed into place, which is atomic within a filesystem, so concurrent runs of one consumer degrade to a clean last-writer-wins — one run's output, whole — instead of corruption. This does not make the losing runs' output survive: a fixed `exportName` still means the last run to finish owns the file, which is what it has always meant. `move` stages the same way, and a staging file is removed when an export fails
+
+- **A failed export finished the run as a success.** The destination drivers catch their own errors, log them and answer `res: false` rather than throwing, and nothing upstream read the flag: the empty key it carries became the run's `resultUri`, so a write that never landed was reported to the caller — API response, usage record, CLI output — as a completed run, with the only trace in the logs. `_unifingResult` made it worse for a consumer writing several files, folding the per-file results with `some`, so one successful file reported the whole export as done. The fold is now `every`, and `launch` refuses the run on a false result, naming the consumer and its destinations and pointing at the logged driver error for the cause. A run that matches no rows is unaffected: writing an empty file is a successful export, not a failed one
+
+- **A remote configuration that failed to extract wiped the one already running.** `Environment._saveConfigFile` deleted every file under the local `remora` directory and only then called `extractAllTo`, on the assumption that an archive which had been constructed was an archive which could be read. It is not: adm-zip parses the entry table lazily, so `new AdmZip(buffer)` succeeds on a malformed archive and the first real read happens inside the extract — after the deletion. A configuration re-uploaded to S3 while a worker was polling it therefore took the worker from a working project to an empty directory, and the process kept serving with no configuration at all. The archive is now read in full before anything live is touched, and extracted to a staging directory beside `remora` that is swapped in only once every entry is on disk, so a refresh either replaces the configuration or leaves the previous one exactly as it was. The commonest way to produce such an archive is to add a file to the existing one rather than rebuild it — an append leaves two entries under the same name, which adm-zip 0.6 rejects as ambiguous because the two can hold different content — so the failure now says that, naming the archive, instead of reporting `Error at bucket ..., while obtainign remora.zip` as though the bucket were at fault; the S3 fetch and the extract are no longer wrapped in the same `catch`
+
+- **`region` and `bucket` could not be `{ENV_VAR}` references on an S3 or Redshift source.** Every other authentication field goes through `SecretManager.replaceSecret`, and these two did not: `RedshiftDriver`, `S3SourceDriver` and `S3DestinationDriver` each read `authentication['region']` (and the S3 pair `authentication['bucket']`) raw and handed it to the AWS SDK. So a source declaring `"region": "{AWS_REGION}"` — the notation the rest of the file uses, and one the project validator already checks and resolves at compile time — failed at run time with `Region not accepted: region="{AWS_REGION}" is not a valid hostname component`, an SDK error naming neither the source nor the variable that went unresolved. The worst of it was that `remora compile` PASSED: the validator confirms the variable is set, so a project could be validated, packaged and deployed, and only fail when a run first touched the source. `QueueManager` had always resolved its region correctly, which is why this survived — the bug was in three drivers and not in the fourth. All four now do the same thing
+
+- **A Postgres source that could not be reached said so without saying why.** A refused or timed-out connection is the commonest thing to go wrong with a SQL source, and it was the one case that reported nothing: node resolves `localhost` to both `::1` and `127.0.0.1`, and when every address fails `pg` rejects with an `AggregateError` whose `message` is the **empty string** and whose actual reasons sit in `errors`. Passing that through produced `Unable to connect to postgres://localhost:5432/analytics for source "s_warehouse": ` — a message naming the host, the database and the source, and then stopping exactly where the reason belongs. The reason is now taken from whichever of `message`, `code` and `errors` carries one, so the same failure reads `...: ECONNREFUSED: connect ECONNREFUSED ::1:5432; connect ECONNREFUSED 127.0.0.1:5432`, and duplicate addresses are collapsed because a host refusing on both of its addresses is one fact, not two. A failure that does report a message — a wrong password, a database that does not exist — is unchanged
+
+- **`synth` no longer fills a column with nulls when a built-in code set is auto-bound to it by name but cannot produce its declared type.** Built-in sets are matched on the field name alone, so a numeric surrogate key like `heb_quality_gaps_measure_code_sk` matched the `measure_code` binding and was written as a HEDIS code (`"CBP"`); read back as its declared `number` it cast to `NaN`, which serializes as `null`, and every row of the column was empty. A code set now has to be able to produce the dimension's type: an automatic binding that cannot is only a name-based guess and is dropped, so the column is generated from its own type (that surrogate key is a dense numeric key again), while an explicit `synth.codeSet` or `synth.allowedValues` that cannot is a configuration error and is refused by name
+
+## V 2.1.1 - 2026-09-23
+
+### Added
+
+- **SQL sources can be run.** `aws-redshift` has been a valid engine since the beginning, but `RedshiftDriver.ready` — the one method every run goes through to stage a producer — threw `Not implemented yet`, so a Redshift producer could be configured, validated and sampled, and never executed. It runs now, and a new **`postgres`** engine runs beside it: Amazon RDS, Aurora PostgreSQL, or a self-hosted server, over the native wire protocol. A producer on either names its table with `sqlTable` (`table` or `schema.table`, falling back to the source's `schema`) and lists the columns it wants as dimensions, with `alias` naming the column where it differs from the dimension name; remora builds the `SELECT` from exactly those dimensions and stages the result as a local CSV dataset, which is the same thing a Delta share does and the reason the rest of the pipeline needed no change. The statement is built in one place from the producer's own config, and every identifier in it is checked against an identifier pattern rather than escaped, so a table or alias that is not a name is refused where it is written instead of reaching the server as syntax. Both engines are now in `STAGEABLE_ENGINES`, so `remora run`, `remora sample` on a consumer and the preview's **Data** tab all reach them
+- **Postgres authenticates with an RDS IAM auth token**, with `method: "iam"` or `"arn"`, so there is no database password in the configuration at all: a short-lived token is signed per connection, by the SDK default credential chain or by an assumed role, through the same `resolveAwsCredentials` every other AWS driver uses. `method: "username-password"` keeps the ordinary path, where the password is an `{ENV_VAR}` reference like any other secret. TLS is on by default for a remote host (`sslMode`: `require`, or `verify-full` to verify the chain) and off for `localhost`, where a local database usually speaks none
+
+### Changed
+
+- A Redshift statement is **polled for up to 30 minutes** with exponential backoff, instead of 20 cycles of a 1 ms sleep. The Data API is submit-and-poll, and the old loop was sized for the one-row warm-up query in `init`: a table scan legitimately sits in `STARTED` for minutes and would have been abandoned as a timeout. Provisioned clusters also work now — a statement carries `ClusterIdentifier`/`DbUser` when the source declares `clusterId` rather than a `workgroup`, where before only Serverless workgroups were ever sent
+
+- **A live run now says what it read, without `REMORA_DEBUG_MODE`.** Everything that reported what a run actually *did* was `logger.log` — DEBUG, which returns on its first line unless the variable was `true` when the process started — so a deployed worker logged a launch line, a completion line and nothing in between. Restarting it with debug on is not an option when the question is about a run that already happened, and turning it on permanently means every diagnostic line of every driver, forever. Three things are now `info`, chosen because each one answers a question that cannot be answered after the fact from anywhere else. **The options the run was invoked with**: the launch line named the consumer, the invoker, the user and the producer count but never the request, and `IUsageStat` does not keep it either, so there was no way to tell whether a run had been handed filters and a limit or had run on the consumer's own config — it now reads `..., producer(s): 2, options: 2 filter(s), limit 100)`, and is unchanged on a run that carried no options. **What a Delta Share query returned**: `Delta Share table "x" returned N file(s) in Yms` is the number that says whether the pushdown pruned anything, and it now names the hints that were sent with it (`(pushdown: predicate, limit 100)`) so the count can be read against what was actually asked of the share; the predicate tree itself stays on the debug line, since it is a JSON document and not a thing to put in every run's log. **What the read staged**: records, dataset files and bytes per producer, which is the run's own account of the volume it pulled. Nothing new is computed and nothing else moved level — the remaining delta-share diagnostics, the per-file skips and the metadata dump are still debug
+
+- **The options an invocation asks a run for are now checked at the door.** A consumer run can be handed `filters` and a `limit` by an API call (`POST` execute) or a queue message, and both entry points took whatever arrived: the queue path checked that `options` was a non-array object and CAST it, the HTTP route read `req.body.options` and passed it straight to the orchestrator. So a `limit` could arrive as `"100"`, as `0` or as `-5` and bound nothing; a `filters` could be an object; an `operator` could be a word `RequestExecutor` has no case for; and a `member` could name no field of the consumer — which is the worst of them, because an unknown member reads `undefined` off every record and so *silently filters nothing*, returning a full dataset that looks like an answer. The rest surfaced as a throw from the middle of a run that had already queried the source and paid for the read. Both entry points now go through one check, `RequestValidator.validate`, before anything is read: `filters` an array of objects, `member` a field the consumer actually outputs (the expanded `alias ?? key` names, which is what a record holds when filters run), `operator` one of `FilterOperator`, `values` an array of strings of the arity the operator needs (`between` exactly two, `in` at least one, `isNull` none), `limit` a whole number between 1 and `MAX_REQUEST_LIMIT`, and `and`/`or` children validated the same way down to `MAX_REQUEST_FILTER_DEPTH`. Every rejection names the offending option and where it sits — `Invalid filter at filters[1].or[0]: consumer "sales" has no field named "regionn"` — and the API answers **400** instead of running and returning 500 or, worse, 200. Two things are refused outright: **`offset` and `order`**, which the request type carries but no stage of a run implements (the only code reading them is `PushdownEngine.resolveLimit`, where their presence merely suppresses the limit pushdown), and the **`sql`** operator, which `RequestExecutor` throws on rather than evaluating — accepting either would mean answering as if the request had been honoured. An unknown option key (`limits`, `filter`) is refused too, since the alternative is ignoring it silently. Bounds on what may be asked for live with the rest: `MAX_REQUEST_LIMIT` (10,000,000), `MAX_REQUEST_FILTERS` (100) and `MAX_REQUEST_FILTER_DEPTH` (5)
+
+### Fixed
+
+- **A ZIP producer read the whole archive into memory — and had in fact never run at all.** `ParseCompression` extracted a zip through AdmZip, which reads the entire archive into one buffer and then decompresses each member into a second one. On a 71 MB archive holding a 441 MB CSV that is **1.06 GB of peak RSS, 2.4x the member**: a file a 2 GB worker stages comfortably as plain CSV exhausts it the moment it arrives zipped. `.gz` and `.tar` beside it already streamed; only ZIP was left behind, the same shape of bug as the delta-share crash. A zip is now read entry by entry with `yauzl`, each member piped from the archive straight to disk, and the same file costs **136 MB peak — about 44 MB above an empty process** — bounded by the pipeline's own buffers rather than by what the archive holds. Two things fell out of the rewrite. The extraction **never ran**: `ExecutorScope.ensurePath` touches the dataset path as an empty *file* before a driver is called, and extracting an archive into that path went straight to `EEXIST: mkdir`, so ZIP and TAR were both dead on arrival on the local source driver. And an archive with more than one member is now read **in full** — `decompressToFile` returns every file it extracted rather than the directory it put them in, and `LocalSourceDriver.ready` fans those out exactly the way it already fans out a wildcard `fileKey`, where before a multi-member archive handed the run a directory path to read as though it were a data file. TAR reports its members the same way. Entry paths are still resolved and checked against the target directory before anything is opened (yauzl refuses a traversal name of its own accord; the `affirm` is the second line), an encrypted member is refused by name instead of writing its ciphertext, and `MAX_ARCHIVE_ENTRIES` (10 000) bounds what a single fileKey may fan out into. `c_canary_zip` in the canary reads a two-member archive and checks the row count against the unsplit source, because reading only the first member is a silent half-answer rather than a failure
+- **Redshift silently returned only the first page of every query.** `getStatementResult` never read `result.NextToken`, and the Data API pages results at roughly 1 MB — so any statement returning more than that came back **short, with no error**: `SQL_MAX_QUERY_ROWS` could not be honoured, a sample of a wide table was a fraction of what was asked for, and nothing anywhere said so. Not a crash but a wrong answer, which is worse, and it had to be fixed before a staging read could be built on the same call. Every read now follows the chain to the end. The in-memory paths (`query`, `execute`) stop at `SQL_MAX_QUERY_ROWS` and **log that they truncated**, rather than doing it quietly; staging accumulates nothing at all — each page is written to the dataset file before the next is fetched — and then **checks what it staged against the row count the cluster reports**, so a lost page fails the run by name instead of producing a short dataset that looks complete
+- Staging a table costs disk rather than memory, on both engines: Postgres reads through a server-side cursor inside a `READ ONLY` transaction, `FETCH`ing 10 000 rows per round trip and writing each batch before asking for the next, and the cursor is closed and the transaction ended even when the read is abandoned half way. Redshift does the same with result pages. Neither ever holds the table
+- **A run that read no rows crashed on the S3 upload.** `S3DestinationDriver` uploads every output through a multipart upload, and an empty dataset — a Delta Share (or any other source) that legitimately returned nothing for the period asked of it — produces zero parts, so the run ended on a `CompleteMultipartUpload` carrying an empty part list, which S3 rejects as `MalformedXML`. The error named neither the consumer nor the reason, so an empty read looked like a broken destination. An upload with no parts is now aborted and the object written directly as a zero-byte `PutObject`, which is what the local destination driver has always done: a no-data run lands an empty file and succeeds. A failed abort is also no longer able to mask the error that triggered it — it is logged with the key and upload id, and the original exception is the one that propagates
+
 ## V 2.1.0 - 2026-09-17
 
 ### Added

package/documentation/README.md

@@ -74,9 +74,8 @@ Producers extract data from sources and define the structure of the data.
 | `measures[].name` | Name of calculated measure | Any string without spaces |
 | `measures[].description` | Description of the measure | Any descriptive text |
 | `measures[].sql` | SQL expression to calculate the measure | Valid SQL expression (e.g., `SUM(field_name)`) |
-| `settings.sqlTable` | Source table name | Table name as string |
+| `settings.sqlTable` | Source table name, `table` or `schema.table` | Table name as string |
 | `settings.direct` | Whether to query directly without creating views | `true`, `false` |
-| `settings.sql` | Custom SQL for this producer | SQL query string |
 | `settings.fileKey` | File key for S3 sources | S3 file path |
 | `settings.fileType` | File format for file-based sources | `JSON`, `JSONL`, `CSV` |
 

package/documentation/default_resources/project.json

@@ -3,10 +3,6 @@
     "name": "Default Remora Project",
     "version": "1.0.0",
     "description": "This is the standard Remora project created when you run init.",
-    "consumers": ["/consumers"],
-    "producers": ["/producers"],
-    "sources": ["/sources"],
-    "schemas": ["/schemas"],
     "settings": {
         "SQL_MAX_QUERY_ROWS": 10000
     }