@forzalabs/remora 2.1.1 → 2.2.0

package/CHANGELOG.md

@@ -6,6 +6,58 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.2.0 - 2026-10-15
+
+**Upgrading from 2.1.** Three changes can alter what an existing project does, despite the minor version. `nullable` on a producer dimension is no longer documentation: it defaults to `true` and keeps a missing value as `null` as before, but a dimension that declares `"nullable": false` now gets the type's own empty value (`""`, `0`, `false`) where it used to get `null`. A `NULL` in a `number` column now stages as `null` rather than as `0`, so a consumer that was summing or filtering over those zeros sees different figures — the fix is in the staging of every CSV-staged source (SQL, Delta Share, local and S3 CSV). And `settings.sql` on a producer is gone from the schema, so a producer still carrying it fails `compile` — the property never did anything; delete it. Also worth a glance: `remora package` writes `./remora.zip` instead of `./remora-config.zip` when no `-o` is given, and the resource folder lists in `project.json` (`sources`, `producers`, `consumers`, `schemas`, `codeSets`) are ignored — the folders are fixed. They still validate, so nothing breaks, but a project pointing them somewhere else was already not being read that way.
+
+### Added
+
+- **`split` on a consumer: the output broken into several files by a field value, by a size cap, or by both.** `"split": { "by": ["country"], "maxFileSizeGb": 2 }` writes one file per country and rotates each of them at 2 Gb. Either property stands on its own — `by` alone never rotates, `maxFileSizeGb` alone is the rotation that has always existed — because they are one operation and not two: every line of the merged dataset is routed to a bucket and each bucket owns a file, so the old `splitBySize` is the degenerate case of the new pass with a single bucket, and it is deleted rather than extended. The order is forced, key first and then size within each key, since rotating across mixed keys would only produce files that need splitting again. The file name carries a label — the sanitized key value, the rotation index, or both joined with an underscore: `sales_Italy.csv`, `sales_0.csv`, `sales_Italy_1.csv` — and an unsplit output keeps the `_0` it has always had, so no existing export is renamed. **Null and empty each get their own bucket** (`_null`, `_empty`) rather than a shared catch-all, which would merge two different things invisibly; a value that cannot go in a file name is sanitized, and two values that sanitize alike are kept apart by a numeric suffix rather than written into one file. Two bounds keep a mis-chosen key from taking the host down: at most 1000 files per consumer (`MAX_SPLIT_FILES` — "split by customer_id" fails the run with a named error instead of exhausting the descriptor table), and at most 64 streams open at once (`MAX_OPEN_SPLIT_STREAMS`), the least recently written bucket being closed and reopened in append mode when its next line arrives — which works because each bucket's byte counter lives in the bucket and not in its stream. Rejected at compile rather than at run time: a `split.by` key the consumer does not have, a hidden key on a positional (`CSV`/`TXT`) output whose file never carries that column, a cap that is not greater than zero, a split on an `API` output (a response is not a file), and a `split` block that sets neither property. `maximumFileSize` keeps working as the shorthand for `split.maxFileSizeGb`; setting both spellings on one consumer is rejected, naming the migration
+
+- **`remora discover <producer> --replace`**, which rewrites a producer's dimensions from the source instead of merging into them: a dimension the source has no field for is dropped, and a declared type the source disagrees with is overwritten. It is the deliberate way back to what discovery used to do to every producer by default (see Fixed), and it exists for one situation — bootstrapping a producer created by `remora create-producer`, whose template ships five placeholder dimensions that nothing else removes. It is also the only way to delete a dimension whose column is gone upstream. Dimensions no discovery could speak for are kept even here: a `sourceFilename` dimension holds the name of the file a row was read from, not a field of the data
+
+- **A consumer's filters and the run's record limit now reach a SQL source as a `WHERE` and a `LIMIT`.** Redshift and Postgres both build their staging statement through the same `SqlQueryBuilder`, so both now stage `SELECT "id", "country" FROM "public"."accounts" WHERE "country" IN ('Italy', 'Spain') LIMIT 100` where they used to read the table whole and discard the excess here — which on a wide table is the difference between a few hundred rows over the wire and all of them. The filters are the consumer's own and the ones an API or queue request carries; the limit is `remora run -l` or `limit` on a request, under the rule the Delta Share hint already used (`PushdownEngine.resolveLimit`: no filters, `distinct`, `distinctOn`, `pivot`, `"onFail": "skip"` validation, `recordSelector` or row-fanning group, since each one can drop a record a bounded read would have to make up for). **Unlike the Delta Share hints this is exact** — whatever the predicate prunes is gone, where a share server may ignore a hint and hand back everything — so the whole design is one-directional: the clause may be WIDER than the per-record filter and must never be narrower, and anything that cannot be guaranteed that way is left out of the statement and applied per record exactly as before. What that rules out, deliberately: a `datetime` column, where an ISO bound names one instant to the client and another to the server depending on how the value was parsed and what the session timezone is; a `boolean` one, which `TypeCaster` reads back through `Boolean(value)` and so reads a staged "false" as TRUE; every negation and null check, which a missing value satisfies; the regex operators, since a JS regular expression is not a POSIX one; a filter value spelled like an ISO date on a `string` column, which the record evaluator compares as an instant and the server as text; and a value carrying a backslash or a control character. What is pushed: `equals`, `in`, `contains`, `startsWith` and `endsWith` on text, and `equals`, `in`, `between` and the four comparisons on numbers — the `%` and `_` a LIKE pattern may pick up from a value only ever match MORE than the literal they stand for, which is the safe direction. One check has no counterpart in the Delta Share path: a filter is pushed only once the SOURCE's own catalog agrees with the dimension's declared type, read from `information_schema` once per producer and only when there is a filter to type, because a producer may declare a `timestamptz` as a `string` and the statement comparing it as one either prunes the wrong rows or fails the staging read outright. A catalog that cannot be read — a Redshift external (Spectrum) table, a late-binding view — costs the pushdown and nothing else. As before, a filter on a field whose value is touched between the source and the filter (a transform, a default, a mask, a validation) is never pushed at all, and `"disablePushdown": true` on the producer turns the lot off. One visible consequence of the limit being exact: `remora run c_big -l 100` against a SQL source now stages 100 rows, so it exports at most 100 — where the run's own limit is enforced per worker and could emit up to `100 x workers`
+
+- **A `presigned-url` engine, usable as a source and as a destination.** It covers the case where the bucket is reachable only through a URL something else signs — typically a Lambda behind API Gateway — so remora needs no AWS credentials for it at all: a signed URL authenticates itself. Every transfer is two calls, mint then transfer. The mint call is an ordinary authenticated HTTP request (`oidc` for an M2M-protected gateway, or `bearer-token`, `api-key`, `username-password`, `none`; `oidc` reuses the same cached, auto-refreshing `OidcTokenProvider` delta-share uses, so the token is fetched per process rather than per file), and the signed URL that comes back is transferred with no credentials whatsoever. Where the URL sits in the response is configuration, not convention: `signedUrlField` takes a dot path, `fileNameParam` names the field the file name travels in — body on a `POST`/`PUT`/`PATCH`, query parameter otherwise — and `signedHeadersField` carries headers the signature covers, which must be echoed verbatim or S3 answers `403 SignatureDoesNotMatch`. As a source the file is staged locally by `ready()` and parsed by the producer's `fileType` like any other file; as a destination `move`/`transformAndMove` PUT it. **The URL is minted per file, immediately before its transfer** — the reason the endpoint is configured rather than the URL: a URL in a config expires, and addresses one object, whereas a consumer rotating by `maximumFileSize` writes several and each now gets its own. Four limits are deliberate and refused or documented rather than worked around: a `%` pattern in `fileKey` is refused outright (the endpoint signs one named object and there is nothing on this side to list the rest); a single object cannot exceed S3's 5 GB limit for one `PUT`, which `maximumFileSize` rotation sidesteps by shipping several objects; there is no resume, so a dropped connection restarts the file; and `init()` deliberately does not probe the way the `http-api` driver does, because minting is not a free health check — it would leave a live signed URL behind every time `verify` ran. The transfers carry no timeout either, since a clock does not measure whether a multi-gigabyte download is healthy; the mint call, a request/response, keeps one. Signed URLs are never logged whole — only origin and path, the same rule `StateEmitter` applies, since the signature in the query string is a working credential
+
+### Changed
+
+- **Output rotation now counts the bytes it WRITES, not the bytes it has read.** `splitBySize` compared `readStream.bytesRead` against the cap, which is the reader's position in the source and includes the 64 KB the stream reads ahead — so the first rotation fired as soon as the read-ahead crossed the limit rather than when the file it was writing reached it, and on a small cap that was effectively the first line. The split pass counts each bucket's own written bytes, which it has to: with a key split there is one reader and many files, and the reader's position says nothing about any of them. A consumer setting a cap therefore gets fewer, fuller files than before — the same data, divided at different boundaries. The export also stops rediscovering those files with a `readdirSync` of the scope folder and taking their order from the directory: the split pass returns the files it wrote, each with the label that names it at the destination
+
+- **`nullable` on a producer dimension now decides what a MISSING value becomes**, where it used to be documentation and nothing else. It defaults to `true`, so a missing value — `null`, `undefined`, or the empty field a SQL `NULL` is staged as — is kept as `null`; a dimension declaring `nullable: false` asks for the empty value of its own type instead (`''` for `string`, `0` for `number`, `false` for `boolean`). A `datetime` has no empty value to give — any instant invented here would be a real date no reader could tell from one the source sent — so it stays `null` whatever is declared. An empty string IS a value for a `string` dimension and is never read as missing, which is also why nothing changes for the string columns of an existing project. The flag is read in exactly one place, `TypeCaster.castDimension`, which is now the single entry point for giving a declared dimension's value its type: the file parsers (CSV, JSON/JSONL, fixed-width), the in-memory read a SQL query returns, and the re-read of remora's own CSV working dataset all go through it. To reject a missing value rather than fill it, use a consumer field `required` validation, as before
+
+- The three stream-to-lines readers (delimited, Excel, XML) that lived on `S3SourceDriver` are now a shared `StreamReader`, which `S3SourceDriver._get` and the new presigned source both call. They never touched S3 state; keeping the file-type switch in one place is what lets a new transport driver be the fetch and nothing else
+
+- **The configuration archive is called `remora.zip`.** `remora package` wrote `./remora-config.zip` when no `-o, --output` was given and the worker's `downloaddeployment` offered the same name, so the file a project deploys carried a name that matches neither the directory it holds (`remora`) nor the command that builds it. Both now write `remora.zip`, which is also the name the worker's own fetch errors have always used for it. Nothing reads the archive by name — `deploy` uploads it as the `remora_config` form field regardless, and `REMORA_CONFIG_URI` is whatever URI it points at — so this only changes what lands on disk: a pipeline that hardcodes `remora-config.zip` either passes `-o` or expects the new name
+
+### Removed
+
+- **The resource folder names in `project.json`.** `sources`, `producers`, `consumers`, `schemas` and `codeSets` each took a list of paths, so a project could put its consumers anywhere and nothing outside the loader could find a resource from its path alone: `compile`, `create-producer`, `create-consumer` and `init` had already hardcoded `remora/producers` and friends, and only `Environment` read the lists — which meant a project that moved them passed `init` and then failed on the first command that wrote a file. The folders are now fixed by `CONSTANTS.RESOURCE_FOLDERS` and spelled in one place: `/sources`, `/producers` and `/consumers` must exist, and `/schemas` and `/code_sets` are read when they are there and ignored when they are not — so a code set no longer needs declaring at all, it just needs the folder. The five keys are still accepted by the schema and are ignored: a project that declares them loads exactly as before and logs one warning naming what to delete, which is also the first warning the loader has ever raised, so `load()` now prints warnings instead of dropping them
+
+- **`settings.sql` on a producer.** It was documented as the name of a file holding a hand-written statement to run instead of the generated one, and nothing in remora ever resolved that file — on a SQL source it was refused by name, and on every other kind of source it was read by nothing at all, so a producer could carry it and quietly not use it. `sqlTable` plus the producer's dimensions say the same thing and say it in one place: remora builds the `SELECT` itself, from the table and the columns the producer declares, which is what lets every identifier be checked against an identifier pattern before anything reaches the server. The option is now gone from the type, the JSON schema, the preview's summary of what a producer reads, and the docs. Because producer `settings` is `additionalProperties: false`, this moves the refusal from staging time to load time and widens it to every source: a project that still declares `settings.sql` fails `remora compile`/`verify` naming the property, instead of validating and then throwing on the first run that touches the source. Remove the property; if it named a query you actually want, the table goes in `sqlTable` and the columns go in `dimensions`. Two unrelated fields keep the name and are untouched: `measures[].sql` (the expression behind a metric) and a consumer join's `sql`
+
+### Fixed
+
+- **`remora discover` overwrote hand-authored dimension properties.** It rewrote the producer's `dimensions` wholesale from what it had just found, so every property discovery knows nothing about was dropped and replaced with the inferred default: `pk` went, every hand-written description became `Auto-mapped field: <name>`, and `alias`, `format`, `mask`, `classification` and `synth` went with them — as did the producer's `measures`, which were rewritten to `[]`. `pk` is not cosmetic, it is what `synth` and the reference checks read, so re-running discovery on a producer anyone had curated since the last run silently un-curated it. That made the command one you could only safely run once, which is exactly backwards now that discovery on a SQL source finally finds the columns a producer has not declared yet: "add a column upstream, re-discover" was the workflow it broke. Discovery now **merges** into the producer as it is written on disk. An existing dimension keeps everything it declares and gains only what it declared nothing for — in practice `nullable: false`, when the source states the column is `NOT NULL` — and a dimension is matched by its `alias` where it has one, including an `alias` that reaches into a field (`address.city` matches the field `address`), so a renamed or decomposed column is recognised rather than re-added as a duplicate. A **declared type is never changed**, even where the source disagrees: a producer is free to read a `timestamptz` as a `string` or a padded `char(n)` as one, so the disagreement is printed and the decision left to the author. A dimension the source has no field for is kept and printed rather than deleted. Everything outside `dimensions` — the producer's description, settings and measures — is now written back exactly as it was, because the merge starts from the file on disk rather than from the loaded (expanded) resource. The command prints what it did: what it added, what it filled in, what it left alone and why
+
+- **`remora discover` on a SQL source reported success and discovered nothing.** It sampled the statement `SqlQueryBuilder.buildProducerSelect` emits — which is built FROM the producer's dimensions — and then derived the dimensions from what came back, so it could only ever rediscover the columns that were already declared. Pointed at a fourteen-column table, a producer naming one column discovered exactly that one column and printed `✔ Producer has been successfully discovered`: the same "wrong answer, no error" shape as the `NextToken` truncation, and it made the command useless for the one thing it exists for, since a table's unmapped columns are precisely what you run it to find. Discovery on a SQL source now reads the table's own columns out of `information_schema.columns` — the table named by `settings.sqlTable`, never the producer's dimensions — through one small statement that reads no data at all. Because the catalog states the types, nothing is inferred either: a column's SQL type decides the dimension type, so a `numeric` is a `number` even when every value sampled from it happened to be whole and a `timestamptz` is a `datetime` even when the sample was empty, where value inference would have called both a string. A column the table declares `NOT NULL` is written back with `"nullable": false`, which no sample could establish; a nullable one carries nothing, since `nullable` already defaults to true. `information_schema` rather than a dialect catalog keeps Redshift and Postgres answering identically, at one known cost: a Redshift external (Spectrum) or late-binding-view column is not in it and so is not discovered. A producer with no `sqlTable` is now refused before anything is sent, naming the setting, and a table name that matches nothing fails saying so — including that names are matched with the case they were created with, since both engines fold an unquoted identifier to lower case
+
+- **A NULL in a `number` column was silently staged as `0`.** Every source that stages through CSV — SQL (Redshift, Postgres), Delta Share, and local or S3 CSV files — writes a NULL as an empty field, and the cast read that empty field as a value: `algo.hasVal` only rejects `null` and `undefined`, so `''` went through to `Number('')`, which is **`0`**. A missing value therefore became a REAL one, and 0 joins, aggregates, sorts and filters like a 0 the source actually holds — a null contract id quietly became contract id 0. It was found by reconciling a 4.3-million-row Redshift table against the cluster's own `count(distinct)`, which came back exactly one too high on each affected column because SQL skips NULLs and remora had turned them into a value; a local Postgres table with a NULL `numeric` and a NULL `integer` reproduced it independently, as did a boolean (`Boolean('')` is `false`) and a datetime (which threw outright). A missing value is now resolved against the dimension's `nullable` declaration — null by default, the type's empty value when the column declares `nullable: false` — so what remora writes for it is a decision the config makes rather than an artefact of the cast. **This changes what existing projects export**: a number column that holds NULLs and has been emitting `0` for them will now emit `null`, and an output declaring `omitNull` will drop the key entirely. Declare `nullable: false` on the dimension to keep the old value. String columns are unaffected: `''` is a value for a string, not a missing one
+
+- **An export could silently lose its last row.** `readline.Interface` is an EventEmitter whose own async iterator pauses and resumes it, so a loop that awaits — `for await (const line of lineReader)` with an `await` in the body — lets the iterator buffer; if the input ends while it is paused, the interface closes and the next pull resumes a closed interface, throwing `ERR_USE_AFTER_CLOSE`. The throw escapes the loop, so the final line is never written. Whether it happened was decided by where the last read landed against the stream's high-water mark, which makes it deterministic per output size and invisible on small inputs: the same consumer lost a row every run at one data volume and none at another, and a project could pass every test it had and still ship short files. A TXT export of 85,928 rows wrote 85,927 of them. Every loop that awaited per line was exposed — the local and S3 `TXT`/`CSV`/`API` exports, the `distinct` and `lineCount` post-processing passes, the dataset captured for `code_lookup` and consumer-as-input, output splitting by `maximumFileSize`, and the worker's own per-chunk read. `JSON`/`JSONL` exports were never affected: they are copied, not streamed line by line. Line reading now goes through `readLines` (`packages/core/src/LineReader.ts`), which reads the stream directly — a plain Readable has no close to be caught out by, so a caller may await freely between lines. Its line splitting is the same as readline's with `crlfDelay: Infinity` (CRLF however the chunks fall across it, LF, a lone CR, and a final line with no terminator), verified against readline over several thousand randomized inputs and chunk boundaries
+
+- **Two runs of one consumer exporting at the same time could produce a file holding neither of them.** A configured `exportName` is a fixed name, which is the point of it — so a run started by CRON while an API call was still exporting targeted the same path, and `transformAndMove` opened it with a truncating write stream. Both streams then wrote into one file at their own offsets, and the result was a file with the two runs' lines interleaved and a torn line where they met: not one run's output, not the other's, and not valid for its format. It reproduced in roughly one of three attempts with four concurrent runs. An export is now built under a unique name beside its destination and renamed into place, which is atomic within a filesystem, so concurrent runs of one consumer degrade to a clean last-writer-wins — one run's output, whole — instead of corruption. This does not make the losing runs' output survive: a fixed `exportName` still means the last run to finish owns the file, which is what it has always meant. `move` stages the same way, and a staging file is removed when an export fails
+
+- **A failed export finished the run as a success.** The destination drivers catch their own errors, log them and answer `res: false` rather than throwing, and nothing upstream read the flag: the empty key it carries became the run's `resultUri`, so a write that never landed was reported to the caller — API response, usage record, CLI output — as a completed run, with the only trace in the logs. `_unifingResult` made it worse for a consumer writing several files, folding the per-file results with `some`, so one successful file reported the whole export as done. The fold is now `every`, and `launch` refuses the run on a false result, naming the consumer and its destinations and pointing at the logged driver error for the cause. A run that matches no rows is unaffected: writing an empty file is a successful export, not a failed one
+
+- **A remote configuration that failed to extract wiped the one already running.** `Environment._saveConfigFile` deleted every file under the local `remora` directory and only then called `extractAllTo`, on the assumption that an archive which had been constructed was an archive which could be read. It is not: adm-zip parses the entry table lazily, so `new AdmZip(buffer)` succeeds on a malformed archive and the first real read happens inside the extract — after the deletion. A configuration re-uploaded to S3 while a worker was polling it therefore took the worker from a working project to an empty directory, and the process kept serving with no configuration at all. The archive is now read in full before anything live is touched, and extracted to a staging directory beside `remora` that is swapped in only once every entry is on disk, so a refresh either replaces the configuration or leaves the previous one exactly as it was. The commonest way to produce such an archive is to add a file to the existing one rather than rebuild it — an append leaves two entries under the same name, which adm-zip 0.6 rejects as ambiguous because the two can hold different content — so the failure now says that, naming the archive, instead of reporting `Error at bucket ..., while obtainign remora.zip` as though the bucket were at fault; the S3 fetch and the extract are no longer wrapped in the same `catch`
+
+- **`region` and `bucket` could not be `{ENV_VAR}` references on an S3 or Redshift source.** Every other authentication field goes through `SecretManager.replaceSecret`, and these two did not: `RedshiftDriver`, `S3SourceDriver` and `S3DestinationDriver` each read `authentication['region']` (and the S3 pair `authentication['bucket']`) raw and handed it to the AWS SDK. So a source declaring `"region": "{AWS_REGION}"` — the notation the rest of the file uses, and one the project validator already checks and resolves at compile time — failed at run time with `Region not accepted: region="{AWS_REGION}" is not a valid hostname component`, an SDK error naming neither the source nor the variable that went unresolved. The worst of it was that `remora compile` PASSED: the validator confirms the variable is set, so a project could be validated, packaged and deployed, and only fail when a run first touched the source. `QueueManager` had always resolved its region correctly, which is why this survived — the bug was in three drivers and not in the fourth. All four now do the same thing
+
+- **A Postgres source that could not be reached said so without saying why.** A refused or timed-out connection is the commonest thing to go wrong with a SQL source, and it was the one case that reported nothing: node resolves `localhost` to both `::1` and `127.0.0.1`, and when every address fails `pg` rejects with an `AggregateError` whose `message` is the **empty string** and whose actual reasons sit in `errors`. Passing that through produced `Unable to connect to postgres://localhost:5432/analytics for source "s_warehouse": ` — a message naming the host, the database and the source, and then stopping exactly where the reason belongs. The reason is now taken from whichever of `message`, `code` and `errors` carries one, so the same failure reads `...: ECONNREFUSED: connect ECONNREFUSED ::1:5432; connect ECONNREFUSED 127.0.0.1:5432`, and duplicate addresses are collapsed because a host refusing on both of its addresses is one fact, not two. A failure that does report a message — a wrong password, a database that does not exist — is unchanged
+
+- **`synth` no longer fills a column with nulls when a built-in code set is auto-bound to it by name but cannot produce its declared type.** Built-in sets are matched on the field name alone, so a numeric surrogate key like `heb_quality_gaps_measure_code_sk` matched the `measure_code` binding and was written as a HEDIS code (`"CBP"`); read back as its declared `number` it cast to `NaN`, which serializes as `null`, and every row of the column was empty. A code set now has to be able to produce the dimension's type: an automatic binding that cannot is only a name-based guess and is dropped, so the column is generated from its own type (that surrogate key is a dense numeric key again), while an explicit `synth.codeSet` or `synth.allowedValues` that cannot is a configuration error and is refused by name
+
 ## V 2.1.1 - 2026-09-23
 
 ### Added

package/documentation/README.md

@@ -74,9 +74,8 @@ Producers extract data from sources and define the structure of the data.
 | `measures[].name` | Name of calculated measure | Any string without spaces |
 | `measures[].description` | Description of the measure | Any descriptive text |
 | `measures[].sql` | SQL expression to calculate the measure | Valid SQL expression (e.g., `SUM(field_name)`) |
-| `settings.sqlTable` | Source table name | Table name as string |
+| `settings.sqlTable` | Source table name, `table` or `schema.table` | Table name as string |
 | `settings.direct` | Whether to query directly without creating views | `true`, `false` |
-| `settings.sql` | Custom SQL for this producer | SQL query string |
 | `settings.fileKey` | File key for S3 sources | S3 file path |
 | `settings.fileType` | File format for file-based sources | `JSON`, `JSONL`, `CSV` |
 

package/documentation/default_resources/project.json

@@ -3,10 +3,6 @@
     "name": "Default Remora Project",
     "version": "1.0.0",
     "description": "This is the standard Remora project created when you run init.",
-    "consumers": ["/consumers"],
-    "producers": ["/producers"],
-    "sources": ["/sources"],
-    "schemas": ["/schemas"],
     "settings": {
         "SQL_MAX_QUERY_ROWS": 10000
     }