@forzalabs/remora 1.6.0 → 1.7.0

package/CHANGELOG.md

@@ -6,6 +6,32 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+### Added
+- Added a new producer input type for **fixed-width (positional) flat files**: `"fileType": "FIXED"`. Such a file has no delimiter and no header, so each dimension declares where its value sits in the line with `position` — `{ "start": 21, "length": 10 }` or `{ "start": 21, "end": 30 }`, 1-based and inclusive, the way a layout spec is written ("columns 21-30"). Values are trimmed, so space- and zero-padded fields read as their real value (`"0000123.45"` → `123.45`) and an all-padding field reads as `null`; gaps (unmapped filler columns) and overlaps (a composite field plus its subfields) are both allowed; `format` still applies, so a date written as bare digits parses with `"format": "YYYYMMDD"`; and a `sourceFilename` dimension needs no position. Works everywhere a flat file already does — local and S3 sources, wildcard `fileKey` patterns, compression, chunked parallel reads, `sample`, `mock` and `synth`
+- Added `settings.recordSelector` for flat files that interleave record types in one file (a header, many detail records, a trailer). It reads only the lines whose marker at a given position matches — `{ "position": { "start": 1, "length": 1 }, "equals": "D" }`, or a list of values — so one producer describes exactly one record layout and every other line is dropped before parsing. Applied per line, so it is unaffected by how a file is chunked across workers
+
+### Fixed
+- Fixed a deployed worker producing no logs. The whole run pipeline logged on the debug channel, which is dropped at `prod` level, so a deployment that did not set `REMORA_DEBUG_MODE=true` emitted nothing at all beyond startup lines and errors. The operational lifecycle events of a run are now logged at `INFO`, so they appear without turning debug on: worker-pool startup, consumer launch/completion, producers skipped as optional or empty, export start/finish per destination, downstream (`run-consumer`) triggers and cycle skips, and CRON/queue scheduler setup, per-trigger execution and stats. Per-line, per-chunk and per-worker detail stays on the debug channel, so `REMORA_DEBUG_MODE=true` is still what you set to see it. File logging is unchanged and remains exclusive: when it is enabled these lines go to the log file rather than stdout
+- Fixed JSON/JSONL producer dimensions ignoring their `format` when casting. The flat-file read path passed the dimension's `format` to the type cast but the JSON path did not, so a date written as a formatted string (e.g. `"20211213"` with `"format": "YYYYMMDD"`) failed to parse with `Invalid time value`. Both paths now cast identically
+- Fixed the `sample` command emitting the header row of a CSV/Excel producer as if it were a data row when `hasHeaderRow` was not set explicitly. Deciding where the data starts is now one shared rule for the run path and the sample path, so the two can no longer disagree
+- Fixed a multi-file (wildcard `fileKey`) read of a headerless flat file failing with a "different header line than the other files" error: the first line of each file was compared as a header even for file types that don't have one, where it is data and legitimately differs
+
+## V 1.7.0 - 2026-07-29
+
+### Added
+- Added IAM-role authentication for loading the project configuration from S3. When `REMORA_CONFIG_URI` (or the URI passed to `Environment.loadFromUri`) points at an S3 URL, the config zip is now fetched with the AWS SDK using the ambient IAM role — the ECS task role or instance profile — instead of a plain HTTP GET, so a deployment no longer needs a presigned URL or static access keys. Bucket, key and region are derived from the URL, and the archive is extracted into the local cache path exactly as the HTTP path does (both share `_saveConfigFile`). Non-S3 remote URIs keep using the existing HTTP download with ETag caching
+
+### Changed
+- **Breaking (config)**: grouped every synthetic-data property of a producer dimension under a single `synth` object. `decimals`, `allowedValues`, `codeSet` and `references` move from the dimension's top level into `synth`, e.g. `{ "name": "gap_status", "type": "string", "synth": { "codeSet": "gap_statuses" } }`. These properties only ever affected the `synth` command, so nesting them makes that structural instead of a caveat on each one, keeps the dimension's real-read properties uncluttered, and gives future synth-only options one obvious home. Behaviour is unchanged. A property left at the dimension's top level is now rejected with an error naming the dimension and the property to move (both at `compile` and at the start of `synth`), rather than being silently ignored — and the producer schema rejects unknown keys inside `synth`, so a typo like `codeSets` fails in the editor
+- Replaced the `decompress` package on the archive-reading path. ZIP extraction now uses `adm-zip` and TAR extraction uses `tar`, both behind explicit path-traversal guards: every zip entry is resolved and asserted to stay inside the target directory before anything is written (zip-slip), and `tar.x` strips leading `/` and `..` segments. `decompress` had no patched release for CVE-2026-53486 (arbitrary file read via a crafted archive), so replacement was the only fix. `GZ` handling is unchanged
+- Upgraded the file-parsing and HTTP dependency surface off known-vulnerable versions: `xlsx` → `@e965/xlsx` 0.20.3 (prototype pollution + ReDoS on untrusted Excel), `multer` 2.0.0 (8 DoS CVEs on the Express upload path), `fast-xml-parser` 5.3.5, `adm-zip` 0.6.0, plus root `overrides` pinning `undici`, `form-data` and `path-to-regexp`. Unused packages were dropped from the published CLI and root dependency sets (`next`, `bcrypt`, `geist`, `react-ui`) so they no longer ship or get scanned. See [CVE_REMEDIATION.md](docs/CVE_REMEDIATION.md) for the scan, per-image triage and remaining items
+- Hardened the three deployed Docker images. Backend and worker now install with a workspace-scoped `npm install` followed by `npm prune --omit=dev` in the same layer, so build tooling (tsc, eslint, vite, tsup, nodemon) no longer ships to production; the base image's bundled global `npm` is patched; both drop to the non-root `node` user; and all three declare a `HEALTHCHECK`. `.dockerignore` now excludes `apps/**/testdata` and `apps/**/test`, which alone removed roughly 1 GB of image weight
+- Queue polling failures now include the underlying error text in the log message instead of only the queue URL, so a poll failure is diagnosable from the log line alone
+
+### Fixed
+- Fixed consumer field transformations rejecting a legitimately `null` referenced field. A `"$fieldName"` operand only requires the field to be *present* in the record: a `null` value is real data and is now passed through to the operation, while a field genuinely absent from the record still throws so the field's `onError` policy applies. Previously both cases threw, so a null in a referenced column failed the row
+- Fixed the worker throwing `new error(...)` — a lowercase import from `node:console` — when the licence key is missing or invalid, which raised a `TypeError` instead of the intended "Invalid Remora licence key" message. It now throws a proper `Error`
+
 ## V 1.6.0 - 2026-07-21
 
 ### Added