@forzalabs/remora 1.2.12 → 1.5.0

package/CHANGELOG.md

@@ -6,6 +6,61 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 1.5.0 - 2026-07-15
+
+### Added
+- Added the `synth` command, which runs the full pipeline on generated fake data: `remora synth [consumer] [-r <records>] [-s <seed>] [-p <project>]`. It generates synthetic INPUT for every producer (one JSONL file per producer, keyed by each dimension), then executes one consumer, all consumers of a project (`--project`), or all consumers through the REAL execution path — filters, field transformations, unions, distinct, pivot, measures, dataset validation and destination export — exporting to their real destinations. Generation is fully deterministic: the same `--seed` reproduces the same input and therefore the same output. Only producer-side source parsing and masking are not exercised, since records are injected already-parsed at the producer boundary. `--records` defaults to `100`, `--seed` to `remora`
+- Added code sets: a new top-level, optional project resource that constrains a producer dimension's synthetic values (used only by `synth`). Point `project.json` at a folder with `"codeSets": ["/code_sets"]` and place one JSON file per set. A set is either a **flat** list (`{ "name": "gap_statuses", "values": ["OPEN", "CLOSED", "COMPLIANT"] }`) or a **structured** table of correlated rows (`{ "name": "hedis_measures", "records": [{ "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" }] }`). A dimension binds via `codeSet` — a string names a flat set (one value picked per row), an object `{ "name", "field" }` binds to a field of a structured set so correlated columns land on the same row. A dimension can also inline a one-off list with `allowedValues` (mutually exclusive with `codeSet`). Validated against a new `code-set-schema.json`
+- Added a library of built-in code sets and healthcare data generators for realistic synthetic data. Built-in sets (HEDIS measures, code systems, gap statuses, product types, specialties, genders, indicators, and more) are applied automatically by field name with no configuration, and a user set of the same name overrides its built-in counterpart. Open-ended generators produce format-correct identifiers and clinical values by field name — Medicare Beneficiary Identifier (MBI), NPI, ICD-10, CPT/HCPCS, NDC, HL7 OIDs, and clinical ranges (HbA1c, systolic/diastolic, etc.)
+- Integrated `@faker-js/faker` to back synthetic generation: each field picks a realistic generator from its declared type and name (email, first/last name, city, phone, price, dates, …), seeded per-cell so output stays deterministic and reproducible regardless of worker order
+- Added the `mock` command, `remora mock <producer> <records>`, which generates a mock source data file for a producer directly from its dimensions (distinct from `synth`, which runs the whole pipeline on synthetic input rather than writing a producer source file)
+- Added a `references` foreign-key declaration on producer dimensions for synthetic data generation (`synth`): `{ "name": "patient_id", "type": "string", "references": { "producer": "p_members", "field": "member_id" } }`. Generated values are drawn from the referenced parent field's key domain, so a consumer's joins/unions over the two producers always line up (no orphan foreign keys) — even when the foreign key and parent key have different names. The referenced parent field is generated as a unique key (a warning is logged if it isn't a `pk`). Single-column relationships only; mutually exclusive with `allowedValues`/`codeSet`. Deterministic under a fixed `--seed`. Canary coverage added (`p_canary_ref_parent`/`p_canary_ref_child` + `c_canary_ref_child`, asserting 0 orphan keys through the real pipeline)
+- Added a descriptive `nullable` flag on producer dimensions (`{ "name": "note", "type": "string", "nullable": true }`). It documents whether a field may hold null/missing values for consumers, schema generation and AI mappers, but is metadata only — it does not change runtime behaviour (null/missing source values are already passed through untouched). Enforce presence with a consumer field `required` validation when a hard check is needed
+- Added an optional `description` property on consumer fields (`{ "key": "total", "description": "Order total in cents" }`) for documenting field intent. Descriptive only, no runtime effect
+- Added a `run-consumer` post-processing action for consumer output `onSuccess` / `onError`: `{ "onSuccess": [{ "action": "run-consumer", "consumers": ["c_downstream", "c_notify"] }] }`. Listed consumers run sequentially in declared order as fresh full runs once the output finishes (on the success or failure path). A downstream failure is logged and swallowed, so it never changes the parent consumer's result or stops the rest of the chain. Runtime cycle detection skips a consumer already on the chain, and a depth cap (`MAX_ON_FINISH_DEPTH`) bounds nesting; `Validator` also rejects unknown/empty targets and static cycles at compile time. Downstream runs are attributed to a new `ON_FINISH` invocation origin
+- Added canary coverage for the `run-consumer` action (`c_onsuccess_run` and `c_onerror_run` triggering `c_onfinish_downstream` on the success and failure paths)
+
+### Changed
+- `remora init` is now interactive: it prompts (via `inquirer`) for the project name and description, seeds a starter project, and prints guidance on the required `REMORA_LICENCE_KEY` environment variable
+- Extended the consumer JSON schema `consumerOutputOnFinish` definition with the `run-consumer` action and its `consumers` list
+- The CLI progress renderer now no-ops instead of crashing when stdout is not a TTY (piped output, CI, Docker logs), where cursor-control methods are unavailable
+
+## V 1.4.0 - 2026-07-08
+
+### Added
+- Added support for using another consumer's full output as a consumer input: name a consumer in a consumer's `producers` array and its output is streamed in as a source, e.g. `{ "producers": [{ "name": "c_upstream" }] }`. The upstream consumer is executed through the normal pipeline, its output captured to disk as JSONL, and read back through a synthetic producer, so arbitrarily large outputs are chunked across workers rather than held in memory (unlike `code_lookup`, which stays a bounded in-memory table). Producers are unioned; nested chains work, with depth/cycle detection (`MAX_CONSUMER_INPUT_DEPTH`) and per-run caching so a referenced consumer is materialized at most once per run
+- Added `ConsumerInputResolver` (executors): before a consumer's workers spawn, it detects `producers` entries that reference a consumer, executes each referenced consumer once (capturing its merged output as JSONL under the run's scope), and exposes it as a synthetic JSONL producer. `ExecutorOrchestrator` gained a `captureFormat` capture path (CSV internal output is reprojected to JSONL; JSON output is copied verbatim) and a synthetic-producer branch in `readySourceFiles`
+- Added the `multiplyBy` and `addBy` field transformations, which multiply/sum a field by the values of several other fields: `{ "multiplyBy": { "fields": ["qty", "price"] } }` and `{ "addBy": { "fields": ["base", "bonus"] } }` — distinct from `multiply`/`add`, which take a single literal or `$fieldName` operand
+- Added `TXT` as a consumer output format with a configurable single-character `delimiter` (defaults to a tab), mirroring the producer-side TXT delimiter: `{ "outputs": [{ "format": "TXT", "delimiter": "|" }] }`
+- Added `onSuccess` and `onError` post-processing actions on consumer outputs. The `move-file` action relocates the consumer's static source input file to another Remora source once the run finishes, on both the success and failure paths: `{ "onSuccess": [{ "action": "move-file", "moveToDestination": "s_archive" }] }`
+- Added the `oidc` authentication method for Delta Share sources (OAuth2 client-credentials via `clientId` / `clientSecret` / `tokenUrl`), extending `ISourceAuthentication` and the source JSON schema
+- Added canary coverage for consumer-as-input (`c_ci_base` → `c_ci_derived` → `c_ci_chain`) with a value-level correctness check (`verifyConsumerInputValues`) that re-derives the expected rows from the source, plus a `c_canary_extract` consumer and a `verifyExtractValues` check for the extract transform
+
+### Changed
+- Extended the consumer JSON schema with the `multiplyBy` / `addBy` transforms, the `TXT` output format and its `delimiter`, the `onSuccess` / `onError` output actions, and consumer names as valid `producers[].name` entries
+- Enhanced HTTP API source configuration in the source JSON schema (widened auth/request-method enums and added API source detail/examples)
+- Consumer producer/source validation now accepts consumer inputs: an entry that resolves to a consumer rather than a producer is treated as a sourceless, local (file) input, and source/engine-uniqueness checks apply only to real producers
+- Internal cleanup: removed a swath of unused code to reduce surface (`EventsEngine`, `JWTManager.verifyCLI`, `LocalSourceDriver.moveFile`, `DatabaseEngine.disconnect`, unused `UserManager` methods, and dead SQL-library / usage-stat interfaces)
+
+### Removed
+- Removed the deprecated `aws-dynamodb` and `postgres` source engines from `DataEngineType` and the source JSON schema
+
+### Fixed
+- Fixed the `extract` transformation (`year` / `month` / `day` / `hour` / `minute`) throwing `date.getTime is not a function` for every input: `TypeCaster.cast(value, 'date')` returns a normalized ISO string rather than a `Date`, so component getters failed. It now rebuilds a `Date` from that ISO string and reads UTC components, making extraction deterministic regardless of host timezone
+- Fixed a consumer-as-input cache-staleness bug: a capture path cached from a nested run's scope (since cleared) could be reused by a later top-level run, causing a "data file is missing" failure. `ConsumerInputResolver` now resets its cache at each top-level run and re-materializes if a cached capture no longer exists on disk
+
+## V 1.3.0 - 2026-07-07
+
+### Added
+- Added the `bit_extract` field transformation to extract a single bit from an integer bitmask (e.g. decoding a NAPS LOB byte): `{ "bit_extract": { "position": 0, "as_boolean": false } }`. `position` is a 0-based index from the least-significant bit (bounded to `[0, 52]`); returns `1`/`0`, or `true`/`false` when `as_boolean` is set. Uses arithmetic rather than JS bitwise operators so masks wider than 31 bits stay correct
+- Added the `date_format` field transformation to reformat a date string from an input format to an output format (dayjs tokens), e.g. `{ "date_format": { "input_fmt": "DDMMYYYY", "output_fmt": "YYYY-MM-DD" } }` — distinct from `cast`, which only parses or formats with a single token set. Backed by a new `TypeCaster.reformatDate` helper
+- Added the `code_lookup` field transformation that maps a code to a value using the output of another consumer as the lookup table: `{ "code_lookup": { "consumer": "c_lookup_plans", "key": "plan_code", "value": "plan_tier", "default": "Unknown Tier" } }`. When the code is not found and no `default` is set, the field's `onError` policy applies
+- Added `LookupResolver` (executors): before a consumer's workers spawn, it detects `code_lookup` references, executes each referenced consumer through the normal pipeline, captures its output, projects it to a bounded `code -> value` table written under the run's scope, and hydrates it into each worker via a new in-memory `LookupRegistry`. Includes cycle/depth detection (`MAX_LOOKUP_DEPTH`), an in-memory row bound (`MAX_LOOKUP_ROWS`), and per-run caching so a referenced consumer is executed at most once
+- Added canary coverage for the new transformations: `c_canary_new_transforms` (exercising `bit_extract`, `date_format`, and `code_lookup`) driven off a `c_lookup_plans` lookup consumer and a `p_lookup_plans` producer
+
+### Changed
+- Extended the consumer JSON schema `singleTransformation` definition with `bit_extract`, `date_format`, and `code_lookup` entries
+
 ## V 1.2.12 - 2026-07-02
 
 ### Added

package/documentation/README.md

@@ -26,8 +26,9 @@ The main project configuration file.
 | `sources` | Path to source definitions | Directory path as string |
 | `schemas` | Path to schema definitions | Directory path as string |
 | `settings.SQL_MAX_QUERY_ROWS` | Maximum rows in query results | Positive integer |
-| `settings.DEFAULT_TIMEOUT_SECONDS` | Maximum query execution time | Positive integer (seconds) |
-| `settings.LOG_LEVEL` | Logging detail level | `DEBUG`, `INFO`, `WARNING`, `ERROR` |
+| `settings.STRING_MAX_CHARACTERS_LENGTH` | Maximum length for string fields | Positive integer |
+| `settings.MAX_ITEMS_IN_MEMORY` | Maximum number of items to keep in memory | Positive integer |
+| `settings.API_QUEUE.source` | Name of an `aws-sqs` source used as an alternative execute-consumer API | Must match a source `name` |
 
 ## Source Configuration
 

package/documentation/default_resources/project.json

@@ -8,8 +8,6 @@
     "sources": ["/sources"],
     "schemas": ["/schemas"],
     "settings": {
-        "SQL_MAX_QUERY_ROWS": 10000,
-        "DEFAULT_TIMEOUT_SECONDS": 300,
-        "LOG_LEVEL": "INFO"
+        "SQL_MAX_QUERY_ROWS": 10000
     }
 }