@forzalabs/remora 2.4.0 → 2.5.0

package/CHANGELOG.md

@@ -6,6 +6,53 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.5.0 - 2026-10-26
+
+**Upgrading from 2.4.** Nothing in an existing project changes. `FIXED` is a new output format and every rule this release adds fires only on a consumer that declares one, so no configuration needs touching, no exported bytes move, and nothing that compiled under 2.4 is refused. The one thing to know before writing a FIXED output is what the receiver has to do with the file: it leaves Remora as UTF-8 text with `lf` or `crlf` line terminators, so a z/OS target still needs the ASCII→EBCDIC and `RECFM=FB` conversion on its own side (see Added).
+
+### Added
+
+- **`"format": "FIXED"`, a positional flat-file output.** Remora has read fixed-width files since 1.8 and could not write one — every output format it had quotes and delimits — so a customer who sends a positional feed and wants one back had no way out of the pipeline. A `FIXED` output declares a `width` per field and writes the record at those columns: no delimiter, no quotes, no header. It reaches the same destinations the other file formats do (local, S3, a pre-signed URL) with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour, because it is a per-line transform at export time and not a new write path — the dataset is staged exactly as a `TXT` one is and re-emitted positionally on its way out, so nothing in the run pipeline changed to carry it.
+
+- **Each visible field gains a `position` block** — the write-side counterpart of the producer dimension's `position`, named the same for symmetry. `width` (required) is how many characters the field occupies; `align` (`left`/`right`), `pad` and `nullFill` are the single characters it is filled out with; `overflow` is what a value too long for its field does. **Column order is field order and the offsets are computed from the widths**, so no offset is ever transcribed by hand — a `start` may be declared, but it is an assertion `compile` recomputes rather than a second source of truth, and a gap or a COBOL FILLER needs no new concept because a `fixed` field with a `width` already is one.
+
+  ```json
+  {
+      "fields": [
+          { "key": "key_serv_plan", "position": { "width": 3, "start": 1 } },
+          { "key": "key_tran_id", "position": { "width": 4, "start": 4 } },
+          { "key": "balance", "position": { "width": 11, "align": "right", "pad": "0" } }
+      ],
+      "outputs": [{
+          "format": "FIXED",
+          "exportDestination": "s_output",
+          "recordLength": 44,
+          "lineEnding": "crlf",
+          "defaults": { "number": { "align": "right", "pad": "0" } }
+      }]
+  }
+  ```
+
+- **Five fill rules, each chosen so a wrong file fails loudly instead of shipping.** A value shorter than its `width` is padded on the side `align` names. **A value longer than its width fails the run**, naming the field, the value and both widths — a truncated account number is a plausible-looking account number, and an `overflow: "truncate"` escape exists per field for the layouts that genuinely specify a cut (the end that survives follows `align`, which is what a COBOL `MOVE` does). A null is the full width in `nullFill`, **which defaults to spaces rather than to `pad`**: zero-filling an empty numeric column writes `00`, and the receiver reads that as the number zero where there was no value at all. An empty string is written as a null, because the staged encoding keeps the two apart and a positional line cannot. And every record is exactly `recordLength`, a record whose last field is null included — the line is never cut short to the last character that carries something.
+
+- **`recordLength`, `defaults` and `lineEnding` on the output.** `recordLength` is the length of the record as written, which is not always the length in the copybook — a NASCO POD 1300 is 44 characters of a 256-column layout, cut after the last field that carries anything — and `compile` fails if the widths do not sum to it. `defaults` sets `align`/`pad`/`nullFill`/`overflow` once per dimension type (`string`, `number`, `datetime`, `boolean`) so a 90-field layout states "numbers are right-aligned and zero-padded" once and names only the exceptions, with a field's own `position` winning where both set the same rule. `lineEnding` is `lf` (the default) or `crlf`, because a real positional feed picks one terminator and holds it on every line including the last.
+
+- **`layouts` — several record layouts in one file.** A positional feed is one file even when it carries three kinds of record, and shipping one file per record type is not the same file. The output names a `discriminator` field, lists its values in `order`, and gives each one a field list and a `recordLength` of its own; the consumer declares the union of every layout's fields and each layout names the ones its records carry. This is the write-side mirror of the producer's `recordSelector`. **The records come out grouped, one contiguous block per value in `order`, keeping the order they were read in within each block** — which is how a real feed is written. The grouping is a two-pass reorder over the staged file (route each line to its block, then append the blocks in order), so it costs one extra copy of the dataset on disk rather than one per block, and `CONSTANTS.defaults.MAX_FIXED_LAYOUTS` (64) bounds how many blocks one output may open at once.
+
+- **Four kinds of layout error are caught before a producer is read, and `remora compile` prints the layout it resolved.** A missing `width`, a `start` that disagrees with the computed offset, widths that do not sum to `recordLength`, a layout field or discriminator that is not one of the consumer's visible fields — all of them are configuration, so all of them are resolved by the same code the export runs and reported together, every problem in the output named in one `compile` rather than one per run. The same check runs again at the start of a run, so a width that does not add up fails before the first producer is read instead of after the whole dataset has been built. `compile` then prints each layout as a table — field, start, end, width, in the order the record is written — which is the whole point of declaring widths instead of columns: the author reads the start columns back and checks them against the spec once, rather than counting characters across 90 fields to find the filler they forgot. `run` and `verify` compile too and stay quiet, since a 90-field table is noise in front of a run.
+
+- **A FIXED output refuses what it has no room for, rather than ignoring it.** `header` is rejected — a field name does not sit on the columns it describes, and the receiver reads by position anyway. `delimiter` is rejected, because a positional line has none. The wildcard field key `"*"` is rejected, since every field of a positional line declares its own width and a wildcard expands to columns nobody has written one for. `omitNull` stays rejected as on every positional format. And the four FIXED-only output properties plus a field `position` are all rejected when the format is something else, the way a `position` on a non-FIXED producer is: silently dropping a rule the author wrote is how a file ships looking almost right.
+
+- **The bar is a byte-identical round trip, and it is a canary stage.** A three-layout positional file is read by three producers, exported as JSON, read back and written out again as one `FIXED` file, and the stage fails on any difference at all — a width the writer gets wrong is a width the reader also got wrong, and only a byte comparison catches the pair. It runs over a committed fixture and over any real batch `POD_SAMPLE_DIR` names.
+
+- **What the receiver has to do is documented rather than assumed.** The file leaves Remora as UTF-8 text, one record per line, one character per column. A z/OS target needs ASCII→EBCDIC and reframing to `RECFM=FB` on the customer's side — the mirror of the conversion job a mainframe feed already requires inbound, and the same shops run it in that direction — and a non-ASCII character leaves the record `recordLength` characters but the line no longer `recordLength` bytes. Implied decimals and overpunch signs are not written: POD-class layouts are `PIC X` and unsigned `9` throughout, and a feed that needs scale or sign needs the read-side work first. [Remora - Consumers.md](docs/public_docs/Remora%20-%20Consumers.md) carries all of it under **FIXED Output**.
+
+- **The full prose reference now ships with the CLI.** `docs/public_docs` is copied into the published package, so the four reference documents (Sources, Producers, Consumers, Code Sets) sit on disk beside the installed version — a user, or an AI assistant working in their project, reads the reference for the release they actually have rather than fetching a copy that may describe a different one.
+
+### Changed
+
+- **The README `remora init` drops into a new project was rewritten.** It used to mirror the JSON schemas key by key, which rotted: the schemas moved and the tables did not. It is now an orientation document — how the three resource types chain together, what `init` just created, the commands, the environment variables, and where the full reference lives — with enum lists but no key-by-key tables, so the inline contract has exactly one home. It also has a section written for an AI assistant working in the project folder.
+
 ## V 2.4.0 - 2026-10-19
 
 **Upgrading from 2.3.** Two configurations that compiled under 2.3 are now refused, and both fail at `compile` with the offending resource named rather than at run time. A producer carrying `measures` must have them deleted — the key is gone from the schema and a producer rejects unknown keys (nothing is lost: no stage of the run path ever evaluated one, see Removed). A consumer declaring **more than one** `outputs` entry must keep the one it actually used — a second entry was never exported, so the one to keep is the first (see Changed). Two further changes alter the bytes an existing project produces, neither of them opt-out: a `CSV`/`TXT` export now writes a null as an unquoted empty field (`"USR-2",,`) where it used to write `""`, and a value containing a newline is no longer split across two records. A destination that reads `''` where a null is now written, or that relied on the old split rows, needs checking before this version runs.

package/docs/Remora - Code Sets.md

@@ -0,0 +1,268 @@
+# Remora - Code Sets
+
+## **Overview**
+
+Code sets are named, reusable lists of allowed values used to drive **synthetic data generation** (the `synth` command). They let you produce realistic, domain-accurate fake data by telling Remora which values a field may take — and, for related fields, which values belong together on the same row.
+
+Code sets affect **only** synthetic generation. They have no effect on normal producer reads, consumer transformations, or any other part of the pipeline.
+
+There are three ways to constrain a producer dimension's synthetic values:
+
+- **Inline** (`synth.allowedValues`) — a quick, one-off list on a single dimension.
+- **Named flat code set** (`synth.codeSet: "name"`) — a reusable list, defined once and referenced from many dimensions.
+- **Named structured code set** (`synth.codeSet: { "name": ..., "field": ... }`) — a table of correlated rows, so related fields (e.g. a code and its description) stay coherent.
+
+All of them are declared under the dimension's `synth` object, which holds every synthetic-only property.
+
+Remora also ships a library of **built-in code sets** (HEDIS measures, code systems, gap statuses, and more) that are applied automatically by field name, with no configuration required.
+
+---
+
+## **How Values Are Chosen**
+
+For each dimension, synthetic generation resolves a value in this order of precedence:
+
+1. Explicit `synth.references` on the dimension (a foreign key — see below)
+2. Explicit `synth.allowedValues` on the dimension
+3. Explicit `synth.codeSet` on the dimension
+4. A **built-in** code set matched by the field's name
+5. A name-based generator (email, city, dates, identifiers, etc.)
+
+`references`, `allowedValues` and `codeSet` are mutually exclusive within a dimension's `synth` object, and any of them wins over a built-in match, so you can override the defaults per field. Picks are uniform (each value equally likely).
+
+---
+
+## **Referential Integrity (Foreign Keys)**
+
+Code sets keep *categorical* fields coherent. To keep *keys* coherent across producers — so a consumer's joins and unions actually line up — declare a foreign key with `synth.references`:
+
+```json
+{ "name": "patient_id", "type": "string",
+  "synth": { "references": { "producer": "p_members", "field": "member_id" } } }
+```
+
+Every synthetic value generated for `patient_id` is drawn from the values `p_members.member_id` emits, so the foreign key is always valid — there are no orphan rows. The relationship works even when the foreign key and the parent key have different names (`patient_id` → `member_id`).
+
+- `producer` / `field` name the parent producer and the parent dimension (by its `name`).
+- The referenced parent field is generated as a **unique key** (like a primary key) so children always find a match. If it isn't marked `pk`, `synth` logs a warning and proceeds.
+- Only single-column relationships are supported. Composite (multi-column) keys are not yet correlated: two `synth.references` fields pointing at the same parent are picked independently, so their pair may not match a single parent row.
+- Generation stays deterministic — the same `--seed` reproduces the same keys and the same joined output.
+
+---
+
+## **Defining Code Sets**
+
+Code sets are a top-level project resource, like sources, producers and consumers. They are **optional** — omit them entirely and the built-in sets still apply.
+
+### **Project Setup**
+
+Create a `code_sets` folder next to `project.json` and place one JSON file per code set inside it. There is nothing to declare: the folder is picked up automatically when it is there, and ignored when it is not.
+
+```
+remora/
+├── project.json
+├── code_sets/
+│   ├── gap_statuses.json
+│   └── hedis_measures.json
+├── producers/
+└── ...
+```
+
+### **Flat Code Sets**
+
+A flat code set is a simple list of values. One value is picked per row.
+
+```json
+{
+  "$schema": "https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/code-set-schema.json",
+  "name": "gap_statuses",
+  "description": "Care-gap compliance statuses.",
+  "values": ["OPEN", "CLOSED", "COMPLIANT", "NON-COMPLIANT", "EXCLUDED", "PENDING"]
+}
+```
+
+Values may be strings, numbers, or booleans:
+
+```json
+{
+  "name": "days_supply",
+  "values": [30, 60, 90]
+}
+```
+
+### **Structured (Correlated) Code Sets**
+
+A structured code set is a table of records. Dimensions bind to a **field** of the set, and every dimension bound to the same set lands on the **same row per record** — so related values stay coherent.
+
+```json
+{
+  "$schema": "https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/code-set-schema.json",
+  "name": "hedis_measures",
+  "description": "Correlated HEDIS measure code, name and coding system.",
+  "records": [
+    { "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" },
+    { "code": "COL", "name": "Colorectal Cancer Screening", "system": "CPT" },
+    { "code": "BCS", "name": "Breast Cancer Screening", "system": "CPT" }
+  ]
+}
+```
+
+> A code set must have **either** `values` **or** `records`, never both.
+
+---
+
+## **Referencing a Code Set from a Producer**
+
+You reference a code set from a producer's dimension's `synth` object (see [Remora - Producers](Remora%20-%20Producers.md)). `allowedValues` and `codeSet` are mutually exclusive on the same dimension.
+
+### **Inline values**
+
+```json
+{
+  "name": "gender_code",
+  "type": "string",
+  "synth": { "allowedValues": ["M", "F", "U"] }
+}
+```
+
+### **Named flat set**
+
+Reference a flat code set by name (a string):
+
+```json
+{
+  "name": "gap_status",
+  "type": "string",
+  "synth": { "codeSet": "gap_statuses" }
+}
+```
+
+### **Named structured set**
+
+Reference a structured code set with `{ "name", "field" }`. Point multiple dimensions at the same set to keep them correlated:
+
+```json
+{
+  "name": "measure_code",
+  "type": "string",
+  "synth": { "codeSet": { "name": "hedis_measures", "field": "code" } }
+},
+{
+  "name": "measure_name",
+  "type": "string",
+  "synth": { "codeSet": { "name": "hedis_measures", "field": "name" } }
+}
+```
+
+With the set above, every generated row pairs a code with its correct name — e.g. `CBP` always appears alongside `Controlling High Blood Pressure`, never a mismatched description.
+
+---
+
+## **Built-in Code Sets**
+
+Remora ships built-in healthcare/payer-quality code sets that are applied **automatically** when a dimension's name matches — no `synth.codeSet` needed. This is why, out of the box, a field named `measure_code` produces real HEDIS codes and `gap_status` produces valid statuses.
+
+| Built-in set | Kind | Applied to fields whose name contains |
+| --- | --- | --- |
+| `hedis_measures` | structured (`code`, `name`, `system`) | `measure_code`, `sub_measure_code`, `measure_classification_code` → `code`; `measure_name`, `measure_description` → `name` |
+| `code_systems` | flat | `code_system` |
+| `gap_statuses` | flat | `gap_code`, `gap_status`, `compliance` |
+| `product_types` | flat | `product_type` |
+| `specialties` | flat | `specialty` |
+| `clinical_events` | flat | `event_name`, `event_category` |
+| `run_types` | flat | `run_type` |
+| `genders` | flat | `gender_code`, `gender`, `sex` |
+| `yn_indicators` | flat | `indicator` |
+| `days_supply` | flat | `days_supply` |
+
+Because `measure_code` and `measure_name` both bind to the built-in `hedis_measures` set, they are automatically correlated on each row — with no configuration.
+
+A built-in set is only applied when it can produce the dimension's declared `type`. A numeric surrogate key such as `heb_quality_gaps_measure_code_sk` matches `measure_code` by name, but HEDIS codes are not numbers, so the binding is skipped and the field is generated from its own type instead. An **explicit** `synth.codeSet` or `synth.allowedValues` that cannot produce the declared type is a configuration error and fails the run by name.
+
+### **Overriding a built-in set**
+
+Define a code set with the **same name** as a built-in one to override it project-wide:
+
+```json
+{
+  "name": "gap_statuses",
+  "values": ["OPEN", "CLOSED"]
+}
+```
+
+Now any field that maps to `gap_statuses` (whether by built-in name matching or an explicit `synth.codeSet`) draws only from your values.
+
+---
+
+## **Determinism**
+
+Synthetic generation is fully deterministic: the same `--seed` always produces the same dataset, including code-set picks and correlated rows. Change the seed to get a different-but-equally-valid dataset. This makes generated data reproducible across runs and machines.
+
+---
+
+## **Code Set Schema Reference**
+
+| Property | Required | Description | Values |
+| --- | --- | --- | --- |
+| `name` | ✅ | Unique code set name, referenced from a dimension's `synth.codeSet` | Any valid identifier |
+| `description` | ❌ | Human-readable description | Descriptive text |
+| `values` | one of `values`/`records` | Flat list; one value picked per row | Array of string / number / boolean |
+| `records` | one of `values`/`records` | Correlated rows; dimensions bind to a field | Array of objects (string / number / boolean fields) |
+
+---
+
+## **End-to-End Example**
+
+`remora/code_sets/hedis_measures.json`:
+
+```json
+{
+  "name": "hedis_measures",
+  "records": [
+    { "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" },
+    { "code": "COL", "name": "Colorectal Cancer Screening", "system": "CPT" }
+  ]
+}
+```
+
+`remora/code_sets/gap_statuses.json`:
+
+```json
+{
+  "name": "gap_statuses",
+  "values": ["OPEN", "CLOSED", "COMPLIANT", "EXCLUDED"]
+}
+```
+
+`remora/producers/p_quality_gaps.json` (excerpt):
+
+```json
+{
+  "name": "p_quality_gaps",
+  "source": "s_local",
+  "dimensions": [
+    { "name": "member_id", "type": "string", "pk": true },
+    { "name": "measure_code", "type": "string", "synth": { "codeSet": { "name": "hedis_measures", "field": "code" } } },
+    { "name": "measure_name", "type": "string", "synth": { "codeSet": { "name": "hedis_measures", "field": "name" } } },
+    { "name": "gap_status",   "type": "string", "synth": { "codeSet": "gap_statuses" } },
+    { "name": "gender_code",  "type": "string", "synth": { "allowedValues": ["M", "F", "U"] } }
+  ],
+  "settings": { "sqlTable": "quality_gaps_vw" },
+  "_version": 1
+}
+```
+
+Generate synthetic data through a consumer that uses this producer:
+
+```bash
+# Run one consumer (100 rows, default seed)
+npx @forzalabs/remora synth c_quality_gaps
+
+# Control row count and seed
+npx @forzalabs/remora synth c_quality_gaps --records 500 --seed my-seed
+
+# Run every consumer tagged with a given project
+npx @forzalabs/remora synth --project my-project
+```
+
+Each generated row will pair `measure_code` with its matching `measure_name`, draw `gap_status` from your flat set, and pick `gender_code` from the inline list.