@forzalabs/remora 2.3.0 → 2.5.0

package/CHANGELOG.md

@@ -6,6 +6,99 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.5.0 - 2026-10-26
+
+**Upgrading from 2.4.** Nothing in an existing project changes. `FIXED` is a new output format and every rule this release adds fires only on a consumer that declares one, so no configuration needs touching, no exported bytes move, and nothing that compiled under 2.4 is refused. The one thing to know before writing a FIXED output is what the receiver has to do with the file: it leaves Remora as UTF-8 text with `lf` or `crlf` line terminators, so a z/OS target still needs the ASCII→EBCDIC and `RECFM=FB` conversion on its own side (see Added).
+
+### Added
+
+- **`"format": "FIXED"`, a positional flat-file output.** Remora has read fixed-width files since 1.8 and could not write one — every output format it had quotes and delimits — so a customer who sends a positional feed and wants one back had no way out of the pipeline. A `FIXED` output declares a `width` per field and writes the record at those columns: no delimiter, no quotes, no header. It reaches the same destinations the other file formats do (local, S3, a pre-signed URL) with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour, because it is a per-line transform at export time and not a new write path — the dataset is staged exactly as a `TXT` one is and re-emitted positionally on its way out, so nothing in the run pipeline changed to carry it.
+
+- **Each visible field gains a `position` block** — the write-side counterpart of the producer dimension's `position`, named the same for symmetry. `width` (required) is how many characters the field occupies; `align` (`left`/`right`), `pad` and `nullFill` are the single characters it is filled out with; `overflow` is what a value too long for its field does. **Column order is field order and the offsets are computed from the widths**, so no offset is ever transcribed by hand — a `start` may be declared, but it is an assertion `compile` recomputes rather than a second source of truth, and a gap or a COBOL FILLER needs no new concept because a `fixed` field with a `width` already is one.
+
+  ```json
+  {
+      "fields": [
+          { "key": "key_serv_plan", "position": { "width": 3, "start": 1 } },
+          { "key": "key_tran_id", "position": { "width": 4, "start": 4 } },
+          { "key": "balance", "position": { "width": 11, "align": "right", "pad": "0" } }
+      ],
+      "outputs": [{
+          "format": "FIXED",
+          "exportDestination": "s_output",
+          "recordLength": 44,
+          "lineEnding": "crlf",
+          "defaults": { "number": { "align": "right", "pad": "0" } }
+      }]
+  }
+  ```
+
+- **Five fill rules, each chosen so a wrong file fails loudly instead of shipping.** A value shorter than its `width` is padded on the side `align` names. **A value longer than its width fails the run**, naming the field, the value and both widths — a truncated account number is a plausible-looking account number, and an `overflow: "truncate"` escape exists per field for the layouts that genuinely specify a cut (the end that survives follows `align`, which is what a COBOL `MOVE` does). A null is the full width in `nullFill`, **which defaults to spaces rather than to `pad`**: zero-filling an empty numeric column writes `00`, and the receiver reads that as the number zero where there was no value at all. An empty string is written as a null, because the staged encoding keeps the two apart and a positional line cannot. And every record is exactly `recordLength`, a record whose last field is null included — the line is never cut short to the last character that carries something.
+
+- **`recordLength`, `defaults` and `lineEnding` on the output.** `recordLength` is the length of the record as written, which is not always the length in the copybook — a NASCO POD 1300 is 44 characters of a 256-column layout, cut after the last field that carries anything — and `compile` fails if the widths do not sum to it. `defaults` sets `align`/`pad`/`nullFill`/`overflow` once per dimension type (`string`, `number`, `datetime`, `boolean`) so a 90-field layout states "numbers are right-aligned and zero-padded" once and names only the exceptions, with a field's own `position` winning where both set the same rule. `lineEnding` is `lf` (the default) or `crlf`, because a real positional feed picks one terminator and holds it on every line including the last.
+
+- **`layouts` — several record layouts in one file.** A positional feed is one file even when it carries three kinds of record, and shipping one file per record type is not the same file. The output names a `discriminator` field, lists its values in `order`, and gives each one a field list and a `recordLength` of its own; the consumer declares the union of every layout's fields and each layout names the ones its records carry. This is the write-side mirror of the producer's `recordSelector`. **The records come out grouped, one contiguous block per value in `order`, keeping the order they were read in within each block** — which is how a real feed is written. The grouping is a two-pass reorder over the staged file (route each line to its block, then append the blocks in order), so it costs one extra copy of the dataset on disk rather than one per block, and `CONSTANTS.defaults.MAX_FIXED_LAYOUTS` (64) bounds how many blocks one output may open at once.
+
+- **Four kinds of layout error are caught before a producer is read, and `remora compile` prints the layout it resolved.** A missing `width`, a `start` that disagrees with the computed offset, widths that do not sum to `recordLength`, a layout field or discriminator that is not one of the consumer's visible fields — all of them are configuration, so all of them are resolved by the same code the export runs and reported together, every problem in the output named in one `compile` rather than one per run. The same check runs again at the start of a run, so a width that does not add up fails before the first producer is read instead of after the whole dataset has been built. `compile` then prints each layout as a table — field, start, end, width, in the order the record is written — which is the whole point of declaring widths instead of columns: the author reads the start columns back and checks them against the spec once, rather than counting characters across 90 fields to find the filler they forgot. `run` and `verify` compile too and stay quiet, since a 90-field table is noise in front of a run.
+
+- **A FIXED output refuses what it has no room for, rather than ignoring it.** `header` is rejected — a field name does not sit on the columns it describes, and the receiver reads by position anyway. `delimiter` is rejected, because a positional line has none. The wildcard field key `"*"` is rejected, since every field of a positional line declares its own width and a wildcard expands to columns nobody has written one for. `omitNull` stays rejected as on every positional format. And the four FIXED-only output properties plus a field `position` are all rejected when the format is something else, the way a `position` on a non-FIXED producer is: silently dropping a rule the author wrote is how a file ships looking almost right.
+
+- **The bar is a byte-identical round trip, and it is a canary stage.** A three-layout positional file is read by three producers, exported as JSON, read back and written out again as one `FIXED` file, and the stage fails on any difference at all — a width the writer gets wrong is a width the reader also got wrong, and only a byte comparison catches the pair. It runs over a committed fixture and over any real batch `POD_SAMPLE_DIR` names.
+
+- **What the receiver has to do is documented rather than assumed.** The file leaves Remora as UTF-8 text, one record per line, one character per column. A z/OS target needs ASCII→EBCDIC and reframing to `RECFM=FB` on the customer's side — the mirror of the conversion job a mainframe feed already requires inbound, and the same shops run it in that direction — and a non-ASCII character leaves the record `recordLength` characters but the line no longer `recordLength` bytes. Implied decimals and overpunch signs are not written: POD-class layouts are `PIC X` and unsigned `9` throughout, and a feed that needs scale or sign needs the read-side work first. [Remora - Consumers.md](docs/public_docs/Remora%20-%20Consumers.md) carries all of it under **FIXED Output**.
+
+- **The full prose reference now ships with the CLI.** `docs/public_docs` is copied into the published package, so the four reference documents (Sources, Producers, Consumers, Code Sets) sit on disk beside the installed version — a user, or an AI assistant working in their project, reads the reference for the release they actually have rather than fetching a copy that may describe a different one.
+
+### Changed
+
+- **The README `remora init` drops into a new project was rewritten.** It used to mirror the JSON schemas key by key, which rotted: the schemas moved and the tables did not. It is now an orientation document — how the three resource types chain together, what `init` just created, the commands, the environment variables, and where the full reference lives — with enum lists but no key-by-key tables, so the inline contract has exactly one home. It also has a section written for an AI assistant working in the project folder.
+
+## V 2.4.0 - 2026-10-19
+
+**Upgrading from 2.3.** Two configurations that compiled under 2.3 are now refused, and both fail at `compile` with the offending resource named rather than at run time. A producer carrying `measures` must have them deleted — the key is gone from the schema and a producer rejects unknown keys (nothing is lost: no stage of the run path ever evaluated one, see Removed). A consumer declaring **more than one** `outputs` entry must keep the one it actually used — a second entry was never exported, so the one to keep is the first (see Changed). Two further changes alter the bytes an existing project produces, neither of them opt-out: a `CSV`/`TXT` export now writes a null as an unquoted empty field (`"USR-2",,`) where it used to write `""`, and a value containing a newline is no longer split across two records. A destination that reads `''` where a null is now written, or that relied on the old split rows, needs checking before this version runs.
+
+### Added
+
+- **`"format": "PARQUET"`, a fourth file output.** It writes an Apache Parquet file — one typed column per visible field, nulls preserved, SNAPPY compressed — to any file destination the other formats reach (local, S3, a pre-signed URL), with the same naming, `split`, `trigger` and `onSuccess`/`onError` behaviour as every other file output. Parquet is what a lake expects to read, and it gives a reader two things a `CSV` export cannot: the file **declares its types**, so nothing downstream guesses them back out of the text, and it is a fraction of the size — a four-column, 2-million-row dataset measures 28 MB as parquet against 115 MB as quoted CSV. The schema is taken from what the consumer already declares, nothing new to configure: `string` -> `STRING`, `number` -> `DOUBLE` (every value remora holds is a double, and an `INT64` would promise a reader precision the record never had), `boolean` -> `BOOLEAN`, `datetime` -> `TIMESTAMP`, and a field with no dimension behind it — a `fixed`, `lineCount` or `copyFrom` field, or a column a pattern key matched — is a string, because there is no declared type to read. Every column is nullable, so **a null stays a null** instead of becoming `0`, `false` or `""`: in a column store those are four different things. A `hidden` field is absent from the file entirely, not an empty column in it. An unparseable value under a `datetime` dimension **fails the run**, naming the field and the value, rather than writing a timestamp column quietly full of 1970.
+
+- Parquet is the one output that is **not** a line-for-line pass. It is a columnar container whose footer can only be written once the last row group is encoded, so there is no per-line mapping to hand a destination driver: the file is built in the run's scope folder and then handed over as bytes with no header, through the `move` that is already byte-faithful in all three destination drivers. **No driver method was added and nothing in the driver layer changed.** The writer is `hyparquet-writer`, the companion of the `hyparquet` reader the Delta Share driver already depends on — same author, MIT, one dependency, and reached by dynamic `import()` exactly as that driver reaches the reader. Rows are pulled from an async generator one row group at a time, so **peak memory is bounded by a row group, not by the dataset**, which is the property the whole pipeline is built on; row groups are sized from a budget of *cells* rather than a row count, since 100,000 rows is 400k values on a four-column consumer and 130 million on a 1,300-field one.
+
+- Two flags behave differently on it, both rejected at compile rather than ignored. `header` is **rejected**: a parquet file carries its column names in its own schema, and a header line would be a first row of data claiming to be one. `omitNull` stays rejected as on every positional format — a parquet row group declares every column up front, and a null is written as a null in it. `delimiter` means nothing to it and is ignored, as it is for every format but `TXT`. One caveat is documented rather than worked around: on a split parquet output, **`split.maxFileSizeGb` caps the internal staged bytes, not the parquet bytes**, so the exported files land comfortably under the cap rather than at it — the cap exists to bound what a destination is asked to take in one object, and rotating a file after it was compressed would mean rewriting it. A `PARQUET` output is declared alone on its consumer, as every output now is (see Changed).
+
+### Changed
+
+- **A `CSV` or `TXT` export now distinguishes a null from an empty string.** Every value in those files is quoted, which left the unquoted empty field free to mean something — and it now means **null**, while `""` means the empty string. The two have been the identical two bytes for as long as the format existed, so a column that was genuinely empty and one that held no value at all arrived at the destination indistinguishable, and nothing downstream could get the difference back. **This changes the bytes of every existing CSV/TXT export that carries a null**: where a row used to read `"USR-2","",""` it now reads `"USR-2",,`. It is the shape most readers already assume — `COPY`, `LOAD DATA` and a pandas `read_csv` all treat an unquoted empty field as a null and a quoted one as the empty string — but a destination that reads the two the same way will now see a null where it saw `''`. A field a dimension declares `nullable: false` is unaffected: it carries its type's own empty value (`"0"`, `"false"`, `""`) as it always did, which is a value and so is quoted
+
+- **A value holding a newline no longer becomes two records.** A line of a remora dataset *is* a record — every pass reads it with a line reader and the record limit counts rows by counting newline bytes — so a value carrying a `\n` used to split the row in two, silently, and every stage after it was reading garbage: the wrong column count, a `distinct` that compared halves of rows, a split that bucketed on whatever landed under the key. Such a value is now written as the two characters `\n` (and a carriage return as `\r`) and the record stays whole. The escape is deliberately **one-way** — nothing turns it back into a newline — because a reversible escape would have to escape the backslash too, and that would rewrite every ordinary `C:\path` in the file. A consumer that must carry an embedded newline verbatim has `JSON` and `PARQUET`, whose encodings hold one natively
+
+- **A consumer declares exactly one output.** `outputs` stays an array — nothing in a config file moves, and lineage, the bug report and `ui-preview` keep iterating it — but a second entry is now refused by the schema (`/outputs must NOT have more than 1 items`) and named by the validator. This is a **cap on something that never worked**, not a feature withdrawn: a run stages its records in ONE encoding, chosen from `outputs[0]`, and then *moves* that staged file to the destination, which unlinks it; the export loop also carried its `return` inside the loop body. So a consumer declaring two outputs has always written the first and silently dropped the rest, whatever the second one said. Failing at `compile` tells its author something they need to know rather than taking a working feature away, and so it rejects from this release with no deprecation window. **Nothing in this repository is affected** — of the 140 consumer configs across the canary, the stress and parallel projects, the demos and the apps' own projects, every single one declares exactly one output. To write the same rows twice, declare a second consumer over the same producers. What the cap removed with it: the duplicate-format check, the half of the `omitNull` validation that existed only because two outputs shared one serialized record (`omitNull` on a `JSON` output is no longer rejected for having a `CSV` sibling — there are no siblings), and the "which of the outputs decides the internal format" question, which is now correct by construction
+
+### Fixed
+
+- **A `hidden` field shifted every column after it, for four passes that read the dataset back.** A staged dataset holds the VISIBLE fields only, and it carries no header — but `distinctOn`, `split.by`, the `unique` dataset validation and the `pivot` discovery each worked the column list out for themselves from the consumer's full field list, hidden fields included. So on any consumer with a hidden field, each of them read every column after it off by one: `distinctOn` deduplicated on the wrong column, `split.by` bucketed files by it, and `unique` reported duplicates in a field it had never actually looked at. Nothing failed, and the output looked plausible. The run now compiles **one** description of what is on a line — the columns, in line order, with the type behind each one — hands it to every writer and every reader, and `LineParser` takes it as a required parameter with no overload that accepts a field list, so the derivation that was wrong is not a check someone has to remember but a parameter that no longer exists. The same description travels to a `code_lookup` table and to a consumer-as-input capture, both of which re-derived it too
+
+- **A `pivot` changed the shape of the dataset and nothing downstream was told.** A pivot replaces the record's columns with its row keys plus one column per pivot value — and the CSV header, the split pass, the dataset-level validations and the parquet schema all still described the record as it was BEFORE the pivot. A pivoted `CSV` export therefore carried a header naming columns that were no longer in the file, and a split or a validation over a pivoted consumer addressed columns by the pre-pivot positions. The pivot now returns the shape it wrote, and everything after it reads that
+
+- **A value re-read by a post-pass came back as text.** Once a record had been written to the dataset, any pass that read it again (`distinctOn`, `pivot`, the `unique` validation, a `code_lookup` table, a split key, the consumer-as-input capture) got a string for a column whose dimension declared a `number`, a `boolean` or a `datetime` unless the field happened to resolve its dimension — so a numeric `distinctOn` with a `max` resolution compared `"9"` against `"10"` and a lookup table keyed on a number held string keys. Each column of the dataset now carries the three properties a cast reads — `type`, `format` and `nullable` — so a value is cast back to what it was declared as, without anything going near the consumer config
+
+- **Every boolean read from a text source was `true`.** The cast was `Boolean(value)`, and `Boolean('false')` is `true` — as is `Boolean('0')`, `Boolean('no')` and `Boolean('N')`. Every non-empty string is. So a `boolean` dimension over a CSV, a Delta share, a SQL extract or a re-read of the staged dataset reported `true` on every row that held anything at all, and the column was worthless without anyone being told. A boolean arriving as text is now matched against the spellings a source actually writes (`1`, `true`, `yes`, `y`, `t`, case-insensitively) and is `false` otherwise; a real boolean and a missing value are unchanged. **This changes what such a consumer produces** — a column that was uniformly `true` will now hold the falses it always had
+
+- **`pivot` never ran on a source under 10 MB.** `distinct` and `distinctOn` have a worker-side counterpart and are redone over the merged file only when there was more than one chunk, so the whole post-pass block sat behind `executorResults.length > 1` — and `pivot`, which has no worker-side counterpart and is the only pass that MUST run there, inherited that gate. A consumer with a `pivot` over a source below `MIN_FILE_SIZE_FOR_PARALLEL` (10 MB) therefore exported its rows unpivoted, with no warning and no error: the configuration asked for a shape and got a different one. `pivot` now runs whenever it is configured. **This changes what such a consumer produces** — the output gains the pivoted columns it always asked for and loses the raw ones — so anything downstream that adapted to the unpivoted shape needs looking at. Nothing changes for a consumer whose source was already over the threshold
+
+- **A `lineCount` field was renumbered against a file that no longer matched it.** The pass attributes line *i* of the merged dataset to the worker that wrote it by walking each chunk's output count in order, and adds that chunk's row offset — which only holds while the merged file still carries exactly `sum(outputCount)` lines in worker order. It ran LAST, after `distinct`, `distinctOn` and `pivot` had rewritten the file and changed both the count and the order, so on any consumer combining one of those with a `lineCount` field the offsets landed on the wrong rows. It now runs first, which is also the right meaning: a row that survives `distinct` keeps the line number it had in its source file. (Recounting afterwards would not have fixed it — that renumbers the survivors 1..N, which is a different thing, and a pivoted row has no source line at all)
+
+- **Two producers feeding one consumer serialized the same record into two different byte strings.** A `JSON`/`API` record was written out in the order its keys happened to be inserted, which is the order the PRODUCER's dimensions were read — the source's layout, not the consumer's. So the same values under the same names came out as `{"id":7,"country":"IT"}` from one producer and `{"country":"IT","id":7}` from the other, and both `distinct` and the `no_duplicates` dataset validation, which compare the serialized line, saw two different rows. Records are now written in the consumer's own field order, which is one order for every writer in the run — and is the order the fields are declared in, so a `JSON` export reads top to bottom like the consumer does. `remora sample` loses the re-ordering it did at the surface to work around this, because there is nothing left to work around
+
+- **A clean exit deleted a concurrent run's temp folder.** The shutdown handler cleared "open" executor scopes by listing the directories under the temp root — which cannot tell this process's leftovers from another process's live run — and it ran on every exit, not only an unexpected one. Two remora processes sharing one `REMORA_DATA_PATH` (a docker worker beside a CLI run, two scheduled runs, the backend beside anything) were enough: whichever finished first deleted the other's dataset mid-run, and the survivor then failed on a file that had vanished under it. `ExecutorScope` now registers the scopes **this process** opened and the cleanup clears only those, which also makes it a true no-op on a clean exit, since a run that finished already cleared its own scope. The directory listing stays for what it is actually good for — watching folders come and go, which the parallel canary does — and deletes nothing
+
+- An export to a destination that is not a source of the project now says so, naming the consumer and the destination, instead of failing as a `TypeError` from inside the driver factory. The two unreachable branches of the export switch are gone with it (`CSV` and `JSON` both match the staged format and are byte-`move`d, so neither could ever reach the re-encoding path), and an `API` output no longer falls into a branch that parsed a JSON line as CSV — it stages as JSON and ships as JSON
+
+- Bumped `proxy-addr` to 2.0.8 (with its `ipaddr.js` dependency) through the lockfile. It is a transitive dependency of `express`, reached by the backend and worker servers and by the `ui-preview` host, and is not a package remora depends on directly.
+
+### Removed
+
+- **`measures` on a producer.** A producer measure was a `{ name, description, sql }` triple meant to define a calculated metric, and nothing ever calculated one. `ResourcesUtils.getAvailableColumns` exposed each measure as a column a consumer could select, `ConsumerManager` carried it through field expansion as `ICompiledColumn.measure`, and there the trail ended: no stage of the run path ever read `measure.sql`, so the expression was never sent to a source, never evaluated here, and never produced a value. What it did produce was a column that passed `compile` and then exported nothing — a consumer naming a measure validated cleanly and wrote an empty field for it, which is worse than the field not existing, because the configuration says it should be there. Not one producer in this repository declared a measure across all 88 config files that mentioned the property: every one carried the inert `"measures": []` the old templates emitted. The property is now gone from `IProducer`, from `IProducerMeasure`, from `ICompiledColumn`/`ICompiledConsumerField`, from the JSON schema, from the field-lineage chain and the `ui-preview` Measures table, from `create-consumer` (which used to map measures into the generated consumer's fields), from the frontend's producer view, and from the project README and the public Producers doc. Aggregation was never a producer's job — a producer declares what it reads, and a consumer is where values are combined. **Because producer configs are `additionalProperties: false`, this moves the refusal to load time and makes it total**: a project still carrying `measures` — including the empty `"measures": []` almost every generated producer has — now fails `remora compile`/`verify` with `Invalid producer configuration: p_x (remora/producers/p_x.json) - / has unknown property "measures"` instead of loading and ignoring it. Delete the property; it was doing nothing. If the metric it named is one you want, it belongs on the consumer that reads the producer
+
 ## V 2.3.0 - 2026-10-15
 
 **Upgrading from 2.2.** One change alters what an existing project produces: a `CSV` or `TXT` output now writes a header line, so every exported file of that kind gains a first line naming its columns. It is opt-out — `"header": false` on the output restores the old headerless shape — but it is on by default, so a destination that reads the first line as data needs that flag set before this version runs. Nothing else here touches an existing configuration: `remora report` is a new command, and the `.env` refusal only fires on an archive that should never have contained one.

package/docs/Remora - Code Sets.md

@@ -0,0 +1,268 @@
+# Remora - Code Sets
+
+## **Overview**
+
+Code sets are named, reusable lists of allowed values used to drive **synthetic data generation** (the `synth` command). They let you produce realistic, domain-accurate fake data by telling Remora which values a field may take — and, for related fields, which values belong together on the same row.
+
+Code sets affect **only** synthetic generation. They have no effect on normal producer reads, consumer transformations, or any other part of the pipeline.
+
+There are three ways to constrain a producer dimension's synthetic values:
+
+- **Inline** (`synth.allowedValues`) — a quick, one-off list on a single dimension.
+- **Named flat code set** (`synth.codeSet: "name"`) — a reusable list, defined once and referenced from many dimensions.
+- **Named structured code set** (`synth.codeSet: { "name": ..., "field": ... }`) — a table of correlated rows, so related fields (e.g. a code and its description) stay coherent.
+
+All of them are declared under the dimension's `synth` object, which holds every synthetic-only property.
+
+Remora also ships a library of **built-in code sets** (HEDIS measures, code systems, gap statuses, and more) that are applied automatically by field name, with no configuration required.
+
+---
+
+## **How Values Are Chosen**
+
+For each dimension, synthetic generation resolves a value in this order of precedence:
+
+1. Explicit `synth.references` on the dimension (a foreign key — see below)
+2. Explicit `synth.allowedValues` on the dimension
+3. Explicit `synth.codeSet` on the dimension
+4. A **built-in** code set matched by the field's name
+5. A name-based generator (email, city, dates, identifiers, etc.)
+
+`references`, `allowedValues` and `codeSet` are mutually exclusive within a dimension's `synth` object, and any of them wins over a built-in match, so you can override the defaults per field. Picks are uniform (each value equally likely).
+
+---
+
+## **Referential Integrity (Foreign Keys)**
+
+Code sets keep *categorical* fields coherent. To keep *keys* coherent across producers — so a consumer's joins and unions actually line up — declare a foreign key with `synth.references`:
+
+```json
+{ "name": "patient_id", "type": "string",
+  "synth": { "references": { "producer": "p_members", "field": "member_id" } } }
+```
+
+Every synthetic value generated for `patient_id` is drawn from the values `p_members.member_id` emits, so the foreign key is always valid — there are no orphan rows. The relationship works even when the foreign key and the parent key have different names (`patient_id` → `member_id`).
+
+- `producer` / `field` name the parent producer and the parent dimension (by its `name`).
+- The referenced parent field is generated as a **unique key** (like a primary key) so children always find a match. If it isn't marked `pk`, `synth` logs a warning and proceeds.
+- Only single-column relationships are supported. Composite (multi-column) keys are not yet correlated: two `synth.references` fields pointing at the same parent are picked independently, so their pair may not match a single parent row.
+- Generation stays deterministic — the same `--seed` reproduces the same keys and the same joined output.
+
+---
+
+## **Defining Code Sets**
+
+Code sets are a top-level project resource, like sources, producers and consumers. They are **optional** — omit them entirely and the built-in sets still apply.
+
+### **Project Setup**
+
+Create a `code_sets` folder next to `project.json` and place one JSON file per code set inside it. There is nothing to declare: the folder is picked up automatically when it is there, and ignored when it is not.
+
+```
+remora/
+├── project.json
+├── code_sets/
+│   ├── gap_statuses.json
+│   └── hedis_measures.json
+├── producers/
+└── ...
+```
+
+### **Flat Code Sets**
+
+A flat code set is a simple list of values. One value is picked per row.
+
+```json
+{
+  "$schema": "https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/code-set-schema.json",
+  "name": "gap_statuses",
+  "description": "Care-gap compliance statuses.",
+  "values": ["OPEN", "CLOSED", "COMPLIANT", "NON-COMPLIANT", "EXCLUDED", "PENDING"]
+}
+```
+
+Values may be strings, numbers, or booleans:
+
+```json
+{
+  "name": "days_supply",
+  "values": [30, 60, 90]
+}
+```
+
+### **Structured (Correlated) Code Sets**
+
+A structured code set is a table of records. Dimensions bind to a **field** of the set, and every dimension bound to the same set lands on the **same row per record** — so related values stay coherent.
+
+```json
+{
+  "$schema": "https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/code-set-schema.json",
+  "name": "hedis_measures",
+  "description": "Correlated HEDIS measure code, name and coding system.",
+  "records": [
+    { "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" },
+    { "code": "COL", "name": "Colorectal Cancer Screening", "system": "CPT" },
+    { "code": "BCS", "name": "Breast Cancer Screening", "system": "CPT" }
+  ]
+}
+```
+
+> A code set must have **either** `values` **or** `records`, never both.
+
+---
+
+## **Referencing a Code Set from a Producer**
+
+You reference a code set from a producer's dimension's `synth` object (see [Remora - Producers](Remora%20-%20Producers.md)). `allowedValues` and `codeSet` are mutually exclusive on the same dimension.
+
+### **Inline values**
+
+```json
+{
+  "name": "gender_code",
+  "type": "string",
+  "synth": { "allowedValues": ["M", "F", "U"] }
+}
+```
+
+### **Named flat set**
+
+Reference a flat code set by name (a string):
+
+```json
+{
+  "name": "gap_status",
+  "type": "string",
+  "synth": { "codeSet": "gap_statuses" }
+}
+```
+
+### **Named structured set**
+
+Reference a structured code set with `{ "name", "field" }`. Point multiple dimensions at the same set to keep them correlated:
+
+```json
+{
+  "name": "measure_code",
+  "type": "string",
+  "synth": { "codeSet": { "name": "hedis_measures", "field": "code" } }
+},
+{
+  "name": "measure_name",
+  "type": "string",
+  "synth": { "codeSet": { "name": "hedis_measures", "field": "name" } }
+}
+```
+
+With the set above, every generated row pairs a code with its correct name — e.g. `CBP` always appears alongside `Controlling High Blood Pressure`, never a mismatched description.
+
+---
+
+## **Built-in Code Sets**
+
+Remora ships built-in healthcare/payer-quality code sets that are applied **automatically** when a dimension's name matches — no `synth.codeSet` needed. This is why, out of the box, a field named `measure_code` produces real HEDIS codes and `gap_status` produces valid statuses.
+
+| Built-in set | Kind | Applied to fields whose name contains |
+| --- | --- | --- |
+| `hedis_measures` | structured (`code`, `name`, `system`) | `measure_code`, `sub_measure_code`, `measure_classification_code` → `code`; `measure_name`, `measure_description` → `name` |
+| `code_systems` | flat | `code_system` |
+| `gap_statuses` | flat | `gap_code`, `gap_status`, `compliance` |
+| `product_types` | flat | `product_type` |
+| `specialties` | flat | `specialty` |
+| `clinical_events` | flat | `event_name`, `event_category` |
+| `run_types` | flat | `run_type` |
+| `genders` | flat | `gender_code`, `gender`, `sex` |
+| `yn_indicators` | flat | `indicator` |
+| `days_supply` | flat | `days_supply` |
+
+Because `measure_code` and `measure_name` both bind to the built-in `hedis_measures` set, they are automatically correlated on each row — with no configuration.
+
+A built-in set is only applied when it can produce the dimension's declared `type`. A numeric surrogate key such as `heb_quality_gaps_measure_code_sk` matches `measure_code` by name, but HEDIS codes are not numbers, so the binding is skipped and the field is generated from its own type instead. An **explicit** `synth.codeSet` or `synth.allowedValues` that cannot produce the declared type is a configuration error and fails the run by name.
+
+### **Overriding a built-in set**
+
+Define a code set with the **same name** as a built-in one to override it project-wide:
+
+```json
+{
+  "name": "gap_statuses",
+  "values": ["OPEN", "CLOSED"]
+}
+```
+
+Now any field that maps to `gap_statuses` (whether by built-in name matching or an explicit `synth.codeSet`) draws only from your values.
+
+---
+
+## **Determinism**
+
+Synthetic generation is fully deterministic: the same `--seed` always produces the same dataset, including code-set picks and correlated rows. Change the seed to get a different-but-equally-valid dataset. This makes generated data reproducible across runs and machines.
+
+---
+
+## **Code Set Schema Reference**
+
+| Property | Required | Description | Values |
+| --- | --- | --- | --- |
+| `name` | ✅ | Unique code set name, referenced from a dimension's `synth.codeSet` | Any valid identifier |
+| `description` | ❌ | Human-readable description | Descriptive text |
+| `values` | one of `values`/`records` | Flat list; one value picked per row | Array of string / number / boolean |
+| `records` | one of `values`/`records` | Correlated rows; dimensions bind to a field | Array of objects (string / number / boolean fields) |
+
+---
+
+## **End-to-End Example**
+
+`remora/code_sets/hedis_measures.json`:
+
+```json
+{
+  "name": "hedis_measures",
+  "records": [
+    { "code": "CBP", "name": "Controlling High Blood Pressure", "system": "LOINC" },
+    { "code": "COL", "name": "Colorectal Cancer Screening", "system": "CPT" }
+  ]
+}
+```
+
+`remora/code_sets/gap_statuses.json`:
+
+```json
+{
+  "name": "gap_statuses",
+  "values": ["OPEN", "CLOSED", "COMPLIANT", "EXCLUDED"]
+}
+```
+
+`remora/producers/p_quality_gaps.json` (excerpt):
+
+```json
+{
+  "name": "p_quality_gaps",
+  "source": "s_local",
+  "dimensions": [
+    { "name": "member_id", "type": "string", "pk": true },
+    { "name": "measure_code", "type": "string", "synth": { "codeSet": { "name": "hedis_measures", "field": "code" } } },
+    { "name": "measure_name", "type": "string", "synth": { "codeSet": { "name": "hedis_measures", "field": "name" } } },
+    { "name": "gap_status",   "type": "string", "synth": { "codeSet": "gap_statuses" } },
+    { "name": "gender_code",  "type": "string", "synth": { "allowedValues": ["M", "F", "U"] } }
+  ],
+  "settings": { "sqlTable": "quality_gaps_vw" },
+  "_version": 1
+}
+```
+
+Generate synthetic data through a consumer that uses this producer:
+
+```bash
+# Run one consumer (100 rows, default seed)
+npx @forzalabs/remora synth c_quality_gaps
+
+# Control row count and seed
+npx @forzalabs/remora synth c_quality_gaps --records 500 --seed my-seed
+
+# Run every consumer tagged with a given project
+npx @forzalabs/remora synth --project my-project
+```
+
+Each generated row will pair `measure_code` with its matching `measure_name`, draw `gap_status` from your flat set, and pick `gender_code` from the inline list.