@forzalabs/remora 2.6.0 → 2.7.0

package/LICENCE.md

@@ -1,31 +1,66 @@
-Business Source License 1.1
+# Remora Licence
 
-Parameters
+Copyright (c) 2026 Forza Labs LLC. All rights reserved.
 
-Licensor: Forza Labs LLC
+**Licensor:** Forza Labs LLC
 
-Licensed Work: Remora
+**Licensed Work:** Remora, including the `@forzalabs/remora` and `@forzalabs/remora-preview`
+packages and the documentation and schemas distributed with them.
 
-Additional Use Grant: You may make use of the Licensed Work for any non-commercial purpose, including personal, academic, research, or open source projects.
+This is a proprietary licence. It is **not** an open source licence, and it does not convert
+into one. The Licensed Work is made available on the terms below and on no others.
 
-Change Date: 02/07/2025
+## 1. Non-commercial use
 
-Terms
+You may use, copy, modify and redistribute the Licensed Work free of charge for any
+non-commercial purpose, including personal, academic, research, evaluation and open source
+projects, provided you comply with sections 3 to 7.
 
-The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make other use of the Licensed Work. The rights granted are subject to the following conditions:
+Use is non-commercial where it is not carried out for, on behalf of, or for the benefit of a
+company or other organisation, and is not directed towards commercial advantage or monetary
+compensation.
 
-1. **Commercial Use**: Any use of the Licensed Work for or on behalf of a company, organization, or any commercial purpose requires a separate commercial license from the Licensor.
+## 2. Commercial use
 
-2. **Non-Commercial Use**: Non-commercial users may freely use, modify, and distribute the Licensed Work, provided that such use is not intended for commercial advantage or monetary compensation.
+Any other use requires a separate written commercial licence from the Licensor. This includes
+any use for, on behalf of, or for the benefit of a company or other organisation, and any use
+directed towards commercial advantage or monetary compensation.
 
-3. **Change Date**: On the Change Date, the terms of this license will change to the Change License specified above.
+Without such a licence, no right to use the Licensed Work for those purposes is granted. To
+obtain one, contact admin@forza-labs.com.
 
-4. **Other Conditions**:
-   - This license does not grant trademark rights.
-   - You must include this license text with any distribution of the Licensed Work.
-   - You may not remove or modify this license notice.
+## 3. Separate agreements take precedence
 
-Use Limitation: If your usage does not comply with the above terms (e.g., if it's for commercial purposes), you must contact the Licensor to purchase a commercial license.
+Where you have entered into a separate written agreement with the Licensor covering the
+Licensed Work, that agreement governs your use of it and supersedes this notice in full to the
+extent of any conflict.
 
-To inquire about commercial licensing, contact: admin@forza-labs.com
+## 4. Notices
 
+You must include this licence text, unmodified, with any copy or redistribution of the Licensed
+Work. You may not remove, alter or obscure this notice, or any copyright, licence or
+attribution notice contained in the Licensed Work.
+
+## 5. Trademarks
+
+This licence grants no right to use the names, logos or trademarks of the Licensor, including
+"Forza Labs" and "Remora", beyond the attribution section 4 requires.
+
+## 6. No warranty and limitation of liability
+
+THE LICENSED WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE, TITLE AND NONINFRINGEMENT. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO
+EVENT SHALL THE LICENSOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE LICENSED
+WORK OR THE USE OF OR OTHER DEALINGS IN IT.
+
+## 7. Termination
+
+If you use the Licensed Work other than as this notice permits, the rights granted to you end
+automatically. They are reinstated only once you cease the non-compliant use and the Licensor
+confirms reinstatement in writing.
+
+---
+
+Commercial licensing enquiries: admin@forza-labs.com

package/README.md

@@ -39,9 +39,19 @@ remora compile
 remora deploy
 ```
 
+## 📋 Release notes
+
+What changed in each release, and what to check before upgrading, is in
+[RELEASE_NOTES.md](RELEASE_NOTES.md), which ships with this package.
+
 ## 📄 License
 
-BSL
+Proprietary. Free for non-commercial use — personal, academic, research, evaluation and open source
+projects. Any use for, on behalf of, or for the benefit of a company or other organisation requires a
+separate written commercial licence.
+
+The full terms are in [LICENCE.md](LICENCE.md), which ships with this package. For commercial
+licensing, contact admin@forza-labs.com.
 
 ---
 

package/RELEASE_NOTES.md

@@ -0,0 +1,192 @@
+# Remora release notes
+
+What changed in each release, and what to check before upgrading. Newest first.
+
+Run `remora -v` to see the version you have installed.
+
+---
+
+## Unreleased
+
+## 2.7.0 — 2026-10-27
+
+**Added**
+
+- **`scale` on a `number` dimension** reads an amount written without its decimal point: with
+  `"scale": 2`, `000012345` reads as `123.45`. Any source, not only fixed-width files; a value that
+  already carries a decimal point fails the run.
+- **`sign` on a `number` dimension** reads a sign written the way a legacy flat file writes one:
+  `"overpunch"` for a sign carried by the field's last byte (`123J` is `-1231`), `"leading"` or
+  `"trailing"` for a `+`/`-` of its own. Combines with `scale`.
+- **`scale` and `sign` on a `FIXED` output field's `position`**, writing a number the way the layout
+  spells it. Declare the same ones the producer reads with and a positional file read and written
+  back is byte-identical. `remora compile` prints both beside the columns.
+- **Repeating groups nest.** A group's child may be a group in turn, positioned relative to one
+  occurrence of the group it sits in, and `{index1}`…`{indexN}` name the enclosing occurrences.
+- **`remora import-copybook <file> [-n name]`** writes a `FIXED` producer from a COBOL copybook,
+  computing every position from the `PIC` widths. A clause it cannot read fails on the line it is
+  on rather than producing a layout that is nearly right.
+- **`usage` on a dimension** records how the source stores a field. `comp` and `comp-3` are
+  rejected at compile: the feed has to expand them to display characters before it is sent.
+
+**Changed**
+
+- **`remora synth` writes to `remora/.synth/<consumer>/`**, not to each consumer's configured
+  `exportDestination`, so it runs against the consumers you deploy. `-o, --output <dir>` picks
+  another folder. The `move-file` on-finish actions no longer run under `synth`.
+- **A licence problem now explains itself.** A missing, expired, unrecognised or malformed
+  `REMORA_LICENCE_KEY` prints what is wrong and how to fix it, instead of a stack trace.
+- **`init`, `version` and `--help` no longer need a licence**, so a new install can set the project
+  up before its key is in place. Every other command still requires one.
+
+**Fixed**
+
+- A local export whose `exportName` carries a folder prefix — `outbound/feed.csv` — now creates the
+  folder instead of failing on a missing directory.
+
+**Upgrading.** Nothing in a project needs changing. Add `remora/.synth` to your `.gitignore`; `init`
+now does it for new projects.
+
+## 2.6.1 — 2026-10-27
+
+**Changed**
+
+- Remora is distributed under a restated proprietary licence. What you may do is unchanged: free for
+  personal, academic, research, evaluation and open source use, commercial use by written licence.
+  Full text in `LICENCE.md`, and `remora ui-preview` now states the same licence.
+- These release notes ship inside the package. The development changelog, included up to 2.6.0, does
+  not — it is an internal record.
+
+**Upgrading.** Nothing in a project needs changing. If anything of yours reads
+`node_modules/@forzalabs/remora/CHANGELOG.md`, point it at `RELEASE_NOTES.md` in the same folder.
+
+## 2.6.0 — 2026-10-26
+
+**Changed**
+
+- A `CSV` or `TXT` export quotes a field only where the quotes carry meaning — a delimiter, a `"`, a
+  line break, or a space at either end. An empty string stays quoted. Files are around 15% smaller.
+
+**Upgrading.** Check any destination that compares exports against a fixture, or parses them by
+splitting on `","` rather than with a CSV reader.
+
+## 2.5.0 — 2026-10-26
+
+**Added**
+
+- `"format": "FIXED"`, a positional flat-file output: no delimiter, no quotes, no header. Each field
+  declares a `width`, with `align`, `pad`, `nullFill`, `overflow`, `recordLength`, `defaults` and
+  `lineEnding` beside it, and `layouts` for several record layouts in one file.
+- The full prose reference ships with the CLI, so the documentation matches your version.
+
+**Upgrading.** Nothing changes unless a consumer declares a `FIXED` output. A FIXED file leaves Remora as
+UTF-8 text, so a z/OS target still needs ASCII→EBCDIC and `RECFM=FB` on its own side.
+
+## 2.4.0 — 2026-10-19
+
+**Added**
+
+- `"format": "PARQUET"`, a fourth file output.
+
+**Changed**
+
+- A `CSV` or `TXT` export distinguishes a null from an empty string.
+- A value containing a newline no longer becomes two records.
+- A consumer declares exactly one output.
+
+**Removed**
+
+- `measures` on a producer. It never calculated anything, so a consumer naming one exported an empty
+  column.
+
+**Fixed**
+
+- A `hidden` field no longer shifts the columns after it.
+- A `pivot` now tells downstream passes that the shape changed, and runs on sources under 10 MB.
+- A boolean read from a text source is no longer always `true`.
+- A `lineCount` field is renumbered against the file it belongs to.
+- A clean exit no longer deletes a concurrent run's temporary folder.
+
+**Upgrading.** A producer carrying `measures` is refused at `compile` — delete the key, including an empty
+`"measures": []`. Run `remora compile` before upgrading in place.
+
+## 2.3.0 — 2026-10-15
+
+**Added**
+
+- `remora report [run-id]` packages a run — versions, redacted configuration, the run record and its slice
+  of the log — into a `.zip` you can send to someone else.
+
+**Changed**
+
+- A `CSV` or `TXT` export writes a header line naming its columns.
+- `remora package` and `remora deploy` refuse an archive containing a `.env` file.
+
+**Upgrading.** Every exported `CSV` and `TXT` file gains a first line. Check any receiver that expects data
+on line one.
+
+## 2.2.0 / 2.2.1 — 2026-10-15
+
+**Added**
+
+- `split` on a consumer: the output broken into several files by a field value, a size cap or a record
+  count.
+- `remora discover <producer> --replace`.
+- A consumer's filters and the run's record limit reach a SQL source as a `WHERE` and a `LIMIT`.
+- A `presigned-url` engine, usable as a source and as a destination.
+
+**Changed**
+
+- `nullable` on a producer dimension decides what a missing value becomes. It defaults to `true`, which
+  keeps a missing value as null.
+- The configuration archive is called `remora.zip`.
+- `settings.sql` on a producer, and the resource folder names in `project.json`, are removed.
+
+**Fixed**
+
+- `remora discover` no longer overwrites hand-authored dimension properties, and reports what it found on
+  a SQL source.
+- A NULL in a `number` column is no longer staged as `0`.
+- An export no longer loses its last row, and a failed export no longer finishes the run as a success.
+- `region` and `bucket` can be `{ENV_VAR}` references on an S3 or Redshift source.
+
+**Upgrading.** `nullable` now affects what a run produces — a `number` column holding NULLs stops emitting
+`0`. Remove `settings.sql` and the folder names from existing configurations.
+
+## 2.1.0 / 2.1.1 — 2026-09
+
+**Added**
+
+- SQL sources can be run directly.
+- Postgres authenticates with an RDS IAM auth token.
+
+**Changed**
+
+- A live run reports what it read without needing `REMORA_DEBUG_MODE`.
+
+**Fixed**
+
+- A delta-share producer no longer exhausts worker memory.
+- A ZIP producer no longer reads the whole archive into memory.
+- A multi-line error arrives as one log event rather than many unrelated ones.
+
+## 2.0.0 — 2026-09-01
+
+A performance release. A run shares one worker pool across every producer, file and chunk, per-record work
+and output serialization are compiled once rather than per record, and workers write in batches.
+
+**Fixed**
+
+- JSON schema validation runs when a project loads.
+- A filter on a `number`, `datetime` or `boolean` field matches.
+- A run's record limit applies to the whole run, not to each worker.
+- A masked producer dimension is no longer shown in the clear when sampled (2.0.3).
+
+**Upgrading.** A configuration that was silently schema-invalid now fails at `compile`. Run `remora compile`
+before upgrading in place.
+
+---
+
+Earlier 1.x releases are superseded by 2.0. Questions and bug reports:
+[@forzalabs/remora](https://www.npmjs.com/package/@forzalabs/remora) — `remora report` packages a run worth
+attaching.

package/docs/Remora - Code Sets.md

@@ -177,7 +177,7 @@ Remora ships built-in healthcare/payer-quality code sets that are applied **auto
 
 Because `measure_code` and `measure_name` both bind to the built-in `hedis_measures` set, they are automatically correlated on each row — with no configuration.
 
-A built-in set is only applied when it can produce the dimension's declared `type`. A numeric surrogate key such as `heb_quality_gaps_measure_code_sk` matches `measure_code` by name, but HEDIS codes are not numbers, so the binding is skipped and the field is generated from its own type instead. An **explicit** `synth.codeSet` or `synth.allowedValues` that cannot produce the declared type is a configuration error and fails the run by name.
+A built-in set is only applied when it can produce the dimension's declared `type`. A numeric surrogate key such as `quality_gaps_measure_code_sk` matches `measure_code` by name, but HEDIS codes are not numbers, so the binding is skipped and the field is generated from its own type instead. An **explicit** `synth.codeSet` or `synth.allowedValues` that cannot produce the declared type is a configuration error and fails the run by name.
 
 ### **Overriding a built-in set**
 
@@ -200,6 +200,24 @@ Synthetic generation is fully deterministic: the same `--seed` always produces t
 
 ---
 
+## **Where the Output Goes**
+
+`synth` touches nothing real at either end. The producers are served from generated files instead of their sources, and the exports are written to a local folder — `remora/.synth/<consumer>/` by default, or wherever `--output` points — instead of each consumer's configured `exportDestination`.
+
+That is what lets you run it against the consumers you deploy. Their destinations are production buckets, shares and endpoints: unreachable from a development machine, and not somewhere fake rows belong, since a consumer with a fixed `exportName` would overwrite the live feed.
+
+```bash
+# Writes to remora/.synth/c_quality_gaps/, not to the consumer's destination
+npx @forzalabs/remora synth c_quality_gaps
+
+# Write somewhere else
+npx @forzalabs/remora synth c_quality_gaps --output ./tmp/fixtures
+```
+
+The export itself still runs in full — format conversion, header, field positions, file splitting, FIXED layouts — so the file you get is what the destination would have received, byte for byte. `move-file` actions on `onSuccess`/`onError` are the one thing skipped: they archive and delete the producers' real source files, which a synthetic run never read. `run-consumer` actions still fire, and the consumers they start are synthesized too.
+
+---
+
 ## **Code Set Schema Reference**
 
 | Property | Required | Description | Values |
@@ -263,6 +281,9 @@ npx @forzalabs/remora synth c_quality_gaps --records 500 --seed my-seed
 
 # Run every consumer tagged with a given project
 npx @forzalabs/remora synth --project my-project
+
+# Write the exports somewhere other than remora/.synth
+npx @forzalabs/remora synth c_quality_gaps --output ./tmp/fixtures
 ```
 
 Each generated row will pair `measure_code` with its matching `measure_name`, draw `gap_status` from your flat set, and pick `gender_code` from the inline list.

package/docs/Remora - Consumers.md

@@ -1155,6 +1155,8 @@ The file is exported with a `.txt` extension, to any file destination (local, S3
 | `position.nullFill` | ❌ | The single character a **null** field is filled with, for its full width. Defaults to a space for every type | `" "`, `"0"` |
 | `position.overflow` | ❌ | What a value longer than `width` does | `"fail"` (default), `"truncate"` |
 | `position.start` | ❌ | 1-based column the field is expected to start at. An assertion, not a second source of truth (see below) | `21` |
+| `position.scale` | ❌ | `number` fields only: implied decimal places the field is written with, the point dropped (see [How a Number Is Spelled](#how-a-number-is-spelled)) | `2` |
+| `position.sign` | ❌ | `number` fields only: how the field carries its sign, where it is not a plain leading `-` (see [How a Number Is Spelled](#how-a-number-is-spelled)) | `"overpunch"`, `"leading"`, `"trailing"` |
 
 Five rules decide what lands on the line:
 
@@ -1172,6 +1174,29 @@ Five rules decide what lands on the line:
 
 **Carry a positional field through as a `string`.** A `number` dimension drops leading zeros — `"00"` read as a number is `0` — so writing it back takes `"align": "right"` with `"pad": "0"`, a rule you have to get right field by field. Declaring the producer dimension a `string` instead carries the characters through untouched.
 
+### **How a Number Is Spelled**
+
+The receivers that read by column also write their numbers by column: an amount **without its decimal point**, and a negative sign carried by a byte that also holds a digit. `position.scale` and `position.sign` are the write side of the producer dimension's `scale` and `sign` (see the Producers guide), and they mean the same things in the other direction:
+
+| Property | The field is written as | `-123.45` is written |
+| --- | --- | --- |
+| `"scale": 2` | digits only, the last 2 of them the fraction | `-12345` |
+| `"sign": "overpunch"` | digits, the last byte carrying the sign as well as its digit | `1234N` |
+| `"sign": "leading"` | a `+` or `-`, then digits | `-12345` |
+| `"sign": "trailing"` | digits, then a `+` or `-` | `12345-` |
+
+```json
+{ "key": "claim_amount", "position": { "width": 10, "scale": 2, "sign": "overpunch" } }
+```
+
+`-123.45` is then written `000001234N`, and `123.45` as `000001234E`.
+
+- `number` fields only, and both are optional on their own. A field writing a plain `-123.45` declares neither.
+- **A field declaring `sign` is a signed field, so a positive value is written in the form the mode declares** — `123A`, `+123`, `123+` — never as bare digits.
+- Declared on the output, not taken from the producer: the file you write is not the file you read, and a value can reach the output from anywhere. Declare the same `scale` and `sign` on both sides and a field read from a positional file is written back byte for byte.
+- The padding goes where the layout wants it. A `leading` sign keeps the first column and the fill goes between it and the digits (`-0001234`), which is where a COBOL `SIGN LEADING SEPARATE` field carries it.
+- `compile` prints both alongside the columns, since a wrong `scale` writes a plausible amount that is 10ⁿ off.
+
 ### **Column Order, Gaps and `start`**
 
 **Column order is field order, and every offset is computed from the widths before it.** No column number is ever transcribed by hand: list the fields in the order the record is written, give each one a width, and the layout falls out of them. That is the same rule the read side follows, and it is why a layout is written once rather than renumbered every time a field moves.
@@ -1194,7 +1219,7 @@ FIXED layout of consumer "claims_export"
     start    end  width  field
         1     12     12  member_id
        13     20      8  service_date
-       21     30     10  amount
+       21     30     10  amount  (2 implied decimal(s), overpunch sign)
 ```
 
 Every problem with a layout is reported in one `compile` rather than one per run, so six fields missing a width come back as six lines. `run` and `verify` resolve the same layout before they read a producer, so a layout that does not add up fails in front of the run instead of after the whole dataset has been built.
@@ -1271,7 +1296,7 @@ How it behaves:
 
 ### **How a Record Ends**
 
-`lineEnding` is `"lf"` (the default) or `"crlf"`, and it ends **every** record, the last one included. A real positional feed picks one and holds it: NASCO POD files, for instance, are CRLF throughout.
+`lineEnding` is `"lf"` (the default) or `"crlf"`, and it ends **every** record, the last one included. A real positional feed picks one and holds it: a POD-style transaction feed, for instance, may be CRLF throughout.
 
 A file with **no terminator at all** — fixed record framing, `RECFM=FB`, where the record length alone says where one record ends — is not expressible here, and asking for it is the signal that what is needed is byte-mode framing rather than a line-based file. Remora does not write one.
 

package/docs/Remora - Producers.md

@@ -179,8 +179,11 @@ Notes:
 - Values are **trimmed**, so space- and zero-padded fields (`"0000123.45"`, `"ABC   "`) read as their real value. A field that is entirely padding reads as `null` — or as its type's empty value, if the dimension declares `nullable: false`.
 - Gaps are fine: filler columns you don't map simply have no dimension. Overlaps are fine too — a composite field and its subfields can both be declared.
 - Combine `position` with `format` to parse dates written as bare digits, as in `service_date` above. Day-of-year ("Julian") dates use `DDD`: `"format": "YYYYDDD"` reads `2007305` as 1 November 2007.
+- An amount written **without its decimal point** — `0012345` meaning `123.45`, the way a COBOL `PIC S9(7)V99` field is written — needs `scale`. See [Implied decimals](#implied-decimals).
+- An amount whose **sign is written in the last byte** — `123J` meaning `-1231`, the way a signed COBOL `DISPLAY` field is — needs `sign`. See [Written signs](#written-signs).
 - A dimension marked `sourceFilename` needs no `position`: it isn't in the line.
-- `discover` cannot be used on a `FIXED` producer — a positional layout can only come from the file's spec. Use `sample` to check that your positions line up.
+- A feed unloaded from a mainframe has to be converted before any of this reads — ASCII, display digits, one record per line. See [Mainframe unloads](#mainframe-unloads).
+- `discover` cannot be used on a `FIXED` producer — a positional layout can only come from the file's spec. Use `sample` to check that your positions line up. If the spec is a COBOL copybook, [`import-copybook`](#importing-a-cobol-copybook) writes the whole producer from it.
 
 ### **Repeating Groups (OCCURS)**
 
@@ -213,7 +216,7 @@ The group expands into one dimension per (occurrence × child) when the project
 | `position` | ✅ | Extent of **one** occurrence: `start` of the first, `length` per occurrence | `{ "start": 32, "length": 30 }` |
 | `occurs` | ❌ | Number of occurrences. Default `1`, max `1000`. An object when the count varies per line — see [Variable Occurrences](#variable-occurrences-and-anchoring) | `3` |
 | `indexFrom` | ❌ | First index used in generated names: `0` (default) or `1` | `1` |
-| `naming` | ❌ | Name template. Tokens `{group}`, `{index}`, `{field}`, `{Field}` (capitalized). `{index}` is required unless `occurs` is `1`. Default `{group}{index}{Field}` | `{group}_{index}_{field}` → `addr_0_street` |
+| `naming` | ❌ | Name template. Tokens `{group}`, `{index}`, `{field}`, `{Field}` (capitalized), plus `{index1}`…`{indexN}` inside a nested group. `{index}` is required unless `occurs` is `1`. Default `{group}{index}{Field}` | `{group}_{index}_{field}` → `addr_0_street` |
 
 Notes:
 
@@ -222,7 +225,41 @@ Notes:
 - `occurs: 1` is useful on its own: a group is then simply a way to **structure** a segment of the layout, and with `"naming": "{field}"` the children keep their own names.
 - Unpopulated occurrences read as `null`, exactly like any all-padding field.
 - A child position that doesn't fit inside one occurrence is rejected — that is the classic transcription slip, an absolute offset pasted where a relative one belongs.
-- Nested groups (a group inside a group) are **not supported yet** and are rejected with a message saying so. Declare the inner group as a sibling with its own absolute `position`.
+- Groups **nest**, the way a copybook's `OCCURS` clauses do — see [Nested groups](#nested-groups).
+
+### **Nested Groups**
+
+A copybook nests `OCCURS` clauses routinely — an address segment that repeats, each occurrence holding two phone slots that repeat in turn. A group's child may itself be a group, and the same two rules apply one level down: its `position` is the extent of **one** of its own occurrences, measured **relative to one occurrence of the group it sits in**.
+
+```json
+{
+  "name": "addr",
+  "position": { "start": 1, "length": 30 },
+  "occurs": 2,
+  "dimensions": [
+    { "name": "street", "type": "string", "position": { "start": 1, "length": 10 } },
+    {
+      "name": "phone",
+      "position": { "start": 11, "length": 10 },
+      "occurs": 2,
+      "dimensions": [
+        { "name": "area", "type": "string", "position": { "start": 1, "length": 3 } },
+        { "name": "num",  "type": "string", "position": { "start": 4, "length": 7 } }
+      ]
+    }
+  ]
+}
+```
+
+Each level names what the level below it generated, so the default template gives `addr0Street`, `addr0Phone0Area`, `addr0Phone0Num`, `addr0Phone1Area`, … `addr1Phone1Num` — ten dimensions at columns 1-10, 11-13, 14-20, 21-23, 24-30, then 31-40, 41-43, … 54-60.
+
+Notes:
+
+- Inside a nested group's `naming`, **`{index}` is that group's own occurrence** and `{index1}`…`{indexN}` reach N levels outward, for a template that wants the outer index somewhere of its own: `"naming": "{group}{index1}_{index}_{field}"` on `phone` above gives `phone0_1_area`.
+- All of a nested group's occurrences must fit inside **one** occurrence of the group it sits in. `phone` above takes columns 11-30 of a 30-character address, and a position that runs past the end is rejected.
+- `occurs` is capped at 1000 **per level**, so a group nested in another is bounded at the product.
+- A nested group repeats a **fixed** number of times: `occurs` as an object, and `emit`, belong to the outermost group only. A per-line count can only be read along the line, not inside another group's occurrence.
+- Everything else is unchanged: the whole tree flattens to plain positioned dimensions when the project loads, so nothing downstream knows a group was nested.
 
 ### **Occurrences as Rows**
 
@@ -375,6 +412,69 @@ A matching line shorter than `minLength` is dropped and logged at debug level (`
 | `minLength` | ❌ | Shortest a valid line of this record type can be, in characters | `2400` |
 | `onLengthMismatch` | ❌ | What to do with a matching line that is too short | `"skip"` (default), `"fail"` |
 
+### **Mainframe Unloads**
+
+Remora does not connect to VSAM. A VSAM file is reached through z/OS access-method calls, not as a byte stream, so there is nothing at the other end of a connection to read. What Remora reads is the **unload**: the flat file an `IDCAMS REPRO` step writes, described by the COBOL copybook of the record it holds. Producing that unload, and converting it, belongs on the mainframe side of the handover — Remora's side starts at the file that arrives.
+
+**The unload has to arrive as ASCII display data.** That is a requirement of reading a mainframe feed, not a preference. Before the file leaves z/OS, a conversion step — the `DFSORT`/`OUTREC` job most shops already run, or the equivalent in whichever sort product is installed — has to do three things:
+
+- **Translate the record from EBCDIC to ASCII.** Remora reads the file as text and carries no EBCDIC codepage.
+- **Expand every `COMP-3` (packed decimal) and `COMP` (binary) field to display digits**, each at the DISPLAY width of its `PIC` clause. Packed decimal holds two digits per byte and binary holds no digits at all, so there are no characters at those columns for a positional read to address.
+- **Write one record per line**, newline-terminated, each record padded to its full width. `RECFM=FB` writes no separator between records and `RECFM=VB` prefixes each record with a 4-byte RDW; neither of those is a line.
+
+Nothing else about the record has to change, and nothing else should. An overpunched sign and an implied decimal point are ordinary characters once the record is ASCII, and Remora reads both as the copybook spells them — see [Implied decimals](#implied-decimals) and [Written signs](#written-signs). Ask for the layout itself to be left exactly as the copybook describes it: the columns are what the producer is written from, so a conversion that re-aligns, re-orders or trims fields makes the copybook wrong.
+
+**Positions are counted in characters**, everywhere in a `FIXED` producer. In a converted single-byte record a character is a byte and both agree with the copybook's offsets — which is why the conversion happens before the file is sent, rather than being described in the layout.
+
+If a feed arrives unconverted:
+
+| What arrives | What happens |
+| --- | --- |
+| A record still in EBCDIC | No column reads as itself: text fields hold unrelated characters and numeric ones fail on them. Sample a first delivery before trusting it. |
+| `RECFM=FB`, no newlines | The whole file reads as one very long line, so only the first record's columns land anywhere. |
+| `RECFM=VB`, no newlines | The same, and each record's four RDW bytes shift every position after them. |
+| A `COMP-3` or `COMP` field | `remora compile` fails, naming the dimension and the conversion the feed needs. [`import-copybook`](#importing-a-cobol-copybook) writes `"usage": "comp-3"` (or `"comp"`) on such a field for exactly that reason: once the conversion job expands it, delete `usage` and the producer compiles. |
+
+`COMP-1` and `COMP-2` (floating point) are not supported, converted or not: `import-copybook` refuses a copybook that declares one rather than writing a field that cannot be read.
+
+**What to ask for.** One request to the team that owns the job covers it — an `IDCAMS REPRO` unload of the file, converted to ASCII with `COMP-3` and `COMP` fields expanded to display digits, written as newline-delimited records of fixed length, together with the copybook that describes the record. With the copybook in hand, [`import-copybook`](#importing-a-cobol-copybook) writes the producer from it.
+
+### **Importing a COBOL Copybook**
+
+A fixed-width feed unloaded from a mainframe comes with a COBOL copybook, and the copybook is the only place the width, the implied decimals and the written sign of each field are stated. `import-copybook` reads it and writes the producer:
+
+```bash
+# Write remora/producers/p_orders.json from the copybook
+remora import-copybook ./layouts/ORDERS.cpy -n p_orders
+
+# Without -n, the producer is named after the copybook file ("ORDERS.cpy" -> "p_orders")
+remora import-copybook ./layouts/ORDERS.cpy
+```
+
+The parse is deterministic, never inferred: level numbers, `PIC` clauses, `USAGE`, `OCCURS` (fixed and `DEPENDING ON`) and `REDEFINES`. **Every `position` is computed from the PIC widths**, so no offset is transcribed by hand — which is the point of the command on a layout of several hundred fields.
+
+| Copybook | Producer |
+| --- | --- |
+| `PIC X(12)` | `"type": "string"`, `"position": { "length": 12 }` |
+| `PIC 9(8)` | `"type": "number"` |
+| `PIC S9(7)V99` | `"type": "number"`, `"scale": 2`, `"sign": "overpunch"` |
+| `PIC S9(5) SIGN IS TRAILING SEPARATE` | `"type": "number"`, `"sign": "trailing"`, one character wider |
+| `PIC ZZ,ZZ9.99` | `"type": "string"` — an edited picture holds its insertion characters, so the field is text |
+| `OCCURS 3` | a [repeating group](#repeating-groups-occurs), `"indexFrom": 1` so the names match the copybook |
+| `OCCURS 1 TO 10 DEPENDING ON ORD-COUNT` | `"occurs": { "max": 10, "min": 1, "from": "$ordCount" }`, and everything after it [anchored](#variable-occurrences-and-anchoring) |
+| `REDEFINES` | the first reading only; the alternates are named in the dimension's `description` |
+| `FILLER` | nothing — it carries width, not a value |
+
+Names are the copybook's own: `ORD-TOTAL-AMT` becomes `ordTotalAmt`, and a name two groups share is qualified with the group it sits in. Groups that do not repeat are flattened, since they structure the copybook rather than the record.
+
+Both source formats are read: a fixed-format copybook (sequence numbers in columns 1-6, an indicator in column 7, text in columns 8-72) and a free-format one. Comment lines are skipped.
+
+**The feed still has to arrive as display characters**, as every mainframe unload does — see [Mainframe unloads](#mainframe-unloads). A `COMP-3` or `COMP` field is emitted with its `usage` recorded (`"usage": "comp-3"`) and `remora compile` rejects it by name, so the conversion job expands the field; the `usage` property is then removed and the producer compiles.
+
+A clause the importer cannot read **fails, naming the copybook line**, rather than producing a layout that is nearly right: a field read from the wrong column holds a plausible value that no run and no validation can catch. `OCCURS DEPENDING ON` inside another repeating group, a `PIC` using `P`, `COMP-1`/`COMP-2`, level 66 `RENAMES` and a second record description in one copybook are all refused this way. Read each record type of a mixed file with its own producer and a [`recordSelector`](#mixed-record-types-hdt).
+
+The written producer points at a placeholder source and file: set `source` and `settings.fileKey`, then run `remora compile` and `remora sample` to check the positions against a real line.
+
 ### **File Pattern Matching**
 
 Use wildcards to process multiple files matching a pattern:
@@ -785,11 +885,14 @@ Dimensions define the structure and characteristics of data fields in your produ
 | `classification` | ❌ | Data privacy classification | `["PII"]`, `["PHI"]`, `["GDPR"]` |
 | `mask` | ❌ | Data masking method | `hash`, `mask`, `crypt`, `random`, `seeded-random`, `none` |
 | `format` | ❌ | Source parse/cast format (e.g. a date format). `DDD` reads a day-of-year ("Julian") date | `"YYYYMMDD"`, `"DD-MM-YYYY"`, `"YYYYDDD"` |
+| `scale` | ❌ | Implied decimal places of a `number` written without a decimal point. See [Implied decimals](#implied-decimals). | `2` |
+| `sign` | ❌ | How a `number` written as text carries its sign. See [Written signs](#written-signs). | `overpunch`, `leading`, `trailing` |
+| `usage` | ❌ | How the source system stores the field, as a copybook's `USAGE` clause declares it. Written by [`import-copybook`](#importing-a-cobol-copybook); `comp` and `comp-3` are rejected at compile | `display`, `comp`, `comp-3` |
 | `sourceFilename` | ❌ | Populate this dimension with the source file name (file producers only; one per producer) | `true`, `false` (default) |
 | `position` | ⚠️ | Where the value sits in the line, for `FIXED` (fixed-width) producers — required on every dimension of one, ignored elsewhere. See [Fixed-Width Files](#fixed-width-files-fixed). | `{ "start": 21, "length": 10 }`, `{ "start": 21, "end": 30 }` |
 | `position.after` | ❌ | Anchors the position to the end of an earlier group's actual occurrences instead of to the start of the line. See [Variable Occurrences](#variable-occurrences-and-anchoring). | `{ "after": "reimb", "length": 18 }` |
-| `dimensions` | ❌ | Makes this a repeating group (`FIXED` only): the fields that repeat, positioned relative to one occurrence. See [Repeating Groups](#repeating-groups-occurs). | see Repeating Groups |
-| `occurs` | ❌ | Occurrences of a repeating group. Default `1`, max `1000`; an object when the count varies per line | `3`, `{ "max": 36, "while": { ... } }` |
+| `dimensions` | ❌ | Makes this a repeating group (`FIXED` only): the fields that repeat, positioned relative to one occurrence. Groups nest — see [Repeating Groups](#repeating-groups-occurs) and [Nested Groups](#nested-groups). | see Repeating Groups |
+| `occurs` | ❌ | Occurrences of a repeating group. Default `1`, max `1000` per level; an object when the count varies per line, on the outermost group only | `3`, `{ "max": 36, "while": { ... } }` |
 | `indexFrom` | ❌ | First index used in the names a group generates | `0` (default), `1` |
 | `naming` | ❌ | Name template for the dimensions a group or `flags` generates | `"{group}_{index}_{field}"`, `"{field}_{flag}"` |
 | `emit` | ❌ | Whether a group's occurrences become columns (default) or one record each. See [Occurrences as Rows](#occurrences-as-rows). | `"columns"`, `"rows"` |
@@ -900,6 +1003,58 @@ Bound the **parent** field of a foreign key, never the child: a `synth.reference
 }
 ```
 
+#### Implied decimals
+
+Flat files from mainframe and other legacy systems write an amount **without its decimal point**: the field holds digits only, and the layout says how many of the trailing ones are the fraction. A COBOL `PIC S9(7)V99` amount of `123.45` is written `000012345`. Read as-is that is `12345` — a perfectly plausible amount, 100× wrong, and nothing downstream can tell.
+
+`scale` says how many of the trailing digits are the fraction:
+
+```json
+{
+  "name": "claim_amount",
+  "type": "number",
+  "scale": 2,
+  "position": { "start": 21, "length": 9 }
+}
+```
+
+`000012345` then reads as `123.45`, and `000000005` as `0.05`.
+
+- `number` dimensions only, and 1 or more. A field written *with* its decimal point, or with no fraction at all, omits `scale` entirely.
+- A field declaring `scale` holds digits and an optional leading sign, nothing else. A decimal point in the value contradicts the layout, so the run fails naming the value rather than guessing which of the two readings was meant.
+- Not restricted to `FIXED`: a CSV column written the same way reads the same way.
+- Only text is decoded. A source that already hands Remora a number (JSON, SQL) hands over the value itself, so `scale` leaves it alone.
+- `synth` generates values with `scale` decimal places, so synthetic data fits the column it describes. `synth.decimals` overrides that.
+
+#### Written signs
+
+The same files rarely write a negative amount with a leading `-`. A signed COBOL `DISPLAY` field carries its sign in the **last byte**, which also holds the last digit: `-1231` is written `123J`, and `+1231` as `123A`. Read as-is, `Number("123J")` is `NaN` — the whole column arrives empty.
+
+`sign` says how the sign is written:
+
+| `sign` | The field holds | `-123` is written |
+|---|---|---|
+| `overpunch` | digits, the last byte carrying the sign as well as its digit: `{` and `A`-`I` positive, `}` and `J`-`R` negative | `12L` |
+| `leading` | an optional `+`/`-`, then digits | `-123` |
+| `trailing` | digits, then an optional `+`/`-` | `123-` |
+
+```json
+{
+  "name": "claim_amount",
+  "type": "number",
+  "scale": 2,
+  "sign": "overpunch",
+  "position": { "start": 21, "length": 10 }
+}
+```
+
+`000001234E` then reads as `123.45`, and `000001234N` as `-123.45`.
+
+- `number` dimensions only, and never inferred. Without `sign`, a letter in a numeric field is a malformed record rather than a sign — so a column whose feed uses overpunched signs has to say so, and one that does not keeps failing loudly on a stray letter.
+- `overpunch` reads overpunched signs only. A field whose conversion writes a real `+`/`-` is `leading` or `trailing` instead; a plain digit in the last byte is the unsigned form and reads as positive.
+- Combines with `scale`, which is applied after the sign is taken off: `0012345-` at `"scale": 2` is `-123.45`.
+- Like `scale`, it decodes text only, and is not restricted to `FIXED`.
+
 For synthetic data (`synth`), use `synth.decimals` to control how many decimal places generated `number` values get, so they load into the destination column. `decimals: 0` forces whole integers (fits an `integer` column); a positive value rounds to that many decimals:
 
 ```json

package/documentation/README.md

@@ -94,6 +94,7 @@ remora synth [consumer] [-r n] [-s seed]  # run the pipeline on generated fake d
 remora discover <producer>           # read the source and fill in the dimensions for you
 remora create-producer <name>        # scaffold a producer
 remora create-consumer <name> [-p producer]   # scaffold a consumer
+remora import-copybook <file> [-n name]       # write a FIXED producer from a COBOL copybook
 remora automap <producer> <schemas...>        # map a producer onto schemas with AI
 remora graph [-f mermaid|dot|json]   # print the lineage — what feeds what
 remora ui-preview [-p port]          # browse the project and its data in a browser
@@ -225,7 +226,7 @@ Set these in the `.env` beside your `remora/` folder.
 
 | Variable | |
 |---|---|
-| `REMORA_LICENCE_KEY` | **Required.** No command runs without a valid licence |
+| `REMORA_LICENCE_KEY` | **Required.** Every command needs a valid licence, except `init`, `version` and `--help` |
 | `REMORA_RUNTIME_CONTEXT` | `cli` — already added by `init` |
 | `OPENAI_API_KEY` | Optional. Enables the AI features (`discover`, `automap`) |
 | `REMORA_WORKER_HOST` | Optional. Your Remora Worker URL, used by `deploy` |