@forzalabs/remora 2.6.1 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/RELEASE_NOTES.md +51 -16
- package/docs/Remora - Code Sets.md +21 -0
- package/docs/Remora - Consumers.md +26 -1
- package/docs/Remora - Producers.md +160 -5
- package/documentation/README.md +2 -1
- package/index.js +1489 -409
- package/json_schemas/consumer-schema.json +11 -0
- package/json_schemas/producer-schema.json +22 -3
- package/package.json +1 -1
- package/workers/ExecutorWorker.js +1119 -136
package/RELEASE_NOTES.md
CHANGED
|
@@ -8,17 +8,54 @@ Run `remora -v` to see the version you have installed.
|
|
|
8
8
|
|
|
9
9
|
## Unreleased
|
|
10
10
|
|
|
11
|
+
## 2.7.0 — 2026-10-27
|
|
12
|
+
|
|
13
|
+
**Added**
|
|
14
|
+
|
|
15
|
+
- **`scale` on a `number` dimension** reads an amount written without its decimal point: with
|
|
16
|
+
`"scale": 2`, `000012345` reads as `123.45`. Any source, not only fixed-width files; a value that
|
|
17
|
+
already carries a decimal point fails the run.
|
|
18
|
+
- **`sign` on a `number` dimension** reads a sign written the way a legacy flat file writes one:
|
|
19
|
+
`"overpunch"` for a sign carried by the field's last byte (`123J` is `-1231`), `"leading"` or
|
|
20
|
+
`"trailing"` for a `+`/`-` of its own. Combines with `scale`.
|
|
21
|
+
- **`scale` and `sign` on a `FIXED` output field's `position`**, writing a number the way the layout
|
|
22
|
+
spells it. Declare the same ones the producer reads with and a positional file read and written
|
|
23
|
+
back is byte-identical. `remora compile` prints both beside the columns.
|
|
24
|
+
- **Repeating groups nest.** A group's child may be a group in turn, positioned relative to one
|
|
25
|
+
occurrence of the group it sits in, and `{index1}`…`{indexN}` name the enclosing occurrences.
|
|
26
|
+
- **`remora import-copybook <file> [-n name]`** writes a `FIXED` producer from a COBOL copybook,
|
|
27
|
+
computing every position from the `PIC` widths. A clause it cannot read fails on the line it is
|
|
28
|
+
on rather than producing a layout that is nearly right.
|
|
29
|
+
- **`usage` on a dimension** records how the source stores a field. `comp` and `comp-3` are
|
|
30
|
+
rejected at compile: the feed has to expand them to display characters before it is sent.
|
|
31
|
+
|
|
32
|
+
**Changed**
|
|
33
|
+
|
|
34
|
+
- **`remora synth` writes to `remora/.synth/<consumer>/`**, not to each consumer's configured
|
|
35
|
+
`exportDestination`, so it runs against the consumers you deploy. `-o, --output <dir>` picks
|
|
36
|
+
another folder. The `move-file` on-finish actions no longer run under `synth`.
|
|
37
|
+
- **A licence problem now explains itself.** A missing, expired, unrecognised or malformed
|
|
38
|
+
`REMORA_LICENCE_KEY` prints what is wrong and how to fix it, instead of a stack trace.
|
|
39
|
+
- **`init`, `version` and `--help` no longer need a licence**, so a new install can set the project
|
|
40
|
+
up before its key is in place. Every other command still requires one.
|
|
41
|
+
|
|
42
|
+
**Fixed**
|
|
43
|
+
|
|
44
|
+
- A local export whose `exportName` carries a folder prefix — `outbound/feed.csv` — now creates the
|
|
45
|
+
folder instead of failing on a missing directory.
|
|
46
|
+
|
|
47
|
+
**Upgrading.** Nothing in a project needs changing. Add `remora/.synth` to your `.gitignore`; `init`
|
|
48
|
+
now does it for new projects.
|
|
49
|
+
|
|
11
50
|
## 2.6.1 — 2026-10-27
|
|
12
51
|
|
|
13
52
|
**Changed**
|
|
14
53
|
|
|
15
54
|
- Remora is distributed under a restated proprietary licence. What you may do is unchanged: free for
|
|
16
|
-
personal, academic, research, evaluation and open source use,
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
installed sits on disk beside it. The development changelog, which was included up to 2.6.0, is an
|
|
21
|
-
internal working record and is no longer part of the package.
|
|
55
|
+
personal, academic, research, evaluation and open source use, commercial use by written licence.
|
|
56
|
+
Full text in `LICENCE.md`, and `remora ui-preview` now states the same licence.
|
|
57
|
+
- These release notes ship inside the package. The development changelog, included up to 2.6.0, does
|
|
58
|
+
not — it is an internal record.
|
|
22
59
|
|
|
23
60
|
**Upgrading.** Nothing in a project needs changing. If anything of yours reads
|
|
24
61
|
`node_modules/@forzalabs/remora/CHANGELOG.md`, point it at `RELEASE_NOTES.md` in the same folder.
|
|
@@ -27,25 +64,23 @@ Run `remora -v` to see the version you have installed.
|
|
|
27
64
|
|
|
28
65
|
**Changed**
|
|
29
66
|
|
|
30
|
-
- A `CSV` or `TXT` export quotes a field only where the quotes carry meaning —
|
|
31
|
-
|
|
32
|
-
distinct from a null. Exported files are around 15% smaller.
|
|
67
|
+
- A `CSV` or `TXT` export quotes a field only where the quotes carry meaning — a delimiter, a `"`, a
|
|
68
|
+
line break, or a space at either end. An empty string stays quoted. Files are around 15% smaller.
|
|
33
69
|
|
|
34
|
-
**Upgrading.**
|
|
35
|
-
|
|
70
|
+
**Upgrading.** Check any destination that compares exports against a fixture, or parses them by
|
|
71
|
+
splitting on `","` rather than with a CSV reader.
|
|
36
72
|
|
|
37
73
|
## 2.5.0 — 2026-10-26
|
|
38
74
|
|
|
39
75
|
**Added**
|
|
40
76
|
|
|
41
77
|
- `"format": "FIXED"`, a positional flat-file output: no delimiter, no quotes, no header. Each field
|
|
42
|
-
declares a `width`, with `align`, `pad`, `nullFill`
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
- The full prose reference now ships with the CLI, so the documentation always matches your version.
|
|
78
|
+
declares a `width`, with `align`, `pad`, `nullFill`, `overflow`, `recordLength`, `defaults` and
|
|
79
|
+
`lineEnding` beside it, and `layouts` for several record layouts in one file.
|
|
80
|
+
- The full prose reference ships with the CLI, so the documentation matches your version.
|
|
46
81
|
|
|
47
82
|
**Upgrading.** Nothing changes unless a consumer declares a `FIXED` output. A FIXED file leaves Remora as
|
|
48
|
-
UTF-8 text, so a z/OS target still needs
|
|
83
|
+
UTF-8 text, so a z/OS target still needs ASCII→EBCDIC and `RECFM=FB` on its own side.
|
|
49
84
|
|
|
50
85
|
## 2.4.0 — 2026-10-19
|
|
51
86
|
|
|
@@ -200,6 +200,24 @@ Synthetic generation is fully deterministic: the same `--seed` always produces t
|
|
|
200
200
|
|
|
201
201
|
---
|
|
202
202
|
|
|
203
|
+
## **Where the Output Goes**
|
|
204
|
+
|
|
205
|
+
`synth` touches nothing real at either end. The producers are served from generated files instead of their sources, and the exports are written to a local folder — `remora/.synth/<consumer>/` by default, or wherever `--output` points — instead of each consumer's configured `exportDestination`.
|
|
206
|
+
|
|
207
|
+
That is what lets you run it against the consumers you deploy. Their destinations are production buckets, shares and endpoints: unreachable from a development machine, and not somewhere fake rows belong, since a consumer with a fixed `exportName` would overwrite the live feed.
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
# Writes to remora/.synth/c_quality_gaps/, not to the consumer's destination
|
|
211
|
+
npx @forzalabs/remora synth c_quality_gaps
|
|
212
|
+
|
|
213
|
+
# Write somewhere else
|
|
214
|
+
npx @forzalabs/remora synth c_quality_gaps --output ./tmp/fixtures
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
The export itself still runs in full — format conversion, header, field positions, file splitting, FIXED layouts — so the file you get is what the destination would have received, byte for byte. `move-file` actions on `onSuccess`/`onError` are the one thing skipped: they archive and delete the producers' real source files, which a synthetic run never read. `run-consumer` actions still fire, and the consumers they start are synthesized too.
|
|
218
|
+
|
|
219
|
+
---
|
|
220
|
+
|
|
203
221
|
## **Code Set Schema Reference**
|
|
204
222
|
|
|
205
223
|
| Property | Required | Description | Values |
|
|
@@ -263,6 +281,9 @@ npx @forzalabs/remora synth c_quality_gaps --records 500 --seed my-seed
|
|
|
263
281
|
|
|
264
282
|
# Run every consumer tagged with a given project
|
|
265
283
|
npx @forzalabs/remora synth --project my-project
|
|
284
|
+
|
|
285
|
+
# Write the exports somewhere other than remora/.synth
|
|
286
|
+
npx @forzalabs/remora synth c_quality_gaps --output ./tmp/fixtures
|
|
266
287
|
```
|
|
267
288
|
|
|
268
289
|
Each generated row will pair `measure_code` with its matching `measure_name`, draw `gap_status` from your flat set, and pick `gender_code` from the inline list.
|
|
@@ -1155,6 +1155,8 @@ The file is exported with a `.txt` extension, to any file destination (local, S3
|
|
|
1155
1155
|
| `position.nullFill` | ❌ | The single character a **null** field is filled with, for its full width. Defaults to a space for every type | `" "`, `"0"` |
|
|
1156
1156
|
| `position.overflow` | ❌ | What a value longer than `width` does | `"fail"` (default), `"truncate"` |
|
|
1157
1157
|
| `position.start` | ❌ | 1-based column the field is expected to start at. An assertion, not a second source of truth (see below) | `21` |
|
|
1158
|
+
| `position.scale` | ❌ | `number` fields only: implied decimal places the field is written with, the point dropped (see [How a Number Is Spelled](#how-a-number-is-spelled)) | `2` |
|
|
1159
|
+
| `position.sign` | ❌ | `number` fields only: how the field carries its sign, where it is not a plain leading `-` (see [How a Number Is Spelled](#how-a-number-is-spelled)) | `"overpunch"`, `"leading"`, `"trailing"` |
|
|
1158
1160
|
|
|
1159
1161
|
Five rules decide what lands on the line:
|
|
1160
1162
|
|
|
@@ -1172,6 +1174,29 @@ Five rules decide what lands on the line:
|
|
|
1172
1174
|
|
|
1173
1175
|
**Carry a positional field through as a `string`.** A `number` dimension drops leading zeros — `"00"` read as a number is `0` — so writing it back takes `"align": "right"` with `"pad": "0"`, a rule you have to get right field by field. Declaring the producer dimension a `string` instead carries the characters through untouched.
|
|
1174
1176
|
|
|
1177
|
+
### **How a Number Is Spelled**
|
|
1178
|
+
|
|
1179
|
+
The receivers that read by column also write their numbers by column: an amount **without its decimal point**, and a negative sign carried by a byte that also holds a digit. `position.scale` and `position.sign` are the write side of the producer dimension's `scale` and `sign` (see the Producers guide), and they mean the same things in the other direction:
|
|
1180
|
+
|
|
1181
|
+
| Property | The field is written as | `-123.45` is written |
|
|
1182
|
+
| --- | --- | --- |
|
|
1183
|
+
| `"scale": 2` | digits only, the last 2 of them the fraction | `-12345` |
|
|
1184
|
+
| `"sign": "overpunch"` | digits, the last byte carrying the sign as well as its digit | `1234N` |
|
|
1185
|
+
| `"sign": "leading"` | a `+` or `-`, then digits | `-12345` |
|
|
1186
|
+
| `"sign": "trailing"` | digits, then a `+` or `-` | `12345-` |
|
|
1187
|
+
|
|
1188
|
+
```json
|
|
1189
|
+
{ "key": "claim_amount", "position": { "width": 10, "scale": 2, "sign": "overpunch" } }
|
|
1190
|
+
```
|
|
1191
|
+
|
|
1192
|
+
`-123.45` is then written `000001234N`, and `123.45` as `000001234E`.
|
|
1193
|
+
|
|
1194
|
+
- `number` fields only, and both are optional on their own. A field writing a plain `-123.45` declares neither.
|
|
1195
|
+
- **A field declaring `sign` is a signed field, so a positive value is written in the form the mode declares** — `123A`, `+123`, `123+` — never as bare digits.
|
|
1196
|
+
- Declared on the output, not taken from the producer: the file you write is not the file you read, and a value can reach the output from anywhere. Declare the same `scale` and `sign` on both sides and a field read from a positional file is written back byte for byte.
|
|
1197
|
+
- The padding goes where the layout wants it. A `leading` sign keeps the first column and the fill goes between it and the digits (`-0001234`), which is where a COBOL `SIGN LEADING SEPARATE` field carries it.
|
|
1198
|
+
- `compile` prints both alongside the columns, since a wrong `scale` writes a plausible amount that is 10ⁿ off.
|
|
1199
|
+
|
|
1175
1200
|
### **Column Order, Gaps and `start`**
|
|
1176
1201
|
|
|
1177
1202
|
**Column order is field order, and every offset is computed from the widths before it.** No column number is ever transcribed by hand: list the fields in the order the record is written, give each one a width, and the layout falls out of them. That is the same rule the read side follows, and it is why a layout is written once rather than renumbered every time a field moves.
|
|
@@ -1194,7 +1219,7 @@ FIXED layout of consumer "claims_export"
|
|
|
1194
1219
|
start end width field
|
|
1195
1220
|
1 12 12 member_id
|
|
1196
1221
|
13 20 8 service_date
|
|
1197
|
-
21 30 10 amount
|
|
1222
|
+
21 30 10 amount (2 implied decimal(s), overpunch sign)
|
|
1198
1223
|
```
|
|
1199
1224
|
|
|
1200
1225
|
Every problem with a layout is reported in one `compile` rather than one per run, so six fields missing a width come back as six lines. `run` and `verify` resolve the same layout before they read a producer, so a layout that does not add up fails in front of the run instead of after the whole dataset has been built.
|
|
@@ -179,8 +179,11 @@ Notes:
|
|
|
179
179
|
- Values are **trimmed**, so space- and zero-padded fields (`"0000123.45"`, `"ABC "`) read as their real value. A field that is entirely padding reads as `null` — or as its type's empty value, if the dimension declares `nullable: false`.
|
|
180
180
|
- Gaps are fine: filler columns you don't map simply have no dimension. Overlaps are fine too — a composite field and its subfields can both be declared.
|
|
181
181
|
- Combine `position` with `format` to parse dates written as bare digits, as in `service_date` above. Day-of-year ("Julian") dates use `DDD`: `"format": "YYYYDDD"` reads `2007305` as 1 November 2007.
|
|
182
|
+
- An amount written **without its decimal point** — `0012345` meaning `123.45`, the way a COBOL `PIC S9(7)V99` field is written — needs `scale`. See [Implied decimals](#implied-decimals).
|
|
183
|
+
- An amount whose **sign is written in the last byte** — `123J` meaning `-1231`, the way a signed COBOL `DISPLAY` field is — needs `sign`. See [Written signs](#written-signs).
|
|
182
184
|
- A dimension marked `sourceFilename` needs no `position`: it isn't in the line.
|
|
183
|
-
-
|
|
185
|
+
- A feed unloaded from a mainframe has to be converted before any of this reads — ASCII, display digits, one record per line. See [Mainframe unloads](#mainframe-unloads).
|
|
186
|
+
- `discover` cannot be used on a `FIXED` producer — a positional layout can only come from the file's spec. Use `sample` to check that your positions line up. If the spec is a COBOL copybook, [`import-copybook`](#importing-a-cobol-copybook) writes the whole producer from it.
|
|
184
187
|
|
|
185
188
|
### **Repeating Groups (OCCURS)**
|
|
186
189
|
|
|
@@ -213,7 +216,7 @@ The group expands into one dimension per (occurrence × child) when the project
|
|
|
213
216
|
| `position` | ✅ | Extent of **one** occurrence: `start` of the first, `length` per occurrence | `{ "start": 32, "length": 30 }` |
|
|
214
217
|
| `occurs` | ❌ | Number of occurrences. Default `1`, max `1000`. An object when the count varies per line — see [Variable Occurrences](#variable-occurrences-and-anchoring) | `3` |
|
|
215
218
|
| `indexFrom` | ❌ | First index used in generated names: `0` (default) or `1` | `1` |
|
|
216
|
-
| `naming` | ❌ | Name template. Tokens `{group}`, `{index}`, `{field}`, `{Field}` (capitalized). `{index}` is required unless `occurs` is `1`. Default `{group}{index}{Field}` | `{group}_{index}_{field}` → `addr_0_street` |
|
|
219
|
+
| `naming` | ❌ | Name template. Tokens `{group}`, `{index}`, `{field}`, `{Field}` (capitalized), plus `{index1}`…`{indexN}` inside a nested group. `{index}` is required unless `occurs` is `1`. Default `{group}{index}{Field}` | `{group}_{index}_{field}` → `addr_0_street` |
|
|
217
220
|
|
|
218
221
|
Notes:
|
|
219
222
|
|
|
@@ -222,7 +225,41 @@ Notes:
|
|
|
222
225
|
- `occurs: 1` is useful on its own: a group is then simply a way to **structure** a segment of the layout, and with `"naming": "{field}"` the children keep their own names.
|
|
223
226
|
- Unpopulated occurrences read as `null`, exactly like any all-padding field.
|
|
224
227
|
- A child position that doesn't fit inside one occurrence is rejected — that is the classic transcription slip, an absolute offset pasted where a relative one belongs.
|
|
225
|
-
-
|
|
228
|
+
- Groups **nest**, the way a copybook's `OCCURS` clauses do — see [Nested groups](#nested-groups).
|
|
229
|
+
|
|
230
|
+
### **Nested Groups**
|
|
231
|
+
|
|
232
|
+
A copybook nests `OCCURS` clauses routinely — an address segment that repeats, each occurrence holding two phone slots that repeat in turn. A group's child may itself be a group, and the same two rules apply one level down: its `position` is the extent of **one** of its own occurrences, measured **relative to one occurrence of the group it sits in**.
|
|
233
|
+
|
|
234
|
+
```json
|
|
235
|
+
{
|
|
236
|
+
"name": "addr",
|
|
237
|
+
"position": { "start": 1, "length": 30 },
|
|
238
|
+
"occurs": 2,
|
|
239
|
+
"dimensions": [
|
|
240
|
+
{ "name": "street", "type": "string", "position": { "start": 1, "length": 10 } },
|
|
241
|
+
{
|
|
242
|
+
"name": "phone",
|
|
243
|
+
"position": { "start": 11, "length": 10 },
|
|
244
|
+
"occurs": 2,
|
|
245
|
+
"dimensions": [
|
|
246
|
+
{ "name": "area", "type": "string", "position": { "start": 1, "length": 3 } },
|
|
247
|
+
{ "name": "num", "type": "string", "position": { "start": 4, "length": 7 } }
|
|
248
|
+
]
|
|
249
|
+
}
|
|
250
|
+
]
|
|
251
|
+
}
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Each level names what the level below it generated, so the default template gives `addr0Street`, `addr0Phone0Area`, `addr0Phone0Num`, `addr0Phone1Area`, … `addr1Phone1Num` — ten dimensions at columns 1-10, 11-13, 14-20, 21-23, 24-30, then 31-40, 41-43, … 54-60.
|
|
255
|
+
|
|
256
|
+
Notes:
|
|
257
|
+
|
|
258
|
+
- Inside a nested group's `naming`, **`{index}` is that group's own occurrence** and `{index1}`…`{indexN}` reach N levels outward, for a template that wants the outer index somewhere of its own: `"naming": "{group}{index1}_{index}_{field}"` on `phone` above gives `phone0_1_area`.
|
|
259
|
+
- All of a nested group's occurrences must fit inside **one** occurrence of the group it sits in. `phone` above takes columns 11-30 of a 30-character address, and a position that runs past the end is rejected.
|
|
260
|
+
- `occurs` is capped at 1000 **per level**, so a group nested in another is bounded at the product.
|
|
261
|
+
- A nested group repeats a **fixed** number of times: `occurs` as an object, and `emit`, belong to the outermost group only. A per-line count can only be read along the line, not inside another group's occurrence.
|
|
262
|
+
- Everything else is unchanged: the whole tree flattens to plain positioned dimensions when the project loads, so nothing downstream knows a group was nested.
|
|
226
263
|
|
|
227
264
|
### **Occurrences as Rows**
|
|
228
265
|
|
|
@@ -375,6 +412,69 @@ A matching line shorter than `minLength` is dropped and logged at debug level (`
|
|
|
375
412
|
| `minLength` | ❌ | Shortest a valid line of this record type can be, in characters | `2400` |
|
|
376
413
|
| `onLengthMismatch` | ❌ | What to do with a matching line that is too short | `"skip"` (default), `"fail"` |
|
|
377
414
|
|
|
415
|
+
### **Mainframe Unloads**
|
|
416
|
+
|
|
417
|
+
Remora does not connect to VSAM. A VSAM file is reached through z/OS access-method calls, not as a byte stream, so there is nothing at the other end of a connection to read. What Remora reads is the **unload**: the flat file an `IDCAMS REPRO` step writes, described by the COBOL copybook of the record it holds. Producing that unload, and converting it, belongs on the mainframe side of the handover — Remora's side starts at the file that arrives.
|
|
418
|
+
|
|
419
|
+
**The unload has to arrive as ASCII display data.** That is a requirement of reading a mainframe feed, not a preference. Before the file leaves z/OS, a conversion step — the `DFSORT`/`OUTREC` job most shops already run, or the equivalent in whichever sort product is installed — has to do three things:
|
|
420
|
+
|
|
421
|
+
- **Translate the record from EBCDIC to ASCII.** Remora reads the file as text and carries no EBCDIC codepage.
|
|
422
|
+
- **Expand every `COMP-3` (packed decimal) and `COMP` (binary) field to display digits**, each at the DISPLAY width of its `PIC` clause. Packed decimal holds two digits per byte and binary holds no digits at all, so there are no characters at those columns for a positional read to address.
|
|
423
|
+
- **Write one record per line**, newline-terminated, each record padded to its full width. `RECFM=FB` writes no separator between records and `RECFM=VB` prefixes each record with a 4-byte RDW; neither of those is a line.
|
|
424
|
+
|
|
425
|
+
Nothing else about the record has to change, and nothing else should. An overpunched sign and an implied decimal point are ordinary characters once the record is ASCII, and Remora reads both as the copybook spells them — see [Implied decimals](#implied-decimals) and [Written signs](#written-signs). Ask for the layout itself to be left exactly as the copybook describes it: the columns are what the producer is written from, so a conversion that re-aligns, re-orders or trims fields makes the copybook wrong.
|
|
426
|
+
|
|
427
|
+
**Positions are counted in characters**, everywhere in a `FIXED` producer. In a converted single-byte record a character is a byte and both agree with the copybook's offsets — which is why the conversion happens before the file is sent, rather than being described in the layout.
|
|
428
|
+
|
|
429
|
+
If a feed arrives unconverted:
|
|
430
|
+
|
|
431
|
+
| What arrives | What happens |
|
|
432
|
+
| --- | --- |
|
|
433
|
+
| A record still in EBCDIC | No column reads as itself: text fields hold unrelated characters and numeric ones fail on them. Sample a first delivery before trusting it. |
|
|
434
|
+
| `RECFM=FB`, no newlines | The whole file reads as one very long line, so only the first record's columns land anywhere. |
|
|
435
|
+
| `RECFM=VB`, no newlines | The same, and each record's four RDW bytes shift every position after them. |
|
|
436
|
+
| A `COMP-3` or `COMP` field | `remora compile` fails, naming the dimension and the conversion the feed needs. [`import-copybook`](#importing-a-cobol-copybook) writes `"usage": "comp-3"` (or `"comp"`) on such a field for exactly that reason: once the conversion job expands it, delete `usage` and the producer compiles. |
|
|
437
|
+
|
|
438
|
+
`COMP-1` and `COMP-2` (floating point) are not supported, converted or not: `import-copybook` refuses a copybook that declares one rather than writing a field that cannot be read.
|
|
439
|
+
|
|
440
|
+
**What to ask for.** One request to the team that owns the job covers it — an `IDCAMS REPRO` unload of the file, converted to ASCII with `COMP-3` and `COMP` fields expanded to display digits, written as newline-delimited records of fixed length, together with the copybook that describes the record. With the copybook in hand, [`import-copybook`](#importing-a-cobol-copybook) writes the producer from it.
|
|
441
|
+
|
|
442
|
+
### **Importing a COBOL Copybook**
|
|
443
|
+
|
|
444
|
+
A fixed-width feed unloaded from a mainframe comes with a COBOL copybook, and the copybook is the only place the width, the implied decimals and the written sign of each field are stated. `import-copybook` reads it and writes the producer:
|
|
445
|
+
|
|
446
|
+
```bash
|
|
447
|
+
# Write remora/producers/p_orders.json from the copybook
|
|
448
|
+
remora import-copybook ./layouts/ORDERS.cpy -n p_orders
|
|
449
|
+
|
|
450
|
+
# Without -n, the producer is named after the copybook file ("ORDERS.cpy" -> "p_orders")
|
|
451
|
+
remora import-copybook ./layouts/ORDERS.cpy
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
The parse is deterministic, never inferred: level numbers, `PIC` clauses, `USAGE`, `OCCURS` (fixed and `DEPENDING ON`) and `REDEFINES`. **Every `position` is computed from the PIC widths**, so no offset is transcribed by hand — which is the point of the command on a layout of several hundred fields.
|
|
455
|
+
|
|
456
|
+
| Copybook | Producer |
|
|
457
|
+
| --- | --- |
|
|
458
|
+
| `PIC X(12)` | `"type": "string"`, `"position": { "length": 12 }` |
|
|
459
|
+
| `PIC 9(8)` | `"type": "number"` |
|
|
460
|
+
| `PIC S9(7)V99` | `"type": "number"`, `"scale": 2`, `"sign": "overpunch"` |
|
|
461
|
+
| `PIC S9(5) SIGN IS TRAILING SEPARATE` | `"type": "number"`, `"sign": "trailing"`, one character wider |
|
|
462
|
+
| `PIC ZZ,ZZ9.99` | `"type": "string"` — an edited picture holds its insertion characters, so the field is text |
|
|
463
|
+
| `OCCURS 3` | a [repeating group](#repeating-groups-occurs), `"indexFrom": 1` so the names match the copybook |
|
|
464
|
+
| `OCCURS 1 TO 10 DEPENDING ON ORD-COUNT` | `"occurs": { "max": 10, "min": 1, "from": "$ordCount" }`, and everything after it [anchored](#variable-occurrences-and-anchoring) |
|
|
465
|
+
| `REDEFINES` | the first reading only; the alternates are named in the dimension's `description` |
|
|
466
|
+
| `FILLER` | nothing — it carries width, not a value |
|
|
467
|
+
|
|
468
|
+
Names are the copybook's own: `ORD-TOTAL-AMT` becomes `ordTotalAmt`, and a name two groups share is qualified with the group it sits in. Groups that do not repeat are flattened, since they structure the copybook rather than the record.
|
|
469
|
+
|
|
470
|
+
Both source formats are read: a fixed-format copybook (sequence numbers in columns 1-6, an indicator in column 7, text in columns 8-72) and a free-format one. Comment lines are skipped.
|
|
471
|
+
|
|
472
|
+
**The feed still has to arrive as display characters**, as every mainframe unload does — see [Mainframe unloads](#mainframe-unloads). A `COMP-3` or `COMP` field is emitted with its `usage` recorded (`"usage": "comp-3"`) and `remora compile` rejects it by name, so the conversion job expands the field; the `usage` property is then removed and the producer compiles.
|
|
473
|
+
|
|
474
|
+
A clause the importer cannot read **fails, naming the copybook line**, rather than producing a layout that is nearly right: a field read from the wrong column holds a plausible value that no run and no validation can catch. `OCCURS DEPENDING ON` inside another repeating group, a `PIC` using `P`, `COMP-1`/`COMP-2`, level 66 `RENAMES` and a second record description in one copybook are all refused this way. Read each record type of a mixed file with its own producer and a [`recordSelector`](#mixed-record-types-hdt).
|
|
475
|
+
|
|
476
|
+
The written producer points at a placeholder source and file: set `source` and `settings.fileKey`, then run `remora compile` and `remora sample` to check the positions against a real line.
|
|
477
|
+
|
|
378
478
|
### **File Pattern Matching**
|
|
379
479
|
|
|
380
480
|
Use wildcards to process multiple files matching a pattern:
|
|
@@ -785,11 +885,14 @@ Dimensions define the structure and characteristics of data fields in your produ
|
|
|
785
885
|
| `classification` | ❌ | Data privacy classification | `["PII"]`, `["PHI"]`, `["GDPR"]` |
|
|
786
886
|
| `mask` | ❌ | Data masking method | `hash`, `mask`, `crypt`, `random`, `seeded-random`, `none` |
|
|
787
887
|
| `format` | ❌ | Source parse/cast format (e.g. a date format). `DDD` reads a day-of-year ("Julian") date | `"YYYYMMDD"`, `"DD-MM-YYYY"`, `"YYYYDDD"` |
|
|
888
|
+
| `scale` | ❌ | Implied decimal places of a `number` written without a decimal point. See [Implied decimals](#implied-decimals). | `2` |
|
|
889
|
+
| `sign` | ❌ | How a `number` written as text carries its sign. See [Written signs](#written-signs). | `overpunch`, `leading`, `trailing` |
|
|
890
|
+
| `usage` | ❌ | How the source system stores the field, as a copybook's `USAGE` clause declares it. Written by [`import-copybook`](#importing-a-cobol-copybook); `comp` and `comp-3` are rejected at compile | `display`, `comp`, `comp-3` |
|
|
788
891
|
| `sourceFilename` | ❌ | Populate this dimension with the source file name (file producers only; one per producer) | `true`, `false` (default) |
|
|
789
892
|
| `position` | ⚠️ | Where the value sits in the line, for `FIXED` (fixed-width) producers — required on every dimension of one, ignored elsewhere. See [Fixed-Width Files](#fixed-width-files-fixed). | `{ "start": 21, "length": 10 }`, `{ "start": 21, "end": 30 }` |
|
|
790
893
|
| `position.after` | ❌ | Anchors the position to the end of an earlier group's actual occurrences instead of to the start of the line. See [Variable Occurrences](#variable-occurrences-and-anchoring). | `{ "after": "reimb", "length": 18 }` |
|
|
791
|
-
| `dimensions` | ❌ | Makes this a repeating group (`FIXED` only): the fields that repeat, positioned relative to one occurrence.
|
|
792
|
-
| `occurs` | ❌ | Occurrences of a repeating group. Default `1`, max `1000
|
|
894
|
+
| `dimensions` | ❌ | Makes this a repeating group (`FIXED` only): the fields that repeat, positioned relative to one occurrence. Groups nest — see [Repeating Groups](#repeating-groups-occurs) and [Nested Groups](#nested-groups). | see Repeating Groups |
|
|
895
|
+
| `occurs` | ❌ | Occurrences of a repeating group. Default `1`, max `1000` per level; an object when the count varies per line, on the outermost group only | `3`, `{ "max": 36, "while": { ... } }` |
|
|
793
896
|
| `indexFrom` | ❌ | First index used in the names a group generates | `0` (default), `1` |
|
|
794
897
|
| `naming` | ❌ | Name template for the dimensions a group or `flags` generates | `"{group}_{index}_{field}"`, `"{field}_{flag}"` |
|
|
795
898
|
| `emit` | ❌ | Whether a group's occurrences become columns (default) or one record each. See [Occurrences as Rows](#occurrences-as-rows). | `"columns"`, `"rows"` |
|
|
@@ -900,6 +1003,58 @@ Bound the **parent** field of a foreign key, never the child: a `synth.reference
|
|
|
900
1003
|
}
|
|
901
1004
|
```
|
|
902
1005
|
|
|
1006
|
+
#### Implied decimals
|
|
1007
|
+
|
|
1008
|
+
Flat files from mainframe and other legacy systems write an amount **without its decimal point**: the field holds digits only, and the layout says how many of the trailing ones are the fraction. A COBOL `PIC S9(7)V99` amount of `123.45` is written `000012345`. Read as-is that is `12345` — a perfectly plausible amount, 100× wrong, and nothing downstream can tell.
|
|
1009
|
+
|
|
1010
|
+
`scale` says how many of the trailing digits are the fraction:
|
|
1011
|
+
|
|
1012
|
+
```json
|
|
1013
|
+
{
|
|
1014
|
+
"name": "claim_amount",
|
|
1015
|
+
"type": "number",
|
|
1016
|
+
"scale": 2,
|
|
1017
|
+
"position": { "start": 21, "length": 9 }
|
|
1018
|
+
}
|
|
1019
|
+
```
|
|
1020
|
+
|
|
1021
|
+
`000012345` then reads as `123.45`, and `000000005` as `0.05`.
|
|
1022
|
+
|
|
1023
|
+
- `number` dimensions only, and 1 or more. A field written *with* its decimal point, or with no fraction at all, omits `scale` entirely.
|
|
1024
|
+
- A field declaring `scale` holds digits and an optional leading sign, nothing else. A decimal point in the value contradicts the layout, so the run fails naming the value rather than guessing which of the two readings was meant.
|
|
1025
|
+
- Not restricted to `FIXED`: a CSV column written the same way reads the same way.
|
|
1026
|
+
- Only text is decoded. A source that already hands Remora a number (JSON, SQL) hands over the value itself, so `scale` leaves it alone.
|
|
1027
|
+
- `synth` generates values with `scale` decimal places, so synthetic data fits the column it describes. `synth.decimals` overrides that.
|
|
1028
|
+
|
|
1029
|
+
#### Written signs
|
|
1030
|
+
|
|
1031
|
+
The same files rarely write a negative amount with a leading `-`. A signed COBOL `DISPLAY` field carries its sign in the **last byte**, which also holds the last digit: `-1231` is written `123J`, and `+1231` as `123A`. Read as-is, `Number("123J")` is `NaN` — the whole column arrives empty.
|
|
1032
|
+
|
|
1033
|
+
`sign` says how the sign is written:
|
|
1034
|
+
|
|
1035
|
+
| `sign` | The field holds | `-123` is written |
|
|
1036
|
+
|---|---|---|
|
|
1037
|
+
| `overpunch` | digits, the last byte carrying the sign as well as its digit: `{` and `A`-`I` positive, `}` and `J`-`R` negative | `12L` |
|
|
1038
|
+
| `leading` | an optional `+`/`-`, then digits | `-123` |
|
|
1039
|
+
| `trailing` | digits, then an optional `+`/`-` | `123-` |
|
|
1040
|
+
|
|
1041
|
+
```json
|
|
1042
|
+
{
|
|
1043
|
+
"name": "claim_amount",
|
|
1044
|
+
"type": "number",
|
|
1045
|
+
"scale": 2,
|
|
1046
|
+
"sign": "overpunch",
|
|
1047
|
+
"position": { "start": 21, "length": 10 }
|
|
1048
|
+
}
|
|
1049
|
+
```
|
|
1050
|
+
|
|
1051
|
+
`000001234E` then reads as `123.45`, and `000001234N` as `-123.45`.
|
|
1052
|
+
|
|
1053
|
+
- `number` dimensions only, and never inferred. Without `sign`, a letter in a numeric field is a malformed record rather than a sign — so a column whose feed uses overpunched signs has to say so, and one that does not keeps failing loudly on a stray letter.
|
|
1054
|
+
- `overpunch` reads overpunched signs only. A field whose conversion writes a real `+`/`-` is `leading` or `trailing` instead; a plain digit in the last byte is the unsigned form and reads as positive.
|
|
1055
|
+
- Combines with `scale`, which is applied after the sign is taken off: `0012345-` at `"scale": 2` is `-123.45`.
|
|
1056
|
+
- Like `scale`, it decodes text only, and is not restricted to `FIXED`.
|
|
1057
|
+
|
|
903
1058
|
For synthetic data (`synth`), use `synth.decimals` to control how many decimal places generated `number` values get, so they load into the destination column. `decimals: 0` forces whole integers (fits an `integer` column); a positive value rounds to that many decimals:
|
|
904
1059
|
|
|
905
1060
|
```json
|
package/documentation/README.md
CHANGED
|
@@ -94,6 +94,7 @@ remora synth [consumer] [-r n] [-s seed] # run the pipeline on generated fake d
|
|
|
94
94
|
remora discover <producer> # read the source and fill in the dimensions for you
|
|
95
95
|
remora create-producer <name> # scaffold a producer
|
|
96
96
|
remora create-consumer <name> [-p producer] # scaffold a consumer
|
|
97
|
+
remora import-copybook <file> [-n name] # write a FIXED producer from a COBOL copybook
|
|
97
98
|
remora automap <producer> <schemas...> # map a producer onto schemas with AI
|
|
98
99
|
remora graph [-f mermaid|dot|json] # print the lineage — what feeds what
|
|
99
100
|
remora ui-preview [-p port] # browse the project and its data in a browser
|
|
@@ -225,7 +226,7 @@ Set these in the `.env` beside your `remora/` folder.
|
|
|
225
226
|
|
|
226
227
|
| Variable | |
|
|
227
228
|
|---|---|
|
|
228
|
-
| `REMORA_LICENCE_KEY` | **Required.**
|
|
229
|
+
| `REMORA_LICENCE_KEY` | **Required.** Every command needs a valid licence, except `init`, `version` and `--help` |
|
|
229
230
|
| `REMORA_RUNTIME_CONTEXT` | `cli` — already added by `init` |
|
|
230
231
|
| `OPENAI_API_KEY` | Optional. Enables the AI features (`discover`, `automap`) |
|
|
231
232
|
| `REMORA_WORKER_HOST` | Optional. Your Remora Worker URL, used by `deploy` |
|