@forzalabs/remora 2.3.0 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +93 -0
- package/docs/Remora - Code Sets.md +268 -0
- package/docs/Remora - Consumers.md +2224 -0
- package/docs/Remora - Producers.md +1658 -0
- package/docs/Remora - Sources.md +1150 -0
- package/documentation/README.md +262 -133
- package/documentation/default_resources/mock_data.csv +2 -2
- package/index.js +2891 -1955
- package/json_schemas/consumer-schema.json +154 -10
- package/json_schemas/producer-schema.json +0 -38
- package/package.json +4 -2
- package/workers/ExecutorWorker.js +1327 -403
package/documentation/README.md
CHANGED
|
@@ -1,133 +1,262 @@
|
|
|
1
|
-
#
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
1
|
+
# Your Remora project
|
|
2
|
+
|
|
3
|
+
Remora moves data: it reads from somewhere, reshapes it, and writes it somewhere else. You describe
|
|
4
|
+
all of that in JSON files — the ones in this folder — and Remora does the work. There is no pipeline
|
|
5
|
+
code to write.
|
|
6
|
+
|
|
7
|
+
This README explains how the pieces fit together and where to find the full reference. It is written
|
|
8
|
+
to be read by you *and* by an AI assistant working in this folder — see
|
|
9
|
+
[Working with an AI assistant](#working-with-an-ai-assistant).
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## How Remora works
|
|
14
|
+
|
|
15
|
+
Three resource types, in a chain:
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
SOURCE PRODUCER CONSUMER
|
|
19
|
+
where data lives what to read from it what to build from it, and where to send it
|
|
20
|
+
|
|
21
|
+
s3 bucket ──▶ a table or a file ──▶ join · filter · transform · validate ──▶ output
|
|
22
|
+
database + its columns + pivot, split, dedupe
|
|
23
|
+
HTTP API ("dimensions")
|
|
24
|
+
local folder
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
- **Source** — a connection. Credentials, host, bucket, folder, URL. Nothing about the data itself.
|
|
28
|
+
One source is normally reused by many producers.
|
|
29
|
+
- **Producer** — one dataset read from one source: which table or file, and the list of fields
|
|
30
|
+
(`dimensions`) it has. A producer is a *description of shape*, not a query you write by hand.
|
|
31
|
+
- **Consumer** — the output. It takes one or more producers, joins/unions them, picks and renames
|
|
32
|
+
fields, transforms values, filters rows, validates, and writes the result to a destination.
|
|
33
|
+
|
|
34
|
+
A **run** is always driven by a consumer: `remora run <consumer>`. Remora resolves which producers
|
|
35
|
+
it needs, reads them (in parallel, in chunks, streaming to disk rather than into memory), applies the
|
|
36
|
+
consumer's logic, and writes the output.
|
|
37
|
+
|
|
38
|
+
Two supporting resource types:
|
|
39
|
+
|
|
40
|
+
- **Schema** — a JSON Schema in `schemas/` used to validate a consumer's output shape.
|
|
41
|
+
- **Code set** — a named list of allowed values in `code_sets/`, used by `remora synth` to generate
|
|
42
|
+
realistic fake data. Affects synthetic generation only, never a real run.
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## What `init` just created
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
remora/
|
|
50
|
+
├── project.json project name, version, and run settings
|
|
51
|
+
├── README.md this file
|
|
52
|
+
├── sources/
|
|
53
|
+
│ └── s_default.json a 'local' source pointing at ./default_remora_data
|
|
54
|
+
├── producers/
|
|
55
|
+
│ └── p_default.json reads mock_data.csv from that folder, 5 dimensions
|
|
56
|
+
├── consumers/
|
|
57
|
+
│ └── c_default.json passes those 5 fields through, exports JSON
|
|
58
|
+
└── schemas/ (empty — add JSON Schemas here to validate consumer output)
|
|
59
|
+
|
|
60
|
+
default_remora_data/
|
|
61
|
+
└── mock_data.csv sample input, 5 columns
|
|
62
|
+
|
|
63
|
+
.env REMORA_RUNTIME_CONTEXT=cli was added
|
|
64
|
+
.gitignore remora/.temp was added (run scratch space — never commit it)
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The folder names above are **fixed**. Remora always looks in `sources/`, `producers/`, `consumers/`,
|
|
68
|
+
`schemas/` and `code_sets/`. You do not configure these paths (see [Deprecated](#deprecated)).
|
|
69
|
+
|
|
70
|
+
Add `code_sets/` yourself if you start using synthetic data.
|
|
71
|
+
|
|
72
|
+
### Try it
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
remora compile # validate every config file, report errors
|
|
76
|
+
remora run c_default # execute the default consumer
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
That reads the 5 rows of `mock_data.csv` and writes them back out as
|
|
80
|
+
`default_remora_data/c_default_<timestamp>_0.jsonl` — one JSON object per line. From there, point
|
|
81
|
+
`s_default` at something real, or add a new source, and build out from the defaults.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Commands
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
remora init # scaffold a project (what you just ran)
|
|
89
|
+
remora compile # validate all configs — run this after every edit
|
|
90
|
+
remora verify [producer] # test that sources actually connect
|
|
91
|
+
remora run [name] [-p project] [-l n] # execute a consumer, a project, or everything
|
|
92
|
+
remora sample <name> [-s size] # print rows from a producer or consumer as a table
|
|
93
|
+
remora synth [consumer] [-r n] [-s seed] # run the pipeline on generated fake data
|
|
94
|
+
remora discover <producer> # read the source and fill in the dimensions for you
|
|
95
|
+
remora create-producer <name> # scaffold a producer
|
|
96
|
+
remora create-consumer <name> [-p producer] # scaffold a consumer
|
|
97
|
+
remora automap <producer> <schemas...> # map a producer onto schemas with AI
|
|
98
|
+
remora graph [-f mermaid|dot|json] # print the lineage — what feeds what
|
|
99
|
+
remora ui-preview [-p port] # browse the project and its data in a browser
|
|
100
|
+
remora package [-o output] # build the deployable .zip
|
|
101
|
+
remora deploy # upload it to a Remora Worker
|
|
102
|
+
remora report [run-id] [-o output] # zip up a run for a bug report
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
The three worth knowing early: **`compile`** is your validator, **`sample`** shows you the data
|
|
106
|
+
without a full run, and **`ui-preview`** draws the whole project.
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## Configuration reference
|
|
111
|
+
|
|
112
|
+
Every resource is validated against a JSON Schema. The schemas are the **authoritative reference** —
|
|
113
|
+
they carry a description for every property, and they are what `remora compile` actually enforces:
|
|
114
|
+
|
|
115
|
+
| Resource | Schema |
|
|
116
|
+
|---|---|
|
|
117
|
+
| Project | [project-schema.json](https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/project-schema.json) |
|
|
118
|
+
| Source | [source-schema.json](https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/source-schema.json) |
|
|
119
|
+
| Producer | [producer-schema.json](https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/producer-schema.json) |
|
|
120
|
+
| Consumer | [consumer-schema.json](https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/consumer-schema.json) |
|
|
121
|
+
| Code set | [code-set-schema.json](https://raw.githubusercontent.com/ForzaLabs/remora-public/refs/heads/main/json_schemas/code-set-schema.json) |
|
|
122
|
+
|
|
123
|
+
Each generated file already has a `$schema` key pointing at its schema, so VS Code and most editors
|
|
124
|
+
give you completion, inline docs and red squiggles for free. Keep that key.
|
|
125
|
+
|
|
126
|
+
**Don't** maintain a copy of the option list anywhere. It goes stale; this README used to be one.
|
|
127
|
+
|
|
128
|
+
### The full reference, offline
|
|
129
|
+
|
|
130
|
+
The CLI ships the complete prose documentation — Sources, Producers, Consumers and Code Sets, with
|
|
131
|
+
a worked example for every option. It is installed alongside the CLI, so it always describes the
|
|
132
|
+
version you actually have:
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
ls "$(npm root -g)/@forzalabs/remora/docs" # global install
|
|
136
|
+
ls node_modules/@forzalabs/remora/docs # local install
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
The canonical schemas are in `json_schemas/` beside it, and those are the exact files
|
|
140
|
+
`remora compile` validates against.
|
|
141
|
+
|
|
142
|
+
### What's available, at a glance
|
|
143
|
+
|
|
144
|
+
So you know what to go looking for in the schemas:
|
|
145
|
+
|
|
146
|
+
- **Source `engine`** — `aws-redshift`, `postgres`, `aws-s3`, `aws-sqs`, `local`, `delta-share`,
|
|
147
|
+
`http-api`, `presigned-url`
|
|
148
|
+
- **`authentication.method`** — `iam`, `username-password`, `access-secret-key`, `arn`, `implicit`,
|
|
149
|
+
`bearer-token`, `api-key`, `oidc`, `none`
|
|
150
|
+
- **Producer `settings.fileType`** — `JSON`, `JSONL`, `CSV`, `TXT`, `XLS`, `XLSX`, `XML`, `PARQUET`,
|
|
151
|
+
`FIXED` (fixed-width/COBOL, with `position`, `occurs`, `flags` on the dimensions).
|
|
152
|
+
`compressionType`: `GZ`, `ZIP`, `TAR`
|
|
153
|
+
- **Dimension `type`** — `string`, `number`, `datetime`, `boolean`. Plus `pk`, `nullable`, `format`,
|
|
154
|
+
`alias`, `classification` (`PII`/`PHI`/`GDPR`) and `mask` (`hash`, `mask`, `crypt`, `random`,
|
|
155
|
+
`seeded-random`, `none`, or your own string / `{ENV_VAR}`)
|
|
156
|
+
- **Consumer field transformations** — 28 of them, chained in order on `fields[].transform`:
|
|
157
|
+
`cast`, `concat`, `combine_fields`, `conditional`, `switch_case`, `code_lookup`, `date_format`,
|
|
158
|
+
`format`, `substring`, `split`, `extract`, `regex_extract`, `regex_match`, `regex_replace`,
|
|
159
|
+
`trim`, `pad_start`, `pad_end`, `prepend`, `append`, `to_uppercase`, `to_lowercase`, `capitalize`,
|
|
160
|
+
`mask`, `add`, `addBy`, `multiply`, `multiplyBy`, `bit_extract`
|
|
161
|
+
- **Filter operators** — `equals`, `notEquals`, `contains`, `notContains`, `startsWith`, `endsWith`,
|
|
162
|
+
`greaterThan`, `greaterThanOrEquals`, `lessThan`, `lessThanOrEquals`, `in`, `notIn`, `between`,
|
|
163
|
+
`notBetween`, `isNull`, `isNotNull`, `true`, `false`, `matches`, `notMatches`
|
|
164
|
+
- **Output `format`** — `API`, `CSV`, `FIXED`, `JSON`, `PARQUET`, `TXT`. **Exactly one output per
|
|
165
|
+
consumer.** `FIXED` writes a positional flat file and needs a `position` (`width`, `align`,
|
|
166
|
+
`pad`, `nullFill`, `overflow`) on every visible field; the output takes `recordLength`,
|
|
167
|
+
`defaults`, `lineEnding` (`lf`/`crlf`) and `layouts` for several record types in one file
|
|
168
|
+
- **Output `trigger.type`** — `CRON`, `API`, `QUEUE`. Plus `onSuccess` / `onError` hooks
|
|
169
|
+
- **Consumer `options`** — `distinct`, `distinctOn` (with a `first`/`last`/`min`/`max` resolution
|
|
170
|
+
strategy), `pivot` (`sum`/`count`/`avg`/`min`/`max`)
|
|
171
|
+
- **Validation** — `validate[]` on a field or on the whole consumer, with `onFail`: `fail`, `warn`,
|
|
172
|
+
`skip`, `set_default`
|
|
173
|
+
- **Splitting** — `split.by` / `split.maxFileSizeGb`, and `maximumFileSize`
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Working with an AI assistant
|
|
178
|
+
|
|
179
|
+
Remora configs are a good fit for Claude Code, Cursor, ChatGPT and friends — they're plain JSON with
|
|
180
|
+
a published schema and a fast validator. A few things make that work much better.
|
|
181
|
+
|
|
182
|
+
**1. Point it at the installed docs, not its memory.** The prose reference and the canonical schemas
|
|
183
|
+
both ship with the CLI (see [The full reference, offline](#the-full-reference-offline)). Tell the
|
|
184
|
+
assistant to read them off disk before it writes or edits a config — no network, and guaranteed to
|
|
185
|
+
match the version you have. The schemas carry a description for every property (~350 across the
|
|
186
|
+
five), so they answer most questions on their own.
|
|
187
|
+
|
|
188
|
+
**2. Make `remora compile` the feedback loop.** It validates every file against the schema and prints
|
|
189
|
+
what's wrong and where. An assistant that runs `compile` after each edit will converge on a working
|
|
190
|
+
config on its own; one that doesn't will hand you something plausible and broken. Then
|
|
191
|
+
`remora sample <name>` to check the data is actually right, and `remora verify` for connectivity.
|
|
192
|
+
|
|
193
|
+
**3. Watch for outdated config.** Remora's older public documentation is still indexed, so an
|
|
194
|
+
assistant may confidently produce options that no longer exist. These are the ones that come up:
|
|
195
|
+
|
|
196
|
+
| If you see | It's wrong — use |
|
|
197
|
+
|---|---|
|
|
198
|
+
| `"engine": "aws-dynamodb"` | Removed. Not a valid engine — pick one from the list above |
|
|
199
|
+
| `"operator": "eq"` / `"neq"` / `"gt"` / `"gte"` / `"lt"` / `"lte"` | `equals`, `notEquals`, `greaterThan`, `greaterThanOrEquals`, `lessThan`, `lessThanOrEquals` |
|
|
200
|
+
| `"operator": "not_contains"` | `notContains` |
|
|
201
|
+
| `authentication.bucketName` | `authentication.bucket` |
|
|
202
|
+
| `outputs[].accellerated` | Removed. Delete it |
|
|
203
|
+
| `outputs[].direct` | Removed. Delete it |
|
|
204
|
+
| `producers[].custom.language` / `.code` | Removed. Use `fields[].transform` |
|
|
205
|
+
| `fields[].grouping` | Removed |
|
|
206
|
+
| Two or more entries in `outputs` | Exactly one is allowed |
|
|
207
|
+
| `"format": "SQL"` as an output | Removed |
|
|
208
|
+
| `project.json` listing `consumers`/`producers`/`sources`/`schemas` paths | Ignored — see [Deprecated](#deprecated) |
|
|
209
|
+
| `decimals`, `allowedValues`, `codeSet`, `references` directly on a dimension | Moved under `dimension.synth` |
|
|
210
|
+
|
|
211
|
+
`remora compile` catches all of these. Run it.
|
|
212
|
+
|
|
213
|
+
**4. Point it at the right files.** `remora graph -f json` prints the whole project's lineage as JSON
|
|
214
|
+
— a compact way to tell an assistant what exists and what depends on what, without it reading every
|
|
215
|
+
file.
|
|
216
|
+
|
|
217
|
+
**5. Keep secrets out of the configs.** Reference environment variables instead of pasting
|
|
218
|
+
credentials into JSON an assistant will read back to you. `mask` accepts `{ENV_VAR}` notation too.
|
|
219
|
+
|
|
220
|
+
---
|
|
221
|
+
|
|
222
|
+
## Environment variables
|
|
223
|
+
|
|
224
|
+
Set these in the `.env` beside your `remora/` folder.
|
|
225
|
+
|
|
226
|
+
| Variable | |
|
|
227
|
+
|---|---|
|
|
228
|
+
| `REMORA_LICENCE_KEY` | **Required.** No command runs without a valid licence |
|
|
229
|
+
| `REMORA_RUNTIME_CONTEXT` | `cli` — already added by `init` |
|
|
230
|
+
| `OPENAI_API_KEY` | Optional. Enables the AI features (`discover`, `automap`) |
|
|
231
|
+
| `REMORA_WORKER_HOST` | Optional. Your Remora Worker URL, used by `deploy` |
|
|
232
|
+
| `REMORA_MAX_WORKERS` | Optional. Caps worker threads. Can only *lower* the computed count |
|
|
233
|
+
| `REMORA_DEBUG_MODE` | Optional. Rotating file logs under `remora/.temp/logs` |
|
|
234
|
+
| `REMORA_LOG_FORMAT` | Optional. `text` or `json` console output |
|
|
235
|
+
|
|
236
|
+
Plus whatever your sources need — AWS credentials, Delta Share tokens, database passwords.
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## Deprecated
|
|
241
|
+
|
|
242
|
+
`project.json` used to name the resource folders:
|
|
243
|
+
|
|
244
|
+
```json
|
|
245
|
+
"consumers": ["/consumers"],
|
|
246
|
+
"producers": ["/producers"],
|
|
247
|
+
"sources": ["/sources"],
|
|
248
|
+
"schemas": ["/schemas"],
|
|
249
|
+
"codeSets": ["/code_sets"]
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
These are **ignored**. The folder layout is fixed. `compile` warns if they're present — delete them.
|
|
253
|
+
|
|
254
|
+
---
|
|
255
|
+
|
|
256
|
+
## Getting unstuck
|
|
257
|
+
|
|
258
|
+
1. `remora compile` — almost every problem is a config problem, and this names it.
|
|
259
|
+
2. `remora verify` — if compile passes but a run can't read anything, it's connectivity.
|
|
260
|
+
3. `remora sample <producer>` — see what Remora actually reads, before any consumer logic.
|
|
261
|
+
4. `remora synth <consumer> -r 100` — exercise the whole pipeline on fake data.
|
|
262
|
+
5. `remora report` — zips the last run (versions, redacted config, logs) for a bug report.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
id,name,age,country,spend
|
|
2
|
-
USR-00001,Alice,28,Italy,0
|
|
2
|
+
USR-00001,Alice,28,Italy,0.48
|
|
3
3
|
USR-00002,Bob,35,Portugal,191.75
|
|
4
4
|
USR-00003,Carla,36,Netherlands,23.5
|
|
5
5
|
USR-00004,Diego,54,France,35.25
|
|
6
|
-
USR-00005,Elisa,62,Germany,47
|
|
6
|
+
USR-00005,Elisa,62,Germany,47.52
|