@elaraai/east-codex-plugin 0.0.0-stage → 1.0.89

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,841 @@
1
+ ---
2
+ name: e3
3
+ description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
4
+ ---
5
+
6
+ # East Execution Engine (e3)
7
+
8
+ e3 runs East programs as a durable dataflow over content-addressed data: every
9
+ value is an object named by its hash, every task execution is cached by its
10
+ inputs, and a change re-runs only what it reaches. It is the platform's
11
+ **Compute** layer — and East + e3 solutions are decision-oriented: a dataflow
12
+ exists to put auditable evidence behind a business decision, not to move data
13
+ for its own sake.
14
+
15
+ ## Before writing code — search the example index
16
+
17
+ Every East API has a tested example in the plugin's index — the index IS the
18
+ API reference, printed from each example's IR in TypeScript or python. Before
19
+ writing or changing East code:
20
+
21
+ 1. Call `search_east_examples` for each capability you
22
+ are about to use — `language: "python"` for east-py, `"typescript"`
23
+ otherwise. Summaries come back first: id, signature, the inputs and the
24
+ expected result, a few hundred bytes each.
25
+ 2. Fetch the one or two that match with `get_east_example`
26
+ and pattern your code on them.
27
+ 3. Do not read `node_modules/@elaraai/**` or `*.examples.ts` files wholesale,
28
+ and do not reason from `.d.ts` signatures: the index holds the same
29
+ programs, exact and far cheaper, and the signatures omit the runtime rules
30
+ that make East code correct.
31
+
32
+ Nothing is injected for you; the search is the step.
33
+
34
+ ## Quick Start
35
+
36
+ ```typescript
37
+ // src/index.ts
38
+ import { East, StringType, variant } from '@elaraai/east';
39
+ import e3 from '@elaraai/e3';
40
+
41
+ const name = e3.input('name', StringType, variant('value', 'World'));
42
+ const greet = e3.task('greet', [name],
43
+ East.function([StringType], StringType, ($, n) => East.str`Hello, ${n}!`));
44
+
45
+ const pkg = e3.package('hello', '1.0.0', greet); // dependencies are collected
46
+ await e3.export(pkg, '/tmp/hello.zip');
47
+ export default pkg;
48
+ ```
49
+
50
+ ```bash
51
+ e3 repo create .
52
+ e3 workspace deploy . dev --from-source ./src/index.ts # or --from-zip /tmp/hello.zip
53
+ e3 dataflow run . dev
54
+ e3 dataset get . dev.greet # flat path: <ws>.<name>
55
+ ```
56
+
57
+ ## Decision Tree
58
+
59
+ ```
60
+ What do you need?
61
+ ├─ Author a package (@elaraai/e3)
62
+ │ ├─ An input dataset → e3.input(name, type, source?) source: variant('value', v) | variant('file', path)
63
+ │ ├─ A task — two questions: does the output fit in memory, can the work be split over an input? (see "Which task?")
64
+ │ │ ├─ Fits, built in one pass → e3.task(name, [inputs], fn, config?) — returns the output
65
+ │ │ ├─ Too large, or built as you go → e3.streamTask(name, { inputs, output }, fn) — emits into e3.output.array | set | dict | fold
66
+ │ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
67
+ │ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
68
+ │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
69
+ │ ├─ Audited state → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
70
+ │ ├─ A record's type changes → e3.migration.value | rows | rekey(name, record, fn, { after? })
71
+ │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
72
+ │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
73
+ │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
74
+ ├─ Operate it from the shell → the e3 CLI (below): deploy, run, read and write datasets, mutate records, gc
75
+ └─ Drive it from code
76
+ ├─ A server, over HTTP → @elaraai/e3-api-client
77
+ ├─ A local repository → @elaraai/e3-core
78
+ ├─ Serve repositories → e3-api-server --repos <dir>, or createServer()
79
+ └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 ("Running e3 in a browser")
80
+ ```
81
+
82
+ ## Authoring a package
83
+
84
+ ### Inputs — `e3.input(name, type, source?)`
85
+
86
+ A dataset at `.inputs.<name>`, addressed `<ws>.<name>` from the CLI. The third
87
+ argument is a source variant — a bare value is refused:
88
+
89
+ | Source | Meaning |
90
+ |---|---|
91
+ | `variant('value', v)` | Inline, carried in the package. |
92
+ | `variant('file', path)` | A beast2 file on the machine that **deploys**. The package carries the path (relative paths resolve against the working directory at export); deploy takes the file in as the value. |
93
+ | omitted | Unassigned until something sets it. |
94
+
95
+ ```typescript
96
+ const name = e3.input('name', StringType, variant('value', 'default'));
97
+ const count = e3.input('count', IntegerType);
98
+ const table = e3.input('table', ArrayType(RowType), variant('file', './deliveries/TABLE.beast2'));
99
+ ```
100
+
101
+ An inline value is a small constant written in the source. Rows computed on
102
+ the host as the package is built — a `Map` filled in a loop, a generated
103
+ array — would ride the package as a snapshot, so a dataset that is generated
104
+ or large is made where data is made: a task's output (an `e3.task` over a
105
+ small authored count, as e3-ui's paged examples generate their rows), a
106
+ `file` source, or a record.
107
+
108
+ A `file` source:
109
+ - is checked against the declared type at `e3.export` (drift is a build error
110
+ naming the input and the first differing field) and again at deploy, before
111
+ the workspace is touched;
112
+ - is hashed first, so a delivery the store already holds costs that one read;
113
+ - is taken in without being read whole or modified — so replace a delivery with
114
+ a new file, never edit it in place. A collection is taken in on the runners —
115
+ east-c, or east-node where there is no east-c — in pieces of its segments, as
116
+ many at once as the budget allows: each row is walked, the bytes the Writer
117
+ writes for it kept as they stand and any other row written again, so the
118
+ delivery is stored as the Writer's value whichever writer wrote it, and a new
119
+ delivery stores only the segments that changed. A deploy or an upload
120
+ stopped part way takes up again from the pieces it finished. Any other value
121
+ is taken in by reflink, hard link or one copy;
122
+ - may hold no segment of more than 64 MiB, the most a collection is read in at
123
+ once. A value encoded whole is one such segment, and so can be a segment of
124
+ wide rows from an older Writer, which bounded a segment by its element count
125
+ alone; either is refused, naming the fix: write it again with a current
126
+ Writer, whose segments stay under 8 MiB;
127
+ - is read on the machine that runs `e3 workspace deploy`, local repository or
128
+ not: against a server the CLI checks every delivery first, then streams each
129
+ after the deploy. A deploy made through the API leaves it unset;
130
+ `--skip-file-sources` defers it and prints the
131
+ `e3 dataset set <repo> <ws>.<name> --from-file <path>` that completes each.
132
+
133
+ Every door into a dataset — `dataset set`, the API, the transfer, a file
134
+ adopt — refuses bytes whose type is not **exactly** the declared type, since
135
+ runners decode by it.
136
+
137
+ ### Tasks — `e3.task(name, inputs, fn, config?)`
138
+
139
+ Runs an East function once, as one unit, and returns its output at
140
+ `.tasks.<name>.output`; chain tasks with `task.output`. A collection it returns
141
+ is written segment by segment, but the body builds it whole: an output too large
142
+ to hold is emitted instead (`e3.streamTask`). `config` is
143
+ `{ runner?, environment?, role? }` — `role` is what e3-ui's `ui()` sets.
144
+
145
+ Task inputs, on every runtime (a mutation's state included):
146
+ - are **deeply frozen**: mutating one raises `cannot mutate a frozen value (task
147
+ inputs are immutable) — copy first`; derive a changed value from `.copy()`;
148
+ - open **lazily**, whatever their size: size, iteration and keyed gets read a
149
+ segment at a time, with the same semantics, and an operation that needs the
150
+ whole value decodes it once, when it first does. A runner's `decode: 'whole'`
151
+ decodes every input before the program runs instead — for a program whose
152
+ reads land at random across more segments than the runner keeps, which `-v`
153
+ names in the task's log. Only elements carrying a `Ref` or a function decode
154
+ whole either way.
155
+
156
+ Data that is not a dataset opens the same way inside a body —
157
+ `FileSystem.openBeast(T, path)` for a beast2 file on the runner's disk,
158
+ `blob.openBeast(T)` for bytes in hand — but the dataflow does not watch it:
159
+ anything a task should react to is an input.
160
+
161
+ | `config.runner` | Runs on |
162
+ |---|---|
163
+ | omitted | `{ runtime: 'east-node', platforms: ['@elaraai/east-node-std'] }` — every project has Node |
164
+ | `{ runtime: 'east-py', platforms: ['east-py-std', 'east-py-io', 'east-py-datascience'] }` | Python |
165
+ | `{ runtime: 'east-c', platforms: ['east-c-std'] }` | native, the lowest overhead |
166
+ | `{ runtime: 'custom', command: ['uv', 'run', 'east-py', 'run', '-p', 'east-py-std'] }` | any command: e3 appends `run`'s arguments (`-i <input>` each, `-o <output>`, the program's file) |
167
+
168
+ Platform names are typed per runtime — a typo, or another runtime's platform, is
169
+ a compile error; `{ custom: '<name>' }` names your own. A stock runtime is handed
170
+ one unit file (`exec <unit>`), and takes `decode: 'lazy' | 'whole'` (`'lazy'`
171
+ when omitted): how its units read their inputs — the unit's `decode`, as every
172
+ runner's `run --decode` takes it. It is the runner's, so a task, each piece of a
173
+ split task, a function, a mutation, a migration and an index build all read
174
+ their inputs as their runner says, and changing it re-runs the task.
175
+
176
+ ```typescript
177
+ const greet = e3.task('greet', [name],
178
+ East.function([StringType], StringType, ($, n) => East.str`Hello, ${n}!`));
179
+ const shout = e3.task('shout', [greet.output],
180
+ East.function([StringType], StringType, ($, s) => s.upperCase()),
181
+ { runner: { runtime: 'east-c', platforms: ['east-c-std'] } });
182
+ ```
183
+
184
+ #### Your own code: platform functions, imported functions, environments
185
+
186
+ - **A project-owned platform function** (native code East cannot express): list
187
+ `{ custom: '<name>' }` in the runner's platforms. For east-node the name is
188
+ your package's **scoped name** (its `./platform` export, a
189
+ `PlatformFunction[]`); for east-py, the Python **module** (its top-level
190
+ `platform` list). The function's declared name, `"<project>.<fn>"`, must
191
+ byte-match its implementation. Authoring and wiring: the **east-project**,
192
+ **east** (`East.platform(...).implement(...)`) and **east-py**
193
+ (`@platform_function`) skills.
194
+ - **An East function written in python, or in another node package**:
195
+ `East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
196
+ resolves it and embeds its IR, so the task is pure IR on any runner, its
197
+ locations kept so an error or a profile names the function's own lines. A uv
198
+ workspace member is found by name in `uv.lock`, and its root module's
199
+ `east_functions` dict is exported with `east-py export-functions` (from the
200
+ project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
201
+ `package.json` name, and its built `./functions` export's `eastFunctions` is
202
+ exported with `east-node export-functions` (`EAST_NODE`). A package built
203
+ elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
204
+ which wins. The declared type must equal the exported one exactly, and each
205
+ platform function it calls must be provided by the task's runner, by name or
206
+ by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
207
+ `east-py-io` ≡ `@elaraai/east-node-io`).
208
+ - **Environments** are derived at export from each `{ custom }` platform: the
209
+ providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
210
+ or `package.json` + lockfile + `npm pack`) is captured into the package and
211
+ materialized per repository, so the task runs where your tree never was. With
212
+ platform code split into packages (`create-e3 --python-packages=a,b`), editing
213
+ one re-runs only its tasks. An explicit `environment` overrides the
214
+ derivation: `{ python: { project } }`, `{ node: { project } }`,
215
+ `{ tools: { files: [...] } }` (prebuilt binaries on PATH, such as a C runner)
216
+ or `{ image: { digest: 'repo@sha256:<64 hex>' } }` (cloud; a tag is refused).
217
+
218
+ ```python
219
+ # packages/pricing/src/pricing/__init__.py — the member's root module
220
+ score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * r.price)
221
+ east_functions = {"score": score}
222
+ ```
223
+
224
+ ```typescript
225
+ const score = East.importFunction('pricing', 'score', FunctionType([RowType], FloatType));
226
+ const total = e3.task('total', [rows],
227
+ East.function([ArrayType(RowType)], FloatType, ($, rs) => rs.map(($, r) => score(r)).sum()));
228
+ ```
229
+
230
+ ### Stream tasks — `e3.streamTask(name, spec, fn)`
231
+
232
+ A stream task **emits** its output instead of returning it, so its memory does
233
+ not grow with its output: emission order is free, and the runner sorts a set or
234
+ a dict in bounded runs and merges them. `spec` is `inputs` (datasets, in the
235
+ body's parameter order, some wrapped in `e3.partition`), `output` (an output
236
+ kind), `runner` (a stock runtime; `custom` is refused) and `environment`. The
237
+ body takes the inputs, then `emit`, and returns nothing. With no partitioned
238
+ input it is one unit with exact left-to-right semantics; a producer has no
239
+ inputs and emits what it reads from platform functions.
240
+
241
+ | Output kind | `emit` | The parts combine by |
242
+ |---|---|---|
243
+ | `e3.output.array(T)` | `emit(t)` | concatenation: emission order within a unit, units in input order |
244
+ | `e3.output.set(T)` | `emit(t)` | union: an element emitted twice is held once |
245
+ | `e3.output.dict(K, V, { merge? })` | `emit(k, v)` | key: a key emitted more than once folds with `merge($, key, a, b)`, in input order; without `merge` it fails the task, naming the key |
246
+ | `e3.output.fold(T, { zero, combine })` | `emit(t)` | every value, folded with `combine($, a, b)` from `zero`, in input order |
247
+
248
+ ```typescript
249
+ const balances = e3.streamTask('balances', {
250
+ inputs: [events], // no partitioned input: one unit, in order
251
+ output: e3.output.array(FloatType),
252
+ }, ($, events, emit) => {
253
+ const balance = $.let(0.0);
254
+ $.for(events, ($, event) => {
255
+ $.assign(balance, balance.add(event.amount));
256
+ $(emit(balance));
257
+ });
258
+ });
259
+
260
+ const ingest = e3.streamTask('ingest', {
261
+ inputs: [], // a producer: emits in any order
262
+ output: e3.output.dict(StringType, RowType, {
263
+ merge: ($, _id, a, b) => East.greater(a.at, b.at).ifElse(($) => a, ($) => b), // the latest row wins
264
+ }),
265
+ }, ($, emit) => { /* page through a platform source, $(emit(row.id, row)) each */ });
266
+ ```
267
+
268
+ ### Splitting the work — `e3.partition(dataset, { by? })`
269
+
270
+ Wrap a stream task's input in `e3.partition` and the body runs once per
271
+ **piece** of it: each piece is a unit of its own — cached on its own, run in
272
+ parallel under the budget — typed as the whole dataset (a key range of a Set or
273
+ Dict, a position range of an Array), and the pieces' outputs combine as the
274
+ output kind says. Pieces are cut by content, most holding 64 to 100 MiB of
275
+ stored bytes, so an append or an insertion re-runs only the pieces it reaches.
276
+ An input not wrapped reaches every piece whole, opened lazily as every input is
277
+ (a keyed get reads only its segments); a change to it re-runs every piece.
278
+
279
+ ```typescript
280
+ // sales: Dict<{ sku, id }, { amount, currency }>. A re-key: a key emitted in
281
+ // several pieces folds with `merge`. `rates` reaches every piece whole.
282
+ const bySku = e3.streamTask('by_sku', {
283
+ inputs: [e3.partition(sales), rates],
284
+ output: e3.output.dict(StringType, FloatType, { merge: ($, _sku, a, b) => a.add(b) }),
285
+ }, ($, sales, rates, emit) => {
286
+ $.for(sales, ($, sale, key) => {
287
+ $(emit(key.sku, sale.amount.multiply(rates.get(sale.currency))));
288
+ });
289
+ });
290
+
291
+ // Huge → small: a fold.
292
+ const revenue = e3.streamTask('revenue', {
293
+ inputs: [e3.partition(sales)],
294
+ output: e3.output.fold(FloatType, { zero: 0.0, combine: ($, a, b) => a.add(b) }),
295
+ }, ($, sales, emit) => {
296
+ $.for(sales, ($, sale) => { $(emit(sale.amount)); });
297
+ });
298
+
299
+ // Per entity, in order. postings: Dict<{ account, at, id }, Float>. Rows whose
300
+ // `by` fields are equal are never split across pieces, and a piece iterates in
301
+ // key order, so each account's rows arrive together, oldest first.
302
+ const statements = e3.streamTask('statements', {
303
+ inputs: [e3.partition(postings, { by: ['account'] })],
304
+ output: e3.output.array(StatementLineType), // { account, at, balance }
305
+ }, ($, postings, emit) => {
306
+ const account = $.let('');
307
+ const balance = $.let(0.0);
308
+ $.for(postings, ($, amount, key) => {
309
+ $.if(East.notEqual(key.account, account), ($) => { // the next account starts from zero
310
+ $.assign(account, key.account);
311
+ $.assign(balance, 0.0);
312
+ });
313
+ $.assign(balance, balance.add(amount));
314
+ $(emit({ account: key.account, at: key.at, balance }));
315
+ });
316
+ });
317
+
318
+ // Reconcile two same-keyed datasets: both partitioned, so cut at the same keys.
319
+ const change = e3.streamTask('change', {
320
+ inputs: [e3.partition(today), e3.partition(yesterday)], // both Dict<SaleKey, Float>
321
+ output: e3.output.dict(SaleKeyType, FloatType),
322
+ }, ($, today, yesterday, emit) => {
323
+ $.for(today, ($, amount, key) => {
324
+ $(emit(key, amount.subtract(yesterday.get(key, ($, _key) => 0.0))));
325
+ });
326
+ });
327
+ ```
328
+
329
+ - `by` names leading key fields, in order: `['account']`, or
330
+ `['account', 'at.day']`, whose last entry reads the first field of `at`.
331
+ - Two or more partitioned inputs are cut at the same keys, so they must be Sets
332
+ or Dicts whose keys, or `by` fields, have the same types.
333
+ - **The author's contract, the only one:** `merge` and `combine` are
334
+ associative, `zero` is an identity of `combine`, and a partitioned body's
335
+ combined result does not depend on where its input was cut.
336
+ - Refused when the task is defined, naming it: `e3.partition` on an `e3.task`
337
+ input; a partitioned input that is not a collection; `by` on an Array, or
338
+ naming anything but leading key fields; co-partitioned inputs with no common
339
+ key.
340
+
341
+ #### Which task?
342
+
343
+ | Workload | Use |
344
+ |---|---|
345
+ | The output fits in memory, computed in one pass | `e3.task` |
346
+ | Global sequential state: running balances, replay, simulation | `e3.streamTask` with no partitioned input |
347
+ | Ingest from external sources | `e3.streamTask` with no inputs |
348
+ | Row-local derive, clean or validate over a huge input | `e3.partition` it; emit each row kept (`e3.output.array`, or a `dict` keyed as the input) |
349
+ | Enrich against a small reference, or sparse keyed reads of another huge dataset | `e3.partition` the big input; pass the other unwrapped |
350
+ | Aggregate huge → small (KPIs, counts, top-k) | `e3.partition` + `e3.output.fold` |
351
+ | Per-key totals, latest per key, a re-key (shuffle) | `e3.partition` + `e3.output.dict` with `merge` (a Set: `e3.output.set`) |
352
+ | Per entity: sequential within it, parallel across entities | `e3.partition(dataset, { by: [<leading key fields>] })` |
353
+ | Reconcile two same-keyed huge datasets | two `e3.partition` inputs |
354
+ | ML training, or work East cannot express | a platform function, or `e3.customTask` |
355
+
356
+ ### Custom tasks, functions, packages
357
+
358
+ | Definition | What it is |
359
+ |---|---|
360
+ | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
361
+ | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
362
+ | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
363
+ | `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
364
+
365
+ ```typescript
366
+ const process = e3.customTask('process', [rawData], StringType,
367
+ ($, inputPaths, outputPath) => East.str`python script.py -i ${inputPaths.get(0n)} -o ${outputPath}`);
368
+ const add = e3.function('add',
369
+ East.function([IntegerType, IntegerType], IntegerType, ($, a, b) => a.add(b)));
370
+ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
371
+ ```
372
+
373
+ ### Records — audited state, written only through mutations
374
+
375
+ `e3.record(name, type, initialValue)` is a dataset at `.records.<name>` that
376
+ tasks read and react to like any input, but that only its **mutations** write:
377
+ a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
378
+ body and no platform call, since the compare-and-swap retry runs it again
379
+ against fresher state — named by an identifier. Each commit records its parent,
380
+ the state, the mutation, its arguments, the actor and the time. Deploy mints a
381
+ `$init` commit from the initial value, and a redeploy keeps state and history.
382
+ The record's version vector carries the commit hash, so even a mutation that
383
+ leaves the state unchanged triggers downstream tasks.
384
+
385
+ | Form | Body | Use it when |
386
+ |---|---|---|
387
+ | `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
388
+ | `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
389
+ | `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
390
+
391
+ Every form commits a delta addressed by key, rewriting only the segments it
392
+ touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
393
+ takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
394
+ value a row already holds changes nothing, and a `delete` or `update` of a key
395
+ the record does not hold is a conflict naming the key: the caller's view of the
396
+ record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
397
+
398
+ ```typescript
399
+ const PlanType = StructType({
400
+ title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
401
+ });
402
+ const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
403
+
404
+ const addPlan = e3.mutation.reduce('add_plan', plans,
405
+ East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
406
+ const next = $.let(state.copy()); // the state is frozen: copy first
407
+ $(next.insert(id, plan));
408
+ return next;
409
+ }));
410
+
411
+ const reschedule = e3.mutation.edit('reschedule', plans,
412
+ East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
413
+ ($, state, id, due, edit) => {
414
+ const plan = $.let(state.get(id)); // decodes one segment, not the record
415
+ $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
416
+ }));
417
+ ```
418
+
419
+ Costs count in segments: a Dict or Set record is cut by key and by each entry's
420
+ encoded size, so wide rows get segments of a few rows and a one-row edit stays
421
+ cheap. A view reads every field of the rows it shows, so keep a bulky payload it
422
+ does not display in a second record keyed the same way.
423
+
424
+ #### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
425
+
426
+ A second collection over a Dict record, in another sort order, maintained in the
427
+ same commit, so a view by an attribute is a page rather than a scan. `on`
428
+ names row fields in index order: one field uses its value as the key, several
429
+ use a Struct in that order, and a Set field gives an entry per element. It
430
+ also carries field metadata for `Data.bindPaged(record, { indexes })` and
431
+ `Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
432
+ `keys` returns a Set, one entry per element (a row
433
+ naming five resources appears five times, an empty set not at all); `value` is a
434
+ projection a view renders without touching the record. The functions are pure
435
+ and synchronous, since an index maintained commit by commit must equal its
436
+ rebuild to the byte. The name is an identifier, never `primary` (the record's own
437
+ order). Read through one with `index=<name>` on a page or key search; `e3
438
+ reindex` rebuilds. A deploy builds an index the package declares and the state
439
+ lacks, drops one it no longer declares, and runs nothing for the rest; a build
440
+ is a split task over the record.
441
+
442
+ ```typescript
443
+ const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
444
+ const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
445
+ const byOwner = e3.recordIndex('by_owner', plans, {
446
+ key: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
447
+ value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
448
+ });
449
+ const byResource = e3.recordIndex('by_resource', plans, {
450
+ keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
451
+ });
452
+ const pkg = e3.package('planning', '1.0.0',
453
+ plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
454
+ ```
455
+
456
+ For events that must remain visible while they overlap a selected time window,
457
+ declare `interval: { from: 'start', to: 'end' }`. Both fields must be DateTime
458
+ or Option<DateTime>. Optional `on` fields are exact leading choices:
459
+
460
+ ```typescript
461
+ const byTime = e3.recordIndex('by_time', events, {
462
+ interval: { from: 'start', to: 'end' },
463
+ });
464
+ const byStatusTime = e3.recordIndex('by_status_time', events, {
465
+ on: ['status'], interval: { from: 'start', to: 'end' },
466
+ });
467
+ ```
468
+
469
+ An interval index uses the ordinary index build, mutation and paging machinery.
470
+ It stores at most three entries per event per leading-field combination,
471
+ independent of duration. The record Slice adapter resolves a half-open
472
+ `[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
473
+ each matching original record occurs once, including events that started
474
+ earlier or span the whole window. It never scans the record or guesses a
475
+ maximum duration. An event ending at `from` is outside; an instant at `from`
476
+ is inside and one at `to` is outside. A missing or non-positive end is an
477
+ instant. A missing start remains selectable as Unset/backlog.
478
+
479
+ Use `on: ['start']` only when the intended query is **starts within**, since it
480
+ cannot preserve carry-in events. When both declarations are available, a Slice
481
+ range on the start field uses the interval declaration. The generated interval
482
+ keys are query machinery; record edits and sessions always target the original
483
+ primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
484
+ requires a matching interval declaration and the primary `entries` binding to
485
+ confirm saves even after an edit leaves the selected window.
486
+
487
+ #### Migrations — how a record's type changes
488
+
489
+ | Form | Function | Runs as |
490
+ |---|---|---|
491
+ | `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
492
+ | `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
493
+ | `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
494
+
495
+ A record's steps are one chain: each names the step before it with `after` (a
496
+ step of the same record, leaving it as this step's input type), passing the last
497
+ step passes the chain, and the last step leaves the record as its declared type.
498
+ A step's name is an identifier, unique on the record, and a workspace records the
499
+ steps it has applied by name — so moving a step's code, or re-exporting it with a
500
+ newer e3, does not run it again, and an edit to an applied step's body is not
501
+ detected. The functions are pure, synchronous East.
502
+
503
+ ```typescript
504
+ const RowV1Type = StructType({ title: StringType });
505
+ const RowV2Type = StructType({ title: StringType, owner: StringType });
506
+ const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
507
+
508
+ const addOwner = e3.migration.rows('add_owner', plans,
509
+ East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
510
+ // The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
511
+ const pkg = e3.package('planning', '2.0.0', plans, addOwner);
512
+ ```
513
+
514
+ What a deploy does with each record, decided before it writes anything:
515
+
516
+ | The workspace holds | Against the package's chain | The deploy |
517
+ |---|---|---|
518
+ | no state | — | mints `$init`; the whole chain counts as applied |
519
+ | state | every step applied, the type unchanged | keeps it, with a `$deploy` commit when the package changed |
520
+ | state | a proper prefix applied | runs the rest, a `$migrate:<name>` commit each |
521
+ | state | every step applied, but the type changed | refused: a type change needs a migration, or `--schema reset` |
522
+ | state | applied steps the chain does not start with | refused, naming the steps applied and those declared |
523
+ | a record the package no longer declares | — | refused unless `--allow-drop-records` |
524
+
525
+ `--schema <policy>` on `e3 workspace deploy` and `e3 watch`: `migrate` (the
526
+ default), `fail` (run no migration; refuse) or `reset` (reset the record to its
527
+ initial value, with a `$reset` commit). `--plan` prints what the deploy would do
528
+ to each record, index and input and writes nothing — from a zip or a source it
529
+ imports nothing, and a server plans only a package it holds. The steps run
530
+ before the deploy writes: a failed step leaves the workspace as it was, and a
531
+ deploy run again is served the steps that finished from the cache.
532
+
533
+ What a deploy does with each input — a dataset an `e3.input` declares, which
534
+ people set:
535
+
536
+ | The input now | Its type | The deploy |
537
+ |---|---|---|
538
+ | holds the value the deployed package gave it, or is new | — | the new package's value (`package`) |
539
+ | someone set it | — | the new package's value under `--inputs reset`, the default, and says it reset it (`reset`) |
540
+ | someone set it | the new package's | kept, under `--inputs keep-edited` (`keep`) |
541
+ | someone set it | changed | the new package's value, and says why (`reset`) |
542
+ | the deployed package took it from a file, and it holds a value | — | the new package's value under either policy, since a value set since cannot be told from the file's, and says so (`reset`) |
543
+ | the new package takes it from a file | — | its file, as every deploy does (`file`); a deploy that reads no file, a server's, leaves it unset for its client to send |
544
+ | the new package no longer declares it | — | gone, with its value (`drop`) |
545
+
546
+ ## The e3 CLI
547
+
548
+ Every `<repo>` is a local path or a server URL, `http(s)://<host>/repos/<name>`;
549
+ where `[repo]` is optional it defaults to `$E3_REPO`, then `.`. Dataset paths are
550
+ flat, `<ws>.<name>` — an input or a task output, with "did you mean" suggestions;
551
+ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
552
+
553
+ | Command | Does |
554
+ |---|---|
555
+ | **Repositories** | |
556
+ | `e3 repo create [repo] [--exist-ok]` | Create one (`--exist-ok`: succeed if it exists). |
557
+ | `e3 repo status [repo]` · `e3 repo remove [repo] [-r]` | Show objects, packages, workspaces · remove (`-r`: its workspaces first). |
558
+ | `e3 repo gc [repo] [--dry-run] [--keep-runs <n>] [--keep-days <d>] [--min-age <ms>]` | Drop old run history, then what nothing names (below). |
559
+ | `e3 repo list <server>` | The repositories a server holds. |
560
+ | **Packages** | |
561
+ | `e3 package import [repo] <zip>` · `export [repo] <pkg[@ver]> <zip>` | Import · export (`--quiet`: errors only). |
562
+ | `e3 package list [repo]` · `remove [repo] <pkg[@ver]>` | |
563
+ | **Workspaces** | |
564
+ | `e3 workspace create [repo] <ws>` · `list [repo]` · `status [repo] <ws>` · `remove [repo] <ws>` | `status` shows tasks, datasets and locks. |
565
+ | `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
566
+ | `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
567
+ | `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
568
+ | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
569
+ | `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
570
+ | **Datasets** | |
571
+ | `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
572
+ | `e3 dataset set [repo] <ws.name> <file> [--type <spec> \| --type-file <path>]` | Write from `.east`, `.beast2`, `.json` or `.csv` (JSON and CSV need the type). |
573
+ | `e3 dataset set [repo] <ws.name> --from-file <path.beast2> [-j <n>] [--memory <size>]` | Take a beast2 file in as the value, as a deploy takes a `file` source in, saying how far it has got; `-j`, `--memory`: the budget its intake units run under (local). |
574
+ | `e3 dataset list [repo] <ws> [-l]` · `status [repo] <ws.name>` · `find [repo] <ws> <pattern>` | Paths (`-l`: kind, type, status, size) · one dataset · by substring or glob. |
575
+ | **Running** | |
576
+ | `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. |
577
+ | `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
578
+ | `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
579
+ | `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
580
+ | `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. |
581
+ | **Records** (`-w <ws>` required) | |
582
+ | `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
583
+ | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target; local). |
584
+ | `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
585
+ | **Development** | |
586
+ | `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
587
+ | `e3 convert [input] [--from <f>] [--to <f>] [--type <spec>] [-o <out>]` | Convert between `.east`, `.json` and `.beast2`. |
588
+ | `e3 completion install [--shell <s>]` · `uninstall` · `bash` \| `zsh` \| `fish` | Shell completion. |
589
+ | **Servers** | |
590
+ | `e3 auth login <server> [--no-browser]` · `logout <server>` · `status` · `token <server>` · `whoami [server]` | OAuth2 device flow; credentials per server (`token` prints one for curl). |
591
+
592
+ **The budget.** `-j`/`--jobs <n>` is the cores — runner processes in flight, a
593
+ task or a unit each — and `--memory <size>` (`8G`, `512M`) what they may reserve
594
+ between them. The defaults are `E3_JOBS` and `E3_MEMORY`, else what e3 may use:
595
+ its CPU affinity capped by a cgroup quota, and the cgroup's `memory.max` or
596
+ physical memory, less a reserve. A unit of a split task reserves the largest
597
+ peak a unit of its stage has reached in the run, so a stage runs its first unit
598
+ alone and then fans out; on Linux and macOS a guard stops the newest unit when
599
+ the runners together pass the budget, and runs it again once it fits. A deploy
600
+ takes its `file` sources in `-j` at a time, their pieces' intake units running
601
+ under the same budget. The commands that run East — `workspace deploy`,
602
+ `dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `reindex`
603
+ and `watch` — take the flags for a local repository; against a server the CLI
604
+ refuses them, since the server runs the work under its own budget
605
+ (`e3-api-server -j`, `--memory`).
606
+
607
+ **A deploy says how far it has got**, on stderr. Each `file` source prints a
608
+ line once it is in: its size, time and rate, and how it was taken in
609
+ (`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
610
+ value that is not a collection), with why the first time a runner fell back to
611
+ another. A terminal also keeps a live line for the files in flight, with their
612
+ pieces, the rate and the time left across them. Against a server, the deploy line
613
+ says what the job is doing, such as migrating a record or building an index;
614
+ the deliveries then upload a few at a time, each printing a line once it is in,
615
+ and the live line says what the server's commit is doing with each in flight.
616
+ While a deploy runs, its lock carries the same progress for `e3-ui` or any
617
+ client to read (`workspaceLockStatus`). A workspace deployed for the first time
618
+ has no status until its deploy ends, so the lock is the only place to see it.
619
+
620
+ **`-v`** passes `-v` to the runners, which print to the task's logs, identically
621
+ on every runtime, how each input was read — opened lazily, and what reading it
622
+ came to, or decoded whole, and the resident memory that added — and a timing
623
+ and peak-memory block. It never changes hashes or caching (add `--force` to see
624
+ it for a cached task), and works against a server.
625
+ **Where a task's time goes.** `EAST_PROFILE=1` in the environment of a local
626
+ `e3 dataflow run` (or `run`, `call`, `mutate`) has the east-c and east-py
627
+ runners profile each unit — the runners inherit e3's environment — and the
628
+ task's log (`e3 task logs`) then ends with every East function and platform
629
+ function it called, by self time, with its calls and where it was written and
630
+ called: the python lines of a function `East.importFunction` linked in
631
+ included. Each input read lazily is an entry too, named as `-v` names it
632
+ (`input 1`): the time its reads spent decoding segments, a call each — keyed
633
+ reads at random beyond the segments the runner keeps, or a scan repeated,
634
+ which `-v`'s account counts — and not the reading function's own time.
635
+ `EAST_PROFILE_INTERVAL=300` adds the profile so far every five minutes while a
636
+ long task runs. east-node does not profile. Like `-v` it changes no hash: add
637
+ `--force` for a cached task.
638
+ **`E3_SCRATCH_DIR`** moves a local run's per-execution scratch directories
639
+ (default `<repo>/tmp/scratch`, on the object store's disk; on tmpfs, outputs sit
640
+ in memory until stored).
641
+
642
+ **`repo gc`** keeps each workspace's last 10 runs (`--keep-runs`), every run
643
+ from the last 7 days (`--keep-days`) and the run its current state came from,
644
+ with every execution those runs used; every execution a workspace's current
645
+ state is served from, so a re-run stays cached; every execution from the last 7
646
+ days; and whatever is still running. An attempt recorded running that the local
647
+ runner says cannot finish is recorded interrupted, and goes as any that ended.
648
+ It then removes the objects nothing names, and staging files older than
649
+ `--min-age` (60 s). `--dry-run` reports what would go.
650
+
651
+ A package zip names the release of e3 that exported it, and an import refuses a
652
+ zip a newer release exported, naming that release: import it with an e3 at least
653
+ as new as the SDK that exported it. A dataset upload names the transfer protocol
654
+ its e3 speaks, and a server refuses one that speaks another, naming both
655
+ releases and which of them to upgrade.
656
+
657
+ ## Driving e3 from code
658
+
659
+ **`@elaraai/e3-api-client`** talks to a server over HTTP. Every call is
660
+ `(url, repo, …, options)`, `options` being
661
+ `{ token: string | null, retry?, verbose?, fetch? }` (`e3 auth token <server>`
662
+ prints a token); `fetch` is the `fetch` every request of the call goes
663
+ through, the global one unless given — an in-page e3's `e3.fetch` (below).
664
+ Dataset paths are `TreePath`s — an input definition's `.path`, for one. A
665
+ request a workspace's lock refuses throws `ApiError` `workspace_locked`,
666
+ naming the holder: its pid, boot id, command and when it took the lock.
667
+
668
+ | Area | Functions |
669
+ |---|---|
670
+ | Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
671
+ | Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
672
+ | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
673
+ | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
674
+ | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
675
+ | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
676
+ | Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
677
+ | From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
678
+
679
+ **`@elaraai/e3-core`** does the same on a local repository, given a storage
680
+ backend (`new LocalStorage()`) and the repository's path; work that runs East
681
+ takes a runner, `new LocalTaskRunner(repo)`.
682
+
683
+ | Area | Functions |
684
+ |---|---|
685
+ | Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
686
+ | Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
687
+ | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
688
+ | Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named) |
689
+ | Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
690
+ | Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex` |
691
+
692
+ `@elaraai/e3-core/test` exports the contract suites another storage backend runs
693
+ over itself.
694
+
695
+ **`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
696
+ repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
697
+ `-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
698
+ every run and call it serves and every upload it takes in), and auth — `--oidc`, a built-in provider for `e3
699
+ auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
700
+ issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
701
+ `createServer({ reposDir | singleRepoPath, port, host, … })` starts one, and its
702
+ route factories mount on another host.
703
+
704
+ ## Running e3 in a browser — `@elaraai/e3-web`
705
+
706
+ e3-web runs e3 entirely in a page, with no server: a repository kept in the
707
+ browser's own storage (records in IndexedDB, objects in OPFS, locks through
708
+ Web Locks), every East program on a pool of Web Workers, and e3's whole API
709
+ answered in the page by e3-api-server's own routes. An app is three scripts:
710
+
711
+ ```typescript
712
+ // unit.worker.ts — where East programs run: east-web-std answers for east-node-std's
713
+ // platform functions, and e3's own (e3-api-client's Platform) reach the e3 worker
714
+ import { serveUnits } from '@elaraai/e3-web/units';
715
+ serveUnits();
716
+ ```
717
+
718
+ ```typescript
719
+ // e3.worker.ts — e3 itself: its storage, orchestrator, runner and API
720
+ import { serveE3 } from '@elaraai/e3-web/worker';
721
+ serveE3({
722
+ units: () => new Worker(new URL('./unit.worker.ts', import.meta.url), { type: 'module' }),
723
+ });
724
+ ```
725
+
726
+ ```typescript
727
+ // main.ts — the page: e3-api-client's calls, every one answered by the e3 worker
728
+ import { createWebE3 } from '@elaraai/e3-web';
729
+ import {
730
+ dataflowExecute, packageImport, repoCreate, repoList, workspaceCreate, workspaceDeploy,
731
+ } from '@elaraai/e3-api-client';
732
+
733
+ const e3 = await createWebE3(new Worker(new URL('./e3.worker.ts', import.meta.url), { type: 'module' }));
734
+ const opts = { token: null, fetch: e3.fetch };
735
+
736
+ // The first visit makes the repository; later visits find it where they left it.
737
+ if (!(await repoList(e3.apiUrl, opts)).includes('default')) {
738
+ await repoCreate(e3.apiUrl, 'default', opts);
739
+ // The package zip e3.export wrote at build time, served beside the page
740
+ const zip = new Uint8Array(await (await fetch('/sales-planning-1.0.0.zip')).arrayBuffer());
741
+ const { name, version } = await packageImport(e3.apiUrl, 'default', zip, opts);
742
+ await workspaceCreate(e3.apiUrl, 'default', 'main', opts);
743
+ await workspaceDeploy(e3.apiUrl, 'default', 'main', `${name}@${version}`, opts);
744
+ await dataflowExecute(e3.apiUrl, 'default', 'main', {}, opts);
745
+ }
746
+ ```
747
+
748
+ `e3.apiUrl` is `https://e3-web.invalid`, an origin reserved so that a request
749
+ sent without `e3.fetch` fails rather than reaching the network, and
750
+ `e3.fetch`, the global `fetch`'s signature, is answered by the e3 worker. Both
751
+ go to e3-api-client (`RequestOptions.fetch`) and to e3-ui-components'
752
+ `<E3Provider>` (`E3Config.fetch`: the **e3-ui** skill). An app's own platform
753
+ package is served beside the standard ones:
754
+ `serveUnits({ platforms: { '@acme/pricing': PricingPlatform } })`.
755
+
756
+ | `serveE3` option | Default | Effect |
757
+ |---|---|---|
758
+ | `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
759
+ | `persist` | `true` | `false` keeps repositories in memory, gone with the page |
760
+ | `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
761
+ | `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
762
+ | `access` | `oneShotAccessByRoles()` with `identify`, else `any` | the grant a caller holds for a one-shot, a split call, or a function call naming its own runner |
763
+ | `wholeIntakeLimit` | 256 MiB | the largest delivery one intake unit takes in whole |
764
+
765
+ What works is the API, as the local server answers it: every API suite of
766
+ e3-api-tests passes against it in Chromium, as an admin and, in its reader
767
+ cases, as a reader. Paged reads and key search, function calls, one-shot and
768
+ split calls, record mutations, deploys that migrate records or build indexes,
769
+ uploads, package import and export, gc, and dataflow runs — tasks in parallel
770
+ up to the pool's width (the cores), split tasks in pieces, re-run reactively.
771
+ A repository outlives a reload, and two tabs share an origin's repositories as
772
+ two processes share a directory. A closed tab's locks are free; a run it left
773
+ is taken as one whose process died, the next run served what finished from
774
+ the cache; and a job or an upload's commit it left is recorded failed, never
775
+ run again.
776
+
777
+ What a browser cannot do — each fails naming it, never a partial answer:
778
+
779
+ - **Commands.** A custom task, or a task on the `custom` runtime, is recorded
780
+ `error`: "a browser runs no commands".
781
+ - **Platform functions with no browser meaning.** east-web-std has no
782
+ FileSystem, Env or large-JSON reader, and east-node-io and the python
783
+ packages have no browser equivalent: a unit calling one fails naming the
784
+ function, and one listing a package its worker does not serve fails naming
785
+ the package.
786
+ - **Segments on demand.** A unit's inputs are staged whole into its worker:
787
+ waiting for a segment needs a cross-origin isolated page, which GitHub Pages
788
+ cannot give. Split a large input into pieces with `e3.partition`, which
789
+ bounds a unit's memory anywhere; a delivery that cannot be cut into pieces
790
+ is refused above `wholeIntakeLimit`, naming the fix.
791
+ - **A memory budget.** Memory is not measured: the pool is as wide as the
792
+ cores, there is no `--memory`, and no record names a peak.
793
+ - **The CLI.** `e3` drives e3 over HTTP, which an in-page e3 does not serve.
794
+
795
+ ## A local repository
796
+
797
+ ```
798
+ repo/
799
+ ├── repository.beast2 # the release that last wrote it, and the upgrades it has had
800
+ ├── repository-upgrade.beast2 # while an upgrade is under way: where its last part stopped
801
+ ├── metadata.beast2 # its name and status
802
+ ├── objects/ # content-addressed: values, segments, manifests, programs
803
+ ├── packages/ # package refs
804
+ ├── workspaces/ # each workspace's state, dataset refs, and its latest run's state and events
805
+ ├── dataflows/ # run records
806
+ ├── executions/ # execution attempts: status, owner, logs
807
+ ├── running/ # an index of the attempts recorded running, by task: what a status reads
808
+ ├── runs/ # an index of each task's runs, its units left out: what its history and logs read
809
+ ├── adoptions/ # the manifest each delivered file, or piece of one, became
810
+ ├── locks/ # locks, their holders, and how far each says it has got
811
+ ├── gc/ # gc beside running work: unreachable notes, a stepped run's parts
812
+ ├── envs/ # built execution environments
813
+ └── tmp/ # scratch and staged uploads
814
+ ```
815
+
816
+ An e3 opening a repository an older release wrote applies the upgrades it has
817
+ not had, in place; one a newer e3 upgraded is refused, naming that release.
818
+
819
+ **Caching.** An execution is keyed by its task — program, runner and
820
+ environment — and its inputs' hashes, so a task re-runs only when one of those
821
+ changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
822
+ it names: a task whose code e3 cannot see changed, such as a platform
823
+ function's, or one reading data from outside e3). A change to an input re-runs its
824
+ consumers, even during a run. A split task is cached a unit at a time: each piece
825
+ and each merge is its own execution, and pieces are cut by content, so an append
826
+ or an insertion re-runs only the pieces it reaches and the merges above them.
827
+
828
+ ## Related skills
829
+
830
+ - **east** — the language task bodies are written in.
831
+ - **e3-ui** — UI tasks (`ui()`) and decision surfaces bound to workspace
832
+ datasets, and rendering them in an app (`<E3Provider>`), over a server or an
833
+ e3 in the page; **east-ui** — their components; **e3-ui-cli** — the terminal
834
+ UI.
835
+ - **e3-create** — scaffold a project (`npm create @elaraai/e3`);
836
+ **east-project** — its build, deploy, run and test lifecycle.
837
+ - **east-py** and **east-py-datascience** — python platform functions and ML
838
+ tasks; **east-node-std** and **east-node-io** — Node platform functions for
839
+ files, HTTP, databases and storage.
840
+ - **east-design** and **east-ontology** — plan the dataflow, and model the
841
+ business, before building.