@elaraai/east-codex-plugin 1.0.99 → 1.0.100

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/.build/mcp/server.js +3 -2
  2. package/.codex-plugin/plugin.json +1 -1
  3. package/index.json +12207 -216
  4. package/package.json +1 -1
  5. package/skills/e3/SKILL.md +29 -802
  6. package/skills/e3/reference/browser.md +93 -0
  7. package/skills/e3/reference/cli.md +120 -0
  8. package/skills/e3/reference/from-code.md +48 -0
  9. package/skills/e3/reference/rack.md +139 -0
  10. package/skills/e3/reference/record-indexes.md +326 -0
  11. package/skills/e3/reference/record-migrations.md +63 -0
  12. package/skills/e3/reference/repository.md +25 -0
  13. package/skills/east/SKILL.md +30 -459
  14. package/skills/east/reference/beast2.md +71 -0
  15. package/skills/east/reference/cross-language-and-json-schema.md +87 -0
  16. package/skills/east/reference/queries.md +46 -0
  17. package/skills/east/reference/runners-and-ir.md +148 -0
  18. package/skills/east/reference/trees-and-recursion.md +56 -0
  19. package/skills/east/reference/values-in-typescript.md +62 -0
  20. package/skills/east-py/SKILL.md +28 -1647
  21. package/skills/east-py/reference/beast2.md +108 -0
  22. package/skills/east-py/reference/cli.md +51 -0
  23. package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
  24. package/skills/east-py/reference/control-flow-and-queries.md +147 -0
  25. package/skills/east-py/reference/cross-language.md +70 -0
  26. package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
  27. package/skills/east-py/reference/patterns.md +369 -0
  28. package/skills/east-py/reference/performance.md +70 -0
  29. package/skills/east-py/reference/value-methods.md +306 -0
  30. package/skills/east-py/reference/values-construction-and-json.md +88 -0
  31. package/skills/east-py/reference/writing-east-functions.md +275 -0
  32. package/skills/east-py-datascience/SKILL.md +11 -28
@@ -33,6 +33,10 @@ writing or changing East code:
33
33
  programs, exact and far cheaper, and the signatures omit the runtime rules
34
34
  that make East code correct.
35
35
 
36
+ Tested e3 packages (inputs, tasks, stream tasks and partitions, custom tasks,
37
+ functions, records with their mutations and migrations) are in the index too:
38
+ search with `package: "e3"`.
39
+
36
40
  Nothing is injected for you; the search is the step.
37
41
 
38
42
  ## Quick Start
@@ -72,162 +76,24 @@ What do you need?
72
76
  │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
73
77
  │ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
74
78
  │ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
75
- │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) })
79
+ │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) }) (reference/record-indexes.md)
76
80
  │ │ ├─ Read by a time window or a 2D window → index.interval | index.area
77
81
  │ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
78
82
  │ │ └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
79
83
  │ │ → index.nested({ collections, tile, summary? })
80
- │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order
84
+ │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order (reference/record-migrations.md)
81
85
  │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
82
86
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
83
87
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
84
- ├─ Operate it from the shell → the e3 CLI (below): deploy, run, read and write datasets, mutate records, gc
85
- ├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack
88
+ ├─ Operate it from the shell → the e3 CLI (reference/cli.md): deploy, run, read and write datasets, mutate records, gc
89
+ ├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack (reference/rack.md)
86
90
  └─ Drive it from code
87
- ├─ A server, over HTTP → @elaraai/e3-api-client
91
+ ├─ A server, over HTTP → @elaraai/e3-api-client (reference/from-code.md)
88
92
  ├─ A local repository → @elaraai/e3-core
89
93
  ├─ Serve repositories → e3-api-server --repos <dir>, or createServer()
90
- └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 ("Running e3 in a browser")
94
+ └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 (reference/browser.md)
91
95
  ```
92
96
 
93
- ## Rack delegation from a local repository
94
-
95
- **Setup and routing**
96
-
97
- | Signature | Description | Example |
98
- | --- | --- | --- |
99
- | `e3 rack enroll --listen <host:port> [--allow <ip,...>] [--advertise-url <origin>] [--label <label>] [--no-wait]` | Starts the user-wide hub and prints the source origin, the hub's certificate SHA-256, the matching release and a single-use token for the companion's hidden prompts. Waits for a new registration, not activation. Behind an SSH reverse tunnel, listen on the loopback and advertise the relay's origin. | `e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://relay.example:7401` |
100
- | `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards. Size: serverless, small, medium, large, xlarge. | `e3 rack policy . set 'train_*' rack --size large` |
101
- | `e3 rack policy [repo] enable\|disable\|mode <opt-in\|auto>` | Enable/disable saved opt-in; `mode auto` selects eligible unmatched tasks. Defaults: disabled, opt-in. | `e3 rack policy . enable` |
102
- | `e3 rack policy [repo] defaults [--no-spill] [--claim-timeout <sec>] [--retry-task-failures-locally]` | Defaults: large, 1440-minute timeout, spill on, 30-second claim recheck, no task-failure retry. | `e3 rack policy . defaults --no-spill` |
103
- | `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
104
- | `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
105
- | `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
106
- | `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub. | `e3 rack status` |
107
- | `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
108
-
109
- The local process owns graph planning, repository and cache. After a cache
110
- miss, eligible plain task, map and merge bodies can execute on the rack; input
111
- closures and every output object transfer by hash. The session verifies the
112
- whole output graph before success. Placement never changes task/cache hashes.
113
- Local bodies and fallback share the original cores/memory budget; rack bodies
114
- use remote capacity. Calls, mutations and intake stay local. Stock runner
115
- configurations qualify; commands, custom environments/platforms and unscannable
116
- IR stay local. Host file/environment/network access stays local unless a rule
117
- deliberately grants `--allow-host-access` and its resources exist on the rack.
118
-
119
- The listener serves HTTPS with the hub's own self-signed certificate, made on
120
- first use and kept in `hub-tls.beast2`; a rack pins it by its SHA-256, which
121
- enrollment prints, so TLS runs end to end through any relay or tunnel. The
122
- usual path is an SSH reverse tunnel from the developer machine to a relay the
123
- rack reaches, with the hub on the loopback (`--allow 127.0.0.1`) and the relay's
124
- origin advertised. Removing the certificate file, with the hub stopped, makes a
125
- new one, which every worker must then pin again. `--no-tls` is for a TLS
126
- terminator in front of the hub; the hub refuses plain HTTP on a wildcard or
127
- public address, or without `--allow`. A listener on every address needs
128
- `--advertise-url` to enroll.
129
-
130
- The handoff targets a prepared independent worker VM with the matching verified
131
- `e3-rack` release. Its `configure --allocation <host-provisioned-path>` prompts
132
- for the source origin, the certificate SHA-256 for an HTTPS origin the hub
133
- serves, the worker label and settings; `enroll --allocation <path>` prompts for
134
- the one-time token. Origins, pins and credentials are encrypted together. A
135
- plain-HTTP origin needs `--allow-vpn-http` on both commands and host
136
- authorization for that exact endpoint, and belongs only on a path encrypted end
137
- to end, such as an SSH tunnel, never across a VPN hub that decrypts traffic.
138
- Never put a source address or token in worker command arguments or plaintext
139
- files. The local mint intentionally displays its token; do not log that output.
140
-
141
- Enrollment does not start services or authorize task claims. The local CLI
142
- observes a new registration (optionally filtered by label); compare it with the
143
- worker's result. Before a first heartbeat, status is `enrolled-not-started`.
144
- After managed resume, a ready heartbeat advertises free plus occupied slots;
145
- busy workers stay eligible. Fenced, draining or stopping workers withdraw their
146
- capacity and show `unavailable`. After 45 seconds without a heartbeat, readiness
147
- expires; lease polling cannot prolong it. Source boot approval, when required,
148
- remains an additional gate. This advertisement does not authorize host resource
149
- admission or removal of service/network quarantine.
150
- Resources and mutable whole-GPU assignments belong to the host lifecycle, not
151
- local enrollment flags. The companion accepts up to 32 mutually trusted sources
152
- in one worker. After draining and stopping its services and sockets, use
153
- `configure --allocation <path> --add-source`, confirm the shared trust boundary,
154
- and enter the additional origin privately. It prints the new source UUID for
155
- `enroll --allocation <path> --source <UUID>`; add `--allow-vpn-http` to both for a
156
- plain-HTTP origin. All sources must be enrolled before startup. Each has its own identity,
157
- client, cache and completion journal; actual execution shares one worker-wide
158
- task/GPU limit. Per-source capacity reports are not additive reservations.
159
- A worker is the trust boundary: a shared agent process can access all its
160
- sources' credentials, so separate clients, and a cloud deployment and developer
161
- hubs, need separate worker VMs. The hub keeps no task data; what a worker keeps
162
- between tasks is its own policy, and an agent that keeps nothing works unchanged. Worker-local `source list --allocation <path>`
163
- prints UUIDs/enrollment status; `--show-origins` adds origins only on a terminal.
164
- After drain/stop and source-side revocation, `source remove` or `source reset`
165
- with `--allocation <path> --source <UUID> --acknowledge-revocation` asks for UUID
166
- confirmation. Reset prints a new UUID for fresh enrollment; old pending results
167
- and caches stay in the retired namespace. Other sources and retained data are
168
- preserved, and host endpoint policy must be reconciled before activation. The
169
- flag acknowledges revocation; it does not revoke remotely.
170
-
171
- Worker-local `registry status --allocation <path>` reports only whether encrypted
172
- registry credentials are present; it can run while active and does not test access.
173
- After drain/stop, `registry configure --allocation <path>` privately prompts for
174
- username/token without re-entering settings. `registry clear --allocation <path>`
175
- asks for the installation UUID and removes only the stored access. Both require
176
- the same release, the writer lock and no unresolved enrollment/service transition;
177
- they preserve current identities, rotations, settings, journals and cached data.
178
- Credentials stay in the existing encrypted bundle, with no Docker login file.
179
- A later activation receives them; services remain stopped and the next boot fenced.
180
- Repeating configure with the same values or clear when absent does not write.
181
- Clearing/replacing does not revoke a token remotely or purge cached images.
182
-
183
- The companion's
184
- `reconcile-services` applies a host-provided next allocation and service budgets
185
- after a completed drain, with stopped units, the previous plan hash and exact
186
- operation/boot binding. It retains ciphertext/journals and network quarantine;
187
- it does not move a GPU, start a worker or change its executable release. For a
188
- compatible release change, invoke the target verified executable's `switch-release`
189
- with the same drain/plan bindings and unchanged resource budgets at the next
190
- allocation generation. It checks both managed-state contracts, updates only the
191
- runtime version in the current encrypted configuration, and preserves identities
192
- and journals through interruption or downgrade. Both four-file releases must stay
193
- available; a downgrade is a new operation, never a saved-state restore. Services
194
- stay stopped and quarantined. If the selected release never resumed, the exact
195
- recorded previous executable's `recover-release` binds the completed release
196
- receipt, current plan and new operation at another generation increase. It
197
- requires the original drain or a valid fenced journal with no operation at the
198
- failed generation, verifies both releases' recorded hashes, and preserves current
199
- credentials and journals. A prior resume attempt, even across reboot, refuses
200
- recovery. General journal repair remains unfinished. Named host management, source aliases, network
201
- activation, incompatible-state migrations and hardware acceptance remain
202
- companion cloud #253 work. This and durable completion retry are implemented in
203
- companion PR #254, separately from the original main protocol pin.
204
-
205
- Policy is `<repo>/rack/policy.beast2`; hub state is `~/.e3/rack` (or
206
- `E3_RACK_HOME`). A hub `dataflow run` starts exits after 15 idle minutes, and a
207
- run waits up to 10 seconds for an enrolled rack to report healthy (with none
208
- enrolled, or a policy that selects no task for a rack, it starts at once): run
209
- `e3 rack hub` as a service, or set `--idle-exit 0`, for a hub racks can always
210
- reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
211
- at most 5 GiB. The control schema currently requires protocol 2 exactly.
212
- Agents must report bundled e3 at least the project's release. Infrastructure
213
- failure falls back locally with a separate execution attempt, as does an
214
- attempt the rack stopped on its own; the run's own cancellation and the task's
215
- timeout never do. One cancelled subscriber does not stop work shared by other
216
- sessions of the same canonical repository and e3 release.
217
-
218
- **Programmatic attachment** (`@elaraai/e3-rack`)
219
-
220
- | Signature | Description | Example |
221
- | --- | --- | --- |
222
- | `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
223
- | `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
224
-
225
- The API server's optional rack package is needed only for
226
- `e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
227
- factory and do not import hub code. Custom environments, remote function
228
- delegation, automatic reconnect after hub restart and embedded UI integration
229
- are deferred.
230
-
231
97
  ## Authoring a package
232
98
 
233
99
  ### Inputs — `e3.input(name, type, source?)`
@@ -608,665 +474,12 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
608
474
  cheap. A view reads every field of the rows it shows, so keep a bulky payload it
609
475
  does not display in a second record keyed the same way.
610
476
 
611
- #### Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
612
-
613
- A second collection over a Dict record, in another sort order, maintained in the
614
- same commit, so a view by an attribute is a page rather than a scan.
615
- `index.on([...])` names row fields in index order: one field uses its value as
616
- the key, several use a Struct in that order, and a Set field gives an entry per
617
- element. It also carries field metadata for `Data.bindPaged(record, { indexes })`
618
- and `Slice.bindRecord` (see **e3-ui**). `index.key(fn)` gives one computed entry
619
- per row; `index.keys(fn)` returns a Set, one entry per element (a row naming five
620
- resources appears five times, an empty set not at all); the `value` option of
621
- each is a projection a view renders without touching the record. The functions
622
- are pure and synchronous, since an index maintained commit by commit must equal
623
- its rebuild to the byte. Its name — its key in `indexes` — is an identifier,
624
- never `primary` (the record's own order). Read through one with `index=<name>` on a page or key search; `e3
625
- reindex` rebuilds. A deploy builds an index the package declares and the state
626
- lacks, drops one it no longer declares, and runs nothing for the rest; a build
627
- is a split task over the record.
628
-
629
- ```typescript
630
- const plans = e3.record('plans', PlansType, {
631
- initial: new Map(),
632
- indexes: (index) => ({
633
- by_due: index.on(['due']),
634
- by_owner_due: index.on(['owner', 'due']),
635
- by_owner: index.key(East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner), {
636
- value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
637
- }),
638
- by_resource: index.keys(East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources)),
639
- }),
640
- mutations: (mutation) => ({ /* add_plan, reschedule, patch — above */ }),
641
- });
642
- const pkg = e3.package('planning', '1.0.0', plans); // the record whole: its indexes and mutations with it
643
- ```
644
-
645
- #### Interval and area indexes — `index.interval`, `index.area`
646
-
647
- These indexes answer more than an order — as do the series and summary indexes
648
- below — each with a builder method of its own whose keys the SDK generates.
649
- Underneath each is an ordinary index — the same build,
650
- mutation, storage, paging and `e3 reindex` — and the record Slice reads a
651
- window through either (see **e3-ui**). Their generated keys are query
652
- machinery: edits and sessions always target the record's primary key.
653
-
654
- A generated key is a struct whose own fields are e3's: `on` holds the
655
- declaration's leading fields, by their names, at their key types (a Set field's
656
- element type) — a struct with no fields when it lists none — and the kind's own
657
- parts sit beside it. So no name is reserved: a row may name a field anything,
658
- `area`, `tile` and `__area` among them, and an index may be led by it.
659
-
660
- | Index | Key | e3's parts |
661
- |---|---|---|
662
- | interval | `{ on, node, at }` | the tree node an entry is filed at; the instant it is ordered by, at the `from` field's type |
663
- | area | `{ on, area }` | the cell's Morton code times 32 plus its level; −1 for a row not yet placed |
664
- | series | `{ on, key, level, tile }` | the value of the field `key` names; the level an event is filed at, and the start of that level's tile (#1433) |
665
- | summary | `{ on, key, tile }` | the value of the field `key` names; the tile's start |
666
-
667
- A definition's `on` is the leading fields alone; its kind's options say the
668
- rest (`events.indexes.by_time.interval`, `racks.indexes.racks_by_area.area`,
669
- `jobs.indexes.jobs_by_press_week.series`).
670
-
671
- **`index.interval({ on?, from, to })`** keeps events visible while they
672
- overlap a selected time window. `from` and `to` are the row's DateTime or
673
- Option<DateTime> fields; optional `on` fields are exact leading choices:
674
-
675
- ```typescript
676
- const events = e3.record('events', EventsType, {
677
- initial: new Map(),
678
- indexes: (index) => ({
679
- by_time: index.interval({ from: 'start', to: 'end' }),
680
- by_status_time: index.interval({ on: ['status'], from: 'start', to: 'end' }),
681
- }),
682
- });
683
- ```
477
+ A record's indexes (`index.on` / `key` / `keys`, `interval`, `area`,
478
+ `series`, `summary`, `nested`) are in
479
+ [reference/record-indexes.md](reference/record-indexes.md), and its migrations
480
+ in [reference/record-migrations.md](reference/record-migrations.md).
684
481
 
685
- Its key is `{ on, node, at }`. It stores at most three entries per event per
686
- leading-field combination, independent of duration. The record Slice adapter resolves a half-open
687
- `[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
688
- each matching original record occurs once, including events that started
689
- earlier or span the whole window. It never scans the record or guesses a
690
- maximum duration. An event ending at `from` is outside; an instant at `from`
691
- is inside and one at `to` is outside. A missing or non-positive end is an
692
- instant. A missing start remains selectable as Unset/backlog.
693
-
694
- Use `on: ['start']` only when the intended query is **starts within**, since it
695
- cannot preserve carry-in events. When both declarations are available, a Slice
696
- range on the start field uses the interval declaration. The generated interval
697
- keys are query machinery; record edits and sessions always target the original
698
- primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
699
- requires a matching interval declaration and the primary `entries` binding to
700
- confirm saves even after an edit leaves the selected window.
701
-
702
- **`index.area({ on?, x, y, width?, height?, at?, cell })`** reads the rows
703
- whose box meets a 2D window — what a floor plan's viewport shows — the interval
704
- index's twin:
705
-
706
- ```typescript
707
- // RacksType: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
708
- const racks = e3.record('racks', RacksType, {
709
- initial: new Map(),
710
- indexes: (index) => ({
711
- racks_by_area: index.area({ x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
712
- racks_by_hall_area: index.area({ on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
713
- }),
714
- });
715
- ```
716
-
717
- | Option | The box's |
718
- |---|---|
719
- | `x`, `y` | place: two Float fields, or two Option<Float> fields whose `none` is a row not yet placed |
720
- | `width`, `height` | size: two Float fields, or neither for a point |
721
- | `at` | `'corner'`, the default — `x`, `y` is the box's minimum corner, as a zone's is — or `'centre'`, as an item's is |
722
- | `cell` | finest cell: its side in world units, about the smallest thing drawn — 0.01 for a floor in metres, 1 for one in millimetres. It is rounded down to a power of two, so the keys' arithmetic is exact on every runtime, and has no default: a wrong one costs a silent tenfold read |
723
- | `on` | exact leading fields, as the interval index's; a Set field fans out |
724
-
725
- Each option takes only the row's fields of its type, so a wrong one does not
726
- compile, and the definition checks them again, naming the option, the field and
727
- its type. The domain is the square ±`cell`·2²⁷ world units around zero
728
- (±1,048 km at a 0.01 cell, which rounds to 2⁻⁷), in 29 levels. Each row is
729
- filed once, under the deepest level whose cell is at least twice its box's
730
- larger side, in the cell holding its box's centre: the key
731
- `{ on, area: Integer }`. A row not yet placed files under `area` −1, which the
732
- record Slice reads as the unplaced clause; one outside the domain, not finite,
733
- or larger than half of it files under the root, which every window reads. The index **carries its rows**: each
734
- entry's projection is the whole row, so a window reads the index alone and no
735
- record segment, and it takes no `value`. The record Slice reads a viewport,
736
- grown by an eighth of its larger side on each side, as at most 266 disjoint
737
- searches per exact prefix, sent in one request, which together hold every row
738
- whose box meets it, each once — beside rows of the cells searched that it
739
- misses, which the reader draws or not.
740
-
741
- #### Series and summary indexes — `index.series`, `index.summary`
742
-
743
- What a planning board reads: the events on a block of keys — a Plan's
744
- resources — over a span of time, and, zoomed out past what a screen can draw,
745
- how busy each key is. Both file a row's event by the key it is on and by time
746
- tiles — `day`, `week` (ISO, from Monday), `month`, `quarter` or `year`, in UTC,
747
- half-open — the key before the time, so a key's entries over any span are
748
- contiguous: a series index's `{ on, key, level, tile }`, a summary index's
749
- `{ on, key, tile }`.
750
-
751
- ```typescript
752
- // JobsType: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
753
- const jobs = e3.record('jobs', JobsType, {
754
- initial: new Map(),
755
- indexes: (index) => ({
756
- jobs_by_press_week: index.series({ key: 'press', from: 'start', to: 'end', tile: 'week' }),
757
- jobs_by_press_month: index.summary({ key: 'press', from: 'start', to: 'end', tile: 'month' }),
758
- }),
759
- });
760
- ```
761
-
762
- | Option | |
763
- |---|---|
764
- | `key` | the field naming what an event is on: any field but a Set |
765
- | `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is filed nowhere; no end, or one at or before the start, is an instant |
766
- | `tile` | a series index's finest level, and a summary index's tile: `day`, `week`, `month`, `quarter` or `year` |
767
- | `on` | exact leading fields, before the key; a Set field fans out |
768
-
769
- **A series index** is an ordinary index whose keys the SDK generates, and it
770
- **carries its rows**, as the area index does. It files each event **once**, at
771
- the finest level from its `tile` up to the year whose shortest tile holds the
772
- event's length — a day, a week, 28 days, 90 days, 365 days — under that level's
773
- tile holding its start; an instant at `tile`; past 365 days, under each year it
774
- touches. An event filed at a level ends within the tile after the one holding
775
- its start, so a reader reads a block of keys over any span from the index alone
776
- as one range a key a level — that level's tiles from two before the span to one
777
- after it — in one request, whatever the span's width. **A summary index** holds,
778
- under each key and tile, `{ count, ms }`: how many rows touch the tile, and the
779
- milliseconds of it they cover — a row's shares
780
- add up to its length, and an instant's is none. Each entry stands for every row
781
- of its key and names none, so its window's `key` is Null, a read of it joins
782
- nothing, and a read that needs its rows — a joined read, a placement
783
- (`datasetPlace`) or a membership read (`datasetWithin`) — is refused, naming the
784
- index; the record's describe says it is a summary. A write
785
- changes its totals by what its rows took away and gave — its program reads the
786
- record, never the summary — and the apply adds that to each total, dropping one
787
- whose count reaches zero; the totals are Integers, so a maintained summary is the
788
- one `e3 reindex` sums, byte for byte, and a build split into pieces adds the
789
- pieces' totals of a key. A change that would leave a total no rebuild sums — a
790
- count below zero, or time with no row — fails the write, naming the total and
791
- `e3 reindex`, which rebuilds it; a total past the largest Integer fails the
792
- write or the build, naming a finer tile.
793
-
794
- A series index files an event past a year under each year it touches, as many
795
- as 4,096: one touching more is refused where it is written, naming it and its
796
- span. A summary splits an event over its tiles, as many as 4,096 — eleven years
797
- of days — and one touching more is refused the same way, naming the tile:
798
- declare a coarser tile. Each option takes only the row's fields of its type, as
799
- the area index's do, and the definition checks them again; a `value` is
800
- refused, and so is an `on` field that is the key, the start or the end: the key
801
- follows the leading fields, and the tiles are cut by the start and the end.
802
-
803
- #### Nested series indexes — `index.nested`
804
-
805
- What a planning board reads when a record's rows nest what it draws: a
806
- machine's jobs and its load a day at a time, a hall's members, an assembly's
807
- parts to any depth. One declaration a record lists every collection its rows
808
- nest, each field with a builder saying what it holds. e3 files each item apart
809
- from its row — once, by its collection, its row, its node and its time — keeps
810
- each node's totals a summary tile at a time, and each parent's subtotals at
811
- every level; and the record's other indexes carry the rows without those
812
- collections, so a page of rows reads none of them and a window of items reads
813
- no row.
814
-
815
- ```typescript
816
- // MachinesType: Dict<String, { name, site, status, jobs: Dict<String, Job>, load: Dict<DateTime, Float> }>
817
- const machines = e3.record('machines', MachinesType, {
818
- initial: new Map(),
819
- indexes: (index) => ({
820
- machines_in_time: index.nested({
821
- collections: {
822
- jobs: index.nested.items({ from: 'start', to: 'end' }), // things people drag: by id, their times fields
823
- load: index.nested.values(), // a value per instant: keyed by the instant
824
- },
825
- tile: 'week',
826
- summary: 'month',
827
- }),
828
- machines_by_site: index.on(['site']), // pages each machine without its jobs and load
829
- }),
830
- });
831
- // Each member's load, and each hall's mean of its members', at every level from the day up.
832
- const halls = e3.record('halls', HallsType, {
833
- indexes: (index) => ({
834
- halls_in_time: index.nested({
835
- collections: { members: index.nested.rows({ load: index.nested.values({ rollup: 'mean' }) }) },
836
- tile: 'day',
837
- summary: 'month',
838
- }),
839
- }),
840
- });
841
- // Parts to any depth, each part's jobs filed under every part above it too.
842
- const assemblies = e3.record('assemblies', AssembliesType, {
843
- indexes: (index) => ({
844
- assemblies_in_time: index.nested({
845
- collections: {
846
- parts: index.nested.rows({
847
- jobs: index.nested.items({ from: 'start', to: 'end' }, { rollup: true }),
848
- parts: index.nested.recursive(),
849
- }),
850
- },
851
- tile: 'week',
852
- summary: 'month',
853
- }),
854
- }),
855
- });
856
- ```
857
-
858
- | Builder (`index.nested.…`) | A field holding |
859
- |---|---|
860
- | `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
861
- | `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
862
- | `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
863
- | `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
864
-
865
- TypeScript types `collections` field by field from the record's type alone: a
866
- misnamed time, or a builder on the wrong kind of field, is an error where it is
867
- written, and its message says what fits. The definition checks the same again,
868
- in words, for a declaration TypeScript could not see. A record has one nested
869
- series index, which takes no `on` and no `value`.
870
-
871
- **What it files.** Every entry is keyed `{ collection, row, path, run, level,
872
- tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
873
- field names from the row down (`members` then `load`); the row's key; the Dict
874
- keys from the row down to the node (`[]` for the row, `m-3` for a member); the
875
- run; the level and tile; and the item. A place and a path are held as a key
876
- holds a list, folded from the last: `.step (key="members", rest=.step
877
- (key="load", rest=.end))`. Each run of a node is one contiguous range a level:
878
-
879
- | Run | Holds |
880
- |---|---|
881
- | `items` | each item once, as a series index files an event — at the level its length fits, past a year under each year — carrying it (`.item .jobs (…)`); a value, and an `{ at }` item, at the declared tile alone |
882
- | `totals` | where `summary` is declared, each node's `{ count, ms, values }` a summary tile at a time; for a Float or Integer collection, `values` is the sum, the least, the greatest and the last there, taken in key order |
883
- | `subtotals(rollup)` | for `values({ rollup })`, every parent's subtotal by each rollup at each level from `tile` to the year, where a descendant's value touches the tile: a node holding values folds its own, a parent aggregates its children's, in key order |
884
- | `subtree`, `subtreeTotals` | for `items(…, { rollup: true })`, each item under every ancestor too, `.under (path, id)`, and each ancestor's totals of them |
885
-
886
- A rollup is `'sum'`, `'mean'` (a mean of means up the tree), `'min'`, `'max'`,
887
- `'count'` (the leaf rows beneath that hold a value), a fold and an aggregate
888
- named apart (`{ fold: 'last', aggregate: 'sum' }`, a balance), or several.
889
-
890
- **Reading.** A reader searches the index's leading fields — `[collection, row,
891
- path, run, level, tile]`, each as East prints it — as a range a row a level: a
892
- block of fifty machines' jobs over three weeks is fifty times four ranges in one
893
- `datasetFindKeys` request, and one `datasetGetRuns` read of the entries they
894
- select, each job carried whole; a parent's week subtotals are one range; a
895
- part's subtree is one range a level.
896
-
897
- **The lean rows.** Beside a nested series index, every other index of the
898
- record that carries nothing or the row carries the row without the listed
899
- collections — each listed collection empty, child rows kept with theirs empty —
900
- so a page of rows read without a join reads no nested item and no segment of
901
- the record's own. An index with a `value` of its own carries its own. A
902
- binding of the record's indexes lists the nested index by name beside any that
903
- carry the lean rows — `Data.bindPaged(machines, { indexes: ['machines_by_site',
904
- 'machines_in_time'] })` (see **e3-ui**).
905
-
906
- **Its span.** Beside every nested series index e3 keeps its span, an index of
907
- the record's named `<index>_span` (e3-types' `nestedSpanName`), built with it
908
- and never declared: under each collection's place and each month its items or
909
- values touch (`NestedSpanKeyType`, `{ collection, tile }`), how many of them
910
- touch it and the milliseconds of it its items cover, summed across the rows as a
911
- summary index's totals are, and a month let go once nothing touches it. A
912
- collection's first and last month held are its span across the record — what a
913
- Plan's horizon covers for the series reading it — and its months' counts are
914
- how much it holds in each, which the horizon's strip draws as its bars (see
915
- **e3-ui**). Reading it is one search a collection, an exact match on its
916
- place, and one read of the first and last entry of each. An index the record
917
- declares under that name is refused.
918
-
919
- **Writes.** A row a write updates by a patch is maintained by the patch: e3
920
- walks it along the declaration — each collection's Dict patch, and each child
921
- rows' — and compares the entries of only the items it names, with the totals of
922
- the summary tiles they touch and the subtotals of the tiles they touch at each
923
- level. A replace on the way, which a recursive type's patch always is, and an
924
- inserted or deleted row are compared whole. Both ways give the same delta, and a maintained
925
- index is the one `e3 reindex` rebuilds, byte for byte.
926
-
927
- #### Migrations — how a record's type changes
928
-
929
- | Form | Function | Runs as |
930
- |---|---|---|
931
- | `migration.value(name, fn)` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
932
- | `migration.rows(name, fn)` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
933
- | `migration.rekey(name, fn)` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
934
-
935
- A record's steps are one chain: the list `migrations` returns, in order, each
936
- step taking the record as the step before it leaves it, and the last leaving it
937
- as its declared type — each checked where the record is declared.
938
- A step's name is an identifier, unique on the record, and a workspace records the
939
- steps it has applied by name — so moving a step's code, or re-exporting it with a
940
- newer e3, does not run it again, and an edit to an applied step's body is not
941
- detected. The functions are pure, synchronous East.
942
-
943
- ```typescript
944
- const RowV1Type = StructType({ title: StringType });
945
- const RowV2Type = StructType({ title: StringType, owner: StringType });
946
- const plans = e3.record('plans', DictType(StringType, RowV2Type), {
947
- initial: new Map(),
948
- migrations: (migration) => [
949
- migration.rows('add_owner', East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' }))),
950
- // The next version adds its step here, after this one.
951
- ],
952
- });
953
- const pkg = e3.package('planning', '2.0.0', plans);
954
- ```
955
-
956
- What a deploy does with each record, decided before it writes anything:
957
-
958
- | The workspace holds | Against the package's chain | The deploy |
959
- |---|---|---|
960
- | no state | — | mints `$init` from the initial value, the whole chain counted applied; declared with no initial value, leaves it unassigned (`unassigned`) for `e3 seed` |
961
- | state | every step applied, the type unchanged | keeps it, with a `$deploy` commit when the package changed |
962
- | state | a proper prefix applied | runs the rest, a `$migrate:<name>` commit each |
963
- | state | every step applied, but the type changed | refused: a type change needs a migration, or `--schema reset` |
964
- | state | applied steps the chain does not start with | refused, naming the steps applied and those declared |
965
- | a record the package no longer declares | — | refused unless `--allow-drop-records` |
966
-
967
- `--schema <policy>` on `e3 workspace deploy` and `e3 watch`: `migrate` (the
968
- default), `fail` (run no migration; refuse) or `reset` (reset the record to its
969
- initial value, with a `$reset` commit (`reset`); one declared with no initial
970
- value goes back to unassigned (`unassign`), for `e3 seed` again). `--plan` prints what the deploy would do
971
- to each record, index and input and writes nothing — from a zip or a source it
972
- imports nothing, and a server plans only a package it holds. The steps run
973
- before the deploy writes: a failed step leaves the workspace as it was, and a
974
- deploy run again is served the steps that finished from the cache.
975
-
976
- What a deploy does with each input — a dataset an `e3.input` declares, which
977
- people set:
978
-
979
- | The input now | Its type | The deploy |
980
- |---|---|---|
981
- | holds the value the deployed package gave it, or is new | — | the new package's value (`package`) |
982
- | someone set it | — | the new package's value under `--inputs reset`, the default, and says it reset it (`reset`) |
983
- | someone set it | the new package's | kept, under `--inputs keep-edited` (`keep`) |
984
- | someone set it | changed | the new package's value, and says why (`reset`) |
985
- | the deployed package took it from a file, and it holds a value | — | the new package's value under either policy, since a value set since cannot be told from the file's, and says so (`reset`) |
986
- | the new package takes it from a file | — | its file, as every deploy does (`file`); a deploy that reads no file, a server's, leaves it unset for its client to send |
987
- | the new package no longer declares it | — | gone, with its value (`drop`) |
988
-
989
- ## The e3 CLI
990
-
991
- Every `<repo>` is a local path or a server URL, `http(s)://<host>/repos/<name>`;
992
- where `[repo]` is optional it defaults to `$E3_REPO`, then `.`. Dataset paths are
993
- flat, `<ws>.<name>` — an input or a task output, with "did you mean" suggestions;
994
- a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
995
-
996
- | Command | Does |
997
- |---|---|
998
- | **Repositories** | |
999
- | `e3 repo create [repo] [--exist-ok]` | Create one (`--exist-ok`: succeed if it exists). |
1000
- | `e3 repo status [repo]` · `e3 repo remove [repo] [-r]` | Show objects, packages, workspaces · remove (`-r`: its workspaces first). |
1001
- | `e3 repo gc [repo] [--dry-run] [--keep-runs <n>] [--keep-days <d>] [--min-age <ms>]` | Drop old run history, then what nothing names (below). |
1002
- | `e3 repo list <server>` | The repositories a server holds. |
1003
- | **Packages** | |
1004
- | `e3 package import [repo] <zip>` · `export [repo] <pkg[@ver]> <zip>` | Import · export (`--quiet`: errors only). |
1005
- | `e3 package list [repo]` · `remove [repo] <pkg[@ver]>` | |
1006
- | **Workspaces** | |
1007
- | `e3 workspace create [repo] <ws>` · `list [repo]` · `status [repo] <ws>` · `remove [repo] <ws>` | `status` shows tasks, datasets and locks. |
1008
- | `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
1009
- | `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
1010
- | `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
1011
- | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
1012
- | `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
1013
- | **Datasets** | |
1014
- | `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
1015
- | `e3 dataset set [repo] <ws.name> <file> [--type <spec> \| --type-file <path>]` | Write from `.east`, `.beast2`, `.json` or `.csv` (JSON and CSV need the type). |
1016
- | `e3 dataset set [repo] <ws.name> --from-file <path.beast2> [-j <n>] [--memory <size>]` | Take a beast2 file in as the value, as a deploy takes a `file` source in, saying how far it has got; `-j`, `--memory`: the budget its intake units run under (local). |
1017
- | `e3 dataset list [repo] <ws> [-l]` · `status [repo] <ws.name>` · `find [repo] <ws> <pattern>` | Paths (`-l`: kind, type, status, size) · one dataset · by substring or glob. |
1018
- | **Running** | |
1019
- | `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. A task whose input nothing has set — an input never set, or a record declared with no initial value and not yet seeded — is left waiting, with its dependents (`[WAIT]`, `Waiting:`), and the run completes without them; they run once it is set. |
1020
- | `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
1021
- | `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
1022
- | `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
1023
- | `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. |
1024
- | **Records** (`-w <ws>` required) | |
1025
- | `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
1026
- | `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
1027
- | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
1028
- | `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
1029
- | **Development** | |
1030
- | `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
1031
- | `e3 convert [input] [--from <f>] [--to <f>] [--type <spec>] [-o <out>]` | Convert between `.east`, `.json` and `.beast2`. |
1032
- | `e3 completion install [--shell <s>]` · `uninstall` · `bash` \| `zsh` \| `fish` | Shell completion. |
1033
- | **Servers** | |
1034
- | `e3 auth login <server> [--no-browser]` · `logout <server>` · `status` · `token <server>` · `whoami [server]` | OAuth2 device flow; credentials per server (`token` prints one for curl). |
1035
-
1036
- **The budget.** `-j`/`--jobs <n>` is the cores — runner processes in flight, a
1037
- task or a unit each — and `--memory <size>` (`8G`, `512M`) what they may reserve
1038
- between them. The defaults are `E3_JOBS` and `E3_MEMORY`, else what e3 may use:
1039
- its CPU affinity capped by a cgroup quota, and the cgroup's `memory.max` or
1040
- physical memory, less a reserve. A unit of a split task reserves the largest
1041
- peak a unit of its stage has reached in the run, so a stage runs its first unit
1042
- alone and then fans out; on Linux and macOS a guard stops the newest unit when
1043
- the runners together pass the budget, and runs it again once it fits. A deploy
1044
- takes its `file` sources in `-j` at a time, their pieces' intake units running
1045
- under the same budget. A source deploy, or a `watch`, exports the workspace
1046
- packages its imported functions come from `-j` at a time, a process each, on
1047
- this machine; against a server, as many at once as this machine has CPUs. The
1048
- commands that run East — `workspace deploy`,
1049
- `dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
1050
- `reindex` and `watch` — take the flags for a local repository; against a server the CLI
1051
- refuses them, since the server runs the work under its own budget
1052
- (`e3-api-server -j`, `--memory`).
1053
-
1054
- **A deploy says how far it has got**, on stderr. Each `file` source prints a
1055
- line once it is in: its size, time and rate, and how it was taken in
1056
- (`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
1057
- value that is not a collection), with why the first time a runner fell back to
1058
- another. A terminal also keeps a live line for the files in flight, with their
1059
- pieces, the rate and the time left across them. A source deploy prints a line
1060
- for each workspace package whose functions it links: `exported <n> function(s)
1061
- of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
1062
- unchanged since its last export` when the export cache held them. Against a
1063
- server, the deploy line
1064
- says what the job is doing, such as migrating a record or building an index;
1065
- the deliveries then upload a few at a time, each printing a line once it is in,
1066
- and the live line says what the server's commit is doing with each in flight.
1067
- While a deploy runs, its lock carries the same progress for `e3-ui` or any
1068
- client to read (`workspaceLockStatus`). A workspace deployed for the first time
1069
- has no status until its deploy ends, so the lock is the only place to see it.
1070
-
1071
- **`-v`** passes `-v` to the runners, which print to the task's logs, identically
1072
- on every runtime, how each input was read — opened lazily, and what reading it
1073
- came to, or decoded whole, and the resident memory that added — and a timing
1074
- and peak-memory block. It never changes hashes or caching (add `--force` to see
1075
- it for a cached task), and works against a server.
1076
- **Where a task's time goes.** `EAST_PROFILE=1` in the environment of a local
1077
- `e3 dataflow run` (or `run`, `call`, `mutate`) has the east-c and east-py
1078
- runners profile each unit — the runners inherit e3's environment — and the
1079
- task's log (`e3 task logs`) then ends with every East function and platform
1080
- function it called, by self time, with its calls and where it was written and
1081
- called: the python lines of a function `East.importFunction` linked in
1082
- included. Each input read lazily is an entry too, named as `-v` names it
1083
- (`input 1`): the time its reads spent decoding segments, a call each — keyed
1084
- reads at random beyond the segments the runner keeps, or a scan repeated,
1085
- which `-v`'s account counts — and not the reading function's own time.
1086
- `EAST_PROFILE_INTERVAL=300` adds the profile so far every five minutes while a
1087
- long task runs. east-node does not profile. Like `-v` it changes no hash: add
1088
- `--force` for a cached task.
1089
- **`E3_SCRATCH_DIR`** moves a local run's per-execution scratch directories
1090
- (default `<repo>/tmp/scratch`, on the object store's disk; on tmpfs, outputs sit
1091
- in memory until stored).
1092
-
1093
- **`repo gc`** keeps each workspace's last 10 runs (`--keep-runs`), every run
1094
- from the last 7 days (`--keep-days`) and the run its current state came from,
1095
- with every execution those runs used; every execution a workspace's current
1096
- state is served from, so a re-run stays cached; every execution from the last 7
1097
- days; and whatever is still running. An attempt recorded running that the local
1098
- runner says cannot finish is recorded interrupted, and goes as any that ended.
1099
- It then removes the objects nothing names, and staging files older than
1100
- `--min-age` (60 s). `--dry-run` reports what would go.
1101
-
1102
- A package zip names the release of e3 that exported it, and an import refuses a
1103
- zip a newer release exported, naming that release: import it with an e3 at least
1104
- as new as the SDK that exported it. A dataset upload names the transfer protocol
1105
- its e3 speaks, and a server refuses one that speaks another, naming both
1106
- releases and which of them to upgrade.
1107
-
1108
- ## Driving e3 from code
1109
-
1110
- **`@elaraai/e3-api-client`** talks to a server over HTTP. Every call is
1111
- `(url, repo, …, options)`, `options` being
1112
- `{ token: string | null, retry?, verbose?, fetch? }` (`e3 auth token <server>`
1113
- prints a token); `fetch` is the `fetch` every request of the call goes
1114
- through, the global one unless given — an in-page e3's `e3.fetch` (below).
1115
- Dataset paths are `TreePath`s — an input definition's `.path`, for one. A
1116
- request a workspace's lock refuses throws `ApiError` `workspace_locked`,
1117
- naming the holder: its pid, boot id, command and when it took the lock.
1118
-
1119
- | Area | Functions |
1120
- |---|---|
1121
- | Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
1122
- | Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
1123
- | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
1124
- | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
1125
- | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
1126
- | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
1127
- | Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
1128
- | From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
1129
-
1130
- **`@elaraai/e3-core`** does the same on a local repository, given a storage
1131
- backend (`new LocalStorage()`) and the repository's path; work that runs East
1132
- takes a runner, `new LocalTaskRunner(repo)`.
1133
-
1134
- | Area | Functions |
1135
- |---|---|
1136
- | Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
1137
- | Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
1138
- | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
1139
- | Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
1140
- | Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
1141
- | Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
1142
-
1143
- `@elaraai/e3-core/test` exports the contract suites another storage backend runs
1144
- over itself.
1145
-
1146
- **`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
1147
- repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
1148
- `-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
1149
- every run and call it serves and every upload it takes in), and auth — `--oidc`, a built-in provider for `e3
1150
- auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
1151
- issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
1152
- `createServer({ reposDir | singleRepoPath, port, host, … })` starts one, and its
1153
- route factories mount on another host.
1154
-
1155
- ## Running e3 in a browser — `@elaraai/e3-web`
1156
-
1157
- e3-web runs e3 entirely in a page, with no server: a repository kept in the
1158
- browser's own storage (records in IndexedDB, objects in OPFS, locks through
1159
- Web Locks), every East program on a pool of Web Workers, and e3's whole API
1160
- answered in the page by e3-api-server's own routes. An app is three scripts:
1161
-
1162
- ```typescript
1163
- // unit.worker.ts — where East programs run: east-web-std answers for east-node-std's
1164
- // platform functions, and e3's own (e3-api-client's Platform) reach the e3 worker
1165
- import { serveUnits } from '@elaraai/e3-web/units';
1166
- serveUnits();
1167
- ```
1168
-
1169
- ```typescript
1170
- // e3.worker.ts — e3 itself: its storage, orchestrator, runner and API
1171
- import { serveE3 } from '@elaraai/e3-web/worker';
1172
- serveE3({
1173
- units: () => new Worker(new URL('./unit.worker.ts', import.meta.url), { type: 'module' }),
1174
- });
1175
- ```
1176
-
1177
- ```typescript
1178
- // main.ts — the page: e3-api-client's calls, every one answered by the e3 worker
1179
- import { createWebE3 } from '@elaraai/e3-web';
1180
- import {
1181
- dataflowExecute, packageImport, repoCreate, repoList, workspaceCreate, workspaceDeploy,
1182
- } from '@elaraai/e3-api-client';
1183
-
1184
- const e3 = await createWebE3(new Worker(new URL('./e3.worker.ts', import.meta.url), { type: 'module' }));
1185
- const opts = { token: null, fetch: e3.fetch };
1186
-
1187
- // The first visit makes the repository; later visits find it where they left it.
1188
- if (!(await repoList(e3.apiUrl, opts)).includes('default')) {
1189
- await repoCreate(e3.apiUrl, 'default', opts);
1190
- // The package zip e3.export wrote at build time, served beside the page
1191
- const zip = new Uint8Array(await (await fetch('/sales-planning-1.0.0.zip')).arrayBuffer());
1192
- const { name, version } = await packageImport(e3.apiUrl, 'default', zip, opts);
1193
- await workspaceCreate(e3.apiUrl, 'default', 'main', opts);
1194
- await workspaceDeploy(e3.apiUrl, 'default', 'main', `${name}@${version}`, opts);
1195
- await dataflowExecute(e3.apiUrl, 'default', 'main', {}, opts);
1196
- }
1197
- ```
1198
-
1199
- `e3.apiUrl` is `https://e3-web.invalid`, an origin reserved so that a request
1200
- sent without `e3.fetch` fails rather than reaching the network, and
1201
- `e3.fetch`, the global `fetch`'s signature, is answered by the e3 worker. Both
1202
- go to e3-api-client (`RequestOptions.fetch`) and to e3-ui-components'
1203
- `<E3Provider>` (`E3Config.fetch`: the **e3-ui** skill). An app's own platform
1204
- package is served beside the standard ones:
1205
- `serveUnits({ platforms: { '@acme/pricing': PricingPlatform } })`.
1206
-
1207
- | `serveE3` option | Default | Effect |
1208
- |---|---|---|
1209
- | `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
1210
- | `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
1211
- | `persist` | `true` | `false` keeps repositories in memory, gone with the page |
1212
- | `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
1213
- | `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
1214
- | `access` | `oneShotAccessByRoles()` with `identify`, else `any` | the grant a caller holds for a one-shot, a split call, or a function call naming its own runner |
1215
- | `wholeIntakeLimit` | 256 MiB | the largest delivery one intake unit takes in whole |
1216
-
1217
- What works is the API, as the local server answers it: every API suite of
1218
- e3-api-tests passes against it in Chromium, as an admin and, in its reader
1219
- cases, as a reader. Paged reads and key search, function calls, one-shot and
1220
- split calls, record mutations, deploys that migrate records or build indexes,
1221
- uploads, package import and export, gc, and dataflow runs — tasks in parallel
1222
- up to the pool's width (the cores), split tasks in pieces, re-run reactively.
1223
- A repository outlives a reload, and two tabs share an origin's repositories as
1224
- two processes share a directory. A closed tab's locks are free; a run it left
1225
- is taken as one whose process died, the next run served what finished from
1226
- the cache; and a job or an upload's commit it left is recorded failed, never
1227
- run again.
1228
-
1229
- What a browser cannot do — each fails naming it, never a partial answer:
1230
-
1231
- - **Commands.** A custom task, or a task on the `custom` runtime, is recorded
1232
- `error`: "a browser runs no commands".
1233
- - **Platform functions with no browser meaning.** east-web-std has no
1234
- FileSystem, Env or large-JSON reader, and east-node-io and the python
1235
- packages have no browser equivalent: a unit calling one fails naming the
1236
- function, and one listing a package its worker does not serve fails naming
1237
- the package.
1238
- - **Segments on demand.** A unit's inputs are staged whole into its worker:
1239
- waiting for a segment needs a cross-origin isolated page, which GitHub Pages
1240
- cannot give. Split a large input into pieces with `e3.partition`, which
1241
- bounds a unit's memory anywhere; a delivery that cannot be cut into pieces
1242
- is refused above `wholeIntakeLimit`, naming the fix.
1243
- - **A memory budget.** Memory is not measured: the pool is as wide as the
1244
- cores, there is no `--memory`, and no record names a peak.
1245
- - **The CLI.** `e3` drives e3 over HTTP, which an in-page e3 does not serve.
1246
-
1247
- ## A local repository
1248
-
1249
- ```
1250
- repo/
1251
- ├── repository.beast2 # the release that last wrote it, and the upgrades it has had
1252
- ├── repository-upgrade.beast2 # while an upgrade is under way: where its last part stopped
1253
- ├── metadata.beast2 # its name and status
1254
- ├── objects/ # content-addressed: values, segments, manifests, programs
1255
- ├── packages/ # package refs
1256
- ├── workspaces/ # each workspace's state, dataset refs, and its latest run's state and events
1257
- ├── dataflows/ # run records
1258
- ├── executions/ # execution attempts: status, owner, logs
1259
- ├── running/ # an index of the attempts recorded running, by task: what a status reads
1260
- ├── runs/ # an index of each task's runs, its units left out: what its history and logs read
1261
- ├── adoptions/ # the manifest each delivered file, or piece of one, became
1262
- ├── locks/ # locks, their holders, and how far each says it has got
1263
- ├── gc/ # gc beside running work: unreachable notes, a stepped run's parts
1264
- ├── envs/ # built execution environments
1265
- └── tmp/ # scratch and staged uploads
1266
- ```
1267
-
1268
- An e3 opening a repository an older release wrote applies the upgrades it has
1269
- not had, in place; one a newer e3 upgraded is refused, naming that release.
482
+ ## Caching
1270
483
 
1271
484
  **Caching.** An execution is keyed by its task — program, runner and
1272
485
  environment — and its inputs' hashes, so a task re-runs only when one of those
@@ -1284,6 +497,20 @@ consumers, even during a run. A split task is cached a unit at a time: each piec
1284
497
  and each merge is its own execution, and pieces are cut by content, so an append
1285
498
  or an insertion re-runs only the pieces it reaches and the merges above them.
1286
499
 
500
+ ## Reference files
501
+
502
+ Read the one a task needs; each holds its subject whole.
503
+
504
+ | File | Holds |
505
+ |---|---|
506
+ | [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes |
507
+ | [reference/record-migrations.md](reference/record-migrations.md) | `migration.value` / `rows` / `rekey`, and how a record's type changes |
508
+ | [reference/cli.md](reference/cli.md) | Every `e3` command: repositories, packages, workspaces, datasets, tasks, runs, calls, records, watch, convert, auth, gc |
509
+ | [reference/from-code.md](reference/from-code.md) | `@elaraai/e3-api-client`, `@elaraai/e3-core`, `e3-api-server` |
510
+ | [reference/browser.md](reference/browser.md) | `@elaraai/e3-web`: serveUnits, serveE3, createWebE3, and what a browser cannot do |
511
+ | [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, `@elaraai/e3-rack` |
512
+ | [reference/repository.md](reference/repository.md) | A local repository's layout, and upgrades |
513
+
1287
514
  ## Related skills
1288
515
 
1289
516
  - **east** — the language task bodies are written in.