@elaraai/east-codex-plugin 1.0.93 → 1.0.95

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@elaraai/east-codex-plugin",
3
- "version": "1.0.93",
3
+ "version": "1.0.95",
4
4
  "description": "East programming language ecosystem - a statically typed, expression-based language for business logic, data processing, UI definitions, and durable execution",
5
5
  "license": "AGPL-3.0-or-later",
6
6
  "type": "module",
@@ -1,8 +1,12 @@
1
1
  ---
2
2
  name: e3
3
- description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
3
+ description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack… See the detailed scope below."
4
4
  ---
5
5
 
6
+ ## Detailed skill scope
7
+
8
+ East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui().
9
+
6
10
  # East Execution Engine (e3)
7
11
 
8
12
  e3 runs East programs as a durable dataflow over content-addressed data: every
@@ -72,6 +76,7 @@ What do you need?
72
76
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
73
77
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
74
78
  ├─ Operate it from the shell → the e3 CLI (below): deploy, run, read and write datasets, mutate records, gc
79
+ ├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack
75
80
  └─ Drive it from code
76
81
  ├─ A server, over HTTP → @elaraai/e3-api-client
77
82
  ├─ A local repository → @elaraai/e3-core
@@ -79,6 +84,144 @@ What do you need?
79
84
  └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 ("Running e3 in a browser")
80
85
  ```
81
86
 
87
+ ## Rack delegation from a local repository
88
+
89
+ **Setup and routing**
90
+
91
+ | Signature | Description | Example |
92
+ | --- | --- | --- |
93
+ | `e3 rack enroll --listen <host:port> [--allow <ip,...>] [--advertise-url <origin>] [--label <label>] [--no-wait]` | Starts the user-wide hub and prints the source origin, the hub's certificate SHA-256, the matching release and a single-use token for the companion's hidden prompts. Waits for a new registration, not activation. Behind an SSH reverse tunnel, listen on the loopback and advertise the relay's origin. | `e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://relay.example:7401` |
94
+ | `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards. Size: serverless, small, medium, large, xlarge. | `e3 rack policy . set 'train_*' rack --size large` |
95
+ | `e3 rack policy [repo] enable\|disable\|mode <opt-in\|auto>` | Enable/disable saved opt-in; `mode auto` selects eligible unmatched tasks. Defaults: disabled, opt-in. | `e3 rack policy . enable` |
96
+ | `e3 rack policy [repo] defaults [--no-spill] [--claim-timeout <sec>] [--retry-task-failures-locally]` | Defaults: large, 1440-minute timeout, spill on, 30-second claim recheck, no task-failure retry. | `e3 rack policy . defaults --no-spill` |
97
+ | `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
98
+ | `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
99
+ | `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
100
+ | `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub. | `e3 rack status` |
101
+ | `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
102
+
103
+ The local process owns graph planning, repository and cache. After a cache
104
+ miss, eligible plain task, map and merge bodies can execute on the rack; input
105
+ closures and every output object transfer by hash. The session verifies the
106
+ whole output graph before success. Placement never changes task/cache hashes.
107
+ Local bodies and fallback share the original cores/memory budget; rack bodies
108
+ use remote capacity. Calls, mutations and intake stay local. Stock runner
109
+ configurations qualify; commands, custom environments/platforms and unscannable
110
+ IR stay local. Host file/environment/network access stays local unless a rule
111
+ deliberately grants `--allow-host-access` and its resources exist on the rack.
112
+
113
+ The listener serves HTTPS with the hub's own self-signed certificate, made on
114
+ first use and kept in `hub-tls.beast2`; a rack pins it by its SHA-256, which
115
+ enrollment prints, so TLS runs end to end through any relay or tunnel. The
116
+ usual path is an SSH reverse tunnel from the developer machine to a relay the
117
+ rack reaches, with the hub on the loopback (`--allow 127.0.0.1`) and the relay's
118
+ origin advertised. Removing the certificate file, with the hub stopped, makes a
119
+ new one, which every worker must then pin again. `--no-tls` is for a TLS
120
+ terminator in front of the hub; the hub refuses plain HTTP on a wildcard or
121
+ public address, or without `--allow`. A listener on every address needs
122
+ `--advertise-url` to enroll.
123
+
124
+ The handoff targets a prepared independent worker VM with the matching verified
125
+ `e3-rack` release. Its `configure --allocation <host-provisioned-path>` prompts
126
+ for the source origin, the certificate SHA-256 for an HTTPS origin the hub
127
+ serves, the worker label and settings; `enroll --allocation <path>` prompts for
128
+ the one-time token. Origins, pins and credentials are encrypted together. A
129
+ plain-HTTP origin needs `--allow-vpn-http` on both commands and host
130
+ authorization for that exact endpoint, and belongs only on a path encrypted end
131
+ to end, such as an SSH tunnel, never across a VPN hub that decrypts traffic.
132
+ Never put a source address or token in worker command arguments or plaintext
133
+ files. The local mint intentionally displays its token; do not log that output.
134
+
135
+ Enrollment does not start services or authorize task claims. The local CLI
136
+ observes a new registration (optionally filtered by label); compare it with the
137
+ worker's result. Before a first heartbeat, status is `enrolled-not-started`.
138
+ After managed resume, a ready heartbeat advertises free plus occupied slots;
139
+ busy workers stay eligible. Fenced, draining or stopping workers withdraw their
140
+ capacity and show `unavailable`. After 45 seconds without a heartbeat, readiness
141
+ expires; lease polling cannot prolong it. Source boot approval, when required,
142
+ remains an additional gate. This advertisement does not authorize host resource
143
+ admission or removal of service/network quarantine.
144
+ Resources and mutable whole-GPU assignments belong to the host lifecycle, not
145
+ local enrollment flags. The companion accepts up to 32 mutually trusted sources
146
+ in one worker. After draining and stopping its services and sockets, use
147
+ `configure --allocation <path> --add-source`, confirm the shared trust boundary,
148
+ and enter the additional origin privately. It prints the new source UUID for
149
+ `enroll --allocation <path> --source <UUID>`; add `--allow-vpn-http` to both for a
150
+ plain-HTTP origin. All sources must be enrolled before startup. Each has its own identity,
151
+ client, cache and completion journal; actual execution shares one worker-wide
152
+ task/GPU limit. Per-source capacity reports are not additive reservations.
153
+ A worker is the trust boundary: a shared agent process can access all its
154
+ sources' credentials, so separate clients, and a cloud deployment and developer
155
+ hubs, need separate worker VMs. The hub keeps no task data; what a worker keeps
156
+ between tasks is its own policy, and an agent that keeps nothing works unchanged. Worker-local `source list --allocation <path>`
157
+ prints UUIDs/enrollment status; `--show-origins` adds origins only on a terminal.
158
+ After drain/stop and source-side revocation, `source remove` or `source reset`
159
+ with `--allocation <path> --source <UUID> --acknowledge-revocation` asks for UUID
160
+ confirmation. Reset prints a new UUID for fresh enrollment; old pending results
161
+ and caches stay in the retired namespace. Other sources and retained data are
162
+ preserved, and host endpoint policy must be reconciled before activation. The
163
+ flag acknowledges revocation; it does not revoke remotely.
164
+
165
+ Worker-local `registry status --allocation <path>` reports only whether encrypted
166
+ registry credentials are present; it can run while active and does not test access.
167
+ After drain/stop, `registry configure --allocation <path>` privately prompts for
168
+ username/token without re-entering settings. `registry clear --allocation <path>`
169
+ asks for the installation UUID and removes only the stored access. Both require
170
+ the same release, the writer lock and no unresolved enrollment/service transition;
171
+ they preserve current identities, rotations, settings, journals and cached data.
172
+ Credentials stay in the existing encrypted bundle, with no Docker login file.
173
+ A later activation receives them; services remain stopped and the next boot fenced.
174
+ Repeating configure with the same values or clear when absent does not write.
175
+ Clearing/replacing does not revoke a token remotely or purge cached images.
176
+
177
+ The companion's
178
+ `reconcile-services` applies a host-provided next allocation and service budgets
179
+ after a completed drain, with stopped units, the previous plan hash and exact
180
+ operation/boot binding. It retains ciphertext/journals and network quarantine;
181
+ it does not move a GPU, start a worker or change its executable release. For a
182
+ compatible release change, invoke the target verified executable's `switch-release`
183
+ with the same drain/plan bindings and unchanged resource budgets at the next
184
+ allocation generation. It checks both managed-state contracts, updates only the
185
+ runtime version in the current encrypted configuration, and preserves identities
186
+ and journals through interruption or downgrade. Both four-file releases must stay
187
+ available; a downgrade is a new operation, never a saved-state restore. Services
188
+ stay stopped and quarantined. If the selected release never resumed, the exact
189
+ recorded previous executable's `recover-release` binds the completed release
190
+ receipt, current plan and new operation at another generation increase. It
191
+ requires the original drain or a valid fenced journal with no operation at the
192
+ failed generation, verifies both releases' recorded hashes, and preserves current
193
+ credentials and journals. A prior resume attempt, even across reboot, refuses
194
+ recovery. General journal repair remains unfinished. Named host management, source aliases, network
195
+ activation, incompatible-state migrations and hardware acceptance remain
196
+ companion cloud #253 work. This and durable completion retry are implemented in
197
+ companion PR #254, separately from the original main protocol pin.
198
+
199
+ Policy is `<repo>/rack/policy.beast2`; hub state is `~/.e3/rack` (or
200
+ `E3_RACK_HOME`). A hub `dataflow run` starts exits after 15 idle minutes, and a
201
+ run waits up to 10 seconds for an enrolled rack to report healthy (with none
202
+ enrolled, or a policy that selects no task for a rack, it starts at once): run
203
+ `e3 rack hub` as a service, or set `--idle-exit 0`, for a hub racks can always
204
+ reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
205
+ at most 5 GiB. The control schema currently requires protocol 2 exactly.
206
+ Agents must report bundled e3 at least the project's release. Infrastructure
207
+ failure falls back locally with a separate execution attempt, as does an
208
+ attempt the rack stopped on its own; the run's own cancellation and the task's
209
+ timeout never do. One cancelled subscriber does not stop work shared by other
210
+ sessions of the same canonical repository and e3 release.
211
+
212
+ **Programmatic attachment** (`@elaraai/e3-rack`)
213
+
214
+ | Signature | Description | Example |
215
+ | --- | --- | --- |
216
+ | `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
217
+ | `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
218
+
219
+ The API server's optional rack package is needed only for
220
+ `e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
221
+ factory and do not import hub code. Custom environments, remote function
222
+ delegation, automatic reconnect after hub restart and embedded UI integration
223
+ are deferred.
224
+
82
225
  ## Authoring a package
83
226
 
84
227
  ### Inputs — `e3.input(name, type, source?)`