@elaraai/east-codex-plugin 1.0.104 → 1.0.106
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/lsp.js +166 -10
- package/.build/daemon/server.js +166 -10
- package/.codex-plugin/plugin.json +1 -1
- package/index.json +117 -5
- package/package.json +1 -1
- package/skills/e3/SKILL.md +5 -3
- package/skills/e3/reference/cli.md +1 -1
- package/skills/e3/reference/from-code.md +8 -4
- package/skills/e3-ui/SKILL.md +22 -8
- package/skills/east/SKILL.md +5 -0
- package/skills/east/reference/beast2.md +10 -1
- package/skills/east/reference/queries.md +13 -1
- package/skills/east-py/reference/beast2.md +1 -1
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@elaraai/east-codex-plugin",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.106",
|
|
4
4
|
"description": "East programming language ecosystem - a statically typed, expression-based language for business logic, data processing, UI definitions, and durable execution",
|
|
5
5
|
"license": "AGPL-3.0-or-later",
|
|
6
6
|
"type": "module",
|
package/skills/e3/SKILL.md
CHANGED
|
@@ -69,7 +69,7 @@ What do you need?
|
|
|
69
69
|
│ │ ├─ Too large, or built as you go → e3.streamTask(name, { inputs, output }, fn) — emits into e3.output.array | set | dict | fold
|
|
70
70
|
│ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
|
|
71
71
|
│ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
|
|
72
|
-
│ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name
|
|
72
|
+
│ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name; a call through a workspace is kept for its lifetime
|
|
73
73
|
│ │ └─ It reads workspace data too → e3.function(name, { inputs, args, output, runner? }, fn) — inputs read at each call
|
|
74
74
|
│ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
|
|
75
75
|
│ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
|
|
@@ -385,7 +385,7 @@ const change = e3.streamTask('change', {
|
|
|
385
385
|
| Definition | What it is |
|
|
386
386
|
|---|---|
|
|
387
387
|
| `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
|
|
388
|
-
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow
|
|
388
|
+
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API, `Func.bind`) and its result returned inline: not in the dataflow. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A call through a workspace is the server's: it outlives the request that starts it — the client starts it and polls it to its end — and runs until its timeout, the server's ceiling (ten minutes locally) unless it asks for less; it can be cancelled; and the server keeps it for its lifetime, an hour for a success or a cancel and a week for any other end: who made it, its arguments, the versions of the inputs it read, its value and its logs (reference/from-code.md). A call through the package runs within one request and keeps nothing. A result is at most 1 MiB, so a large one belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
|
|
389
389
|
| `e3.function(name, { inputs, args, output, runner?, environment?, async? }, fn)` | A function that reads the workspace as well: `inputs` (an input, a task's output or a record, as a task takes them) are read as stored when each call launches, pinned at their hashes and named in the result's `inputs`; the caller passes `args` alone. The body takes the inputs, then the arguments. `e3.package` collects the inputs and what they depend on; nothing reruns when one changes. The inputs are read one at a time and the call waits for no run, so a call during a run may read one input before the run rewrites it and another after: the hashes in `inputs` say which. Called through a workspace only (`e3 call -w`); an unassigned input is `invalid`, naming it and why. ❗ Refused: an `e3.partition` input, and the `custom` runtime with inputs. `FunctionSpec` names the spec's type. |
|
|
390
390
|
| `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
|
|
391
391
|
| `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
|
|
@@ -445,7 +445,9 @@ part, and the record carries each by its name — `plans.mutations.add_plan`,
|
|
|
445
445
|
|
|
446
446
|
Every form commits a delta addressed by key, rewriting only the segments it
|
|
447
447
|
touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
|
|
448
|
-
takes `{ runner? }
|
|
448
|
+
takes `{ runner? }`: with none, its program — and an index's, which takes one
|
|
449
|
+
too — runs on east-c, naming no platform, since it calls none; a local e3 whose
|
|
450
|
+
east-c cannot run (none found, or no native binary) runs it on east-node. In an `edit`, repeated edits of one key fold, a `set` of the
|
|
449
451
|
value a row already holds changes nothing, and a `delete` or `update` of a key
|
|
450
452
|
the record does not hold is a conflict naming the key: the caller's view of the
|
|
451
453
|
record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
|
|
@@ -34,7 +34,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
|
|
|
34
34
|
| `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
|
|
35
35
|
| `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
|
|
36
36
|
| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
|
|
37
|
-
| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. A function that declares `inputs` reads them from the workspace itself, so it takes its arguments alone and is called with `-w`; without it, it is refused, naming its inputs. |
|
|
37
|
+
| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. A function that declares `inputs` reads them from the workspace itself, so it takes its arguments alone and is called with `-w`; without it, it is refused, naming its inputs. The value goes to stdout, and the function's stdout and stderr to stderr. Through a server's workspace the call is the server's, kept for its lifetime: it runs as long as its timeout allows, a call longer than a second is named (`Call <id> runs on`), and Ctrl-C asks the server to stop it (exit 130). |
|
|
38
38
|
| **Records** (`-w <ws>` required) | |
|
|
39
39
|
| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
|
|
40
40
|
| `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
|
|
@@ -18,7 +18,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
|
|
|
18
18
|
| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
|
|
19
19
|
| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
|
|
20
20
|
| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
|
|
21
|
-
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
|
|
21
|
+
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; within one request, keeping nothing; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts, { signal?, onStarted?, startWaitMs? })` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`), within one request, keeping nothing; `oneShotCall(url, repo, ws, request, opts, { signal?, onStarted?, startWaitMs? })` — the same, under the same rule, started as a kept call through the workspace (below), and run within one request on a server that keeps none (`oneShotStart` its start apart); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress?, signal? })` → `{ result, output, report }` — a program over a dataset's pieces, as a job it polls, under the same rule; `output` is `array`, `set`, `dict { merge }`, `fold { combine, zero }`, or `first { limit }` — an `array`'s first `limit` elements in input order, its pieces run in order and stopped once they hold them; `then`, the final function, takes the assembled output and then the arguments not partitioned, in order, and one of another arity is `invalid`, and how it ran: its pieces, the units it ran and the cache served, and its peak; a value over the call's result limit is `too_large`, naming its stored `output`, which `collectionOpen(url, repo, output, opts)` → `{ type, totalElements, page({ offset, limit }) }` reads a window at a time, fetching only the segments each window overlaps; aborting `signal` cancels the job, whose units the server stops (or `splitCallLaunch`, `splitCallStatus`, `splitCallCancel` — `cancelled`, or the status the job had already ended with — and `splitCallExplain`) |
|
|
22
|
+
| Calls through a workspace | A call through a workspace is the server's, and outlives the request that starts it: `workspaceFunctionCall` starts it, waits up to the server's window (20 s; `startWaitMs` less), and polls a call still running to its end, each poll held until it ends or a while passes, however long its timeout lets it run — the server's ceiling, ten minutes locally, unless its `limits` ask for less; aborting `signal` asks the server to stop it, and `onStarted(id)` is told the id of one that runs on; a server that keeps no calls is called through `workspaceFunctionCallSync`, the sync route, which a caller may name itself. Apart: `workspaceFunctionStart(…, opts, { waitMs? })` → `{ id, status }` (`?wait=0` answers `running` at once), `workspaceFunctionCallStatus(url, repo, ws, id, opts, { waitMs?, signal? })` → `running`, `completed` with the result, `cancelled` naming who asked, or `lost`, `workspaceFunctionCallCancel(url, repo, ws, id, opts)`. The server keeps each call for its lifetime, an hour for a success or a cancel and a week for any other end: `workspaceCallList(url, repo, ws, opts, { function?, state?, before?, limit? })` — a page of them, the latest first, its `next` the next page's `before` (`FUNCTION_CALLS_PAGE_DEFAULT` 50, `…_MAX` 200), and `workspaceCallGet(url, repo, ws, id, opts)` — one whole: its function and package (`none` for a one-shot), its arguments (values by the objects they were stored as, datasets by the hashes they were pinned at), the inputs it read, its limits, its value's object, its log tails and when it is forgotten (`expiresAt`); `objectGet` reads a value by its hash. A one-shot started as a call (`oneShotCall`) is kept, polled, cancelled and listed as a function's call is, its program the caller's, under its one-shot grant. A workspace keeps no call that another made: `internal`, "keeps no call" |
|
|
22
23
|
| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
|
|
23
24
|
| From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
|
|
24
25
|
|
|
@@ -41,8 +42,11 @@ over itself.
|
|
|
41
42
|
**`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
|
|
42
43
|
repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
|
|
43
44
|
`-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
|
|
44
|
-
every run and call it serves and every upload it takes in),
|
|
45
|
+
every run and call it serves and every upload it takes in), the call limits —
|
|
46
|
+
`--call-deadline` (a one-shot or sync function call's, 120s), `--job-timeout`
|
|
47
|
+
(a split call's job, or a kept call, 10m), `--max-result` (1M, held in the server's memory while
|
|
48
|
+
it is answered) and `--max-log` (256K) — and auth: `--oidc`, a built-in provider for `e3
|
|
45
49
|
auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
|
|
46
50
|
issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
|
|
47
|
-
`createServer({ reposDir | singleRepoPath, port, host, … })` starts one,
|
|
48
|
-
route factories mount on another host.
|
|
51
|
+
`createServer({ reposDir | singleRepoPath, port, host, calls?, … })` starts one,
|
|
52
|
+
and its route factories mount on another host.
|
package/skills/e3-ui/SKILL.md
CHANGED
|
@@ -118,7 +118,7 @@ Task → What do you need?
|
|
|
118
118
|
│ ├─ Lifecycle (variant) → .status() (idle|running|succeeded|failed|cancelled)
|
|
119
119
|
│ ├─ Failure detail → .error() (Option — message/kind/stdout/stderr)
|
|
120
120
|
│ ├─ Spinner boolean → .pending()
|
|
121
|
-
│ ├─ Stop
|
|
121
|
+
│ ├─ Stop the call → .cancel() (asks the server to stop it; a later .call() supersedes one in flight the same way)
|
|
122
122
|
│ └─ A function that reads workspace data too → e3.function(name, { inputs, args, output }, fn); .call(args…) takes the args alone
|
|
123
123
|
│
|
|
124
124
|
├─ Edit a graph / ontology dataset
|
|
@@ -311,12 +311,18 @@ conflict naming the entry, and nothing is overwritten. `Record.onApply(record,
|
|
|
311
311
|
|
|
312
312
|
### `Func.bind(fn)`
|
|
313
313
|
|
|
314
|
-
A call handle for an `e3.function`
|
|
315
|
-
|
|
316
|
-
`
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
314
|
+
A call handle for an `e3.function`. `call(args…)` is fire-and-forget and safe
|
|
315
|
+
in a sync `onClick`, latest-wins; the outcome arrives reactively through
|
|
316
|
+
`read()`, `status()`, `error()` and `pending()`. All handles bound to one
|
|
317
|
+
function share one channel, so one component can launch while another shows the
|
|
318
|
+
spinner. A call is the server's: it outlives the request that starts it, and
|
|
319
|
+
runs until its timeout — the server's ceiling, ten minutes locally — however
|
|
320
|
+
long that is, so an optimiser of several minutes is called the same way; the
|
|
321
|
+
server keeps it for its lifetime, with its arguments, the inputs it read, its
|
|
322
|
+
value and its logs (the **e3** skill). `cancel()` asks the server to stop the
|
|
323
|
+
call, and `status()` reads `cancelled`; a `call` that supersedes one in flight
|
|
324
|
+
asks the server to stop that one. A result is at most 1 MiB: a large one
|
|
325
|
+
belongs in a task, read with `Data.bind`.
|
|
320
326
|
|
|
321
327
|
A function that declares `inputs` (`e3.function(name, { inputs, args, output },
|
|
322
328
|
fn)`, the **e3** skill) is bound the same way: its handle's `call` takes the
|
|
@@ -426,7 +432,15 @@ start and end are `Option<DateTime>` when it can wait in the backlog.
|
|
|
426
432
|
saved jq queries; `Query.value(sources, queries)` ships some, checked at
|
|
427
433
|
build. `<Query.Builder queries datasets />` edits a query as steps or jq,
|
|
428
434
|
checks it, runs it on e3 (split over a large dataset's pieces) and saves it;
|
|
429
|
-
`<Query.Library>` lists them.
|
|
435
|
+
`<Query.Library>` lists them. A run of one program is a call the workspace
|
|
436
|
+
keeps, as it keeps a function's (the **e3** skill): held to its own timeout,
|
|
437
|
+
never a request's, and listed among the workspace's calls, naming no
|
|
438
|
+
function. A split run's plan read-out says how many pieces it cut and how
|
|
439
|
+
many units e3's cache served; `first(…)` and `limit(n; …)` over a dataset's
|
|
440
|
+
rows split too, stopping once they have their rows; a run given up — a new
|
|
441
|
+
run, or the builder going — cancels its call, or its split call's job; and a
|
|
442
|
+
split answer too large for one, with no final step, is read a page at a time
|
|
443
|
+
in the Table and the Tree.
|
|
430
444
|
|
|
431
445
|
## Rendering surfaces in an app — `<E3Provider>`
|
|
432
446
|
|
package/skills/east/SKILL.md
CHANGED
|
@@ -69,6 +69,8 @@ Task → What do you need?
|
|
|
69
69
|
│ ├─ Collection → ArrayType(T), SetType(K), DictType(K, V), RefType(T)
|
|
70
70
|
│ ├─ Numeric → VectorType(T), MatrixType(T), T one of FloatType, IntegerType, BooleanType
|
|
71
71
|
│ ├─ Compound → StructType({...}), VariantType({...}), OptionType(T), RecursiveType(self => ...)
|
|
72
|
+
│ │ (a field named like an integer → StructType([["b", T], ["10", T]]), its fields as a list in order:
|
|
73
|
+
│ │ an object lists "10" before "b", so the object form refuses one)
|
|
72
74
|
│ ├─ Function → FunctionType([inputs], output), AsyncFunctionType([inputs], output)
|
|
73
75
|
│ └─ Patch → PatchType(T) (the patch type of any type); dictPatchOpsType(V) / setPatchOpsType(E)
|
|
74
76
|
│ for the op a touched key carries, when building a sparse key-addressed change set yourself
|
|
@@ -186,6 +188,9 @@ Task → What do you need?
|
|
|
186
188
|
├─ A JSON contract for a type → jsonSchemaFor(T) · back from a vendored schema → typeFromJsonSchema(schema)
|
|
187
189
|
└─ Data larger than memory → reference/beast2.md; inside a body blob.openBeast(T),
|
|
188
190
|
and a file on the runner's disk FileSystem.openBeast(T, path) (east-node-std / east-c-std / east-py-std)
|
|
191
|
+
└─ A loop over a lazy input (a large task input, an openBeast) decodes only the fields its body reads:
|
|
192
|
+
read the row through fields (row.total, $.for(row.lines, …)), not bound whole ($.let(row));
|
|
193
|
+
a runner's -v says "paged loops: N segments decoded to the fields read, M in full"
|
|
189
194
|
```
|
|
190
195
|
|
|
191
196
|
## Types and their values
|
|
@@ -20,7 +20,7 @@ reads any container version its type was written in.
|
|
|
20
20
|
| **Collections larger than memory** (v5) |
|
|
21
21
|
| `new Beast2ElementWriter(T, sink, opts?)` | The canonical writer: `.add(element)` one Array/Set element or Dict `[key, value]` — Set elements and Dict keys strictly ascending in East order — and the content-defined cut rule places the segments, so the bytes are a function of the value, identical from east-c and east-py, and a one-row edit re-cuts only the segments around it. Memory is one segment; `.finish()` writes the last, the terminator and the index. `sink` is `(bytes) => void`, or `{ segment(s) }` for one standalone segment blob at a time (`s`: `{ count, fence, logicalBytes, blob }`, under `writer.header`) |
|
|
22
22
|
| `encodeBeast2PagedFor(T)(value)` | One value in memory through the element writer; plain `Map`/`Set` inputs are sorted first |
|
|
23
|
-
| `new Beast2RunSorter(T, openRun, { merge } \| { union })` | Elements in ANY order in, sorted canonical runs out (a key added twice folds with `merge(key, acc, value)`, or `union` for a Set; without one it throws); then `mergeBeast2For(T, { merge } \| { union })(runs, sink)` writes the value. Memory is one run, then one segment per run. A run closes at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES`, platform constants, so every runtime writes the same runs |
|
|
23
|
+
| `new Beast2RunSorter(T, openRun, { merge } \| { union })` | Elements in ANY order in, sorted canonical runs out (a key added twice folds with `merge(key, acc, value)`, or `union` for a Set; without one it throws); then `mergeBeast2For(T, { merge } \| { union })(runs, sink)` writes the value. Memory is one run, then one segment per run. A run closes at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES`, platform constants, so every runtime writes the same runs. A fold whose values do not grow — `merge` over a bounded value type (Null, Boolean, Integer, Float, DateTime, Structs and Variants of those), or `union` — folds each key's values as they are added, in a table of at most `FOLD_TABLE_MAX_KEYS` keys and `FOLD_TABLE_MAX_KEY_BYTES` that spills into the open run, so few keys make one small run |
|
|
24
24
|
| `mergeBeast2For(T, opts?)(sources, sink)` | Sorted collections — blobs, range readers or manifests — into their union's blob or manifest, segment by segment; a key several hold folds in input order; `opts.from` / `opts.to` merge a key range |
|
|
25
25
|
| `recutBeast2For(T)(pieces, sink)` | A collection in pieces, some already written, into its canonical segments: a segment the whole shares with its piece is carried unread (`sink.carried`), the rest re-cut (`sink.written`) |
|
|
26
26
|
| `intakeBeast2For(T)(delivery, sink, opts?)` | A collection from outside — a `Beast2SyncRangeReader` over a v4 or v5 blob whose header names exactly `T` — through the Writer into a `Beast2ManifestSink`, a segment at a time: each row is walked by its type, the Writer's own bytes go on as they stand, and any other row is decoded and written again. `opts.segments` takes in the index's segments `[from, to)`, a piece of a large delivery. Refuses another type, a malformed segment or one over `RUN_MAX_BYTES`, a row that does not decode, and keys that do not ascend, with `Beast2IntakeError` in the words every runner uses. What e3's intake units run, on every runner |
|
|
@@ -69,3 +69,12 @@ openBeast2PagesFor(Rows)(blob).element(4_000_000); // one row: decodes ONE segm
|
|
|
69
69
|
`$.for` cost one segment, mutation is refused (`.copy()` first), and the
|
|
70
70
|
header type is checked against `T` first. `East.Blob.encodeBeast` writes no
|
|
71
71
|
index, so a paged blob comes from the writers above or a runner's output.
|
|
72
|
+
- **A `$.for` over a lazy input decodes each segment to the fields its body
|
|
73
|
+
reads** of each row — an Array's element, a Dict's value — on every runtime:
|
|
74
|
+
read through field accesses (`row.total`, `row.address.city`), inner loops
|
|
75
|
+
over its arrays and dicts (`$.for(row.lines, …)`), their sizes, and functions
|
|
76
|
+
called on the row or its fields. Binding the row (`$.let(row)`), putting it in
|
|
77
|
+
a container or passing it to a function value decodes every field, as does a
|
|
78
|
+
Set's loop. Every runner's `-v` says `paged loops: N segments decoded to the
|
|
79
|
+
fields read, M in full`; in process, `beast2PagedLoopStats()` counts
|
|
80
|
+
them. The bytes are read either way: a dropped field is only not built.
|
|
@@ -22,6 +22,18 @@ const revenue = East.function([ArrayType(Order)], FloatType, ($, orders) =>
|
|
|
22
22
|
East.jq({ orders }, ".orders | map(.total) | add", FloatType));
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
+
A stage that collects rows before a step that reads them one at a time —
|
|
26
|
+
`map(f) | add`, `| length`, `| any`, `| first`, `| .[:n]`, a `reduce` over
|
|
27
|
+
`.[]` — runs as one loop, so the array between them is never built; `first`
|
|
28
|
+
and `.[:n]` stop the stage once they have their rows, unless a later row could
|
|
29
|
+
raise (QUERY.md §15.1–15.2).
|
|
30
|
+
|
|
31
|
+
A row a stage builds or keeps holds only the fields the steps after it read:
|
|
32
|
+
`map(. + {region: …}) | group_by(.region) | map(… .total …)` builds each row of
|
|
33
|
+
its region and total, and over an e3 root `map(select(p))` keeps each row as
|
|
34
|
+
the fields read after it, so a paged dataset decodes those alone. A row read
|
|
35
|
+
whole — the output, `keys`, `tojson`, `==` — is built whole (QUERY.md §15.1).
|
|
36
|
+
|
|
25
37
|
| Signature | Description | Example |
|
|
26
38
|
|-----------|-------------|---------|
|
|
27
39
|
| **In an East body** |
|
|
@@ -32,7 +44,7 @@ const revenue = East.function([ArrayType(Order)], FloatType, ($, orders) =>
|
|
|
32
44
|
| `printJq(program: JqNode, options?: { layout?: "line" \| "pipeline" }): PrintedJq` | A program's canonical text, and its nodes' spans; `pipeline` puts each top-level stage on a line of its own | `printJq(checked.program!).text` |
|
|
33
45
|
| `translateJq(checked: CheckJqResult, options?: { maxOutputs?: number, tooling?: boolean }): JqTranslation` **❗** | The translation: `fn()`, an East function of the inputs the query reads (`inputs`), and its `resultType` | `East.compile(translateJq(checked).fn(), [])` |
|
|
34
46
|
| `evaluateJq(program: string \| CheckJqResult, input: unknown, options?: { inputType?, root?, tooling?, platform? }): unknown` **❗** | Checks, translates, compiles (cached) and runs a query over a TypeScript value. Throws `QueryError`: the checker's diagnostics, or one `runtime` diagnostic at the node that raised | `evaluateJq("[.[] \| .id]", orders, { inputType: ArrayType(Order) })` |
|
|
35
|
-
| `splitJq(checked: CheckJqResult, options?: { maxOutputs?: number }): JqSplit` | A query checked as an e3 root, split over one dataset's pieces for an e3 split call: `piece()` each piece runs, the `output` kind its outputs combine by (`array`, `set`, `dict` with `merge()`, `fold` with `zero` and `combine()`), and `then()` over the result; or, `kind: "whole"`, why it runs as one unit. QUERY.md §17.1 | `splitJq(checkJq(".orders \| group_by(.region) \| map(length)", Root, { root: true })).output.kind` |
|
|
47
|
+
| `splitJq(checked: CheckJqResult, options?: { maxOutputs?: number }): JqSplit` | A query checked as an e3 root, split over one dataset's pieces for an e3 split call: `piece()` each piece runs, the `output` kind its outputs combine by (`array`, `set`, `dict` with `merge()`, `fold` with `zero` and `combine()`, or `first` with its `limit`, a stream's first rows in input order for `first(f)` and `limit(n; f)`), and `then()` over the result; or, `kind: "whole"`, why it runs as one unit. QUERY.md §17.1 | `splitJq(checkJq(".orders \| group_by(.region) \| map(length)", Root, { root: true })).output.kind` |
|
|
36
48
|
|
|
37
49
|
- **The values are East's.** A Dict is an object keyed by its key type, so
|
|
38
50
|
`.byId[1035]` looks up an Integer key; a variant reads as `{type, value}`,
|
|
@@ -70,7 +70,7 @@ with open_beast2_file("table.beast2") as t: # Dict<String, Float>
|
|
|
70
70
|
| `encode_beast2_for(T)` / `decode_beast2_for(T)` | **Headerless** type-directed bytes: the reader must know `T` exactly, and mutable containers (Array/Set/Dict/Ref) are refused. In TypeScript `encodeBeast2For` is the FULL container — do not port a call site by name |
|
|
71
71
|
| `Beast2ElementWriter(T, stream, *, codec="deflate", parallel=False)` | The canonical writer (a context manager): `.add(el)` — an Array/Set element or a Dict `(key, value)` pair — and `.add_all(batch)` (the loop in east-c); Set elements and Dict keys strictly ascending in East order. Segments fall where the cut rule places them — the bytes the managed writer, TypeScript and east-c write; memory is one open segment; `.close()` writes the last segment, the terminator and the index; `.segments` counts the closed ones. An element that does not ascend, or fails to encode, raises and leaves the writer as it was |
|
|
72
72
|
| `encode_beast2_paged_for(T, *, codec="deflate") -> (value) -> bytes` | A whole collection through the canonical writer — the write side of `open_beast2_pages_for` |
|
|
73
|
-
| `Beast2RunSorter(T, open_run, *, merge=None, union=False, codec="deflate", parallel=False)` | A Set's or Dict's elements in ANY order in, sorted canonical runs out: `.add(el)` encodes at once; at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES` it sorts the run — stably, so a key's values keep the order they came in — folds a repeated key (`merge`, a compiled `(K, V, V) -> V` East function, for a Dict; `union=True` for a Set; neither: it raises) and writes the run to `open_run(run)` (anything with `write(bytes)` and `close()`) as its value's canonical blob. `.finish()` writes the last run; `.runs` counts them. The runs are the ones east-c and TypeScript write; a sink's exception comes back out of the call that wrote the run; a key repeated across runs is the merge's to fold |
|
|
73
|
+
| `Beast2RunSorter(T, open_run, *, merge=None, union=False, codec="deflate", parallel=False)` | A Set's or Dict's elements in ANY order in, sorted canonical runs out: `.add(el)` encodes at once; at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES` it sorts the run — stably, so a key's values keep the order they came in — folds a repeated key (`merge`, a compiled `(K, V, V) -> V` East function, for a Dict; `union=True` for a Set; neither: it raises) and writes the run to `open_run(run)` (anything with `write(bytes)` and `close()`) as its value's canonical blob. `.finish()` writes the last run; `.runs` counts them. A fold whose values do not grow — `merge` over a bounded value type (Null, Boolean, Integer, Float, DateTime, Structs and Variants of those), or `union` — folds each key's values as they are added, in a table of at most `FOLD_TABLE_MAX_KEYS` keys and `FOLD_TABLE_MAX_KEY_BYTES` that spills into the open run, so few keys make one small run. The runs are the ones east-c and TypeScript write; a sink's exception comes back out of the call that wrote the run; a key repeated across runs is the merge's to fold |
|
|
74
74
|
| `Beast2Writer(T, stream, *, codec="deflate", self_contained=True, index=True, parallel=False)` | Segments of your choosing (a context manager): `.write(batch)` appends one segment per non-empty batch; `.close()` writes the terminator and the index; `.segments` counts. Set/Dict batches must ascend strictly across the stream. **Keep both defaults** — `index` and `self_contained` are what `open_beast2_pages_for` needs, and turning either off silently forfeits random access. `codec="none"` for already-compressed payloads or raw write throughput |
|
|
75
75
|
| `encode_beast2_segments_for(T, **opts) -> (batches) -> bytes` | In-memory `Beast2Writer`: one segment per non-empty batch |
|
|
76
76
|
| `encode_beast2_v5_for(T, *, codec="deflate", index=False) -> (value) -> bytes` | A whole value of any root type as v5; decode with `decode_beast2_with_header_for` |
|