@elaraai/east-codex-plugin 1.0.98 → 1.0.99

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,11 +1,11 @@
1
1
  ---
2
2
  name: e3
3
- description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex, e3.recordIndex.interval / e3.recordIndex.area for reads by a time window or a 2D window, and e3.recordIndex.series / e3.recordIndex.summary for a block of keys over a span of time tiles and their totals) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code:… See the detailed scope below."
3
+ description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package,… See the detailed scope below."
4
4
  ---
5
5
 
6
6
  ## Detailed skill scope
7
7
 
8
- East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex, e3.recordIndex.interval / e3.recordIndex.area for reads by a time window or a 2D window, and e3.recordIndex.series / e3.recordIndex.summary for a block of keys over a span of time tiles and their totals) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui().
8
+ East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui().
9
9
 
10
10
  # East Execution Engine (e3)
11
11
 
@@ -70,10 +70,14 @@ What do you need?
70
70
  │ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
71
71
  │ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
72
72
  │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
73
- │ ├─ Audited state → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
74
- │ │ ├─ Read by a time window or a 2D window → e3.recordIndex.interval | e3.recordIndex.area
75
- │ │ └─ Read a block of keys over time tiles, or their totals → e3.recordIndex.series | e3.recordIndex.summary
76
- │ ├─ A record's type changes → e3.migration.value | rows | rekey(name, record, fn, { after? })
73
+ │ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
74
+ │ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
75
+ │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) })
76
+ │ │ ├─ Read by a time window or a 2D window → index.interval | index.area
77
+ │ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
78
+ │ │ └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
79
+ │ │ → index.nested({ collections, tile, summary? })
80
+ │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order
77
81
  │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
78
82
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
79
83
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
@@ -339,17 +343,26 @@ const shout = e3.task('shout', [greet.output],
339
343
  - **An East function written in python, or in another node package**:
340
344
  `East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
341
345
  resolves it and embeds its IR, so the task is pure IR on any runner, its
342
- locations kept so an error or a profile names the function's own lines. A uv
346
+ locations kept so an error or a profile names the function's own lines (so
347
+ moving them re-runs the task: **Caching**, below). A uv
343
348
  workspace member is found by name in `uv.lock`, and its root module's
344
- `east_functions` dict is exported with `east-py export-functions` (from the
345
- project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
346
- `package.json` name, and its built `./functions` export's `eastFunctions` is
347
- exported with `east-node export-functions` (`EAST_NODE`). A package built
348
- elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
349
- which wins. The declared type must equal the exported one exactly, and each
350
- platform function it calls must be provided by the task's runner, by name or
351
- by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
352
- `east-py-io` ≡ `@elaraai/east-node-io`).
349
+ `east_functions` dict is exported with `east-py export-functions` (`EAST_PY`
350
+ when set, else the project's `.venv`, else PATH); an npm member is found by
351
+ its `package.json` name, and its built `./functions` export's `eastFunctions`
352
+ is exported with `east-node export-functions` (`EAST_NODE`). Each member is
353
+ exported in one process however many tasks import from it, the members side
354
+ by side, as many at once as the export has cores (a deploy's `-j`,
355
+ `e3.export`'s `cores`). A python member's manifests are kept in
356
+ `node_modules/.cache/e3/functions` under the directory the export runs in,
357
+ keyed on its sources and those of the local packages it depends on, the
358
+ workspace's `uv.lock` and `pyproject.toml`, the east-py exporting it and the
359
+ functions and providers each manifest is for, so a redeploy whose python has
360
+ not changed runs no export; deleting the directory clears it, and a node
361
+ member is exported every time. A package built elsewhere is passed as its
362
+ manifest — `{ functions: [...] }` / `--functions` — which wins. The declared
363
+ type must equal the exported one exactly, and each platform function it calls
364
+ must be provided by the task's runner, by name or by family (`east-py-std` ≡
365
+ `@elaraai/east-node-std` ≡ `east-c-std`; `east-py-io` ≡ `@elaraai/east-node-io`).
353
366
  - **Environments** are derived at export from each `{ custom }` platform: the
354
367
  providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
355
368
  or `package.json` + lockfile + `npm pack`) is captured into the package and
@@ -505,7 +518,7 @@ const change = e3.streamTask('change', {
505
518
  | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
506
519
  | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
507
520
  | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
508
- | `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
521
+ | `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
509
522
 
510
523
  ```typescript
511
524
  const process = e3.customTask('process', [rawData], StringType,
@@ -517,15 +530,19 @@ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
517
530
 
518
531
  ### Records — audited state, written only through mutations
519
532
 
520
- `e3.record(name, type, initialValue?)` is a dataset at `.records.<name>` that
521
- tasks read and react to like any input, but that only its **mutations** write:
533
+ `e3.record(name, type, { initial?, indexes?, mutations?, migrations? })` is a
534
+ dataset at `.records.<name>` that tasks read and react to like any input, but
535
+ that only its **mutations** write:
522
536
  a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
523
537
  body and no platform call, since the compare-and-swap retry runs it again
524
538
  against fresher state — named by an identifier. Each commit records its parent,
525
539
  the state, the mutation, its arguments, the actor and the time. Deploy mints a
526
540
  `$init` commit from the initial value, and a redeploy keeps state and history.
527
- The record's version vector carries the commit hash, so even a mutation that
528
- leaves the state unchanged triggers downstream tasks.
541
+ A large initial value is given as a function that makes it —
542
+ `initial: () => East.compile(generate, [])(50_000n)` — called once, as the
543
+ package is exported, never where the declaration is only read (a page binding
544
+ the record). The record's version vector carries the commit hash, so even a
545
+ mutation that leaves the state unchanged triggers downstream tasks.
529
546
 
530
547
  Declared with no initial value, a record is **unassigned** until its first
531
548
  state is committed — `e3 seed <repo> <record> <file.beast2> -w <ws>` from a
@@ -536,38 +553,54 @@ and a mutation on it is refused until then. So a record whose first state is
536
553
  data that arrives after the deploy — a customer list, a reference table —
537
554
  never runs its dependents on an empty value first.
538
555
 
556
+ A record is **declared whole**: its type, its initial value and its parts, in
557
+ one call, never added to after. Each part comes from a builder the call hands
558
+ its declaration, typed by the record's type: `indexes` gets `index` and returns
559
+ the indexes by name, `mutations` gets `mutation` and returns the mutations by
560
+ name, and `migrations` gets `migration` and returns its steps in order. Each is
561
+ checked where the record is declared, every refusal naming the record and the
562
+ part, and the record carries each by its name — `plans.mutations.add_plan`,
563
+ `plans.indexes.by_due` — which a package, a binding (`Record.bind(plans)`,
564
+ `Data.bindPaged(plans, { indexes: ['by_due'] })`) and TypeScript all read.
565
+
539
566
  | Form | Body | Use it when |
540
567
  |---|---|---|
541
- | `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
542
- | `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
543
- | `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
568
+ | `mutation.reduce(fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
569
+ | `mutation.edit(fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
570
+ | `mutation.patch()` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
544
571
 
545
572
  Every form commits a delta addressed by key, rewriting only the segments it
546
573
  touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
547
574
  takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
548
575
  value a row already holds changes nothing, and a `delete` or `update` of a key
549
576
  the record does not hold is a conflict naming the key: the caller's view of the
550
- record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
577
+ record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
578
+ mutation is named by its key, an identifier; a part's functions are typed by the
579
+ record's type, so name the type (`PlansType`) to write them, in the declaration
580
+ or beside it.
551
581
 
552
582
  ```typescript
553
583
  const PlanType = StructType({
554
584
  title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
555
585
  });
556
- const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
557
-
558
- const addPlan = e3.mutation.reduce('add_plan', plans,
559
- East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
560
- const next = $.let(state.copy()); // the state is frozen: copy first
561
- $(next.insert(id, plan));
562
- return next;
563
- }));
564
-
565
- const reschedule = e3.mutation.edit('reschedule', plans,
566
- East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
567
- ($, state, id, due, edit) => {
568
- const plan = $.let(state.get(id)); // decodes one segment, not the record
569
- $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
570
- }));
586
+ const PlansType = DictType(StringType, PlanType);
587
+
588
+ const plans = e3.record('plans', PlansType, {
589
+ initial: new Map(),
590
+ mutations: (mutation) => ({
591
+ add_plan: mutation.reduce(East.function([PlansType, StringType, PlanType], PlansType, ($, state, id, plan) => {
592
+ const next = $.let(state.copy()); // the state is frozen: copy first
593
+ $(next.insert(id, plan));
594
+ return next;
595
+ })),
596
+ reschedule: mutation.edit(East.function([PlansType, StringType, DateTimeType, e3.mutation.editType(PlansType)], NullType,
597
+ ($, state, id, due, edit) => {
598
+ const plan = $.let(state.get(id)); // decodes one segment, not the record
599
+ $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
600
+ })),
601
+ patch: mutation.patch(), // a view's save
602
+ }),
603
+ });
571
604
  ```
572
605
 
573
606
  Costs count in segments: a Dict or Set record is cut by key and by each entry's
@@ -575,61 +608,82 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
575
608
  cheap. A view reads every field of the rows it shows, so keep a bulky payload it
576
609
  does not display in a second record keyed the same way.
577
610
 
578
- #### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
611
+ #### Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
579
612
 
580
613
  A second collection over a Dict record, in another sort order, maintained in the
581
- same commit, so a view by an attribute is a page rather than a scan. `on`
582
- names row fields in index order: one field uses its value as the key, several
583
- use a Struct in that order, and a Set field gives an entry per element. It
584
- also carries field metadata for `Data.bindPaged(record, { indexes })` and
585
- `Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
586
- `keys` returns a Set, one entry per element (a row
587
- naming five resources appears five times, an empty set not at all); `value` is a
588
- projection a view renders without touching the record. The functions are pure
589
- and synchronous, since an index maintained commit by commit must equal its
590
- rebuild to the byte. The name is an identifier, never `primary` (the record's own
591
- order). Read through one with `index=<name>` on a page or key search; `e3
614
+ same commit, so a view by an attribute is a page rather than a scan.
615
+ `index.on([...])` names row fields in index order: one field uses its value as
616
+ the key, several use a Struct in that order, and a Set field gives an entry per
617
+ element. It also carries field metadata for `Data.bindPaged(record, { indexes })`
618
+ and `Slice.bindRecord` (see **e3-ui**). `index.key(fn)` gives one computed entry
619
+ per row; `index.keys(fn)` returns a Set, one entry per element (a row naming five
620
+ resources appears five times, an empty set not at all); the `value` option of
621
+ each is a projection a view renders without touching the record. The functions
622
+ are pure and synchronous, since an index maintained commit by commit must equal
623
+ its rebuild to the byte. Its name — its key in `indexes` — is an identifier,
624
+ never `primary` (the record's own order). Read through one with `index=<name>` on a page or key search; `e3
592
625
  reindex` rebuilds. A deploy builds an index the package declares and the state
593
626
  lacks, drops one it no longer declares, and runs nothing for the rest; a build
594
627
  is a split task over the record.
595
628
 
596
629
  ```typescript
597
- const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
598
- const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
599
- const byOwner = e3.recordIndex('by_owner', plans, {
600
- key: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
601
- value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
602
- });
603
- const byResource = e3.recordIndex('by_resource', plans, {
604
- keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
630
+ const plans = e3.record('plans', PlansType, {
631
+ initial: new Map(),
632
+ indexes: (index) => ({
633
+ by_due: index.on(['due']),
634
+ by_owner_due: index.on(['owner', 'due']),
635
+ by_owner: index.key(East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner), {
636
+ value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
637
+ }),
638
+ by_resource: index.keys(East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources)),
639
+ }),
640
+ mutations: (mutation) => ({ /* add_plan, reschedule, patch — above */ }),
605
641
  });
606
- const pkg = e3.package('planning', '1.0.0',
607
- plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
642
+ const pkg = e3.package('planning', '1.0.0', plans); // the record whole: its indexes and mutations with it
608
643
  ```
609
644
 
610
- #### Interval and area indexes — `e3.recordIndex.interval`, `e3.recordIndex.area`
645
+ #### Interval and area indexes — `index.interval`, `index.area`
611
646
 
612
647
  These indexes answer more than an order — as do the series and summary indexes
613
- below — each with a constructor of its own whose keys the SDK generates;
614
- `e3.recordIndex` refuses an `interval` option, naming its constructor. Underneath each is an ordinary index — the same build,
648
+ below — each with a builder method of its own whose keys the SDK generates.
649
+ Underneath each is an ordinary index — the same build,
615
650
  mutation, storage, paging and `e3 reindex` — and the record Slice reads a
616
651
  window through either (see **e3-ui**). Their generated keys are query
617
652
  machinery: edits and sessions always target the record's primary key.
618
653
 
619
- **`e3.recordIndex.interval(name, record, { on?, from, to })`** keeps events
620
- visible while they overlap a selected time window. `from` and `to` are the
621
- row's DateTime or Option<DateTime> fields; optional `on` fields are exact
622
- leading choices:
654
+ A generated key is a struct whose own fields are e3's: `on` holds the
655
+ declaration's leading fields, by their names, at their key types (a Set field's
656
+ element type) — a struct with no fields when it lists none — and the kind's own
657
+ parts sit beside it. So no name is reserved: a row may name a field anything,
658
+ `area`, `tile` and `__area` among them, and an index may be led by it.
659
+
660
+ | Index | Key | e3's parts |
661
+ |---|---|---|
662
+ | interval | `{ on, node, at }` | the tree node an entry is filed at; the instant it is ordered by, at the `from` field's type |
663
+ | area | `{ on, area }` | the cell's Morton code times 32 plus its level; −1 for a row not yet placed |
664
+ | series | `{ on, key, level, tile }` | the value of the field `key` names; the level an event is filed at, and the start of that level's tile (#1433) |
665
+ | summary | `{ on, key, tile }` | the value of the field `key` names; the tile's start |
666
+
667
+ A definition's `on` is the leading fields alone; its kind's options say the
668
+ rest (`events.indexes.by_time.interval`, `racks.indexes.racks_by_area.area`,
669
+ `jobs.indexes.jobs_by_press_week.series`).
670
+
671
+ **`index.interval({ on?, from, to })`** keeps events visible while they
672
+ overlap a selected time window. `from` and `to` are the row's DateTime or
673
+ Option<DateTime> fields; optional `on` fields are exact leading choices:
623
674
 
624
675
  ```typescript
625
- const byTime = e3.recordIndex.interval('by_time', events, { from: 'start', to: 'end' });
626
- const byStatusTime = e3.recordIndex.interval('by_status_time', events, {
627
- on: ['status'], from: 'start', to: 'end',
676
+ const events = e3.record('events', EventsType, {
677
+ initial: new Map(),
678
+ indexes: (index) => ({
679
+ by_time: index.interval({ from: 'start', to: 'end' }),
680
+ by_status_time: index.interval({ on: ['status'], from: 'start', to: 'end' }),
681
+ }),
628
682
  });
629
683
  ```
630
684
 
631
- It stores at most three entries per event per leading-field combination,
632
- independent of duration. The record Slice adapter resolves a half-open
685
+ Its key is `{ on, node, at }`. It stores at most three entries per event per
686
+ leading-field combination, independent of duration. The record Slice adapter resolves a half-open
633
687
  `[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
634
688
  each matching original record occurs once, including events that started
635
689
  earlier or span the whole window. It never scans the record or guesses a
@@ -645,17 +699,18 @@ primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
645
699
  requires a matching interval declaration and the primary `entries` binding to
646
700
  confirm saves even after an edit leaves the selected window.
647
701
 
648
- **`e3.recordIndex.area(name, record, { on?, x, y, width?, height?, at?, cell })`**
649
- reads the rows whose box meets a 2D window — what a floor plan's viewport
650
- shows — the interval index's twin:
702
+ **`index.area({ on?, x, y, width?, height?, at?, cell })`** reads the rows
703
+ whose box meets a 2D window — what a floor plan's viewport shows — the interval
704
+ index's twin:
651
705
 
652
706
  ```typescript
653
- // racks: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
654
- const racksByArea = e3.recordIndex.area('racks_by_area', racks, {
655
- x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01,
656
- });
657
- const racksByHallArea = e3.recordIndex.area('racks_by_hall_area', racks, {
658
- on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01,
707
+ // RacksType: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
708
+ const racks = e3.record('racks', RacksType, {
709
+ initial: new Map(),
710
+ indexes: (index) => ({
711
+ racks_by_area: index.area({ x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
712
+ racks_by_hall_area: index.area({ on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
713
+ }),
659
714
  });
660
715
  ```
661
716
 
@@ -669,14 +724,13 @@ const racksByHallArea = e3.recordIndex.area('racks_by_hall_area', racks, {
669
724
 
670
725
  Each option takes only the row's fields of its type, so a wrong one does not
671
726
  compile, and the definition checks them again, naming the option, the field and
672
- its type; a row field named `__area` is refused, the name being the key's. The
673
- domain is the square ±`cell`·2²⁷ world units around zero (±1,048 km at a 0.01
674
- cell, which rounds to 2⁻⁷), in 29 levels. Each row is filed once, under the
675
- deepest level whose cell is at least twice its box's larger side, in the cell
676
- holding its box's centre: the key `{ …on, __area: Integer }`. A row not yet
677
- placed files under `__area` −1, which the record Slice reads as the unplaced
678
- clause; one outside the domain, not finite, or larger than half of it files
679
- under the root, which every window reads. The index **carries its rows**: each
727
+ its type. The domain is the square ±`cell`·2²⁷ world units around zero
728
+ (±1,048 km at a 0.01 cell, which rounds to 2⁻⁷), in 29 levels. Each row is
729
+ filed once, under the deepest level whose cell is at least twice its box's
730
+ larger side, in the cell holding its box's centre: the key
731
+ `{ on, area: Integer }`. A row not yet placed files under `area` −1, which the
732
+ record Slice reads as the unplaced clause; one outside the domain, not finite,
733
+ or larger than half of it files under the root, which every window reads. The index **carries its rows**: each
680
734
  entry's projection is the whole row, so a window reads the index alone and no
681
735
  record segment, and it takes no `value`. The record Slice reads a viewport,
682
736
  grown by an eighth of its larger side on each side, as at most 266 disjoint
@@ -684,37 +738,45 @@ searches per exact prefix, sent in one request, which together hold every row
684
738
  whose box meets it, each once — beside rows of the cells searched that it
685
739
  misses, which the reader draws or not.
686
740
 
687
- #### Series and summary indexes — `e3.recordIndex.series`, `e3.recordIndex.summary`
741
+ #### Series and summary indexes — `index.series`, `index.summary`
688
742
 
689
743
  What a planning board reads: the events on a block of keys — a Plan's
690
744
  resources — over a span of time, and, zoomed out past what a screen can draw,
691
- how busy each key is. Both file a row's event by the key it is on and by each
692
- time tile it touches — `day`, `week` (ISO, from Monday), `month`, `quarter` or
693
- `year`, in UTC, half-open — their key `{ …on, __tile, <key> }`. The tile leads
694
- the key, so a block of keys in one tile is one contiguous range:
745
+ how busy each key is. Both file a row's event by the key it is on and by time
746
+ tiles — `day`, `week` (ISO, from Monday), `month`, `quarter` or `year`, in UTC,
747
+ half-open — the key before the time, so a key's entries over any span are
748
+ contiguous: a series index's `{ on, key, level, tile }`, a summary index's
749
+ `{ on, key, tile }`.
695
750
 
696
751
  ```typescript
697
- // jobs: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
698
- const jobsByPressWeek = e3.recordIndex.series('jobs_by_press_week', jobs, {
699
- key: 'press', from: 'start', to: 'end', tile: 'week',
700
- });
701
- const jobsByPressMonth = e3.recordIndex.summary('jobs_by_press_month', jobs, {
702
- key: 'press', from: 'start', to: 'end', tile: 'month',
752
+ // JobsType: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
753
+ const jobs = e3.record('jobs', JobsType, {
754
+ initial: new Map(),
755
+ indexes: (index) => ({
756
+ jobs_by_press_week: index.series({ key: 'press', from: 'start', to: 'end', tile: 'week' }),
757
+ jobs_by_press_month: index.summary({ key: 'press', from: 'start', to: 'end', tile: 'month' }),
758
+ }),
703
759
  });
704
760
  ```
705
761
 
706
762
  | Option | |
707
763
  |---|---|
708
764
  | `key` | the field naming what an event is on: any field but a Set |
709
- | `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is in no tile; no end, or one at or before the start, is an instant, in the one tile holding it |
710
- | `tile` | `day`, `week`, `month`, `quarter` or `year` |
711
- | `on` | exact leading fields, before the tile; a Set field fans out |
712
-
713
- **A series index** is an ordinary index whose keys the SDK generates, an entry
714
- per tile an event touches, and it **carries its rows**, as the area index does:
715
- a block of keys over a span of tiles reads the index alone, a search a tile, in
716
- one request. **A summary index** holds, under each key, `{ count, ms }`: how
717
- many rows touch its tile, and the milliseconds of it they cover — a row's shares
765
+ | `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is filed nowhere; no end, or one at or before the start, is an instant |
766
+ | `tile` | a series index's finest level, and a summary index's tile: `day`, `week`, `month`, `quarter` or `year` |
767
+ | `on` | exact leading fields, before the key; a Set field fans out |
768
+
769
+ **A series index** is an ordinary index whose keys the SDK generates, and it
770
+ **carries its rows**, as the area index does. It files each event **once**, at
771
+ the finest level from its `tile` up to the year whose shortest tile holds the
772
+ event's length — a day, a week, 28 days, 90 days, 365 days — under that level's
773
+ tile holding its start; an instant at `tile`; past 365 days, under each year it
774
+ touches. An event filed at a level ends within the tile after the one holding
775
+ its start, so a reader reads a block of keys over any span from the index alone
776
+ as one range a key a level — that level's tiles from two before the span to one
777
+ after it — in one request, whatever the span's width. **A summary index** holds,
778
+ under each key and tile, `{ count, ms }`: how many rows touch the tile, and the
779
+ milliseconds of it they cover — a row's shares
718
780
  add up to its length, and an instant's is none. Each entry stands for every row
719
781
  of its key and names none, so its window's `key` is Null, a read of it joins
720
782
  nothing, and a read that needs its rows — a joined read, a placement
@@ -729,25 +791,150 @@ count below zero, or time with no row — fails the write, naming the total and
729
791
  `e3 reindex`, which rebuilds it; a total past the largest Integer fails the
730
792
  write or the build, naming a finer tile.
731
793
 
732
- An event touching more than 4,096 tiles — eleven years of days — is refused
733
- where it is written, naming it, its span and the tile: declare a coarser tile.
734
- Each option takes only the row's fields of its type, as the area index's do,
735
- and the definition checks them again; a row field named `__tile` is refused, the
736
- name being the key's, and so are a `value` and an `on` field that is the key,
737
- the start or the end: the key follows the tile, and the tiles are cut by the
738
- start and the end.
794
+ A series index files an event past a year under each year it touches, as many
795
+ as 4,096: one touching more is refused where it is written, naming it and its
796
+ span. A summary splits an event over its tiles, as many as 4,096 — eleven years
797
+ of days — and one touching more is refused the same way, naming the tile:
798
+ declare a coarser tile. Each option takes only the row's fields of its type, as
799
+ the area index's do, and the definition checks them again; a `value` is
800
+ refused, and so is an `on` field that is the key, the start or the end: the key
801
+ follows the leading fields, and the tiles are cut by the start and the end.
802
+
803
+ #### Nested series indexes — `index.nested`
804
+
805
+ What a planning board reads when a record's rows nest what it draws: a
806
+ machine's jobs and its load a day at a time, a hall's members, an assembly's
807
+ parts to any depth. One declaration a record lists every collection its rows
808
+ nest, each field with a builder saying what it holds. e3 files each item apart
809
+ from its row — once, by its collection, its row, its node and its time — keeps
810
+ each node's totals a summary tile at a time, and each parent's subtotals at
811
+ every level; and the record's other indexes carry the rows without those
812
+ collections, so a page of rows reads none of them and a window of items reads
813
+ no row.
814
+
815
+ ```typescript
816
+ // MachinesType: Dict<String, { name, site, status, jobs: Dict<String, Job>, load: Dict<DateTime, Float> }>
817
+ const machines = e3.record('machines', MachinesType, {
818
+ initial: new Map(),
819
+ indexes: (index) => ({
820
+ machines_in_time: index.nested({
821
+ collections: {
822
+ jobs: index.nested.items({ from: 'start', to: 'end' }), // things people drag: by id, their times fields
823
+ load: index.nested.values(), // a value per instant: keyed by the instant
824
+ },
825
+ tile: 'week',
826
+ summary: 'month',
827
+ }),
828
+ machines_by_site: index.on(['site']), // pages each machine without its jobs and load
829
+ }),
830
+ });
831
+ // Each member's load, and each hall's mean of its members', at every level from the day up.
832
+ const halls = e3.record('halls', HallsType, {
833
+ indexes: (index) => ({
834
+ halls_in_time: index.nested({
835
+ collections: { members: index.nested.rows({ load: index.nested.values({ rollup: 'mean' }) }) },
836
+ tile: 'day',
837
+ summary: 'month',
838
+ }),
839
+ }),
840
+ });
841
+ // Parts to any depth, each part's jobs filed under every part above it too.
842
+ const assemblies = e3.record('assemblies', AssembliesType, {
843
+ indexes: (index) => ({
844
+ assemblies_in_time: index.nested({
845
+ collections: {
846
+ parts: index.nested.rows({
847
+ jobs: index.nested.items({ from: 'start', to: 'end' }, { rollup: true }),
848
+ parts: index.nested.recursive(),
849
+ }),
850
+ },
851
+ tile: 'week',
852
+ summary: 'month',
853
+ }),
854
+ }),
855
+ });
856
+ ```
857
+
858
+ | Builder (`index.nested.…`) | A field holding |
859
+ |---|---|
860
+ | `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
861
+ | `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
862
+ | `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
863
+ | `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
864
+
865
+ TypeScript types `collections` field by field from the record's type alone: a
866
+ misnamed time, or a builder on the wrong kind of field, is an error where it is
867
+ written, and its message says what fits. The definition checks the same again,
868
+ in words, for a declaration TypeScript could not see. A record has one nested
869
+ series index, which takes no `on` and no `value`.
870
+
871
+ **What it files.** Every entry is keyed `{ collection, row, path, run, level,
872
+ tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
873
+ field names from the row down (`members` then `load`); the row's key; the Dict
874
+ keys from the row down to the node (`[]` for the row, `m-3` for a member); the
875
+ run; the level and tile; and the item. A place and a path are held as a key
876
+ holds a list, folded from the last: `.step (key="members", rest=.step
877
+ (key="load", rest=.end))`. Each run of a node is one contiguous range a level:
878
+
879
+ | Run | Holds |
880
+ |---|---|
881
+ | `items` | each item once, as a series index files an event — at the level its length fits, past a year under each year — carrying it (`.item .jobs (…)`); a value, and an `{ at }` item, at the declared tile alone |
882
+ | `totals` | where `summary` is declared, each node's `{ count, ms, values }` a summary tile at a time; for a Float or Integer collection, `values` is the sum, the least, the greatest and the last there, taken in key order |
883
+ | `subtotals(rollup)` | for `values({ rollup })`, every parent's subtotal by each rollup at each level from `tile` to the year, where a descendant's value touches the tile: a node holding values folds its own, a parent aggregates its children's, in key order |
884
+ | `subtree`, `subtreeTotals` | for `items(…, { rollup: true })`, each item under every ancestor too, `.under (path, id)`, and each ancestor's totals of them |
885
+
886
+ A rollup is `'sum'`, `'mean'` (a mean of means up the tree), `'min'`, `'max'`,
887
+ `'count'` (the leaf rows beneath that hold a value), a fold and an aggregate
888
+ named apart (`{ fold: 'last', aggregate: 'sum' }`, a balance), or several.
889
+
890
+ **Reading.** A reader searches the index's leading fields — `[collection, row,
891
+ path, run, level, tile]`, each as East prints it — as a range a row a level: a
892
+ block of fifty machines' jobs over three weeks is fifty times four ranges in one
893
+ `datasetFindKeys` request, and one `datasetGetRuns` read of the entries they
894
+ select, each job carried whole; a parent's week subtotals are one range; a
895
+ part's subtree is one range a level.
896
+
897
+ **The lean rows.** Beside a nested series index, every other index of the
898
+ record that carries nothing or the row carries the row without the listed
899
+ collections — each listed collection empty, child rows kept with theirs empty —
900
+ so a page of rows read without a join reads no nested item and no segment of
901
+ the record's own. An index with a `value` of its own carries its own. A
902
+ binding of the record's indexes lists the nested index by name beside any that
903
+ carry the lean rows — `Data.bindPaged(machines, { indexes: ['machines_by_site',
904
+ 'machines_in_time'] })` (see **e3-ui**).
905
+
906
+ **Its span.** Beside every nested series index e3 keeps its span, an index of
907
+ the record's named `<index>_span` (e3-types' `nestedSpanName`), built with it
908
+ and never declared: under each collection's place and each month its items or
909
+ values touch (`NestedSpanKeyType`, `{ collection, tile }`), how many of them
910
+ touch it and the milliseconds of it its items cover, summed across the rows as a
911
+ summary index's totals are, and a month let go once nothing touches it. A
912
+ collection's first and last month held are its span across the record — what a
913
+ Plan's horizon covers for the series reading it — and its months' counts are
914
+ how much it holds in each, which the horizon's strip draws as its bars (see
915
+ **e3-ui**). Reading it is one search a collection, an exact match on its
916
+ place, and one read of the first and last entry of each. An index the record
917
+ declares under that name is refused.
918
+
919
+ **Writes.** A row a write updates by a patch is maintained by the patch: e3
920
+ walks it along the declaration — each collection's Dict patch, and each child
921
+ rows' — and compares the entries of only the items it names, with the totals of
922
+ the summary tiles they touch and the subtotals of the tiles they touch at each
923
+ level. A replace on the way, which a recursive type's patch always is, and an
924
+ inserted or deleted row are compared whole. Both ways give the same delta, and a maintained
925
+ index is the one `e3 reindex` rebuilds, byte for byte.
739
926
 
740
927
  #### Migrations — how a record's type changes
741
928
 
742
929
  | Form | Function | Runs as |
743
930
  |---|---|---|
744
- | `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
745
- | `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
746
- | `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
931
+ | `migration.value(name, fn)` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
932
+ | `migration.rows(name, fn)` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
933
+ | `migration.rekey(name, fn)` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
747
934
 
748
- A record's steps are one chain: each names the step before it with `after` (a
749
- step of the same record, leaving it as this step's input type), passing the last
750
- step passes the chain, and the last step leaves the record as its declared type.
935
+ A record's steps are one chain: the list `migrations` returns, in order, each
936
+ step taking the record as the step before it leaves it, and the last leaving it
937
+ as its declared type — each checked where the record is declared.
751
938
  A step's name is an identifier, unique on the record, and a workspace records the
752
939
  steps it has applied by name — so moving a step's code, or re-exporting it with a
753
940
  newer e3, does not run it again, and an edit to an applied step's body is not
@@ -756,12 +943,14 @@ detected. The functions are pure, synchronous East.
756
943
  ```typescript
757
944
  const RowV1Type = StructType({ title: StringType });
758
945
  const RowV2Type = StructType({ title: StringType, owner: StringType });
759
- const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
760
-
761
- const addOwner = e3.migration.rows('add_owner', plans,
762
- East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
763
- // The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
764
- const pkg = e3.package('planning', '2.0.0', plans, addOwner);
946
+ const plans = e3.record('plans', DictType(StringType, RowV2Type), {
947
+ initial: new Map(),
948
+ migrations: (migration) => [
949
+ migration.rows('add_owner', East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' }))),
950
+ // The next version adds its step here, after this one.
951
+ ],
952
+ });
953
+ const pkg = e3.package('planning', '2.0.0', plans);
765
954
  ```
766
955
 
767
956
  What a deploy does with each record, decided before it writes anything:
@@ -819,7 +1008,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
819
1008
  | `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
820
1009
  | `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
821
1010
  | `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
822
- | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
1011
+ | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
823
1012
  | `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
824
1013
  | **Datasets** | |
825
1014
  | `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
@@ -853,7 +1042,10 @@ peak a unit of its stage has reached in the run, so a stage runs its first unit
853
1042
  alone and then fans out; on Linux and macOS a guard stops the newest unit when
854
1043
  the runners together pass the budget, and runs it again once it fits. A deploy
855
1044
  takes its `file` sources in `-j` at a time, their pieces' intake units running
856
- under the same budget. The commands that run East — `workspace deploy`,
1045
+ under the same budget. A source deploy, or a `watch`, exports the workspace
1046
+ packages its imported functions come from `-j` at a time, a process each, on
1047
+ this machine; against a server, as many at once as this machine has CPUs. The
1048
+ commands that run East — `workspace deploy`,
857
1049
  `dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
858
1050
  `reindex` and `watch` — take the flags for a local repository; against a server the CLI
859
1051
  refuses them, since the server runs the work under its own budget
@@ -864,7 +1056,11 @@ line once it is in: its size, time and rate, and how it was taken in
864
1056
  (`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
865
1057
  value that is not a collection), with why the first time a runner fell back to
866
1058
  another. A terminal also keeps a live line for the files in flight, with their
867
- pieces, the rate and the time left across them. Against a server, the deploy line
1059
+ pieces, the rate and the time left across them. A source deploy prints a line
1060
+ for each workspace package whose functions it links: `exported <n> function(s)
1061
+ of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
1062
+ unchanged since its last export` when the export cache held them. Against a
1063
+ server, the deploy line
868
1064
  says what the job is doing, such as migrating a record or building an index;
869
1065
  the deliveries then upload a few at a time, each printing a line once it is in,
870
1066
  and the live line says what the server's commit is doing with each in flight.
@@ -924,7 +1120,7 @@ naming the holder: its pid, boot id, command and when it took the lock.
924
1120
  |---|---|
925
1121
  | Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
926
1122
  | Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
927
- | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
1123
+ | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
928
1124
  | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
929
1125
  | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
930
1126
  | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
@@ -939,7 +1135,7 @@ takes a runner, `new LocalTaskRunner(repo)`.
939
1135
  |---|---|
940
1136
  | Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
941
1137
  | Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
942
- | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
1138
+ | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
943
1139
  | Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
944
1140
  | Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
945
1141
  | Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
@@ -1011,6 +1207,7 @@ package is served beside the standard ones:
1011
1207
  | `serveE3` option | Default | Effect |
1012
1208
  |---|---|---|
1013
1209
  | `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
1210
+ | `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
1014
1211
  | `persist` | `true` | `false` keeps repositories in memory, gone with the page |
1015
1212
  | `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
1016
1213
  | `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
@@ -1075,7 +1272,14 @@ not had, in place; one a newer e3 upgraded is refused, naming that release.
1075
1272
  environment — and its inputs' hashes, so a task re-runs only when one of those
1076
1273
  changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
1077
1274
  it names: a task whose code e3 cannot see changed, such as a platform
1078
- function's, or one reading data from outside e3). A change to an input re-runs its
1275
+ function's, or one reading data from outside e3). A task's program holds the
1276
+ source locations of its body and of every function it embeds, an imported
1277
+ python or node function's too, since its errors, a caught error's `stack` and
1278
+ the closures it returns name them: an edit that moves one of those lines — a
1279
+ comment or an import added above a function — re-runs the task. One that leaves
1280
+ them where they were — another function, the end of a module, another package —
1281
+ re-runs only the tasks whose environment it is in, their runner's `{ custom }`
1282
+ packages. A change to an input re-runs its
1079
1283
  consumers, even during a run. A split task is cached a unit at a time: each piece
1080
1284
  and each merge is its own execution, and pieces are cut by content, so an append
1081
1285
  or an insertion re-runs only the pieces it reaches and the merges above them.