@elaraai/east-codex-plugin 1.0.99 → 1.0.100

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/.build/mcp/server.js +3 -2
  2. package/.codex-plugin/plugin.json +1 -1
  3. package/index.json +12207 -216
  4. package/package.json +1 -1
  5. package/skills/e3/SKILL.md +29 -802
  6. package/skills/e3/reference/browser.md +93 -0
  7. package/skills/e3/reference/cli.md +120 -0
  8. package/skills/e3/reference/from-code.md +48 -0
  9. package/skills/e3/reference/rack.md +139 -0
  10. package/skills/e3/reference/record-indexes.md +326 -0
  11. package/skills/e3/reference/record-migrations.md +63 -0
  12. package/skills/e3/reference/repository.md +25 -0
  13. package/skills/east/SKILL.md +30 -459
  14. package/skills/east/reference/beast2.md +71 -0
  15. package/skills/east/reference/cross-language-and-json-schema.md +87 -0
  16. package/skills/east/reference/queries.md +46 -0
  17. package/skills/east/reference/runners-and-ir.md +148 -0
  18. package/skills/east/reference/trees-and-recursion.md +56 -0
  19. package/skills/east/reference/values-in-typescript.md +62 -0
  20. package/skills/east-py/SKILL.md +28 -1647
  21. package/skills/east-py/reference/beast2.md +108 -0
  22. package/skills/east-py/reference/cli.md +51 -0
  23. package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
  24. package/skills/east-py/reference/control-flow-and-queries.md +147 -0
  25. package/skills/east-py/reference/cross-language.md +70 -0
  26. package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
  27. package/skills/east-py/reference/patterns.md +369 -0
  28. package/skills/east-py/reference/performance.md +70 -0
  29. package/skills/east-py/reference/value-methods.md +306 -0
  30. package/skills/east-py/reference/values-construction-and-json.md +88 -0
  31. package/skills/east-py/reference/writing-east-functions.md +275 -0
  32. package/skills/east-py-datascience/SKILL.md +11 -28
@@ -0,0 +1,108 @@
1
+ # Beast2 streaming: collections larger than memory
2
+
3
+ `write_beast2_file` / `open_beast2_file`, parallel writes and splicing, the writers and page readers. Part of the `east-py` skill.
4
+
5
+ ## Beast2 streaming — bounded-memory collections (`from east.serialization.beast2 import ...`)
6
+
7
+ Beast2 v5 encodes a large Array/Set/Dict as an append-only stream of
8
+ independently decodable segments: writer memory is one segment, never the whole
9
+ collection, and the decoders read v4 and v5 through the same entry points. The
10
+ canonical writers — the managed file writer, `Beast2ElementWriter`,
11
+ `encode_beast2_paged_for` — cut segments by one content-defined rule, the same in
12
+ TypeScript and east-c, so a value's bytes are its own whoever writes them;
13
+ `Beast2Writer` makes each batch you give it a segment instead.
14
+
15
+ **Managed files — start here (`open_beast2_file` / `write_beast2_file`).** Path
16
+ in, East values out. The file is self-describing, so a read needs no declared
17
+ type (a write does; a type declared on a read is validated at open). The file
18
+ object owns the fd and the mmap (closed on `with`-exit), east-c does every byte,
19
+ and the segments are the canonical ones — no buffers, iterators or batch sizes in
20
+ your code. The read flavour mirrors the root collection's read surface name for
21
+ name.
22
+
23
+ | Signature | Description |
24
+ |-----------|-------------|
25
+ | `write_beast2_file(path, T, value, *, codec="deflate")` | One call writes a collection of any size as one indexed v5 file — the value's canonical blob: segments where the content-defined cut rule places them, bounded in elements and bytes (wide rows never pile into one), key-disjoint for a Dict/Set. The bytes TypeScript and east-c write, and what `encode_beast2_paged_for(T)(value)` returns |
26
+ | `open_beast2_file(path, T, mode="w", *, codec=)` | The streaming writer: `.write()` takes East collections or python `list`/`dict`/`set` batches of any size, which add up to one canonical blob (a Set/Dict batch continues strictly ascending from the last); `.segments` counts the segments closed so far |
27
+ | `open_beast2_file(path, T=None, *, project=None)` | The read: `Beast2ArrayFile` / `Beast2DictFile` / `Beast2SetFile`, a READ-ONLY East collection VALUE subclassing `EastArray`/`EastDict`/`EastSet` — `isinstance`/`type_of` answer, every eager method works (streamed, below), mutation raises, and it binds into functions or passes into compiled calls by reference, where keyed reads answer from the pager, one frame per hit or miss, through a byte-budgeted segment cache (`EAST_PAGED_CACHE_BYTES`); `close()` defers while a bind holds it. `T` is optional (the header supplies it, also as `f.wire_type`); a declared `T` is checked against the header at open, never decoded as garbage |
28
+ | `read_beast2_type(source) -> EastType` | The root type of any beast2 blob (v4 or v5), from a path or a buffer, nothing decoded — for a file you know nothing about |
29
+ | `write_beast2_file_parallel(path, T, partitions, produce, *, processes=, strategy="auto", codec=, keep_shards=False, verify=False)` | N workers, ONE file: `produce(partition)` returns that partition's batches (or one collection); each worker writes a shard cut canonically from its start, and the shards splice **in partition order** as they finish. `strategy="auto"` forks on Linux/macOS — what `produce` closes over is inherited copy-on-write, so build the expensive context before the call (and before starting threads) — and runs inline on Windows, byte-identically. Any worker failure fails the call with the worker's traceback and leaves nothing behind |
30
+ | `splice_beast2_files(path, T, sources, *, verify=False) -> (segments, elements)` | Merges indexed v5 files by **byte copy** — east-c parses the geometry, `os.sendfile` moves the frames, nothing re-encodes. `sources` may be a lazy generator (spliced as they complete, in row order). Each must be v5, indexed and self-contained with an identical type section; a refusal names the path and leaves no destination. `verify=True` re-walks the result with east-c's strict reader |
31
+ | `f.load()` | The whole collection, decoded in east-c off the mmap — input memory stays one segment at any size; also the mutable escape hatch, like `f.copy()` |
32
+ | `f.segments()` | DEPRECATED — the file IS its collection value; still works (with a warning) for per-batch migration code |
33
+ | `len(f)` · `f.segment_count` · `f.self_contained` · `f.indexed` | O(1) from the trailing index, exact for every root kind |
34
+ | Array: `f[i]` / `f[a:b]` · `f.get(i)` ❗bounds · `f.get_or_default(i, d)` · `f.try_get(i)` · `f.has(i)` · `f.slice(a, b)` · `f.get_keys(rows)` | `EastArray`'s names, signatures and errors; a point read decodes only the owning segment, `get_keys` each owning segment once |
35
+ | Dict: `f[k]` ❗KeyError · `f.get(k[, default \| fn(k)])` · `f.get_or_default(k, d)` · `f.try_get(k)` · `f.has(k)` / `k in f` · `f.get_keys(keys, fill)` · `f.items()` / `values()` / iteration (streaming) · `f.keys()` (the Set) · `f.size()` — Set: `x in f` / `f.has(x)` | east-c binary-searches the segment *fences* (each segment's first key, from a bounded probe, cached), then decodes ONLY the owning segment (a small LRU keeps hot ones); `get_keys` merges the sorted keys against the fences and calls `fill` per missing key. The first keyed read verifies the fences: a corrupt or pre-contract blob raises `segments are not disjoint ascending key ranges` instead of reporting false misses |
36
+ | Array sorted search: `f.find_sorted_first/last(target)` → a global index · `f.find_sorted_range(target)` → `{start, end}` | `EastArray`'s contract over the whole file, only the boundary segment decoded; no `key=` projection (the file pages in element order) — pair with `f.slice(start, end)` |
37
+ | Compute: `map` `filter` `filter_map` `first_map` `reduce` `scan` `map_reduce` `sum` `mean` `maximum` `minimum` `every` `some` `find_*` `is_sorted` `to_set` `unique` `to_dict` `to_array` `to_columns` `map_batches` `string_join` `flat_map` (Array) / `flatten_to_array` (Set, Dict) `flatten_to_set` `flatten_to_dict` `for_each`, the whole `group_*` family, Set algebra | Each segment runs the ordinary eager method, and partials combine in east-c in stream order: order-dependent folds thread ONE accumulator and grouped folds seed each segment from the running per-group accumulators, so results equal `load()` exactly, float order included. Array `(el, idx)` callbacks, and the indices `find_*` / `group_find_*` report, are GLOBAL; `first_map`/`some`/`every`/`is_superset_of` stop decoding at the answer. Dict/Set compute streams disjointness-verified segments. A re-keyed collision (`to_dict`, `flatten_to_dict`, `group_to_dicts`) combines left-associatively in stream order — use an associative `combine`. `sort`/`reverse`/`copy`/`concat`/`union` materialize the collection — `load()` first |
38
+ | Column projection — INFERRED on the compute family; EXPLICIT with `open_beast2_file(path, project=NARROW)` | The compute family builds its callbacks FIRST and decodes each segment to just the struct fields their IR reads (the rest are hopped over, never built — building values, not walking bytes, is what decoding costs). Fields subset by name at any depth; a subtree used any other way stays whole, so results never change; Dict keys and Set elements never narrow. Runner-opened task inputs infer the same from the body's loop IR. What cannot be inferred decodes whole and is COUNTED in `eager_stats()` (`beast2_segments_projected/whole`, `beast2_projection_declined_*` by reason) — a projection that silently stopped applying would be an invisible cliff. The explicit form serves the subset to EVERY read (point reads, keyed gets, `load()`); `project` must be a subset of the wire type (a missing field raises `ValueError` naming it), a declared `T` keeps its meaning, and `find_sorted_*` refuse under it. A segment decoded under one mask is never served to a read needing more. No wire change |
39
+ | Degraded blobs | v4 → refused (`decode_beast2_with_header_for` still decodes it whole); index-less v5 → `load()` works, random access refuses; not self-contained → point reads refuse |
40
+ | **Manifest directories** — one object per segment plus a manifest naming them: how e3 stores a collection, and stages a task input |
41
+ | `Beast2ManifestWriter(T, path, *, codec="deflate")` | The canonical writer (a context manager): `.add(el)` / `.add_all(batch)`, as `Beast2ElementWriter` takes them; each segment goes to `<path>.segments/<sha256>.beast2`, a standalone blob under the shared header, and `.close()` writes the manifest to `path` — the directory TypeScript's `Beast2ManifestWriter` and east-c write. A writer left by an exception writes no manifest; `.segments` counts them; east-c does every byte, hashing included |
42
+ | `load_beast2_manifest(path, T=None)` ❗ | The whole collection, segment by segment in east-c — the value its segments spliced into one blob decode to. `ValueError` when `path` holds no manifest, the types differ, or a segment is missing or malformed |
43
+ | `read_beast2_manifest(source)` | The manifest — `kind`, `level`, `type`, `rule`, `header`, `entries` (each `{hash, fence, count, bytes}`) — or `None`; a blob holding a value is read no further than its type section |
44
+
45
+ ```python
46
+ from east.serialization.beast2 import open_beast2_file, write_beast2_file
47
+
48
+ write_beast2_file(path, rows_t, rows) # any size, one call
49
+
50
+ with open_beast2_file(path) as f: # self-describing: type from the
51
+ row = f[1_234_567] # header (f.wire_type); pass rows_t
52
+ totals = f.group_sum(lambda b, r: r["sku"], # instead to VALIDATE it at open
53
+ lambda b, r: r["qty"]) # whole-file compute: segment folds,
54
+ top = f.maximum(by=lambda b, r: r["qty"]) # never materialized, == load() exactly
55
+ table = f.load() # whole table when you truly need it
56
+
57
+ # The file IS a collection value: bind it into a function and the
58
+ # compiled body's keyed reads answer from the pager — one frame per lookup.
59
+ with open_beast2_file("table.beast2") as t: # Dict<String, Float>
60
+ lookup = East.function([StringType, DictType(StringType, FloatType)], FloatType,
61
+ lambda b, k, d: d.get_or_default(k, 0.0)).bind(t)
62
+ joined = rows.map(lambda b, r: r.v + lookup(r.k)) # loop + callee + pager: all east-c
63
+ ```
64
+
65
+ **Buffer level — you hold bytes, not a path. `_for` is NOT `_with_header_for`:**
66
+
67
+ | Signature | Description |
68
+ |-----------|-------------|
69
+ | `encode_beast2_with_header_for(T, *, version=None)` / `decode_beast2_with_header_for(T)` | **The one you want**: the self-describing container — magic, type schema, value. Encode writes the current default container (v5; `version=4` only for a reader that predates it); decode reads v4 and v5, whole (Set/Dict segments must hold the canonical value — sorted, disjoint — or the blob is rejected as corrupt). ❗The header's type must be `T`, or a subtype whose variant cases line up with `T`'s (a `none` reads as any `Option`), else `ValueError: beast2: cannot decode a blob of type … as …` before anything decodes — the words TypeScript and east-c use |
70
+ | `encode_beast2_for(T)` / `decode_beast2_for(T)` | **Headerless** type-directed bytes: the reader must know `T` exactly, and mutable containers (Array/Set/Dict/Ref) are refused. In TypeScript `encodeBeast2For` is the FULL container — do not port a call site by name |
71
+ | `Beast2ElementWriter(T, stream, *, codec="deflate", parallel=False)` | The canonical writer (a context manager): `.add(el)` — an Array/Set element or a Dict `(key, value)` pair — and `.add_all(batch)` (the loop in east-c); Set elements and Dict keys strictly ascending in East order. Segments fall where the cut rule places them — the bytes the managed writer, TypeScript and east-c write; memory is one open segment; `.close()` writes the last segment, the terminator and the index; `.segments` counts the closed ones. An element that does not ascend, or fails to encode, raises and leaves the writer as it was |
72
+ | `encode_beast2_paged_for(T, *, codec="deflate") -> (value) -> bytes` | A whole collection through the canonical writer — the write side of `open_beast2_pages_for` |
73
+ | `Beast2RunSorter(T, open_run, *, merge=None, union=False, codec="deflate", parallel=False)` | A Set's or Dict's elements in ANY order in, sorted canonical runs out: `.add(el)` encodes at once; at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES` it sorts the run — stably, so a key's values keep the order they came in — folds a repeated key (`merge`, a compiled `(K, V, V) -> V` East function, for a Dict; `union=True` for a Set; neither: it raises) and writes the run to `open_run(run)` (anything with `write(bytes)` and `close()`) as its value's canonical blob. `.finish()` writes the last run; `.runs` counts them. The runs are the ones east-c and TypeScript write; a sink's exception comes back out of the call that wrote the run; a key repeated across runs is the merge's to fold |
74
+ | `Beast2Writer(T, stream, *, codec="deflate", self_contained=True, index=True, parallel=False)` | Segments of your choosing (a context manager): `.write(batch)` appends one segment per non-empty batch; `.close()` writes the terminator and the index; `.segments` counts. Set/Dict batches must ascend strictly across the stream. **Keep both defaults** — `index` and `self_contained` are what `open_beast2_pages_for` needs, and turning either off silently forfeits random access. `codec="none"` for already-compressed payloads or raw write throughput |
75
+ | `encode_beast2_segments_for(T, **opts) -> (batches) -> bytes` | In-memory `Beast2Writer`: one segment per non-empty batch |
76
+ | `encode_beast2_v5_for(T, *, codec="deflate", index=False) -> (value) -> bytes` | A whole value of any root type as v5; decode with `decode_beast2_with_header_for` |
77
+ | `iter_beast2_segments_for(T) -> (source) -> iterator` | One decoded collection per segment, O(segment) memory; `source` is bytes, an `mmap` or a binary stream |
78
+ | `read_beast2_index(T, blob) -> (segments, elements) \| None` | O(1) totals from a v5 blob's trailing index |
79
+ | `open_beast2_pages_for(T) -> (source) -> Beast2Pages` | Random access: `.segment_count` `.element_count` `.self_contained` `.counts`, `.segment(i)`, `.element(row)` (also `len()` / `[]`), each decoding ONE segment. ❗Needs `index=True` and `self_contained=True` (the defaults); `.element()` is for Array roots. ❗Borrows the buffer — keep it (and an mmap) alive for the pages' lifetime, or use `open_beast2_file`, which owns it |
80
+
81
+ **Batch size (`Beast2Writer` only — the canonical writers cut by the rule).** A
82
+ batch is at once your memory ceiling, one segment, one compression window and
83
+ the granularity of random access. ~1000 rows is a good default: one row per
84
+ batch costs about 4× the bytes, and the curve is flat past ~100. `write()` takes
85
+ a batch, never a row — accumulate and flush yourself, or hand rows to
86
+ `Beast2ElementWriter`, which needs no batch at all.
87
+
88
+ ```python
89
+ import mmap
90
+ from east.serialization.beast2 import (
91
+ Beast2Writer, iter_beast2_segments_for, open_beast2_pages_for,
92
+ )
93
+ rows_t = ArrayType(row_type)
94
+
95
+ with open(path, "wb") as f, Beast2Writer(rows_t, f) as w:
96
+ for batch in produce_batches(): # each an EastArray(row_type, ...)
97
+ w.write(batch) # O(batch) memory, one segment each
98
+
99
+ # Sequential re-read. mmap, NOT .read() — .read() pulls the whole file in and
100
+ # throws away the bounded-memory property you just paid for.
101
+ with open(path, "rb") as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
102
+ for batch in iter_beast2_segments_for(rows_t)(mm):
103
+ consume(batch) # O(segment) decoded at a time
104
+
105
+ # Or jump straight to a row — decodes only the segment that owns it.
106
+ pages = open_beast2_pages_for(rows_t)(mm)
107
+ row = pages.element(1_234_567)
108
+ ```
@@ -0,0 +1,51 @@
1
+ # The east-py CLI
2
+
3
+ Part of the `east-py` skill.
4
+
5
+ `east-py` (the `east-py-cli` package) runs East IR with python platform
6
+ functions — the runner e3 drives for `runtime: 'east-py'` — and carries the
7
+ authoring tools:
8
+
9
+ ```bash
10
+ east-py run prog.beast2 -p east-py-std -i a.beast2 -i b.json -o out.beast2 # -v: timings, peak memory
11
+ east-py run prog.beast2 -p east-py-std --profile # time per East and platform function
12
+ east-py exec unit.beast2 # one unit: the runner protocol e3 speaks
13
+ east-py convert value.beast2 # a beast2 value as East text (-o value.east to write it)
14
+ east-py transpile prog.beast2 -o prog.py # IR → a python module that rebuilds it
15
+ east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
16
+ east-py lint src/ ; east-py check src/ ; east-py lsp
17
+ east-py version -p east-py-std # the runner's version, and each package's
18
+ ```
19
+
20
+ | Command | Does |
21
+ |---|---|
22
+ | `run <ir> [-p PACKAGE]… [-i FILE]… [-o FILE] [--decode lazy\|whole] [-v] [--profile]` | Compiles an IR file (`.beast2`, `.beast`, `.east` or `.json`) and calls it with one `-i` file per parameter, in order, each decoded by its extension to the parameter's type and FROZEN (a task input is immutable). An indexed beast2 collection input opens lazily, whatever it weighs — mapped, one segment decoded at a time, and decoded whole, once, when an operation the pager cannot serve first needs it — and a manifest file opens as the collection it names; `--decode whole` (east-c's and east-node's flag too) decodes every input before the program runs instead. `-o` writes the result in its extension's format (a collection as canonical indexed beast2); without it the result prints as East text. `-v` adds timings, peak memory and how each input was read: a lazy input's segment reads, or the resident memory a whole decode added |
23
+ | `exec <unit> [-v] [--profile]` | The runner protocol: one unit file says what to run — a program over its inputs, read as its `decode` says (as `run --decode`), or a merge of an output's parts — and where; the output is written by its kind and a result recorded. Exit 0 when it succeeded, 1 when it failed (the message and locations also go to stderr), 2 when the unit cannot be read or the result written. Relative paths in the unit are relative to its directory |
24
+ | `convert <file> [-o FILE.east] [-v]` | Decodes a beast2 value (its header names the type) and prints it as East text, or writes it to a `.east` file |
25
+ | `transpile <ir> [-o FILE] [--name NAME] [-p PACKAGE]…` | IR as a python module rebuilding it (`--name` binds it, default `main`; each `-p` prints its calls as that package's own functions) — [IR ↔ python](ir-and-diagnostics.md#ir--python-east-py-transpile-and-the-east-c-ir-toolbox) |
26
+ | `export-functions <module> -o FILE [-p PACKAGE]… [--name NAME] [--package-version V] [--only NAME]…` | A module's (a dotted name, or a `.py` path) `east_functions` as a function manifest; every platform call must be provided by a `-p` package; `--name` defaults to the module's top-level name, the version to the installed distribution's (else `0.0.0`); `--only` exports just those, building only them and what they call (a sibling that does not build fails nothing). `--batch FILE` in place of `-o`/`-p`/`--only` writes several manifests from one import, FILE a beast2 batch (`FunctionExportBatchType`: each entry's `output`, `packages` and `only`) — what `e3.export` runs per workspace package — [Cross-language functions](cross-language.md#cross-language-functions-east-py-export-functions-and-eastimport_function) |
27
+ | `lint [paths…] [--format text\|json] [--disable RULE]… [--exclude DIR]… [--list-rules]` | The rules (default path `.`); exit 1 on any finding, a warning included, 2 on a usage error — [Diagnostics](ir-and-diagnostics.md#diagnostics-at-edit-time--east-py-lint-east-py-check-flake8-east-py-lsp) |
28
+ | `check <targets…> [--format text\|json] [--only-if-enabled]` | The build's own errors; exit 1 on any |
29
+ | `lsp [--probe]` | The Language Server over stdio (needs `pygls`); `--probe` reports whether it can start here |
30
+ | `version [-p PACKAGE]…` | The `east-py-cli` and `east-py` versions (and the Cython extensions loaded), and each package's version and function count |
31
+
32
+ A `-p` package is a python module name (`east-py-std` imports `east_py_std`)
33
+ exporting a top-level `platform` list; a project's own module is loaded the same
34
+ way (see [Project-owned platform module](patterns.md#project-owned-platform-module-calling-your-python-from-e3)).
35
+ `run` and `exec` take `--exit-with-parent`: exit once stdin reaches end of file,
36
+ for a parent that holds a stdin pipe it never writes to.
37
+
38
+ **Profiling.** `run` and `exec` take `--profile`, east-c's profiler and report:
39
+ after the work (and after a failure's error), every East function and platform
40
+ function it called, by self time, with its calls, its total time and where it
41
+ is — the `file:line:column` it was written at and, for a helper called inside
42
+ another body, where it was called. A platform function (torch, a solver) is an
43
+ entry of its own, so its time is not its caller's, and so is each input read
44
+ lazily, named as `-v` names it (`input 1`): the segments its reads decode, a
45
+ call each — keyed reads at random beyond the segments the runner keeps, or a
46
+ scan repeated, show there, not in the reading function's self time. `EAST_PROFILE=1` profiles
47
+ without the flag — e3 hands its environment to the runner, so a task's report
48
+ lands in its log (`e3 task logs`) — and `EAST_PROFILE_INTERVAL=N` prints the
49
+ profile so far every N seconds while the work runs. A python function linked
50
+ into a TypeScript task carries its locations through the manifest, so the
51
+ report names its python lines.
@@ -0,0 +1,87 @@
1
+ # Columnar escape hatches, platform functions, and what is not east-c
2
+
3
+ Part of the `east-py` skill.
4
+
5
+ ## Columnar escape hatches — when the logic must stay python
6
+
7
+ When per-element logic genuinely needs python (numpy, a model, an external
8
+ library), don't touch rows one at a time — cross the boundary **once per
9
+ column** instead of once per row × field:
10
+
11
+ ```python
12
+ cols = rows.to_columns() # {"price": np.float64[...], "qty": np.int64[...],
13
+ # "sku": [str, ...] (interned), ...} — one crossing/column
14
+ amount = cols["price"] * cols["qty"].astype(np.float64) # vectorised numpy
15
+ out = EastArray.from_columns(Out, {"sku": cols["sku"], "amount": amount})
16
+
17
+ result = rows.map_batches(f, out=Out, batch_size=50_000) # f sees columnar chunks;
18
+ # batches may shrink (filter-like)
19
+
20
+ acc = EastDict(StringType, FloatType) # dicts as accumulators:
21
+ acc.update_many(keys, values, combine=lambda b, cur, new: cur + new) # one crossing,
22
+ # combine builds -> collisions in C
23
+ arr.extend(np_array) # bulk push, one crossing
24
+
25
+ solver_input = coerce_to( # struct fields take 1-D numpy
26
+ {"start_nodes": np_i64, "capacities": np_i64, ...}, # columns directly: each
27
+ MinCostFlowInputType) # Array<Int/Float/Bool> fills C-side
28
+ ```
29
+
30
+ `Float`/`Integer`/`Boolean` columns move through numpy buffers filled in C
31
+ (`Option<Float>` ↔ float64 with NaN for `none`); `String` columns box once
32
+ through a bounded intern table (repeated categories/ids come back as the
33
+ same python object); other field types fall back to boxed lists. The same
34
+ contract governs the INPUT direction: `from_columns`/`coerce_to` fill C-side
35
+ only when a numeric column arrives as a numpy array of the matching dtype
36
+ (float64/int64/bool; float64-with-NaN for `Option<Float>`) — a plain python
37
+ list converts per cell, so `np.asarray(col)` first when the source is a
38
+ list. Composition rule: **East functions for East-expressible transforms,
39
+ columns/batches for the genuinely-python remainder.**
40
+
41
+ **Put the logic in the platform function, not a pure-python shim.** A
42
+ `@East.platform_function` is just a typed, validated python function; its one
43
+ added cost, validating the declared output, is a feature. A separate helper layer
44
+ over `list`/`dict` that the platform function converts into and out of costs
45
+ **testability** (the typed `inputs`/`output` is the contract you test; untyped
46
+ helpers turn bugs into silent corruption instead of a named `EastTypeError`),
47
+ **portability** (a platform function over East values moves to an e3 task,
48
+ another runtime or a TypeScript mirror unchanged), a **forced sandwich**
49
+ (East→python on the way in, back on the way out) and **purity** (a platform
50
+ function pure in its East inputs is memoisable). Small reusable functions are
51
+ fine — have them take and return East values, pay output validation only at the
52
+ real East↔python edge, and never call a platform function per element in a loop.
53
+
54
+ ## Platform functions
55
+
56
+ | Signature | Description |
57
+ |-----------|-------------|
58
+ | `@East.platform_function(*, inputs, output, name=None, is_async=None, validate_output=True, validate_input=False)` | Register a Python fn; infers sync/async from the def (`is_async=` declares it when they differ — a function East calls asynchronously on every runtime whose python is a plain `def`); validates output against `output`. DUAL-MODE: called with values it runs the python (an async def's coroutine, to await); called inside an `East.function` / `East.asyncFunction` body it emits the `Platform` node with this signature — an async one inside a sync body is the build-time error. Paired at compile by name (the def's, or `name=`). Also importable bare: `from east import platform_function` |
59
+ | `@East.generic_platform_function(*, type_parameters, name=None, is_async=False, inputs=None, output=None, type_erased=False)` | Type-parameterized factory: the decorated fn is `fn(platform, *type_params) -> impl`; `is_async` is **explicit** (not inferred). With `inputs=`/`output=` (placeholders allowed anywhere: `output="T"`, `output=ArrayType("T")`) it is dual-mode too: `fs_open_beast(DictType(K, V), path)` in a body — the type arguments first, as TypeScript reads — emits the generic node; without them a body call is a build error naming what to declare. `type_erased=True` when the implementation ignores the type arguments (it reads the values): the decorated fn IS the implementation, python calls it directly (`causal_experiment(rows, config)`) and the factory the runtime binds is derived |
60
+ | a platform function RETURNING a lazy value | A `@platform_function` may return a paged hold — an `open_beast2_file(...)` value, `open_paged_file(T, path)` (a C-owned mapping), `fs_open_beast(...)`'s result — and the compiled body keeps its O(segment) cost model: the C value crosses the return seam by pointer (checked against the declared output type, never re-marshalled through python), and nothing python-side has to outlive it: the mapping is the value's own. The mirror of a paged ARGUMENT crossing in |
61
+ | `East.platform_functions(module) -> list` | Collects every decorated fn in `module` (pass `__name__`). Two consumers: `East.compile()` for in-process use, and a package's top-level `platform` list that `east-py run -p <module>` (and the e3 `{ custom }` runner) loads |
62
+ | `@memoize` / `@memoize(salt="…")` / `memoized = memoize(fn, salt="…")` | Content-addressed memo over ONE platform function. Apply **above** `@East.platform_function` (or inline on an imported one). Key = sha256(name + salts + per-input digests of the with-header BEAST2 encodings via the declared input types); value = with-header BEAST2 of the output, decoded via the declared output type. Inert by default |
63
+ | `configure_memo(directory, salt="")` | Activate (`None` deactivates) memoization for `@memoize` functions; overrides `EAST_MEMO_DIR` / `EAST_MEMO_SALT` env vars. Bump `salt` to invalidate after code edits — input-derived keys can't see them |
64
+
65
+ ## What is NOT east-c
66
+
67
+ Nearly everything above delegates to native builtins. What still runs python,
68
+ so you can reason about cost and semantics:
69
+
70
+ - **`EastMatrix.map_elements`** — DEPRECATED: a python callback per element,
71
+ the one shape the strict surface removes; it warns and will go. Use the
72
+ arithmetic / reduction / mask / sparse methods, `map_rows`, or
73
+ `to_numpy()`/`to_torch()`. (`Vector.map`/`reduce` and `Matrix.map_rows` are the
74
+ native VectorMap / VectorFold / MatrixMapRows builtins.)
75
+ - **`Dict.get_or_insert`** composes membership and get python-side so `fn` runs
76
+ only on a miss (lazier than East's strict default). The other singles
77
+ (`insert`/`update`/`swap`/`delete`/`try_delete`/`insert_or_update`) are the
78
+ native builtins, error messages included.
79
+ - **Reduction sugar** — `mean`, `group_mean`, `group_size`, …: several native
80
+ passes, zero python per element, but not one fused builtin (as in TypeScript).
81
+ - **Boundary utilities** — `coerce_to`/`assert_value_of`/`type_of`, `variant`/
82
+ `struct` validation, `match()` dispatch, `compare_for`/`make_east_key`: python
83
+ walkers, by design — they are the python↔East edge.
84
+ - **Iteration** — `for x in arr` / `list(arr)` boxes per element (lazily, under
85
+ East's iteration lock, so mutating during the loop raises `Cannot modify …
86
+ during iteration`, as an East for-loop does); cross once with
87
+ `to_columns`/`to_numpy` instead.
@@ -0,0 +1,147 @@
1
+ # Control flow, jq, and what a body may reference
2
+
3
+ The expression forms of control flow, `East.jq`, and the captures a body accepts or refuses. Part of the `east-py` skill.
4
+
5
+ ## Contents
6
+
7
+ - Control flow — the expression forms (`East.while_`, `East.for_`, …)
8
+ - Queries — `East.jq`
9
+ - What a body may reference
10
+
11
+ ## Control flow — the expression forms (`East.while_`, `East.for_`, …)
12
+
13
+ Prefer the collection methods: `map`/`filter`/`reduce`/`group_reduce`
14
+ express most work and are the fastest thing in the runtime. Reach here when
15
+ the next step **depends on the last** — a worklist, a BFS, a fixpoint, a
16
+ topological replay — which no data-parallel method can express. The
17
+ statement forms above (`b.while_`, `b.for_`) build the same nodes; these are
18
+ their EXPRESSION twins, usable inside a callback or a one-expression body.
19
+
20
+ Python cannot overload `=`, and `while`/`if` collapse to a `bool` before any
21
+ build sees them. That is the constraint that produced `East.if_else`, and the
22
+ answer is the same shape: **`while_` is to `while` what `if_else` is to `if`**.
23
+ The body is a pure function of the state that RETURNS the next state, and the
24
+ state struct IS the loop's local variables:
25
+
26
+ ```python
27
+ total = East.while_({"i": 0, "acc": 0},
28
+ cond=lambda b, s: s.i < n,
29
+ body=lambda b, s: {"acc": s.acc + s.i, "i": s.i + 1}).acc
30
+
31
+ # `East.if_else` is the `if`; a field left as `s.field` is the empty else;
32
+ # `{**s, …}` changes one field and keeps the rest
33
+ East.for_(rows, {"n": 0, "hi": 0.0},
34
+ lambda b, s, r: {"n": s.n + 1,
35
+ "hi": East.if_else(r.price > s.hi, r.price, s.hi)})
36
+ ```
37
+
38
+ This lowers to a `Ref` holding the state, a `While`/`For*` node whose body is
39
+ one `RefUpdate`, and a final read — the whole loop runs inside east-c. Every
40
+ construct is dual-mode like `East.if_else`: outside a build it runs the plain
41
+ python loop, so one body serves both a callback and a direct call on plain
42
+ East values. A worked example is in
43
+ [Key Patterns](patterns.md#sequential-logic-that-stays-in-east-c-worklist--replay).
44
+
45
+ | Call | Emits | Notes |
46
+ |---|---|---|
47
+ | `East.if_else(cond, value, …, otherwise)` | `IfElse` | cond/value pairs then the else — an if/elif/else chain is ONE node; exactly one arm evaluates |
48
+ | `East.while_(state, cond, body, label=…)` | `While` | `cond(b, s) -> Boolean`, `body(b, s) -> next state` |
49
+ | `East.for_(coll, state, body, label=…)` | `ForArray`/`ForSet`/`ForDict` | Array `body(b, s, el[, i])`, Set `body(b, s, el)`, Dict `body(b, s, k, v)` |
50
+ | `East.block(a, b, …)` | `Block` | evaluates in order, yields the last — the sequencing point for mutators (`East.block(lambda b: …)` is the statement form) |
51
+ | `East.let(value, fn)` | `Let` | `fn(b, bound)` — bind once, use many times, inside an expression body (`b.const` is the statement form) |
52
+ | `East.ref(v)` | `NewRef` | a cell — `.get()` / `.update(v)` / `.merge(v, fn(b, current, patch))` |
53
+ | `East.label(name=None)` | — | names a loop, for `break_`/`continue_` from a NESTED one |
54
+ | `East.break_(state=…, label=…)` / `East.continue_(state=…, label=…)` | `Break` / `Continue` | leave / next iteration, optionally committing a last state |
55
+ | `East.try_catch(body, handler, finally_=None)` | `TryCatch` | `body(b)`, `handler(b, message[, stack])`, same East type as `body` |
56
+ | `East.new_array/new_set/new_dict(…)` | `NewArray`/`NewSet`/`NewDict` | a FRESH collection per evaluation — the loop accumulator |
57
+ | `East.new_vector(T, values)` / `East.new_matrix(T, rows, cols, values)` | `NewVector`/`NewMatrix` | a fresh tensor from scalar expressions |
58
+
59
+ **Accumulate in place.** Threading a collection through the state rebuilds it
60
+ every iteration (`order.concat(…)` copies — O(n²) over the loop). The
61
+ mutators extend it in O(1). Each returns what its EAGER twin returns, so
62
+ sequence with `East.block`; a mutation written as a bare statement is
63
+ evaluated and thrown away, and the build says so rather than compiling a
64
+ loop that silently does nothing:
65
+
66
+ ```python
67
+ counts = East.for_(rows, {"counts": East.new_dict(StringType, IntegerType)},
68
+ lambda b, s, r: East.block(
69
+ s.counts.insert_or_update(r.sku, 1, lambda b, x, y: x + y),
70
+ s)).counts
71
+ ```
72
+
73
+ ⚠️ Seed accumulators with `East.new_array/new_set/new_dict`. A captured
74
+ `EastArray(T)` works too — a captured collection is a snapshot of its
75
+ contents, rebuilt wherever the body reads it, so a seed is fresh per call —
76
+ but mutating one anywhere ELSE changes a copy nothing else sees, and the
77
+ build raises rather than compile a mutation that persists nowhere.
78
+
79
+ ## Queries — `East.jq`
80
+
81
+ `East.jq(input, program, result_type)` is a jq query as East code: the jq is
82
+ parsed, checked against the input's East type and translated to ordinary East
83
+ IR when the program is built — loops, `Match`, the builtins — so it runs
84
+ wherever East runs, with no interpreter. In a body it is a call of the `Query`
85
+ builtin, which carries the program as written beside its translation and gives
86
+ the translation, so `to_python_source` prints it back as `East.jq(…)`. It is
87
+ the twin of TypeScript's `East.jq` and builds the same IR. Dual-mode, like the
88
+ standard library: in a body it builds the call; on values it compiles the
89
+ translation (once per query and input type) and runs it now.
90
+
91
+ ```python
92
+ from east import East, ArrayType, FloatType, IntegerType, StructType, array
93
+
94
+ Order = StructType([("id", IntegerType), ("total", FloatType)])
95
+ orders = array(Order, [{"id": 1, "total": 250.0}, {"id": 2, "total": 1200.0}])
96
+
97
+ # in a body: an expression of `result_type` (here an ArrayExpression), so its methods chain
98
+ count_big = East.function([ArrayType(Order)], IntegerType,
99
+ lambda b, orders: East.jq(orders, "[.[] | select(.total > 1000) | .id]",
100
+ ArrayType(IntegerType)).size())
101
+ count_big(orders) # 1
102
+
103
+ East.jq(orders, "map(.total) | add", FloatType) # 1450.0 — on values, run now
104
+
105
+ # several inputs: a dict, checked as an e3 root — `.orders` reads the `orders` input alone
106
+ East.jq({"orders": orders, "limit": 1000.0},
107
+ ".limit as $l | [.orders[] | select(.total > $l) | .id]", ArrayType(IntegerType)) # [2]
108
+ ```
109
+
110
+ | | |
111
+ |---|---|
112
+ | ❗ `result_type` | Required: the outputs' type for a query that gives one output (`length`, `[…]`, `reduce`), an `Option` of it for at most one (`first(f)`), an `Array` of it for any number (`.[] \| .id`). The query must check to exactly it, or the build raises `type_mismatch: the query gives Array<Float>, not the Integer it was given.` |
113
+ | A query that does not check | `QueryError` when the program is built (a `QueryBuildError`, an `ExpressionError` too), its diagnostics each `jq <line>:<column>: <message>`: `unknown_field: .totl is not a field of Struct{id: Integer, total: Float}. Did you mean .total?` |
114
+ | An error the query raises | `error(v)`, an integer `% 0`, a date that does not parse: an East runtime error located in the jq text (`jq 1:5`), which `b.try_` catches; on values, a `QueryError` with one `runtime` diagnostic |
115
+ | The input's type | In a body, the expression's; on values, `type_of(value)`. Where that is narrower than the type meant (a variant's is its one case), run `evaluate_jq(program, value, input_type=T)` |
116
+ | The language | jq 1.8 over East's types: Dict keys of any type, DateTime a type (an ISO literal compares with it), Integers exact, a variant read as `{type, value}`. Every departure is listed in `libs/east/devdocs/QUERY.md` §13 |
117
+ | Host tools (`from east.query import …`) | `check_jq(program, input_type, root=False)`, without building: the program as written (`.program`, a `JqType` tree whose constants are typed East values; `None` when it does not check), the type it was checked against (`.input_type`), its `.element_type` and `.multiplicity` (`"one"` / `"maybe"` / `"many"`; `None` when it does not check), the datasets a root query reads (`.reads`) and its `.diagnostics`. A query's types are never stored: check the program against its data wherever they are needed · `evaluate_jq(program, value, input_type=T)`: a query whose input type is data · `translate_jq(checked)` · `parse_jq` / `print_jq` / `lex_jq` · `QueryError` |
118
+
119
+ ## What a body may reference
120
+
121
+ - Its own parameters; plain scalar constants (closure floats / ints / strings /
122
+ datetimes bake in — the same value per element either way); East types and
123
+ values (`east_null` included); the `East` namespace, `East.if_else` included;
124
+ the `struct`/`variant`/`some`/`none` constructors (dual-mode: they build IR
125
+ when handed expression fields); **East.function artifacts** (dual-mode: a
126
+ call of their inline Function node at any nesting depth); **compiled East function values**
127
+ (`.bind` results, `compile_from_*` functions — a CALL on one lowers to a native
128
+ IR `Call`); and, two wrapper levels deep — enough for helper lambdas that
129
+ compose a callback — other python functions that pass the same rules.
130
+ - Anything else RAISES an `ExpressionError` **naming the binding**: a module
131
+ (`random.…`, `np.…`), a python builtin (`len`, `str`), a mutable python
132
+ capture, closure mutation (`nonlocal x; x += 1`). A closed-over East
133
+ *collection* raises in an eager callback too — a capture snapshots, `.bind`
134
+ stays live, and which you meant is your choice: an explicit `East.function`
135
+ snapshots a side table, `.bind(table)` keeps it live. For genuine python
136
+ semantics write an explicit python loop outside the body. Side effects are
137
+ therefore never lost or silently doubled: the body runs once, at build time, or
138
+ not at all.
139
+ - Arithmetic follows East types exactly: no implicit Integer↔Float mixing
140
+ (`.to_float()` / `.to_integer()` convert), and `/` is Float division.
141
+ - Captures are CACHED: an eager callback whose code object, captured bindings
142
+ and declared signature match an earlier call reuses its compiled function, so
143
+ the per-group aggregate `group_to_arrays(key).to_array(lambda b, es, k: {…})`
144
+ builds each inner lambda once, not once per group. ⚠️ A lambda whose CAPTURES
145
+ change per call (`lambda b, r: r.v > g` in a python loop over `g`) rebuilds
146
+ every call, since each capture bakes into a different function — hoist an
147
+ `East.function(...)` and pass the varying value as a bound parameter.
@@ -0,0 +1,70 @@
1
+ # Cross-language functions
2
+
3
+ `east-py export-functions` and `East.import_function`. Part of the `east-py` skill.
4
+
5
+ ## Cross-language functions: `east-py export-functions` and `East.import_function`
6
+
7
+ A python `East.function` is *called* from a TypeScript e3 task — and a
8
+ TypeScript one from python — as pure IR: no python at run time, no platform
9
+ bridge. The functions travel as a **function manifest** (each function's IR,
10
+ declared type and platform dependencies with the package providing each);
11
+ the importer names the function and declares its type; linking checks the
12
+ type exactly and embeds the IR. Declare them on the package's root module:
13
+
14
+ ```python
15
+ # packages/pricing/src/pricing/__init__.py — the package's root module
16
+ from east import East
17
+ from east.types.types import FloatType, StructType, IntegerType
18
+
19
+ Row = StructType([("qty", IntegerType), ("price", FloatType)])
20
+ score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * r.price)
21
+ east_functions = {"score": score} # name -> East.function artifact. CLOSED values: the
22
+ # exported IR must BIND every name it reads. A captured
23
+ # lookup table or a stdlib format string is a literal in
24
+ # the body and closes over nothing; a .bind result (no IR
25
+ # of its own) and an unresolved import are refused, and a
26
+ # genuinely free variable is refused BY NAME
27
+ ```
28
+
29
+ ```typescript
30
+ // the e3 task, in TypeScript (see the e3 skill) — the reference is all there is to write:
31
+ // e3.export finds `pricing` in the uv workspace, exports it (east-py export-functions, in the
32
+ // project's .venv) and links; the providers come from the task's runner
33
+ const score = East.importFunction("pricing", "score", FunctionType([RowType], FloatType));
34
+ await e3.export(pkg, "out.zip");
35
+ ```
36
+
37
+ e3 exports each workspace package in one `east-py export-functions --batch`
38
+ process, however many tasks import from it, and keeps the manifests, so a
39
+ redeploy whose python has not changed runs no export (the **e3** skill). A
40
+ function's manifest entry is its own: the same bytes whatever else the process
41
+ built first and whatever directory it ran in, so a task embedding it keeps its
42
+ hash until the function, or a line it was written at, changes.
43
+
44
+ For a package built elsewhere (published, another repo), or to link
45
+ in-process, write the manifest where the package lives and pass it:
46
+
47
+ ```bash
48
+ east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
49
+ # -p names the platform package implementing each platform call the functions make;
50
+ # a call no package provides fails the export (the manifest records the provider)
51
+ # → e3.export(pkg, "out.zip", { functions: ["./pricing.functions.beast2"] })
52
+ ```
53
+
54
+ The other direction, in python:
55
+
56
+ ```python
57
+ manifest = East.decode_function_manifest(Path("maths.functions.beast2").read_bytes()) # east-node export-functions wrote it
58
+ double = East.import_function("maths", "double", FunctionType([IntegerType], IntegerType))
59
+ user = East.function([IntegerType], IntegerType, lambda b, x: double(x) + 1) # callable in a body
60
+ ir, imports = East.link_imports(user, [manifest]) # exact type check; the IR embedded as a Let
61
+ compile_from_value(ir, [])(20) # 41 — pure IR, runs on any runner
62
+ ```
63
+
64
+ Unlinked, the reference is a `Platform` node named `east.importFunction`, and
65
+ compiling it raises naming that platform. `East.export_functions(pkg, version,
66
+ {…}, providers)` and `East.encode_function_manifest` are the API behind the CLI,
67
+ and `East.platform_dependencies(fn)` lists the platform functions an artifact
68
+ calls, in first-use order (unresolved imports skipped). Each also answers to its
69
+ TypeScript camelCase name (`East.exportFunctions`, `East.linkImports`, …). The
70
+ contract and the runner rules: `docs/conventions/EAST_CODEGEN.md`.