@elaraai/east-codex-plugin 1.0.99 → 1.0.101

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/.build/mcp/server.js +3 -2
  2. package/.codex-plugin/plugin.json +1 -1
  3. package/index.json +12207 -216
  4. package/package.json +1 -1
  5. package/skills/e3/SKILL.md +37 -809
  6. package/skills/e3/reference/browser.md +93 -0
  7. package/skills/e3/reference/cli.md +123 -0
  8. package/skills/e3/reference/from-code.md +48 -0
  9. package/skills/e3/reference/rack.md +139 -0
  10. package/skills/e3/reference/record-indexes.md +326 -0
  11. package/skills/e3/reference/record-migrations.md +63 -0
  12. package/skills/e3/reference/repository.md +25 -0
  13. package/skills/e3-ui/SKILL.md +273 -3623
  14. package/skills/e3-ui-cli/SKILL.md +1 -5
  15. package/skills/east/SKILL.md +31 -464
  16. package/skills/east/reference/beast2.md +71 -0
  17. package/skills/east/reference/cross-language-and-json-schema.md +87 -0
  18. package/skills/east/reference/queries.md +46 -0
  19. package/skills/east/reference/runners-and-ir.md +148 -0
  20. package/skills/east/reference/trees-and-recursion.md +56 -0
  21. package/skills/east/reference/values-in-typescript.md +62 -0
  22. package/skills/east-py/SKILL.md +29 -1652
  23. package/skills/east-py/reference/beast2.md +108 -0
  24. package/skills/east-py/reference/cli.md +51 -0
  25. package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
  26. package/skills/east-py/reference/control-flow-and-queries.md +147 -0
  27. package/skills/east-py/reference/cross-language.md +70 -0
  28. package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
  29. package/skills/east-py/reference/patterns.md +369 -0
  30. package/skills/east-py/reference/performance.md +70 -0
  31. package/skills/east-py/reference/value-methods.md +306 -0
  32. package/skills/east-py/reference/values-construction-and-json.md +88 -0
  33. package/skills/east-py/reference/writing-east-functions.md +275 -0
  34. package/skills/east-py-datascience/SKILL.md +12 -33
  35. package/skills/east-py-io/SKILL.md +1 -5
  36. package/skills/east-py-std/SKILL.md +1 -5
  37. package/skills/east-ui/SKILL.md +220 -1332
@@ -0,0 +1,150 @@
1
+ # IR and diagnostics
2
+
3
+ Printing IR as python (`east-py transpile`), the `east-c ir` toolbox, and the edit-time checks (`east-py lint`, `check`, flake8, `lsp`). Part of the `east-py` skill.
4
+
5
+ ## Contents
6
+
7
+ - IR ↔ python: `east-py transpile` and the `east-c ir` toolbox
8
+ - Diagnostics at edit time — `east-py lint`, `east-py check`, flake8, `east-py lsp`
9
+
10
+ ## IR ↔ python: `east-py transpile` and the `east-c ir` toolbox
11
+
12
+ Any East IR — an `East.function` artifact, or a `.json` / `.beast2` export from
13
+ TypeScript, east-c or east-node — prints as an idiomatic python module that
14
+ REBUILDS it through the surface above:
15
+
16
+ ```python
17
+ from east.codegen import to_python_source
18
+ print(to_python_source(classify)) # `@East.function(...)` / `def main(b, …)`
19
+ ```
20
+
21
+ ```bash
22
+ east-py transpile program.json -o program.py --name main # the same, from a file
23
+ east-py transpile program.json -p east-py-std -p east-py-io # platform calls print as the packages' own
24
+ # functions and named types, not restated declarations
25
+ east-c ir normalize program.json -o canonical.json # the canonical form
26
+ east-c ir diff a.json b.beast2 # first difference, or "identical"
27
+ east-c ir convert program.json -o program.beast2 # json <-> beast2, source map intact
28
+ ```
29
+
30
+ - **The contract:** `build(print(IR))` equals `IR` under `east-c ir normalize`
31
+ (location ids stripped, variables and labels renamed in the TypeScript
32
+ lowering's order, captures recomputed, recursive type ids renumbered — one
33
+ normalizer, in libeast-c). The TypeScript printer (`East.toSource`, `east-node
34
+ transpile`) is its twin: IR → python → IR → TypeScript → IR is equal at every
35
+ leg, over the whole compliance corpus and every example
36
+ (`docs/conventions/EAST_CODEGEN.md`).
37
+ - **Builtins print through one spelling table** (`east.codegen.spellings`):
38
+ operators only where they are exact — `+ - *` on numbers, `/` on Floats, the
39
+ comparisons, `& | ^ ~` on Booleans, `+` on Strings — named `East.<Type>.*` or
40
+ method spellings elsewhere, and `East.builtin(...)` for `RAW_ONLY`.
41
+ - **A function used as a VALUE** (`$.let(East.DateTime.roundDownWeek)`) prints
42
+ as the inline `@East.function` it is — the IR carries the body, not the name.
43
+ - **A platform call** prints as a hoisted `East.platform(...)` declaration — or,
44
+ given the implementing packages
45
+ (`to_python_source(ir, providers=providers_for(["east_py_std"]))`, `east-py
46
+ transpile -p east-py-std`), as `from east_py_std import fs_read_file` and the
47
+ call itself, with a type the package names (`GzipOptionsType`) printed by its
48
+ name; the plugin's example index is rendered this way. A provider is used only
49
+ where its declared signature IS the node's, so a drifted signature keeps the
50
+ declaration rather than printing a wrong import. A package whose function is
51
+ implemented in C exports the DECLARATION under the same name
52
+ (`simulation_run`, `optimization_iterative`), which a body calls identically.
53
+
54
+ ## Diagnostics at edit time — `east-py lint`, `east-py check`, flake8, `east-py lsp`
55
+
56
+ Everything the build refuses — a body without the block, `//` on an
57
+ expression, an f-string over one, `if` on one, a callback reaching for `np` — is
58
+ also a **rule** in `east.diagnostics` (the python twin of
59
+ `@elaraai/east-diagnostics`). The rules read a file's `ast`, find its East bodies
60
+ (an `East.function` body and everything nested in it, an eager callback on an
61
+ East value, a `@East.platform_function`'s East inputs), and say at edit time what
62
+ the build would say: every rule's message IS the build's refusal for the same
63
+ code, pinned by building the very source the rules read.
64
+
65
+ Two tiers, at two costs. The **rules** read the `ast` — instant, and useful even
66
+ on a file that does not parse. The **check** runs the build — the only type
67
+ checker a python East body has, since East's type checker IS the builder — at
68
+ the cost of importing the module.
69
+
70
+ ```bash
71
+ east-py lint src/ # file:line:col: category [rule] message — exit 1 on any finding
72
+ east-py lint src/ --format json # the findings as records
73
+ east-py lint src/ --disable no-deprecated-alias --exclude fixtures
74
+ east-py lint --list-rules # every rule, with its EAS code
75
+ east-py check src/ # the BUILD's errors — a file, a directory, or a module; --format json
76
+ flake8 --select EAS src/ # the same rules inside flake8 (east-py-cli registers the plugin)
77
+ east-py lsp # a Language Server over stdio, both tiers (--probe: can it start here?)
78
+ # pylsp # python-lsp-server runs the rules too (east-py-cli registers the plugin)
79
+ ```
80
+
81
+ - **`east-py check`** imports each target — a `.py` file, a directory (walked
82
+ as `lint` walks it) or a dotted module name, each executed afresh — builds
83
+ every East function in it, and reports EVERY failure at its authoring line: a
84
+ body whose type differs from the declared `out`, a callback the capture
85
+ refuses, an `IRAnalysisError`. Importing runs the module, so `EAST_CHECK=1` is
86
+ set for the duration — skip import-time work when you see it.
87
+ `--only-if-enabled` honours the project's opt-in (below), as the plugin's read
88
+ hook does, so the hook and the language server agree.
89
+ - **`east-py lsp`** (needs `pygls`) runs the rules on every change and the build
90
+ tier on **open** (debounced) and **save** (at once), on one long-lived worker
91
+ thread, and only when the project opts in: the build reads the module from
92
+ DISK, so on an unsaved buffer it would report the saved version's errors. A
93
+ save evicts the module from the warm process, so its importers are checked
94
+ against the new version. Pyright/Pylance has no plugin API, so flake8 and
95
+ pylsp are the editor paths that need no East-specific server; the Claude Code
96
+ plugin launches the project's own `east-py lsp` for `.py` files, and falls back
97
+ to `east-py lint` per change when that server cannot start.
98
+
99
+ The rules — `east-py lint --list-rules` prints them:
100
+ | Rule | Flags | What the build says |
101
+ |---|---|---|
102
+ | `body-takes-block-first` (EAS001) | `lambda x: …`, a body whose parameter count is not the declared count plus the block, `b.price`, `x + b` | `a body takes the block first` |
103
+ | `no-operator-fork` (EAS002) | `//` `%` `**` on an expression, `xs[-1]` | the operator-fork texts — `East.Integer.divide` / `remainder` / `pow`, `xs.get(xs.size() - 1)` |
104
+ | `no-python-formatting` (EAS003) | `f"{x}"`, `str(x)`, `print(x)`, `format(x)`, `"{}".format(x)`, `"%d" % x` | `f-strings / str() cannot be traced … East.String.print(T, value)` |
105
+ | `no-python-boolean` (EAS004) | `if`/`while`/`assert`/`and`/`or`/`not`/`x if c else y` on an expression, `in`, `for … in xs`, `len`/`int`/`float`/`bool`/`sum`/`sorted`/`max`… over one | `python \`if/and/or/not\` cannot be traced …` (`iteration`, `len()`, …) |
106
+ | `no-python-round` (EAS005) | `round(x)` on a Float expression | `East.Float.round_half(x) rounds half away from zero …` |
107
+ | `no-python-work` (EAS006) | an EAGER callback loading a module (`np`, `math`), a python builtin the capture does not admit (all but `abs`/`bool`/`isinstance`), a name imported from the standard library or an installed package that has no East form (`from math import floor`; a constant like `pi` lifts), a mutable East collection, or a `def` doing any of those (a clean macro `def` is fine) | `the callback cannot be captured automatically: it references np …` |
108
+ | `no-statement-on-outer-block` (EAS007) | `b.if_(p, lambda _b: b.assign(…))` — a statement on an enclosing body's block | `b.assign() was called on an OUTER block …` |
109
+ | `no-deprecated-alias` (EAS008, warning) | `.fold`, `.lower`, `East.Boolean.and_`, … (read off the surface's own deprecation docstrings) | `.fold() is deprecated: the spelling is .reduce() (the TypeScript name)` |
110
+ | `no-discarded-expression` (EAS009) | a bare `acc.push_last(x)` / `East.error(…)` / `xs.size()` line in a body | `.push_last() was evaluated and thrown away … b.do(…)` |
111
+ | `prefer-some-none` (EAS010, warning) | `variant("some", x)` / `variant("none", None)` | use `some(value)` / `none` |
112
+ | `no-handrolled-variant` (EAS011, warning) | a `{"type": …, "value": …}` dict | `variant("Tag", value, Type)` — the encoder needs what it constructs |
113
+ | `prefer-explicit-east-type` (EAS012) | `b.let([])` / `b.let([1, 2])` / `b.const(list())` — a bare python list, set or tuple, empty or not | `cannot lift python value of type list …` — pass the East type: `b.let([], ArrayType(IntegerType))`, or build it with `East.new_array` |
114
+ | `no-let-const-in-expression` (EAS013, warning) | `b.let` / `b.const` buried in a call argument, an element, a chain target (a tuple of declarations is fine) | give the declaration its own statement |
115
+ | `no-untracked-east-data` (EAS014, suggestion) | a plain python list/dict local reaching an expression's method | `b.const(rows, Type)` — a local carries no East type and re-inlines |
116
+ | `no-redundant-east-cast` (EAS015, warning) | `b.let(East.value(x, T), T)` | pass the value and type to `b.let` directly |
117
+ | `prefer-let-const-over-east-value` (EAS016, suggestion) | `East.value(...)` assigned or returned inside a body | `b.const(value, Type)` — `East.value` erases the type at the binding |
118
+ | `no-host-comparison-on-east-values` (EAS017) | `<` / `>` on a decoded VARIANT or OPTION outside a body — it raises `TypeError` (`==` is structural and fine; a decoded Integer is an `int`) | `compare_for(T)` / `less_for(T)`, `make_east_key(T)` for `sorted` |
119
+ | `no-module-scope-east-macro` (EAS018, warning) | a module-scope helper a body calls to build IR, or to assemble a composite `f"{a}|{b}"` key | make it an `East.function`; model typed / nested East data |
120
+ | `no-build-time-clock` (EAS019, warning) | `datetime.now()` / `time.time()` at module scope (a script's `if __name__ == "__main__":` is not module scope) | author the constant, or read the clock at runtime — a platform function, or `east_py_std`'s `time_now()` in a body |
121
+ | `no-inline-credentials` (EAS020, warning) | a literal `password` / `token` / `secret_access_key` (a localhost sibling host, a header NAME like `X-Api-Key`, and a `${TEMPLATE}` are exempt) | read it at runtime: `east_py_std`'s `env_get("VAR")` in a body, `os.environ` in a platform function — IR is content-addressed and replicated |
122
+ | `no-compile-time-data-injection` (EAS021, warning) | `open()` / `json.load` / `Path.read_text` / `os.environ` at module import, when the result REACHES East (a body reads it, or a boundary call takes it) | load at runtime: an e3 input, a dataset, a platform function |
123
+ | `no-python-east-data` (EAS022, warning) | East rows assembled by a module-scope comprehension or loop, then handed to a body | write the rows out, or produce them at runtime |
124
+ | `no-python-string-building` (EAS023, warning) | an f-string assembling an East string CONSTANT — a regex, a template, a key | spell the constant out |
125
+ | `no-derived-struct-fields` (EAS024, warning) | `Derived = StructType([… for f in Other.value])` | a type declaration is a wire format — spell the fields |
126
+ | `no-python-data-work` (EAS025, warning) | a python helper doing parse / strip / null-check / coerce work on an EXPRESSION a body hands it | express it in East, where it runs on every row |
127
+ | `no-reinlined-east-binding` (EAS026, warning) | a python local holding an EXPRESSION (`n = xs.size()`, not a `b.let`/`b.const` binding) read more than once in one body, or read inside a per-element callback, a loop body or a nested function below the body that bound it | each read re-emits — and re-evaluates — the expression: bind it once, `n = b.const(xs.size())`, where it is needed (the TypeScript rule of the same name) |
128
+
129
+ Configure it once in the project's `pyproject.toml`; every surface reads it, and
130
+ `east-py lint`'s own flags add to it:
131
+
132
+ ```toml
133
+ [tool.east-py]
134
+ check = true # let an EDITOR run the build tier (off by default)
135
+ disable = ["no-deprecated-alias"] # rules to skip
136
+ exclude = ["fixtures", "vendor"] # extra directory names not to walk
137
+ ```
138
+
139
+ `check` is off unless a project asks: the rules READ a file, the build RUNS it,
140
+ and an editor should not import someone's modules on save because a language
141
+ server happened to be installed. `east-py check` on the command line is consent
142
+ in itself and ignores the setting. Unreadable configuration falls back to the
143
+ defaults — a diagnostics tool must never stop a project building.
144
+
145
+ A file that does not import `east` is never diagnosed; a line ending in `# noqa`
146
+ (or `# noqa: EAS002` / `# noqa: no-operator-fork`) is skipped; `.venv`, `venv`,
147
+ `node_modules`, `dist`, `build`, `.git`, `__pycache__`, `tests` and `test` are not
148
+ walked. The rules are syntactic, so a clean `lint` is necessary, not sufficient —
149
+ the type errors live behind `east-py check`. The rules are disjoint (each
150
+ mistake is one rule's to report).
@@ -0,0 +1,369 @@
1
+ # Key patterns
2
+
3
+ Part of the `east-py` skill.
4
+
5
+ ## Contents
6
+
7
+ - Python values vs East expressions
8
+ - The canonical platform function
9
+ - Project-owned platform module (calling your Python from e3)
10
+ - Memoize expensive pure stages (dev/test harnesses)
11
+ - Fork-parallel export of a huge table (one file out)
12
+ - Keyed lookups against a file too big to load
13
+ - Sequential logic that stays in east-c (worklist / replay)
14
+ - Trees from flat rows, and recursion
15
+ - Sort uses East's total order
16
+ - Scalars: use the `East.<Type>` utilities for consistency — above all String & DateTime
17
+ - torch interop (in a torch-having package, inline — no helpers)
18
+
19
+
20
+ ## Python values vs East expressions
21
+
22
+ The `east` skill's first rule, in python: a body's PARAMETERS are expressions;
23
+ a python value used inside a body must lift. Scalars lift on their own; a
24
+ collection must be an explicit choice.
25
+
26
+ ```python
27
+ RATES = EastDict(StringType, FloatType, {"AUD": 0.65}) # python-side East value
28
+ CUTOFF = 100
29
+
30
+ @East.function([Row], FloatType)
31
+ def usd(b, r):
32
+ return East.if_else(r.amount > CUTOFF, # a scalar constant lifts (TS East.value)
33
+ r.amount * RATES.get_or_default(r.ccy, 1.0), # an EXPLICIT build snapshots the dict: a literal, rebuilt per call
34
+ 0.0)
35
+
36
+ rows.map(lambda b, r: r.amount * RATES.get_or_default(r.ccy, 1.0)) # ❌ an eager callback REFUSES
37
+ # the collection capture:
38
+ conv = East.function([Row, DictType(StringType, FloatType)], FloatType,
39
+ lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
40
+ rows.map(conv.bind(RATES)) # ✅ bind it: live, zero-copy
41
+ ```
42
+
43
+ ## The canonical platform function
44
+
45
+ ```python
46
+ from east import East, FloatType, StringType, StructType, ArrayType, struct
47
+
48
+ LineItem = StructType([("name", StringType), ("price", FloatType)]) # Struct<String, Float>
49
+
50
+ @East.platform_function(inputs=[FloatType, ArrayType(LineItem)], output=ArrayType(LineItem))
51
+ def convert_prices(fx_rate, items):
52
+ # items: an east-c-backed array with eager methods; row["price"] is a plain float
53
+ return items.map(lambda b, row: struct(
54
+ {"name": row["name"], "price": row["price"] * fx_rate}, # plain f64 * f64
55
+ LineItem, # tag + validate the result
56
+ ))
57
+ ```
58
+ `.map` builds the lambda into an East function — the loop and the row construction run
59
+ in east-c, with `fx_rate` baked in as a constant; `struct(..., LineItem)` is dual-mode, so
60
+ the same spelling builds the row here and on plain values; the decorator validates the
61
+ `Array<LineItem>` result — a named `EastTypeError` instead of silent corruption.
62
+
63
+ ## Project-owned platform module (calling your Python from e3)
64
+
65
+ To call a Python platform function from an **e3 task**, package it so `east-py
66
+ run -p <module>` can load it: each module ends with `<name>_impl =
67
+ East.platform_functions(__name__)`, and the package `__init__.py` aggregates them into
68
+ a top-level `platform` list (the same shape as east-py-std / east-py-datascience).
69
+
70
+ ```python
71
+ # platform_module/forecast.py
72
+ @East.platform_function(inputs=[ArrayType(FloatType)], output=FloatType,
73
+ name="my_project.forecast") # dotted "<project>.<fn>"; MUST byte-match
74
+ def forecast(history): # the TS East.platform(...) declaration
75
+ return sum(history) / len(history) if history else 0.0
76
+ forecast_impl = East.platform_functions(__name__)
77
+
78
+ # platform_module/__init__.py
79
+ from .forecast import forecast_impl
80
+ platform = [*forecast_impl] # what `east-py run -p platform_module` loads
81
+ ```
82
+
83
+ The e3 task wires it with `{ runtime: "east-py", platforms: [{ custom:
84
+ "platform_module" }, "east-py-std"] }`. East code needs a TS `East.platform(
85
+ "my_project.forecast", [...], ...)` **declaration** with the identical name (no
86
+ codegen). Add native deps (numpy, …) to `pyproject.toml` and run `uv sync`. See
87
+ **east-project** for the full scaffold (`--platform`) and the setuptools
88
+ packaging; **e3** for the runner.
89
+
90
+ ## Memoize expensive pure stages (dev/test harnesses)
91
+
92
+ ```python
93
+ from east import East, configure_memo, memoize
94
+
95
+ @memoize # eligibility — the author asserts purity
96
+ @East.platform_function(inputs=[ArrayType(RowType), ConfigType], output=ModelType)
97
+ def train_model(rows, config): ...
98
+
99
+ # A test harness flips the whole package on with one call (or EAST_MEMO_DIR):
100
+ configure_memo("/tmp/my_project", salt="v1") # None deactivates
101
+ loaded = memoize(other_pkg.load_csv, salt=file_digest) # inline, per-call salt
102
+ ```
103
+
104
+ The platform-function boundary is the cache boundary, e3-style: hit = decode the
105
+ stored BEAST2 blob via the declared output type (skipping the body entirely),
106
+ miss = compute + atomic save. Only mark functions **pure in their East inputs** —
107
+ a hit on a function that writes files/rows/requests silently drops those effects.
108
+ Stochastic fits memoize their first realization. Code edits don't change keys —
109
+ bump the salt. Inert with no directory configured: under e3, the dataflow's
110
+ content-addressed task cache is the real memo and this stays off.
111
+
112
+ ## Fork-parallel export of a huge table (one file out)
113
+
114
+ The shape for "N CPUs on one table": build the expensive shared context once,
115
+ let forked workers inherit it copy-on-write, and ship a single indexed file —
116
+ consumers never learn the export was parallel. (Windows runs the same contract
117
+ inline, sequentially; the output is byte-identical either way.)
118
+
119
+ ```python
120
+ from east.serialization.beast2 import open_beast2_file, write_beast2_file_parallel
121
+
122
+ ROW = StructType([("order_id", StringType), ("qty", IntegerType), ("total", FloatType)])
123
+
124
+ lookups = load_reference_tables() # multi-GB keyed dicts: built ONCE, pre-fork,
125
+ # inherited COW — never pickled, never rebuilt
126
+
127
+ def produce(span): # runs in the worker process
128
+ start, count = span # yield batches of any size — the managed
129
+ for chunk in chunk_ranges(start, count, 8192): # writer cuts the canonical segments
130
+ yield compute_rows(lookups, chunk) # eager methods + East functions: native
131
+
132
+ write_beast2_file_parallel(
133
+ "orders.beast2", ArrayType(ROW),
134
+ partitions=row_ranges(total_records, shards=13), # partition order = row order
135
+ produce=produce,
136
+ processes=13, # a worker failure kills the rest, cleans up, re-raises
137
+ )
138
+
139
+ # Consumers — e.g. the table's @East.platform_function loader — see ONE file, no
140
+ # catalog, no shard names, and read it at one segment of memory:
141
+ @East.platform_function(inputs=[StringType], output=ArrayType(ROW))
142
+ def load_orders(path):
143
+ with open_beast2_file(path, ArrayType(ROW)) as f:
144
+ return f.load() # or compute on f directly, never holding the table
145
+ ```
146
+
147
+ Shards produced by your own process topology (or on another machine)? Merge
148
+ them directly — a pure byte copy, nothing re-encoded:
149
+
150
+ ```python
151
+ splice_beast2_files("orders.beast2", ArrayType(ROW), sorted(shard_paths), verify=True)
152
+ ```
153
+
154
+ ## Keyed lookups against a file too big to load
155
+
156
+ A reference table exported as a Dict file answers point reads without ever
157
+ being held in memory: east-c fence-searches the segment index and decodes
158
+ one segment per lookup, so a sparse join against a multi-GB file stays
159
+ cheap. (A *dense* pass over most keys is still better as whole-file compute
160
+ — `f.group_sum(...)`, `f.filter(...)`, any read method directly on the file.)
161
+
162
+ ```python
163
+ from east.serialization.beast2 import open_beast2_file
164
+
165
+ with open_beast2_file("orders.beast2") as orders: # Dict<String, Order>; Order = Struct{id, total}
166
+ order = orders[order_id] # fence search → ONE segment decode
167
+ hot = orders.get_keys(wanted_ids, # each owning segment decodes once;
168
+ lambda b, k: {"id": k, "total": 0.0}) # `fill` is a body (a python dict
169
+ if candidate_id in orders: # captured from outside is refused)
170
+ ...
171
+
172
+ # A sorted Array file answers range queries the same way — global insertion
173
+ # indices, then slice exactly the covered rows:
174
+ with open_beast2_file("events.beast2") as events: # Array<Event> sorted by timestamp
175
+ span = events.find_sorted_range(cutoff_event)
176
+ window = events.slice(span["start"], span["end"])
177
+ ```
178
+
179
+ ## Sequential logic that stays in east-c (worklist / replay)
180
+
181
+ When the next step depends on the last — a worklist, a BFS, a fixpoint, a
182
+ topological replay — no `map`/`filter`/`group_reduce` expresses it, and a
183
+ python loop over decoded rows is the whole job's cost. The statement surface
184
+ threads the state through the loop natively, so the whole thing is ONE
185
+ compiled function. Two rules carry most of the weight. **Accumulate in
186
+ place** — a collection threaded through a state is rebuilt every iteration,
187
+ so `order.concat(...)` is O(n²) over the loop where `order.push_last(...)` is
188
+ O(1). And **a mutation is a statement** — `b.do(acc.push_last(x))` in a
189
+ statement body, `East.block(acc.push_last(x), result)` in an expression body;
190
+ a bare `acc.push_last(x)` line is evaluated at build time and thrown away
191
+ (the build raises rather than compile a loop that silently does nothing).
192
+
193
+ ```python
194
+ from east import ArrayType, DictType, East, IntegerType, StringType
195
+
196
+ Node, Edges = StringType, DictType(StringType, ArrayType(StringType))
197
+ Indeg = DictType(StringType, IntegerType)
198
+
199
+ # Kahn's algorithm — ONE compiled function, the TypeScript `$` shape: a
200
+ # worklist the loop appends to, a cursor into it, and per-node successors
201
+ # decremented in place.
202
+ @East.function([ArrayType(Node), Edges, Indeg], ArrayType(Node))
203
+ def topo_order(b, roots, succ, indeg):
204
+ ready = b.const(roots.copy()) # task inputs arrive frozen: copy
205
+ remaining = b.const(indeg.copy())
206
+ order = b.const(East.new_array(Node))
207
+ i = b.let(0)
208
+
209
+ def step(b, label):
210
+ node = b.const(ready.get(i))
211
+ b.do(order.push_last(node))
212
+
213
+ def visit(b, v):
214
+ b.do(remaining.insert_or_update(v, -1, lambda _b, old, d: old + d))
215
+ b.if_(remaining.get(v) == 0, lambda b: b.do(ready.push_last(v)))
216
+
217
+ b.for_(succ.get_or_default(node, East.new_array(Node)), visit)
218
+ b.assign(i, i + 1)
219
+
220
+ b.while_(i < ready.size(), step)
221
+ return order
222
+
223
+ @East.platform_function(inputs=[ArrayType(Node), Edges, Indeg], output=ArrayType(Node))
224
+ def replay_order(roots, succ, indeg):
225
+ return topo_order(roots, succ, indeg)
226
+ ```
227
+
228
+ The same algorithm as EXPRESSION forms — a state struct threaded through
229
+ `East.while_`/`East.for_`, the shape that fits inside a one-expression
230
+ callback:
231
+
232
+ ```python
233
+ topo_order = East.function(
234
+ [ArrayType(Node), Edges, Indeg], ArrayType(Node),
235
+ lambda b, roots, succ, indeg: East.while_(
236
+ {"ready": roots.copy(), "indeg": indeg.copy(),
237
+ "order": East.new_array(Node), "i": 0},
238
+ cond=lambda b, s: s.i < s.ready.size(),
239
+ body=lambda b, s: East.let(s.ready.get(s.i), lambda b, node: East.block(
240
+ s.order.push_last(node),
241
+ East.for_(
242
+ succ.get_or_default(node, East.new_array(Node)),
243
+ {**s, "i": s.i + 1}, # inner loop's state
244
+ lambda b, t, v: East.block(
245
+ t.indeg.insert_or_update(v, -1, lambda b, old, d: old + d),
246
+ East.if_else(t.indeg.get(v) == 0, # newly ready?
247
+ East.block(t.ready.push_last(v), t),
248
+ t))))), # else: unchanged
249
+ ).order)
250
+
251
+ # any carried state — a forward fill is a `for_` whose state remembers the last value
252
+ forward_fill = East.function(
253
+ [ArrayType(StringType)], ArrayType(StringType),
254
+ lambda b, cells: East.for_(
255
+ cells, {"last": "", "out": East.new_array(StringType)},
256
+ lambda b, s, cell: East.let(
257
+ East.if_else(cell == "", s.last, cell),
258
+ lambda b, v: East.block(s.out.push_last(v), {"last": v, "out": s.out}))).out)
259
+ ```
260
+
261
+ To stop early, put `East.break_(state)` in an `if_else` arm — the state
262
+ argument commits before the jump, so the answer survives; `East.label(...)` on
263
+ an outer loop lets an inner one break all the way out. Reach for all of this
264
+ only when the work is genuinely sequential: a `group_reduce` or a `reduce` is
265
+ both shorter and faster when it fits.
266
+
267
+ ## Trees from flat rows, and recursion
268
+
269
+ `to_tree` builds nested data — any depth, your own node type — from rows that
270
+ name their key and their parent's key. `build` runs once per row, children
271
+ first; the result is the roots in source order. The walk is iterative, in
272
+ east-c, on values and in a body alike:
273
+
274
+ ```python
275
+ from east import ArrayType, OptionType, StringType, StructType, recursive_type
276
+
277
+ Row = StructType([("id", StringType), ("parent", OptionType(StringType)), ("name", StringType)])
278
+ Node = recursive_type(lambda self: StructType([("name", StringType), ("children", ArrayType(self))]))
279
+
280
+ roots = rows.to_tree(Node,
281
+ lambda b, r: r.id,
282
+ lambda b, r: r.parent,
283
+ lambda b, r, _i, children: {"name": r.name, "children": children})
284
+ ```
285
+
286
+ - A row whose parent is `none` is a root, and so is an orphan (its parent key
287
+ is not in the rows). A repeated key ❗ `toTree: duplicate key <key>`; a cycle
288
+ ❗ `toTree: cycle through key <key>` — both before any `build`.
289
+ - A non-recursive `Node` folds bottom-up: `IntegerType` with
290
+ `children.sum() + 1` is each root's subtree size. A Set's element and a
291
+ Dict's key are their own keys, so those take no `key`.
292
+
293
+ `flatten_tree` is the way back: every node of nested data, each with its path
294
+ of keys (a Dict's) or indices (an Array's) from its root, depth first — each
295
+ node before its descendants, siblings in key or array order. `children` names a
296
+ node's children, of the collection's own type:
297
+
298
+ ```python
299
+ from east import DictType, FloatType, StringType, StructType, recursive_type
300
+
301
+ Part = recursive_type(lambda self: StructType([("name", StringType), ("cost", FloatType), ("parts", DictType(StringType, self))]))
302
+
303
+ nodes = parts.flatten_tree(lambda b, part: part.unwrap().parts)
304
+ # an Array of {path, node}: ["pump"], ["pump", "motor"], ["pump", "motor", "rotor"], ["pump", "seal"]
305
+ ```
306
+
307
+ - An Array of `{path, node}`, not a Dict keyed by path: a Dict key is
308
+ immutable, and a path is an Array. `to_tree` over it, keyed by `path` and
309
+ parented by `path` less its last step, rolls the tree up at every root.
310
+ - `children` runs once per node, in the result's order. Each path is an array
311
+ of its own, so a chain n deep holds n(n+1)/2 path elements.
312
+ - **Recursion** — a function calling itself through a captured variable — is
313
+ for shallow depths, in the hundreds. Deeper, a call ❗ `call stack exhausted:
314
+ East calls nested too deeply`, a catchable `EastError` on every runtime. Walk
315
+ deep data with `b.while_`, `to_tree` or `flatten_tree`; a recursive VALUE's depth is still
316
+ bounded, at tens of thousands of levels, by east-c's recursive collect and
317
+ free.
318
+
319
+ ## Sort uses East's total order
320
+
321
+ ```python
322
+ # WRONG — Python's default ordering (incorrect for floats/NaN, mixed, type-specific)
323
+ sorted(list(arr))
324
+
325
+ # CORRECT — East total order, in east-c (the TypeScript names)
326
+ arr.sort() # new array
327
+ arr.sort_in_place() # in place
328
+ arr.sort(lambda b, r: r["x"]) # by a projection, still East-ordered
329
+ ```
330
+
331
+ ## Scalars: use the `East.<Type>` utilities for consistency — above all String & DateTime
332
+
333
+ East scalars *are* Python scalars, so Python operators run on them — but the
334
+ `East.<Type>` namespaces (and `East.less`/`compare`/`equal`) are the standardised
335
+ ops that give the **same** answer in Python, C, TS, and cached e3 tasks. Python's
336
+ `str` methods, `re`, `datetime`, `strftime`, `//`, and `<` drift by
337
+ engine / locale / timezone; the East functions do not. Use them for anything with
338
+ divergent semantics — and **always** for strings and dates.
339
+
340
+ ```python
341
+ # String — East.String.*, not Python str/re: split edge cases, the regex engine,
342
+ # trim's whitespace set, and JSON encoding all differ from Python
343
+ East.String.split("a,b,c", ",") # East semantics, not str.split
344
+ East.String.regex_replace(s, r"\d+", "#") # East regex engine, not Python re
345
+ East.String.trim(s) # East's whitespace set
346
+ East.String.print_json(IntegerType, 5) # '"5"' — Integer is a JSON *string* in East
347
+
348
+ # DateTime — East.DateTime.* only. East DateTime is UTC; tokens are Day.js, not strftime
349
+ East.DateTime.add_milliseconds(dt, 86_400_000) # +1 day, not dt + timedelta(days=1)
350
+ East.DateTime.duration_milliseconds(a, b) # a − b (ms), standardised
351
+ East.DateTime.get_day_of_week(dt) # Monday == 1, not dt.weekday() (== 0)
352
+ East.DateTime.print_formatted(dt, "YYYY-MM-DD HH:mm:ss") # Day.js tokens, not dt.strftime("%Y-…")
353
+ East.Float.print_currency(total) # "$1,234.57" — the stdlib, same in TS
354
+
355
+ # Ordering / equality — East total order (NaN-correct, mixed types, variants-by-name)
356
+ East.less(FloatType, a, b) # not a < b / sorted(...)
357
+ # Integer is i64 — divide truncates; Python's unbounded int can overflow i64
358
+ East.Integer.divide(7, 2) # 3
359
+ # (you also can't method-call a Python float/str — the ops live on the namespace)
360
+ East.Float.sqrt(2.0) # not (2.0).sqrt()
361
+ ```
362
+
363
+ ## torch interop (in a torch-having package, inline — no helpers)
364
+
365
+ ```python
366
+ import torch, numpy as np
367
+ t = mat.to_torch() # writeable torch copy of the buffer
368
+ out = EastMatrix(FloatType, model(t).detach().cpu().numpy()) # bridge canonicalizes dtype
369
+ ```
@@ -0,0 +1,70 @@
1
+ # Performance
2
+
3
+ Part of the `east-py` skill.
4
+
5
+ ## Performance — the levers, ranked
6
+
7
+ The point of the machinery: **data stays in east-c and python never runs per
8
+ element**.
9
+
10
+ 1. **Keep values East end-to-end.** `EastArray`/`EastSet`/`EastDict` are
11
+ C-backed, and every eager method runs its loop natively however deeply the
12
+ element type nests. `list(...)`/`dict(...)` or a python loop pays a boxing
13
+ crossing per element — convert at most once, at the very end.
14
+ 2. **Let the callbacks build, and chain on the results** — a chain of
15
+ `map`/`filter`/`reduce` stays native between steps. Return a dict literal
16
+ (`lambda b, r: {"a": …, "b": …}`) to derive every column in ONE pass. **Bind
17
+ shared work with `b.const`** — the builder emits exactly what the body
18
+ spells, as TypeScript's does, so a python variable holding an expression is
19
+ re-emitted (and re-evaluated) at every read: `fields = r.data.split("|")`
20
+ read for 30 columns runs the split 30 times per row, and `table =
21
+ derive(rec)` read inside a `.map` runs `derive` once per element. `fields =
22
+ b.const(r.data.split("|"))` and `table = b.const(derive(rec))` run each
23
+ once, as `$.const` does — the `no-reinlined-east-binding` rule (`east-py
24
+ lint`, the LSP) points at every such read. There is no optimisation pass
25
+ (#1312 removed the trace-time CSE: it moved reads across mutations it could
26
+ not see), so what you write is what runs — a callback that pushes onto
27
+ `order` and returns `order.size() - 1` reads the size after its push.
28
+ 3. **Side tables: `b.const` a captured one, `bind` big ones** — opposite contracts:
29
+ - **Capture (snapshot).** A collection captured by an `East.function` body
30
+ is snapshot into its IR as a literal of its contents — exactly as a
31
+ TypeScript literal is — and rebuilt wherever the body reads it: per call
32
+ at the function's top level, per element inside a callback. **Bind it
33
+ once per call with `t = b.const(TABLE)`** and read `t`. Right for small
34
+ tables (up to a few thousand entries): the literal's size rides the IR,
35
+ every call rebuilds it, and later mutations are **not** seen. Only an
36
+ EXPLICIT build snapshots: an eager callback refuses a mutable capture.
37
+ - **`East.function(...).bind(table)` (by reference, live).** Declare the
38
+ table as a trailing parameter and pre-bind it: zero copy, O(1) at any
39
+ size, nothing rebuilt per call, and the function **sees later
40
+ mutations**. Rebinding gives independent callables, the unbound function
41
+ stays usable, and a wrong-typed value raises `TypeError`.
42
+
43
+ ```python
44
+ fx = EastDict(StringType, FloatType, rates) # small table → capture
45
+ @East.function([Row], FloatType) # (the EXPLICIT build
46
+ def to_usd(b, r): # opts into the snapshot)
47
+ t = b.const(fx) # the literal, once per call
48
+ return r.amount * t.get_or_default(r.ccy, 1.0)
49
+
50
+ TableT = DictType(StringType, FloatType) # huge table → bind
51
+ conv = East.function([Row, TableT], FloatType,
52
+ lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
53
+ rows.map(conv.bind(big_table)) # loop + lookup stay in east-c
54
+ conv.bind(t1, t2) # multi-table: binds the TRAILING parameters in order
55
+ ```
56
+
57
+ Function *parameters* always cross by reference (zero copy, any size, any
58
+ depth); `bind` is what lets an eager method use a parameter-taking function
59
+ where the callback signature is fixed.
60
+ 4. **Hoist `East.function(...)` out of python loops** and reuse it — building
61
+ is cheap, not free.
62
+ 5. **When logic must stay python, go columnar** — one crossing per column or
63
+ batch instead of per row × field, then back with `from_columns`/`extend`
64
+ ([Columnar escape hatches](columns-and-platform-functions.md#columnar-escape-hatches--when-the-logic-must-stay-python)).
65
+ 6. **See how a hot call ran** — `east.runtime.compiler.eager_stats()`:
66
+ `function_direct` counts callbacks that rode a precompiled function value
67
+ straight in, `c_to_py_decodes` counts values boxed C→python (an eager method
68
+ quietly decoding a whole collection shows here), and the `beast2_*` counters
69
+ report column projection. There is no per-element python counter because
70
+ there is no per-element python path.