@elaraai/east-codex-plugin 0.0.0-stage → 1.0.89
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/east-py-lsp.js +568 -0
- package/.build/daemon/lsp.js +3737 -0
- package/.build/daemon/server.js +3535 -0
- package/.build/hooks/codex-tools.js +584 -0
- package/.build/hooks/diagnose-bash.js +399 -0
- package/.build/hooks/diagnose.js +347 -0
- package/.build/hooks/pre-read.js +100 -0
- package/.build/hooks/pre-write.js +247 -0
- package/.build/hooks/session-start.js +263 -0
- package/.build/hooks/subagent-start.js +212 -0
- package/.build/mcp/server.js +23577 -0
- package/.build/status/report.js +2163 -0
- package/.codex-plugin/plugin.json +36 -0
- package/.lsp.json +21 -0
- package/.mcp.json +11 -0
- package/LICENSE.md +682 -0
- package/README.md +192 -2
- package/hooks/hooks.json +51 -0
- package/index.json +44926 -0
- package/package.json +36 -4
- package/skills/e3/SKILL.md +841 -0
- package/skills/e3-create/SKILL.md +265 -0
- package/skills/e3-ui/SKILL.md +2260 -0
- package/skills/e3-ui-cli/SKILL.md +332 -0
- package/skills/east/SKILL.md +690 -0
- package/skills/east-contribute/SKILL.md +348 -0
- package/skills/east-design/SKILL.md +233 -0
- package/skills/east-node-io/SKILL.md +143 -0
- package/skills/east-node-std/SKILL.md +289 -0
- package/skills/east-ontology/SKILL.md +181 -0
- package/skills/east-project/SKILL.md +189 -0
- package/skills/east-py/SKILL.md +2023 -0
- package/skills/east-py-datascience/SKILL.md +280 -0
- package/skills/east-py-io/SKILL.md +186 -0
- package/skills/east-py-std/SKILL.md +301 -0
- package/skills/east-ui/SKILL.md +1529 -0
|
@@ -0,0 +1,280 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: east-py-datascience
|
|
3
|
+
description: "Data science and machine learning platform functions for the East language (TypeScript types + directly-callable Python implementations). Use when writing East programs that need optimization (MADS, Optuna, SimAnneal, ALNS, Scipy, Optimization, GoogleOr), machine learning (XGBoost, LightGBM, NGBoost, Torch MLP, Lightning, GP), Bayesian inference (PyMC), causal inference (Causal: DoWhy, EconML DML, ALE), simulation (Simulation DES), ML utilities (Sklearn preprocessing, metrics, splits), conformal prediction (MAPIE), or model explainability (SHAP). Triggers for: (1) Writing East programs with @elaraai/east-py-datascience, (2) Derivative-free optimization with MADS, (3) Bayesian optimization with Optuna, (4) Discrete/combinatorial optimization with SimAnneal or ALNS, (5) Gradient boosting with XGBoost or LightGBM, (6) Probabilistic predictions with NGBoost or GP, (7) Neural networks with Torch MLP or Lightning, (8) Data… See the detailed scope below."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Detailed skill scope
|
|
7
|
+
|
|
8
|
+
Data science and machine learning platform functions for the East language (TypeScript types + directly-callable Python implementations). Use when writing East programs that need optimization (MADS, Optuna, SimAnneal, ALNS, Scipy, Optimization, GoogleOr), machine learning (XGBoost, LightGBM, NGBoost, Torch MLP, Lightning, GP), Bayesian inference (PyMC), causal inference (Causal: DoWhy, EconML DML, ALE), simulation (Simulation DES), ML utilities (Sklearn preprocessing, metrics, splits), conformal prediction (MAPIE), or model explainability (SHAP). Triggers for: (1) Writing East programs with @elaraai/east-py-datascience, (2) Derivative-free optimization with MADS, (3) Bayesian optimization with Optuna, (4) Discrete/combinatorial optimization with SimAnneal or ALNS, (5) Gradient boosting with XGBoost or LightGBM, (6) Probabilistic predictions with NGBoost or GP, (7) Neural networks with Torch MLP or Lightning, (8) Data preprocessing and metrics with Sklearn, (9) Conformal prediction intervals with MAPIE, (10) Model explainability with Shap, (11) Iterative coordinate descent with Optimization, (12) Constraint programming, vehicle routing, LP/MIP, or graph algorithms with GoogleOr, (13) Bayesian regression, hierarchical models, and multi-layer estimation with PyMC, (14) Economic ontology simulation via discrete event simulation with Simulation, (15) One declarative causal experiment — naive vs adjusted effect, overlap, robustness, and an honesty verdict — with Causal.experiment, (16) Calling the east_py_datascience functions (xgboost_train_regressor, mads_optimize, …) from a project's own Python @East.platform_function, or inside an East.function body — the same object does both.
|
|
9
|
+
|
|
10
|
+
# East Data Science
|
|
11
|
+
|
|
12
|
+
Data science and machine learning platform functions for East: optimisation,
|
|
13
|
+
ML models, Bayesian and causal inference, simulation, preprocessing and
|
|
14
|
+
explainability. This is the platform's **Reason** layer — a model earns its
|
|
15
|
+
place by improving a decision (a forecast or optimisation that feeds a
|
|
16
|
+
`decision`), so wire its outputs toward one. The functions run on the python
|
|
17
|
+
runtime only: an e3 task needs a python runner (see Related skills).
|
|
18
|
+
|
|
19
|
+
## Before writing code — search the example index
|
|
20
|
+
|
|
21
|
+
Every East API has a tested example in the plugin's index — the index IS the
|
|
22
|
+
API reference, printed from each example's IR in TypeScript or python. Before
|
|
23
|
+
writing or changing East code:
|
|
24
|
+
|
|
25
|
+
1. Call `search_east_examples` for each capability you
|
|
26
|
+
are about to use — `language: "python"` for east-py, `"typescript"`
|
|
27
|
+
otherwise. Summaries come back first: id, signature, the inputs and the
|
|
28
|
+
expected result, a few hundred bytes each.
|
|
29
|
+
2. Fetch the one or two that match with `get_east_example`
|
|
30
|
+
and pattern your code on them.
|
|
31
|
+
3. Do not read `node_modules/@elaraai/**` or `*.examples.ts` files wholesale,
|
|
32
|
+
and do not reason from `.d.ts` signatures: the index holds the same
|
|
33
|
+
programs, exact and far cheaper, and the signatures omit the runtime rules
|
|
34
|
+
that make East code correct.
|
|
35
|
+
|
|
36
|
+
Nothing is injected for you; the search is the step.
|
|
37
|
+
|
|
38
|
+
## Quick Start
|
|
39
|
+
|
|
40
|
+
```typescript
|
|
41
|
+
import { East, FloatType, VectorType, none, some } from "@elaraai/east";
|
|
42
|
+
import { MADS } from "@elaraai/east-py-datascience";
|
|
43
|
+
|
|
44
|
+
// Tune feed rate and temperature to minimise a defect rate
|
|
45
|
+
export const tune = East.function([], MADS.Types.ResultType, $ => {
|
|
46
|
+
const objective = $.const(East.function([VectorType(FloatType)], FloatType, ($, x) => {
|
|
47
|
+
const feed = $.let(x.get(0n).subtract(5.0));
|
|
48
|
+
const temp = $.let(x.get(1n).subtract(180.0));
|
|
49
|
+
$.return(feed.multiply(feed).add(temp.multiply(temp).multiply(0.01)));
|
|
50
|
+
}));
|
|
51
|
+
const x0 = $.let(East.Vector.fromArray([8.0, 200.0]));
|
|
52
|
+
const bounds = $.let({ lower: East.Vector.fromArray([1.0, 100.0]), upper: East.Vector.fromArray([10.0, 250.0]) });
|
|
53
|
+
const config = $.let({
|
|
54
|
+
max_bb_eval: some(100n), display_degree: some(0n), direction_type: none,
|
|
55
|
+
initial_mesh_size: none, min_mesh_size: none, seed: some(42n),
|
|
56
|
+
});
|
|
57
|
+
$.return(MADS.optimize(objective, x0, bounds, none, config)); // none: no constraints
|
|
58
|
+
});
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Decision Tree: Which Module to Use
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
Task → What do you need?
|
|
65
|
+
│
|
|
66
|
+
├─ MADS (derivative-free continuous optimisation) → .optimize(objective, x0, bounds, constraints: Option, config)
|
|
67
|
+
├─ Optuna (Bayesian hyperparameter tuning) → .optimize()
|
|
68
|
+
├─ SimAnneal (discrete / combinatorial) → .optimize(), .optimizePermutation(), .optimizeSubset()
|
|
69
|
+
├─ ALNS (adaptive large neighbourhood search) → .optimize([SolutionType], initial, objective, destroys, repairs, config)
|
|
70
|
+
│ — generic over the solution type S: define your own struct
|
|
71
|
+
├─ Optimization (iterative coordinate descent over integer vectors)
|
|
72
|
+
│ ├─ .iterative(objective, paramSpaces, config) — the whole objective per move
|
|
73
|
+
│ ├─ .iterativeIncremental(elementObjective, paramSpaces, config) — a per-element contribution, recomputed only
|
|
74
|
+
│ │ where a move changed it (coordinate or swap mode)
|
|
75
|
+
│ └─ .iterativeGrouped(groupObjective, paramSpaces, config) — contributions grouped by VALUE (employees, bins,
|
|
76
|
+
│ vehicles): a move from A to B recomputes groups A and B
|
|
77
|
+
├─ GoogleOr (OR-Tools)
|
|
78
|
+
│ ├─ CP-SAT → .cpsatSolve(), .cpsatSolveAll()
|
|
79
|
+
│ ├─ Routing → .routingSolve() (TSP, CVRP, VRPTW, VRPPD)
|
|
80
|
+
│ ├─ Linear → .linearSolve() (LP, MIP)
|
|
81
|
+
│ └─ Graph → .minCostFlow(), .maxFlow(), .assignment() (dense cost matrix),
|
|
82
|
+
│ .minCostAssignment() (sparse arcs, task capacity, opt-out penalty)
|
|
83
|
+
├─ Scipy
|
|
84
|
+
│ ├─ Optimisation → .optimizeMinimize(), .optimizeMinimizeQuadratic(), .optimizeDualAnnealing()
|
|
85
|
+
│ ├─ Statistics → .statsDescribe(), .statsPearsonr(), .statsSpearmanr(), .statsPercentile(), .statsPercentileOfScore(),
|
|
86
|
+
│ │ .statsIqr(), .statsMedian(), .statsMad(), .statsRobust()
|
|
87
|
+
│ ├─ Histogram / KDE → .histogram(), .kdeFit(), .kdeEvaluate()
|
|
88
|
+
│ ├─ Curve fitting → .curveFit()
|
|
89
|
+
│ └─ Interpolation → .interpolate1dFit(), .interpolate1dPredict()
|
|
90
|
+
├─ XGBoost → train .trainRegressor(), .trainClassifier(), .trainQuantile() · predict .predict(), .predictClass(),
|
|
91
|
+
│ .predictProba(), .predictQuantile()
|
|
92
|
+
├─ LightGBM → train .trainRegressor(), .trainClassifier() · predict .predict(), .predictClass(), .predictProba()
|
|
93
|
+
├─ NGBoost (probabilistic boosting) → .trainRegressor() · .predict(), .predictDist()
|
|
94
|
+
├─ Torch (MLP) → train .mlpTrain(), .mlpTrainMulti() · predict .mlpPredict(), .mlpPredictMulti() · embeddings
|
|
95
|
+
│ .mlpEncode(), .mlpDecode()
|
|
96
|
+
├─ Lightning (PyTorch Lightning)
|
|
97
|
+
│ ├─ .train(X, y, config, masks, group_weights, conditions) · .predict(model, X, masks, conditions)
|
|
98
|
+
│ ├─ Embeddings (autoencoders) → .encode(), .decode(), .decodeConditional()
|
|
99
|
+
│ ├─ Generation (sequential models) → .generateSequence(model, prefix, condition: Option, config{n_steps,
|
|
100
|
+
│ │ temperature (0 = argmax), return_probs}) — the generated steps, not the prefix
|
|
101
|
+
│ ├─ Architectures → mlp · autoencoder · conv1d (temporal) · sequential (LSTM/GRU, temporal) · transformer (temporal)
|
|
102
|
+
│ ├─ Output modes → regression (MSE) · binary (BCE; per-position pos_weights, masks) ·
|
|
103
|
+
│ │ multi_head (N CE heads; per-head class_weights, masks)
|
|
104
|
+
│ └─ condition_dim for conditional generation in the temporal architectures; early stopping, gradient clipping,
|
|
105
|
+
│ epoch callbacks, group_weights
|
|
106
|
+
├─ GP (Gaussian process regression) → .train() · .predict(), .predictStd()
|
|
107
|
+
├─ MAPIE (conformal prediction)
|
|
108
|
+
│ ├─ Regression → .trainConformalRegressor(), .trainCQR() · classification → .trainConformalClassifier()
|
|
109
|
+
│ ├─ Predict → .predictInterval(), .predictSet()
|
|
110
|
+
│ └─ For SHAP → .uncertaintyPredictorRegressor(), .uncertaintyPredictorClassifier()
|
|
111
|
+
├─ Sklearn
|
|
112
|
+
│ ├─ Splitting → .split() (N-way, stratify, overlap, multi_overlap) · overlap filtering → .overlap()
|
|
113
|
+
│ ├─ Scaling → .standardScalerFit/Transform(), .minMaxScalerFit/Transform(), .robustScalerFit/Transform()
|
|
114
|
+
│ ├─ Encoding → .labelEncoderFit/Transform/InverseTransform(), .ordinalEncoderFit/Transform()
|
|
115
|
+
│ ├─ Class weights → .computeClassWeight()
|
|
116
|
+
│ ├─ Metrics → .computeMetrics(), .computeMetricsMulti() · .computeClassificationMetrics(),
|
|
117
|
+
│ │ .computeClassificationMetricsMulti() · .rocAucScore(), .logLoss(), .confusionMatrix()
|
|
118
|
+
│ ├─ Multi-target → .regressorChainTrain(), .regressorChainPredict()
|
|
119
|
+
│ └─ GMM clustering → .gmmFit(), .gmmPredict(), .gmmPredictProba(), .gmmScoreSamples(), .gmmSample(), .gmmBic(),
|
|
120
|
+
│ .gmmAic() · .silhouetteScore()
|
|
121
|
+
├─ PyMC (Bayesian inference)
|
|
122
|
+
│ ├─ Train → .trainRegression(), .trainHierarchical(), .trainMultiLayer() · predict → .predict(), .predictDistribution()
|
|
123
|
+
│ └─ Posterior → .posteriorSummary(), .posteriorSamples() · diagnostics → .diagnostics(), .posteriorPredictiveCheck()
|
|
124
|
+
├─ Simulation (economic-ontology simulation by DES) → .run([R, E], initialState, initialEvents, process, config)
|
|
125
|
+
├─ Causal (one declarative experiment and an honesty verdict) — generic over the row struct: data is an
|
|
126
|
+
│ Array<Struct> (fields = columns), the row type a type argument, config naming columns by field
|
|
127
|
+
│ ├─ Did X change Y, and can I trust it? → .experiment([Row], data, config)
|
|
128
|
+
│ │ (binary treatment; naive vs adjusted effect, confounder balance, propensity overlap, a placebo / E-value
|
|
129
|
+
│ │ robustness check, and a verdict: causal / modest / adjustment_insufficient / non_identifiable_positivity /
|
|
130
|
+
│ │ not_estimable; `adjusted` is none when the engine refuses. DoWhy / EconML / PyALE are internal.)
|
|
131
|
+
│ └─ What real trial would confirm it? → .designValidation([Row], data, config, result, designConfig)
|
|
132
|
+
│ (a randomised-trial recipe: sample size, split options, match-on categories, power curve, rationale)
|
|
133
|
+
│ Honesty caveats: `causal` means robust to the OBSERVED backdoor set and the refuters that ran — NOT correctly
|
|
134
|
+
│ signed, free of reverse causation or of unobserved confounding. overlap.support_strength (refused / thin /
|
|
135
|
+
│ strong vs config.strong_overlap, default 0.55) tempers thin support to modest; opt-in config.evalue_floor folds
|
|
136
|
+
│ a weak E-value (risk-ratio scale) into modest; opt-in config.expected_sign flags an implausibly signed effect
|
|
137
|
+
│ (refutation.expected_sign_ok = some(false), verdict adjustment_insufficient). Clustered designs
|
|
138
|
+
│ (bootstrap.cluster_column) cluster the placebo and the naive CI too.
|
|
139
|
+
└─ Shap (explainability)
|
|
140
|
+
├─ Create → .treeExplainerCreate() (XGBoost only), .kernelExplainerCreate() (any model)
|
|
141
|
+
├─ Compute → .computeValues(), .featureImportance()
|
|
142
|
+
└─ KernelExplainer takes XGBoost, LightGBM, NGBoost, GP, Torch, RegressorChain and MAPIE models
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
## Data and Types
|
|
146
|
+
|
|
147
|
+
Data is East tensors: features `MatrixType(FloatType)` (rows × features),
|
|
148
|
+
targets `VectorType(FloatType)`, class labels `VectorType(IntegerType)`; the
|
|
149
|
+
predictions come back as vectors. Build them with
|
|
150
|
+
`East.Matrix.fromArray([[…], […]])` and `East.Vector.fromArray([…])` in
|
|
151
|
+
TypeScript, `EastMatrix` / `EastVector` (numpy) in python. The package
|
|
152
|
+
re-exports East's constructors as `VectorType` / `SharedVectorType` and
|
|
153
|
+
`SharedMatrixType`.
|
|
154
|
+
|
|
155
|
+
Each module's own types are under `Module.Types.*` — and at the top level
|
|
156
|
+
under a prefixed name (`MADSConfigType`, `XGBoostModelBlobType`, …):
|
|
157
|
+
|
|
158
|
+
```typescript
|
|
159
|
+
import { MADS, Optuna, ALNS, Sklearn, XGBoost, ModelBlobType } from "@elaraai/east-py-datascience";
|
|
160
|
+
|
|
161
|
+
MADS.Types.BoundsType // { lower, upper } vectors
|
|
162
|
+
MADS.Types.ConfigType // every field an Option: some(v) / none
|
|
163
|
+
MADS.Types.ConstraintType // variant eb / pb over a constraint function
|
|
164
|
+
MADS.Types.ResultType // { x_best, f_best, success, … }
|
|
165
|
+
Optuna.Types.ParamSpaceType // one parameter's search space
|
|
166
|
+
Optuna.Types.StudyResultType
|
|
167
|
+
ALNS.Types.ConfigType // and ALNS.Types.ResultType, over the solution type
|
|
168
|
+
Sklearn.Types.SplitConfigType
|
|
169
|
+
XGBoost.Types.ModelBlobType // a trained model
|
|
170
|
+
ModelBlobType // the union of every library's model blob, for an arbitrary trained model
|
|
171
|
+
// (AnyModelBlobType is the SHAP explainer's input union)
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
A model blob round-trips: what a train function returns is exactly what the
|
|
175
|
+
predict and explain functions take — store it in a dataset, pass it between
|
|
176
|
+
tasks, or hand it to `Shap`.
|
|
177
|
+
|
|
178
|
+
## Common Patterns
|
|
179
|
+
|
|
180
|
+
### Train and predict
|
|
181
|
+
|
|
182
|
+
```typescript
|
|
183
|
+
const fit = East.function([], VectorType(FloatType), $ => {
|
|
184
|
+
const X = $.let(East.Matrix.fromArray([[1.0, 2.0], [2.0, 1.0], [3.0, 4.0], [4.0, 3.0]]));
|
|
185
|
+
const y = $.let(East.Vector.fromArray([3.0, 3.0, 7.0, 7.0]));
|
|
186
|
+
const config = $.let({ // every field an Option: some(value) / none
|
|
187
|
+
n_estimators: some(100n), max_depth: some(3n), learning_rate: none, min_child_weight: none,
|
|
188
|
+
subsample: none, colsample_bytree: none, reg_alpha: none, reg_lambda: none, gamma: none,
|
|
189
|
+
random_state: some(42n), n_jobs: none, sample_weight: none, categorical_features: none,
|
|
190
|
+
categorical_n: none, max_cat_to_onehot: none, max_cat_threshold: none, scale_pos_weight: none,
|
|
191
|
+
}, XGBoost.Types.XGBoostConfigType);
|
|
192
|
+
const model = $.let(XGBoost.trainRegressor(X, y, config));
|
|
193
|
+
$.return(XGBoost.predict(model, X));
|
|
194
|
+
});
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### Optimisation
|
|
198
|
+
|
|
199
|
+
The objective is an East function of the decision vector —
|
|
200
|
+
`East.function([VectorType(FloatType)], FloatType, …)` — bound with `$.const`
|
|
201
|
+
and passed with the start point, the bounds and the config; the result
|
|
202
|
+
carries `x_best` and `f_best` (see the Quick Start).
|
|
203
|
+
|
|
204
|
+
## Calling from Python (the functions directly)
|
|
205
|
+
|
|
206
|
+
The platform functions are exported from `east_py_datascience` as **plain
|
|
207
|
+
callables over East values** — no IR, no compile — and, the same objects,
|
|
208
|
+
callable inside an `East.function` body, where the call is the `Platform`
|
|
209
|
+
node with the declared signature. A project's own `@East.platform_function`
|
|
210
|
+
imports and calls them (the preferred way to use lightning / torch / xgboost
|
|
211
|
+
/ sklearn from project python); a python East body calls them the same way and
|
|
212
|
+
compiles against the package's `platform` list:
|
|
213
|
+
|
|
214
|
+
```python
|
|
215
|
+
from east import East, FloatType, MatrixType, VectorType, coerce_to, none, some
|
|
216
|
+
from east_py_datascience import xgboost_predict, xgboost_train_regressor
|
|
217
|
+
from east_py_datascience.xgboost import XGBoostConfigType
|
|
218
|
+
|
|
219
|
+
@East.platform_function(inputs=[MatrixType(FloatType), VectorType(FloatType), MatrixType(FloatType)],
|
|
220
|
+
output=VectorType(FloatType))
|
|
221
|
+
def forecast(X_train, y_train, X_new):
|
|
222
|
+
config = coerce_to({
|
|
223
|
+
"n_estimators": some(200), "max_depth": some(4), "learning_rate": some(0.05),
|
|
224
|
+
"min_child_weight": none, "subsample": none, "colsample_bytree": none,
|
|
225
|
+
"reg_alpha": none, "reg_lambda": none, "gamma": none,
|
|
226
|
+
"random_state": some(42), "n_jobs": none, "sample_weight": none,
|
|
227
|
+
"categorical_features": none, "categorical_n": none,
|
|
228
|
+
"max_cat_to_onehot": none, "max_cat_threshold": none,
|
|
229
|
+
"scale_pos_weight": none, # the binary class-imbalance weight (classifiers)
|
|
230
|
+
}, XGBoostConfigType)
|
|
231
|
+
model = xgboost_train_regressor(X_train, y_train, config) # a model blob, East in and out
|
|
232
|
+
return xgboost_predict(model, X_new)
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Rules:
|
|
236
|
+
|
|
237
|
+
- **Inputs are East values, not numpy**: `EastMatrix` / `EastVector` for data,
|
|
238
|
+
an `EastStruct` config built with `coerce_to(dict, ConfigType)` — every
|
|
239
|
+
`Option` field spelled `some(value)` or `none` (a plain value or `None` is
|
|
240
|
+
refused) — and `variant(case, value, Type)` for a variant.
|
|
241
|
+
- **Config and blob types live in each submodule**
|
|
242
|
+
(`east_py_datascience.xgboost.XGBoostConfigType`, `…XGBoostModelBlobType`),
|
|
243
|
+
mirroring the TypeScript `Module.Types.*`.
|
|
244
|
+
- **Names are the platform names** (`xgboost_train_regressor`,
|
|
245
|
+
`mads_optimize`, …), except GoogleOr's, exported without the `google_or_`
|
|
246
|
+
prefix: `cpsat_solve`, `cpsat_solve_all`, `routing_solve`, `linear_solve`,
|
|
247
|
+
`min_cost_flow`, `max_flow`, `assignment`, `min_cost_assignment`.
|
|
248
|
+
- **Inside an East body**: `East.function([...], ..., lambda b, X, y, cfg:
|
|
249
|
+
xgboost_predict(xgboost_train_regressor(X, y, cfg), X))` — the calls are
|
|
250
|
+
`Platform` nodes; `East.compile(fn, platform=east_py_datascience.platform)`
|
|
251
|
+
runs it (the list the e3 python runner registers).
|
|
252
|
+
- **The generic ones take the type argument FIRST in a body**, as TypeScript
|
|
253
|
+
reads: `causal_experiment(RowType, rows, config)`,
|
|
254
|
+
`alns_optimize(SolutionType, initial, objective, destroy, repair, config)`.
|
|
255
|
+
From python they take the values alone (`causal_experiment(rows, config)`):
|
|
256
|
+
they read the values, not the type.
|
|
257
|
+
- **`simulation_run` and `optimization_iterative*` run in C**, so there is no
|
|
258
|
+
python to call: the name exports the declaration, and a body calls it the
|
|
259
|
+
same way — `simulation_run(Resources, Events, state, events, process,
|
|
260
|
+
config)`.
|
|
261
|
+
- Optional dependencies gate at call time: a function raises
|
|
262
|
+
`NotImplementedError` naming the missing extra (`east-py-datascience[causal]`).
|
|
263
|
+
|
|
264
|
+
For the East-value API (eager methods, `coerce_to`, `to_numpy` / `to_torch`,
|
|
265
|
+
`@East.platform_function`), load the **east-py** skill.
|
|
266
|
+
|
|
267
|
+
## Related skills
|
|
268
|
+
|
|
269
|
+
- **e3** — **required to run these**: they need the python runtime, so wrap each
|
|
270
|
+
call in an `e3.task` with a python runner (`{ runner: { runtime: 'east-py',
|
|
271
|
+
platforms: ['east-py-datascience'] } }` — the typed runner resolves east-py
|
|
272
|
+
from the project's `.venv`, no `uv run` wrapper needed). They do not run on
|
|
273
|
+
the Node or C runtimes.
|
|
274
|
+
- **east** — the language for objective functions, configs and results.
|
|
275
|
+
- **east-py** — the python runtime: East values as plain python data, eager
|
|
276
|
+
methods, and the `@East.platform_function` on-ramp the direct calls live in.
|
|
277
|
+
- **east-ontology** — the decisions these models improve are the `decision`
|
|
278
|
+
nodes of the business's economic ontology.
|
|
279
|
+
- **east-design** — place the forecast or optimisation in a decision-oriented
|
|
280
|
+
architecture.
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: east-py-io
|
|
3
|
+
description: "I/O platform functions for the East language on the Python runtime - SQL databases (SQLite, PostgreSQL, MySQL, Access), NoSQL (Redis, MongoDB), S3 storage, file transfers (FTP, SFTP), file formats (XLSX, XML), and compression (Gzip, Zip, Tar). Use when writing Python (not the TypeScript DSL) that calls or registers these platform functions. Triggers for: (1) Calling east_py_io functions (sqlite_select, mongo_find, s3_put_object, ftp_get, xlsx_read, gzip_compress, ...) from a project's own Python @East.platform_function, or inside an East.function body — the same object does both, (2) Registering the east_py_io platform list with compile() so East programs can use SQL/NoSQL/Storage/Transfer/Format/Compression on the Python runtime, (3) Building connection configs (SqliteConfigType, PostgresConfigType, RedisConfigType, S3ConfigType, FtpConfigType, ...) in Python, (4) The connect -) handle -) operate -) close connection… See the detailed scope below."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
## Detailed skill scope
|
|
7
|
+
|
|
8
|
+
I/O platform functions for the East language on the Python runtime - SQL databases (SQLite, PostgreSQL, MySQL, Access), NoSQL (Redis, MongoDB), S3 storage, file transfers (FTP, SFTP), file formats (XLSX, XML), and compression (Gzip, Zip, Tar). Use when writing Python (not the TypeScript DSL) that calls or registers these platform functions. Triggers for: (1) Calling east_py_io functions (sqlite_select, mongo_find, s3_put_object, ftp_get, xlsx_read, gzip_compress, ...) from a project's own Python @East.platform_function, or inside an East.function body — the same object does both, (2) Registering the east_py_io platform list with compile() so East programs can use SQL/NoSQL/Storage/Transfer/Format/Compression on the Python runtime, (3) Building connection configs (SqliteConfigType, PostgresConfigType, RedisConfigType, S3ConfigType, FtpConfigType, ...) in Python, (4) The connect -> handle -> operate -> close connection lifecycle. For authoring East programs in TypeScript against these functions, use the east-node-io skill instead.
|
|
9
|
+
|
|
10
|
+
# East.py I/O Platform Functions
|
|
11
|
+
|
|
12
|
+
`east_py_io` is the Python implementation of the East I/O platform: SQL and
|
|
13
|
+
NoSQL databases, S3 object storage, FTP/SFTP transfers, XLSX/XML formats and
|
|
14
|
+
compression. Each function is **dual-mode**: a plain python callable over East
|
|
15
|
+
values — call it from your own `@East.platform_function` — and, the same
|
|
16
|
+
object, callable inside an `East.function` / `East.asyncFunction` body, where
|
|
17
|
+
the call IS the `Platform` node with the function's declared signature.
|
|
18
|
+
Nothing restates the signature: register the `platform` list at
|
|
19
|
+
`East.compile` / `East.compileAsync` (or let the runner) and it runs.
|
|
20
|
+
|
|
21
|
+
Every function but `xlsx_*`, `xml_*`, `zip_*` and `ftp_close` is **async**: a
|
|
22
|
+
direct call returns a coroutine to `await`, and a body calling one is an
|
|
23
|
+
`East.asyncFunction`. Exported names are the platform names, except MongoDB's,
|
|
24
|
+
exported as `mongo_*` (`mongo_find` is the platform's `mongodb_find_many`).
|
|
25
|
+
|
|
26
|
+
This skill is for **Python**. To author East programs in TypeScript against
|
|
27
|
+
these functions (`SQL.SQLite.query`, `Storage.S3.putObject`, …), load
|
|
28
|
+
**east-node-io** — the same functions, under the same platform names.
|
|
29
|
+
|
|
30
|
+
## Before writing code — search the example index
|
|
31
|
+
|
|
32
|
+
Every East API has a tested example in the plugin's index — the index IS the
|
|
33
|
+
API reference, printed from each example's IR in TypeScript or python. Before
|
|
34
|
+
writing or changing East code:
|
|
35
|
+
|
|
36
|
+
1. Call `search_east_examples` for each capability you
|
|
37
|
+
are about to use — `language: "python"` for east-py, `"typescript"`
|
|
38
|
+
otherwise. Summaries come back first: id, signature, the inputs and the
|
|
39
|
+
expected result, a few hundred bytes each.
|
|
40
|
+
2. Fetch the one or two that match with `get_east_example`
|
|
41
|
+
and pattern your code on them.
|
|
42
|
+
3. Do not read `node_modules/@elaraai/**` or `*.examples.ts` files wholesale,
|
|
43
|
+
and do not reason from `.d.ts` signatures: the index holds the same
|
|
44
|
+
programs, exact and far cheaper, and the signatures omit the runtime rules
|
|
45
|
+
that make East code correct.
|
|
46
|
+
|
|
47
|
+
Nothing is injected for you; the search is the step.
|
|
48
|
+
|
|
49
|
+
## Quick Start
|
|
50
|
+
|
|
51
|
+
```python
|
|
52
|
+
from east import ArrayType, BlobType, East, FloatType, StringType, StructType, coerce_to, none, some
|
|
53
|
+
from east_py_io import (GzipOptionsType, SqliteConfigType, gzip_compress, platform,
|
|
54
|
+
sqlite_close, sqlite_connect, sqlite_select)
|
|
55
|
+
|
|
56
|
+
OrderRow = StructType([("sku", StringType), ("qty", FloatType)])
|
|
57
|
+
|
|
58
|
+
@East.platform_function(inputs=[StringType], output=ArrayType(OrderRow))
|
|
59
|
+
async def load_orders(db_path):
|
|
60
|
+
# An Option field takes some(x) / none — a plain value (or None) is refused
|
|
61
|
+
config = coerce_to({"path": db_path, "readOnly": some(True), "memory": none}, SqliteConfigType)
|
|
62
|
+
handle = await sqlite_connect(config) # a connection handle: String
|
|
63
|
+
try:
|
|
64
|
+
select = sqlite_select(None, OrderRow) # the typed read — rows decode to YOUR struct
|
|
65
|
+
return await select(handle, "SELECT sku, qty FROM orders", [])
|
|
66
|
+
finally:
|
|
67
|
+
await sqlite_close(handle)
|
|
68
|
+
|
|
69
|
+
# The same functions inside an East body — the call is the Platform node, the
|
|
70
|
+
# package's own named types spell the options; an async implementation needs an
|
|
71
|
+
# East.asyncFunction body, compiled with East.compileAsync and the package's list
|
|
72
|
+
packed = East.asyncFunction([BlobType], BlobType,
|
|
73
|
+
lambda b, data: gzip_compress(data, East.value({"level": some(9)}, GzipOptionsType)))
|
|
74
|
+
East.compileAsync(packed, platform=platform)
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Run a compiled async body through the runner (`east-py run`, an e3 task): the
|
|
78
|
+
bridge runs async implementations on its own event loop, so the compiled
|
|
79
|
+
coroutine cannot be awaited under `asyncio.run` once it calls one.
|
|
80
|
+
|
|
81
|
+
## Connection Lifecycle
|
|
82
|
+
|
|
83
|
+
Databases, transfers and caches share one pattern: `*_connect` (Access:
|
|
84
|
+
`access_open`) takes a config and returns an opaque **connection handle** (a
|
|
85
|
+
`String`, valid for the process lifetime), every operation takes the handle,
|
|
86
|
+
and `*_close(handle)` / `*_close_all()` release it — always close, since
|
|
87
|
+
handles are pooled per process. S3 has no connection: each call takes the
|
|
88
|
+
`S3ConfigType`.
|
|
89
|
+
|
|
90
|
+
## Decision Tree: What Do You Need?
|
|
91
|
+
|
|
92
|
+
```
|
|
93
|
+
Task → What do you need?
|
|
94
|
+
│
|
|
95
|
+
├─ SQL — sqlite_* (apsw) · postgres_* (asyncpg) · mysql_* (aiomysql): connect / query / select / close / close_all
|
|
96
|
+
│ ├─ Typed rows → *_select, generic: in a body sqlite_select(Row, handle, sql, params) (the row type FIRST, as
|
|
97
|
+
│ │ TypeScript reads); from python the factory sqlite_select(None, Row)(handle, sql, params)
|
|
98
|
+
│ ├─ Anything else → *_query(handle, sql, params) -> SqlResultType, a variant by statement kind:
|
|
99
|
+
│ │ select {rows: each a Dict<String, LiteralValue>} · insert {rowsAffected, lastInsertId: Option} ·
|
|
100
|
+
│ │ update / delete {rowsAffected}
|
|
101
|
+
│ ├─ params → an Array of LiteralValueType (Variant<Blob, Boolean, DateTime, Float, Integer, Null, String>)
|
|
102
|
+
│ └─ MS Access, read-only (access-parser) → access_open(config) · access_tables(handle) -> {tables} ·
|
|
103
|
+
│ access_query(Row, handle, AccessQueryOptionsType{table, columns, rowOffset, rowLimit: Options}) — a table
|
|
104
|
+
│ read, generic like *_select (the factory from python) · access_close / access_close_all
|
|
105
|
+
│
|
|
106
|
+
├─ NoSQL
|
|
107
|
+
│ ├─ MongoDB (motor; python names mongo_*) → mongo_connect(MongoConfigType{uri, database, collection}) ·
|
|
108
|
+
│ │ mongo_insert_one(h, doc) -> id String · mongo_find_one(h, filter) -> Option<doc> ·
|
|
109
|
+
│ │ mongo_find(h, filter, MongoFindOptionsType{limit, skip}) -> Array<doc> · mongo_update_one(h, filter, update) ·
|
|
110
|
+
│ │ mongo_delete_one / mongo_delete_many(h, filter) -> count · mongo_close / mongo_close_all
|
|
111
|
+
│ │ (a document is a Dict<String, BsonValueType>; _id comes back as a String)
|
|
112
|
+
│ └─ Redis (redis) → redis_connect · redis_get(h, key) -> Option<String> · redis_set(h, key, value) ·
|
|
113
|
+
│ redis_setex(h, key, value, seconds) · redis_del(h, key) -> count · redis_close / redis_close_all
|
|
114
|
+
│
|
|
115
|
+
├─ Object storage (boto3; each call takes the S3ConfigType first)
|
|
116
|
+
│ └─ s3_put_object(cfg, key, blob) · s3_get_object(cfg, key) -> Blob · s3_head_object(cfg, key) ·
|
|
117
|
+
│ s3_delete_object(cfg, key) · s3_list_objects(cfg, prefix, max_keys, continuation_token: Option) -> S3ListResultType ·
|
|
118
|
+
│ s3_presign_url(cfg, key, expires_in_seconds) -> String
|
|
119
|
+
│
|
|
120
|
+
├─ File transfer — ftp_* (aioftp) · sftp_* (asyncssh)
|
|
121
|
+
│ └─ connect · list(h, path) -> FileListType · get(h, path) -> Blob · put(h, path, blob) · delete(h, path) · close / close_all
|
|
122
|
+
│
|
|
123
|
+
├─ File formats (sync)
|
|
124
|
+
│ ├─ XLSX (openpyxl) → xlsx_read(blob, XlsxReadOptionsType{sheetName: Option}) -> Array<Array<LiteralValue>>, the
|
|
125
|
+
│ │ sheet as a grid of cells · xlsx_write(grid, options) -> Blob · xlsx_info(blob) -> {sheets: [{name, rowCount, columnCount}]}
|
|
126
|
+
│ └─ XML (defusedxml) → xml_parse(blob, XmlParseConfigType) -> XmlNodeType (recursive) · xml_serialize(node, XmlSerializeConfigType) -> Blob
|
|
127
|
+
│
|
|
128
|
+
└─ Compression (core — the standard library, no extra)
|
|
129
|
+
├─ Gzip → gzip_compress(blob, GzipOptionsType{level: Option}) · gzip_decompress(blob) (async)
|
|
130
|
+
├─ Zip → zip_compress(ZipEntriesType, ZipOptionsType) -> Blob · zip_decompress(blob) -> Dict<name, Blob> (sync)
|
|
131
|
+
└─ Tar → tar_create(TarEntriesType) -> Blob · tar_extract(blob) -> Dict<name, Blob> (async)
|
|
132
|
+
(entries: Array<{name, data: Blob}>)
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## Optional Dependencies
|
|
136
|
+
|
|
137
|
+
Each driver is an extra named after its module; a function whose extra is
|
|
138
|
+
missing raises `NotImplementedError` naming it (`SQLite support requires the
|
|
139
|
+
'sqlite' extra. Add east-py-io[sqlite] to your pyproject.toml dependencies.`):
|
|
140
|
+
|
|
141
|
+
| Extra | Driver | Extra | Driver |
|
|
142
|
+
|-------|--------|-------|--------|
|
|
143
|
+
| `sqlite` | apsw | `redis` | redis |
|
|
144
|
+
| `postgres` | asyncpg | `mongodb` | motor |
|
|
145
|
+
| `mysql` | aiomysql | `xlsx` | openpyxl |
|
|
146
|
+
| `access` | access-parser | `xml` | defusedxml |
|
|
147
|
+
| `s3` | boto3 | `ftp` / `sftp` | aioftp / asyncssh |
|
|
148
|
+
|
|
149
|
+
Convenience groups: `sql`, `nosql`, `all`. Compression is core (the standard
|
|
150
|
+
library).
|
|
151
|
+
|
|
152
|
+
## East Type Definitions
|
|
153
|
+
|
|
154
|
+
Every config and result type is importable from `east_py_io`. The ones you
|
|
155
|
+
build:
|
|
156
|
+
|
|
157
|
+
| Type | Fields |
|
|
158
|
+
|------|--------|
|
|
159
|
+
| `SqliteConfigType` | `path`, `readOnly: Option<Boolean>`, `memory: Option<Boolean>` |
|
|
160
|
+
| `PostgresConfigType` / `MySqlConfigType` | `host`, `port`, `database`, `user`, `password`, `ssl: Option<Boolean>`, `maxConnections: Option<Integer>` |
|
|
161
|
+
| `AccessConfigType` | `path`, `password: Option<String>` |
|
|
162
|
+
| `RedisConfigType` | `host`, `port`, `password: Option<String>`, `db: Option<Integer>`, `keyPrefix: Option<String>` |
|
|
163
|
+
| `MongoConfigType` | `uri`, `database`, `collection` |
|
|
164
|
+
| `S3ConfigType` | `region`, `bucket`, `accessKeyId: Option<String>`, `secretAccessKey: Option<String>`, … |
|
|
165
|
+
| `FtpConfigType` / `SftpConfigType` | `host`, `port`, `user`, `password`, `secure: Boolean` / `host`, `port`, `username`, `password: Option<String>`, `privateKey: Option<String>` |
|
|
166
|
+
| `XlsxReadOptionsType` · `XmlParseConfigType` · `XmlSerializeConfigType` | `sheetName: Option<String>` · `preserveWhitespace`, `decodeEntities` · `indent: Option<String>`, `includeXmlDeclaration`, `encodeEntities`, `selfClosingTags` |
|
|
167
|
+
| `GzipOptionsType` / `ZipOptionsType` · `ZipEntriesType` / `TarEntriesType` | `level: Option<Integer>` · `Array<{name, data: Blob}>` |
|
|
168
|
+
|
|
169
|
+
Results come back as the documented types — `SqlResultType`, `BsonValueType`
|
|
170
|
+
documents, `S3ListResultType` (`objects`, `isTruncated`,
|
|
171
|
+
`continuationToken`), `FileListType` (`name`, `path`, `size`, `isDirectory`,
|
|
172
|
+
`modifiedTime`), `XmlNodeType`, `LiteralValueType` cells. Build a config with
|
|
173
|
+
`coerce_to({...}, ConfigType)`: an `Option` field takes `some(value)` or
|
|
174
|
+
`none`, never a plain value or `None`.
|
|
175
|
+
|
|
176
|
+
## Related skills
|
|
177
|
+
|
|
178
|
+
- **east-py** — the python runtime: East values as plain data, eager methods,
|
|
179
|
+
`coerce_to`, and the `@East.platform_function` on-ramp these calls live
|
|
180
|
+
inside (and the dual-mode rule that makes them callable in a body).
|
|
181
|
+
- **east-py-std** — the standard-library sibling: console, filesystem, fetch,
|
|
182
|
+
crypto, time, path, random, large JSON.
|
|
183
|
+
- **east-py-datascience** — ML and optimisation platform functions.
|
|
184
|
+
- **east-node-io** — the TypeScript authoring surface for the same functions.
|
|
185
|
+
- **e3** — the execution engine whose python runner registers this list for
|
|
186
|
+
dataflow tasks.
|