@aid-framework/runtime 0.1.39 → 0.1.51

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +171 -41
  2. package/dist/.tsbuildinfo +1 -1
  3. package/package.json +7 -7
package/README.md CHANGED
@@ -1,53 +1,183 @@
1
- # `@aid-framework/runtime` (TypeScript)
1
+ # `@aid-framework/runtime`
2
2
 
3
- The TypeScript runtime for the AID framework. It mirrors the Python runtime in
4
- `runtimes/python/aid_runtime` module for module, so an application generated for
5
- either language meets the same gateway semantics, the same pinned-model rules, the
6
- same eval-gate arithmetic and the same retrieval surface.
3
+ The TypeScript runtime for [AID Framework](https://github.com/aid-framework/aid-framework): the library a
4
+ generated application calls at runtime. Pin a model, run a prompt, call a tool, trace it, stay inside a budget,
5
+ and prove that quality did not regress.
6
+
7
+ Generated code is thin glue. This is where the behaviour lives — which is why it is a normal dependency of the
8
+ generated app, not part of the compiler.
9
+
10
+ ## Install
11
+
12
+ ```sh
13
+ npm install @aid-framework/runtime
14
+ ```
15
+
16
+ Requires Node.js 24 or newer. Ships as ESM with TypeScript declarations. `zod` is used for structured output and
17
+ nothing else.
18
+
19
+ ## Quick start
20
+
21
+ A gateway resolves aliases to pinned models, enforces budgets, records cost and opens a span per call:
22
+
23
+ ```ts
24
+ import {
25
+ Budget, ModelGateway, OpenAICompatibleProvider, PinnedModel,
26
+ } from '@aid-framework/runtime';
27
+
28
+ const gateway = new ModelGateway({
29
+ providers: {
30
+ 'openai-compatible': new OpenAICompatibleProvider({ apiKey: process.env.OPENAI_API_KEY }),
31
+ },
32
+ aliases: {
33
+ small: new PinnedModel({
34
+ alias: 'small',
35
+ spec: {
36
+ name: 'gpt-4o-2024-08-06',
37
+ provider: 'openai-compatible',
38
+ contextWindow: 128_000,
39
+ maxOutputTokens: 4_096,
40
+ cost: { inputPer1kUsd: 0.0025, outputPer1kUsd: 0.01 },
41
+ },
42
+ }),
43
+ },
44
+ budget: new Budget({ maxUsd: 5, scope: 'nightly' }),
45
+ });
46
+
47
+ const result = await gateway.generate('small', { prompt: 'hello' });
48
+
49
+ result.text;
50
+ result.usage;
51
+ result.costUsd;
52
+ result.attempts; // every alias tried, in order
53
+ result.fellBack;
54
+ ```
55
+
56
+ **An alias resolves to an exact, pinned model id.** That is not cosmetic: a baseline recorded against `gpt-4o`
57
+ means nothing if the alias floats to a later snapshot on the next run. An unknown alias therefore raises
58
+ `UnknownModelAliasError` rather than falling through to a provider default.
59
+
60
+ ## A pipeline
61
+
62
+ Control flow is plain async TypeScript. There is no graph engine and no orchestration framework: a generated app
63
+ is ordinary code that is easy to read, step through and test.
64
+
65
+ ```ts
66
+ import { Pipeline, PipelineContext } from '@aid-framework/runtime';
67
+
68
+ const pipeline = new Pipeline([
69
+ { name: 'load', run: (ctx) => ctx.withValues({ question: ctx.require('request').question }) },
70
+ { name: 'answer', run: async (ctx) => ({ answer: await ask(ctx.require('question')) }) },
71
+ ]);
72
+
73
+ const result = await pipeline.run(new PipelineContext({ request }));
74
+ ```
75
+
76
+ `PipelineContext` is immutable — `get` / `require` / `withValues` — so a step cannot mutate what an earlier step
77
+ observed, and a failed run's state stays inspectable. Each step runs in its own span; a step that throws aborts
78
+ the run with the original error chained; `StopPipeline` is the supported early exit. A pipeline rejects an empty
79
+ or duplicate-named step list at construction.
80
+
81
+ Composite steps — `branch`, `parallel`, `mapStep`, `reduceStep` — are provided as step builders, with bounded
82
+ nesting (`MAX_STEP_DEPTH`).
7
83
 
8
84
  ## What is in the package
9
85
 
10
- | Area | Module | Notes |
86
+ | Area | Exports | Notes |
11
87
  | --- | --- | --- |
12
- | Types | `types` | `Message`, `ModelSpec`, `PinnedModel`, `TokenUsage`, requests and responses |
13
- | Errors | `errors` | One `AidRuntimeError` base so an application catches framework refusals in one place |
14
- | Cost | `cost` | `Budget`, `CostLedger`, spend summaries |
15
- | Tracing | `tracing` | OpenTelemetry GenAI semantic-convention span attributes |
16
- | Prompts | `prompts` | Versioned registry with ordered version resolution |
17
- | Tools | `tools` | Schema-validated dispatch with a documented JSON-schema subset |
18
- | Pipelines | `pipeline` | Sequential reducer over an immutable context |
19
- | Structured output | `structured` | Parse and repair loop over zod schemas |
20
- | Gateway | `gateway` | Pinned aliases, capability checks, fallback chains, budget enforcement |
21
- | Providers | `providers/*` | Deterministic fake and OpenAI-compatible HTTP adapter |
22
- | Evals | `evals/*` | Datasets, metrics, N-run sampling and the statistical gate |
23
- | Retrieval | `retrieval/*` | Chunkers, embedders, stores, hybrid search, reranking, citations, gate |
24
-
25
- ## Deliberate parity notes
26
-
27
- The eval gate reproduces the Python runtime's arithmetic exactly: the same
28
- z-critical table, the same baseline comparison and the same N-run sampling
29
- semantics. A gate whose statistics differed per language would let a TypeScript
30
- regression pass a Python baseline.
88
+ | Types | `Message`, `ModelSpec`, `PinnedModel`, `TokenUsage`, `CompletionRequest`, `CompletionResponse`, `Capability` | The vocabulary everything else is typed against |
89
+ | Errors | `AidRuntimeError` and 25 subclasses | One base, so an app catches framework refusals in one place |
90
+ | Gateway | `ModelGateway`, `estimateUsage` | Pinned aliases, capability negotiation, ordered fallback, budget preflight |
91
+ | Providers | `OpenAICompatibleProvider`, `FakeProvider`, `supports` | One HTTP adapter, one deterministic fake, plus the `Provider` protocol |
92
+ | Cost | `Budget`, `CostLedger`, `estimateCostUsd` | Pre-call enforcement, per-call ledger entries, spend summaries |
93
+ | Tracing | `getTracer`, `llmSpan`, `recordUsage`, `recordCost`, `recordException` | OTel GenAI semantic conventions; AID values namespaced `aid.*` |
94
+ | Prompts | `PromptRegistry`, `PromptTemplate`, `Example` | Versioned registry, typed rendering, few-shot assembly |
95
+ | Tools | `Tool`, `ToolExecutor`, `validateArguments` | Schema-validated dispatch over a documented JSON-Schema subset |
96
+ | Structured output | `buildAgent`, `parseStructured`, `runStructured`, `repairPrompt` | Parse/repair loop over zod schemas |
97
+ | Pipelines | `Pipeline`, `PipelineContext`, `StopPipeline`, `branch`, `parallel`, `mapStep`, `reduceStep` | Sequential reducer over an immutable context |
98
+ | Concurrency | `ConcurrencyLimiter`, `holdsSlot` | Bounded in-flight work, including inside composite steps |
99
+ | Policy | `evaluatePolicy`, `Principal`, `POLICY_BUILTINS` | Tool-auth expressions, fail-closed |
100
+ | Agent loop | `agents/*` | Native loop: transfer vs. delegate, budgets, degrade, memory, HITL checkpoint |
101
+ | Evals | `Dataset`, `EvalCase`, `runEvaluation`, `BaselineGate`, `EvalBaseline` | N-run sampling and the statistical gate |
102
+ | Metrics | `ExactMatch`, `NormalizedContains`, `JsonSchemaValid`, `LlmJudge` | Plus `defaultMetrics` and `metricNames` |
103
+ | Retrieval | `Retriever`, `RetrievalGate`, `chunkDocument`, `openConnector`, `createStore`, `fuse` | Connectors, chunkers, embedders, stores, hybrid search, reranking, citations |
31
104
 
32
- Two things are intentionally not byte-identical, and both are documented at the
33
- point of difference:
105
+ ## Model pinning and capability checks
34
106
 
35
- - `providers/fake.ts` derives its accuracy draw from a SHA-256 hash of
36
- `seed:index:prompt` rather than CPython's Mersenne Twister, because the Mersenne
37
- Twister's string seeding is a CPython implementation detail with no portable
38
- equivalent. The draw is still deterministic and still varies per call index,
39
- which is the property the gate depends on.
40
- - `tools.ts` takes an explicit JSON-Schema subset instead of inferring a schema
41
- from handler reflection, because JavaScript has no `inspect.signature`.
107
+ `gateway.require(alias, capabilities)` checks **both** sides — the model's declared capabilities and the
108
+ adapter's ability to serialize them — and returns the pinned model. A model that supports tool calling behind an
109
+ adapter that does not serialize tool definitions cannot tool-call, and `CapabilityMismatchError` names what is
110
+ missing rather than failing later at the provider.
42
111
 
43
- ## Determinism
112
+ `generate(alias, request, { fallbacks })` tries the alias, then each fallback **in the order given**. Fallbacks
113
+ are ordered, not conditional, and `result.attempts` reports what was actually tried.
44
114
 
45
- `FakeDeterministicEmbedder` hashes with SHAKE-128 and `chunkIdFor` hashes with
46
- SHA-256, so re-ingesting a corpus reproduces byte-identical ids and vectors.
115
+ ## Evals gate on statistics, not equality
47
116
 
48
- ## Build
117
+ A model is *expected* to vary between runs, so a metric comparing two runs for equality is either always red or
118
+ measuring nothing. Non-determinism is the property the gate is built to tolerate:
49
119
 
50
- ```sh
51
- corepack pnpm install
52
- corepack pnpm build
120
+ ```ts
121
+ import { BaselineGate, runEvaluation, ExactMatch, defaultMetrics } from '@aid-framework/runtime';
122
+
123
+ const run = await runEvaluation(dataset, runner, defaultMetrics(), { runs: 5 });
124
+ const verdict = new BaselineGate(thresholds).evaluate(run, baseline);
125
+
126
+ verdict.passed;
127
+ verdict.metrics; // per-metric z-score, allowance and outcome
53
128
  ```
129
+
130
+ `runEvaluation` samples the runner **N times** and reports per-run metric means, so one unlucky run does not
131
+ decide the verdict. `EvalBaseline.fromRun(run)` stores per metric the sample count, mean and spread; the gate
132
+ compares with a one-sided normal test. A metric passes by clearing `minMean`, by staying within `maxRegression`
133
+ of the baseline, or by regressing only within sampling noise.
134
+
135
+ `LlmJudge` is implemented and never CI-required: a gate that depends on one non-deterministic call inside the
136
+ gate you are trying to trust is the problem the design avoids.
137
+
138
+ The arithmetic matches the Python runtime exactly — the same z-critical table, the same baseline comparison, the
139
+ same sampling semantics — because a gate whose statistics differed per language would let a TypeScript
140
+ regression pass a Python baseline.
141
+
142
+ ## Retrieval
143
+
144
+ `retrieval/*` is IR-driven: `bindingFromSpec(retriever, { embedding, store })` folds the IR's `retrievers`,
145
+ `embeddings` and `stores` entries into one `RetrieverBinding`, which a `Retriever` then ingests and searches.
146
+
147
+ Chunk ids are content hashes, so re-ingesting the same input produces the same ids and a no-op diff.
148
+ `FakeDeterministicEmbedder` hashes with SHAKE-128 and `chunkIdFor` with SHA-256, which is what lets a retrieval
149
+ gate run in CI with no network and no key.
150
+
151
+ `RetrievalMetric` (`RecallAtK`, `MeanReciprocalRank`, `CitationGrounding`) is a separate protocol from `Metric`,
152
+ because retrieval scores a *retrieved set* rather than a single string. Retrieval is sampled through
153
+ `RetrievalGate.sample(...)` into a standard `EvalRun` and judged by the unchanged `BaselineGate`.
154
+
155
+ ## Limits, stated plainly
156
+
157
+ - **Streaming is declared, not implemented.** `Capability` includes it so a spec can require it and a build-time
158
+ gate can check it; calling `ModelGateway.stream` raises `FeatureDeferredError`. The transport is
159
+ request/response only.
160
+ - **One HTTP provider and one fake.** Breadth of vendor coverage is a later concern; a narrow, correct seam is
161
+ worth more than a wide shallow one.
162
+ - `PgVectorStore` needs a live `vector` extension and is not exercised by CI; the in-memory store carries the
163
+ store tests, and both share the same `VectorStore` protocol and `similarity` semantics.
164
+
165
+ ## API
166
+
167
+ The full surface is the package's named exports; the table above is the map. The one deliberate asymmetry with
168
+ the Python runtime is `providers/fake.ts`, which derives its accuracy draw from a SHA-256 hash of
169
+ `seed:index:prompt` rather than CPython's Mersenne Twister, because string seeding there is a CPython
170
+ implementation detail with no portable equivalent. The draw is still deterministic and still varies per call
171
+ index, which is the property the gate depends on.
172
+
173
+ ## Related packages
174
+
175
+ - [`aid-runtime-python`](https://pypi.org/project/aid-runtime-python/) — the same runtime for Python
176
+ - [`@aid-framework/generator-ts-vercel`](https://www.npmjs.com/package/@aid-framework/generator-ts-vercel) — the
177
+ generator that emits apps against this runtime
178
+ - [`@aid-framework/cli`](https://www.npmjs.com/package/@aid-framework/cli) — `aid`
179
+
180
+ ## License
181
+
182
+ [Apache-2.0](https://github.com/aid-framework/aid-framework/blob/main/LICENSE). Code generated by AID Framework is
183
+ not subject to this license.