@niscorp/vex 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/DESIGN.md +881 -0
  2. package/DOCS.md +1295 -0
  3. package/LICENSE +202 -0
  4. package/README.md +97 -0
  5. package/dist/adapters/express/index.cjs +1406 -0
  6. package/dist/adapters/express/index.cjs.map +1 -0
  7. package/dist/adapters/express/index.d.cts +25 -0
  8. package/dist/adapters/express/index.d.ts +25 -0
  9. package/dist/adapters/express/index.js +60 -0
  10. package/dist/adapters/express/index.js.map +1 -0
  11. package/dist/adapters/hono/index.cjs +1426 -0
  12. package/dist/adapters/hono/index.cjs.map +1 -0
  13. package/dist/adapters/hono/index.d.cts +25 -0
  14. package/dist/adapters/hono/index.d.ts +25 -0
  15. package/dist/adapters/hono/index.js +78 -0
  16. package/dist/adapters/hono/index.js.map +1 -0
  17. package/dist/adapters/pglite/index.cjs +17 -0
  18. package/dist/adapters/pglite/index.cjs.map +1 -0
  19. package/dist/adapters/pglite/index.d.cts +21 -0
  20. package/dist/adapters/pglite/index.d.ts +21 -0
  21. package/dist/adapters/pglite/index.js +14 -0
  22. package/dist/adapters/pglite/index.js.map +1 -0
  23. package/dist/agent/index.cjs +478 -0
  24. package/dist/agent/index.cjs.map +1 -0
  25. package/dist/agent/index.d.cts +29 -0
  26. package/dist/agent/index.d.ts +29 -0
  27. package/dist/agent/index.js +313 -0
  28. package/dist/agent/index.js.map +1 -0
  29. package/dist/chunk-FPKW4TRX.js +1528 -0
  30. package/dist/chunk-FPKW4TRX.js.map +1 -0
  31. package/dist/chunk-HF25Q53I.js +165 -0
  32. package/dist/chunk-HF25Q53I.js.map +1 -0
  33. package/dist/handler-Cql8SJ3F.d.cts +133 -0
  34. package/dist/handler-mICf9ins.d.ts +133 -0
  35. package/dist/index.cjs +4078 -0
  36. package/dist/index.cjs.map +1 -0
  37. package/dist/index.d.cts +199 -0
  38. package/dist/index.d.ts +199 -0
  39. package/dist/index.js +2340 -0
  40. package/dist/index.js.map +1 -0
  41. package/dist/introspect-D3OaXHBc.d.cts +16 -0
  42. package/dist/introspect-D3OaXHBc.d.ts +16 -0
  43. package/dist/types-BRvMB6IM.d.cts +1491 -0
  44. package/dist/types-BRvMB6IM.d.ts +1491 -0
  45. package/package.json +125 -0
package/DOCS.md ADDED
@@ -0,0 +1,1295 @@
1
+ # Vex Documentation
2
+
3
+ In-depth usage guide for `@niscorp/vex`. For the high-level pitch see
4
+ [README.md](./README.md); for architecture and rationale see
5
+ [DESIGN.md](./DESIGN.md).
6
+
7
+ ## Contents
8
+
9
+ - [Concepts](#concepts)
10
+ - [Installation](#installation)
11
+ - [Creating an engine](#creating-an-engine)
12
+ - [Configuration reference](#configuration-reference)
13
+ - [The request/response contract](#the-requestresponse-contract)
14
+ - [Engine methods](#engine-methods)
15
+ - [The DSL](#the-dsl)
16
+ - [Scope policies](#scope-policies)
17
+ - [Grants and behaviors](#grants-and-behaviors)
18
+ - [Mutations](#mutations)
19
+ - [Caching](#caching)
20
+ - [Wiring the LLM agents](#wiring-the-llm-agents)
21
+ - [Framework adapters](#framework-adapters)
22
+ - [Events](#events)
23
+ - [Error handling](#error-handling)
24
+ - [Local development](#local-development)
25
+
26
+ ---
27
+
28
+ ## Concepts
29
+
30
+ **Request → shape, not query.** A caller sends an `intent` (optional natural
31
+ language), a `shape` (an example of the structure they want back), and a
32
+ `context` object (runtime values). They do not write SQL or even the DSL.
33
+
34
+ **The DSL is the intermediate.** Either you hand-write a DSL `Query` and call
35
+ `compile`/`test`/`execute`, or an LLM hook generates one from the request. The
36
+ DSL is validated by Zod before anything compiles it.
37
+
38
+ **Deterministic core.** Everything except DSL generation and result mapping is
39
+ pure and LLM-free: discover entities → apply scope → resolve to columns/joins →
40
+ analyze → compile to parameterized SQL → bind → execute. You can run this whole
41
+ path with no model and no network beyond the database.
42
+
43
+ **Fingerprint caching.** Every generation is cached under a **fingerprint** —
44
+ server-minted (`fp_…`, returned in `meta.cache.fingerprint`) or caller-chosen
45
+ (a name like `deals/list`). Replaying a fingerprint skips generation entirely
46
+ and re-runs only the deterministic path — so scope and context stay current
47
+ while LLM cost drops to zero. A minted fingerprint is an immutable pin; a
48
+ named one is a mutable slot whose content follows your request. Design and
49
+ rationale: [DESIGN.md](./DESIGN.md) **Cache v2 — fingerprints**.
50
+
51
+ **Scope is separate from context.** `context` is caller data, visible to the
52
+ agent and referenced via `{ $context: "key" }`. `scope` is server-side access
53
+ control, invisible to the agent, injected as filters via `{ $scope: "key" }`.
54
+ They are passed through different channels and must never be conflated.
55
+
56
+ **Writes are replay-only, same wire.** A mutation is a dev-authored cache
57
+ entry (`kind: 'mutation'`) invoked exactly like a read replay —
58
+ `{ fingerprint, context }` — and the entry's kind picks the pipeline. The
59
+ def never travels and no generation path exists for writes. See
60
+ [Mutations](#mutations).
61
+
62
+ **Reserved sort keys.** `sortBy` (an `entity.field`) and `sortDir` (`asc`/`desc`)
63
+ are reserved keys in `context`: the engine reads them straight into the `ORDER BY`
64
+ (the column is schema-validated) and they are never bound as parameters. The
65
+ caller's column **leads**; the query's own `sort` keys stay behind it as
66
+ tiebreakers, minus a duplicate of the chosen column. That matters for a keyset
67
+ page: an entry sorted `(name, id)` keeps `id` behind whatever the caller asks
68
+ for, so the order stays total and no row can straddle a page boundary
69
+ unreachably. An empty `sortBy` leaves the authored order alone.
70
+
71
+ **Optional conditions.** `{ optional: { key, then } }` lets a caller switch a
72
+ condition ON by supplying its key — omit it and the condition is removed from
73
+ the query before it compiles. It is *not there*, as opposed to there and
74
+ matching everything, so one entry answers "everyone", "everyone matching a
75
+ search", and "one by id" without sentinel values like `'%'`. Absent means
76
+ missing, `undefined` or `null` (`''` is a value); a condition reading several
77
+ keys names them all, and the direction of the change follows position — under
78
+ `and` supplying the key narrows, under `or` it widens. Reads only: the seed lint
79
+ refuses one inside a mutation, and the correlation inside an `exists` must hold
80
+ unconditionally. `meta.context` marks such keys `optional: true`, plus
81
+ `absent: true` on a run that did not supply them, so a caller can learn what a
82
+ fingerprint accepts without failing a request first.
83
+
84
+ ---
85
+
86
+ ## Installation
87
+
88
+ ```bash
89
+ pnpm add @niscorp/vex zod
90
+ ```
91
+
92
+ `zod` is the only required peer. Add the others for the paths you use:
93
+
94
+ ```bash
95
+ pnpm add pg # Postgres adapter + Postgres cache
96
+ pnpm add hono # or: express # framework adapter
97
+ pnpm add @niscorp/cortex @niscorp/signal @niscorp/prism # reference LLM agents
98
+ ```
99
+
100
+ For semantic (vector) search you need a Postgres with `pgvector` (the dev setup
101
+ uses the `pgvector/pgvector:pg16` image) and an embedding function wired onto the
102
+ engine — see [Semantic search](#semantic-search).
103
+
104
+ ---
105
+
106
+ ## Creating an engine
107
+
108
+ ### Minimal (deterministic, no LLM)
109
+
110
+ ```typescript
111
+ import pg from 'pg';
112
+ import { createQueryEngine, createPostgresAdapter } from '@niscorp/vex';
113
+
114
+ const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
115
+ const adapter = createPostgresAdapter({ pool });
116
+
117
+ const engine = createQueryEngine({ adapter });
118
+
119
+ await engine.introspect(); // REQUIRED before compile/test/execute
120
+ ```
121
+
122
+ `introspect()` discovers tables, columns, normalized types, relations, and
123
+ indexes, and caches the result plus a schema fingerprint. The engine throws
124
+ `execution_error` ("schema not loaded") if you call `compile`/`execute`/`test`
125
+ before introspecting.
126
+
127
+ With no `generateDsl` hook, `execute` serves only requests whose shape is
128
+ already cached (and otherwise throws `agent_failed`). `compile` and `test` work
129
+ fully because you pass them a DSL directly.
130
+
131
+ ### With LLM-backed generation and mapping
132
+
133
+ ```typescript
134
+ const engine = createQueryEngine({
135
+ adapter,
136
+ cache, // optional; defaults to in-memory
137
+ scope: scopePolicy, // optional
138
+ onEvent: handler, // optional
139
+ generateDsl, // natural language → DSL (see "Wiring the LLM agents")
140
+ mapToShape, // raw rows → requested shape
141
+ });
142
+ await engine.introspect();
143
+ ```
144
+
145
+ ---
146
+
147
+ ## Configuration reference
148
+
149
+ ### `QueryEngineConfig`
150
+
151
+ ```typescript
152
+ type QueryEngineConfig = {
153
+ adapter: DatabaseAdapter; // required
154
+ scope?: ScopePolicy; // server-side access control
155
+ cache?: CacheBackend; // default: createMemoryCache()
156
+ onEvent?: VexEventHandler; // pipeline event stream
157
+ generateDsl?: (request, schema) => Promise<Query>;
158
+ mapToShape?: (rows, shape) => Promise<{ ir: CompiledIr; transformed: JsonValue }>;
159
+ embed?: (text, dimensions?) => Promise<number[]>; // text→vector for semantic filters
160
+ config?: {
161
+ maxNestingDepth?: number; // default: 2
162
+ defaultLimit?: number; // default: 100
163
+ maxLimit?: number; // default: 1000
164
+ rejectCartesianProducts?: boolean; // default: true
165
+ warnUnindexedFilters?: boolean; // default: true
166
+ rejectUnindexedFilters?: boolean; // default: false
167
+ entities?: string[]; // whitelist for introspection
168
+ unsatisfiableTtlMs?: number; // negative-cache TTL, default 300_000 (5 min)
169
+ };
170
+ };
171
+ ```
172
+
173
+ - `config.entities`, when set, restricts `introspect()` to those tables.
174
+ - `defaultLimit` is applied when a DSL omits `limit`; `maxLimit` clamps any
175
+ larger value.
176
+ - **`embed`** turns text into a vector for `semantic` filters, at parameter-
177
+ binding time. It's an injected provider (wire it to Signal) — *not* a database
178
+ concern, so it lives on the engine, not the adapter (the adapter only emits the
179
+ vector-distance SQL). Required only if your schema has vector columns you
180
+ search over.
181
+
182
+ ### `ExecuteOptions`
183
+
184
+ ```typescript
185
+ type ExecuteOptions = {
186
+ scope?: ScopeValues; // { [key]: value } for $scope references
187
+ entities?: string[]; // restrict generation to this entity subset for this call
188
+ locked?: boolean; // replay-only: unknown/changed fingerprints throw `locked` instead of generating
189
+ };
190
+ ```
191
+
192
+ There is no cache mode — the request body itself expresses
193
+ replay/generate/replace semantics via its `fingerprint` field (see the
194
+ request contract below).
195
+
196
+ An engine with a policy (configured, or `options.scopePolicy` per request)
197
+ enforces it on every run. `options.scope` supplies the values its row rules
198
+ compare against; it never decides whether the policy applies. An engine with no
199
+ policy is open and needs no values; a policy with no row rules needs none
200
+ either. A statement whose `$scope` slot the caller left empty is refused with
201
+ `missing_scope` — it is never read as "everybody".
202
+
203
+ ---
204
+
205
+ ## The request/response contract
206
+
207
+ ### Request
208
+
209
+ ```typescript
210
+ type QueryRequest = {
211
+ fingerprint?: string; // the cache identity (see postures below)
212
+ intent?: string; // natural language (drives the agent)
213
+ shape?: unknown; // example of the desired output structure
214
+ context?: Record<string, unknown>; // runtime values for { $context } refs; reserved `sortBy`/`sortDir` drive ORDER BY (default {})
215
+ };
216
+ ```
217
+
218
+ The request is Zod-validated; `execute` throws `invalid_request` on failure.
219
+ It must carry a `fingerprint`, a `shape`, or both — the three postures:
220
+
221
+ 1. **Explore** — `{ intent, shape, context }`: generate, cache, mint a
222
+ fingerprint (`fp_…`, returned in `meta.cache.fingerprint`).
223
+ 2. **Replay** — `{ fingerprint, context }`: serve the stored entry; never
224
+ generates. Unknown fingerprint → `cache_miss`. Context values may vary per
225
+ call (they are runtime data, not identity).
226
+ 3. **Own a slot** — `{ fingerprint, intent, shape, context }`: hit when the
227
+ stored request matches (intent + normalized shape + context key names),
228
+ regenerate and REPLACE the slot when it differs
229
+ (`meta.cache.replaced: true`) — unless the entry is `protected` → throws
230
+ `fingerprint_protected`.
231
+
232
+ The **shape** is an example, not a schema: use empty strings, zeros, and
233
+ booleans as type markers. `[{ id: '', total: 0 }]` means "an array of objects
234
+ with a string `id` and a number `total`." A bare object means a single record.
235
+
236
+ The shape also picks the result envelope: an **array** shape maps over the whole
237
+ row set (the mapping sees `$.result` as the rows array — a `$map`/identity); a
238
+ **non-array** shape maps the single (first) row (the mapping sees `$.result` as
239
+ that row, so a detail reads `$.result.field`). So the mapping — not Vex — owns
240
+ the output: array, single object, or scalar. On a fingerprint-only replay the
241
+ entry's STORED shape drives this — the caller sends none.
242
+
243
+ Two more roots sit beside `$.result` for the mapping to read:
244
+
245
+ | root | what it holds |
246
+ |---|---|
247
+ | `$.result` | the rows (or the single row) — what every mapping is written against |
248
+ | `$.context` | the request's own context values |
249
+ | `$.scope` | the engine-side scope values (`studioId`, `today`, `locale`, …) |
250
+
251
+ `$.scope` is what lets a mapping produce words in the **reader's** language:
252
+ a mapping is where a row becomes `"Active"`, `"Fri 14 Mar"`, `"€45"`, and words
253
+ have a language. Reading it here widens nothing a caller can reach — scope
254
+ values are injected by the host per session and are unauthorable by a request.
255
+
256
+ None of this touches the cache. Vex caches the query **plan** (`dsl` +
257
+ `prismIr`), never rows, and a compiled mapping holds the *lookup*, not the
258
+ looked-up value — so a locale-parameterised mapping shares one cache entry
259
+ across every language.
260
+
261
+ ### Response
262
+
263
+ ```typescript
264
+ type QueryResponse = {
265
+ result: JsonValue; // the mapping's output (array / object / scalar); raw Row[] if no mapping
266
+ meta: {
267
+ cache: {
268
+ hit: boolean;
269
+ fingerprint?: string; // the entry's identity — minted or your name
270
+ replaced?: boolean; // true when a named slot was regenerated + replaced
271
+ intent?: string; // the stored intent (descriptive only)
272
+ };
273
+ context: Record<string, { // the resolved parameter contract
274
+ type?: 'string' | 'number' | 'boolean' | 'string[]' | 'number[]';
275
+ kind: 'context' | 'scope' | 'semantic';
276
+ optional?: true; // supplying it switches a condition on
277
+ absent?: true; // optional, and not supplied this run (so: no type)
278
+ }>;
279
+ timing?: { agentMs?: number; executionMs: number; mappingMs?: number };
280
+ warnings?: string[]; // analyzer warnings, if any
281
+ missingContext?: string[]; // present when REQUIRED keys were not supplied;
282
+ // optional keys never appear here
283
+ };
284
+ };
285
+ ```
286
+
287
+ If the compiled query needs context/scope values the caller did not provide,
288
+ `execute` returns a **valid** response with an empty `result` and
289
+ `meta.missingContext` listing the missing keys — it does not throw. Retry with
290
+ the values filled in. `meta.context` always describes the full parameter
291
+ contract so a caller knows what to provide.
292
+
293
+ ---
294
+
295
+ ## Engine methods
296
+
297
+ ```typescript
298
+ type QueryEngine = {
299
+ introspect: () => Promise<DatabaseSchema>;
300
+ execute: (request: QueryRequest, options?: ExecuteOptions) => Promise<QueryResponse>;
301
+ compile: (dsl: Query, scope?: ScopeValues) => CompiledQuery;
302
+ test: (dsl: Query, scope?: ScopeValues) => Promise<TestResult>;
303
+ getDslSchema: () => object;
304
+ getSchema: () => DatabaseSchema | undefined;
305
+ cache: CacheBackend;
306
+ };
307
+ ```
308
+
309
+ - **`introspect()`** — discover and cache the schema + fingerprint. Call once at
310
+ startup (and again after a schema migration).
311
+ - **`execute(request, options)`** — the full pipeline described in DESIGN.md.
312
+ - **`compile(dsl)`** — run the deterministic pipeline (clamp limit →
313
+ discover → scope → resolve → analyze → adapter compile) and return the
314
+ `CompiledQuery` (`{ sql, paramSlots, contextContract }`). No execution, no
315
+ LLM. Throws `invalid_dsl` if the analyzer finds errors.
316
+ - **`test(dsl, scope?)`** — compile with `limit` forced to 5, build synthetic
317
+ context values, execute, and return `{ rows, warnings, errors }`. Errors are
318
+ caught and returned in `errors` rather than thrown. Useful for validating a
319
+ hand-written DSL against real data.
320
+ - **`getDslSchema()`** — the DSL as a JSON Schema (draft-7). This is what you
321
+ feed an LLM so it learns the DSL.
322
+ - **`getSchema()`** — the last introspected `DatabaseSchema`, or `undefined`.
323
+ - **`cache`** — the live `CacheBackend` for inspection or manual eviction.
324
+
325
+ ---
326
+
327
+ ## The DSL
328
+
329
+ A `Query` is a constrained JSON object. The model never produces SQL — it fills
330
+ this in, and the schema's `.describe()` annotations teach it how.
331
+
332
+ ```typescript
333
+ type Query = {
334
+ from: Source[]; // min 1
335
+ fields?: FieldRef[]; // optional with `aggregate`; "entity.field" or { field, as }
336
+ filter?: Filter;
337
+ compute?: Record<string, ComputeExpression>; // alias → expression
338
+ aggregate?: Record<string, AggregateExpression>; // alias → function
339
+ groupBy?: string[];
340
+ sort?: { field: string; dir?: 'asc' | 'desc' }[];
341
+ limit?: number;
342
+ distinct?: boolean;
343
+ };
344
+
345
+ type Source = string | { as: string; query: Query }; // entity name or subquery
346
+ type FieldRef = string | { field: string; as: string }; // bare column, or aliased output key
347
+ ```
348
+
349
+ ### Fields
350
+
351
+ Each entry is `entity.field`, or `{ field, as }` to set a distinct output key
352
+ (so three joined `name` columns can become `company` / `stage` / `owner`
353
+ without a compute). There is no `SELECT *`. `fields` is **optional** — omit it
354
+ for an aggregate-only query (e.g. a bare `COUNT(*)`); a query must have at least
355
+ one of `fields` / `aggregate`. Every entity referenced anywhere must also appear
356
+ in `from`.
357
+
358
+ ```json
359
+ { "from": ["order", "customer"],
360
+ "fields": ["order.id", "order.total", { "field": "customer.name", "as": "customer" }] }
361
+ ```
362
+
363
+ ### Filters
364
+
365
+ ```typescript
366
+ // Comparison — both sides are field paths, literals, or { $context }/{ $scope }
367
+ { eq: [a, b] } { neq: [a, b] }
368
+ { gt: [a, b] } { gte: [a, b] }
369
+ { lt: [a, b] } { lte: [a, b] }
370
+
371
+ // Collection — first arg is a field path; second is an array or a ref
372
+ { in: ["entity.field", [v1, v2]] }
373
+ { notIn: ["entity.field", { $context: "ids" }] }
374
+
375
+ // String pattern (% wildcard)
376
+ { like: ["entity.field", "%abc%"] } // case-sensitive
377
+ { ilike: ["entity.field", "%abc%"] } // case-insensitive
378
+
379
+ // Null
380
+ { isNull: "entity.field" }
381
+ { isNotNull: "entity.field" }
382
+
383
+ // Logic ('and'/'or' require ≥ 2 conditions)
384
+ { and: [f1, f2, ...] }
385
+ { or: [f1, f2, ...] }
386
+ { not: f }
387
+
388
+ // OPTIONAL — a condition the CALLER turns on, by supplying its key(s).
389
+ // Absent, the condition is removed before the query compiles: it is not
390
+ // there, rather than there and matching everything.
391
+ { optional: { key: "q", then: { ilike: ["people.name", { $context: "q" }] } } }
392
+ { optional: { key: ["after", "afterId"], then: SEEK } } // needs both, so it names both
393
+
394
+ // EXISTS — is there a row over there that points back at this one?
395
+ // The subquery is the same { from, filter } a query is, and NOTHING else:
396
+ // no fields, no sort, no limit. EXISTS asks whether, not what.
397
+ // The correlation is an ordinary field-to-field comparison against the
398
+ // OUTER entity — no special syntax, because a dotted string is already a
399
+ // field path.
400
+ { exists: { from: ["task"], filter: { eq: ["task.issue_id", "issue.id"] } } }
401
+
402
+ // NOT EXISTS is `not` around it — "issues nobody has been sent to":
403
+ { not: { exists: { from: ["task"], filter: { eq: ["task.issue_id", "issue.id"] } } } }
404
+
405
+ // Prefer this over fetching a list of ids and passing them back in a
406
+ // `notIn`. Two round trips do not scale, and the set can change between them.
407
+
408
+ // Semantic vector similarity (query MUST be a $context/$scope ref)
409
+ { semantic: { field: "product.embedding", query: { $context: "search" }, minScore?: 0.7 } }
410
+
411
+ // Fuzzy string match (Levenshtein)
412
+ { fuzzy: { field: "customer.name", query: { $context: "q" }, maxDistance?: 2 } }
413
+ ```
414
+
415
+ There is intentionally **no** `offset` (use a cursor: `{ gt: ["entity.id",
416
+ { $context: "cursor" }] }`) and **no** `between` (use `{ and: [{ gte }, { lte }] }`).
417
+
418
+ Wrap the cursor in an `optional` so the first page is one that sends no cursor
419
+ at all rather than one that sends a sentinel — and name **every** key the seek
420
+ reads, because a surviving clause with a missing key returns nothing, which on a
421
+ paging loop reads as the end of the list.
422
+
423
+ ### Values
424
+
425
+ ```typescript
426
+ "order.status" // a field path — any string in "entity.field" form (a dot, not at either end)
427
+ "published" // a literal string — any string without a usable dot
428
+ 42, true, null // literals
429
+ { $context: "userId" } // caller-provided value, from request.context
430
+ { $scope: "accountId" } // server-injected value, from options.scope
431
+ ```
432
+
433
+ The resolver distinguishes a field path from a literal string purely by the
434
+ `entity.field` shape (a dot that is neither the first nor last character). So
435
+ `"order.status"` resolves to a column, while `"published"` is a literal value.
436
+
437
+ ### Compute
438
+
439
+ Record of `alias → expression`. The alias is the output key.
440
+
441
+ ```json
442
+ {
443
+ "compute": {
444
+ "total": { "add": ["order.price", "order.tax"] },
445
+ "displayName": { "coalesce": ["user.nickname", "user.name"] },
446
+ "tier": { "case": {
447
+ "when": [{ "condition": { "gte": ["order.total", 100] }, "then": "premium" }],
448
+ "else": "standard"
449
+ } }
450
+ }
451
+ }
452
+ ```
453
+
454
+ Operators: `add`, `subtract`, `multiply`, `divide` (binary), `concat`,
455
+ `coalesce` (≥ 2 args), `case` (`when[]` of `{ condition, then }` + `else`).
456
+
457
+ ### Aggregates
458
+
459
+ Record of `alias → function`. Pair with `groupBy` for grouped results; without
460
+ it the analyzer warns and you get a single aggregated row. Omit `fields` for an
461
+ aggregate-only query (a bare count):
462
+
463
+ ```json
464
+ { "from": ["order"], "aggregate": { "orders": { "count": "*" } } }
465
+ ```
466
+
467
+ `sum` / `avg` / `min` / `max` take a field path **or a compute expression**, so
468
+ you can aggregate a derived value (no extra column needed):
469
+
470
+ ```json
471
+ {
472
+ "from": ["order", "stage"],
473
+ "aggregate": { "weighted": { "sum": { "multiply": ["order.value", "stage.win_probability"] } } },
474
+ "filter": { "eq": ["order.status", "open"] }
475
+ }
476
+ ```
477
+
478
+ Grouped, with a renamed field:
479
+
480
+ ```json
481
+ {
482
+ "from": ["order", "customer"],
483
+ "fields": [{ "field": "customer.name", "as": "customer" }],
484
+ "aggregate": { "revenue": { "sum": "order.total" }, "orders": { "count": "*" } },
485
+ "groupBy": ["customer.name"]
486
+ }
487
+ ```
488
+
489
+ Functions: `count` (field path or `"*"`); `sum` / `avg` / `min` / `max` (a field
490
+ path or a compute expression).
491
+
492
+ ### Subqueries
493
+
494
+ A `from` entry can be `{ as, query }`. Nesting is bounded by
495
+ `config.maxNestingDepth` (default 2). Scope, discovery, and resolution all
496
+ recurse into subqueries. A subquery's output columns are referenceable from the
497
+ outer query as `alias.field` — every selected field (under its `as`, if
498
+ aliased) plus every `compute`/`aggregate` alias.
499
+
500
+ **Joins.** String entity sources are joined automatically by foreign key; a
501
+ source with no FK path to the others is an error. A **nullable** FK compiles to
502
+ a `LEFT JOIN` — a null FK never silently drops the referencing row from the
503
+ read (the joined fields come back null); a non-nullable FK stays an inner
504
+ `JOIN` (equivalent — the column can't be null). Reverse one-to-many joins are
505
+ inner by design (rows multiply per child). Subquery sources are **never**
506
+ FK-joined — any beyond the first are `CROSS JOIN`ed. That makes a row of
507
+ independent aggregates expressible: cross-join N single-row `COUNT(*)`
508
+ subqueries and select each one's count, and you get one row with N counts in a
509
+ single read (use an object shape so Vex maps that single row).
510
+
511
+ ```json
512
+ { "from": [{ "as": "c", "query": { "from": ["contact"], "aggregate": { "n": { "count": "*" } } } },
513
+ { "as": "co", "query": { "from": ["company"], "aggregate": { "n": { "count": "*" } } } }],
514
+ "fields": [{ "field": "c.n", "as": "contacts" }, { "field": "co.n", "as": "companies" }] }
515
+ ```
516
+
517
+ (Limitation: subquery SQL params are embedded verbatim, so at most one bound
518
+ `$context`/`$scope` param across all subqueries is supported today.)
519
+
520
+ ### Semantic search
521
+
522
+ `{ semantic: { field, query } }` needs three things wired up: a `vector` column
523
+ in the schema, an HNSW or IVFFlat index on it (else the analyzer warns it will
524
+ be slow), and an `embed` function on the engine:
525
+
526
+ ```typescript
527
+ import { createSignal } from '@niscorp/signal';
528
+
529
+ const embedder = createSignal('openai').model('text-embedding-3-small');
530
+ const engine = createQueryEngine({
531
+ adapter,
532
+ embed: (text, dimensions) =>
533
+ embedder.embed(text, dimensions ? { dimensions } : undefined),
534
+ });
535
+ ```
536
+
537
+ At execution, the `query` ref's text is read from context, embedded, and bound
538
+ as the vector parameter. (Never select a vector column directly in `fields` — it
539
+ is for `semantic` filters only.)
540
+
541
+ ---
542
+
543
+ ## Scope policies
544
+
545
+ Server-side, LLM-invisible access control, applied to the resolved query/mutation
546
+ *after* it is authored — so a generated or injected DSL can never reference,
547
+ dodge, or forge it. Two roles, named for the SQL they produce:
548
+
549
+ - **`match`** (RLS) — the row's column must equal a scope value. A `WHERE` filter
550
+ on read/update/delete; on insert the column is pinned to that value.
551
+ - **`set`** (identity) — the engine writes the column from a scope value on every
552
+ column-writing mutation: insert values and update set alike (`write` means every
553
+ write; delete writes no columns, so it is structurally exempt).
554
+
555
+ ```typescript
556
+ type ScopePolicy = {
557
+ default: 'allow' | 'deny'; // fallback for unlisted entities / absent phases
558
+ entities: Record<string, ScopeEntityRule>;
559
+ };
560
+
561
+ type ScopeEntityRule =
562
+ | { public: true } // fully open
563
+ | { deny: true } // fully closed
564
+ | {
565
+ read?: ScopeMatch[]; // SELECT WHERE (match only)
566
+ write?: (ScopeMatch | ScopeSet)[]; // UMBRELLA — grants + rules for insert/update/delete
567
+ insert?: (ScopeMatch | ScopeSet)[]; // specific: just INSERT, rules stack on write's
568
+ update?: (ScopeMatch | ScopeSet)[]; // specific: just UPDATE, rules stack on write's
569
+ delete?: ScopeMatch[]; // specific: just DELETE (match only — nothing to set)
570
+ };
571
+
572
+ type ScopeMatch = { match: string; to: string }; // row.<match> = scope[to]
573
+ type ScopeSet = { set: string; to: string }; // INSERT/UPDATE row.<set> := scope[to]
574
+ ```
575
+
576
+ Example:
577
+
578
+ ```typescript
579
+ const scope: ScopePolicy = {
580
+ default: 'deny',
581
+ entities: {
582
+ order: { read: [{ match: 'accountId', to: 'accountId' }],
583
+ write: [{ match: 'accountId', to: 'accountId' }, { set: 'owner_id', to: 'userId' }] },
584
+ country: { public: true },
585
+ secret: { deny: true },
586
+ },
587
+ };
588
+
589
+ const engine = createQueryEngine({ adapter, scope });
590
+ await engine.execute(request, { scope: { accountId: 'acc-99', userId: 'u-1' } });
591
+ ```
592
+
593
+ Behavior:
594
+
595
+ - The query engine enforces the **`read`** rules: each `match` becomes
596
+ `{ eq: ["order.accountId", { $scope: "accountId" }] }`, AND-merged into the DSL.
597
+ - `default` covers a discovered entity with no rule *and* a listed entity with no
598
+ matching phase: `'deny'` throws `VexScopeError` (`scope_denied`), `'allow'`
599
+ passes through unfiltered. `{ public: true }` passes; `{ deny: true }` throws.
600
+ - The write phases are the contract for a mutation executor (the query engine
601
+ is read-only). **`write`** is the umbrella: its presence grants insert, update
602
+ and delete, and its rules apply to all of them. **`insert`**/**`update`**/**`delete`**
603
+ are specific phases: each grants just its op, its rules stacking on the
604
+ umbrella's (`delete` is match-only — nothing to set). Rule mechanics: `match`
605
+ filters update/delete and pins the column on insert; `set` writes the column
606
+ on insert AND update. Applied server-side, never authored in the mutation
607
+ — so identity/ownership can't be forged.
608
+ - A missing `$scope` value throws `missing_scope` — on a read, and on a write,
609
+ where binding it would stamp NULL. A missing `$context` value is different: it
610
+ is the caller's, and a read surfaces it via `meta.missingContext`
611
+ (empty-but-valid).
612
+ - `match` is equality today; an `op` for `in` / multi-valued boundaries is additive
613
+ when a real one appears.
614
+
615
+ The policy applies whenever one is configured; the values fill it. Omitting
616
+ the values means nobody, never everybody.
617
+
618
+ ---
619
+
620
+
621
+ ## Grants and behaviors
622
+
623
+ A `ScopePolicy` written by hand is fine for one app. A policy *layer* — a
624
+ charter, a role system, a config file — wants to hand vex a flat set of strings
625
+ and get vex's native contract back. That is the grant dialect.
626
+
627
+ A grant is `<table>.<verb>` where the verb is one of `SCOPE_VERBS`:
628
+ `read`, `write.insert`, `write.update`, `write.delete`. The strings are opaque to
629
+ the layer that resolves them and native to vex, so neither imports the other.
630
+
631
+ ```typescript
632
+ scopeGrants(['orders', 'users']) // every grant those tables can carry
633
+ createScopePolicy(grants, behaviors, scoping?) // → ScopePolicy
634
+ ```
635
+
636
+ **Grants decide whether a phase exists. Behaviors decide what a granted phase
637
+ does.** Listing a table in `behaviors` grants nothing.
638
+
639
+ ```typescript
640
+ type ScopeRules = { read?: ScopeMatch[]; write?: ScopeRule[]; insert?: ScopeRule[]; update?: ScopeRule[]; delete?: ScopeMatch[] };
641
+
642
+ type ScopeBehaviors = Record<string, ScopeRules | Record<string, ScopeRules>>;
643
+ ```
644
+
645
+ ### Scoping profiles
646
+
647
+ A table carries **either** one rule set — the shape above — **or several named
648
+ ones**, of which `default` is what an unprofiled principal gets.
649
+
650
+ The problem they solve: scoping used to be a property of the *table*, so every
651
+ role holding any grant on it got the same reach. *"The desk reads every booking;
652
+ a member reads their own"* was unsayable, and the workaround was a second table
653
+ carrying the tighter rule — one fact in two places, kept level by a trigger.
654
+
655
+ Reach is a property of the **principal**, so it is chosen once and applies to
656
+ every table they touch:
657
+
658
+ ```typescript
659
+ const behaviors: ScopeBehaviors = {
660
+ bookings: {
661
+ default: { read: [{ match: 'studio_id', to: 'studioId' }] },
662
+ personal: { read: [{ match: 'studio_id', to: 'studioId' },
663
+ { match: 'membership_id', to: 'membershipId' }] },
664
+ },
665
+ // No variant: everybody reads it the same way, and a profiled principal
666
+ // falls back to this without needing an entry per profile.
667
+ class_sessions: { default: { read: [{ match: 'studio_id', to: 'studioId' }] } },
668
+ };
669
+
670
+ createScopePolicy(new Set(['bookings.read']), behaviors) // studio-wide
671
+ createScopePolicy(new Set(['bookings.read']), behaviors, 'personal') // + their own rows
672
+ ```
673
+
674
+ The two shapes are told apart by their keys: a rule set has at least one of
675
+ `read`/`write`/`insert`/`update`/`delete`; a named map has none of them. The plain
676
+ shape is unchanged and still means "one reach for everyone".
677
+
678
+ **It fails closed.** An unknown profile denies *everything* — not just the table
679
+ it could not find. A mistyped profile that quietly meant "the default" would
680
+ widen a principal to every row of every table they hold a grant on, which is the
681
+ one failure mode a policy layer may not have. A table declaring named variants
682
+ and no `default` likewise refuses an unprofiled principal rather than guessing
683
+ that "no rule" is safe.
684
+
685
+ `scopeProfiles(behaviors)` lists every profile name a map declares — what a
686
+ policy layer validates its own names against.
687
+
688
+ ### A principal with several roles
689
+
690
+ Reach belongs to the role, and a principal may hold more than one — somebody who
691
+ teaches at a studio and trains there is staff on the roster and a member in the
692
+ class. No single profile describes them: studio-wide on the read they do as
693
+ staff, their own rows on the write they do as a member.
694
+
695
+ So compile **one policy per role**, each with that role's own profile, and merge:
696
+
697
+ ```typescript
698
+ mergeScopePolicies(
699
+ roles.map((role) => createScopePolicy(grantsFor(role), behaviors, profileFor(role))),
700
+ )
701
+ ```
702
+
703
+ A principal may do anything **any** of their roles permits, which is what makes
704
+ holding two additive rather than a conflict to arbitrate.
705
+
706
+ **Merge rule:** per entity and phase, the *broadest* rule set wins — fewest
707
+ `match` rules, since every match narrows. Profiles are refinements of a default
708
+ (`personal` is the tenant rule plus one more), so "fewest" is "widest" in
709
+ practice. A phase only one role grants keeps that role's rules intact, which is
710
+ what carries a member's identity stamp onto a write no other role grants.
711
+
712
+ ### A read that must stay narrow
713
+
714
+ The merge above widens. For a read whose answer legitimately widens with the
715
+ reader — a roster, a schedule, a members list — that is right. For one that means
716
+ *mine* it is a leak: an instructor who also trains would open "the classes you
717
+ have booked" and get the studio's.
718
+
719
+ So the ENTRY names the reach it must be served at, and the host recompiles the
720
+ same principal under that profile:
721
+
722
+ ```typescript
723
+ { kind: "ok", dsl: { from: ["bookings"], ... }, reach: "personal" }
724
+ ```
725
+
726
+ Grants are unchanged — the union of every role the caller holds — so this can
727
+ only narrow rows, never widen phases. A caller with no verb for the table is
728
+ still refused.
729
+
730
+ The host supplies `policyForReach(reach)` (hono: `getPolicyForReach(c, reach)`),
731
+ because vex holds neither grants nor behaviors. **It fails closed both ways:** an
732
+ entry declaring a reach no host can resolve is a 500, and a host returning
733
+ `undefined` is a 403. Falling back to the caller's own reach would be silent,
734
+ and silence is the whole risk — the read would answer, with too much.
735
+
736
+ ## Mutations
737
+
738
+ Writes are first-class and deliberately narrower than reads: **no generation
739
+ path exists.** A mutation is authored by a developer, seeded into the cache
740
+ as a `kind: 'mutation'` entry under a named fingerprint (normally
741
+ `protected`), and replayed over the same wire shape reads use. An inline
742
+ `{ mutation }` body is not a request shape at all.
743
+
744
+ ### The grammar
745
+
746
+ Five ops, closed and strict. Values are literals, `{ $context: "key" }`, or a
747
+ `{ $lookup }` scalar subquery (below) — **never `$scope`** (identity is
748
+ engine-injected, unforgeable) and never a field path. `update`/`delete`
749
+ REQUIRE a `where` (a real vex `Filter`).
750
+
751
+ ```ts
752
+ { op: 'insert', table: 'tasks', values: { title: { $context: 'title' } } }
753
+ { op: 'update', table: 'tasks', set: { done: { $context: 'done' } },
754
+ where: { eq: ['tasks.id', { $context: 'id' }] } }
755
+ { op: 'delete', table: 'tasks', where: { eq: ['tasks.id', { $context: 'id' }] } }
756
+ // one INSERT per element of a caller-sized array (see insertEach below)
757
+ { op: 'insertEach', table: 'slots', items: { $context: 'days' },
758
+ values: { weekday: { $item: 'weekday' }, name: { $context: 'name' } } }
759
+ // sugar — insert-or-update keyed on `key`; `insert` columns apply on create only
760
+ { op: 'upsert', table: 'tasks', key: 'id',
761
+ columns: { title: { $context: 'title' } },
762
+ insert: { deal_id: { $context: 'deal_id' } } }
763
+ ```
764
+
765
+ An array of ops is a batch and runs in ONE transaction (the client must
766
+ expose `transaction`; a batch on a non-transactional client is refused).
767
+
768
+ ### `onConflict` — the database arbitrates
769
+
770
+ `insert` and `insertEach` take an optional `onConflict`. Where the `upsert`
771
+ sugar branches on what the CALLER sent (key present → update that row),
772
+ `onConflict` branches on what the DATABASE holds — atomically, in one
773
+ statement, the answer to every "look up, then insert if absent" race:
774
+
775
+ ```ts
776
+ // create-or-fetch: the no-op "touch" keeps RETURNING * yielding the row on
777
+ // BOTH paths — this is the idiom for "ensure this person exists, give me their id"
778
+ { op: 'insert', table: 'people',
779
+ values: { email: { $context: 'email' }, name: { $context: 'name' } },
780
+ onConflict: { target: ['email'], set: { email: { $context: 'email' } } } }
781
+
782
+ // insert-if-absent: DO NOTHING returns NO row on conflict — callers read the
783
+ // absence as "already existed" (a batch reply's row count tells the story)
784
+ { op: 'insert', table: 'memberships',
785
+ values: { person_id: { $context: 'personId' } },
786
+ onConflict: { target: ['studio_id', 'person_id'] } }
787
+ ```
788
+
789
+ Two guarantees, both enforced before any SQL runs:
790
+
791
+ - **`target` must name a real unique constraint.** It is validated against
792
+ the introspected schema (primary key + unique indexes, order-insensitive);
793
+ an ON CONFLICT that arrests nothing is an authoring error with the actual
794
+ unique column sets in the message — never a runtime surprise.
795
+ - **DO UPDATE is an update and is governed like one.** Declaring `set`
796
+ requires the update (or `write` umbrella) phase — an insert-only grant
797
+ refuses it. Update-phase `set` rules stamp identity into the DO UPDATE SET;
798
+ `match` rules whose column the conflict target already pins are inherently
799
+ satisfied (the inserted, scope-pinned value must equal the existing row's
800
+ for a conflict to arise), and any other `match` compiles to a WHERE on the
801
+ DO UPDATE half — the RLS boundary, again. `mutationEffect` declares the
802
+ entry as the insert-plus-update it really is, so discovery visibility
803
+ follows the same rule.
804
+
805
+ ### `$lookup` — a value read from another table
806
+
807
+ Any value position (insert `values`, update `set`, `onConflict.set`) may be a
808
+ scalar subquery — how a statement references a row it cannot otherwise name:
809
+
810
+ ```ts
811
+ { op: 'insert', table: 'memberships',
812
+ values: {
813
+ person_id: { $lookup: { from: 'people', field: 'id',
814
+ where: { eq: ['people.email', { $context: 'email' }] } } },
815
+ } }
816
+ // → INSERT INTO memberships (person_id, …) VALUES ((SELECT id FROM people WHERE …), …)
817
+ ```
818
+
819
+ The lookup READS, so the read-phase scope rules of `from` are ANDed into its
820
+ WHERE by the engine — a mutation entry is never a read-scope bypass, and a
821
+ table the principal cannot read refuses the whole write. The WHERE should hit
822
+ a unique key: more than one matching row is a runtime error (deliberately —
823
+ silently picking an arbitrary row would be worse), and zero rows write NULL
824
+ (the column's constraints decide whether that lands).
825
+
826
+ ### `insertEach` — one statement for a caller-sized list
827
+
828
+ `items` names a context key holding an array of objects; `{ $item: "key" }`
829
+ values read from the current element (cast to the column's type from the
830
+ schema); every other value — literal, `$context`, `$lookup`, engine-injected
831
+ `$scope` — is constant across rows. Compiles to a single
832
+ `INSERT … SELECT … FROM jsonb_array_elements($items)`, so "one row per ticked
833
+ weekday" is one authored statement, not a code loop:
834
+
835
+ ```ts
836
+ { op: 'insertEach', table: 'class_templates', items: { $context: 'slots' },
837
+ values: { weekday: { $item: 'weekday' }, starts_at: { $item: 'startsAt' },
838
+ name: { $context: 'name' } },
839
+ onConflict: { target: ['studio_id', 'weekday', 'starts_at'] } } // composes
840
+ ```
841
+
842
+ Scope-wise it is an insert that happens N times: the insert phase gates it,
843
+ and `set`/`match` rules pin their columns identically on every row. A
844
+ non-array `items` value is refused loudly before any SQL runs; an empty array
845
+ inserts nothing and returns no rows.
846
+
847
+ ### Replay
848
+
849
+ The handler dispatches by the entry's kind — one wire shape:
850
+
851
+ ```
852
+ POST { "fingerprint": "tasks/setDone", "context": { "id": "task_1", "done": true } }
853
+ → 200 { "result": <affected row — or rows for a batch/bulk write> }
854
+ ```
855
+
856
+ The pipeline per statement: parse → desugar → **require context** (a write
857
+ never executes with holes — missing keys are a hard 400 `missing_context`
858
+ whose `details.expected` carries the FULL derived signature) → scope (the
859
+ same `ScopePolicy` reads use: `write` `set` rules write identity on insert
860
+ and update, `match` rules pin rows) → column check against the introspected schema →
861
+ compile (parameterized, via the read pipeline's own operators) → execute
862
+ with `RETURNING *`.
863
+
864
+ Enable replay on a handler by configuring the client and policy:
865
+
866
+ ```ts
867
+ handleQuery({ engine, locked: true, mutations: { client: db, policy } }, body, scope);
868
+ ```
869
+
870
+ `client` is structural (`MutationClient` — PGlite, a pg wrapper, a test
871
+ double). `locked` does not affect writes: they are replay-only under every
872
+ posture. Direct library use: `executeMutation(client, def, { context, scope, policy, schema })`.
873
+
874
+ A host that resolves a policy **per principal** (e.g. compiled from an ACL
875
+ layer) passes it per request: `VexHandlerConfig.scopePolicy` overrides the
876
+ engine's configured read policy for that request (`ExecuteOptions.scopePolicy`
877
+ at the library level), and the same policy goes in `mutations.policy` — one
878
+ policy, both sides of the wire. Omitted, the engine default applies.
879
+
880
+ ### Derived context signatures
881
+
882
+ The input contract of ANY entry — read or write — is computed from its
883
+ stored def: each `{ $context }` ref sits at a position whose column the
884
+ schema types. Discovery lists it per fingerprint (`context`, plus `shape`
885
+ for reads / `effect` = `{ op, table, columns }` for writes), and the
886
+ `missing_context` error teaches it. Nothing is authored; nothing drifts.
887
+
888
+ ```ts
889
+ collectMutationContext(def, schema); // { id: { type: 'string', column: 'tasks.id', note: 'upsert key — …' }, … }
890
+ collectQueryContext(dsl, schema); // same idea over a query's filter/compute/subqueries
891
+ mutationEffect(def); // [{ op: 'update', table: 'tasks', columns: ['done'] }]
892
+ ```
893
+
894
+ ### Authoring lint
895
+
896
+ `lintMutation(def)` flags an `update`/`delete` whose WHERE binds no
897
+ `$context` — such a write is not caller-bounded (its only row limit is the
898
+ scope policy). Run it in your seed path so an unkeyed write never ships:
899
+ loud at boot, never at runtime.
900
+
901
+ ### The write observer
902
+
903
+ Vex is the choke point every application write passes through, which makes
904
+ it the one place a host can hear about all of them. `mutations.onWrite`
905
+ (handler config; the hono adapter passes it through verbatim) fires once per
906
+ successful mutation replay, AFTER the commit:
907
+
908
+ ```ts
909
+ mutations: {
910
+ client, policy,
911
+ onWrite: ({ fingerprint, writes, scope }) => { … },
912
+ // writes: [{ table, op, rows }] — per statement, resolved post-desugar
913
+ // (an upsert reports the branch it took; insertEach reports 'insert'),
914
+ // rows as the database returned them. scope: what the request could
915
+ // not forge.
916
+ }
917
+ ```
918
+
919
+ The grain is the point: a two-statement mutation is two writes, each with
920
+ its own `RETURNING` rows, and an UPDATE the scope narrowed to nothing
921
+ carries zero rows — the observer is never told "a write landed" about a
922
+ write that changed nothing. Reads never fire it; a refused or failed
923
+ mutation never reaches it; a listener that throws is logged and contained,
924
+ because the response must tell the truth about a commit that already
925
+ happened.
926
+
927
+ What the host does with the news is not the engine's business — moss mints
928
+ tide write facts from it and fans row-less pings to the app. If you hand
929
+ `rows` onward, know who is on the other end: a row given to code is a row
930
+ outside every scope policy this engine compiles.
931
+
932
+ `executeWrites(client, def, mctx)` is the same per-statement shape at the
933
+ library level; `executeMutation` remains the flattened `Row[]` every
934
+ existing caller expects.
935
+
936
+ ---
937
+
938
+ ## Caching
939
+
940
+ ### Fingerprints — the one identity
941
+
942
+ The positive cache is keyed by the request's `fingerprint` (see the request
943
+ contract above). There are no cache modes — the body's posture says
944
+ everything:
945
+
946
+ - **No fingerprint** → generate, cache, mint (`fp_…`). The mint is an
947
+ immutable pin: embed it and replay that exact generation forever.
948
+ - **Fingerprint alone** → replay-or-`cache_miss`. Never generates. Context
949
+ values may vary per call — sorting and parameterized reloads are legal
950
+ against one cached DSL.
951
+ - **Fingerprint + request** → a named slot: hit when the stored request
952
+ matches, regenerate-and-replace when it differs; `protected` entries throw
953
+ `fingerprint_protected` instead of being replaced.
954
+
955
+ Concurrent identical generations collapse to one (single-flight, keyed by the
956
+ request hash).
957
+
958
+ ### Lifetime and protection
959
+
960
+ - Every replay stamps `lastUsedAt`; `sweepCache(cache, { maxIdleMs })` evicts
961
+ entries idle past the limit, skipping protected ones. Alive = used.
962
+ - `protected: true` is written only by a seed path or an explicit
963
+ `handleFingerprintPatch` — never by a runtime query.
964
+ - `ExecuteOptions.locked` (and `VexHandlerConfig.locked`) makes an engine call
965
+ or endpoint replay-only: asking it to generate throws `locked`.
966
+
967
+ ### Supporting hashes
968
+
969
+ - **Request hash** — intent + normalized shape + the *names* of context keys
970
+ (values excluded). Stored with named slots (the match/replace test) and the
971
+ key of the **negative cache**: a `cannotSatisfy` result is cached with a TTL
972
+ (`config.unsatisfiableTtlMs`, default 5 min) so an impossible request isn't
973
+ re-run until it expires.
974
+ - Every entry stores a **schema fingerprint**. After a schema change it no
975
+ longer matches and the entry is treated as a miss and evicted — its cached
976
+ DSL might reference columns that changed.
977
+
978
+ ### Backends
979
+
980
+ ```typescript
981
+ import {
982
+ createMemoryCache,
983
+ createPostgresCache,
984
+ createTieredCache,
985
+ } from '@niscorp/vex';
986
+ ```
987
+
988
+ **In-memory** (default) — a `Map`. Fast, volatile, single-process.
989
+
990
+ ```typescript
991
+ const cache = createMemoryCache();
992
+ ```
993
+
994
+ **Postgres** — durable L2. Stores entries as `jsonb`, enforces TTL server-side,
995
+ validates on write, and evicts corrupt/stale rows on read. Used directly it
996
+ reads from Postgres on every `get`, so it is meant to sit behind an L1.
997
+
998
+ ```typescript
999
+ const l2 = createPostgresCache({
1000
+ pool,
1001
+ schema: 'public', // default
1002
+ table: 'vex_cache', // default
1003
+ onError: (err) => console.error(err),
1004
+ });
1005
+ await l2.init(); // applies l2.sequence through strata's ledger — once, recorded
1006
+ ```
1007
+
1008
+ **Tiered** — L1 (memory) + L2 (durable), read-through/write-through. Reads hit
1009
+ L1; on an L1 miss it promotes a validated L2 entry into L1. Writes go to L1
1010
+ synchronously and L2 fire-and-forget.
1011
+
1012
+ ```typescript
1013
+ const cache = createTieredCache({
1014
+ l1: createMemoryCache(),
1015
+ l2: createPostgresCache({ pool }),
1016
+ warmup: 'full', // 'full' (default) | 'lazy' | { mode: 'partial', fingerprints: [...] }
1017
+ onError: (err) => console.error(err),
1018
+ });
1019
+ await cache.init(); // runs L2/L1 init + warm-up
1020
+
1021
+ const engine = createQueryEngine({ adapter, cache });
1022
+ ```
1023
+
1024
+ Warm-up modes: `full` loads all of L2 into L1 at startup; `lazy` fills L1 on
1025
+ demand; `partial` preloads only the listed fingerprints and lazy-fills the
1026
+ rest. The tiered cache is single-instance: separate processes keep separate
1027
+ L1s.
1028
+
1029
+ ---
1030
+
1031
+ ## Wiring the LLM agents
1032
+
1033
+ The engine consumes two hooks. Both are optional; supply them to answer
1034
+ free-form requests.
1035
+
1036
+ ```typescript
1037
+ type GenerateDsl = (request: QueryRequest, schema: DatabaseSchema) => Promise<Query>;
1038
+ type MapToShape = (rows: Row[], shape: unknown) =>
1039
+ Promise<{ ir: CompiledIr; transformed: unknown[] }>;
1040
+ ```
1041
+
1042
+ - `generateDsl` runs on a cache miss; it must return a DSL that passes
1043
+ `QuerySchema`. To signal an impossible request, throw
1044
+ `new VexError('unsatisfiable', reason)` — the runtime negative-caches it.
1045
+ - `mapToShape` runs when the requested shape differs from the raw rows; it
1046
+ returns a compiled Prism `ir` (cached for reuse) plus the transformed rows.
1047
+
1048
+ You can implement these however you like. Vex also ships a **reference
1049
+ implementation** built on `@niscorp/cortex` and `@niscorp/prism`, exported from
1050
+ the `@niscorp/vex/agent` subpath (`@niscorp/cortex`, `@niscorp/signal`, and
1051
+ `@niscorp/prism` become required peers only if you import it):
1052
+
1053
+ ```typescript
1054
+ import { createQueryDsl, createShapeMapper } from '@niscorp/vex/agent';
1055
+ import { createSignal } from '@niscorp/signal';
1056
+
1057
+ const schema = await engine.introspect();
1058
+ const queryJsonSchema = engine.getDslSchema();
1059
+
1060
+ const llm = createSignal('openrouter', { apiKey, model: 'qwen/qwen3.8-27b' });
1061
+
1062
+ const generateDsl = createQueryDsl({ llm, queryJsonSchema });
1063
+ const mapToShape = createShapeMapper(llm);
1064
+
1065
+ const engine = createQueryEngine({ adapter, generateDsl, mapToShape, cache });
1066
+ await engine.introspect();
1067
+ ```
1068
+
1069
+ - `createQueryDsl(config)` builds the `generateDsl` hook by running
1070
+ `vexQueryDslAgent`. It is built from a model and the DSL spec only — who it
1071
+ runs for arrives with every call. The engine calls the hook as
1072
+ `generateDsl(request, schema, caller)`: `schema` holds only the tables the
1073
+ caller's policy can read, and `caller.read(dsl)` runs a DSL through the
1074
+ engine's own pipeline under the caller's policy and scope values (`$context`
1075
+ binds NULL). That capability is the tools' only way to the database.
1076
+ - `createShapeMapper(llm)` builds the `mapToShape` hook by running Prism's
1077
+ `mappingAgent`. The `@niscorp/vex/agent` subpath also exports `vexQueryDslAgent`
1078
+ itself plus `createQueryTools` for customizing the tool set.
1079
+
1080
+ The reference query agent (`vexQueryDslAgent`, id `vex.query`) is a Cortex
1081
+ agent whose envelope payload is a `QuerySchema` query, with six tools:
1082
+
1083
+ | Tool | Input | Purpose |
1084
+ |------|-------|---------|
1085
+ | `getSchema` | `{ entities? }` | Read the schema (optionally a subset) |
1086
+ | `getSampleRows` | `{ entity, limit? }` | See real rows (default 5) |
1087
+ | `getDistinctValues` | `{ entity, field, limit? }` | Field cardinality (default 20) |
1088
+ | `describeField` | `{ entity, field }` | type, nullable, cardinality, null count, min/max |
1089
+ | `testQuery` | `{ dsl }` | Validate + compile + execute with synthetic params (LIMIT 5) |
1090
+ | `cannotSatisfy` | `{ reason }` | Declare the request impossible (`createQueryDsl` aborts the run and throws `unsatisfiable`) |
1091
+
1092
+ Bounds: `stepCount(20)` + `outputRetries(3)` stop conditions on the agent. Every
1093
+ data-touching tool is a DSL run through `caller.read` — there is no SQL in the
1094
+ tool set — so a sample row, a distinct value and a statistic are all things the
1095
+ caller could have read. (If you build your own generator,
1096
+ none of this applies — only the `Query` you return matters.)
1097
+
1098
+ `testQuery` executes drafts with **NULL synthetic params** — valid SQL against
1099
+ a column of any type (Postgres infers from the column side), so a draft is
1100
+ judged on executability, never on a guessed value's cast.
1101
+
1102
+ ### `vexGuide()` — the exported agent-facing contract
1103
+
1104
+ Any agent that reads or embeds Vex endpoints should inject `vexGuide()` (a
1105
+ string): the three request postures, fingerprint semantics, per-call context
1106
+ values, reserved `sortBy`/`sortDir`, the response envelope, and discovery.
1107
+ Attach it as a cortex tool `guide` or context producer. Apps never hand-write
1108
+ their own Vex explanation — that is how docs drift.
1109
+
1110
+ ---
1111
+
1112
+ ## Framework adapters
1113
+
1114
+ Vex does not own a server, but ships thin adapters that expose a resource with a
1115
+ discovery endpoint (`GET`) and a query endpoint (`POST`). Both are optional
1116
+ sub-exports.
1117
+
1118
+ ### Hono
1119
+
1120
+ ```typescript
1121
+ import { Hono } from 'hono';
1122
+ import { vex } from '@niscorp/vex/hono';
1123
+
1124
+ const app = new Hono();
1125
+
1126
+ app.route('/api/orders/vex', vex({
1127
+ engine,
1128
+ entities: ['orders', 'order_items'], // optional: restrict this resource
1129
+ getScope: async (c) => ({ accountId: await resolveAccount(c) }),
1130
+ }));
1131
+ ```
1132
+
1133
+ ### Express
1134
+
1135
+ ```typescript
1136
+ import express from 'express';
1137
+ import { vex } from '@niscorp/vex/express';
1138
+
1139
+ const app = express();
1140
+ app.use(express.json());
1141
+
1142
+ app.all('/api/orders/vex', vex({
1143
+ engine,
1144
+ entities: ['orders', 'order_items'],
1145
+ getScope: (req) => ({ accountId: req.user.accountId }),
1146
+ }));
1147
+ ```
1148
+
1149
+ ### The endpoints
1150
+
1151
+ - **`GET`** → discovery JSON: `vex` protocol version, description, the filtered
1152
+ entities (fields, relations, row counts), the request body contract (the
1153
+ three fingerprint postures), the protection summary (`protection:
1154
+ "all"|"some"|"none"`, `locked`), the fingerprint overview (`[{ fingerprint,
1155
+ kind, protected, schemaFresh, intent, context, shape|effect, lastUsedAt }]`
1156
+ — `context` is the derived, typed input signature; `effect` summarizes a
1157
+ write), and the full DSL JSON Schema. A client or agent can read this to
1158
+ learn how to call the resource — and exactly what each fingerprint takes.
1159
+ - **`POST`** → runs a query OR replays a write, one body shape. A
1160
+ `fingerprint` naming a `kind: 'mutation'` entry dispatches to the write
1161
+ pipeline (when `mutations` is configured); anything else is the read path
1162
+ (fingerprint replay, generation, or named slot); scope comes from
1163
+ `getScope`. Returns `{ status, body }` mapped to the HTTP response —
1164
+ unknown fingerprint → 404 `cache_miss`, protected mismatch → 409
1165
+ `fingerprint_protected`, locked → 403, a write with missing context → 400
1166
+ `missing_context` with the full derived signature in `details.expected`,
1167
+ other `VexError`s → 400 with `{ error, message, details }`, unexpected
1168
+ errors → 500.
1169
+ - **`PATCH`** with `{ fingerprint, protected }` → flip the protection bit.
1170
+ **`DELETE`** with `{ fingerprint }` → evict (unprotected only). Fingerprints
1171
+ ride the body — names contain `/`.
1172
+
1173
+ Under the hood all of these call the framework-agnostic `handleDiscovery` /
1174
+ `handleQuery` / `handleFingerprintPatch` / `handleFingerprintDelete` from
1175
+ `handler.ts`, which you can use directly with any framework. A handler config
1176
+ with `locked: true` makes the whole endpoint replay-only.
1177
+
1178
+ ---
1179
+
1180
+ ## Events
1181
+
1182
+ Pass `onEvent` to observe the pipeline. Every event is a discriminated union
1183
+ member on `type`:
1184
+
1185
+ ```typescript
1186
+ import type { VexEvent } from '@niscorp/vex';
1187
+
1188
+ const onEvent = (e: VexEvent) => {
1189
+ switch (e.type) {
1190
+ case 'query.start': /* intent, shape, fingerprint, entities */ break;
1191
+ case 'query.cache': /* hit, fingerprint, replaced */ break;
1192
+ case 'query.dsl': /* dsl, agentMs */ break;
1193
+ case 'query.sql': /* sql, warnings */ break;
1194
+ case 'query.rows': /* count, executionMs */ break;
1195
+ case 'query.mapped': /* mappingMs */ break;
1196
+ case 'query.done': /* totalMs */ break;
1197
+ case 'query.error': /* code, message */ break;
1198
+ // llm.request / llm.response — only if you emit them from your LLM wrapper
1199
+ }
1200
+ };
1201
+
1202
+ const engine = createQueryEngine({ adapter, onEvent });
1203
+ ```
1204
+
1205
+ The `llm.request`/`llm.response` variants exist for callers that wrap their LLM
1206
+ client to record a transcript (the dev server does this to write a full debug
1207
+ log); the engine itself emits the `query.*` events.
1208
+
1209
+ ---
1210
+
1211
+ ## Error handling
1212
+
1213
+ `execute` throws `VexError` on failure (the framework handlers convert these to
1214
+ HTTP responses). Missing context is **not** an error — it returns an empty
1215
+ response with `meta.missingContext`.
1216
+
1217
+ ```typescript
1218
+ import { VexError } from '@niscorp/vex';
1219
+
1220
+ try {
1221
+ const res = await engine.execute(request, { scope });
1222
+ } catch (err) {
1223
+ if (err instanceof VexError) {
1224
+ // err.code, err.message, err.details
1225
+ }
1226
+ }
1227
+ ```
1228
+
1229
+ `VexError.code` is one of:
1230
+
1231
+ | Code | Meaning |
1232
+ |------|---------|
1233
+ | `invalid_request` | The request failed Zod validation |
1234
+ | `invalid_dsl` | The analyzer rejected the resolved query (cartesian product, nesting, etc.) |
1235
+ | `missing_scope` | A `$scope` slot the host did not fill. The host's fault, not the request's: the HTTP handler answers 500 and keeps the key names in its log |
1236
+ | `scope_denied` | An entity is denied by the scope policy (`VexScopeError`) |
1237
+ | `missing_context` | A required context value was absent. Reads surface it softly via `meta.missingContext`; a WRITE hard-400s with the full derived signature in `details.expected` |
1238
+ | `execution_error` | Schema not loaded, or a database/runtime failure |
1239
+ | `agent_failed` | No `generateDsl` hook and no cached DSL for the shape |
1240
+ | `cache_miss` | Cache mode `only` and nothing cached |
1241
+ | `unsatisfiable` | The agent declared the request impossible (negative-cached) |
1242
+
1243
+ ---
1244
+
1245
+ ## Local development
1246
+
1247
+ The package ships a Postgres-backed dev environment and a runnable HTTP server.
1248
+
1249
+ ### Database
1250
+
1251
+ ```bash
1252
+ pnpm --filter @niscorp/vex docker:up # pgvector/pgvector:pg16 on host port 5433
1253
+ pnpm --filter @niscorp/vex seed # idempotent schema + sample data
1254
+ pnpm --filter @niscorp/vex docker:down
1255
+ ```
1256
+
1257
+ The dev database is `vex_dev` (user/pass `vex`/`vex`), exposed on **5433** to
1258
+ avoid clashing with a local 5432. `scripts/init.sql` runs on first boot; `seed`
1259
+ loads a realistic schema (customers, orders, products, categories, reviews, …)
1260
+ including a vector column for semantic search.
1261
+
1262
+ ### Environment
1263
+
1264
+ `.env` (see `.env.example`):
1265
+
1266
+ ```bash
1267
+ OPENROUTER_API_KEY= # enables the query + mapping agents
1268
+ OPENAI_API_KEY= # enables embeddings (text-embedding-3-small) for semantic search
1269
+ ```
1270
+
1271
+ The dev server also reads `DATABASE_URL` (default
1272
+ `postgresql://vex:vex@localhost:5433/vex_dev`), `PORT` (default 3456), and
1273
+ `VEX_CACHE` (`memory` | `postgres`) with `VEX_CACHE_SCHEMA` / `VEX_CACHE_TABLE`.
1274
+
1275
+ ### Run the server
1276
+
1277
+ ```bash
1278
+ pnpm --filter @niscorp/vex dev:server
1279
+ ```
1280
+
1281
+ It seeds if needed, introspects, wires the reference agents when
1282
+ `OPENROUTER_API_KEY` is set (falling back to deterministic-only otherwise),
1283
+ mounts several scoped resources via the Hono adapter, and logs per-stage timing
1284
+ to the console and a full LLM transcript to `logs/vex-debug.log`. `GET` any
1285
+ endpoint for discovery; `POST` to query.
1286
+
1287
+ ### Tests
1288
+
1289
+ ```bash
1290
+ pnpm --filter @niscorp/vex test
1291
+ ```
1292
+
1293
+ Unit tests (schemas, discovery, scope, hashing, resolver, analyzer) need no
1294
+ database or LLM. Integration tests (introspection, compilation, execution, cache
1295
+ backends, handlers) run against the seeded Postgres.