@niscorp/vex 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/DESIGN.md +48 -31
- package/DOCS.md +250 -55
- package/README.md +10 -7
- package/dist/adapters/express/index.cjs +1 -1
- package/dist/adapters/express/index.cjs.map +1 -1
- package/dist/adapters/express/index.js +1 -1
- package/dist/adapters/hono/index.cjs +1 -1
- package/dist/adapters/hono/index.cjs.map +1 -1
- package/dist/adapters/hono/index.js +1 -1
- package/dist/{chunk-FPKW4TRX.js → chunk-5CBKESBJ.js} +3 -3
- package/dist/chunk-5CBKESBJ.js.map +1 -0
- package/dist/index.cjs +1 -1
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +2 -2
- package/package.json +7 -7
- package/dist/chunk-FPKW4TRX.js.map +0 -1
package/DOCS.md
CHANGED
|
@@ -86,15 +86,15 @@ fingerprint accepts without failing a request first.
|
|
|
86
86
|
## Installation
|
|
87
87
|
|
|
88
88
|
```bash
|
|
89
|
-
pnpm add @niscorp/vex zod
|
|
89
|
+
pnpm add @niscorp/vex @niscorp/prism @niscorp/strata zod
|
|
90
90
|
```
|
|
91
91
|
|
|
92
|
-
`
|
|
92
|
+
`@niscorp/prism` (mappings), `@niscorp/strata` (the cache table's ledger) and `zod` are required peers. Add the others for the paths you use:
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
95
|
pnpm add pg # Postgres adapter + Postgres cache
|
|
96
96
|
pnpm add hono # or: express # framework adapter
|
|
97
|
-
pnpm add @niscorp/cortex @niscorp/signal
|
|
97
|
+
pnpm add @niscorp/cortex @niscorp/signal # reference LLM agents
|
|
98
98
|
```
|
|
99
99
|
|
|
100
100
|
For semantic (vector) search you need a Postgres with `pgvector` (the dev setup
|
|
@@ -124,9 +124,11 @@ indexes, and caches the result plus a schema fingerprint. The engine throws
|
|
|
124
124
|
`execution_error` ("schema not loaded") if you call `compile`/`execute`/`test`
|
|
125
125
|
before introspecting.
|
|
126
126
|
|
|
127
|
-
With no `generateDsl` hook, `execute` serves only requests
|
|
128
|
-
|
|
129
|
-
|
|
127
|
+
With no `generateDsl` hook, `execute` serves only requests that replay a
|
|
128
|
+
cached fingerprint — a seeded entry or a named slot. An unknown fingerprint sent
|
|
129
|
+
alone throws `cache_miss`; a request that would need generation throws
|
|
130
|
+
`agent_failed`. `compile` and `test` work fully because you pass them a DSL
|
|
131
|
+
directly.
|
|
130
132
|
|
|
131
133
|
### With LLM-backed generation and mapping
|
|
132
134
|
|
|
@@ -154,9 +156,11 @@ type QueryEngineConfig = {
|
|
|
154
156
|
scope?: ScopePolicy; // server-side access control
|
|
155
157
|
cache?: CacheBackend; // default: createMemoryCache()
|
|
156
158
|
onEvent?: VexEventHandler; // pipeline event stream
|
|
157
|
-
generateDsl?: (request, schema) => Promise<Query>;
|
|
159
|
+
generateDsl?: (request, schema, caller) => Promise<Query>;
|
|
158
160
|
mapToShape?: (rows, shape) => Promise<{ ir: CompiledIr; transformed: JsonValue }>;
|
|
159
161
|
embed?: (text, dimensions?) => Promise<number[]>; // text→vector for semantic filters
|
|
162
|
+
behaviors?: ScopeBehaviors; // which columns a scope key is bound to (see below)
|
|
163
|
+
rows?: LiveRowsConfig; // the rows cache behind `refresh: 'reactive'`
|
|
160
164
|
config?: {
|
|
161
165
|
maxNestingDepth?: number; // default: 2
|
|
162
166
|
defaultLimit?: number; // default: 100
|
|
@@ -166,11 +170,26 @@ type QueryEngineConfig = {
|
|
|
166
170
|
rejectUnindexedFilters?: boolean; // default: false
|
|
167
171
|
entities?: string[]; // whitelist for introspection
|
|
168
172
|
unsatisfiableTtlMs?: number; // negative-cache TTL, default 300_000 (5 min)
|
|
173
|
+
statementTimeoutMs?: number; // how long one read may run, default 10_000 (0: unbounded)
|
|
174
|
+
maxPresenceVariants?: number; // optional-key combinations per entry before a warning, default 32
|
|
169
175
|
};
|
|
170
176
|
};
|
|
171
177
|
```
|
|
172
178
|
|
|
173
179
|
- `config.entities`, when set, restricts `introspect()` to those tables.
|
|
180
|
+
- `config.statementTimeoutMs` is enforced by the database (`statement_timeout`),
|
|
181
|
+
set up at `introspect()`. A database that cannot enforce it (PGlite) runs
|
|
182
|
+
reads unbounded and the engine warns each time `introspect()` runs.
|
|
183
|
+
- `config.maxPresenceVariants` is not a limit on what runs: past it, a response
|
|
184
|
+
carries a warning that the entry has compiled that many distinct
|
|
185
|
+
optional-key combinations.
|
|
186
|
+
- **`behaviors`** is the same `ScopeBehaviors` document the host's policies are
|
|
187
|
+
compiled from (see [Grants and behaviors](#grants-and-behaviors)). The engine
|
|
188
|
+
reads it for one thing: which columns a scope key is bound to, so a generation
|
|
189
|
+
can be told who its caller is (`GenerationCaller.bindings`). `introspect()`
|
|
190
|
+
throws `execution_error` if a rule names a column the database does not have.
|
|
191
|
+
- **`rows`** sizes the reactive rows cache — see
|
|
192
|
+
[Reactive reads](#reactive-reads-refresh).
|
|
174
193
|
- `defaultLimit` is applied when a DSL omits `limit`; `maxLimit` clamps any
|
|
175
194
|
larger value.
|
|
176
195
|
- **`embed`** turns text into a vector for `semantic` filters, at parameter-
|
|
@@ -186,6 +205,9 @@ type ExecuteOptions = {
|
|
|
186
205
|
scope?: ScopeValues; // { [key]: value } for $scope references
|
|
187
206
|
entities?: string[]; // restrict generation to this entity subset for this call
|
|
188
207
|
locked?: boolean; // replay-only: unknown/changed fingerprints throw `locked` instead of generating
|
|
208
|
+
scopePolicy?: ScopePolicy; // per-request policy; overrides the engine's configured one for this call
|
|
209
|
+
signal?: AbortSignal; // with onChange: follow a `refresh: 'reactive'` entry until it aborts
|
|
210
|
+
onChange?: (response: QueryResponse) => void; // every later answer that differs from the last
|
|
189
211
|
};
|
|
190
212
|
```
|
|
191
213
|
|
|
@@ -271,7 +293,7 @@ type QueryResponse = {
|
|
|
271
293
|
intent?: string; // the stored intent (descriptive only)
|
|
272
294
|
};
|
|
273
295
|
context: Record<string, { // the resolved parameter contract
|
|
274
|
-
type?: 'string' | 'number' | 'boolean' | 'string[]' | 'number[]';
|
|
296
|
+
type?: 'string' | 'number' | 'boolean' | 'string[]' | 'number[]' | 'json';
|
|
275
297
|
kind: 'context' | 'scope' | 'semantic';
|
|
276
298
|
optional?: true; // supplying it switches a condition on
|
|
277
299
|
absent?: true; // optional, and not supplied this run (so: no type)
|
|
@@ -285,9 +307,10 @@ type QueryResponse = {
|
|
|
285
307
|
```
|
|
286
308
|
|
|
287
309
|
If the compiled query needs context/scope values the caller did not provide,
|
|
288
|
-
`execute` returns a **valid** response with an empty `result` and
|
|
310
|
+
`execute` returns a **valid** response with an empty `result` (`[]`) and
|
|
289
311
|
`meta.missingContext` listing the missing keys — it does not throw. Retry with
|
|
290
|
-
the values filled in.
|
|
312
|
+
the values filled in. (A missing **scope** value is different: it throws
|
|
313
|
+
`missing_scope`.) `meta.context` always describes the full parameter
|
|
291
314
|
contract so a caller knows what to provide.
|
|
292
315
|
|
|
293
316
|
---
|
|
@@ -298,11 +321,13 @@ contract so a caller knows what to provide.
|
|
|
298
321
|
type QueryEngine = {
|
|
299
322
|
introspect: () => Promise<DatabaseSchema>;
|
|
300
323
|
execute: (request: QueryRequest, options?: ExecuteOptions) => Promise<QueryResponse>;
|
|
301
|
-
compile: (dsl: Query
|
|
324
|
+
compile: (dsl: Query) => CompiledQuery;
|
|
302
325
|
test: (dsl: Query, scope?: ScopeValues) => Promise<TestResult>;
|
|
303
326
|
getDslSchema: () => object;
|
|
304
327
|
getSchema: () => DatabaseSchema | undefined;
|
|
305
328
|
cache: CacheBackend;
|
|
329
|
+
invalidate: (tables: readonly string[]) => void;
|
|
330
|
+
rows: { stats: () => LiveRowsStats; stop: () => void };
|
|
306
331
|
};
|
|
307
332
|
```
|
|
308
333
|
|
|
@@ -312,15 +337,24 @@ type QueryEngine = {
|
|
|
312
337
|
- **`compile(dsl)`** — run the deterministic pipeline (clamp limit →
|
|
313
338
|
discover → scope → resolve → analyze → adapter compile) and return the
|
|
314
339
|
`CompiledQuery` (`{ sql, paramSlots, contextContract }`). No execution, no
|
|
315
|
-
LLM. Throws `invalid_dsl` if the analyzer finds errors.
|
|
340
|
+
LLM. Throws `invalid_dsl` if the analyzer finds errors. Every `optional`
|
|
341
|
+
condition is kept, so this is the widest form of the query.
|
|
316
342
|
- **`test(dsl, scope?)`** — compile with `limit` forced to 5, build synthetic
|
|
317
343
|
context values, execute, and return `{ rows, warnings, errors }`. Errors are
|
|
318
344
|
caught and returned in `errors` rather than thrown. Useful for validating a
|
|
319
|
-
hand-written DSL against real data.
|
|
345
|
+
hand-written DSL against real data. Scope is never synthetic: a query whose
|
|
346
|
+
policy binds `$scope` values needs them passed, or the missing-scope refusal
|
|
347
|
+
comes back in `errors`.
|
|
320
348
|
- **`getDslSchema()`** — the DSL as a JSON Schema (draft-7). This is what you
|
|
321
349
|
feed an LLM so it learns the DSL.
|
|
322
350
|
- **`getSchema()`** — the last introspected `DatabaseSchema`, or `undefined`.
|
|
323
351
|
- **`cache`** — the live `CacheBackend` for inspection or manual eviction.
|
|
352
|
+
- **`invalidate(tables)`** — these tables were written: every reactive read
|
|
353
|
+
over any of them is refetched (once per burst). The vex handler calls it after
|
|
354
|
+
every committed mutation; a host calls it for writes vex did not make.
|
|
355
|
+
- **`rows`** — the reactive rows cache: `stats()` (entries, bytes, follows,
|
|
356
|
+
followers, hits, misses, evictions, oversized, refreshes) and `stop()` (clears
|
|
357
|
+
its timers, for a host that outlives the engine).
|
|
324
358
|
|
|
325
359
|
---
|
|
326
360
|
|
|
@@ -486,8 +520,9 @@ Grouped, with a renamed field:
|
|
|
486
520
|
}
|
|
487
521
|
```
|
|
488
522
|
|
|
489
|
-
Functions: `count` (field path or `"*"`); `
|
|
490
|
-
path or a
|
|
523
|
+
Functions: `count` (field path or `"*"`); `countDistinct` (a field path —
|
|
524
|
+
`COUNT(DISTINCT field)`); `sum` / `avg` / `min` / `max` (a field path or a
|
|
525
|
+
compute expression).
|
|
491
526
|
|
|
492
527
|
### Subqueries
|
|
493
528
|
|
|
@@ -569,7 +604,9 @@ type ScopeEntityRule =
|
|
|
569
604
|
delete?: ScopeMatch[]; // specific: just DELETE (match only — nothing to set)
|
|
570
605
|
};
|
|
571
606
|
|
|
572
|
-
type ScopeMatch =
|
|
607
|
+
type ScopeMatch =
|
|
608
|
+
| { match: string; to: string } // row.<match> = scope[to]
|
|
609
|
+
| { match: string; in: string }; // row.<match> = ANY(scope[in]) — filters only, never pins an INSERT
|
|
573
610
|
type ScopeSet = { set: string; to: string }; // INSERT/UPDATE row.<set> := scope[to]
|
|
574
611
|
```
|
|
575
612
|
|
|
@@ -609,8 +646,12 @@ Behavior:
|
|
|
609
646
|
where binding it would stamp NULL. A missing `$context` value is different: it
|
|
610
647
|
is the caller's, and a read surfaces it via `meta.missingContext`
|
|
611
648
|
(empty-but-valid).
|
|
612
|
-
- `match` is equality
|
|
613
|
-
|
|
649
|
+
- `match` is equality (`to`) or set membership (`in`, compiled to
|
|
650
|
+
`col = ANY($n)`). The set form only filters — a read, or the rows an
|
|
651
|
+
UPDATE/DELETE may touch: an empty set matches no row, an absent one is refused
|
|
652
|
+
with `missing_scope` like any other scope value, and a set-valued rule that
|
|
653
|
+
would have to pin an INSERT is refused (`scope_denied`) — a write's subject
|
|
654
|
+
must be a single value.
|
|
614
655
|
|
|
615
656
|
The policy applies whenever one is configured; the values fill it. Omitting
|
|
616
657
|
the values means nobody, never everybody.
|
|
@@ -725,7 +766,8 @@ same principal under that profile:
|
|
|
725
766
|
|
|
726
767
|
Grants are unchanged — the union of every role the caller holds — so this can
|
|
727
768
|
only narrow rows, never widen phases. A caller with no verb for the table is
|
|
728
|
-
still refused.
|
|
769
|
+
still refused. A mutation entry takes `reach` the same way (`SeedMutation.reach`),
|
|
770
|
+
and the write is compiled at that profile whoever asks.
|
|
729
771
|
|
|
730
772
|
The host supplies `policyForReach(reach)` (hono: `getPolicyForReach(c, reach)`),
|
|
731
773
|
because vex holds neither grants nor behaviors. **It fails closed both ways:** an
|
|
@@ -893,10 +935,12 @@ mutationEffect(def); // [{ op: 'update', table: 'tasks', column
|
|
|
893
935
|
|
|
894
936
|
### Authoring lint
|
|
895
937
|
|
|
896
|
-
`lintMutation(def)`
|
|
897
|
-
`$context` — such a write is not
|
|
898
|
-
|
|
899
|
-
|
|
938
|
+
`lintMutation(def)` returns a list of issues (empty when clean). It flags an
|
|
939
|
+
`update`/`delete` whose WHERE binds no `$context` — such a write is not
|
|
940
|
+
caller-bounded (its only row limit is the scope policy) — and any statement
|
|
941
|
+
containing an `optional` condition. `seedCache` runs it on every mutation entry
|
|
942
|
+
and throws on an issue, so an unkeyed write never ships: loud at boot, never at
|
|
943
|
+
runtime.
|
|
900
944
|
|
|
901
945
|
### The write observer
|
|
902
946
|
|
|
@@ -953,14 +997,16 @@ everything:
|
|
|
953
997
|
`fingerprint_protected` instead of being replaced.
|
|
954
998
|
|
|
955
999
|
Concurrent identical generations collapse to one (single-flight, keyed by the
|
|
956
|
-
request hash
|
|
1000
|
+
request hash and the active policy — two callers who can see different tables
|
|
1001
|
+
are not asking the same thing).
|
|
957
1002
|
|
|
958
1003
|
### Lifetime and protection
|
|
959
1004
|
|
|
960
|
-
-
|
|
961
|
-
entries idle past the limit,
|
|
962
|
-
|
|
963
|
-
|
|
1005
|
+
- A replay stamps `lastUsedAt` (at most once a minute per entry);
|
|
1006
|
+
`sweepCache(cache, { maxIdleMs })` evicts entries idle past the limit,
|
|
1007
|
+
skipping protected ones, and returns how many it evicted. Alive = used.
|
|
1008
|
+
- `protected: true` is written only by the seed path (`seedCache`) or an
|
|
1009
|
+
explicit `handleFingerprintPatch` — never by a runtime query.
|
|
964
1010
|
- `ExecuteOptions.locked` (and `VexHandlerConfig.locked`) makes an engine call
|
|
965
1011
|
or endpoint replay-only: asking it to generate throws `locked`.
|
|
966
1012
|
|
|
@@ -968,7 +1014,7 @@ request hash).
|
|
|
968
1014
|
|
|
969
1015
|
- **Request hash** — intent + normalized shape + the *names* of context keys
|
|
970
1016
|
(values excluded). Stored with named slots (the match/replace test) and the
|
|
971
|
-
key of the **negative cache**: a `cannotSatisfy` result is cached with a TTL
|
|
1017
|
+
key (with the active policy) of the **negative cache**: a `cannotSatisfy` result is cached with a TTL
|
|
972
1018
|
(`config.unsatisfiableTtlMs`, default 5 min) so an impossible request isn't
|
|
973
1019
|
re-run until it expires.
|
|
974
1020
|
- Every entry stores a **schema fingerprint**. After a schema change it no
|
|
@@ -1005,6 +1051,11 @@ const l2 = createPostgresCache({
|
|
|
1005
1051
|
await l2.init(); // applies l2.sequence through strata's ledger — once, recorded
|
|
1006
1052
|
```
|
|
1007
1053
|
|
|
1054
|
+
`init()` runs a strata migration, so the pool must expose `transaction` — the
|
|
1055
|
+
PGlite pool has one; a bare `pg.Pool` needs a wrapper that checks a client out
|
|
1056
|
+
per transaction (see `PgPool`). A host that migrates several owners in one run
|
|
1057
|
+
collects `l2.sequence` (`nisc.vex.cache`) instead of calling `init()`.
|
|
1058
|
+
|
|
1008
1059
|
**Tiered** — L1 (memory) + L2 (durable), read-through/write-through. Reads hit
|
|
1009
1060
|
L1; on an L1 miss it promotes a validated L2 entry into L1. Writes go to L1
|
|
1010
1061
|
synchronously and L2 fire-and-forget.
|
|
@@ -1026,6 +1077,90 @@ demand; `partial` preloads only the listed fingerprints and lazy-fills the
|
|
|
1026
1077
|
rest. The tiered cache is single-instance: separate processes keep separate
|
|
1027
1078
|
L1s.
|
|
1028
1079
|
|
|
1080
|
+
### Seeding
|
|
1081
|
+
|
|
1082
|
+
`seedCache(cache, entries)` turns authored entries into **protected** cache
|
|
1083
|
+
rows — the API surface of a locked endpoint. Call it at boot, after
|
|
1084
|
+
`cache.init()`.
|
|
1085
|
+
|
|
1086
|
+
```typescript
|
|
1087
|
+
import { seedCache } from '@niscorp/vex';
|
|
1088
|
+
import type { SeedEntry, SeedMutation } from '@niscorp/vex';
|
|
1089
|
+
|
|
1090
|
+
const ordersRecent: SeedEntry = {
|
|
1091
|
+
fingerprint: 'orders/recent',
|
|
1092
|
+
intent: 'A customer\'s orders, newest first',
|
|
1093
|
+
shape: [{ order_id: '', total: 0, created_at: '' }],
|
|
1094
|
+
dsl: {
|
|
1095
|
+
from: ['orders'],
|
|
1096
|
+
fields: [{ field: 'orders.id', as: 'order_id' }, 'orders.total', 'orders.created_at'],
|
|
1097
|
+
filter: { eq: ['orders.customer_id', { $context: 'customerId' }] },
|
|
1098
|
+
sort: [{ field: 'orders.created_at', dir: 'desc' }],
|
|
1099
|
+
},
|
|
1100
|
+
// mapping?: uncompiled Prism over { result, context, scope }; omitted = identity
|
|
1101
|
+
// reach?: the scope profile this read must be served at
|
|
1102
|
+
// refresh?: 'snapshot' (default) | 'reactive'
|
|
1103
|
+
};
|
|
1104
|
+
|
|
1105
|
+
const taskSetDone: SeedMutation = {
|
|
1106
|
+
fingerprint: 'tasks/setDone',
|
|
1107
|
+
mutation: { op: 'update', table: 'tasks', set: { done: { $context: 'done' } },
|
|
1108
|
+
where: { eq: ['tasks.id', { $context: 'id' }] } },
|
|
1109
|
+
};
|
|
1110
|
+
|
|
1111
|
+
await seedCache(engine.cache, [ordersRecent, taskSetDone]);
|
|
1112
|
+
```
|
|
1113
|
+
|
|
1114
|
+
- A read's `mapping` is compiled to Prism IR here; with none, the identity
|
|
1115
|
+
(`{ $ref: '$.result' }`) is stored, so a replay never falls through to
|
|
1116
|
+
`mapToShape`.
|
|
1117
|
+
- A mutation is linted (`lintMutation`); a failing statement throws.
|
|
1118
|
+
- It is idempotent against a durable cache: a stored row that already matches
|
|
1119
|
+
its authored definition is left alone; anything else at that fingerprint
|
|
1120
|
+
converges to the authored definition, protected.
|
|
1121
|
+
|
|
1122
|
+
`@niscorp/vex/pglite` exports `createPglitePool(db, parsers?)`, which shapes a
|
|
1123
|
+
PGlite database as the `PgPool` the Postgres adapter and cache take (with
|
|
1124
|
+
`transaction`), and `RAW_DATE_PARSERS`, which returns `DATE`/`TIMESTAMPTZ`
|
|
1125
|
+
columns as raw strings. Pass the parsers to the adapter's pool; give the
|
|
1126
|
+
Postgres cache a pool without them — it reads its own timestamp columns as
|
|
1127
|
+
`Date`s.
|
|
1128
|
+
|
|
1129
|
+
### Reactive reads (`refresh`)
|
|
1130
|
+
|
|
1131
|
+
An entry's `refresh` says when its answer is refreshed: `'snapshot'` (the
|
|
1132
|
+
default) answers when asked; `'reactive'` answers again whenever a write lands
|
|
1133
|
+
on a table the query reads, for as long as somebody follows it. The mode is
|
|
1134
|
+
authored on a seeded entry — never generated (`REFRESH_MODES`, `isRefresh`,
|
|
1135
|
+
type `Refresh`).
|
|
1136
|
+
|
|
1137
|
+
- **Following.** Pass `{ signal, onChange }` in `ExecuteOptions`: the returned
|
|
1138
|
+
promise is the first answer; every later answer that differs goes to
|
|
1139
|
+
`onChange` until `signal` aborts. Over HTTP there is no callback, so
|
|
1140
|
+
`handleQuery(config, body, scope, live?)` takes an in-process `VexLive`
|
|
1141
|
+
(`{ signal, onChange }`), and the hono adapter reads one from the request's
|
|
1142
|
+
env under `VEX_LIVE_ENV` (the third argument of `app.request`).
|
|
1143
|
+
- **Invalidation.** The handler calls
|
|
1144
|
+
`engine.invalidate(tablesChangedBy(writes, schema))` after every committed
|
|
1145
|
+
mutation; for a delete, `tablesChangedBy` adds every table that references a
|
|
1146
|
+
deleted one. Writes vex never sees are the host's to announce with
|
|
1147
|
+
`engine.invalidate`.
|
|
1148
|
+
- **The rows cache** is per engine, in process memory, sized by
|
|
1149
|
+
`QueryEngineConfig.rows`:
|
|
1150
|
+
|
|
1151
|
+
```typescript
|
|
1152
|
+
type LiveRowsConfig = {
|
|
1153
|
+
ttlMs?: number; // how long stored rows are trusted; default 60_000
|
|
1154
|
+
maxEntries?: number; // default 5_000
|
|
1155
|
+
maxBytes?: number; // JSON length of stored results; default 64 MiB
|
|
1156
|
+
maxEntryBytes?: number; // largest single result stored; default maxBytes / 10
|
|
1157
|
+
debounceMs?: number; // burst window before followed reads refetch; default 10
|
|
1158
|
+
};
|
|
1159
|
+
```
|
|
1160
|
+
|
|
1161
|
+
A reactive entry should `sort`, and reads time from `$scope`, never context.
|
|
1162
|
+
Design: [DESIGN.md](./DESIGN.md) **Reactive reads**.
|
|
1163
|
+
|
|
1029
1164
|
---
|
|
1030
1165
|
|
|
1031
1166
|
## Wiring the LLM agents
|
|
@@ -1034,38 +1169,46 @@ The engine consumes two hooks. Both are optional; supply them to answer
|
|
|
1034
1169
|
free-form requests.
|
|
1035
1170
|
|
|
1036
1171
|
```typescript
|
|
1037
|
-
type GenerateDsl = (request: QueryRequest, schema: DatabaseSchema) => Promise<Query>;
|
|
1172
|
+
type GenerateDsl = (request: QueryRequest, schema: DatabaseSchema, caller: GenerationCaller) => Promise<Query>;
|
|
1038
1173
|
type MapToShape = (rows: Row[], shape: unknown) =>
|
|
1039
|
-
Promise<{ ir: CompiledIr; transformed:
|
|
1174
|
+
Promise<{ ir: CompiledIr; transformed: JsonValue }>;
|
|
1175
|
+
|
|
1176
|
+
type GenerationCaller = {
|
|
1177
|
+
read: (dsl: Query) => Promise<{ rows: Row[]; sql: string; warnings: string[] }>;
|
|
1178
|
+
bindings: ScopeBinding[]; // { entity, field, key } — columns bound to scope keys the caller carries
|
|
1179
|
+
};
|
|
1040
1180
|
```
|
|
1041
1181
|
|
|
1042
1182
|
- `generateDsl` runs on a cache miss; it must return a DSL that passes
|
|
1043
1183
|
`QuerySchema`. To signal an impossible request, throw
|
|
1044
1184
|
`new VexError('unsatisfiable', reason)` — the runtime negative-caches it.
|
|
1045
|
-
- `mapToShape` runs when
|
|
1046
|
-
|
|
1185
|
+
- `mapToShape` runs when a shape is in play (the request's, or the stored one on
|
|
1186
|
+
a fingerprint-only replay) and the entry has no stored mapping; it returns a
|
|
1187
|
+
compiled Prism `ir` plus the mapped result. The `ir` is stored with the entry
|
|
1188
|
+
only on a miss — a replayed entry that has none calls the hook every time.
|
|
1189
|
+
Without the hook the raw rows are returned.
|
|
1047
1190
|
|
|
1048
1191
|
You can implement these however you like. Vex also ships a **reference
|
|
1049
1192
|
implementation** built on `@niscorp/cortex` and `@niscorp/prism`, exported from
|
|
1050
|
-
the `@niscorp/vex/agent` subpath (`@niscorp/cortex
|
|
1051
|
-
|
|
1193
|
+
the `@niscorp/vex/agent` subpath (`@niscorp/cortex` is an optional peer, needed
|
|
1194
|
+
only if you import it; the model client comes from `@niscorp/signal`):
|
|
1052
1195
|
|
|
1053
1196
|
```typescript
|
|
1054
1197
|
import { createQueryDsl, createShapeMapper } from '@niscorp/vex/agent';
|
|
1055
1198
|
import { createSignal } from '@niscorp/signal';
|
|
1056
1199
|
|
|
1057
|
-
const schema = await engine.introspect();
|
|
1058
|
-
const queryJsonSchema = engine.getDslSchema();
|
|
1059
|
-
|
|
1060
1200
|
const llm = createSignal('openrouter', { apiKey, model: 'qwen/qwen3.8-27b' });
|
|
1061
1201
|
|
|
1062
|
-
const generateDsl = createQueryDsl({ llm, queryJsonSchema });
|
|
1202
|
+
const generateDsl = createQueryDsl({ llm, queryJsonSchema: bare.getDslSchema() });
|
|
1063
1203
|
const mapToShape = createShapeMapper(llm);
|
|
1064
1204
|
|
|
1065
1205
|
const engine = createQueryEngine({ adapter, generateDsl, mapToShape, cache });
|
|
1066
1206
|
await engine.introspect();
|
|
1067
1207
|
```
|
|
1068
1208
|
|
|
1209
|
+
(`bare` is any engine — `getDslSchema()` needs no introspection, so
|
|
1210
|
+
`createQueryEngine({ adapter })` is enough to read it from.)
|
|
1211
|
+
|
|
1069
1212
|
- `createQueryDsl(config)` builds the `generateDsl` hook by running
|
|
1070
1213
|
`vexQueryDslAgent`. It is built from a model and the DSL spec only — who it
|
|
1071
1214
|
runs for arrives with every call. The engine calls the hook as
|
|
@@ -1074,8 +1217,12 @@ await engine.introspect();
|
|
|
1074
1217
|
engine's own pipeline under the caller's policy and scope values (`$context`
|
|
1075
1218
|
binds NULL). That capability is the tools' only way to the database.
|
|
1076
1219
|
- `createShapeMapper(llm)` builds the `mapToShape` hook by running Prism's
|
|
1077
|
-
`mappingAgent`.
|
|
1078
|
-
|
|
1220
|
+
`mappingAgent`. When the rows already are the shape (`rowsFitShape` — a flat
|
|
1221
|
+
shape whose keys and value kinds the first row matches exactly) it stores the
|
|
1222
|
+
identity mapping and makes no model call. The `@niscorp/vex/agent` subpath
|
|
1223
|
+
also exports `vexQueryDslAgent` itself, `createQueryTools` for customizing the
|
|
1224
|
+
tool set, and `describeCaller`, which turns `caller.bindings` into the
|
|
1225
|
+
context line the agent reads.
|
|
1079
1226
|
|
|
1080
1227
|
The reference query agent (`vexQueryDslAgent`, id `vex.query`) is a Cortex
|
|
1081
1228
|
agent whose envelope payload is a `QuerySchema` query, with six tools:
|
|
@@ -1086,10 +1233,10 @@ agent whose envelope payload is a `QuerySchema` query, with six tools:
|
|
|
1086
1233
|
| `getSampleRows` | `{ entity, limit? }` | See real rows (default 5) |
|
|
1087
1234
|
| `getDistinctValues` | `{ entity, field, limit? }` | Field cardinality (default 20) |
|
|
1088
1235
|
| `describeField` | `{ entity, field }` | type, nullable, cardinality, null count, min/max |
|
|
1089
|
-
| `testQuery` |
|
|
1236
|
+
| `testQuery` | the draft query itself (not wrapped) | Validate + compile + execute with synthetic params (LIMIT 5) |
|
|
1090
1237
|
| `cannotSatisfy` | `{ reason }` | Declare the request impossible (`createQueryDsl` aborts the run and throws `unsatisfiable`) |
|
|
1091
1238
|
|
|
1092
|
-
Bounds: `stepCount(20)
|
|
1239
|
+
Bounds: `stepCount(20)`, `outputRetries(3)` and `repeatedCalls(2)` stop conditions on the agent. Every
|
|
1093
1240
|
data-touching tool is a DSL run through `caller.read` — there is no SQL in the
|
|
1094
1241
|
tool set — so a sample row, a distinct value and a statistic are all things the
|
|
1095
1242
|
caller could have read. (If you build your own generator,
|
|
@@ -1130,6 +1277,16 @@ app.route('/api/orders/vex', vex({
|
|
|
1130
1277
|
}));
|
|
1131
1278
|
```
|
|
1132
1279
|
|
|
1280
|
+
The rest of `VexHonoConfig`, all optional:
|
|
1281
|
+
|
|
1282
|
+
| Option | Purpose |
|
|
1283
|
+
|--------|---------|
|
|
1284
|
+
| `locked` | Replay-only endpoint: no generation, no fingerprint management |
|
|
1285
|
+
| `getPolicy(c)` | Per-request `ScopePolicy`; governs reads, mutation replay and discovery on this endpoint |
|
|
1286
|
+
| `getPolicyForReach(c, reach)` | The same principal's policy at a reach an entry demands |
|
|
1287
|
+
| `mutations` | `{ client, policy?, onWrite? }` — enables mutation replay; `policy` is the fallback when `getPolicy` is absent |
|
|
1288
|
+
| `onExecute` | The execution observer (see [Events](#events)) |
|
|
1289
|
+
|
|
1133
1290
|
### Express
|
|
1134
1291
|
|
|
1135
1292
|
```typescript
|
|
@@ -1146,29 +1303,41 @@ app.all('/api/orders/vex', vex({
|
|
|
1146
1303
|
}));
|
|
1147
1304
|
```
|
|
1148
1305
|
|
|
1306
|
+
`VexExpressConfig` also takes `locked` and `mutations`
|
|
1307
|
+
(`{ client, policy, onWrite? }`). It has no per-request policy hook. Any method
|
|
1308
|
+
other than `GET`/`POST`/`PATCH`/`DELETE` answers 405.
|
|
1309
|
+
|
|
1149
1310
|
### The endpoints
|
|
1150
1311
|
|
|
1151
1312
|
- **`GET`** → discovery JSON: `vex` protocol version, description, the filtered
|
|
1152
1313
|
entities (fields, relations, row counts), the request body contract (the
|
|
1153
1314
|
three fingerprint postures), the protection summary (`protection:
|
|
1154
1315
|
"all"|"some"|"none"`, `locked`), the fingerprint overview (`[{ fingerprint,
|
|
1155
|
-
kind, protected, schemaFresh, intent,
|
|
1156
|
-
— `context` is the derived, typed input signature; `effect`
|
|
1157
|
-
write), and the full DSL JSON Schema. A client or agent can read
|
|
1158
|
-
learn how to call the resource — and exactly what each fingerprint
|
|
1316
|
+
kind, protected, schemaFresh, intent, createdAt, lastUsedAt, context,
|
|
1317
|
+
shape|effect }]` — `context` is the derived, typed input signature; `effect`
|
|
1318
|
+
summarizes a write), and the full DSL JSON Schema. A client or agent can read
|
|
1319
|
+
this to learn how to call the resource — and exactly what each fingerprint
|
|
1320
|
+
takes. Under a per-request policy, entities and entries the policy cannot
|
|
1321
|
+
touch are not listed.
|
|
1159
1322
|
- **`POST`** → runs a query OR replays a write, one body shape. A
|
|
1160
1323
|
`fingerprint` naming a `kind: 'mutation'` entry dispatches to the write
|
|
1161
1324
|
pipeline (when `mutations` is configured); anything else is the read path
|
|
1162
1325
|
(fingerprint replay, generation, or named slot); scope comes from
|
|
1163
1326
|
`getScope`. Returns `{ status, body }` mapped to the HTTP response —
|
|
1164
1327
|
unknown fingerprint → 404 `cache_miss`, protected mismatch → 409
|
|
1165
|
-
`fingerprint_protected`,
|
|
1328
|
+
`fingerprint_protected`, a write with missing context → 400
|
|
1166
1329
|
`missing_context` with the full derived signature in `details.expected`,
|
|
1167
|
-
other `VexError`s
|
|
1168
|
-
|
|
1330
|
+
other `VexError`s (including `locked` and `scope_denied`) → 400 with
|
|
1331
|
+
`{ error, message, details }`. A reach the host cannot serve for this
|
|
1332
|
+
principal → 403 `scope_denied`. 500s are the host's faults: `missing_scope`
|
|
1333
|
+
(key names stay in the log), an entry demanding a reach with no
|
|
1334
|
+
`policyForReach`, a mutation fingerprint on a handler with no `mutations`
|
|
1335
|
+
configured, and unexpected errors.
|
|
1169
1336
|
- **`PATCH`** with `{ fingerprint, protected }` → flip the protection bit.
|
|
1170
|
-
**`DELETE`** with `{ fingerprint }` → evict (
|
|
1171
|
-
|
|
1337
|
+
**`DELETE`** with `{ fingerprint }` → evict (a protected entry, read or mutation,
|
|
1338
|
+
is refused with 409 `fingerprint_protected`). Both are refused with 403 `locked` on a
|
|
1339
|
+
locked endpoint and answer 404 `cache_miss` for an unknown fingerprint.
|
|
1340
|
+
Fingerprints ride the body — names contain `/`.
|
|
1172
1341
|
|
|
1173
1342
|
Under the hood all of these call the framework-agnostic `handleDiscovery` /
|
|
1174
1343
|
`handleQuery` / `handleFingerprintPatch` / `handleFingerprintDelete` from
|
|
@@ -1195,6 +1364,8 @@ const onEvent = (e: VexEvent) => {
|
|
|
1195
1364
|
case 'query.mapped': /* mappingMs */ break;
|
|
1196
1365
|
case 'query.done': /* totalMs */ break;
|
|
1197
1366
|
case 'query.error': /* code, message */ break;
|
|
1367
|
+
case 'rows.evict': /* reason: 'capacity' | 'oversized', bytes */ break;
|
|
1368
|
+
case 'rows.error': /* message — a reactive refetch failed; the last good answer stands */ break;
|
|
1198
1369
|
// llm.request / llm.response — only if you emit them from your LLM wrapper
|
|
1199
1370
|
}
|
|
1200
1371
|
};
|
|
@@ -1204,7 +1375,29 @@ const engine = createQueryEngine({ adapter, onEvent });
|
|
|
1204
1375
|
|
|
1205
1376
|
The `llm.request`/`llm.response` variants exist for callers that wrap their LLM
|
|
1206
1377
|
client to record a transcript (the dev server does this to write a full debug
|
|
1207
|
-
log); the engine itself emits the `query.*` events.
|
|
1378
|
+
log); the engine itself emits the `query.*` and `rows.*` events.
|
|
1379
|
+
|
|
1380
|
+
### The execution observer
|
|
1381
|
+
|
|
1382
|
+
`onExecute` (handler config; the hono adapter passes it through) fires once per
|
|
1383
|
+
`handleQuery`, read or write, after the outcome is known:
|
|
1384
|
+
|
|
1385
|
+
```typescript
|
|
1386
|
+
type ExecuteRecord = {
|
|
1387
|
+
kind: 'query' | 'mutation';
|
|
1388
|
+
status: 'ok' | 'error' | 'refused'; // 200 | anything else | 403
|
|
1389
|
+
fingerprint?: string;
|
|
1390
|
+
reach?: string;
|
|
1391
|
+
cacheHit?: boolean; // reads
|
|
1392
|
+
rows?: number; // returned (reads) or written (mutations)
|
|
1393
|
+
startUnixNano: number;
|
|
1394
|
+
endUnixNano: number;
|
|
1395
|
+
scope: ScopeValues;
|
|
1396
|
+
};
|
|
1397
|
+
```
|
|
1398
|
+
|
|
1399
|
+
With no observer the record is not built. A listener that throws is logged and
|
|
1400
|
+
contained.
|
|
1208
1401
|
|
|
1209
1402
|
---
|
|
1210
1403
|
|
|
@@ -1236,8 +1429,10 @@ try {
|
|
|
1236
1429
|
| `scope_denied` | An entity is denied by the scope policy (`VexScopeError`) |
|
|
1237
1430
|
| `missing_context` | A required context value was absent. Reads surface it softly via `meta.missingContext`; a WRITE hard-400s with the full derived signature in `details.expected` |
|
|
1238
1431
|
| `execution_error` | Schema not loaded, or a database/runtime failure |
|
|
1239
|
-
| `agent_failed` |
|
|
1240
|
-
| `cache_miss` |
|
|
1432
|
+
| `agent_failed` | Generation was needed and there is no `generateDsl` hook, or the reference agent's run failed |
|
|
1433
|
+
| `cache_miss` | A fingerprint was sent alone and nothing is stored under it |
|
|
1434
|
+
| `fingerprint_protected` | A protected entry was asked to change: a named-slot request that no longer matches its stored one |
|
|
1435
|
+
| `locked` | A replay-only call (`locked: true`) would have needed generation |
|
|
1241
1436
|
| `unsatisfiable` | The agent declared the request impossible (negative-cached) |
|
|
1242
1437
|
|
|
1243
1438
|
---
|
package/README.md
CHANGED
|
@@ -7,11 +7,12 @@ The LLM never writes SQL. It fills in a validated JSON DSL; a deterministic pipe
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
9
9
|
```bash
|
|
10
|
-
pnpm add @niscorp/vex zod
|
|
11
|
-
#
|
|
12
|
-
|
|
13
|
-
#
|
|
14
|
-
pnpm add
|
|
10
|
+
pnpm add @niscorp/vex @niscorp/prism @niscorp/strata zod
|
|
11
|
+
# prism (mappings), strata (the cache table's ledger) and zod are required
|
|
12
|
+
# peers. Add the others for the paths you use:
|
|
13
|
+
pnpm add pg # a Postgres pool for the adapter and the cache
|
|
14
|
+
pnpm add hono # the Hono adapter (or: express)
|
|
15
|
+
pnpm add @niscorp/cortex @niscorp/signal # only if you wire the bundled LLM agents
|
|
15
16
|
```
|
|
16
17
|
|
|
17
18
|
## Quick Example
|
|
@@ -85,10 +86,12 @@ engine.cache; // CacheBackend — the live cache
|
|
|
85
86
|
|
|
86
87
|
| Area | Provided |
|
|
87
88
|
|------|----------|
|
|
88
|
-
| Database adapters | PostgreSQL (`createPostgresAdapter`) — pluggable `DatabaseAdapter` interface for others |
|
|
89
|
+
| Database adapters | PostgreSQL (`createPostgresAdapter`) — pluggable `DatabaseAdapter` interface for others. `createPglitePool` (`@niscorp/vex/pglite`) shapes a PGlite database as the pool it takes |
|
|
89
90
|
| Cache backends | In-memory, Postgres (durable), tiered L1/L2 with warm-up |
|
|
90
91
|
| Framework adapters | Hono (`@niscorp/vex/hono`), Express (`@niscorp/vex/express`) — discovery + query endpoints |
|
|
91
|
-
| LLM integration | `generateDsl` / `mapToShape` hooks; reference agents built on `@niscorp/cortex` |
|
|
92
|
+
| LLM integration | `generateDsl` / `mapToShape` hooks; reference agents built on `@niscorp/cortex` (`@niscorp/vex/agent`: `createQueryDsl`, `createShapeMapper`) |
|
|
93
|
+
| Writes | Mutation entries in the same cache — a closed grammar, replay-only by fingerprint, never generated; `lintMutation` checks a statement at seed |
|
|
94
|
+
| Seeding and freshness | `seedCache` turns authored `SeedEntry` / `SeedMutation` rows into protected cache entries; an entry's `refresh` is `'snapshot'` or `'reactive'` |
|
|
92
95
|
| Safety | Server-side scope policies, query analyzer (cartesian/nesting/index checks), parameterized SQL only |
|
|
93
96
|
| Its own table | The Postgres cache's table is a [strata](../strata/README.md) sequence (`cache.sequence`, `nisc.vex.cache`): `cache.init()` applies it through the ledger once, instead of `IF NOT EXISTS` on every boot. Requires a pool with `transaction` (the PGlite pool has one). |
|
|
94
97
|
|
|
@@ -1340,7 +1340,7 @@ var handleFingerprintDelete = async (config, fingerprint) => {
|
|
|
1340
1340
|
if (entry === void 0) {
|
|
1341
1341
|
return { status: 404, body: { error: "cache_miss", message: `Unknown fingerprint "${fingerprint}"` } };
|
|
1342
1342
|
}
|
|
1343
|
-
if (entry.kind === "ok" && entry.protected === true) {
|
|
1343
|
+
if ((entry.kind === "ok" || entry.kind === "mutation") && entry.protected === true) {
|
|
1344
1344
|
return { status: 409, body: { error: "fingerprint_protected", message: `"${fingerprint}" is protected \u2014 unprotect it first.` } };
|
|
1345
1345
|
}
|
|
1346
1346
|
await config.engine.cache.delete(fingerprint);
|