@niscorp/vex 0.3.1 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/CHANGELOG.md +27 -0
  2. package/DESIGN.md +11 -7
  3. package/DOCS.md +52 -9
  4. package/dist/adapters/express/index.cjs +1 -1
  5. package/dist/adapters/express/index.cjs.map +1 -1
  6. package/dist/adapters/express/index.d.cts +3 -2
  7. package/dist/adapters/express/index.d.ts +3 -2
  8. package/dist/adapters/express/index.js +3 -2
  9. package/dist/adapters/express/index.js.map +1 -1
  10. package/dist/adapters/hono/index.cjs +1 -1
  11. package/dist/adapters/hono/index.cjs.map +1 -1
  12. package/dist/adapters/hono/index.d.cts +3 -2
  13. package/dist/adapters/hono/index.d.ts +3 -2
  14. package/dist/adapters/hono/index.js +3 -2
  15. package/dist/adapters/hono/index.js.map +1 -1
  16. package/dist/agent/index.cjs +144 -21
  17. package/dist/agent/index.cjs.map +1 -1
  18. package/dist/agent/index.d.cts +3 -2
  19. package/dist/agent/index.d.ts +3 -2
  20. package/dist/agent/index.js +146 -22
  21. package/dist/agent/index.js.map +1 -1
  22. package/dist/{chunk-3HORTMZ7.js → chunk-AKR6HU34.js} +9 -5
  23. package/dist/chunk-AKR6HU34.js.map +1 -0
  24. package/dist/chunk-F7RDSWYF.js +7 -0
  25. package/dist/chunk-F7RDSWYF.js.map +1 -0
  26. package/dist/{chunk-J7JYKCUA.js → chunk-UPQMKAAX.js} +3 -18
  27. package/dist/chunk-UPQMKAAX.js.map +1 -0
  28. package/dist/chunk-XTOK7YOR.js +18 -0
  29. package/dist/chunk-XTOK7YOR.js.map +1 -0
  30. package/dist/{handler-B4NAi4k_.d.cts → handler-CoJxInw-.d.ts} +2 -1
  31. package/dist/{handler-BRVhWuNk.d.ts → handler-CsFbanZA.d.cts} +2 -1
  32. package/dist/index.cjs +26 -13
  33. package/dist/index.cjs.map +1 -1
  34. package/dist/index.d.cts +6 -4
  35. package/dist/index.d.ts +6 -4
  36. package/dist/index.js +30 -20
  37. package/dist/index.js.map +1 -1
  38. package/dist/migrations/index.cjs +173 -0
  39. package/dist/migrations/index.cjs.map +1 -0
  40. package/dist/migrations/index.d.cts +20 -0
  41. package/dist/migrations/index.d.ts +20 -0
  42. package/dist/migrations/index.js +23 -0
  43. package/dist/migrations/index.js.map +1 -0
  44. package/dist/query.schema-B-LVNfyX.d.cts +461 -0
  45. package/dist/query.schema-B-LVNfyX.d.ts +461 -0
  46. package/dist/{types-DUJp5B3o.d.cts → types-B2HEmyNq.d.cts} +11 -463
  47. package/dist/{types-DUJp5B3o.d.ts → types-DC68ZUuq.d.ts} +11 -463
  48. package/package.json +12 -1
  49. package/dist/chunk-3HORTMZ7.js.map +0 -1
  50. package/dist/chunk-J7JYKCUA.js.map +0 -1
package/CHANGELOG.md CHANGED
@@ -1,5 +1,32 @@
1
1
  # @niscorp/vex
2
2
 
3
+ ## 0.4.1
4
+
5
+ ### Patch Changes
6
+
7
+ - 6680ad6: `createQueryDsl` hands the query agent the schema and the DSL spec as compact text instead of JSON: an entity per line with its relations and indexes, and the DSL as types that keep every description, property and allowed value. On a 15-entity CRM schema the fixed part of each call drops from about 36,000 characters to about 13,000. On 19 queries a screen builder asked for, the agent answered all 19, with the same row counts as before, at 44% of the cost and a median of 3.5 s per query instead of 5.1 s. The query it writes is still validated against the full DSL schema.
8
+ - 6680ad6: A shape can declare a field as a Prism expression over the row's stored fields, and `createShapeMapper` computes it with no model call.
9
+
10
+ `[{ close_date: '', close_date_display: { $date: { value: { $ref: '$.close_date' }, format: 'MMM D' } } }]`: `createQueryDsl` asks the query for the stored fields a declared field reads, and `createShapeMapper` computes the declared field with Prism, so the stored value beside it is never overwritten; a mapping a model wrote overwrote it about a quarter of the time. `normalizeShape` keeps a declared field whole, so two formats are two cache entries.
11
+
12
+ A flat shape whose fields the first row carries, each plain one of the kind its example shows, is mapped with no model call, as rows that fit the shape exactly were; rows with more columns than the shape names are now mapped the same way rather than by the agent. Any other shape — nested, a field the rows lack, a value of another kind — goes to the mapping agent, whose mapping is run on the rows before it is used: one that fails there is written again, the agent told why, up to three attempts in all. The agent is told the rows already answer the request (filtered, sorted, limited) and keeps every row, in order.
13
+
14
+ - 1c78d6d: The engine hands `mapToShape` the request's intent, and `createShapeMapper` gives it to the mapping agent.
15
+
16
+ `MapToShape` is `(rows, shape, intent?)`: the engine passes the intent a query was asked in, so the mapping can give a value the form the request named (money as `$1.2M`, a date as `Apr 6`). Before, the mapper saw the rows and the shape alone, whose placeholders say nothing of a form. `createShapeMapper` passes it to prism's `mappingAgent` as its `notes`. A mapper that takes two arguments keeps working.
17
+
18
+ ## 0.4.0
19
+
20
+ ### Minor Changes
21
+
22
+ - 4068f98: Vex keeps what an entry it writes was written from, and says who asks when it looks one up. **Breaking.**
23
+
24
+ BREAKING — approved by moccadroid, 2026-10-14: a `mapToShape` an app writes answers `config`, the Prism config its `ir` was compiled from, beside `ir` and `transformed`. `createShapeMapper` does.
25
+ - **A query vex writes keeps its mapping's Prism config** (`mapping`, on the entry), beside the compiled form. A compiled form can neither be read back as a config nor upgraded when Prism changes.
26
+ - **The Postgres cache's table gets a `mapping` column**, in a new migration, `nisc.vex.cache/2`. A deployment that runs its migrations before it boots runs that one first.
27
+ - **`CacheBackend.get(key, scope?)`**: the two lookups that replay an entry hand the cache the scope of whoever asks, so a cache can answer an entry to some callers and not to others. A backend that takes no scope works as before.
28
+ - **`@niscorp/vex/migrations`** exports `VEX_SEQUENCE`, one stored kind, `nisc.vex/entry` (an entry as written, its `mapping` a `nisc.prism/config`), and `VEX_SCHEMAS`.
29
+
3
30
  ## 0.3.1
4
31
 
5
32
  ### Patch Changes
package/DESIGN.md CHANGED
@@ -266,7 +266,7 @@ Identity is the **fingerprint** (see Cache v2 below). The supporting hashes:
266
266
  impossible, so it isn't re-run until the TTL lapses or the schema changes).
267
267
  - **Backends.** In-memory (`createMemoryCache`), durable Postgres
268
268
  (`createPostgresCache`, jsonb table, server-side TTL, `protected`/
269
- `last_used_at`/`request_hash` columns; the table is a strata sequence,
269
+ `last_used_at`/`request_hash`/`mapping` columns; the table is a strata sequence,
270
270
  `nisc.vex.cache`, applied once through the ledger by `init()`;
271
271
  validate-on-write and evict-on-read), and tiered (`createTieredCache`, L1 +
272
272
  L2 read-through/write-through with `full`/`lazy`/`partial` warm-up). Every
@@ -587,8 +587,11 @@ The engine takes two optional hooks:
587
587
  what the person asking could see — and it is keyed (single-flight, negative
588
588
  cache) by request AND policy, because two callers who see different tables
589
589
  are asking different agents.
590
- - `mapToShape(rows, shape) → { ir, transformed }` — raw rows → requested shape,
591
- returning a compiled Prism IR that is cached for reuse.
590
+ - `mapToShape(rows, shape, intent?) → { ir, config, transformed }` — raw rows → requested
591
+ shape, returning a compiled Prism IR that is cached for reuse, and the Prism
592
+ config it was compiled from, which is kept beside it (`mapping`). `intent` is
593
+ the request's own words, when it has them, so the mapping can give a value the
594
+ form they ask for (money as `$1.2M`, a date as `Apr 6`).
592
595
 
593
596
  This is the seam that keeps the core deterministic. Vex ships a reference
594
597
  implementation of both, built on `@niscorp/cortex` and `@niscorp/prism` and
@@ -600,15 +603,16 @@ exported from the `@niscorp/vex/agent` subpath — `createQueryDsl` (fills
600
603
  `getDistinctValues`, `describeField`, `testQuery`, `cannotSatisfy` — and three
601
604
  stop conditions (`stepCount(20)`, `outputRetries(3)`, `repeatedCalls(2)`);
602
605
  `createQueryDsl` aborts the run when `cannotSatisfy` is called. Context
603
- producers inject the live schema, the caller line and the DSL JSON Schema so the
604
- static instructions stay stable. Every data-touching tool is a DSL run through
606
+ producers inject the live schema and the DSL specification as compact text, and
607
+ the caller line, so the static instructions stay stable. Every data-touching tool is a DSL run through
605
608
  `caller.read`; the tool set holds no adapter and no SQL.
606
609
  `createQueryDsl` wraps it into the `generateDsl` hook.
607
610
  - The **mapping agent** is Prism's exported `mappingAgent`; `createShapeMapper`
608
611
  builds the same envelope the runtime replays against — `{ result: rows }` for an
609
612
  array shape, `{ result: rows[0] ?? null }` for an object shape — runs the agent
610
- once to get a transform config, compiles it to IR, and applies the IR once over
611
- that envelope (the identity, with no model call, when the rows already fit).
613
+ once, with the intent as its `notes`, to get a transform config, compiles it to IR, and applies the IR once over
614
+ that envelope. A flat shape whose fields the rows carry is mapped by Prism with no model call, its declared
615
+ fields computed from the stored ones; the agent writes the mapping for any other.
612
616
 
613
617
  > The reference agents are a separate concern from the deterministic core: they
614
618
  > live behind the `@niscorp/vex/agent` subpath, so `@niscorp/cortex` (and
package/DOCS.md CHANGED
@@ -157,7 +157,7 @@ type QueryEngineConfig = {
157
157
  cache?: CacheBackend; // default: createMemoryCache()
158
158
  onEvent?: VexEventHandler; // pipeline event stream
159
159
  generateDsl?: (request, schema, caller) => Promise<Query>;
160
- mapToShape?: (rows, shape) => Promise<{ ir: CompiledIr; transformed: JsonValue }>;
160
+ mapToShape?: (rows, shape) => Promise<{ ir: CompiledIr; config: unknown; transformed: JsonValue }>;
161
161
  embed?: (text, dimensions?) => Promise<number[]>; // text→vector for semantic filters
162
162
  behaviors?: ScopeBehaviors; // which columns a scope key is bound to (see below)
163
163
  rows?: LiveRowsConfig; // the rows cache behind `refresh: 'reactive'`
@@ -266,6 +266,21 @@ The **shape** is an example, not a schema: use empty strings, zeros, and
266
266
  booleans as type markers. `[{ id: '', total: 0 }]` means "an array of objects
267
267
  with a string `id` and a number `total`." A bare object means a single record.
268
268
 
269
+ A field can be **declared** rather than shown by example: a Prism expression over
270
+ the row's own stored fields, where `$.close_date` is the row's `close_date`.
271
+
272
+ ```typescript
273
+ shape: [{ id: '', close_date: '', close_date_display: { $date: { value: { $ref: '$.close_date' }, format: 'MMM D' } } }]
274
+ // → [{ id: 'deal_085', close_date: '2026-06-08', close_date_display: 'Jun 8' }]
275
+ ```
276
+
277
+ The query is written for the stored fields a declared field reads, and
278
+ `createShapeMapper` computes the declared field with Prism, never a model, so
279
+ the stored value beside it is never overwritten. A declared field is part of the
280
+ request's identity: two formats are two cache entries. A field the rows lack,
281
+ such as a form shown only by example (`close_date_display: 'Apr 6'`), is left
282
+ to the mapping agent; a field the rows already carry is copied as it is.
283
+
269
284
  The shape also picks the result envelope: an **array** shape maps over the whole
270
285
  row set (the mapping sees `$.result` as the rows array — a `$map`/identity); a
271
286
  **non-array** shape maps the single (first) row (the mapping sees `$.result` as
@@ -1232,6 +1247,27 @@ demand; `partial` preloads only the listed fingerprints and lazy-fills the
1232
1247
  rest. The tiered cache is single-instance: separate processes keep separate
1233
1248
  L1s.
1234
1249
 
1250
+ **Who asks.** The engine hands the two lookups that replay an entry the scope
1251
+ of whoever asks (`get(key, scope)`), so a cache a host writes can answer an
1252
+ entry to some callers and not to others. The three backends here answer every
1253
+ caller alike.
1254
+
1255
+ **What a query vex wrote keeps.** The query, the compiled mapping (`prismIr`)
1256
+ and the Prism config it was compiled from (`mapping`), which `mapToShape`
1257
+ answers as `config`. A compiled form can neither be read back as a config nor
1258
+ upgraded when Prism changes, so a host that keeps such an entry for longer
1259
+ keeps the config. The Postgres table has a column for it from
1260
+ `nisc.vex.cache/2` on: a deployment that runs its migrations before it boots
1261
+ runs that one first.
1262
+
1263
+ **An entry as a stored document.** `@niscorp/vex/migrations` exports
1264
+ `VEX_SEQUENCE`, a strata sequence with one kind, `nisc.vex/entry`: an entry as
1265
+ written, `{ dsl, mapping?, shape?, intent?, reach?, refresh? }`, without the
1266
+ fingerprint it is kept under, its `mapping` a `nisc.prism/config`. A host that
1267
+ stores entries stamps them with it and upgrades them where it reads them, as it
1268
+ does nova's and Prism's documents (`VEX_SCHEMAS` is what the grammar check
1269
+ snapshots).
1270
+
1235
1271
  ### Seeding
1236
1272
 
1237
1273
  `seedCache(cache, entries)` turns authored entries into **protected** cache
@@ -1335,7 +1371,7 @@ free-form requests.
1335
1371
  ```typescript
1336
1372
  type GenerateDsl = (request: QueryRequest, schema: DatabaseSchema, caller: GenerationCaller) => Promise<Query>;
1337
1373
  type MapToShape = (rows: Row[], shape: unknown) =>
1338
- Promise<{ ir: CompiledIr; transformed: JsonValue }>;
1374
+ Promise<{ ir: CompiledIr; config: unknown; transformed: JsonValue }>;
1339
1375
 
1340
1376
  type GenerationCaller = {
1341
1377
  read: (dsl: Query) => Promise<{ rows: Row[]; sql: string; warnings: string[] }>;
@@ -1348,8 +1384,9 @@ type GenerationCaller = {
1348
1384
  `new VexError('unsatisfiable', reason)` — the runtime negative-caches it.
1349
1385
  - `mapToShape` runs when a shape is in play (the request's, or the stored one on
1350
1386
  a fingerprint-only replay) and the entry has no stored mapping; it returns a
1351
- compiled Prism `ir` plus the mapped result. The `ir` is stored with the entry
1352
- only on a miss — a replayed entry that has none calls the hook every time.
1387
+ compiled Prism `ir`, the Prism `config` it was compiled from, and the mapped
1388
+ result. Both are stored with the entry only on a miss — a replayed entry that
1389
+ has none calls the hook every time.
1353
1390
  Without the hook the raw rows are returned.
1354
1391
 
1355
1392
  You can implement these however you like. Vex also ships a **reference
@@ -1375,15 +1412,21 @@ await engine.introspect();
1375
1412
 
1376
1413
  - `createQueryDsl(config)` builds the `generateDsl` hook by running
1377
1414
  `vexQueryDslAgent`. It is built from a model and the DSL spec only — who it
1378
- runs for arrives with every call. The engine calls the hook as
1415
+ runs for arrives with every call. The agent reads the schema and the DSL spec
1416
+ as compact text — an entity per line with its relations and indexes, the DSL
1417
+ as types with every description — and, for a shape with declared fields, the
1418
+ stored fields those read. The engine calls the hook as
1379
1419
  `generateDsl(request, schema, caller)`: `schema` holds only the tables the
1380
1420
  caller's policy can read, and `caller.read(dsl)` runs a DSL through the
1381
1421
  engine's own pipeline under the caller's policy and scope values (`$context`
1382
1422
  binds NULL). That capability is the tools' only way to the database.
1383
- - `createShapeMapper(llm)` builds the `mapToShape` hook by running Prism's
1384
- `mappingAgent`. When the rows already are the shape (`rowsFitShape` — a flat
1385
- shape whose keys and value kinds the first row matches exactly) it stores the
1386
- identity mapping and makes no model call. The `@niscorp/vex/agent` subpath
1423
+ - `createShapeMapper(llm)` builds the `mapToShape` hook. A flat shape whose
1424
+ fields the first row carries, each plain one of the kind its example shows —
1425
+ plain fields by key, declared fields computed from the stored fields they
1426
+ read — is mapped by Prism with no model call. Any other shape (nested, a field
1427
+ the rows lack, a value of another kind) goes to Prism's `mappingAgent`, handed the request's intent and run on the rows
1428
+ before it is used: a mapping that fails there is written again, up to three
1429
+ attempts. `rowsFitShape` is still exported. The `@niscorp/vex/agent` subpath
1387
1430
  also exports `vexQueryDslAgent` itself, `createQueryTools` for customizing the
1388
1431
  tool set, and `describeCaller`, which turns `caller.bindings` into the
1389
1432
  context line the agent reads. A run of `vexQueryDslAgent` given no `caller`
@@ -1405,7 +1405,7 @@ var runQuery = async (config, body, scope, probe, live) => {
1405
1405
  try {
1406
1406
  let entryReach;
1407
1407
  if (parsed.data.fingerprint !== void 0) {
1408
- const entry = await engine.cache.get(parsed.data.fingerprint);
1408
+ const entry = await engine.cache.get(parsed.data.fingerprint, scope);
1409
1409
  if (entry !== void 0 && entry.kind === "ok") entryReach = entry.reach;
1410
1410
  if (probe !== void 0 && entryReach !== void 0) probe.reach = entryReach;
1411
1411
  if (entry !== void 0 && entry.kind === "mutation") {