@zenera/rag 1.1.3 → 1.1.5

package/README.md

@@ -46,9 +46,27 @@ zen rag schema stats                  # what is in the index, and what built it
 zen rag schema show Type:Invoice      # a named node, with no search in between
 ```
 
+Search ranks, which means it returns the top of a list — useful when the
+question is vague, and no use at all when the question is whether something
+exists. For that there is exact matching, which needs no embedder, no
+credential and no network:
+
+```sh
+zen rag schema list methods --path "*/users*"   # every route under /users
+zen rag schema list types --name "*Password*"   # every schema so named
+zen rag schema grep password                    # every literal occurrence
+zen rag schema grep "pass(word|phrase)" --regex
+zen rag schema show --method GetCurrentUserInfo --format openapi --exact
+```
+
+`list` and `grep` report `found` as the true total even when `--limit` cuts the
+printed rows, so a shortened answer still tells you how much there is. Nothing
+matching exits 0 — an empty answer is an answer, and here it is a trustworthy
+one: if `grep` finds nothing, the word is not in the description.
+
 Non-interactive search is a machine interface: every field is a flag, the whole
-query can arrive as one JSON object, `--json` is a stable shape, no terminal is
-required, and an empty result exits 0.
+query can arrive as one JSON object, the `--json` output keeps the same
+structure from run to run, no terminal is required, and an empty result exits 0.
 
 ```sh
 zen rag schema search --query - --format ts <<'JSON'
@@ -66,7 +84,9 @@ JSON
 ```
 zen rag schema index <spec...>   Read the documents and write a searchable index.
 zen rag schema search            Ask it something. --interactive for a prompt.
-zen rag schema show <id...>      Print named nodes, with no search in between.
+zen rag schema list <what>       Every method, type or property. No ranking.
+zen rag schema grep <pattern>    Every literal match across the whole index.
+zen rag schema show [id...]      Print named nodes, with no search in between.
 zen rag schema stats             What is in an index, and what built it.
 ```
 
@@ -76,6 +96,18 @@ by `--direction`, `--method-type`, `--limit`, `--max-hops`, `--max-nodes` and
 the four `--exclude-*` filters, and rendered by `--format text | mermaid |
 mermaid-flowchart | ts | openapi`. `zen help rag` prints the full table.
 
+`list` takes `--name` and `--path`, `grep` takes `--regex`, `--case-sensitive`,
+`--kind` and `--ids-only`. A pattern with `*` or `?` in it is a glob matched
+against the whole name; a plain word is a substring, so `--name password` finds
+`ResetPasswordPayload` rather than nothing. `show` takes ids, or `--method` and
+`--type` by name, or `--source` for a whole document, and `--exact` to print
+only what was named instead of its neighbourhood.
+
+```sh
+# Everything that mentions a token, rendered as TypeScript.
+zen rag schema grep token --ids-only | xargs zen rag schema show --format ts
+```
+
 ## From an agent
 
 ```ts
@@ -89,33 +121,60 @@ const index = await SchemaIndex.open(
 const project = await loadProject('./my-project', { tools: schemaTools(index) });
 ```
 
-Four tools in the group `schema`, selectable as `schema:*`:
+Five tools in the group `schema`, selectable as `schema:*`:
 
 | Tool                       | For                                                     |
 | -------------------------- | ------------------------------------------------------- |
 | `search_api`               | the connected piece of the API that matches an intent   |
 | `describe_types`           | named schemas as declarations that compile on their own |
 | `find_types_with_property` | which types have a field of this name — no search       |
-| `list_methods`             | the shape of the API, by path                           |
+| `list_api`                 | the shape of the API: methods, types or fields          |
+| `grep_api`                 | every literal occurrence of a string — no search        |
 
-`find_types_with_property` is the one for the repair loop: when `tsc` says
+Only the first of those ranks anything. The rest are exact, because a model
+told "no results" by a vector search has learned nothing: a ranking returns the
+top of a list, so an empty answer and an absent thing look identical.
+`find_types_with_property` is the one for the repair loop — when `tsc` says
 `'password' does not exist in type 'PublicUserProfile'`, the model does not
 need the word explained again, it needs the list of types that have one.
+`grep_api` is the same instinct widened to the whole description.
 
 ## What an index is
 
 ```
 schema-db/
+├── README.md         what this index holds — a live progress report while it builds
 ├── manifest.json     written last — its absence means "not indexed"
 ├── graph.json        topology and light attributes, read whole
-├── schemas.json      the raw schemas, read on first hydrate
+├── schemas.json      the raw schemas, read the first time one is needed in full
 ├── operations.json   likewise, for the OpenAPI subset
+├── sources/          the documents themselves, bundled, exactly as indexed
 └── lance/            one table: a row per node, one text column, one vector
 ```
 
 The manifest records which embedder made the vectors, and a search with a
 different one is refused rather than answered with noise.
 
+Indexing a large document is minutes of silence, so the directory says what is
+happening to it. `README.md` appears first as a progress report — the documents,
+the embedder, the step, how many entities have been embedded of how many, and
+how long it has been going — rewritten at most every five seconds, and replaced
+on completion by a description of what the index turned out to hold. A build
+that dies leaves it saying so. While one runs, `.lock` names the process; a
+second build of the same directory is refused, unless the lock is stale.
+
+**An index is one portable thing.** Nothing in it names a path outside itself:
+a document is known by a short name taken from its filename, and the document
+itself is copied into `sources/` as it was indexed — bundled, so every external
+`$ref` is already resolved and the copy stands alone. The directory can be moved,
+committed or shipped whole and still says what it is made of. This matters
+because a project's `assets/` is mounted at `/assets` inside an agent's sandbox,
+so an index built here is read under a name this machine never sees. `--no-sources`
+leaves the copies out, for an index that will never travel.
+
+The copies are a record, not an input: rebuilding reads the files you name, not
+the ones in `sources/`.
+
 ## Notes
 
 - Documents are **bundled, not dereferenced**: `#/components/schemas/User` is
@@ -125,8 +184,8 @@ different one is refused rather than answered with noise.
 - A **query parameter is a property**, like any field in a body. Nobody should
   have to know in advance which one `page_size` is.
 - Every type carries a **direction** — `input`, `output` or `both` — worked out
-  by propagating from the operations through composition, so a shared DTO is
-  honestly both rather than whichever side was read last.
+  by propagating from the operations through composition, so a type used on both
+  sides is honestly both rather than whichever side was read last.
 - Filters reaching the store are **closed enums only**. Exclusion lists are
   applied in JavaScript afterwards, so nothing a model wrote ever reaches a SQL
   predicate.

package/dist/command.js

@@ -1,13 +1,15 @@
-import { bold, CliError, cyan, dim, ensureHome, envNames, EXIT, form, isInteractive, json, KeyStore, note, parse, PROVIDERS, table, usageError, write, } from '@zenera/cli/lib';
+import { bold, CliError, CURATED, cyan, dim, ensureHome, envNames, EXIT, form, isInteractive, json, KeyStore, note, parse, PROVIDERS, table, usageError, write, } from '@zenera/cli/lib';
 import { createEmbedder } from '@zenera/neo';
 import { relative, resolve } from 'node:path';
 import { isFormat, present } from "./present.js";
 import { isEmpty, parseQuery, QueryError } from "./query.js";
 import { repl } from "./repl.js";
 import { buildIndex } from "./schema/build.js";
-import { assertSameEmbedding, openIndex, readManifest } from "./schema/files.js";
+import { assertSameEmbedding, openIndex, readManifest, readSource, } from "./schema/files.js";
+import { fields, grepNodes, listNodes, propertyCount } from "./schema/lookup.js";
+import { isGlob, loose, matcher, PatternError, wildcard } from "./schema/match.js";
 import { SchemaIndex } from "./schema/search.js";
-import { stitch } from "./schema/subgraph.js";
+import { select, stitch } from "./schema/subgraph.js";
 // ---------------------------------------------------------------------------
 // zen rag — an api description, as something to search
 //
@@ -18,10 +20,19 @@ import { stitch } from "./schema/subgraph.js";
 // required, and exit 0 when nothing matched — an empty answer is an answer, and
 // a caller that has to tell "no results" from "the index is missing" by parsing
 // stderr will get it wrong.
+//
+// `list`, `grep` and `show` are the other half, and they are deliberately not
+// searches. A ranking can only ever hand back the top of a list, so it cannot
+// answer "is there a field called `password` anywhere" — the honest answer to
+// that question is every match or none, and these three give it without asking
+// a model or a credential for permission.
 // ---------------------------------------------------------------------------
-const USAGE = 'zen rag schema <index|search|show|stats> [spec...]';
+const USAGE = 'zen rag schema <index|search|list|grep|show|stats> [spec...]';
 const INDEX_USAGE = 'zen rag schema index --embedding <ref> [--out <dir>] <spec...>';
 const SEARCH_USAGE = 'zen rag schema search [--dir <dir>] [query...]';
+const LIST_USAGE = 'zen rag schema list <methods|types|properties> [--dir <dir>]';
+const GREP_USAGE = 'zen rag schema grep <pattern> [--dir <dir>]';
+const SHOW_USAGE = 'zen rag schema show [id...] [--method <name>] [--type <name>]';
 const DEFAULT_DIR = './schema-db';
 export const command = {
     summary: 'Search an openapi/swagger document as a graph.',
@@ -31,7 +42,9 @@ export const command = {
         ...table([
             ['  index <spec...>', dim('Read the documents and write a searchable index.')],
             ['  search', dim('Ask it something. --interactive for a prompt.')],
-            ['  show <id...>', dim('Print named nodes, with no search in between.')],
+            ['  list <what>', dim('Every method, type or property matching a pattern.')],
+            ['  grep <pattern>', dim('Every literal match, ranked by nothing.')],
+            ['  show [id...]', dim('Print named nodes, with no search in between.')],
             ['  stats', dim('What is in an index, and what built it.')],
         ]),
         '',
@@ -46,6 +59,7 @@ export const command = {
                 '  --batch <n>',
                 dim('Texts per embedding request, and how often progress prints. Default 96.'),
             ],
+            ['  --no-sources', dim('Do not keep a copy of each document in the index.')],
         ]),
         '',
         'Search terms (repeatable)',
@@ -81,6 +95,31 @@ export const command = {
             ['  --quiet', dim('No narration.')],
         ]),
         '',
+        'Exact listing — no embedder, no credential',
+        ...table([
+            ['  list methods', dim('Operations. Filter with --path and --name.')],
+            ['  list types', dim('Schemas. Filter with --name.')],
+            ['  list properties', dim('Fields and parameters. Filter with --name.')],
+            ['  grep <pattern>', dim('Substring over every node; --regex for a regex.')],
+            ['  --case-sensitive', dim('grep: match the capitals too.')],
+            ['  --kind <k>', dim('grep: method | type | property. Repeatable.')],
+            ['  --ids-only', dim('grep: bare ids, to pipe into show.')],
+            ['  --source <name>', dim('Only this document, as `stats` names it.')],
+            ['  --limit <n>', dim('Keep at most n; the count still reports them all.')],
+        ]),
+        '',
+        dim('  A pattern with * or ? is a glob over the whole name; otherwise it is'),
+        dim('  a substring, so --name password finds ResetPasswordPayload.'),
+        '',
+        'Show',
+        ...table([
+            ['  <id...>', dim('Node ids, e.g. Type:User or Property:User.email.')],
+            ['  --method <name>', dim('An operation by name. * to take more. Repeatable.')],
+            ['  --type <name>', dim('A schema by name. * to take more. Repeatable.')],
+            ['  --source <name>', dim('A whole document, as it was indexed.')],
+            ['  --exact', dim('Only what was named, without the neighbours.')],
+        ]),
+        '',
         dim(`Credentials come from the ${cyan('zen')} keyring — try ${cyan('zen key ls')}.`),
     ],
     async run(ctx) {
@@ -93,6 +132,10 @@ export const command = {
                 return await index(tail, ctx);
             case 'search':
                 return await search(tail, ctx);
+            case 'list':
+                return await list(tail, ctx);
+            case 'grep':
+                return await grep(tail, ctx);
             case 'show':
                 return await show(tail, ctx);
             case 'stats':
@@ -107,6 +150,7 @@ async function index(args, ctx) {
         out: { type: 'string', short: 'o' },
         embedding: { type: 'string' },
         batch: { type: 'string' },
+        'no-sources': { type: 'boolean' },
         quiet: { type: 'boolean' },
     }, INDEX_USAGE);
     if (positionals.length === 0) {
@@ -123,9 +167,10 @@ async function index(args, ctx) {
         embeddingRef: values.embedding,
         indexer: 'zenera-rag',
         batch: values.batch ? count(values.batch, '--batch') : undefined,
+        sources: !values['no-sources'],
         onRead: loud
             ? (summary) => {
-                printSources(summary.sources, ctx.cwd);
+                printSources(summary.sources);
                 // The first batch can take a while and says nothing while it
                 // does; this is the line that makes that a wait, not a hang.
                 note(dim(`  embedding ${summary.counts.entities} entities with ${chosen.id} …`));
@@ -153,9 +198,9 @@ function elapsed(since) {
     const seconds = Math.round((Date.now() - since) / 1000);
     return seconds < 60 ? `${seconds}s` : `${Math.floor(seconds / 60)}m${seconds % 60}s`;
 }
-function printSources(sources, cwd) {
+function printSources(sources) {
     const rows = sources.map((s) => ({
-        name: relative(cwd, s.path) || s.path,
+        name: s.file,
         dialect: s.dialect,
         cells: [s.paths, s.methods, s.types, s.properties],
     }));
@@ -314,33 +359,248 @@ function check(value) {
     }
 }
 // ---------------------------------------------------------------------------
-// show, stats
+// list, grep
+//
+// The deterministic half. Neither takes an embedder, because neither ranks
+// anything: `list` filters on the attributes a node already has and `grep`
+// reads the same materialized string the index was built from. What comes back
+// is every match, and where a limit cut the list the count still reports the
+// total — being shown three of three hundred is only useful if you are told
+// which of the two happened.
+// ---------------------------------------------------------------------------
+const SUBJECTS = {
+    methods: 'method',
+    types: 'type',
+    properties: 'property',
+};
+async function list(args, ctx) {
+    const { values, positionals } = parse(args, {
+        dir: { type: 'string', short: 'd' },
+        name: MANY,
+        path: MANY,
+        source: { type: 'string' },
+        'method-type': { type: 'string' },
+        direction: { type: 'string' },
+        limit: { type: 'string' },
+        quiet: { type: 'boolean' },
+    }, LIST_USAGE);
+    const subject = positionals[0];
+    const kind = subject ? SUBJECTS[subject] : undefined;
+    if (!kind) {
+        throw usageError(subject ? `cannot list "${subject}"` : 'nothing named to list', `expected one of ${Object.keys(SUBJECTS).join(', ')}`);
+    }
+    if (positionals.length > 1) {
+        throw usageError('one subject at a time', LIST_USAGE);
+    }
+    const index = await openIndex(resolve(ctx.cwd, values.dir ?? DEFAULT_DIR));
+    const found = listNodes(index.graph, {
+        kind,
+        name: globs(values.name, '--name'),
+        path: globs(values.path, '--path'),
+        source: values.source,
+        methodType: oneOf(values['method-type'], ['read_only', 'read_write'], '--method-type'),
+        direction: oneOf(values.direction, ['input', 'output'], '--direction'),
+        limit: values.limit ? count(values.limit, '--limit') : undefined,
+    });
+    if (ctx.json) {
+        json({ found: found.found, truncated: found.truncated, rows: found.rows });
+        return;
+    }
+    const lines = rowLines(index.graph, kind, found.rows);
+    if (lines.length > 0) {
+        write(lines.join('\n'));
+    }
+    if (!values.quiet) {
+        note(dim(`  ${found.found} ${subject}${shown(found.found, found.rows.length)}`));
+    }
+}
+function rowLines(graph, kind, rows) {
+    if (kind === 'method') {
+        return table(rows.map((r) => [`${r.httpMethod} ${r.path}`, r.name, doc(r.doc)]));
+    }
+    if (kind === 'type') {
+        return table(rows.map((r) => [
+            r.name,
+            dim(fields(propertyCount(graph, r.id))),
+            dim(r.direction === 'none' ? '' : `(${r.direction})`),
+            doc(r.doc),
+        ]));
+    }
+    return table(rows.map((r) => [
+        `${r.parent ? `${r.parent}.` : ''}${r.name}${r.required ? '' : '?'}`,
+        `: ${r.signature || 'unknown'}`,
+        doc(r.doc),
+    ]));
+}
+async function grep(args, ctx) {
+    const { values, positionals } = parse(args, {
+        dir: { type: 'string', short: 'd' },
+        regex: { type: 'boolean' },
+        'case-sensitive': { type: 'boolean' },
+        kind: MANY,
+        source: { type: 'string' },
+        limit: { type: 'string' },
+        'ids-only': { type: 'boolean' },
+        quiet: { type: 'boolean' },
+    }, GREP_USAGE);
+    if (positionals.length === 0) {
+        throw usageError('no pattern given', GREP_USAGE);
+    }
+    if (positionals.length > 1) {
+        throw usageError('one pattern at a time — quote it if it has spaces', GREP_USAGE);
+    }
+    const kinds = (values.kind ?? []).map((k) => oneOf(k, ['method', 'type', 'property'], '--kind'));
+    const index = await openIndex(resolve(ctx.cwd, values.dir ?? DEFAULT_DIR));
+    const result = pattern(() => grepNodes(index.graph, matcher(positionals[0], {
+        regex: values.regex,
+        caseSensitive: values['case-sensitive'],
+    }), {
+        kinds,
+        source: values.source,
+        limit: values.limit ? count(values.limit, '--limit') : undefined,
+    }));
+    if (ctx.json) {
+        json({
+            found: result.found,
+            truncated: result.truncated,
+            matches: result.matches.map((m) => ({ id: m.id, ...m.attributes, text: m.text })),
+        });
+        return;
+    }
+    if (result.matches.length > 0) {
+        const lines = values['ids-only']
+            ? result.matches.map((m) => m.id)
+            : table(result.matches.map((m) => [m.id, dim(clip(m.text, 140))]));
+        write(lines.join('\n'));
+    }
+    if (!values.quiet && !values['ids-only']) {
+        note(dim(`  ${result.found} match(es)${shown(result.found, result.matches.length)}`));
+    }
+}
 // ---------------------------------------------------------------------------
+const shown = (found, kept) => (kept < found ? `, showing ${kept}` : '');
+function globs(patterns, flag) {
+    if (!patterns || patterns.length === 0) {
+        return undefined;
+    }
+    return patterns.map((p) => pattern(() => loose(p), flag));
+}
+/** A bad pattern is a bad invocation, not a failure of the index. */
+function pattern(run, flag) {
+    try {
+        return run();
+    }
+    catch (err) {
+        if (err instanceof PatternError) {
+            throw usageError(`${flag ? `${flag}: ` : ''}${err.message}`, USAGE);
+        }
+        throw err;
+    }
+}
+function oneOf(value, allowed, flag) {
+    if (value === undefined || value === 'any') {
+        return undefined;
+    }
+    if (!allowed.includes(value)) {
+        throw usageError(`${flag} cannot be "${value}"`, `expected ${allowed.join(' or ')}`);
+    }
+    return value;
+}
+const doc = (text) => (text ? dim(`— ${clip(text.replace(/\s+/g, ' '), 90)}`) : '');
+const clip = (text, max) => text.length <= max ? text : `${text.slice(0, max - 1)}…`;
 /**
  * No embedder and no store: naming a node is a graph lookup, and asking for a
  * credential to print something already on disk would be theatre.
+ *
+ * Ids are the precise way in, and `--method`/`--type` are the way in for
+ * someone who has a name rather than an id — which, with `--format openapi
+ * --exact`, is how a resolved slice of the document is got out.
  */
 async function show(args, ctx) {
-    const usage = 'zen rag schema show <id...>';
-    const { values, positionals } = parse(args, SEARCH_OPTIONS, usage);
-    if (positionals.length === 0) {
-        throw usageError('no node named', usage);
-    }
+    const { values, positionals } = parse(args, {
+        dir: { type: 'string', short: 'd' },
+        method: MANY,
+        type: MANY,
+        source: { type: 'string' },
+        exact: { type: 'boolean' },
+        format: { type: 'string' },
+        'max-nodes': { type: 'string' },
+        'no-docs': { type: 'boolean' },
+        quiet: { type: 'boolean' },
+    }, SHOW_USAGE);
     const format = formatOf(values.format);
-    const index = await openIndex(resolve(ctx.cwd, values.dir ?? DEFAULT_DIR));
-    const missing = positionals.filter((id) => !index.graph.hasNode(id));
-    if (missing.length > 0) {
-        throw new CliError(`no such node: ${missing.join(', ')}`, EXIT.failed, 'ids look like `Type:User` or `Property:User.email`');
+    const dir = resolve(ctx.cwd, values.dir ?? DEFAULT_DIR);
+    // A whole document, verbatim: the copy kept at index time is the resolved
+    // original, and anything rebuilt from the graph would be a paraphrase.
+    if (values.source && format === 'openapi' && positionals.length === 0 && !named(values)) {
+        const document = await readSource(dir, values.source);
+        if (document) {
+            write(document);
+            return;
+        }
+        if (!values.quiet) {
+            note(dim('  this index kept no copy of the documents — rebuilding it from the graph'));
+        }
     }
-    const subgraphs = stitch(index.graph, positionals.map((id) => ({ id, term: id, field: 'show', score: 1 })), { maxNodes: values['max-nodes'] ? count(values['max-nodes'], '--max-nodes') : undefined });
+    const index = await openIndex(dir);
+    const ids = resolveIds(index.graph, positionals, values);
+    const subgraphs = values.exact
+        ? [select(index.graph, ids)]
+        : stitch(index.graph, ids.map((id) => ({ id, term: id, field: 'show', score: 1 })), {
+            maxNodes: values['max-nodes']
+                ? count(values['max-nodes'], '--max-nodes')
+                : undefined,
+        });
     const text = await present(index, subgraphs, format, { docs: !values['no-docs'] });
     if (ctx.json) {
-        json({ subgraphs, rendered: text });
+        json({ ids, subgraphs, rendered: text });
     }
     else if (text) {
         write(text);
     }
 }
+const named = (values) => Boolean(values.method?.length || values.type?.length);
+/** Ids as given, plus whatever the name selectors resolve to. */
+function resolveIds(graph, ids, values) {
+    if (ids.length === 0 && !named(values) && !values.source) {
+        throw usageError('no node named', SHOW_USAGE);
+    }
+    const missing = ids.filter((id) => !graph.hasNode(id));
+    if (missing.length > 0) {
+        throw new CliError(`no such node: ${missing.join(', ')}`, EXIT.failed, 'ids look like `Type:User` or `Property:User.email`');
+    }
+    const out = new Set(ids);
+    for (const kind of ['method', 'type']) {
+        for (const wanted of values[kind] ?? []) {
+            // Selecting, not searching: a bare name means that name. A star is
+            // the way to ask for more than one.
+            const match = isGlob(wanted)
+                ? pattern(() => wildcard(wanted), `--${kind}`)
+                : (name) => name === wanted;
+            const rows = listNodes(graph, { kind, name: [match], source: values.source });
+            // A selector that matched nothing is a wrong answer, not an empty
+            // one: the caller named something they believe is there.
+            if (rows.found === 0) {
+                throw new CliError(`no ${kind} called ${wanted}`, EXIT.failed, `try: zen rag schema list ${kind}s --name "${wanted}"`);
+            }
+            for (const row of rows.rows) {
+                out.add(row.id);
+            }
+        }
+    }
+    // `--source` on its own means the whole document.
+    if (out.size === 0 && values.source) {
+        for (const kind of ['method', 'type']) {
+            for (const row of listNodes(graph, { kind, source: values.source }).rows) {
+                out.add(row.id);
+            }
+        }
+        if (out.size === 0) {
+            throw new CliError(`nothing in this index came from ${values.source}`, EXIT.failed);
+        }
+    }
+    return [...out];
+}
 async function stats(args, ctx) {
     const { values } = parse(args, { dir: { type: 'string', short: 'd' } }, 'zen rag schema stats [--dir <dir>]');
     const dir = resolve(ctx.cwd, values.dir ?? DEFAULT_DIR);
@@ -359,7 +619,7 @@ async function stats(args, ctx) {
             `fts ${yes(manifest.indexes.fts)} · vector ${yes(manifest.indexes.vector)}`,
         ],
     ]));
-    printSources(manifest.sources, ctx.cwd);
+    printSources(manifest.sources);
     notes(table([['  entities', String(manifest.counts.entities)]]).map(dim));
 }
 function notes(lines) {
@@ -383,22 +643,18 @@ async function embedder(ref) {
     return createEmbedder(ref);
 }
 /**
- * Well-known embedding models per provider. A list rather than a lookup: any
- * ref the registry can parse works, and these are the ones worth typing.
- * Anthropic is absent because it publishes no embeddings API at all.
+ * Well-known embedding models per provider, read off the CLI's catalog table so
+ * there is one list rather than two that drift. Any ref the registry can parse
+ * works; these are the ones worth typing. Anthropic has none because it
+ * publishes no embeddings API at all.
  */
-const EMBEDDINGS = {
-    openai: ['text-embedding-3-small', 'text-embedding-3-large'],
-    google: ['gemini-embedding-001'],
-    vertex: ['gemini-embedding-001', 'text-embedding-005'],
-    openrouter: ['openai/text-embedding-3-small'],
-};
+const embeddingsOf = (provider) => CURATED[provider].filter((m) => m.roles.includes('embedding')).map((m) => m.id);
 /** What could be passed, with the ones this machine can actually use first. */
 function choices(keys, fromEnv) {
     const rows = [];
     const rest = [];
     for (const provider of PROVIDERS) {
-        for (const model of EMBEDDINGS[provider] ?? []) {
+        for (const model of embeddingsOf(provider)) {
             const source = fromEnv.has(provider)
                 ? 'environment'
                 : keys.active(provider)
@@ -415,7 +671,9 @@ function choices(keys, fromEnv) {
         note(dim('  no provider on this machine has a credential — try: zen key add openai'));
         note('');
     }
-    return usageError('no embedder named', 'pass --embedding <ref>, one of the above');
+    // `pick` is the one that ends the question rather than restating it: it
+    // tries them and prints the first that answers.
+    return usageError('no embedder named', 'pass --embedding <ref>, or run: zen models pick --embedding');
 }
 function formatOf(value) {
     if (value === undefined) {

package/dist/index.d.ts

@@ -5,6 +5,8 @@ export * from './schema/entities.ts';
 export * from './schema/files.ts';
 export * from './schema/graph.ts';
 export * from './schema/hydrate.ts';
+export * from './schema/lookup.ts';
+export * from './schema/match.ts';
 export * from './schema/render.ts';
 export * from './schema/schema.ts';
 export * from './schema/search.ts';

package/dist/index.js

@@ -5,6 +5,8 @@ export * from "./schema/entities.js";
 export * from "./schema/files.js";
 export * from "./schema/graph.js";
 export * from "./schema/hydrate.js";
+export * from "./schema/lookup.js";
+export * from "./schema/match.js";
 export * from "./schema/render.js";
 export * from "./schema/schema.js";
 export * from "./schema/search.js";

package/dist/schema/build.d.ts

@@ -11,6 +11,8 @@ export interface BuildOptions {
     indexer: string;
     /** texts sent to the embedder at once */
     batch?: number;
+    /** keep a bundled copy of each document in the index. On by default. */
+    sources?: boolean;
     signal?: AbortSignal;
     /** what the documents turned out to hold, before a vector has been paid for */
     onRead?: (summary: BuildSummary) => void;