@zenera/rag 1.1.9 → 1.1.11

package/README.md

@@ -1,7 +1,8 @@
 # @zenera/rag
 
-**An OpenAPI description, indexed as a graph and searched by meaning — for
-agents that have to call an API they have not read.**
+**A corpus, indexed and searched by meaning — an OpenAPI description as a
+graph, a pile of markdown as quotable passages — for agents that have to work
+with something they have not read.**
 
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/andreyryabov/ZeneraNeo/blob/main/LICENSE)
 [![Node](https://img.shields.io/badge/node-%E2%89%A524-brightgreen.svg)](https://nodejs.org)
@@ -11,7 +12,23 @@ agents that have to call an API they have not read.**
 > [`zen`](https://github.com/andreyryabov/ZeneraNeo/blob/main/packages/cli/README.md),
 > which is also where the credentials already are.
 
-## Why
+## Two subjects
+
+A subject is a kind of corpus with its own index format, its own verbs and its
+own flags — not a variation on one command, because what `list` means to an API
+description is not what it means to a folder of notes.
+
+| Subject          | The corpus                | The answer                                     |
+| ---------------- | ------------------------- | ---------------------------------------------- |
+| `zen rag schema` | openapi/swagger documents | the connected piece of the API that matched    |
+| `zen rag docs`   | markdown and plain text   | the passages that matched, quoted with numbers |
+
+Both are built the same way — [LanceDB](https://lancedb.com) for hybrid vector
+
+- full-text retrieval, a manifest that records which embedder made the vectors,
+  and exact commands beside the ranking ones that need no credential at all.
+
+## Schema — an API description as a graph
 
 A large specification does not fit in a prompt, and the parts of it that answer
 a question are scattered: the field is on a schema, the schema is on a request
@@ -117,11 +134,75 @@ zen rag schema search --query - --format ts <<'JSON'
 JSON
 ```
 
+## Docs — markdown as quotable passages
+
+The other subject. Point it at files, directories or globs; `.md`, `.markdown`,
+`.txt` and `.text` are read, hidden directories and `node_modules` are not.
+
+```sh
+zen rag docs index --embedding openai:text-embedding-3-small ./docs
+zen rag docs search "how are rate limits counted"
+```
+
+The answer is the documents themselves — the passages that matched, quoted
+verbatim with their line numbers, and a marker wherever something between two
+of them was left out:
+
+```
+## nsx_4.2.0/api/routing.md — 9 of 148 lines
+
+  5 | ## Rate limits
+  7 | Requests are counted per tenant and rejected past the limit.
+... 12 lines omitted (Retries, Backoff) ...
+ 24 | | route | limit | window |
+ 25 | | --- | --- | --- |
+ 27 | | /api/users | 250 | 1m |
+```
+
+Nobody finds the paragraph they want on the first ask, so **narrowing is the
+interface**, not an afterthought. The second call is the same question inside
+one part of the tree:
+
+```sh
+zen rag docs search --file "nsx_4.2.*/api/**" "rate limit for the users route"
+zen rag docs search --section "Rate limits" --kind table "requests per minute"
+zen rag docs search --mode text "X-RateLimit-Remaining"   # exact wording only
+zen rag docs search --interactive                          # narrow by typing
+```
+
+`--file` is a glob when it has `*` or `?` and a substring otherwise, matched
+against the document's **name**, which is its path relative to the common root
+of everything indexed. That is what keeps
+two releases of the same file apart. `--section` takes a heading title, and
+covers whatever nests inside it. `--kind` takes `paragraph`, `list`, `table`,
+`table_row`, `code`, `frontmatter` or `html`, for when the answer is a table
+and not the prose around it. `-B/-A` widen each passage, `--max-lines` caps the
+whole answer, `--exclude-id` moves on from what was already seen.
+
+Tables are indexed twice over: once as a descriptor carrying the caption and
+the column names, and once per row, with the header row travelling alongside so
+the columns are still named wherever a row lands. A row too wide to be one
+chunk is cut into column groups, with the key column repeated in each.
+
+And beside all that, the exact half — no embedder, no credential, no network:
+
+```sh
+zen rag docs list files                      # every document, and what it holds
+zen rag docs list sections --file "api/**"   # every heading, with its line span
+zen rag docs list tables                     # every table, with its columns
+zen rag docs grep "Retry-After"              # every matching line, and its section
+zen rag docs show api/routing.md --section "Rate limits"
+zen rag docs show api/routing.md --lines 40-80
+```
+
+`grep` reports `found` as the true total even when `--limit` cuts the rows, so
+unlike a search it can answer whether a string appears at all.
+
 ## Which index
 
-Every reading command takes `-d, --dir`. Without one, `$ZEN_SCHEMA_DB` is used
-if it is set; without that, the nearest index to the working directory is found
-and named on stderr as it is used.
+Every reading command takes `-d, --dir`. Without one, `$ZEN_SCHEMA_DB` or
+`$ZEN_DOCS_DB` is used if it is set; without that, the nearest index to the
+working directory is found and named on stderr as it is used.
 
 Nearest means what it says: this directory, then a short way down into it, then
 up a level and again, stopping at your home directory. What is looked for is a
@@ -131,7 +212,10 @@ the same. `schema-db` is only the name a new one is given.
 
 Two indexes the same distance away is a question, not a tie to break, and it is
 refused: the wrong index does not fail, it answers confidently about a
-different API. Name one with `--dir`, or set `ZEN_SCHEMA_DB`.
+different API. Name one with `--dir`, or set the environment variable.
+
+The search is scoped by kind, so a `docs` index and a `schema` index can sit in
+the same tree without either shadowing the other.
 
 ## Commands
 
@@ -147,8 +231,8 @@ zen rag schema stats             What is in an index, and what built it.
 
 Search terms are one flag each — `--all`, `--method`, `--type`, `--input-type`,
 `--output-type`, `--property`, `--input-property`, `--output-property` — shaped
-by `--direction`, `--method-type`, `--limit`, `--max-hops`, `--max-nodes` and
-the four `--exclude-*` filters, and rendered by `--format text | mermaid |
+by `--direction`, `--method-type`, `--limit`, `--max-hops`, `--max-nodes`,
+`--source` and the four `--exclude-*` filters, and rendered by `--format text | mermaid |
 mermaid-flowchart | ts | openapi`. `zen help rag` prints the full table.
 
 `list` and `grep` share `--name`, `--path`, `--regex`, `--case-sensitive`,
@@ -167,6 +251,23 @@ only what was named instead of its neighbourhood.
 zen rag schema grep token --ids-only | xargs zen rag schema show --format ts
 ```
 
+And for documents:
+
+```
+zen rag docs index <path...>     Read the documents and write a searchable index.
+zen rag docs search [text]       Ask it something. --interactive for a prompt.
+zen rag docs list <what>         Every document, section or table. No ranking.
+zen rag docs grep <pattern>      Every matching line, with the section it sits in.
+zen rag docs show <file>         A document, a section of one, or a line range.
+zen rag docs stats               What is in an index, and what built it.
+```
+
+Search takes the question as a bare phrase, narrowed by `--file`,
+`--exclude-file`, `--section`, `--kind` and `--mode`, shaped by `--limit`,
+`-B/--before`, `-A/--after` and `--max-lines`, and moved along by
+`--exclude-id`. `list` and `grep` share `--file`, `--section`, `--regex`,
+`--case-sensitive` and `--limit`.
+
 ## From an agent
 
 ```ts
@@ -201,6 +302,28 @@ need the word explained again, it needs the list of types that have one.
 `trace_api` is the step after both: a field is of no use until the call that
 carries it is known.
 
+Documents come with four, in the group `docs`, selectable as `docs:*`:
+
+```ts
+import { docs } from '@zenera/rag';
+
+const index = await docs.DocsIndex.open('./docs-db', embedder);
+const project = await loadProject('./my-project', { tools: docs.docsTools(index) });
+```
+
+| Tool          | For                                                          |
+| ------------- | ------------------------------------------------------------ |
+| `search_docs` | the passages that match, quoted with their line numbers      |
+| `list_docs`   | the documents, their headings, or their tables — no search   |
+| `grep_docs`   | every matching line, counted in full — no search             |
+| `read_docs`   | a section or a line range, verbatim and with nothing omitted |
+
+Same division, same reason. `search_docs` is the way in when the question is
+vague; `grep_docs` is how "it is not in here" can actually be concluded. Every
+answer carries line numbers and `read_docs` takes them, which is the loop the
+subject exists for: find the passage, read around it, then edit the file the
+passage came from.
+
 ## What an index is
 
 ```
@@ -217,6 +340,21 @@ schema-db/
 The manifest records which embedder made the vectors, and a search with a
 different one is refused rather than answered with noise.
 
+A document index is the same idea with a different middle:
+
+```
+docs-db/
+├── README.md         what this index holds — a live progress report while it builds
+├── manifest.json     written last — its absence means "not indexed"
+├── outline.json      every heading and table, with the lines they cover
+├── sources/          the documents themselves, verbatim — where the quotes come from
+└── lance/            one table: a row per chunk, two texts, one vector
+```
+
+There the copies are not a record but the answer: a search returns line ranges
+and the lines are read back out of `sources/`, so what is quoted is the document
+rather than a reconstruction of it.
+
 Indexing a large document is minutes of silence, so the directory says what is
 happening to it. `README.md` appears first as a progress report — the documents,
 the embedder, the step, how many entities have been embedded of how many, and
@@ -235,7 +373,49 @@ so an index built here is read under a name this machine never sees. `--no-sourc
 leaves the copies out, for an index that will never travel.
 
 The copies are a record, not an input: rebuilding reads the files you name, not
-the ones in `sources/`.
+the ones in `sources/`. A **document** index has no `--no-sources`, because
+there the copies are what every quoted line is read from.
+
+**Vectors are paid for once.** Embedding is the whole cost of a build — the
+reading and the writing are milliseconds, the round trips are minutes, and they
+are the only part anyone is billed for. Almost none of it is new work: editing
+one paragraph re-embeds every other one unchanged, and a build killed at 90%
+starts again from nothing. So the vectors are kept in this machine's shared
+cache, `~/.zenera/neo/cache/vectors/`, keyed by a hash of the exact text that
+produced them together with the model that made them. Re-indexing an unchanged
+corpus embeds nothing; re-indexing after an edit pays for the chunks that
+changed.
+
+The store is the machine's rather than the index's, which is the point: the same
+corpus indexed into a second directory costs nothing the second time, and two
+projects that quote the same handbook pay for it once between them.
+
+The key is the text, never the chunk's position — inserting a sentence at the
+top of a document shifts every later chunk's ordinal without changing a word of
+it. Two documents that share a paragraph share its vector. Vectors another model
+made are not evicted, they are simply never asked for, because the model is part
+of the key. It is a cache, so every failure in it is a miss and nothing more.
+`--no-cache` ignores it; `--cache-dir <dir>` keeps the work somewhere else.
+
+Nothing is ever evicted by a build: what a corpus stops referring to is still
+work somebody paid for. Getting rid of it is [`zen cache`](../cli/README.md)'s
+job — `zen cache ls` to see what is there, `zen cache prune --older-than 30d` to
+drop what has gone unread.
+
+**Parsing is spread across cores, and also cached.** With the vectors cached,
+parsing is what is left: reading a document, cutting it into chunks, and
+recording its shape. It is CPU-bound and every document is independent of every
+other, so a **document** index does it on a pool of worker threads — one per
+core, less the one running the build. The pool is skipped for a handful of
+documents, where starting the threads costs more than it saves.
+
+The result is cached the same way the vectors are, under
+`~/.zenera/neo/cache/docs-parse/`. The key covers the file's bytes, the name
+stamped on its chunks, and every setting that decides where a chunk ends, so
+changing `--chunk-tokens` misses on all of it and editing one file misses on one
+file. Both caches are governed by the same `--no-cache` and the same
+`--cache-dir`, and both fall back to doing the work when anything about them
+looks wrong.
 
 ## Notes
 
@@ -251,6 +431,11 @@ the ones in `sources/`.
 - Filters reaching the store are **closed enums only**. Exclusion lists are
   applied in JavaScript afterwards, so nothing a model wrote ever reaches a SQL
   predicate.
+- A document chunk knows **exactly which lines** of the original it stands for,
+  headings and table headers included. That is what makes an answer quotable,
+  and what lets the next question be phrased in line numbers.
+- Plain text is read as paragraphs and given **no invented headings**: a `.txt`
+  file has one section, which is the document.
 
 ## The rest of the family
 

package/dist/command.js

@@ -1,4 +1,5 @@
 import { bold, cyan, dim, table, usageError, write, } from '@zenera/cli/lib';
+import { command as docs } from "./docs/command.js";
 import { command as schema } from "./schema/command.js";
 // ---------------------------------------------------------------------------
 // zen rag — retrieval, by subject
@@ -18,7 +19,7 @@ import { command as schema } from "./schema/command.js";
 // `help <subject>` is a verb rather than a flag because `--help` never gets
 // here: the frame lifts it out of the arguments and answers with this page.
 // ---------------------------------------------------------------------------
-const SUBJECTS = { schema };
+const SUBJECTS = { schema, docs };
 const USAGE = 'zen rag <subject> <command> [args...]';
 export const command = {
     summary: 'Retrieval over a corpus: index it, then ask it something.',

package/dist/common/cache.d.ts

@@ -0,0 +1,66 @@
+import type { Embedder } from '@zenera/neo';
+export declare const VECTOR_KIND = "vectors";
+export interface VectorCache {
+    /** the vector this text already has, if it has one */
+    get(text: string): Float32Array | undefined;
+    /** keeps a vector for next time */
+    put(text: string, vector: Float32Array): void;
+    /** says the entries this build read are still wanted, so age means unused */
+    commit(): void;
+    /** for a build that failed; entries land as they are paid for, so nothing unwinds */
+    abandon(): void;
+    readonly hits: number;
+}
+/** A cache that remembers nothing, for `--no-cache`. */
+export declare const NO_CACHE: VectorCache;
+export interface VectorCacheOptions {
+    /** the reference as it was typed, which is part of what the vectors mean */
+    ref: string;
+    /** somewhere other than the shared store */
+    dir?: string;
+    /** when the caller asked for a width the model does not default to */
+    dimensions?: number;
+}
+export declare function openCache(embedder: Embedder, options: VectorCacheOptions): VectorCache;
+export interface CachedEmbedOptions {
+    embedder: Embedder;
+    cache: VectorCache;
+    texts: readonly string[];
+    signal?: AbortSignal;
+    onProgress?: (done: number, total: number) => void;
+}
+/**
+ * Embeds only what the cache does not already have, and hands each request's
+ * answer to the cache as it lands rather than at the end, so a build killed
+ * half way keeps the half it paid for.
+ *
+ * Identical texts are embedded once. A corpus repeats itself more than it looks
+ * like it does — shared boilerplate, a table copied between two documents — and
+ * a duplicate is a whole vector's worth of request for an answer already held.
+ */
+export declare function embedCached(options: CachedEmbedOptions): Promise<Float32Array[]>;
+export interface EmbedStreamOptions<R> {
+    embedder: Embedder;
+    cache: VectorCache;
+    records: readonly R[];
+    textOf: (record: R) => string;
+    window?: number;
+    signal?: AbortSignal;
+    /** counted over every record, not over the window being worked on */
+    onProgress?: (done: number, total: number) => void;
+    onWindow: (records: readonly R[], vectors: readonly Float32Array[]) => Promise<void>;
+}
+/**
+ * Embeds in windows and hands each one straight to whatever stores it, so that
+ * peak memory is the window rather than the corpus. A corpus of 200k chunks at
+ * 3072 dimensions held about 12 GB of vectors this way round; it now holds
+ * whatever one window is, however large the corpus gets.
+ *
+ * Duplicate texts spanning two windows still cost one embedding, because the
+ * first window has already written them to the cache by the time the second
+ * asks. Under `--no-cache` they cost two, which is what `--no-cache` means.
+ *
+ * Returns the width the model answered with, for the manifest.
+ */
+export declare function embedStream<R>(options: EmbedStreamOptions<R>): Promise<number>;
+//# sourceMappingURL=cache.d.ts.map

package/dist/common/cache.js

@@ -0,0 +1,172 @@
+import { Cache, cacheKey } from '@zenera/cli/lib';
+// ---------------------------------------------------------------------------
+// Not paying twice for the same vector
+//
+// Embedding is the whole cost of a build: the reading and the writing are
+// seconds, the round trips are minutes and the only part anyone is billed for.
+// And almost none of it is new work. Re-indexing a corpus after editing one
+// paragraph re-embeds every other paragraph unchanged, and a build killed at
+// 90% starts again from nothing.
+//
+// The vectors live in the machine's shared cache, so two indexes built from the
+// same corpus — or one corpus indexed twice into different directories — pay
+// for it once between them.
+//
+// The key is the *text*, together with the model that would embed it. Never the
+// chunk's position: inserting a sentence at the top of a document shifts every
+// later chunk's ordinal without changing a word of it, and an ordinal key would
+// miss all of them. Two documents that happen to share a paragraph share its
+// vector; a different model shares nothing, because it asks a different
+// question and so has a different key. That is also why there is nothing here
+// that invalidates anything.
+//
+// It is a cache, so every error is a miss, and the shared store guarantees that
+// much on its own: nothing in this file can fail a build.
+// ---------------------------------------------------------------------------
+export const VECTOR_KIND = 'vectors';
+/** Bumped only if what a key means changes; the model is already in the key. */
+const VECTOR_VERSION = 'v1';
+/** A cache that remembers nothing, for `--no-cache`. */
+export const NO_CACHE = {
+    get: () => undefined,
+    put: () => { },
+    commit: () => { },
+    abandon: () => { },
+    hits: 0,
+};
+export function openCache(embedder, options) {
+    return new StoredVectors(new Cache(VECTOR_KIND, { dir: options.dir }), embedder, options);
+}
+class StoredVectors {
+    #store;
+    #prefix;
+    constructor(store, embedder, options) {
+        this.#store = store;
+        this.#prefix = [VECTOR_VERSION, options.ref, embedder.id, options.dimensions];
+    }
+    get hits() {
+        return this.#store.hits;
+    }
+    /** Everything the vector is a function of, ending with the text itself. */
+    #key(text) {
+        return cacheKey(...this.#prefix, text);
+    }
+    get(text) {
+        const found = this.#store.get(this.#key(text));
+        if (typeof found !== 'string' || found.length === 0) {
+            return undefined;
+        }
+        const bytes = Buffer.from(found, 'base64');
+        if (bytes.byteLength === 0 || bytes.byteLength % 4 !== 0) {
+            return undefined;
+        }
+        // Buffer.from can land at any offset in the shared pool, and a
+        // Float32Array needs a multiple of four. Slicing copies to its own.
+        return new Float32Array(bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength));
+    }
+    put(text, vector) {
+        if (vector.length > 0) {
+            const bytes = new Uint8Array(vector.buffer, vector.byteOffset, vector.byteLength);
+            this.#store.put(this.#key(text), Buffer.from(bytes).toString('base64'));
+        }
+    }
+    commit() {
+        this.#store.commit();
+    }
+    abandon() { }
+}
+/**
+ * Embeds only what the cache does not already have, and hands each request's
+ * answer to the cache as it lands rather than at the end, so a build killed
+ * half way keeps the half it paid for.
+ *
+ * Identical texts are embedded once. A corpus repeats itself more than it looks
+ * like it does — shared boilerplate, a table copied between two documents — and
+ * a duplicate is a whole vector's worth of request for an answer already held.
+ */
+export async function embedCached(options) {
+    const { cache, texts } = options;
+    const vectors = new Array(texts.length);
+    const wanted = new Map();
+    for (const [at, text] of texts.entries()) {
+        const hit = cache.get(text);
+        if (hit) {
+            vectors[at] = hit;
+            continue;
+        }
+        const waiting = wanted.get(text);
+        if (waiting) {
+            waiting.push(at);
+        }
+        else {
+            wanted.set(text, [at]);
+        }
+    }
+    const input = [...wanted.keys()];
+    const known = texts.length - [...wanted.values()].reduce((n, at) => n + at.length, 0);
+    if (input.length > 0) {
+        const response = await options.embedder.embed({
+            input,
+            taskType: 'document',
+            signal: options.signal,
+            onSlice: (at, slice) => {
+                for (const [i, vector] of slice.entries()) {
+                    cache.put(input[at + i], Float32Array.from(vector));
+                }
+            },
+            onProgress: (done) => options.onProgress?.(known + done, texts.length),
+        });
+        for (const [i, vector] of response.vectors.entries()) {
+            const shared = Float32Array.from(vector);
+            // `put` rewrites the same bytes, so what a slice already saved costs
+            // nothing here. Not every embedder reports slices, and the cache
+            // cannot depend on it.
+            cache.put(input[i], shared);
+            for (const at of wanted.get(input[i])) {
+                vectors[at] = shared;
+            }
+        }
+    }
+    options.onProgress?.(texts.length, texts.length);
+    return vectors;
+}
+/**
+ * How many records are embedded, written and let go of before the next are
+ * looked at. At 3072 dimensions a window costs about 250 MB while it is in
+ * flight, and the embedder still sees enough texts at once to keep every
+ * request slot busy.
+ */
+const WINDOW = 4096;
+/**
+ * Embeds in windows and hands each one straight to whatever stores it, so that
+ * peak memory is the window rather than the corpus. A corpus of 200k chunks at
+ * 3072 dimensions held about 12 GB of vectors this way round; it now holds
+ * whatever one window is, however large the corpus gets.
+ *
+ * Duplicate texts spanning two windows still cost one embedding, because the
+ * first window has already written them to the cache by the time the second
+ * asks. Under `--no-cache` they cost two, which is what `--no-cache` means.
+ *
+ * Returns the width the model answered with, for the manifest.
+ */
+export async function embedStream(options) {
+    const size = options.window ?? WINDOW;
+    const total = options.records.length;
+    let dimensions = 0;
+    for (let at = 0; at < total; at += size) {
+        const window = options.records.slice(at, at + size);
+        const vectors = await embedCached({
+            embedder: options.embedder,
+            cache: options.cache,
+            texts: window.map(options.textOf),
+            signal: options.signal,
+            onProgress: (done) => options.onProgress?.(at + done, total),
+        });
+        if (dimensions === 0) {
+            dimensions = vectors[0]?.length ?? 0;
+        }
+        await options.onWindow(window, vectors);
+    }
+    return dimensions;
+}
+//# sourceMappingURL=cache.js.map

package/dist/common/embedder.d.ts

@@ -1,3 +1,13 @@
 import { type Embedder } from '@zenera/neo';
-export declare function resolveEmbedder(ref: string | undefined): Promise<Embedder>;
+/**
+ * What a shorthand ref cannot say. Both are ceilings on what the embedder would
+ * otherwise work out for itself from the model and from what the provider
+ * refuses, so leaving them unset is the normal case.
+ */
+export interface EmbedderTuning {
+    maxBatch?: number;
+    /** narrower vectors, where the model is trained to be truncated */
+    dimensions?: number;
+}
+export declare function resolveEmbedder(ref: string | undefined, tuning?: EmbedderTuning): Promise<Embedder>;
 //# sourceMappingURL=embedder.d.ts.map

package/dist/common/embedder.js

@@ -1,20 +1,6 @@
 import { bold, CliError, CURATED, cyan, dim, ensureHome, envNames, form, KeyStore, note, PROVIDERS, table, usageError, } from '@zenera/cli/lib';
-import { createEmbedder } from '@zenera/neo';
-// ---------------------------------------------------------------------------
-// Getting an embedder, and saying what the choices are when there is none
-//
-// The keyring is materialised here, and only here. The `zen` frame does not do
-// it, and a command that forgets to looks exactly like a machine with no key:
-// `zen key ls` shows the credential live and the command still says the
-// environment variable is not set. Real env always wins, so a shell that names
-// a key is never overridden by one on disk.
-//
-// Nothing in here is about a particular index. Which model made the vectors is
-// recorded in the manifest and enforced on every later search, so this only has
-// to turn a reference into an embedder — or, given none, into a list worth
-// choosing from.
-// ---------------------------------------------------------------------------
-export async function resolveEmbedder(ref) {
+import { createEmbedder, defaultModels } from '@zenera/neo';
+export async function resolveEmbedder(ref, tuning) {
     ensureHome();
     const keys = await KeyStore.open();
     // Asked before materialising, because materialising is what erases the
@@ -24,7 +10,10 @@ export async function resolveEmbedder(ref) {
     if (!ref) {
         throw choices(keys, fromEnv);
     }
-    return createEmbedder(ref);
+    if (tuning?.maxBatch === undefined && tuning?.dimensions === undefined) {
+        return createEmbedder(ref);
+    }
+    return createEmbedder({ ...defaultModels.parseEmbedding(ref), ...tuning });
 }
 /**
  * Well-known embedding models per provider, read off the CLI's catalog table so

package/dist/common/manifest.d.ts

@@ -12,11 +12,17 @@ export interface IndexHead {
     kind: IndexKind;
     createdAt: string;
     indexer: string;
-    /** `ref` as it was typed, `id` as the embedder answers to it */
+    /**
+     * `ref` as it was typed, `id` as the embedder answers to it, `dimensions`
+     * as the vectors actually came back. `requested` only when a width was
+     * asked for out loud: a search has to ask for the same one, and asking for
+     * the model's own default is not the same as not asking.
+     */
     embedding: {
         ref: string;
         id: string;
         dimensions: number;
+        requested?: number;
     };
     /** whether the table carries an fts index, and whether it carries a vector one */
     indexes: {

package/dist/common/progress.d.ts

@@ -2,6 +2,11 @@ export declare const LOCK_FILE = ".lock";
 export declare const README_FILE = "README.md";
 /** The floor on how often README.md is rewritten. */
 export declare const INTERVAL_MS = 5000;
+/** How long one phase took, by its key rather than its sentence. */
+export interface PhaseTiming {
+    name: string;
+    ms: number;
+}
 export interface Building<S> {
     documents: readonly string[];
     embedding: string;
@@ -9,8 +14,14 @@ export interface Building<S> {
     now: number;
     /** the current phase, as it should be said out loud */
     step: string;
+    /** the current phase's key, for a report that counts different things per phase */
+    phase: string;
     done: number;
     total: number;
+    /** what the phase is still waiting on, when it is able to say */
+    pending: readonly string[];
+    /** every phase so far; the last is the one still running */
+    timings: readonly PhaseTiming[];
     /** what the documents turned out to hold, once they have been read */
     summary: S | undefined;
 }
@@ -18,12 +29,14 @@ export interface Completed<M> {
     dir: string;
     manifest: M;
     ms: number;
+    timings: readonly PhaseTiming[];
 }
 export interface Failed {
     documents: readonly string[];
     step: string;
     reason: unknown;
     started: number;
+    timings: readonly PhaseTiming[];
 }
 /** The three states a README can be in, written by whoever knows the subject. */
 export interface Report<S, M> {
@@ -45,9 +58,11 @@ export interface BuildPlan<S, M, P extends string> {
 export interface Journal<S, M, P extends string> {
     phase(name: P): void;
     read(summary: S, total: number): void;
-    progress(done: number, total: number): void;
+    progress(done: number, total: number, pending?: readonly string[]): void;
     finish(manifest: M): void;
     fail(reason: unknown): void;
+    /** what each phase cost, for a caller that wants to say so out loud */
+    readonly timings: readonly PhaseTiming[];
 }
 /**
  * Takes the directory, or refuses it. Two builds writing one index would