@wirecat/cli-core 0.433.0 → 0.435.0

package/CHANGELOG.md

@@ -5,7 +5,49 @@ Notable changes to `@wirecat/cli-core` (`@wirecat/cli-messaging` up to 0.261.0,
 minor version may break callers, and says how under "Changed — may break callers". `pnpm docs:check` checks the
 shape of this file.
 
-## Unreleased
+## 0.435.0 — 25.10.2026
+
+### Added
+
+- **`search messages --discover` reranks its first 50 hits, on its own, once the reranker is downloaded.** A cross-encoder reads the question with each hit, headed by its chat, author and date and preceded by the message it replies to, and its score is mixed with discovery's own, not put in its place: both scaled to 0–1 over those 50, then 0.3 · discovery + 0.7 · reranker; the hits after keep their order. This is the winning variant of the [evidence-pipeline benchmark](https://github.com/WireCatLabs/cli-testing/tree/main/performance/search/evidence-pipeline#reranker-variants) (holdout Answer@3 67% → 81% with mMARCO). Each hit it read has `discovery.reranker_score`; `score` stays discovery's. The default is `mmarco` (136 MB, `models download mmarco`); `bge-reranker-v2-m3` (588 MB) or `off` is the profile setting `search_reranker`. Loading and scoring do not count against the word search's 2 s: scoring has 15 s of its own, and a reranking that would not finish in them is dropped, never the search. Measured on a Ryzen AI 9 HX 470 through Core's WebAssembly runtime, 50 hits take mmarco about 0.9–1.7 s and BGE about 7–10 s. Without the model, discovery keeps its order and says so: a `not reranked: …` note in the terminal, `missing`/`next` (`models.download`) in `--json` and MCP. A search never downloads a model and never sends text anywhere; `--exact`, `--newest` and strict queries are untouched. `models list` shows the rerankers as kind `reranker`.
+- **`search all` finds messages by meaning too, on its own, once it can.** It runs the same meaning lookup as
+  `search messages --discover` (e5-small over each message's own vector, the 100 nearest, every account the search
+  reads) and fuses it with the word hits by reciprocal rank fusion (k = 60) into one messages list, before the merge
+  with mail and notes. The query's filters apply to both; a query with NOT, or with a filter under OR or in a group,
+  stays words only, since the meaning hits cannot follow it (`cat OR fish` still uses meaning). Each item's `foundBy` says which search found it, and the answer's
+  `messages` says how they were searched. Without the model or the vectors, `search all` reads words as before and
+  says so: a `messages: meaning search not used: …` note, and `missing`/`next` (`models.download` or
+  `conversations.rebuild`) in `--json` and MCP. `--exact` and `--backend server` stay words only. A search never
+  downloads a model, embeds anything or sends text anywhere. `searchAll`, `searchAllWithMeetings` and
+  `searchAllIncludingMeetings` take the meaning lane as an optional last argument; a caller that passes none
+  (memo's `searchEverywhere` today) searches messages by words alone. `meaningLane` is exported and needs only
+  `{ env, store, messenger: { app: { command } } }`, so memo can build one without the messenger's services.
+  Messages and notes share one loaded model within a `search all`, so searching both by meaning loads e5-small once:
+  on a 10,000-message store with 200 notes the median went from 1.7 s to 0.9 s.
+
+### Fixed
+
+- **A vector goes once no text in the store uses it, by one rule for every kind.** Writing messages, images, notes,
+  documents and their pictures, mail and meeting transcripts, and purging an account or a chat, each had their own
+  query; they now all share one, which drops every model's vector of a released hash that no chunk
+  holds any more, and keeps one a chunk of any kind still holds. Deleting a note, document, memory or email still
+  goes through its delete trigger, which applies the same rule. Three writes left vectors behind and no longer do: an image whose
+  picture changed, a meeting transcript whose text changed, and a purged chat (`store clear --left` and a chat
+  merged into another), which also left its messages' and images' chunks. The rule reaches each vector by its key
+  (model, hash), where most of the old queries scanned every vector once per hash.
+- **`conversations embeddings clear --chat` drops the chat's message vectors too**, not only its conversations'; a
+  vector a chunk outside the chat shares stays, and `conversations embed --chat` makes them again.
+- **`store rebuild` drops the vectors no stored text uses**, which older builds left behind, and says how many in
+  `unused_vectors_dropped`. `store check` counts them (`vectors.unused`) and, when there are any, notes that
+  `store rebuild` drops them.
+
+## 0.434.0 — 25.10.2026
+
+### Changed — may break callers
+
+- **Search by meaning ranks by a sqlite-vec index in the store (store version 34), and every store connection loads the extension.** Each embedding model gets a `vec0` table, `vector_index_<id>`, listed in the new `vector_indexes`; triggers on `chunks`, `embeddings` and `messages` keep it in step inside each write's own transaction. `nearestMessages`, `nearestImages` and conversation search (without a person, project, scope or conversation list) take candidates from it, check them with the scan's own SQL and score them as before, so they answer exactly what the scan answers; the scan stays for the filters the index cannot express. A store that held vectors is indexed on its first open by this version, in short batches. **Version 34 raises `minCompatible` to 34**: a build without the extension cannot write chunks to a store with these triggers, so older CLIs refuse the file and ask to be upgraded. The extension ships in `@wirecat/cli-core-sqlite` 1.1.0 (now a required runtime piece on every platform, Windows included); when it is missing or does not load, opening the store fails with `store.vector-extension-missing` or `store.vector-extension-failed` and says to reinstall. Under Node the store is opened with `allowExtension`, and SQL's `load_extension()` is switched off again right after.
+- **`store check` reports `vector_indexes`** — per model, the rows its index holds and the rows it should, and how many differ either way (`missing`, `extra`), compared whole — and fails `checks.vector_indexes` when a ready index is off.
+- **Vectors are stored at length one.** `saveVectors` and a document picture's vector are scaled to unit length before they are kept, so search by dot product and by the index rank alike; a vector with a non-finite value is refused (`store.vector-invalid`). Vectors stored before keep their length. The `not_applicable.extensions` field ("SQLite needs none") is gone.
 
 ## 0.433.0 — 25.10.2026
 

package/dist/cli/messenger/context.js

@@ -414,6 +414,7 @@ export const messengerContext = (command, messenger, resolving = {}) => {
                 history: base.settings.keepFailedRuns,
                 searchCatchUp: base.settings.searchCatchUp,
                 pictures: base.settings.pictureSearch,
+                reranker: base.settings.searchReranker,
                 guard,
                 ...(embedders ? { embedders } : {}),
                 connection: async () => {

package/dist/cli/messenger/conversation-answers.d.ts

@@ -66,6 +66,7 @@ export declare const conversationShown: (account: AccountKey, { summary, message
             missing_terms: string[];
             parent?: string | undefined;
             found_by?: ("meaning" | "words")[] | undefined;
+            reranker_score?: number | undefined;
         } | undefined;
         context?: {
             ref: string;

package/dist/cli/messenger/conversations-command.js

@@ -125,8 +125,8 @@ const embeddingStatusCommand = (messenger) => withModelOptions(annotate(new Comm
         "\n");
 });
 const embeddingClearCommand = (messenger) => withModelOptions(destroysNothing(annotate(new Command("clear"), { output: { entity: "conversation_change" }, effects: LOCAL_WRITE }))
-    .summary("remove stored conversation vectors")
-    .description("drop a chat's vectors, or only one model's; messages and conversations are never touched")
+    .summary("remove a chat's stored conversation and message vectors")
+    .description("drop a chat's conversation and message vectors, or only one model's; messages and conversations are never touched")
     .option("--chat <chat>", messenger.chatArgument)).action(async function () {
     const options = this.optsWithGlobals();
     const chat = chatOf(options);

package/dist/cli/messenger/message-answers.d.ts

@@ -43,6 +43,7 @@ export declare const threadAnswer: ({ items, ...thread }: ThreadContext, reader:
             missing_terms: string[];
             parent?: string | undefined;
             found_by?: ("meaning" | "words")[] | undefined;
+            reranker_score?: number | undefined;
         } | undefined;
         context?: {
             ref: string;

package/dist/cli/messenger/message-writes.d.ts

@@ -47,6 +47,7 @@ export declare const writtenAnswer: ({ operationId, sendId, message, messages, s
             missing_terms: string[];
             parent?: string | undefined;
             found_by?: ("meaning" | "words")[] | undefined;
+            reranker_score?: number | undefined;
         } | undefined;
         context?: {
             ref: string;
@@ -236,6 +237,7 @@ export declare const writtenAnswer: ({ operationId, sendId, message, messages, s
             missing_terms: string[];
             parent?: string | undefined;
             found_by?: ("meaning" | "words")[] | undefined;
+            reranker_score?: number | undefined;
         } | undefined;
         context?: {
             ref: string;

package/dist/cli/messenger/messages-search-command.js

@@ -12,7 +12,7 @@ import { renderAnswer } from "../envelope.js";
 import { positiveCount } from "../paging.js";
 import { archiveSummary } from "./archive-summary.js";
 import { messengerContext } from "./context.js";
-import { messagesAnswer, offsetOf } from "./search-answers.js";
+import { meaningNote, messagesAnswer, offsetOf } from "./search-answers.js";
 import { backendOptions, backendRequest, noteServer } from "./search-backend-options.js";
 import { syncOptions, syncRequest, syncsFirst } from "./search-sync-options.js";
 import { threadOptions, threadRequest } from "./thread-options.js";
@@ -130,15 +130,11 @@ const searchAction = (messenger) => async function (given) {
         context.renderer.note(`${from} → ${to.join(", ")}`);
     for (const { source, reason } of found.skipped ?? [])
         context.renderer.note(`${source} not searched: ${reason}`);
-    const meaning = found.meaningSkipped;
-    if (meaning) {
-        const step = meaning.step === "models.download"
-            ? ` — \`${command} models download ${meaning.model}\`, then \`${command} conversations rebuild\` gives messages their vectors`
-            : meaning.step === "conversations.rebuild"
-                ? ` — \`${command} conversations rebuild\` gives messages their vectors, a batch a run`
-                : "";
-        context.renderer.note(`meaning search not used: ${meaning.reason}${step}`);
-    }
+    if (found.meaningSkipped)
+        context.renderer.note(meaningNote(found.meaningSkipped, command));
+    const rerank = found.rerankSkipped;
+    if (rerank)
+        context.renderer.note(`not reranked: ${rerank.reason}${rerank.step === "models.download" ? ` — \`${command} models download ${rerank.model}\`` : ""}`);
     if (found.picturesSkipped && found.picturesModel === undefined)
         context.renderer.note(`pictures not searched: ${found.picturesSkipped}`);
     const stemming = found.query?.stemming;

package/dist/cli/messenger/models-command.js

@@ -4,10 +4,10 @@ import { annotate } from "@wirecat/cli-base/commands";
 import { Command } from "commander";
 import { isTextModelInstalled, textModelsDirectory } from "../../embeddings/embed.js";
 import { embeddingKeys } from "../../embeddings/keys.js";
-import { DEFAULT_TEXT_MODEL, TEXT_MODELS } from "../../embeddings/models.js";
+import { DEFAULT_RERANK_MODEL, DEFAULT_TEXT_MODEL, RERANK_MODELS, TEXT_MODELS, } from "../../embeddings/models.js";
 import { model as modelEntity } from "../../entities/model.js";
 import { DEFAULT_IMAGE_MODEL, IMAGE_MODELS, imageModelsDirectory, isImageModelInstalled, } from "../../images/picture/models.js";
-import { downloadImageModel, downloadTextModel } from "../../services/model-downloads.js";
+import { downloadImageModel, downloadRerankModel, downloadTextModel } from "../../services/model-downloads.js";
 import { install, installedBytes, isInstalled, megabytes, modelPath, modelsDirectory, vadPath, } from "../../speech/install.js";
 import { orderedModels, speechModel, VAD } from "../../speech/models.js";
 import { choose } from "../../speech/transcribe.js";
@@ -44,6 +44,17 @@ const textItem = (model, directory) => ({
     default: model.id === DEFAULT_TEXT_MODEL,
     path: join(directory, model.id),
 });
+const rerankItem = (model, directory, chosen) => ({
+    id: model.id,
+    kind: "reranker",
+    title: model.title,
+    languages: model.languages,
+    licence: model.licence,
+    size_bytes: model.files.reduce((sum, file) => sum + file.bytes, 0),
+    downloaded: isTextModelInstalled(model, directory),
+    default: model.id === chosen,
+    path: join(directory, model.id),
+});
 const imageItem = (model, directory) => ({
     id: model.id,
     kind: "image",
@@ -93,6 +104,16 @@ const downloadEmbeddingModel = async (command, messenger, id, options) => {
         renderAnswer(command, context, itemOf(textItem(model, directory)), downloaded);
     }, { unbounded: true });
 };
+const downloadReranker = async (command, messenger, id) => {
+    const context = baseContext(command, messenger.resolveSettings);
+    await context.run(async () => {
+        const { model, directory } = await downloadRerankModel(id, {
+            env: context.env,
+            progress: (line) => context.renderer.note(line),
+        });
+        renderAnswer(command, context, itemOf(rerankItem(model, directory, context.settings.searchReranker ?? DEFAULT_RERANK_MODEL)), downloaded);
+    }, { unbounded: true });
+};
 const modelKeysCommand = (messenger, tool) => {
     const key = new Command("keys").description("API keys for embedding and analysis providers");
     grantsAuthority(annotate(key.command("set"), {
@@ -136,7 +157,7 @@ export const modelsCommand = (messenger) => {
         effects: LOCAL_READ,
         examples: [`${tool} models list`, `${tool} models list kind:audio`],
     }), ROOT_MODELS_QUERY)
-        .description("speech and embedding models, their download status and the default of each kind")
+        .description("speech, embedding and reranker models: downloaded or not, each kind's default")
         .action(async function (words) {
         const terms = nativeTerms(listQuery(typedQuery(words), ROOT_MODELS_QUERY, "models list"), "models list");
         const context = baseContext(this, messenger.resolveSettings);
@@ -150,25 +171,29 @@ export const modelsCommand = (messenger) => {
             const textModels = kinds === undefined || kinds === "text"
                 ? TEXT_MODELS.filter((model) => id === undefined || model.id === id)
                 : [];
+            const rerankers = kinds === undefined || kinds === "reranker"
+                ? RERANK_MODELS.filter((model) => id === undefined || model.id === id)
+                : [];
             const chosen = audioModels.length > 0 ? choose(messenger, context.settings, {}, context.env).model.id : "";
             const catalogue = [
                 ...audioModels.map((model) => audioItem(model, modelsDirectory(context.env), chosen)),
                 ...textModels.map((model) => textItem(model, textModelsDirectory(context.env))),
+                ...rerankers.map((model) => rerankItem(model, textModelsDirectory(context.env), context.settings.searchReranker ?? DEFAULT_RERANK_MODEL)),
             ];
             const items = catalogue.filter((item) => kept === undefined || item.downloaded === (kept === "true"));
             const next = items
                 .filter((item) => item.default && !item.downloaded)
                 .map((item) => nextStep("models.download", { args: { model: item.id } }));
-            renderAnswer(this, context, everyModel("models list", items, next), () => items.map((item) => `${item.kind.padEnd(5)} ${modelLine(item, 18)}  ${item.languages}\n`).join(""));
+            renderAnswer(this, context, everyModel("models list", items, next), () => items.map((item) => `${item.kind.padEnd(8)} ${modelLine(item, 18)}  ${item.languages}\n`).join(""));
         });
     });
     reachesOtherService(annotate(models.command("download"), {
         effects: FETCH,
         output: { entity: "model" },
-        examples: [`${tool} models download ${DEFAULT_TEXT_MODEL}`],
+        examples: [`${tool} models download ${DEFAULT_TEXT_MODEL}`, `${tool} models download ${DEFAULT_RERANK_MODEL}`],
     }))
-        .summary("download and verify one speech or embedding model")
-        .description("download and verify one speech or embedding model: tens to hundreds of MB, each size in models list")
+        .summary("download and verify one speech, embedding or reranker model")
+        .description("download and verify one speech, embedding or reranker model: tens to hundreds of MB, each size in models list")
         .argument("<model>", "a model id from models list")
         .option("--accept-terms", "accept the model's licence terms, for a model that has its own")
         .action(async function (id, options) {
@@ -176,6 +201,8 @@ export const modelsCommand = (messenger) => {
             return downloadAudioModel(this, messenger, id);
         if (TEXT_MODELS.some((model) => model.id === id))
             return downloadEmbeddingModel(this, messenger, id, options);
+        if (RERANK_MODELS.some((model) => model.id === id))
+            return downloadReranker(this, messenger, id);
         throw new CliError("validation_error", "unknown model; choose an id from models list");
     });
     return models;

package/dist/cli/messenger/search-answers.d.ts

@@ -2,6 +2,7 @@ import type { Missing, NextStep } from "@wirecat/cli-base/standards";
 import type { InferOutput } from "valibot";
 import type { ENTITIES } from "../../entities/index.js";
 import type { FoundMessage, SearchFound } from "../../services/messages.js";
+import type { MeaningSkipped } from "../../services/messages-discovery.js";
 import type { FoundNote, NotesFoundOf } from "../../services/notes-search.js";
 import type { SearchAllItem } from "../../services/search-all.js";
 import type { MeetingSearchItem, SearchAllIncludingMeetingsFound } from "../../services/search-all-meetings.js";
@@ -20,6 +21,8 @@ export declare const coverageGaps: (found: Pick<SearchFound, "coverage" | "refre
     next: NextStep[];
     missing: Missing[];
 };
+/** The pretty note for messages searched by words only, with the command that would add meaning. */
+export declare const meaningNote: (meaning: MeaningSkipped, command: string) => string;
 export interface Paging {
     limit: number;
     page?: number;

package/dist/cli/messenger/search-answers.js

@@ -31,6 +31,7 @@ export const messageHitItem = (hit, query = "") => {
                 missing_terms: hit.discovery.missingTerms,
                 parent: hit.discovery.parent,
                 found_by: hit.discovery.foundBy,
+                reranker_score: hit.discovery.rerankerScore,
             })
             : undefined,
         email: hit.email
@@ -88,6 +89,31 @@ export const coverageGaps = (found, shown) => {
         ],
     };
 };
+/** Messages searched by words only, and the step that would add meaning: `--discover` and `search all` alike. */
+const messageMeaningGap = (meaning, field) => {
+    const next = [];
+    if (!meaning)
+        return { next, missing: [] };
+    if (meaning.step === "models.download")
+        next.push(nextStep("models.download", { args: { model: meaning.model } }));
+    if (meaning.step === "conversations.rebuild")
+        next.push(nextStep("conversations.rebuild", { args: {} }));
+    return {
+        next,
+        missing: [
+            { field, why: `searched by words only: ${meaning.reason}`, ...(meaning.step ? { step: meaning.step } : {}) },
+        ],
+    };
+};
+/** The pretty note for messages searched by words only, with the command that would add meaning. */
+export const meaningNote = (meaning, command) => {
+    const step = meaning.step === "models.download"
+        ? ` — \`${command} models download ${meaning.model}\`, then \`${command} conversations rebuild\` gives messages their vectors`
+        : meaning.step === "conversations.rebuild"
+            ? ` — \`${command} conversations rebuild\` gives messages their vectors, a batch a run`
+            : "";
+    return `meaning search not used: ${meaning.reason}${step}`;
+};
 const gapsOf = (found) => {
     const { next, missing } = coverageGaps(found, found.items.length);
     const building = !found.wordsReady || (found.query?.stemming?.applied === false && found.query.stemming.reason === "building");
@@ -101,16 +127,19 @@ const gapsOf = (found) => {
             step: "store.migrate",
         });
     }
-    const meaning = found.meaningSkipped;
-    if (meaning) {
-        if (meaning.step === "models.download")
-            next.push(nextStep("models.download", { args: { model: meaning.model } }));
-        if (meaning.step === "conversations.rebuild")
-            next.push(nextStep("conversations.rebuild", { args: {} }));
+    const meaning = messageMeaningGap(found.meaningSkipped, "discovery");
+    for (const step of meaning.next)
+        next.push(step);
+    for (const gap of meaning.missing)
+        missing.push(gap);
+    const rerank = found.rerankSkipped;
+    if (rerank) {
+        if (rerank.step === "models.download")
+            next.push(nextStep("models.download", { args: { model: rerank.model } }));
         missing.push({
             field: "discovery",
-            why: `searched by words only: ${meaning.reason}`,
-            ...(meaning.step ? { step: meaning.step } : {}),
+            why: `not reranked: ${rerank.reason}`,
+            ...(rerank.step ? { step: rerank.step } : {}),
         });
     }
     if (found.picturesModel !== undefined) {
@@ -197,6 +226,7 @@ export const notesAnswer = (command, found, { limit, page, again }, exact = fals
 export const allAnswer = (command, found, { limit, page, again }, exact = false) => {
     const scan = found.meetings?.complete === false ? found.meetings : undefined;
     const notes = meaningGap(found.notes?.meaningSkipped, exact);
+    const messages = messageMeaningGap(found.messages?.meaningSkipped, "found_by");
     return listOf(command, {
         items: found.items.map(allHitItem),
         limit,
@@ -214,6 +244,7 @@ export const allAnswer = (command, found, { limit, page, again }, exact = false)
                 ]
                 : []),
             ...notes.next,
+            ...messages.next.filter((step) => !notes.next.some((asked) => JSON.stringify(asked) === JSON.stringify(step))),
         ],
         missing: [
             ...(scan
@@ -226,6 +257,7 @@ export const allAnswer = (command, found, { limit, page, again }, exact = false)
                 ]
                 : []),
             ...notes.missing,
+            ...messages.missing,
         ],
         meta: metaOf(found.accounts, found.skipped),
     });

package/dist/cli/messenger/search-command.js

@@ -11,7 +11,7 @@ import { commandAt } from "../steps.js";
 import { messengerContext } from "./context.js";
 import { conversationsSearchCommand } from "./conversations-command.js";
 import { messagesSearchCommand } from "./messages-search-command.js";
-import { allAnswer, notesAnswer, offsetOf, SOURCE_SEARCHED } from "./search-answers.js";
+import { allAnswer, meaningNote, notesAnswer, offsetOf, SOURCE_SEARCHED, } from "./search-answers.js";
 import { backendOptions, backendRequest, noteServer } from "./search-backend-options.js";
 import { topicsSearchCommand } from "./topics-command.js";
 const onlyOf = (value) => {
@@ -79,6 +79,8 @@ const allCommand = (messenger) => {
             context.renderer.note(`${resource ?? source} not searched: ${reason}`);
         if (found.notes?.meaningSkipped)
             context.renderer.note(`notes by words only: ${found.notes.meaningSkipped}`);
+        if (found.messages?.meaningSkipped)
+            context.renderer.note(`messages: ${meaningNote(found.messages.meaningSkipped, messenger.app.command)}`);
         const again = {
             query: words.join(" "),
             ...(only === undefined ? {} : { only: only.join(",") }),

package/dist/cli/messenger/store-maintenance-command.js

@@ -12,7 +12,7 @@ import { migrate } from "../../store/migrations.js";
 import { openCache } from "../../store/open.js";
 import { storePath } from "../../store/path.js";
 import { deleteCopy, repairStore } from "../../store/repair.js";
-import { drainNoteIndex, fillSearchIndex, fillStems, involvementStoreOver, noteIndexState, openSqlite, openStore, orphanPointers, pendingNormalization, resetAttachmentWords, resetNoteIndex, resetSearchIndex, resetStems, searchIndexState, stemmerCache, stemsState, } from "../../store/store.js";
+import { drainNoteIndex, dropAllUnusedVectors, fillSearchIndex, fillStems, involvementStoreOver, noteIndexState, openSqlite, openStore, orphanPointers, pendingNormalization, resetAttachmentWords, resetNoteIndex, resetSearchIndex, resetStems, searchIndexState, stemmerCache, stemsState, vectorIndexCounts, } from "../../store/store.js";
 import { environmentOf, outputFor } from "../context.js";
 import { destroysNothing, LOCAL_READ, LOCAL_WRITE } from "../effects.js";
 import { itemOf, renderAnswer, snakeKeys } from "../envelope.js";
@@ -217,6 +217,8 @@ const checkCommand = (messenger) => annotate(new Command("check"), { output: { e
     const orphans = answer.orphanPointers.reduce((sum, { count }) => sum + count, 0);
     if (orphans > 0)
         renderer.note(`${orphans} pointers name a row that is gone or a type the store does not know — see orphan_pointers`);
+    if (answer.vectors && answer.vectors.unused > 0)
+        renderer.note(`${answer.vectors.unused} vectors belong to no text the store still holds — \`${command} store rebuild\` drops them`);
     const older = answer.conversations.filter((chat) => chat.provider === messenger.provider && !chat.current).length;
     if (older > 0) {
         renderer.note(`${older} chats' conversations were built by older rules — \`${command} conversations links rebuild --chat <chat>\``);
@@ -241,11 +243,14 @@ const inspect = (path) => reading(path, (database) => {
     const { bavail, bsize } = statfsSync(dirname(path));
     const free = Number(bavail) * Number(bsize);
     const behind = schema.version > 0 ? chatsBehind(database) : [];
+    const vectorIndexes = schema.version >= 34 ? vectorIndexCounts(database) : [];
     const checks = {
         schema: schema.version >= SPEAKS && schema.writable,
         integrity: integrity.length === 1 && integrity[0] === "ok",
         foreignKeys: foreignKeys === 0,
         searchIndexes: Object.values(searchIndexes).every((state) => state === "ok"),
+        // One that is still filling is short by design; a ready one must hold exactly the rows its sources give.
+        vectorIndexes: vectorIndexes.every(({ ready, missing, extra }) => !ready || missing + extra === 0),
         // A backup or a VACUUM needs about as much again.
         disk: free >= size,
     };
@@ -268,7 +273,7 @@ const inspect = (path) => reading(path, (database) => {
         orphanPointers: schema.version > 0 ? orphanPointers(database) : [],
         conversations: conversationsBuilt(database),
         vectors: vectorsHeld(database),
-        notApplicable: { extensions: "SQLite needs none" },
+        vectorIndexes,
     };
 });
 /** `rank = 1` compares the index with its table; without it a stale external-content index passes. */
@@ -282,8 +287,8 @@ const indexIntegrity = (database, index, rank = 1) => {
     }
 };
 /**
- * Vectors per model, and how many no chunk of any build points at any more — a rebuild that changed a
- * conversation's text leaves its old vector behind until `conversations embed clear` (phase 5).
+ * Vectors per model, and how many no chunk points at any more. Each write that drops chunks drops those
+ * (`dropUnusedVectors`), so a count above zero was most likely left by an older build; `store rebuild` drops them.
  */
 const vectorsHeld = (database) => {
     const exists = database.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'embeddings'").get();
@@ -415,7 +420,7 @@ const buildNoteIndex = (database, note, { reset = false } = {}) => {
 };
 const reindexCommand = (messenger) => destroysNothing(annotate(new Command("rebuild"), { output: { entity: "store_reindex" }, effects: LOCAL_WRITE }))
     .summary("rebuild the local archive's search indexes")
-    .description("rebuild the word index, its typo vocabulary, the stems, the files' word index, the notes' indexes and who took part in what, from what is stored; loses nothing")
+    .description("rebuild the word index, its typo vocabulary, the stems, the files' word index, the notes' indexes and who took part in what, from what is stored, and drop the vectors no stored text uses; loses nothing")
     .action(async function () {
     const { renderer } = outputFor(this);
     const path = storePath(environmentOf(this).env ?? process.env);
@@ -443,6 +448,7 @@ const reindexCommand = (messenger) => destroysNothing(annotate(new Command("rebu
             ...buildStems(database, (note) => renderer.note(note), { force: true }),
             ...buildNoteIndex(database, (note) => renderer.note(note), { reset: true }),
             involvements: involvementStoreOver({ database, now: Date.now }).rebuild(),
+            unusedVectorsDropped: dropAllUnusedVectors(database),
         };
     });
     result(this, answer);

package/dist/cli/model-catalogue-command.js

@@ -15,7 +15,7 @@ export const MODEL_CATALOGUE_QUERY = defineQuery({
     entity: "model",
     backend: "native",
     fields: [
-        termField("kind", "enum", "kind:audio", { values: ["audio", "text"] }),
+        termField("kind", "enum", "kind:audio", { values: ["audio", "text", "reranker"] }),
         termField("id", "kw", "id:e5-small"),
         termField("downloaded", "bool", "downloaded:true"),
     ],

package/dist/cli/settings.d.ts

@@ -82,6 +82,8 @@ export interface Settings extends AISettings {
     searchCatchUp?: boolean;
     /** Search and `images embed` read the pictures themselves, with a model on this machine. Off until set. */
     pictureSearch?: boolean;
+    /** The reranker that orders `search messages --discover`'s first hits once it is downloaded, or `off`. */
+    searchReranker?: string;
     limit: number;
     /** Which page, 1-based. Per invocation only — a page number in a file is a setting nobody wants twice. */
     page: number;
@@ -271,6 +273,7 @@ export declare const settingsFor: <T extends object = Record<never, never>>(app:
             speech_model: v.OptionalSchema<v.StringSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
             picture_search: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: v.OptionalSchema<v.PicklistSchema<[...string[], "off"], (issue: v.BaseIssue<unknown>) => string>, undefined>;
             update_check: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
             skill_hint: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
         }, (issue: v.StrictObjectIssue) => string>, undefined>;
@@ -343,6 +346,7 @@ export declare const settingsFor: <T extends object = Record<never, never>>(app:
             speech_model: v.OptionalSchema<v.StringSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
             picture_search: v.OptionalSchema<v.BooleanSchema<(issue: v.BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: v.OptionalSchema<v.PicklistSchema<[...string[], "off"], (issue: v.BaseIssue<unknown>) => string>, undefined>;
         }, (issue: v.StrictObjectIssue) => string>, "has to be an object of profiles, by name">, {}>;
         personal: v.OptionalSchema<v.StrictObjectSchema<{
             readonly defaults: v.OptionalSchema<v.StrictObjectSchema<v.ObjectEntries, (issue: v.StrictObjectIssue) => string>, undefined>;

package/dist/cli/settings.js

@@ -3,6 +3,7 @@ import { CliError, configFilePath, resolvePaths, saveConfigFile } from "@wirecat
 import * as v from "valibot";
 import { AI_ENTRIES, resolveAISettings } from "../analysis/settings.js";
 import { parseDuration } from "../domain/duration.js";
+import { DEFAULT_RERANK_MODEL, RERANK_MODELS } from "../embeddings/models.js";
 import { envName, readOnlyLockOf } from "../ports/app.js";
 import { assertOverridesTighten, CEILING_KEY, LEVELS, layerPermissions, permissionOverrides, RESOURCES, } from "../sends/permissions.js";
 import { storePath } from "../store/path.js";
@@ -63,6 +64,7 @@ const SHARED_PROFILE_ENTRIES = {
     speech_model: v.optional(v.string(plain("has to be a model id from `models list`, in quotes"))),
     search_catch_up: v.optional(flag),
     picture_search: v.optional(flag),
+    search_reranker: v.optional(v.picklist([...RERANK_MODELS.map((model) => model.id), "off"], plain(`has to be off or a reranker id: ${RERANK_MODELS.map((model) => model.id).join(", ")}`))),
     ...AI_ENTRIES,
 };
 /**
@@ -228,6 +230,7 @@ export const settingsFor = (app, extension = {}) => {
         const senderColors = fromLayers("sender_colors", false);
         const searchCatchUp = fromLayers("search_catch_up", false);
         const pictureSearch = fromLayers("picture_search", false);
+        const searchReranker = fromLayers("search_reranker", DEFAULT_RERANK_MODEL);
         const keepRunsForDays = fromLayers("keep_runs_for_days", DEFAULT_KEEP_RUNS_FOR_DAYS);
         // Set where an agent runs unattended; the config file and `--permission` cannot undo it.
         const readOnlyLock = readOnlyLockOf(app, env);
@@ -275,6 +278,7 @@ export const settingsFor = (app, extension = {}) => {
             senderColors: senderColors.value,
             searchCatchUp: searchCatchUp.value,
             pictureSearch: pictureSearch.value,
+            searchReranker: searchReranker.value,
             limit: limit.value,
             page: flags.page ?? 1,
             all: flags.all === true,
@@ -308,6 +312,7 @@ export const settingsFor = (app, extension = {}) => {
                 senderColors: senderColors.from,
                 searchCatchUp: searchCatchUp.from,
                 pictureSearch: pictureSearch.from,
+                searchReranker: searchReranker.from,
                 record: record.from,
                 keepRunsForDays: keepRunsForDays.from,
                 sendsPerHour: sendsPerHour.from,

package/dist/cli/shared-command-exports.d.ts

@@ -87,6 +87,7 @@ export declare const configCommand: (app: import("./entry.js").AppIdentity, conf
             speech_model: import("valibot").OptionalSchema<import("valibot").StringSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             picture_search: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: import("valibot").OptionalSchema<import("valibot").PicklistSchema<[...string[], "off"], (issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             update_check: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             skill_hint: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
         }, (issue: import("valibot").StrictObjectIssue) => string>, undefined>;
@@ -159,6 +160,7 @@ export declare const configCommand: (app: import("./entry.js").AppIdentity, conf
             speech_model: import("valibot").OptionalSchema<import("valibot").StringSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             picture_search: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: import("valibot").OptionalSchema<import("valibot").PicklistSchema<[...string[], "off"], (issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
         }, (issue: import("valibot").StrictObjectIssue) => string>, "has to be an object of profiles, by name">, {}>;
         personal: import("valibot").OptionalSchema<import("valibot").StrictObjectSchema<{
             readonly defaults: import("valibot").OptionalSchema<import("valibot").StrictObjectSchema<import("valibot").ObjectEntries, (issue: import("valibot").StrictObjectIssue) => string>, undefined>;
@@ -286,6 +288,7 @@ export declare const completeCommand: (messenger: import("./index.js").Messenger
             speech_model: import("valibot").OptionalSchema<import("valibot").StringSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             picture_search: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: import("valibot").OptionalSchema<import("valibot").PicklistSchema<[...string[], "off"], (issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             update_check: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             skill_hint: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
         }, (issue: import("valibot").StrictObjectIssue) => string>, undefined>;
@@ -358,6 +361,7 @@ export declare const completeCommand: (messenger: import("./index.js").Messenger
             speech_model: import("valibot").OptionalSchema<import("valibot").StringSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             search_catch_up: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
             picture_search: import("valibot").OptionalSchema<import("valibot").BooleanSchema<(issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
+            search_reranker: import("valibot").OptionalSchema<import("valibot").PicklistSchema<[...string[], "off"], (issue: import("valibot").BaseIssue<unknown>) => string>, undefined>;
         }, (issue: import("valibot").StrictObjectIssue) => string>, "has to be an object of profiles, by name">, {}>;
         personal: import("valibot").OptionalSchema<import("valibot").StrictObjectSchema<{
             readonly defaults: import("valibot").OptionalSchema<import("valibot").StrictObjectSchema<import("valibot").ObjectEntries, (issue: import("valibot").StrictObjectIssue) => string>, undefined>;

package/dist/embeddings/child.js

@@ -1,18 +1,28 @@
 import { createInterface } from "node:readline";
 import { openEmbedder } from "./embed.js";
-const { model, directory, threads } = JSON.parse(process.argv[2] ?? "{}");
-const embedder = await openEmbedder(model, directory, { threads });
+import { openReranker } from "./rerank.js";
+const { model, directory, threads, rerank } = JSON.parse(process.argv[2] ?? "{}");
+const session = rerank
+    ? { reranker: await openReranker(model, directory, { threads }) }
+    : { embedder: await openEmbedder(model, directory, { threads }) };
 // stdout carries the answers alone, one JSON line each: anything else there breaks the parent's reading.
 const answer = (line) => process.stdout.write(`${JSON.stringify(line)}\n`);
 answer({ ready: true });
 for await (const line of createInterface({ input: process.stdin })) {
-    const { id, texts, kind } = JSON.parse(line);
+    const request = JSON.parse(line);
+    const { id } = request;
     try {
-        answer({ id, vectors: (await embedder.embed(texts, kind)).map((vector) => Array.from(vector)) });
+        if (session.reranker)
+            answer({ id, scores: await session.reranker.score(request.question, request.texts) });
+        else if (session.embedder)
+            answer({
+                id,
+                vectors: (await session.embedder.embed(request.texts, request.kind)).map((vector) => Array.from(vector)),
+            });
     }
     catch (error) {
         answer({ id, error: error instanceof Error ? error.message : String(error) });
     }
 }
-await embedder.close();
+await (session.reranker ?? session.embedder)?.close();
 process.exit(0);