@columenlabs/lumen-ai 0.1.141 → 0.1.143

package/README.md

@@ -163,7 +163,9 @@ The standard way to search an app's records (Columen's canonical setup, 2026-10-
 
 - **Filters** (tags, number and date ranges) are exact SQL.
 - **Words** are matched by an FTS5 index (`porter unicode61`, so "egg" finds "eggs"): `any`, `all`, `none` (leave out: "no milk"), and `in` to say which fields to look in.
-- **Meaning** comes from Vectorize: candidates for a loose ask, trimmed by the same filters and words. It is the next step on this part; until it lands, the model does the fuzzy part over what words and filters return.
+- **Meaning** comes from Vectorize, opt in per app: `like` ("something light for a hot day") asks the index for the nearest rows, and the same filters and words trim them.
+- **Worked-out fields**: a number, date or tag field may be the app's own SQL over the row (`sql: 'json_array_length(t.steps)'`) instead of a column.
+- **Counts**: the result carries `count` (all matches), `returned` and `same_name` (different rows that share a name), so the model says the right number. A search by meaning carries `nearest: true` and no `count`: its rows are the nearest few, not every row that fits, so the model never quotes them as a number of matches.
 
 ```ts
 import { d1Sql, defineFind, findSql, findTool } from '@columenlabs/lumen-ai/assistant';
@@ -188,3 +190,22 @@ plainTools([await findTool(notesFind, d1Sql(env.DB))]);
 - `findTool` makes `<name>_find` with a flat input schema (every provider takes it) and reads each tag field's values once to list them. The assistant tells the model to put a whole question in one find call.
 - `runFind(def, sql, query)` is the same query without the tool: bound parameters only, never model-written SQL. It gives up after 3 s.
 - `findSql(def)` gives the index's SQL. `create` is an external-content FTS5 table plus triggers that keep it in step on every insert, update and delete, whoever writes. `fill` indexes the rows already there, and `drop` takes it all out. Put `create` and `fill` in a migration (D1) or a budgeted background step (a Durable Object), never on a request. Until the index exists, words fall back to a plain text match and the result says so, so code can ship before its migration.
+
+Meaning, for the apps whose people ask loosely (a recipe book, a career log; not for exact records like card benefits):
+
+```ts
+import { meaningStep, vectorMeaning, workersEmbed } from '@columenlabs/lumen-ai/assistant';
+
+// On the definition: what a row means, and the column every write moves.
+meaning: { fields: ['title', 'body', 'tags'], changed: 'updated_at' },
+
+const embed = workersEmbed(env.AI);              // bge-base-en-v1.5: a 768-dimension cosine index
+await findTool(notesFind, d1Sql(env.DB), { meaning: vectorMeaning({ embed, index: env.NOTES_INDEX }) });
+
+// Off the request path (waitUntil after a write and when the tool opens, an alarm, a queue):
+ctx.waitUntil(meaningStep({ def: notesFind, sql: d1Sql(env.DB), embed, index: env.NOTES_INDEX, mark }));
+```
+
+- `meaningStep` embeds only the rows whose `changed` column moved past its bookmark (`mark`: get and put, kept by the app), at most 50 a step in one embedding call and one upsert, and takes rows the find hides (its `where`) out of the index. It resumes where it stopped and runs safely twice at once; `done` is false while rows are left.
+- A person's own rows get a `namespace` each (`vectorMeaning` and `meaningStep` take one), so one index serves one app.
+- When the index cannot answer in 3 s, `like` is left out, the rest of the find runs and the result says so.

package/dist/assistant/engine.js

@@ -960,7 +960,7 @@ export class Assistant {
             'Answer in short plain sentences. Use the tools for facts; never guess a number a tool can give you. Call several tools at once when they do not depend on each other.',
             ...([...fns.values()].some((f) => f.def.find)
                 ? [
-                    "Questions about an app's records go through its find tool (the tool named <app>_find): put every condition of the question in one call (filters, words to include or leave out, which fields to look in), ask only for the fields the answer needs, then judge the rest yourself from what it returns. Never list or open records one by one to filter them. Rows a find already returned in this chat you use again without asking for them again.",
+                    'Questions about an app\'s records go through its find tool (the tool named <app>_find): put every condition of the question in one call (filters, words to include or leave out, which fields to look in), ask only for the fields the answer needs, then judge the rest yourself from what it returns. Never list or open records one by one to filter them. A loose ask (a mood, an occasion, "something light", "good for a party") goes in its like field when the tool has one; then pick from what it returns yourself. How many match is the result\'s count: say that number (a like search has no count, only the nearest rows: never give a number of matches for it), and name each row in same_name, which are different records that share a name. Rows a find already returned you use again only when they answer the whole new question; a new condition (another ingredient, a kind of dish, how quick) is a new find call.',
                 ]
                 : []),
             ...([...fns.values()].some((f) => !f.draft && !f.write) && !fns.has(BATCH)
@@ -983,7 +983,7 @@ export class Assistant {
             const reach = this.above(main.tier);
             if (reach.length) {
                 const own = policy === 'any' ? reach : policy === 'next' ? reach.slice(0, 1) : [];
-                lines.push(`Tiers: you run on ${main.tier.label}. The escalate tool hands one task to a higher tier (${reach.map((t) => t.label).join(' or ')}) with the same tools; it works in its own section of the chat and returns a memo. It costs more than answering yourself.`, `Call it when the person names a tier or says "go deeper"${own.length ? `, and on your own (${own.map((t) => t.label).join(' or ')}) when the question needs judgement a lookup cannot settle (why "judgement"), after two of your tool calls failed (why "stuck"), or when a fix of a check still fails (why "check_failed")` : ''}. Lookups you answer yourself: reading or checking many records against a plain rule (an ingredient, a date, a number, a name) is a lookup, done with a find call or ${BATCH}, never escalated.`, 'Its task must carry every fact from this chat it needs: it does not see the chat. Then answer from its memo in your own short words.');
+                lines.push(`Tiers: you run on ${main.tier.label}. The escalate tool hands one task to a higher tier (${reach.map((t) => t.label).join(' or ')}) with the same tools; it works in its own section of the chat and returns a memo. It costs more than answering yourself.`, `Call it when the person names a tier or says "go deeper"${own.length ? `, and on your own (${own.map((t) => t.label).join(' or ')}) when the question needs judgement a lookup cannot settle (why "judgement"), after two of your tool calls failed (why "stuck"), or when a fix of a check still fails (why "check_failed")` : ''}. Lookups you answer yourself: reading or checking many records against a plain rule (an ingredient, a date, a number, a name) is a lookup, done with a find call or ${BATCH}, never escalated. Suggesting a few of an app's records for a loose ask (a dish for a hot day, something for a party) is a lookup too: one find call, then your own pick, never escalated.`, 'Its task must carry every fact from this chat it needs: it does not see the chat. Then answer from its memo in your own short words.');
             }
             if (advisor)
                 lines.push(`Advisor: the advisor tool (no input) asks ${advisor.session.tier.label} to read this chat and send you a plan; it runs no tools. Call it before a question that needs judgement or after a tool failed, then do the work yourself. At most ${ADVICE_MAX} times an answer. Ask the advisor before you escalate.`);

package/dist/assistant/find.d.ts

@@ -13,7 +13,13 @@
  *
  * Words are matched by the FTS5 index (stems: "egg" finds "eggs"). Until that index exists, words
  * fall back to a plain text match, so a build that lands before its index still answers.
+ *
+ * Meaning (opt in, `meaning` on the definition plus a `FindMeaning` given to `findTool`): `like`
+ * asks a vector index for the rows nearest a loose ask ("something light for a hot day") and the
+ * same SQL trims them, so filters and words still hold. `meaning.ts` gives the Vectorize index and
+ * the budgeted background step that keeps it in step.
  */
+import type { FindMeaning } from './meaning.js';
 import type { SqlRunner } from './store.js';
 import { type PlainTool } from './tools.js';
 /** text: searchable words. tag: an exact value to filter on (listed to the model). number, date: ranges. */
@@ -22,6 +28,11 @@ export interface FindField {
     kind: FindKind;
     /** The table's column, when it is named differently. */
     column?: string;
+    /**
+     * Not a column but the app's own SQL over the row, its columns as `t.<column>`
+     * (`json_array_length(t.steps)`): a number, date or tag worked out per row. Never text.
+     */
+    sql?: string;
     /** Returned unless the call names its fields. False keeps it searchable but out of the default. */
     show?: boolean;
     /** The column holds JSON (a list of ingredients): returned parsed. */
@@ -49,9 +60,20 @@ export interface FindDef {
     /** Rows a call returns by default (25) and at most (200). */
     limit?: number;
     max?: number;
+    /** The field that names a row (the first text field by default): rows that share one are flagged. */
+    title?: string;
+    /**
+     * Search by meaning: which fields make up what a row means (embedded in the background) and
+     * the column that changes on every write, so the index step picks up only what changed.
+     */
+    meaning?: {
+        fields: string[];
+        changed: string;
+    };
 }
 /** What the model sends: flat, so every provider takes the schema. Tag fields are their own keys. */
 export interface FindQuery {
+    like?: string;
     any?: string[];
     all?: string[];
     none?: string[];
@@ -64,14 +86,28 @@ export interface FindQuery {
     [tagOrRange: string]: unknown;
 }
 export interface FindResult {
-    /** Rows that match, all of them, before `limit`. */
-    count: number;
+    /**
+     * Rows that match, all of them, before `limit`. Say this number when asked how many. Left out
+     * of a search by meaning, whose rows are the nearest few, not every row that fits.
+     */
+    count?: number;
+    /** Set on a search by meaning: the rows are the nearest in meaning, nearest first, to pick from. */
+    nearest?: true;
+    /** Rows in this answer (at most `limit`). */
+    returned: number;
     rows: Record<string, unknown>[];
-    /** Set when words were matched without the word index (it is not set up yet). */
+    /** Different rows that share a name: each is its own row, so say both. */
+    same_name?: {
+        name: string;
+        ids: unknown[];
+    }[];
+    /** Set when a part of the query could not run as asked (no word index yet, meaning down). */
     note?: string;
 }
 /** Checks a definition once, where the app declares it. */
 export declare function defineFind(def: FindDef): FindDef;
+/** A field as SQL over the row aliased `t`: its column, or its own expression. */
+export declare const findRef: (def: FindDef, f: string) => string;
 /**
  * The SQL that sets up the word index: an external-content FTS5 table over the text columns, the
  * triggers that keep it in step on every insert, update and delete, and (`fill`) the one statement
@@ -83,10 +119,15 @@ export declare function findSql(def: FindDef): {
     fill: string;
     drop: string[];
 };
-/** Runs one find: one query, the rows' asked fields (JSON parsed) and how many match in all. */
-export declare function runFind(def: FindDef, sql: SqlRunner, q: FindQuery): Promise<FindResult>;
+/**
+ * Runs one find: one query, the rows' asked fields (JSON parsed) and how many match in all. With
+ * `like` and a `meaning`, the index's nearest rows come first and the same SQL trims them.
+ */
+export declare function runFind(def: FindDef, sql: SqlRunner, q: FindQuery, meaning?: FindMeaning): Promise<FindResult>;
 /**
  * The find tool for one table, opened for one answer: its tag values are read once (one small query)
  * so the model picks real ones. A read; `find` marks it so the assistant puts a whole question in it.
  */
-export declare function findTool(def: FindDef, sql: SqlRunner): Promise<PlainTool>;
+export declare function findTool(def: FindDef, sql: SqlRunner, opts?: {
+    meaning?: FindMeaning;
+}): Promise<PlainTool>;

package/dist/assistant/find.js

@@ -1,8 +1,22 @@
 import { ToolError, withDeadline } from './tools.js';
-const RESERVED = new Set(['any', 'all', 'none', 'in', 'ids', 'fields', 'sort', 'desc', 'limit']);
+const RESERVED = new Set([
+    'like',
+    'any',
+    'all',
+    'none',
+    'in',
+    'ids',
+    'fields',
+    'sort',
+    'desc',
+    'limit',
+]);
 const FIND_MS = 3_000;
 const TAG_VALUES = 60;
 const WORDS = 20;
+/** Rows nearest in meaning that `like` asks the index for, before the SQL trims them. */
+const NEAR_MIN = 20;
+const NEAR_MAX = 50;
 /** Checks a definition once, where the app declares it. */
 export function defineFind(def) {
     const ident = /^[A-Za-z_][A-Za-z0-9_]*$/;
@@ -14,12 +28,25 @@ export function defineFind(def) {
             throw new Error(`find ${def.name}: field "${n}" is not a plain name`);
         if (RESERVED.has(n))
             throw new Error(`find ${def.name}: field "${n}" is a reserved word`);
+        if (f.sql && f.kind === 'text')
+            throw new Error(`find ${def.name}: field "${n}" is text, which needs a column, not sql`);
     }
     if (!Object.values(def.fields).some((f) => f.kind === 'text'))
         throw new Error(`find ${def.name}: give at least one text field`);
+    if (def.title && def.fields[def.title]?.kind !== 'text')
+        throw new Error(`find ${def.name}: title "${def.title}" is not a text field`);
+    if (def.meaning) {
+        if (!ident.test(def.meaning.changed))
+            throw new Error(`find ${def.name}: meaning.changed "${def.meaning.changed}" is not a plain name`);
+        const bad = def.meaning.fields.filter((f) => !def.fields[f]);
+        if (bad.length || !def.meaning.fields.length)
+            throw new Error(`find ${def.name}: meaning.fields ${bad.join(', ') || '(none)'} are not fields`);
+    }
     return def;
 }
 const col = (def, f) => def.fields[f]?.column ?? f;
+/** A field as SQL over the row aliased `t`: its column, or its own expression. */
+export const findRef = (def, f) => def.fields[f]?.sql ?? `t.${col(def, f)}`;
 const ftsName = (def) => def.fts ?? `${def.table}_fts`;
 const textFields = (def) => Object.entries(def.fields)
     .filter(([, f]) => f.kind === 'text')
@@ -98,6 +125,7 @@ function checked(def, q) {
     const max = def.max ?? 200;
     const limit = Math.max(1, Math.min(max, Math.trunc(Number(q.limit ?? def.limit ?? 25)) || 25));
     return {
+        like: typeof q.like === 'string' ? q.like.replace(/\s+/g, ' ').trim().slice(0, 300) : '',
         any: words(q.any),
         all: words(q.all),
         none: words(q.none),
@@ -118,8 +146,11 @@ function filterKeys(def) {
                 ? [`${n}_from`, `${n}_to`]
                 : []);
 }
-/** One find as SQL and its bound values. `fts` false matches words with LIKE (no index yet). */
-function compile(def, q, fts) {
+/**
+ * One find as SQL and its bound values. `fts` false matches words with LIKE (no index yet). `near`
+ * is the rows nearest in meaning, nearest first: only they can match, in that order.
+ */
+function compile(def, q, fts, near) {
     const c = checked(def, q);
     const t = 't';
     const where = [];
@@ -127,7 +158,7 @@ function compile(def, q, fts) {
     if (def.where)
         where.push(`(${def.where})`);
     for (const [n, f] of Object.entries(def.fields)) {
-        const column = `${t}.${col(def, n)}`;
+        const column = findRef(def, n);
         if (f.kind === 'tag') {
             const vals = strings(q[n]).slice(0, 50);
             if (vals.length) {
@@ -153,6 +184,13 @@ function compile(def, q, fts) {
         where.push(`${t}.${def.key} IN (${c.ids.map(() => '?').join(', ')})`);
         args.push(...c.ids);
     }
+    let nearOrder = '';
+    if (near?.length) {
+        where.push(`${t}.${def.key} IN (${near.map(() => '?').join(', ')})`);
+        args.push(...near);
+        // Literal integers, not bound: the ORDER BY comes after the WHERE's bound values.
+        nearOrder = `CASE ${t}.${def.key} ${near.map((id, i) => `WHEN ${Math.trunc(id)} THEN ${i}`).join(' ')} END`;
+    }
     const fname = ftsName(def);
     const cols = c.in.map((f) => col(def, f));
     const scope = `{${cols.join(' ')}} : `;
@@ -187,8 +225,9 @@ function compile(def, q, fts) {
             where.push(`NOT ${like(w)}`);
     }
     const order = c.sort
-        ? `${t}.${col(def, c.sort)} ${c.desc ? 'DESC' : 'ASC'}`
-        : rank ||
+        ? `${findRef(def, c.sort)} ${c.desc ? 'DESC' : 'ASC'}`
+        : nearOrder ||
+            rank ||
             (def.order
                 ? def.order
                     .split(',')
@@ -196,16 +235,42 @@ function compile(def, q, fts) {
                     .join(', ')
                 : `${t}.${def.key}`);
     const select = [...new Set([def.key, ...c.fields])]
-        .map((f) => (f === def.key ? `${t}.${def.key} AS ${def.key}` : `${t}.${col(def, f)} AS ${f}`))
+        .map((f) => (f === def.key ? `${t}.${def.key} AS ${def.key}` : `${findRef(def, f)} AS ${f}`))
         .join(', ');
     const sql = `SELECT ${select}, COUNT(*) OVER () AS _count FROM ${def.table} ${t}${join}${where.length ? ` WHERE ${where.join(' AND ')}` : ''} ORDER BY ${order} LIMIT ${c.limit}`;
-    return { sql, args, fields: c.fields, words: c.any.length + c.all.length + c.none.length > 0 };
+    return {
+        sql,
+        args,
+        fields: c.fields,
+        like: c.like,
+        limit: c.limit,
+        words: c.any.length + c.all.length + c.none.length > 0,
+    };
 }
 const missingIndex = (e) => e instanceof Error && /no such table|no such module|fts5/i.test(e.message);
-/** Runs one find: one query, the rows' asked fields (JSON parsed) and how many match in all. */
-export async function runFind(def, sql, q) {
+/**
+ * Runs one find: one query, the rows' asked fields (JSON parsed) and how many match in all. With
+ * `like` and a `meaning`, the index's nearest rows come first and the same SQL trims them.
+ */
+export async function runFind(def, sql, q, meaning) {
     let built = compile(def, q, true);
-    let note;
+    const notes = [];
+    let near;
+    if (built.like) {
+        if (!meaning || !def.meaning)
+            throw new ToolError('like: this find has no search by meaning; use words instead.');
+        const k = Math.min(NEAR_MAX, Math.max(NEAR_MIN, built.limit * 2));
+        try {
+            near = await withDeadline(`${def.name} meaning`, FIND_MS, meaning.near(built.like, k));
+        }
+        catch {
+            near = undefined;
+        }
+        if (near?.length)
+            built = compile(def, q, true, near);
+        else
+            notes.push('Search by meaning is not available now: like was left out, the rest applied.');
+    }
     let rows;
     try {
         rows = await withDeadline(`${def.name} find`, FIND_MS, sql.all(built.sql, ...built.args));
@@ -213,41 +278,65 @@ export async function runFind(def, sql, q) {
     catch (e) {
         if (!built.words || !missingIndex(e))
             throw e;
-        built = compile(def, q, false);
-        note = 'Words were matched as plain text: the word index is not set up yet.';
+        built = compile(def, q, false, near);
+        notes.push('Words were matched as plain text: the word index is not set up yet.');
         rows = await withDeadline(`${def.name} find`, FIND_MS, sql.all(built.sql, ...built.args));
     }
     const count = Number(rows[0]?._count ?? 0);
-    return {
-        count,
-        rows: rows.map((r) => {
-            const out = { [def.key]: r[def.key] };
-            for (const f of built.fields) {
-                const v = r[f];
-                if (def.fields[f]?.json && typeof v === 'string') {
-                    try {
-                        out[f] = JSON.parse(v);
-                    }
-                    catch {
-                        out[f] = v;
-                    }
+    const out = rows.map((r) => {
+        const row = { [def.key]: r[def.key] };
+        for (const f of built.fields) {
+            const v = r[f];
+            if (def.fields[f]?.json && typeof v === 'string') {
+                try {
+                    row[f] = JSON.parse(v);
+                }
+                catch {
+                    row[f] = v;
                 }
-                else
-                    out[f] = v;
             }
-            return out;
-        }),
-        ...(note ? { note } : {}),
+            else
+                row[f] = v;
+        }
+        return row;
+    });
+    const same = sameNames(def, out, built.fields);
+    const byMeaning = !!near?.length;
+    if (byMeaning)
+        notes.push('These are the nearest in meaning, nearest first, not every row that fits: pick from them and do not say how many match.');
+    return {
+        ...(byMeaning ? { nearest: true } : { count }),
+        returned: out.length,
+        rows: out,
+        ...(same.length ? { same_name: same } : {}),
+        ...(notes.length ? { note: notes.join(' ') } : {}),
     };
 }
+/** Rows in the answer that share a name, so the model counts and lists each of them. */
+function sameNames(def, rows, fields) {
+    const title = def.title ?? textFields(def)[0];
+    if (!title || !fields.includes(title))
+        return [];
+    const by = new Map();
+    for (const r of rows) {
+        const name = String(r[title] ?? '').trim();
+        if (!name)
+            continue;
+        const k = name.toLowerCase();
+        const g = by.get(k) ?? { name, ids: [] };
+        g.ids.push(r[def.key]);
+        by.set(k, g);
+    }
+    return [...by.values()].filter((g) => g.ids.length > 1);
+}
 /** The values a tag field holds, to list in the tool's description. */
 async function tagValues(def, sql, field) {
-    const c = col(def, field);
-    const rows = await sql.all(`SELECT DISTINCT ${c} AS v FROM ${def.table}${def.where ? ` WHERE ${def.where}` : ''} ORDER BY ${c} LIMIT ${TAG_VALUES + 1}`);
+    const c = findRef(def, field);
+    const rows = await sql.all(`SELECT DISTINCT ${c} AS v FROM ${def.table} t${def.where ? ` WHERE ${def.where}` : ''} ORDER BY v LIMIT ${TAG_VALUES + 1}`);
     return rows.map((r) => String(r.v ?? '')).filter(Boolean);
 }
 /** The tool's flat JSON Schema, from the fields. */
-function schemaOf(def, tags) {
+function schemaOf(def, tags, meaning) {
     const names = Object.keys(def.fields);
     const text = textFields(def);
     const list = (description, items = { type: 'string' }) => ({
@@ -256,6 +345,14 @@ function schemaOf(def, tags) {
         description,
     });
     const props = {
+        ...(meaning
+            ? {
+                like: {
+                    type: 'string',
+                    description: `A loose ask in plain words ("something light for a hot day", "good for a party"): the ${def.label}s nearest in meaning come first, and every other condition still applies. Use it for a mood, an occasion or a kind of thing, not for one exact word.`,
+                },
+            }
+            : {}),
         any: list('Rows with at least one of these words or phrases.'),
         all: list('Rows with every one of these words or phrases.'),
         none: list('Leave out rows with any of these words or phrases (an ingredient to avoid).'),
@@ -300,7 +397,8 @@ function schemaOf(def, tags) {
  * The find tool for one table, opened for one answer: its tag values are read once (one small query)
  * so the model picks real ones. A read; `find` marks it so the assistant puts a whole question in it.
  */
-export async function findTool(def, sql) {
+export async function findTool(def, sql, opts = {}) {
+    const meaning = def.meaning ? opts.meaning : undefined;
     const tags = {};
     await Promise.all(Object.entries(def.fields)
         .filter(([, f]) => f.kind === 'tag')
@@ -312,10 +410,10 @@ export async function findTool(def, sql) {
         .join(', ');
     return {
         name: `${def.name}_find`,
-        description: `Finds ${def.label}s in one query${def.about ? ` (${def.about})` : ''}: put every condition of the question in one call (filters, words to include or leave out, which fields to look in) and ask only for the fields the answer needs. Fields: ${fields}. Returns the matching ${def.label}s and how many match in all.`,
-        inputSchema: schemaOf(def, tags),
+        description: `Finds ${def.label}s in one query${def.about ? ` (${def.about})` : ''}: put every condition of the question in one call (filters, words to include or leave out, which fields to look in${meaning ? ', and like for a loose ask' : ''}) and ask only for the fields the answer needs. Fields: ${fields}. Returns the matching ${def.label}s, how many match in all (count; a like search returns the nearest instead, marked nearest, with no count) and any that share a name (same_name).`,
+        inputSchema: schemaOf(def, tags, Boolean(meaning)),
         read: true,
         find: true,
-        run: (args) => runFind(def, sql, (args ?? {})),
+        run: (args) => runFind(def, sql, (args ?? {}), meaning),
     };
 }

package/dist/assistant/index.d.ts

@@ -16,6 +16,7 @@ export * from './bearings.js';
 export * from './chatgpt.js';
 export * from './engine.js';
 export * from './find.js';
+export * from './meaning.js';
 export * from './models.js';
 export * from './routes.js';
 export { DEFAULT_CAP, type KeyDoc, keyDocName, type KeyTest, monthIn, type Paying, type PersonDoc, type SpendDoc, State, type TokenDoc, waitText, } from './state.js';

package/dist/assistant/index.js

@@ -16,6 +16,7 @@ export * from './bearings.js';
 export * from './chatgpt.js';
 export * from './engine.js';
 export * from './find.js';
+export * from './meaning.js';
 export * from './models.js';
 export * from './routes.js';
 export { DEFAULT_CAP, keyDocName, monthIn, State, waitText, } from './state.js';

package/dist/assistant/meaning.d.ts

@@ -0,0 +1,87 @@
+/**
+ * Search by meaning for a find tool: a vector index of what each row means, asked for the rows
+ * nearest a loose ask, which the find's own SQL then trims (filters, words, rows the person never
+ * sees). Opt in per app: `meaning` on the definition, then
+ *
+ *   const meaning = vectorMeaning({ embed: workersEmbed(env.AI), index: env.NOTES_INDEX });
+ *   await findTool(notesFind, d1Sql(env.DB), { meaning });
+ *
+ * The index is kept in step by `meaningStep`, a small budgeted background step (`waitUntil`, an
+ * alarm or a queue) that embeds only the rows whose `changed` column moved past its bookmark, so
+ * it never runs on the request path and resumes wherever it stopped, whoever wrote the rows.
+ */
+import { type FindDef } from './find.js';
+import type { SqlRunner } from './store.js';
+/** The rows nearest in meaning to a loose ask, by key, nearest first. */
+export interface FindMeaning {
+    near(text: string, k: number): Promise<number[]>;
+}
+/** Texts to vectors, in order. */
+export type Embed = (texts: string[], as: 'query' | 'passage') => Promise<number[][]>;
+/** What a vector index needs (Cloudflare Vectorize has it). */
+export interface VectorIndex {
+    query(vector: number[], options: {
+        topK: number;
+        namespace?: string;
+    }): Promise<{
+        matches: {
+            id: string;
+            score: number;
+        }[];
+    }>;
+    upsert(vectors: {
+        id: string;
+        values: number[];
+        namespace?: string;
+    }[]): Promise<unknown>;
+    deleteByIds(ids: string[]): Promise<unknown>;
+}
+/** Workers AI's text embeddings (`AI.run`). */
+export interface EmbedAi {
+    run(model: string, input: {
+        text: string[];
+    }): Promise<unknown>;
+}
+export declare const EMBED_MODEL = "@cf/baai/bge-base-en-v1.5";
+/** Embeddings from Workers AI: bge-base-en-v1.5 (768 dimensions, cosine) unless `model` says. */
+export declare function workersEmbed(ai: EmbedAi, model?: string): Embed;
+/** Search by meaning over a vector index: one embedding, one query. */
+export declare function vectorMeaning(o: {
+    embed: Embed;
+    index: VectorIndex;
+    /** One per person when rows are a person's own. */
+    namespace?: string;
+}): FindMeaning;
+/** Where the step stopped: the last row's `changed` value and key. */
+export interface MeaningMark {
+    at: string | number;
+    key: number;
+}
+export interface MeaningStepOptions {
+    def: FindDef;
+    sql: SqlRunner;
+    embed: Embed;
+    index: VectorIndex;
+    namespace?: string;
+    /** The bookmark, kept by the app (a settings row, its object's storage). */
+    mark: {
+        get(): Promise<MeaningMark | null>;
+        put(m: MeaningMark): Promise<void>;
+    };
+    /** Rows one step reads at most (50). */
+    rows?: number;
+}
+/** What a row means, as text: its meaning fields with their names, JSON lists flattened. */
+export declare function meaningText(def: FindDef, row: Record<string, unknown>): string;
+/**
+ * One budgeted step of keeping the index in step: the next rows (at most `rows`) whose `changed`
+ * column moved past the bookmark, embedded in one call and written in one upsert; rows the find
+ * never shows (its `where`) leave the index. Safe to run twice at once (writes are idempotent).
+ * `done` is false while rows are left: run it again (another `waitUntil`, the next alarm).
+ * A row deleted outright keeps its vector, which the find's SQL never matches.
+ */
+export declare function meaningStep(o: MeaningStepOptions): Promise<{
+    indexed: number;
+    removed: number;
+    done: boolean;
+}>;

package/dist/assistant/meaning.js

@@ -0,0 +1,115 @@
+/**
+ * Search by meaning for a find tool: a vector index of what each row means, asked for the rows
+ * nearest a loose ask, which the find's own SQL then trims (filters, words, rows the person never
+ * sees). Opt in per app: `meaning` on the definition, then
+ *
+ *   const meaning = vectorMeaning({ embed: workersEmbed(env.AI), index: env.NOTES_INDEX });
+ *   await findTool(notesFind, d1Sql(env.DB), { meaning });
+ *
+ * The index is kept in step by `meaningStep`, a small budgeted background step (`waitUntil`, an
+ * alarm or a queue) that embeds only the rows whose `changed` column moved past its bookmark, so
+ * it never runs on the request path and resumes wherever it stopped, whoever wrote the rows.
+ */
+import { findRef } from './find.js';
+import { withDeadline } from './tools.js';
+export const EMBED_MODEL = '@cf/baai/bge-base-en-v1.5';
+/** bge models are asymmetric: a query (not a passage) needs this, or scores bunch together. */
+const BGE_QUERY = 'Represent this sentence for searching relevant passages: ';
+/** Texts one embedding call takes. */
+const EMBED_BATCH = 100;
+/** Embeddings from Workers AI: bge-base-en-v1.5 (768 dimensions, cosine) unless `model` says. */
+export function workersEmbed(ai, model = EMBED_MODEL) {
+    const prefix = /\/bge-/.test(model) ? BGE_QUERY : '';
+    return async (texts, as) => {
+        const out = [];
+        for (let i = 0; i < texts.length; i += EMBED_BATCH) {
+            const batch = texts.slice(i, i + EMBED_BATCH).map((t) => (as === 'query' ? prefix + t : t));
+            const res = (await ai.run(model, { text: batch }));
+            if (!res.data || res.data.length !== batch.length)
+                throw new Error('embedding failed');
+            out.push(...res.data);
+        }
+        return out;
+    };
+}
+/**
+ * A vector id: the row's key, after the namespace when there is one, so ids stay unique across
+ * people in one index (a Durable Object app gives each person a namespace).
+ */
+const vectorId = (key, namespace) => namespace ? `${namespace}:${key}` : `${key}`;
+const keyOf = (id) => Number(id.slice(id.lastIndexOf(':') + 1));
+/** Search by meaning over a vector index: one embedding, one query. */
+export function vectorMeaning(o) {
+    return {
+        async near(text, k) {
+            const [vector] = await o.embed([text], 'query');
+            if (!vector)
+                return [];
+            const res = await o.index.query(vector, {
+                topK: k,
+                ...(o.namespace ? { namespace: o.namespace } : {}),
+            });
+            return res.matches.map((m) => keyOf(m.id)).filter(Number.isInteger);
+        },
+    };
+}
+/** What a row means, as text: its meaning fields with their names, JSON lists flattened. */
+export function meaningText(def, row) {
+    const parts = [];
+    for (const f of def.meaning?.fields ?? []) {
+        let v = row[f];
+        if (def.fields[f]?.json && typeof v === 'string') {
+            try {
+                v = JSON.parse(v);
+            }
+            catch { }
+        }
+        const flat = (x) => Array.isArray(x)
+            ? x.map(flat).filter(Boolean).join('; ')
+            : x && typeof x === 'object'
+                ? Object.values(x).map(flat).filter(Boolean).join(': ')
+                : x === null || x === undefined
+                    ? ''
+                    : String(x);
+        const text = flat(v).replace(/\s+/g, ' ').trim();
+        if (text)
+            parts.push(`${f}: ${text}`);
+    }
+    return parts.join('\n').slice(0, 2_000);
+}
+const STEP_MS = 20_000;
+/**
+ * One budgeted step of keeping the index in step: the next rows (at most `rows`) whose `changed`
+ * column moved past the bookmark, embedded in one call and written in one upsert; rows the find
+ * never shows (its `where`) leave the index. Safe to run twice at once (writes are idempotent).
+ * `done` is false while rows are left: run it again (another `waitUntil`, the next alarm).
+ * A row deleted outright keeps its vector, which the find's SQL never matches.
+ */
+export async function meaningStep(o) {
+    const { def } = o;
+    if (!def.meaning)
+        return { indexed: 0, removed: 0, done: true };
+    const n = Math.max(1, Math.min(EMBED_BATCH, o.rows ?? 50));
+    const changed = `t.${def.meaning.changed}`;
+    const mark = await o.mark.get();
+    const cols = [...new Set(def.meaning.fields)].map((f) => `${findRef(def, f)} AS ${f}`).join(', ');
+    const live = def.where ? `CASE WHEN (${def.where}) THEN 1 ELSE 0 END` : '1';
+    const rows = await withDeadline(`${def.name} meaning`, STEP_MS, o.sql.all(`SELECT t.${def.key} AS _key, ${changed} AS _at, ${live} AS _live, ${cols} FROM ${def.table} t${mark ? ` WHERE (${changed} > ? OR (${changed} = ? AND t.${def.key} > ?))` : ''} ORDER BY ${changed}, t.${def.key} LIMIT ${n}`, ...(mark ? [mark.at, mark.at, mark.key] : [])));
+    if (!rows.length)
+        return { indexed: 0, removed: 0, done: true };
+    const keep = rows.filter((r) => Number(r._live) === 1);
+    const gone = rows.filter((r) => Number(r._live) !== 1);
+    if (keep.length) {
+        const vectors = await withDeadline(`${def.name} embed`, STEP_MS, o.embed(keep.map((r) => meaningText(def, r)), 'passage'));
+        await o.index.upsert(keep.map((r, i) => ({
+            id: vectorId(Number(r._key), o.namespace),
+            values: vectors[i] ?? [],
+            ...(o.namespace ? { namespace: o.namespace } : {}),
+        })));
+    }
+    if (gone.length)
+        await o.index.deleteByIds(gone.map((r) => vectorId(Number(r._key), o.namespace)));
+    const last = rows[rows.length - 1];
+    await o.mark.put({ at: last._at, key: Number(last._key) });
+    return { indexed: keep.length, removed: gone.length, done: rows.length < n };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@columenlabs/lumen-ai",
-  "version": "0.1.141",
+  "version": "0.1.143",
   "description": "Lumen's assistant and agent parts: a chat assistant's server (Claude tiers, tools over MCP, the web, hand off, spend caps), serving an MCP server from a Worker, read-only tool sets and metered tool calls.",
   "license": "MIT",
   "type": "module",
@@ -41,8 +41,8 @@
   },
   "dependencies": {
     "@anthropic-ai/sdk": "0.133.0",
-    "@columenlabs/lumen-ops": "0.1.141",
-    "@columenlabs/lumen-ui-types": "0.1.141",
+    "@columenlabs/lumen-ops": "0.1.143",
+    "@columenlabs/lumen-ui-types": "0.1.143",
     "openai": "7.32.0",
     "zod": "^4.3.6"
   },