@zukhruf/embeddings 0.0.0-stage → 0.3.47

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ezzabuzaid
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,77 @@
-# Temporary Holding Version
+# @zukhruf/embeddings
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A program that compares texts by meaning needs a vector for each text, and the model call that makes the vector is slow. This package keeps the vectors of documents in a SQLite database. An upsert calls the model only for new or changed text. A search gives the nearest documents of a document, by exact cosine similarity, from the documents published near it. This kind of package is also known as a vector store.
+
+The words in these documents have one meaning each. See the glossary in [CONTEXT.md](./CONTEXT.md).
+
+## Use it
+
+This project is an experiment. It uses `node:sqlite` and an embedding model of the [AI SDK](https://ai-sdk.dev). The package does not start a model or a server. Give it a model that is ready.
+
+```sh
+npm install @zukhruf/embeddings ai
+```
+
+```ts
+import { DatabaseSync } from 'node:sqlite';
+
+import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
+
+import { Collection } from '@zukhruf/embeddings';
+
+const local = createOpenAICompatible({
+  name: 'llama.cpp',
+  baseURL: 'http://127.0.0.1:8080/v1',
+});
+using database = new DatabaseSync('embeddings.sqlite');
+const collection = new Collection({
+  database,
+  model: local.embeddingModel('embeddinggemma'),
+});
+
+await collection.upsert([
+  {
+    id: 'a',
+    text: 'Storm closes the port',
+    publishedAt: new Date('2026-10-20'),
+  },
+  {
+    id: 'b',
+    text: 'Port shut as storm hits',
+    publishedAt: new Date('2026-10-21'),
+  },
+  { id: 'c', text: 'Port opens again', publishedAt: new Date('2026-11-30') },
+]);
+
+collection.nearest('a', { k: 10, withinDays: 7 }); // [{ id: 'b', similarity: … }]
+collection.similarity('a', 'c');
+```
+
+## API
+
+- `new Collection({ database, model })`. The caller opens and closes `database`. The collection makes its table `embedding_document` in it. `model` is an AI SDK embedding model object.
+- `upsert(documents)` stores each document. It calls the model, through `embedMany`, only for a document whose text or model changed. It stores the new publish date of each document.
+- `nearest(id, { k, withinDays })` gives at most `k` neighbours of the document, with the highest similarity first. A neighbour has a publish date at most `withinDays` days before or after the publish date of the document. Equal similarities sort by id.
+- `neighbourGraph({ from, to, k, withinDays })` gives an iterator of `[id, neighbours]` entries, one for each document published from `from` to `to`, in the order of the publish dates and then of the ids. The neighbours of each entry are equal to the result of `nearest(id, { k, withinDays })`. A document outside the range has no entry, but it can be a neighbour of a document in the range. The call reads and decodes each vector one time ([ADR 0004](./docs/adr/0004-a-neighbour-graph-decodes-each-vector-once.md)). Use it when you need the neighbours of many documents.
+- `similarity(a, b)` gives the cosine similarity of two stored documents.
+
+## Errors
+
+- `UnknownDocumentError`: `nearest` or `similarity` got an id that the collection does not store.
+- `DimensionMismatchError`: the model gave a vector with a dimension that is not the dimension of the stored vectors. The upsert stores none of its documents. To change to a model with another dimension, upsert all documents in one call.
+
+## Limits
+
+- One database holds one collection.
+- The collection stores the vector of the text, not the text. The caller chooses the text, for example a title and the first sentences.
+- A document has one vector. A long text gives one mean vector, and its details become a small part of it.
+- The search reads each document in the window ([ADR 0001](./docs/adr/0001-search-compares-each-document-in-the-window.md)). A window of many thousand documents is slow. A neighbour graph decodes each vector once, but it still calculates a similarity for each pair of documents in a window.
+- After a model change, upsert all documents again ([ADR 0002](./docs/adr/0002-the-hash-includes-the-model.md)).
+
+## Decisions
+
+- [ADR 0001: A search compares each document in the window, with no index](./docs/adr/0001-search-compares-each-document-in-the-window.md)
+- [ADR 0002: The hash includes the model](./docs/adr/0002-the-hash-includes-the-model.md)
+- [ADR 0003: All vectors of a collection have one dimension](./docs/adr/0003-all-vectors-of-a-collection-have-one-dimension.md)
+- [ADR 0004: A neighbour graph decodes each vector once](./docs/adr/0004-a-neighbour-graph-decodes-each-vector-once.md)
+- [Code copied from @deepagents/retrieval](./docs/copied-from-deepagents-retrieval.md)

package/dist/collection.d.ts

@@ -0,0 +1,32 @@
+import type { DatabaseSync } from 'node:sqlite';
+import { type EmbeddingModel } from 'ai';
+export type Document = {
+    readonly id: string;
+    readonly text: string;
+    readonly publishedAt: Date;
+};
+export type Neighbour = {
+    readonly id: string;
+    readonly similarity: number;
+};
+export type CollectionOptions = {
+    readonly database: DatabaseSync;
+    readonly model: Exclude<EmbeddingModel, string>;
+};
+export declare class Collection {
+    #private;
+    constructor(options: CollectionOptions);
+    upsert(documents: readonly Document[]): Promise<void>;
+    nearest(id: string, options: {
+        readonly k: number;
+        readonly withinDays: number;
+    }): Neighbour[];
+    neighbourGraph(options: {
+        readonly from: Date;
+        readonly to: Date;
+        readonly k: number;
+        readonly withinDays: number;
+    }): IteratorObject<[string, Neighbour[]]>;
+    similarity(a: string, b: string): number;
+}
+//# sourceMappingURL=collection.d.ts.map

package/dist/collection.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"collection.d.ts","sourceRoot":"","sources":["../src/collection.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAEhD,OAAO,EAAE,KAAK,cAAc,EAAa,MAAM,IAAI,CAAC;AAWpD,MAAM,MAAM,QAAQ,GAAG;IACrB,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IAEpB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;CAC5B,CAAC;AAEF,MAAM,MAAM,SAAS,GAAG;IACtB,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;CAC7B,CAAC;AAEF,MAAM,MAAM,iBAAiB,GAAG;IAE9B,QAAQ,CAAC,QAAQ,EAAE,YAAY,CAAC;IAGhC,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC,cAAc,EAAE,MAAM,CAAC,CAAC;CACjD,CAAC;AAKF,qBAAa,UAAU;;gBAIT,OAAO,EAAE,iBAAiB;IAQhC,MAAM,CAAC,SAAS,EAAE,SAAS,QAAQ,EAAE,GAAG,OAAO,CAAC,IAAI,CAAC;IAsD3D,OAAO,CACL,EAAE,EAAE,MAAM,EACV,OAAO,EAAE;QAAE,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAA;KAAE,GAC3D,SAAS,EAAE;IAoBd,cAAc,CAAC,OAAO,EAAE;QACtB,QAAQ,CAAC,IAAI,EAAE,IAAI,CAAC;QACpB,QAAQ,CAAC,EAAE,EAAE,IAAI,CAAC;QAClB,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC;QACnB,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;KAC7B,GAAG,cAAc,CAAC,CAAC,MAAM,EAAE,SAAS,EAAE,CAAC,CAAC;IAgCzC,UAAU,CAAC,CAAC,EAAE,MAAM,EAAE,CAAC,EAAE,MAAM,GAAG,MAAM;CAqDzC"}

package/dist/collection.js

@@ -0,0 +1,159 @@
+import { createHash } from 'node:crypto';
+import { embedMany } from 'ai';
+import { transaction } from '@zukhruf/sqlite-transaction';
+import { DimensionMismatchError } from "./dimension-mismatch-error.js";
+import { collectionSchema } from "./schema.js";
+import { UnknownDocumentError } from "./unknown-document-error.js";
+import { UnitVector } from "./vector.js";
+const DAY_MS = 86_400_000;
+// The embeddings of documents, one vector for each document, in a SQLite
+// database. A search compares the document with each document published
+// inside the window, with no index (docs/adr/0001).
+export class Collection {
+    #database;
+    #model;
+    constructor(options) {
+        this.#database = options.database;
+        this.#model = options.model;
+        collectionSchema.apply(this.#database);
+    }
+    // Embeds the documents whose text or model changed since their last upsert,
+    // and stores the publish date of each document.
+    async upsert(documents) {
+        const stored = this.#database.prepare('SELECT "hash" FROM "embedding_document" WHERE "id" = ?');
+        const hashed = documents.map((document) => ({
+            document,
+            hash: this.#hash(document.text),
+        }));
+        const { changed = [], unchanged = [] } = Object.groupBy(hashed, ({ document, hash }) => stored.get(document.id)?.hash === hash ? 'unchanged' : 'changed');
+        const vectors = await this.#embed(changed.map(({ document }) => document.text));
+        transaction(this.#database, () => {
+            const write = this.#database.prepare(`INSERT INTO "embedding_document" ("id", "hash", "published_at", "vector") VALUES (?, ?, ?, ?)
+          ON CONFLICT ("id") DO UPDATE SET "hash" = excluded."hash", "published_at" = excluded."published_at", "vector" = excluded."vector"`);
+            for (const [index, { document, hash }] of changed.entries()) {
+                const vector = vectors[index];
+                if (vector === undefined)
+                    throw new Error(`The model gave ${vectors.length} embeddings for ${changed.length} texts.`);
+                write.run(document.id, hash, document.publishedAt.getTime(), vector.toBlob());
+            }
+            const move = this.#database.prepare('UPDATE "embedding_document" SET "published_at" = ? WHERE "id" = ?');
+            for (const { document } of unchanged)
+                move.run(document.publishedAt.getTime(), document.id);
+            this.#checkDimension(changed.map(({ document }) => document.id), vectors);
+        }, { mode: 'immediate' });
+    }
+    // The k documents most similar to the document, by cosine similarity, from
+    // those published at most withinDays before or after it. The document
+    // itself is not one of them. Equal similarities sort by id.
+    nearest(id, options) {
+        const { vector, publishedAt } = this.#read(id);
+        const window = options.withinDays * DAY_MS;
+        const candidates = this.#database
+            .prepare('SELECT "id", "vector" FROM "embedding_document" WHERE "published_at" BETWEEN ? AND ? AND "id" != ?')
+            .all(publishedAt - window, publishedAt + window, id)
+            .map((row) => ({
+            id: String(row.id),
+            vector: UnitVector.fromBlob(blob(row.vector)),
+        }));
+        return ranked(vector, candidates, options.k);
+    }
+    // The k nearest neighbours of each document published from `from` to `to`.
+    // Each entry is what nearest gives for the document with the same k and
+    // withinDays. The call reads and decodes each vector once
+    // (docs/adr/0004). The entries come in the order of the publish dates, then
+    // of the ids.
+    neighbourGraph(options) {
+        const from = options.from.getTime();
+        const to = options.to.getTime();
+        const window = options.withinDays * DAY_MS;
+        const documents = this.#database
+            .prepare('SELECT "id", "published_at", "vector" FROM "embedding_document" WHERE "published_at" BETWEEN ? AND ? ORDER BY "published_at", "id"')
+            .all(from - window, to + window)
+            .map((row) => ({
+            id: String(row.id),
+            publishedAt: Number(row.published_at),
+            vector: UnitVector.fromBlob(blob(row.vector)),
+        }));
+        return documents
+            .values()
+            .filter(({ publishedAt }) => publishedAt >= from && publishedAt <= to)
+            .map((document) => [
+            document.id,
+            ranked(document.vector, documents.filter((other) => other.id !== document.id &&
+                Math.abs(other.publishedAt - document.publishedAt) <= window), options.k),
+        ]);
+    }
+    // The cosine similarity of two stored documents.
+    similarity(a, b) {
+        return this.#read(a).vector.cosine(this.#read(b).vector);
+    }
+    // No texts make no call to the model.
+    async #embed(texts) {
+        if (texts.length === 0)
+            return [];
+        const { embeddings } = await embedMany({
+            model: this.#model,
+            values: texts,
+        });
+        return embeddings.map((embedding) => UnitVector.normalize(embedding));
+    }
+    // The model is part of the hash, so a new model embeds the text again.
+    #hash(text) {
+        return createHash('sha256')
+            .update(JSON.stringify([this.#model.provider, this.#model.modelId, text]))
+            .digest('hex');
+    }
+    #read(id) {
+        const row = this.#database
+            .prepare('SELECT "vector", "published_at" FROM "embedding_document" WHERE "id" = ?')
+            .get(id);
+        if (row === undefined)
+            throw new UnknownDocumentError(id);
+        return {
+            vector: UnitVector.fromBlob(blob(row.vector)),
+            publishedAt: Number(row.published_at),
+        };
+    }
+    // Before the upsert, each stored vector has one dimension. The new vectors
+    // must have the dimension of a vector that the upsert did not replace, and
+    // of each other.
+    #checkDimension(replaced, vectors) {
+        const [first] = vectors;
+        if (first === undefined)
+            return;
+        const kept = this.#database
+            .prepare('SELECT length("vector") AS "bytes" FROM "embedding_document" WHERE "id" NOT IN (SELECT "value" FROM json_each(?)) LIMIT 1')
+            .get(JSON.stringify(replaced));
+        const expected = kept === undefined
+            ? first.dimension
+            : Number(kept.bytes) / Float32Array.BYTES_PER_ELEMENT;
+        for (const vector of vectors)
+            if (vector.dimension !== expected)
+                throw new DimensionMismatchError(expected, vector.dimension);
+    }
+}
+// The k candidates most similar to the vector. Equal similarities sort by id.
+function ranked(vector, candidates, k) {
+    return candidates
+        .map((candidate) => ({
+        id: candidate.id,
+        similarity: vector.cosine(candidate.vector),
+    }))
+        .toSorted((left, right) => right.similarity - left.similarity || compareIds(left.id, right.id))
+        .slice(0, k);
+}
+// Compares by code unit, not by locale, so the order is the same on every
+// machine.
+function compareIds(left, right) {
+    if (left < right)
+        return -1;
+    if (left > right)
+        return 1;
+    return 0;
+}
+function blob(value) {
+    if (!(value instanceof Uint8Array))
+        throw new TypeError('A stored vector must be a BLOB.');
+    return value;
+}
+//# sourceMappingURL=collection.js.map

package/dist/collection.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"collection.js","sourceRoot":"","sources":["../src/collection.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAGzC,OAAO,EAAuB,SAAS,EAAE,MAAM,IAAI,CAAC;AAEpD,OAAO,EAAE,WAAW,EAAE,MAAM,6BAA6B,CAAC;AAE1D,OAAO,EAAE,sBAAsB,EAAE,MAAM,+BAA+B,CAAC;AACvE,OAAO,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAC/C,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC;AACnE,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAEzC,MAAM,MAAM,GAAG,UAAU,CAAC;AAsB1B,yEAAyE;AACzE,wEAAwE;AACxE,oDAAoD;AACpD,MAAM,OAAO,UAAU;IACZ,SAAS,CAAe;IACxB,MAAM,CAAkC;IAEjD,YAAY,OAA0B;QACpC,IAAI,CAAC,SAAS,GAAG,OAAO,CAAC,QAAQ,CAAC;QAClC,IAAI,CAAC,MAAM,GAAG,OAAO,CAAC,KAAK,CAAC;QAC5B,gBAAgB,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACzC,CAAC;IAED,4EAA4E;IAC5E,gDAAgD;IAChD,KAAK,CAAC,MAAM,CAAC,SAA8B;QACzC,MAAM,MAAM,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,CACnC,wDAAwD,CACzD,CAAC;QACF,MAAM,MAAM,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;YAC1C,QAAQ;YACR,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC;SAChC,CAAC,CAAC,CAAC;QACJ,MAAM,EAAE,OAAO,GAAG,EAAE,EAAE,SAAS,GAAG,EAAE,EAAE,GAAG,MAAM,CAAC,OAAO,CACrD,MAAM,EACN,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE,EAAE,EAAE,CACrB,MAAM,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC,EAAE,IAAI,KAAK,IAAI,CAAC,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS,CACnE,CAAC;QACF,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,MAAM,CAC/B,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,QAAQ,CAAC,IAAI,CAAC,CAC7C,CAAC;QAEF,WAAW,CACT,IAAI,CAAC,SAAS,EACd,GAAG,EAAE;YACH,MAAM,KAAK,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,CAClC;4IACkI,CACnI,CAAC;YACF,KAAK,MAAM,CAAC,KAAK,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,IAAI,OAAO,CAAC,OAAO,EAAE,EAAE,CAAC;gBAC5D,MAAM,MAAM,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;gBAC9B,IAAI,MAAM,KAAK,SAAS;oBACtB,MAAM,IAAI,KAAK,CACb,kBAAkB,OAAO,CAAC,MAAM,mBAAmB,OAAO,CAAC,MAAM,SAAS,CAC3E,CAAC;gBACJ,KAAK,CAAC,GAAG,CACP,QAAQ,CAAC,EAAE,EACX,IAAI,EACJ,QAAQ,CAAC,WAAW,CAAC,OAAO,EAAE,EAC9B,MAAM,CAAC,MAAM,EAAE,CAChB,CAAC;YACJ,CAAC;YACD,MAAM,IAAI,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,CACjC,mEAAmE,CACpE,CAAC;YACF,KAAK,MAAM,EAAE,QAAQ,EAAE,IAAI,SAAS;gBAClC,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,WAAW,CAAC,OAAO,EAAE,EAAE,QAAQ,CAAC,EAAE,CAAC,CAAC;YACxD,IAAI,CAAC,eAAe,CAClB,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,QAAQ,CAAC,EAAE,CAAC,EAC1C,OAAO,CACR,CAAC;QACJ,CAAC,EACD,EAAE,IAAI,EAAE,WAAW,EAAE,CACtB,CAAC;IACJ,CAAC;IAED,2EAA2E;IAC3E,sEAAsE;IACtE,4DAA4D;IAC5D,OAAO,CACL,EAAU,EACV,OAA4D;QAE5D,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC/C,MAAM,MAAM,GAAG,OAAO,CAAC,UAAU,GAAG,MAAM,CAAC;QAC3C,MAAM,UAAU,GAAG,IAAI,CAAC,SAAS;aAC9B,OAAO,CACN,oGAAoG,CACrG;aACA,GAAG,CAAC,WAAW,GAAG,MAAM,EAAE,WAAW,GAAG,MAAM,EAAE,EAAE,CAAC;aACnD,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC;YACb,EAAE,EAAE,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YAClB,MAAM,EAAE,UAAU,CAAC,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;SAC9C,CAAC,CAAC,CAAC;QACN,OAAO,MAAM,CAAC,MAAM,EAAE,UAAU,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC;IAC/C,CAAC;IAED,2EAA2E;IAC3E,wEAAwE;IACxE,0DAA0D;IAC1D,4EAA4E;IAC5E,cAAc;IACd,cAAc,CAAC,OAKd;QACC,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QACpC,MAAM,EAAE,GAAG,OAAO,CAAC,EAAE,CAAC,OAAO,EAAE,CAAC;QAChC,MAAM,MAAM,GAAG,OAAO,CAAC,UAAU,GAAG,MAAM,CAAC;QAC3C,MAAM,SAAS,GAAG,IAAI,CAAC,SAAS;aAC7B,OAAO,CACN,oIAAoI,CACrI;aACA,GAAG,CAAC,IAAI,GAAG,MAAM,EAAE,EAAE,GAAG,MAAM,CAAC;aAC/B,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC;YACb,EAAE,EAAE,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;YAClB,WAAW,EAAE,MAAM,CAAC,GAAG,CAAC,YAAY,CAAC;YACrC,MAAM,EAAE,UAAU,CAAC,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;SAC9C,CAAC,CAAC,CAAC;QACN,OAAO,SAAS;aACb,MAAM,EAAE;aACR,MAAM,CAAC,CAAC,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC,WAAW,IAAI,IAAI,IAAI,WAAW,IAAI,EAAE,CAAC;aACrE,GAAG,CAAC,CAAC,QAAQ,EAAE,EAAE,CAAC;YACjB,QAAQ,CAAC,EAAE;YACX,MAAM,CACJ,QAAQ,CAAC,MAAM,EACf,SAAS,CAAC,MAAM,CACd,CAAC,KAAK,EAAE,EAAE,CACR,KAAK,CAAC,EAAE,KAAK,QAAQ,CAAC,EAAE;gBACxB,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,GAAG,QAAQ,CAAC,WAAW,CAAC,IAAI,MAAM,CAC/D,EACD,OAAO,CAAC,CAAC,CACV;SACF,CAAC,CAAC;IACP,CAAC;IAED,iDAAiD;IACjD,UAAU,CAAC,CAAS,EAAE,CAAS;QAC7B,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;IAC3D,CAAC;IAED,sCAAsC;IACtC,KAAK,CAAC,MAAM,CAAC,KAAe;QAC1B,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAC;QAClC,MAAM,EAAE,UAAU,EAAE,GAAG,MAAM,SAAS,CAAC;YACrC,KAAK,EAAE,IAAI,CAAC,MAAM;YAClB,MAAM,EAAE,KAAK;SACd,CAAC,CAAC;QACH,OAAO,UAAU,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,UAAU,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC,CAAC;IACxE,CAAC;IAED,uEAAuE;IACvE,KAAK,CAAC,IAAY;QAChB,OAAO,UAAU,CAAC,QAAQ,CAAC;aACxB,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC,CAAC;aACzE,MAAM,CAAC,KAAK,CAAC,CAAC;IACnB,CAAC;IAED,KAAK,CAAC,EAAU;QACd,MAAM,GAAG,GAAG,IAAI,CAAC,SAAS;aACvB,OAAO,CACN,0EAA0E,CAC3E;aACA,GAAG,CAAC,EAAE,CAAC,CAAC;QACX,IAAI,GAAG,KAAK,SAAS;YAAE,MAAM,IAAI,oBAAoB,CAAC,EAAE,CAAC,CAAC;QAC1D,OAAO;YACL,MAAM,EAAE,UAAU,CAAC,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,CAAC;YAC7C,WAAW,EAAE,MAAM,CAAC,GAAG,CAAC,YAAY,CAAC;SACtC,CAAC;IACJ,CAAC;IAED,2EAA2E;IAC3E,2EAA2E;IAC3E,iBAAiB;IACjB,eAAe,CAAC,QAA2B,EAAE,OAA8B;QACzE,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,CAAC;QACxB,IAAI,KAAK,KAAK,SAAS;YAAE,OAAO;QAChC,MAAM,IAAI,GAAG,IAAI,CAAC,SAAS;aACxB,OAAO,CACN,2HAA2H,CAC5H;aACA,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC,CAAC;QACjC,MAAM,QAAQ,GACZ,IAAI,KAAK,SAAS;YAChB,CAAC,CAAC,KAAK,CAAC,SAAS;YACjB,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,YAAY,CAAC,iBAAiB,CAAC;QAC1D,KAAK,MAAM,MAAM,IAAI,OAAO;YAC1B,IAAI,MAAM,CAAC,SAAS,KAAK,QAAQ;gBAC/B,MAAM,IAAI,sBAAsB,CAAC,QAAQ,EAAE,MAAM,CAAC,SAAS,CAAC,CAAC;IACnE,CAAC;CACF;AAED,8EAA8E;AAC9E,SAAS,MAAM,CACb,MAAkB,EAClB,UAA2E,EAC3E,CAAS;IAET,OAAO,UAAU;SACd,GAAG,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;QACnB,EAAE,EAAE,SAAS,CAAC,EAAE;QAChB,UAAU,EAAE,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,MAAM,CAAC;KAC5C,CAAC,CAAC;SACF,QAAQ,CACP,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CACd,KAAK,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,IAAI,UAAU,CAAC,IAAI,CAAC,EAAE,EAAE,KAAK,CAAC,EAAE,CAAC,CACtE;SACA,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;AACjB,CAAC;AAED,0EAA0E;AAC1E,WAAW;AACX,SAAS,UAAU,CAAC,IAAY,EAAE,KAAa;IAC7C,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC,CAAC;IAC5B,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC;IAC3B,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAS,IAAI,CAAC,KAAc;IAC1B,IAAI,CAAC,CAAC,KAAK,YAAY,UAAU,CAAC;QAChC,MAAM,IAAI,SAAS,CAAC,iCAAiC,CAAC,CAAC;IACzD,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/dimension-mismatch-error.d.ts

@@ -0,0 +1,6 @@
+export declare class DimensionMismatchError extends Error {
+    readonly expected: number;
+    readonly actual: number;
+    constructor(expected: number, actual: number);
+}
+//# sourceMappingURL=dimension-mismatch-error.d.ts.map

package/dist/dimension-mismatch-error.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"dimension-mismatch-error.d.ts","sourceRoot":"","sources":["../src/dimension-mismatch-error.ts"],"names":[],"mappings":"AAEA,qBAAa,sBAAuB,SAAQ,KAAK;IAC/C,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;gBAEZ,QAAQ,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM;CAQ7C"}

package/dist/dimension-mismatch-error.js

@@ -0,0 +1,13 @@
+// The model gave a vector whose dimension is not the dimension of the vectors
+// that the collection stores.
+export class DimensionMismatchError extends Error {
+    expected;
+    actual;
+    constructor(expected, actual) {
+        super(`The collection stores vectors of dimension ${expected}, and the model gave a vector of dimension ${actual}. Upsert every document again with the new model, in one call.`);
+        this.name = 'DimensionMismatchError';
+        this.expected = expected;
+        this.actual = actual;
+    }
+}
+//# sourceMappingURL=dimension-mismatch-error.js.map

package/dist/dimension-mismatch-error.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"dimension-mismatch-error.js","sourceRoot":"","sources":["../src/dimension-mismatch-error.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,8BAA8B;AAC9B,MAAM,OAAO,sBAAuB,SAAQ,KAAK;IACtC,QAAQ,CAAS;IACjB,MAAM,CAAS;IAExB,YAAY,QAAgB,EAAE,MAAc;QAC1C,KAAK,CACH,8CAA8C,QAAQ,8CAA8C,MAAM,gEAAgE,CAC3K,CAAC;QACF,IAAI,CAAC,IAAI,GAAG,wBAAwB,CAAC;QACrC,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACvB,CAAC;CACF"}

package/dist/index.d.ts

@@ -0,0 +1,4 @@
+export { Collection, type CollectionOptions, type Document, type Neighbour, } from './collection.ts';
+export { DimensionMismatchError } from './dimension-mismatch-error.ts';
+export { UnknownDocumentError } from './unknown-document-error.ts';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,UAAU,EACV,KAAK,iBAAiB,EACtB,KAAK,QAAQ,EACb,KAAK,SAAS,GACf,MAAM,iBAAiB,CAAC;AACzB,OAAO,EAAE,sBAAsB,EAAE,MAAM,+BAA+B,CAAC;AACvE,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC"}

package/dist/index.js

@@ -0,0 +1,4 @@
+export { Collection, } from "./collection.js";
+export { DimensionMismatchError } from "./dimension-mismatch-error.js";
+export { UnknownDocumentError } from "./unknown-document-error.js";
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,UAAU,GAIX,MAAM,iBAAiB,CAAC;AACzB,OAAO,EAAE,sBAAsB,EAAE,MAAM,+BAA+B,CAAC;AACvE,OAAO,EAAE,oBAAoB,EAAE,MAAM,6BAA6B,CAAC"}

package/dist/schema.d.ts

@@ -0,0 +1,3 @@
+import { Declaration } from '@zukhruf/sqlite-drift';
+export declare const collectionSchema: Declaration;
+//# sourceMappingURL=schema.d.ts.map

package/dist/schema.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"schema.d.ts","sourceRoot":"","sources":["../src/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AAKpD,eAAO,MAAM,gBAAgB,aAc5B,CAAC"}

package/dist/schema.js

@@ -0,0 +1,17 @@
+import { Declaration } from '@zukhruf/sqlite-drift';
+// The documents of a collection. Every row derives from a caller's document
+// and the model, so a stale table is replaced with its rows: the next upsert
+// of the caller embeds the documents again (docs/adr/0003).
+export const collectionSchema = new Declaration([
+    `CREATE TABLE embedding_document (
+  id TEXT PRIMARY KEY,
+  -- sha256 of the model's provider and id and the text (docs/adr/0002).
+  hash TEXT NOT NULL,
+  -- Milliseconds since the Unix epoch.
+  published_at INTEGER NOT NULL,
+  -- A unit vector of float32 values, in the byte order of the platform.
+  vector BLOB NOT NULL
+) STRICT`,
+    `CREATE INDEX embedding_document_published_at ON embedding_document (published_at)`,
+], { rows: 'replace' });
+//# sourceMappingURL=schema.js.map

package/dist/schema.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"schema.js","sourceRoot":"","sources":["../src/schema.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,uBAAuB,CAAC;AAEpD,4EAA4E;AAC5E,6EAA6E;AAC7E,4DAA4D;AAC5D,MAAM,CAAC,MAAM,gBAAgB,GAAG,IAAI,WAAW,CAC7C;IACE;;;;;;;;SAQK;IACL,mFAAmF;CACpF,EACD,EAAE,IAAI,EAAE,SAAS,EAAE,CACpB,CAAC"}

package/dist/unknown-document-error.d.ts

@@ -0,0 +1,5 @@
+export declare class UnknownDocumentError extends Error {
+    readonly id: string;
+    constructor(id: string);
+}
+//# sourceMappingURL=unknown-document-error.d.ts.map

package/dist/unknown-document-error.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"unknown-document-error.d.ts","sourceRoot":"","sources":["../src/unknown-document-error.ts"],"names":[],"mappings":"AACA,qBAAa,oBAAqB,SAAQ,KAAK;IAC7C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAKvB"}

package/dist/unknown-document-error.js

@@ -0,0 +1,10 @@
+// The collection stores no document with this id.
+export class UnknownDocumentError extends Error {
+    id;
+    constructor(id) {
+        super(`The collection stores no document with the id ${id}.`);
+        this.name = 'UnknownDocumentError';
+        this.id = id;
+    }
+}
+//# sourceMappingURL=unknown-document-error.js.map

package/dist/unknown-document-error.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"unknown-document-error.js","sourceRoot":"","sources":["../src/unknown-document-error.ts"],"names":[],"mappings":"AAAA,kDAAkD;AAClD,MAAM,OAAO,oBAAqB,SAAQ,KAAK;IACpC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CAAC,iDAAiD,EAAE,GAAG,CAAC,CAAC;QAC9D,IAAI,CAAC,IAAI,GAAG,sBAAsB,CAAC;QACnC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF"}

package/dist/vector.d.ts

@@ -0,0 +1,10 @@
+export declare class UnitVector {
+    #private;
+    private constructor();
+    static normalize(values: readonly number[]): UnitVector;
+    static fromBlob(blob: Uint8Array): UnitVector;
+    get dimension(): number;
+    toBlob(): Uint8Array;
+    cosine(other: UnitVector): number;
+}
+//# sourceMappingURL=vector.d.ts.map

package/dist/vector.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"vector.d.ts","sourceRoot":"","sources":["../src/vector.ts"],"names":[],"mappings":"AAEA,qBAAa,UAAU;;IAGrB,OAAO;IAKP,MAAM,CAAC,SAAS,CAAC,MAAM,EAAE,SAAS,MAAM,EAAE,GAAG,UAAU;IAavD,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,UAAU,GAAG,UAAU;IAI7C,IAAI,SAAS,IAAI,MAAM,CAEtB;IAED,MAAM,IAAI,UAAU;IAWpB,MAAM,CAAC,KAAK,EAAE,UAAU,GAAG,MAAM;CAQlC"}

package/dist/vector.js

@@ -0,0 +1,38 @@
+// A unit vector of float32 values. A dot product of two unit vectors is their
+// cosine similarity.
+export class UnitVector {
+    #values;
+    constructor(values) {
+        this.#values = values;
+    }
+    // Divides each value by the length of the vector.
+    static normalize(values) {
+        const length = Math.sqrt(values.reduce((sum, value) => sum + value * value, 0));
+        if (length === 0 || !Number.isFinite(length))
+            throw new RangeError(`An embedding must have a finite length that is not zero. Its length is ${length}.`);
+        return new UnitVector(Float32Array.from(values, (value) => value / length));
+    }
+    // The bytes that SQLite gives are copied, because a Float32Array needs an
+    // offset that is a multiple of 4.
+    static fromBlob(blob) {
+        return new UnitVector(new Float32Array(Uint8Array.from(blob).buffer));
+    }
+    get dimension() {
+        return this.#values.length;
+    }
+    toBlob() {
+        return new Uint8Array(this.#values.buffer, this.#values.byteOffset, this.#values.byteLength);
+    }
+    // An indexed loop: an entries() iterator with destructuring made the
+    // nearest search of each document in a window of 1,001 documents about two
+    // times slower.
+    cosine(other) {
+        const values = this.#values;
+        const others = other.#values;
+        let sum = 0;
+        for (let index = 0; index < values.length; index++)
+            sum += (values[index] ?? 0) * (others[index] ?? 0);
+        return sum;
+    }
+}
+//# sourceMappingURL=vector.js.map

package/dist/vector.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"vector.js","sourceRoot":"","sources":["../src/vector.ts"],"names":[],"mappings":"AAAA,8EAA8E;AAC9E,qBAAqB;AACrB,MAAM,OAAO,UAAU;IACZ,OAAO,CAAe;IAE/B,YAAoB,MAAoB;QACtC,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC;IACxB,CAAC;IAED,kDAAkD;IAClD,MAAM,CAAC,SAAS,CAAC,MAAyB;QACxC,MAAM,MAAM,GAAG,IAAI,CAAC,IAAI,CACtB,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,GAAG,KAAK,GAAG,KAAK,EAAE,CAAC,CAAC,CACtD,CAAC;QACF,IAAI,MAAM,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC;YAC1C,MAAM,IAAI,UAAU,CAClB,0EAA0E,MAAM,GAAG,CACpF,CAAC;QACJ,OAAO,IAAI,UAAU,CAAC,YAAY,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,GAAG,MAAM,CAAC,CAAC,CAAC;IAC9E,CAAC;IAED,0EAA0E;IAC1E,kCAAkC;IAClC,MAAM,CAAC,QAAQ,CAAC,IAAgB;QAC9B,OAAO,IAAI,UAAU,CAAC,IAAI,YAAY,CAAC,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC;IACxE,CAAC;IAED,IAAI,SAAS;QACX,OAAO,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;IAC7B,CAAC;IAED,MAAM;QACJ,OAAO,IAAI,UAAU,CACnB,IAAI,CAAC,OAAO,CAAC,MAAM,EACnB,IAAI,CAAC,OAAO,CAAC,UAAU,EACvB,IAAI,CAAC,OAAO,CAAC,UAAU,CACxB,CAAC;IACJ,CAAC;IAED,qEAAqE;IACrE,2EAA2E;IAC3E,gBAAgB;IAChB,MAAM,CAAC,KAAiB;QACtB,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC;QAC5B,MAAM,MAAM,GAAG,KAAK,CAAC,OAAO,CAAC;QAC7B,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,MAAM,CAAC,MAAM,EAAE,KAAK,EAAE;YAChD,GAAG,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC;QACrD,OAAO,GAAG,CAAC;IACb,CAAC;CACF"}

package/package.json

@@ -1,6 +1,51 @@
 {
   "name": "@zukhruf/embeddings",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.3.47",
+  "description": "A collection of document embeddings in SQLite. It embeds only new or changed text through an AI SDK embedding model, and finds the nearest documents of a document by exact cosine similarity inside a publish-date window.",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/JanuaryLabs/zukhruf",
+    "directory": "packages/embeddings"
+  },
+  "bugs": {
+    "url": "https://github.com/JanuaryLabs/zukhruf/issues"
+  },
+  "homepage": "https://github.com/JanuaryLabs/zukhruf/tree/main/packages/embeddings#readme",
+  "keywords": [
+    "embeddings",
+    "vector-search",
+    "nearest-neighbour",
+    "cosine-similarity",
+    "sqlite",
+    "node-sqlite",
+    "ai-sdk"
+  ],
+  "type": "module",
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "@zukhruf/source": "./src/index.ts",
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "default": "./dist/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "!dist/spec/**",
+    "!**/*.tsbuildinfo",
+    "!**/*.test.*"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "@zukhruf/sqlite-drift": "^0.3.47",
+    "@zukhruf/sqlite-transaction": "^0.3.47",
+    "ai": "^7.0.154"
+  },
+  "devDependencies": {
+    "@zukhruf/testing": "^0.3.47"
+  }
+}