mini-semantic-search 0.0.0-stage → 0.1.0

package/LICENSE

@@ -0,0 +1,24 @@
+MIT License
+
+Copyright (c) 2026 ShakirasHips
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+This licence covers the code. The files in models/ come from others and keep
+their own licences: see models/README.md.

package/README.md

@@ -1,3 +1,192 @@
-# Temporary Holding Version
+# mini-semantic-search
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Semantic search with [MiniSearch](https://github.com/lucaong/minisearch)'s API. Documents are matched by meaning
+rather than shared words: "stop sending me newsletters" finds "How do I unsubscribe from your emails?", and "the mug
+came smashed" finds "My item arrived damaged or broken". It runs in browsers and Node with no dependencies, no
+server and no API calls, and the package is 2.2 MB with its model (1.8 MB gzipped).
+
+```ts
+import MiniSemanticSearch from 'mini-semantic-search';
+
+const search = await MiniSemanticSearch.create({
+  fields: ['title', 'text'],       // fields to embed and search
+  storeFields: ['title'],          // fields to return with results
+});
+
+search.addAll([
+  { id: 1, title: 'Returns', text: 'Start a return under Orders in your account.' },
+  { id: 2, title: 'Damaged items', text: 'Send us a photo of the damage and we will replace it.' },
+  { id: 3, title: 'Shipping times', text: 'Standard shipping takes 3 to 5 business days.' },
+]);
+
+search.search('the mug came smashed');
+// => [
+//   { id: 2, score: 0.39, title: 'Damaged items' },
+//   { id: 3, score: 0.15, title: 'Shipping times' },
+//   { id: 1, score: 0.08, title: 'Returns' },
+// ]
+```
+
+The embeddings come from [Model2Vec](https://github.com/MinishLab/model2vec), a static embedding model. It keeps one
+vector per token, and a text's vector is the average of its tokens' vectors. There is no neural network to run, so
+embedding a document takes about 0.1 ms on a CPU.
+
+## Install
+
+```sh
+npm install mini-semantic-search
+```
+
+- **Node 18+:** `import MiniSemanticSearch from 'mini-semantic-search'`. The bundled model loads from the package.
+- **Bundlers (Vite, webpack 5, esbuild, Rollup):** the same import. The model is found with
+  `new URL('../models/…', import.meta.url)`, which bundlers copy as an asset. Or pass `model:` your own URL.
+- **Straight from a CDN, no build step:**
+
+  ```html
+  <script type="module">
+    import MiniSemanticSearch from 'https://cdn.jsdelivr.net/npm/mini-semantic-search@0.1.0/dist/index.js';
+    const search = await MiniSemanticSearch.create({ fields: ['title', 'text'] });
+  </script>
+  ```
+
+  The model (2.2 MB) then loads from the CDN too.
+
+## API
+
+It follows MiniSearch, so code written for one mostly works with the other. The one difference in setting it up is
+that the model has to load first, so you either `await MiniSemanticSearch.create(options)` or pass a loaded model
+to the constructor:
+
+```ts
+import MiniSemanticSearch, { StaticModel } from 'mini-semantic-search';
+
+const model = await StaticModel.load();  // load once, share between indexes
+const search = new MiniSemanticSearch({ model, fields: ['title', 'text'] });
+```
+
+| Option | Default | |
+|---|---|---|
+| `fields` | required | The fields embedded and searched. A field holding an array is embedded item by item, and its closest item counts. |
+| `storeFields` | `[]` | The fields returned with each result. |
+| `idField` | `'id'` | The field holding each document's unique id. |
+| `extractField` | `doc[field]` | `(document, fieldName) => value`, for nested or computed fields. |
+| `searchOptions` | `{}` | Defaults for `search`. |
+| `model` | bundled potion-base-8M | A `StaticModel`, or (in `create`) a URL, path or bytes of a packed model. |
+
+| Method | |
+|---|---|
+| `add(doc)`, `addAll(docs)` | Index documents. They can be searched straight away. A duplicate id throws. |
+| `addAllAsync(docs, { chunkSize: 100 })` | The same, yielding between chunks so a page stays responsive. |
+| `remove(doc)`, `discard(id)`, `discardAll(ids)` | Remove documents, by document or by id. |
+| `removeAll(docs?)` | Remove the given documents, or all of them. |
+| `replace(doc)` | Replace the document with the same id. |
+| `has(id)`, `getStoredFields(id)`, `documentCount` | What's indexed. |
+| `search(query, options?)` | Every document, best first, as `{ id, score, ...storeFields }`. |
+
+Search options:
+
+| Option | |
+|---|---|
+| `fields` | Search only these fields, e.g. `{ fields: ['title'] }`. |
+| `boost` | Weights of fields against each other, e.g. `{ boost: { title: 3 } }`. |
+| `filter` | `(result) => boolean`; the result includes its stored fields. |
+| `minScore` | Leave out results scoring under this. |
+
+**How documents are scored.** Each document is embedded once per field and once with all its fields together. Its
+score is half the whole document's cosine similarity to the query, plus half its fields' similarities averaged
+(weighted by `boost`). Scores run from about −0.1 to 1. A document always scores something, so every document comes
+back: `slice` the results, or set `minScore` to drop the ones too far off to count as a match. Where to set it
+depends on your documents' wording. Try your own off-topic questions and put it just above the scores they get.
+
+**How it differs from MiniSearch.** Results have no `terms`, `queryTerms` or `match`, because nothing is matched
+word for word. There's no `prefix`, `fuzzy`, `combineWith`, `autoSuggest` or query syntax, and no
+`toJSON`/`loadJSON`: indexing costs about 0.1 ms per document, so you can just index again on load.
+
+Indexing takes 0.05–0.2 ms per document. A search over 10,000 documents takes about 8 ms.
+
+## Embeddings
+
+```ts
+import { StaticModel, similarity } from 'mini-semantic-search';
+
+const model = await StaticModel.load();            // or StaticModel.load(urlOrPathOrBytes)
+const a = model.embed('how do I cancel my order');  // Float32Array(128), unit length
+const b = model.embed('stop my purchase before it ships');
+similarity(a, b);                                   // cosine similarity, 1 = same meaning
+model.embedMany(['refund', 'exchange']);           // several at once
+model.tokenize('Café');                             // token ids
+```
+
+This matches the Python library. Token ids are identical to Hugging Face's tokenizer, and with a full-precision
+model the vectors match `model2vec` to within 1e-6. The tests check the token ids and the bundled model's vectors
+against fixtures made with `scripts/make-fixture.py`.
+
+Any Model2Vec model with a lower-casing BERT tokenizer and no token weights can be packed (most `minishlab/potion-*`
+models). A bigger model is more accurate. The full 256 dimensions of potion-base-8M at 4 bits are 4.2 MB:
+
+```sh
+npx mini-semantic-search pack ./potion-base-8M ./potion-base-8M.bin --bits 4
+```
+
+```ts
+const search = await MiniSemanticSearch.create({ fields: ['title', 'text'], model: '/potion-base-8M.bin' });
+```
+
+## Command line
+
+```sh
+npx mini-semantic-search search docs.json "how do I get a refund" --fields title,text [--k 5]
+npx mini-semantic-search embed "some text"                  # the vector as JSON
+npx mini-semantic-search pack <model folder> out.bin [--bits 4|8|32] [--group 64] [--dims N]
+```
+
+## How well it works
+
+On the example FAQ in `examples/` (a made-up shop, 45 entries searched on `section`, `q` and `a`, and 135 casual
+test questions), with the right entry ranked first or in the top 3:
+
+| Model | Size | `boost` | First | Top 3 |
+|---|---|---|---|---|
+| potion-base-8M, 256 dims at 4 bits | 4.2 MB | `{ q: 3 }` | 63.0% | 87.4% |
+| **potion-base-8M, 128 dims at 4 bits (bundled)** | **2.2 MB** | none | **59.3%** | **80.0%** |
+| **potion-base-8M, 128 dims at 4 bits (bundled)** | **2.2 MB** | `{ q: 3 }` | **60.7%** | **83.7%** |
+
+The test questions were written with the entries in view, so expect real questions to do somewhat worse. The set
+is small: one question is 0.7%. The bundled model keeps potion-base-8M's first 128 dimensions (its PCA components
+with the most variance), and its vectors are within cosine 0.98 of the full-precision ones cut to 128.
+
+**What it can't do:**
+- Averaging tokens loses word order, so "can I add a shopper" and "can a shopper add me" look alike.
+- It only knows words its training text had. Brand names and jargon ("amex") need other phrasings, such as an
+  array field of the ways people ask.
+- Misspelt words break into word pieces that mean something else, so typos hurt.
+- English only.
+
+## Development
+
+```sh
+npm install
+npm test                       # builds, then runs the tests (Node 21+)
+sh scripts/smoke.sh            # packs the package, installs it in an empty project and uses it from there
+python scripts/make-fixture.py <potion-base-8M folder>   # rebuilds the reference fixture (needs model2vec)
+```
+
+The bundled model is rebuilt with
+`node dist/cli.js pack <potion-base-8M folder> models/potion-base-8M-128d.bin --bits 4 --dims 128 --group 128`.
+
+CI (`.github/workflows/ci.yml`) builds every push and pull request on Node 18, 20, 22 and 24, runs the tests (22
+and 24) and the install check (all four).
+
+To release, bump the version and push the tag: `npm version patch && git push --follow-tags`. The release workflow
+checks that the tag matches `package.json`, tests again, publishes to npm with provenance and creates a GitHub
+release. It publishes with npm's trusted publishing (the package's Settings → Trusted publishing: GitHub Actions,
+repository `ShakirasHips/model2vec-js`, workflow `release.yml`, environment `npm`), so no token is stored anywhere.
+
+## Credits and licence
+
+The code is MIT-licensed (`LICENSE`). The bundled model is
+[potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) by MinishLab, MIT, repacked at 128 dimensions and
+4 bits (see `models/README.md`). The API follows [MiniSearch](https://github.com/lucaong/minisearch) by Luca Ongaro.
+
+This package reimplements Model2Vec's inference and BERT's tokenizer in TypeScript; it contains no code from
+Model2Vec, BERT or MiniSearch.

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,60 @@
+#!/usr/bin/env node
+import { readFile, writeFile } from 'node:fs/promises';
+import { MiniSemanticSearch, packModel, StaticModel } from './node.js';
+const HELP = `mini-semantic-search: semantic search with Model2Vec static embeddings
+
+  mini-semantic-search search <docs.json> "<query>" --fields title,text [--id id] [--k 5] [--model m.bin]
+      The documents closest in meaning to the query. docs.json is an array of objects.
+  mini-semantic-search embed "<text>" [--model m.bin]
+      The text's vector as JSON.
+  mini-semantic-search pack <model folder> <out.bin> [--bits 4|8|32] [--group 64] [--dims N]
+      Packs a Model2Vec model from Hugging Face (model.safetensors + tokenizer.json).
+
+--model defaults to the bundled potion-base-8M (128 dimensions at 4 bits).`;
+const argv = process.argv.slice(2);
+const flags = new Map();
+const args = [];
+for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a.startsWith('--'))
+        flags.set(a.slice(2), argv[++i]);
+    else
+        args.push(a);
+}
+const flag = (name) => flags.get(name);
+const num = (name, def) => (flags.has(name) ? Number(flag(name)) : def);
+const model = () => StaticModel.load(flag('model'));
+async function main() {
+    const [cmd, ...rest] = args;
+    if (cmd === 'search' && rest.length === 2 && flag('fields')) {
+        const fields = flag('fields').split(',');
+        const idField = flag('id') ?? 'id';
+        const docs = JSON.parse(await readFile(rest[0], 'utf8'));
+        const search = await MiniSemanticSearch.create({ fields, storeFields: fields, idField, model: flag('model') });
+        search.addAll(docs);
+        for (const r of search.search(rest[1]).slice(0, num('k', 5))) {
+            console.log(`${r.score.toFixed(3)}  ${String(r.id)}  ${String(r[fields[0]] ?? '')}`);
+        }
+    }
+    else if (cmd === 'embed' && rest.length === 1) {
+        console.log(JSON.stringify(Array.from((await model()).embed(rest[0]), (x) => Number(x.toFixed(6)))));
+    }
+    else if (cmd === 'pack' && rest.length === 2) {
+        const bits = num('bits', 4);
+        if (bits !== 4 && bits !== 8 && bits !== 32)
+            throw new Error('--bits must be 4, 8 or 32');
+        const bytes = await packModel(rest[0], { bits, group: num('group', 64), dims: flags.has('dims') ? num('dims', 0) : undefined });
+        await writeFile(rest[1], bytes);
+        const m = StaticModel.fromBytes(bytes);
+        console.log(`${m.tokens.length} tokens × ${m.dims} dims at ${bits} bits -> ${rest[1]} (${(bytes.length / 1e6).toFixed(2)} MB)`);
+    }
+    else {
+        console.log(HELP);
+        if (cmd && cmd !== 'help' && cmd !== '--help')
+            process.exitCode = 1;
+    }
+}
+main().catch((err) => {
+    console.error(`mini-semantic-search: ${err.message}`);
+    process.exit(1);
+});

package/dist/format.d.ts

@@ -0,0 +1,16 @@
+export interface ModelData {
+    tokens: string[];
+    dims: number;
+    /** tokens.length × dims values, row by row. */
+    vectors: Float32Array;
+}
+export interface EncodeOptions {
+    /** 32, 8 or 4 (default 4). */
+    bits?: 32 | 8 | 4;
+    /** Values sharing one scale (default 64); must divide the number of dimensions. */
+    group?: number;
+}
+export declare function fromF16(h: number): number;
+export declare function toF16(x: number): number;
+export declare function decodeModel(bytes: Uint8Array): ModelData;
+export declare function encodeModel({ tokens, dims, vectors }: ModelData, { bits, group }?: EncodeOptions): Uint8Array;

package/dist/format.js

@@ -0,0 +1,101 @@
+// The packed model file. Little-endian: magic "FAQM", version, vocabulary size, dimensions, bits (32, 8 or 4),
+// group size and the vocabulary's byte length; then the tokens joined by "\n", padded to 4 bytes; then the
+// vectors. At 32 bits they are plain f32. At 8 bits each value is an int8, and at 4 bits two values share a byte
+// (low nibble first, each nibble − 8); either way every `group` values share one f16 scale, and all the scales
+// come first, padded to 4 bytes.
+const MAGIC = 0x4d514146;
+const VERSION = 1;
+export function fromF16(h) {
+    const e = (h >> 10) & 31, m = h & 1023, s = h & 0x8000 ? -1 : 1;
+    if (e === 0)
+        return s * m * 2 ** -24;
+    if (e === 31)
+        return m ? NaN : s * Infinity;
+    return s * (1 + m / 1024) * 2 ** (e - 15);
+}
+export function toF16(x) {
+    const u = new Uint32Array(new Float32Array([x]).buffer)[0];
+    const s = (u >> 16) & 0x8000, e = ((u >> 23) & 255) - 127 + 15;
+    let m = u & 0x7fffff;
+    if (e <= 0)
+        return s | Math.round((m | 0x800000) / 2 ** (14 - e));
+    if (e >= 31)
+        return s | 0x7c00;
+    m = Math.round(m / 8192);
+    return m === 1024 ? s | ((e + 1) << 10) : s | (e << 10) | m;
+}
+const pad4 = (n) => (n + 3) & ~3;
+export function decodeModel(bytes) {
+    const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+    if (bytes.byteLength < 28 || dv.getUint32(0, true) !== MAGIC)
+        throw new Error('not a model2vec-js model file');
+    const header = Array.from({ length: 7 }, (_, i) => dv.getUint32(i * 4, true));
+    const [, version, n, dims, bits, group, vocabLen] = header;
+    if (version !== VERSION)
+        throw new Error(`unsupported model file version ${version}`);
+    let off = 28;
+    const tokens = new TextDecoder().decode(bytes.subarray(off, off + vocabLen)).split('\n');
+    if (tokens.length !== n)
+        throw new Error('model file is corrupt: vocabulary size');
+    off += pad4(vocabLen);
+    const count = n * dims;
+    const vectors = new Float32Array(count);
+    if (bits === 32) {
+        vectors.set(new Float32Array(bytes.slice(off, off + count * 4).buffer));
+    }
+    else if (bits === 8 || bits === 4) {
+        const groups = count / group;
+        const scales = new Float32Array(groups);
+        for (let g = 0; g < groups; g++)
+            scales[g] = fromF16(dv.getUint16(off + g * 2, true));
+        off += pad4(groups * 2);
+        for (let i = 0; i < count; i++) {
+            const q = bits === 8 ? (bytes[off + i] << 24) >> 24 : ((bytes[off + (i >> 1)] >> ((i & 1) * 4)) & 15) - 8;
+            vectors[i] = q * scales[(i / group) | 0];
+        }
+    }
+    else {
+        throw new Error(`unsupported bits: ${bits}`);
+    }
+    return { tokens, dims, vectors };
+}
+export function encodeModel({ tokens, dims, vectors }, { bits = 4, group = 64 } = {}) {
+    const n = tokens.length, count = n * dims;
+    if (vectors.length !== count)
+        throw new Error('vectors must hold tokens.length × dims values');
+    if (tokens.some((t) => t.includes('\n')))
+        throw new Error('tokens must not contain newlines');
+    if (bits !== 32 && dims % group)
+        throw new Error(`group (${group}) must divide dims (${dims})`);
+    const vocab = new TextEncoder().encode(tokens.join('\n'));
+    let body;
+    if (bits === 32) {
+        body = new Uint8Array(new Float32Array(vectors).buffer);
+    }
+    else {
+        const levels = bits === 8 ? 127 : 7, groups = count / group;
+        const scaleBytes = pad4(groups * 2);
+        body = new Uint8Array(scaleBytes + (bits === 8 ? count : Math.ceil(count / 2)));
+        const dv = new DataView(body.buffer);
+        for (let g = 0; g < groups; g++) {
+            let max = 0;
+            for (let i = g * group; i < (g + 1) * group; i++)
+                max = Math.max(max, Math.abs(vectors[i]));
+            const h = toF16(max / levels), scale = fromF16(h) || 1;
+            dv.setUint16(g * 2, h, true);
+            for (let i = g * group; i < (g + 1) * group; i++) {
+                const v = Math.max(-levels, Math.min(levels, Math.round(vectors[i] / scale)));
+                if (bits === 8)
+                    dv.setInt8(scaleBytes + i, v);
+                else
+                    body[scaleBytes + (i >> 1)] |= (v + 8) << ((i & 1) * 4);
+            }
+        }
+    }
+    const out = new Uint8Array(28 + pad4(vocab.length) + body.length);
+    const dv = new DataView(out.buffer);
+    [MAGIC, VERSION, n, dims, bits, group, vocab.length].forEach((v, i) => dv.setUint32(i * 4, v, true));
+    out.set(vocab, 28);
+    out.set(body, 28 + pad4(vocab.length));
+    return out;
+}

package/dist/index.d.ts

@@ -0,0 +1,7 @@
+import { MiniSemanticSearch } from './search.js';
+export default MiniSemanticSearch;
+export { MiniSemanticSearch, type Options, type SearchOptions, type SearchResult } from './search.js';
+export { StaticModel, similarity } from './model.js';
+export { normalize, preTokenize, wordPiece } from './tokenizer.js';
+export { decodeModel, encodeModel, type ModelData, type EncodeOptions } from './format.js';
+export { loadBytes, DEFAULT_MODEL, type Source } from './load.js';

package/dist/index.js

@@ -0,0 +1,7 @@
+import { MiniSemanticSearch } from './search.js';
+export default MiniSemanticSearch;
+export { MiniSemanticSearch } from './search.js';
+export { StaticModel, similarity } from './model.js';
+export { normalize, preTokenize, wordPiece } from './tokenizer.js';
+export { decodeModel, encodeModel } from './format.js';
+export { loadBytes, DEFAULT_MODEL } from './load.js';

package/dist/load.d.ts

@@ -0,0 +1,9 @@
+/** Where to load a file from: a URL, a path (Node), or the bytes themselves. */
+export type Source = string | URL | ArrayBuffer | ArrayBufferView;
+type FileReader = (path: string | URL) => Promise<Uint8Array>;
+/** @internal Used by the Node entry point. */
+export declare function setFileReader(reader: FileReader): void;
+/** The bundled model: potion-base-8M's first 128 dimensions, packed at 4 bits. */
+export declare const DEFAULT_MODEL: URL;
+export declare function loadBytes(src: Source): Promise<Uint8Array>;
+export {};

package/dist/load.js

@@ -0,0 +1,23 @@
+// Fetching files: URLs everywhere, and local paths in Node (the Node entry point registers a file reader, so the
+// browser build never imports node:fs).
+let readFile = null;
+/** @internal Used by the Node entry point. */
+export function setFileReader(reader) {
+    readFile = reader;
+}
+/** The bundled model: potion-base-8M's first 128 dimensions, packed at 4 bits. */
+export const DEFAULT_MODEL = new URL('../models/potion-base-8M-128d.bin', import.meta.url);
+export async function loadBytes(src) {
+    if (src instanceof ArrayBuffer)
+        return new Uint8Array(src);
+    if (ArrayBuffer.isView(src))
+        return new Uint8Array(src.buffer, src.byteOffset, src.byteLength);
+    const url = src instanceof URL ? src : null;
+    if (readFile && (url ? url.protocol === 'file:' : !/^[a-z][a-z0-9+.-]*:\/\//i.test(src))) {
+        return readFile(url ?? src);
+    }
+    const res = await fetch(src);
+    if (!res.ok)
+        throw new Error(`${String(src)}: HTTP ${res.status}`);
+    return new Uint8Array(await res.arrayBuffer());
+}

package/dist/model.d.ts

@@ -0,0 +1,31 @@
+import { type ModelData } from './format.js';
+import { type Source } from './load.js';
+/**
+ * A Model2Vec static embedding model: one vector per token, and a sentence's vector is the mean of its tokens'
+ * vectors, normalised to length 1. No network runs, so embedding takes microseconds. Long texts are cut as the
+ * Python library cuts them: to `maxTokens` × the median token length in characters, then to `maxTokens` tokens,
+ * before unknown tokens are left out.
+ */
+export declare class StaticModel {
+    readonly tokens: string[];
+    readonly dims: number;
+    readonly vectors: Float32Array;
+    readonly unk: number;
+    /** Tokens read per text; Model2Vec's default is 512. */
+    maxTokens: number;
+    private readonly vocab;
+    private readonly medianTokenLength;
+    constructor({ tokens, dims, vectors }: ModelData);
+    /** Loads a packed model file; by default the bundled potion-base-8M (128 dimensions at 4 bits, 2.2 MB). */
+    static load(src?: Source): Promise<StaticModel>;
+    static fromBytes(bytes: Uint8Array): StaticModel;
+    /** Token ids, as Hugging Face's tokenizer gives them without special tokens (unknown words included as UNK). */
+    tokenize(text: string): number[];
+    /** Whether `token` (e.g. "voucher" or "##ing") is in the vocabulary. */
+    has(token: string): boolean;
+    /** The text's vector: unit length, or all zeros if it has no known tokens. */
+    embed(text: string): Float32Array;
+    embedMany(texts: string[]): Float32Array[];
+}
+/** Dot product: the cosine similarity of two unit vectors, such as `embed` returns (1 = same meaning). */
+export declare function similarity(a: Float32Array, b: Float32Array): number;

package/dist/model.js

@@ -0,0 +1,74 @@
+import { decodeModel } from './format.js';
+import { DEFAULT_MODEL, loadBytes } from './load.js';
+import { wordPiece } from './tokenizer.js';
+/**
+ * A Model2Vec static embedding model: one vector per token, and a sentence's vector is the mean of its tokens'
+ * vectors, normalised to length 1. No network runs, so embedding takes microseconds. Long texts are cut as the
+ * Python library cuts them: to `maxTokens` × the median token length in characters, then to `maxTokens` tokens,
+ * before unknown tokens are left out.
+ */
+export class StaticModel {
+    tokens;
+    dims;
+    vectors;
+    unk;
+    /** Tokens read per text; Model2Vec's default is 512. */
+    maxTokens = 512;
+    vocab;
+    medianTokenLength;
+    constructor({ tokens, dims, vectors }) {
+        this.tokens = tokens;
+        this.dims = dims;
+        this.vectors = vectors;
+        this.vocab = new Map(tokens.map((t, i) => [t, i]));
+        this.unk = this.vocab.get('[UNK]') ?? -1;
+        const lengths = tokens.map((t) => Array.from(t).length).sort((a, b) => a - b), mid = lengths.length >> 1;
+        this.medianTokenLength = Math.trunc(lengths.length % 2 ? lengths[mid] : (lengths[mid - 1] + lengths[mid]) / 2);
+    }
+    /** Loads a packed model file; by default the bundled potion-base-8M (128 dimensions at 4 bits, 2.2 MB). */
+    static async load(src = DEFAULT_MODEL) {
+        return StaticModel.fromBytes(await loadBytes(src));
+    }
+    static fromBytes(bytes) {
+        return new StaticModel(decodeModel(bytes));
+    }
+    /** Token ids, as Hugging Face's tokenizer gives them without special tokens (unknown words included as UNK). */
+    tokenize(text) {
+        return wordPiece(this.vocab, this.unk, text);
+    }
+    /** Whether `token` (e.g. "voucher" or "##ing") is in the vocabulary. */
+    has(token) {
+        return this.vocab.has(token);
+    }
+    /** The text's vector: unit length, or all zeros if it has no known tokens. */
+    embed(text) {
+        const { dims, vectors } = this;
+        const v = new Float32Array(dims);
+        const chars = this.maxTokens * this.medianTokenLength;
+        if (text.length > chars)
+            text = Array.from(text).slice(0, chars).join('');
+        for (const id of this.tokenize(text).slice(0, this.maxTokens)) {
+            if (id === this.unk)
+                continue;
+            for (let d = 0, o = id * dims; d < dims; d++)
+                v[d] += vectors[o + d];
+        }
+        let norm = 0;
+        for (let d = 0; d < dims; d++)
+            norm += v[d] * v[d];
+        norm = Math.sqrt(norm) || 1;
+        for (let d = 0; d < dims; d++)
+            v[d] /= norm;
+        return v;
+    }
+    embedMany(texts) {
+        return texts.map((t) => this.embed(t));
+    }
+}
+/** Dot product: the cosine similarity of two unit vectors, such as `embed` returns (1 = same meaning). */
+export function similarity(a, b) {
+    let s = 0;
+    for (let i = 0; i < a.length; i++)
+        s += a[i] * b[i];
+    return s;
+}

package/dist/node.d.ts

@@ -0,0 +1,4 @@
+import { MiniSemanticSearch } from './index.js';
+export default MiniSemanticSearch;
+export * from './index.js';
+export { packModel, type PackOptions } from './pack.js';

package/dist/node.js

@@ -0,0 +1,8 @@
+// The Node entry point: everything in the browser one, plus loading from local paths and packing models.
+import { readFile } from 'node:fs/promises';
+import { setFileReader } from './load.js';
+import { MiniSemanticSearch } from './index.js';
+setFileReader(async (path) => new Uint8Array(await readFile(path)));
+export default MiniSemanticSearch;
+export * from './index.js';
+export { packModel } from './pack.js';

package/dist/pack.d.ts

@@ -0,0 +1,10 @@
+import { type EncodeOptions } from './format.js';
+export interface PackOptions extends EncodeOptions {
+    /** Keep only the leading dimensions (Model2Vec's are PCA components, so they hold the most variance). */
+    dims?: number;
+}
+/**
+ * Packs a Model2Vec model folder (model.safetensors and tokenizer.json, as on Hugging Face, e.g.
+ * minishlab/potion-base-8M) into the file `StaticModel.load` reads.
+ */
+export declare function packModel(dir: string, { dims, ...encode }?: PackOptions): Promise<Uint8Array>;

package/dist/pack.js

@@ -0,0 +1,37 @@
+import { readFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { encodeModel } from './format.js';
+/**
+ * Packs a Model2Vec model folder (model.safetensors and tokenizer.json, as on Hugging Face, e.g.
+ * minishlab/potion-base-8M) into the file `StaticModel.load` reads.
+ */
+export async function packModel(dir, { dims, ...encode } = {}) {
+    const st = await readFile(join(dir, 'model.safetensors'));
+    const headerLen = Number(st.readBigUInt64LE(0));
+    const header = JSON.parse(st.subarray(8, 8 + headerLen).toString());
+    const others = Object.keys(header).filter((k) => k !== 'embeddings' && k !== '__metadata__');
+    if (others.length)
+        throw new Error(`unsupported tensors: ${others.join(', ')} (token weights or mappings)`);
+    const info = header.embeddings;
+    if (!info || info.dtype !== 'F32')
+        throw new Error('expected an F32 "embeddings" tensor');
+    const [n, full] = info.shape;
+    const keep = Math.min(dims ?? full, full);
+    const data = st.subarray(8 + headerLen + info.data_offsets[0], 8 + headerLen + info.data_offsets[1]);
+    const src = new Float32Array(data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength));
+    const tok = JSON.parse(await readFile(join(dir, 'tokenizer.json'), 'utf8'));
+    if (tok.model?.type !== 'WordPiece' || tok.normalizer?.type !== 'BertNormalizer' || tok.normalizer.lowercase === false) {
+        throw new Error('only lower-casing BERT (WordPiece) tokenizers are supported');
+    }
+    const tokens = [];
+    for (const [t, id] of Object.entries(tok.model.vocab))
+        tokens[id] = t;
+    for (const a of tok.added_tokens ?? [])
+        tokens[a.id] = a.content;
+    if (tokens.length !== n || tokens.includes(undefined))
+        throw new Error('vocabulary does not match the table');
+    const vectors = new Float32Array(n * keep);
+    for (let i = 0; i < n; i++)
+        vectors.set(src.subarray(i * full, i * full + keep), i * keep);
+    return encodeModel({ tokens, dims: keep, vectors }, encode);
+}

package/dist/search.d.ts

@@ -0,0 +1,75 @@
+import { type Source } from './load.js';
+import { StaticModel } from './model.js';
+type AnyDocument = any;
+type DocumentId = any;
+export interface SearchOptions {
+    /** Only these fields count (default: all of `fields`). */
+    fields?: string[];
+    /** Weights of fields against each other, e.g. `{ title: 2 }` (default 1 each). */
+    boost?: Record<string, number>;
+    /** Only results for which this is true. */
+    filter?: (result: SearchResult) => boolean;
+    /** Leave out results scoring under this (default: none). */
+    minScore?: number;
+}
+export interface Options<T = AnyDocument> {
+    /** The fields embedded and searched. */
+    fields: string[];
+    /** The fields returned with each result (default: none). */
+    storeFields?: string[];
+    /** The field holding each document's unique id (default "id"). */
+    idField?: string;
+    /** Reads a field from a document (default: `document[fieldName]`). Arrays are embedded item by item. */
+    extractField?: (document: T, fieldName: string) => unknown;
+    /** Defaults for `search`. */
+    searchOptions?: SearchOptions;
+    /** A loaded model, or where to load one from in `create` (default: the bundled potion-base-8M). */
+    model?: StaticModel | Source;
+}
+export type SearchResult = {
+    id: DocumentId;
+    /** About −0.1 to 1: how close in meaning the document is to the query. */
+    score: number;
+    [storedField: string]: unknown;
+};
+/**
+ * Semantic search with MiniSearch's API: documents are found by meaning rather than shared words, so "stop sending me
+ * newsletters" finds "How do I unsubscribe from your emails?". Each document is embedded once per field and once
+ * whole. It scores half its whole vector's cosine similarity to the query, plus half its fields' (weighted by
+ * `boost`; an array field counts its closest item). Searching takes no network and no server.
+ */
+export declare class MiniSemanticSearch<T = AnyDocument> {
+    readonly model: StaticModel;
+    private readonly options;
+    private readonly index;
+    /** Loads the model (the bundled one unless `options.model` says otherwise), then makes the index. */
+    static create<T = AnyDocument>(options: Options<T>): Promise<MiniSemanticSearch<T>>;
+    /** For a model you have loaded already (`StaticModel.load()`); `create` loads it for you. */
+    constructor(options: Options<T> & {
+        model: StaticModel;
+    });
+    /** How many documents are indexed. */
+    get documentCount(): number;
+    /** Adds a document; it can be searched straight away. Embedding takes about 0.1 ms. */
+    add(document: T): void;
+    addAll(documents: readonly T[]): void;
+    /** Adds documents `chunkSize` at a time, yielding between chunks so a page stays responsive. */
+    addAllAsync(documents: readonly T[], { chunkSize }?: {
+        chunkSize?: number;
+    }): Promise<void>;
+    /** Removes a document, found by its id. */
+    remove(document: T): void;
+    /** Removes the document with this id. */
+    discard(id: DocumentId): void;
+    discardAll(ids: readonly DocumentId[]): void;
+    /** Removes the given documents, or every document if none are given. */
+    removeAll(documents?: readonly T[]): void;
+    /** Replaces the document that has the same id. */
+    replace(document: T): void;
+    has(id: DocumentId): boolean;
+    /** The document's `storeFields`, or undefined if it isn't indexed. */
+    getStoredFields(id: DocumentId): Record<string, unknown> | undefined;
+    /** Every document, best first, with its id, score and stored fields. */
+    search(query: string, searchOptions?: SearchOptions): SearchResult[];
+}
+export {};

package/dist/search.js

@@ -0,0 +1,129 @@
+import { DEFAULT_MODEL } from './load.js';
+import { similarity, StaticModel } from './model.js';
+const texts = (value) => {
+    if (value === undefined || value === null)
+        return [];
+    return (Array.isArray(value) ? value : [value]).map(String).filter((s) => s.trim());
+};
+/**
+ * Semantic search with MiniSearch's API: documents are found by meaning rather than shared words, so "stop sending me
+ * newsletters" finds "How do I unsubscribe from your emails?". Each document is embedded once per field and once
+ * whole. It scores half its whole vector's cosine similarity to the query, plus half its fields' (weighted by
+ * `boost`; an array field counts its closest item). Searching takes no network and no server.
+ */
+export class MiniSemanticSearch {
+    model;
+    options;
+    index = new Map();
+    /** Loads the model (the bundled one unless `options.model` says otherwise), then makes the index. */
+    static async create(options) {
+        const { model = DEFAULT_MODEL } = options;
+        return new MiniSemanticSearch({ ...options, model: model instanceof StaticModel ? model : await StaticModel.load(model) });
+    }
+    /** For a model you have loaded already (`StaticModel.load()`); `create` loads it for you. */
+    constructor(options) {
+        const { model, fields, storeFields = [], idField = 'id', extractField = (d, f) => d[f], searchOptions = {} } = options;
+        if (!(model instanceof StaticModel))
+            throw new Error('MiniSemanticSearch: the constructor needs a loaded model; use MiniSemanticSearch.create to load one');
+        if (!fields?.length)
+            throw new Error('MiniSemanticSearch: option "fields" must list at least one field');
+        this.model = model;
+        this.options = { fields, storeFields, idField, extractField, searchOptions };
+    }
+    /** How many documents are indexed. */
+    get documentCount() {
+        return this.index.size;
+    }
+    /** Adds a document; it can be searched straight away. Embedding takes about 0.1 ms. */
+    add(document) {
+        const { fields, storeFields, idField, extractField } = this.options;
+        const id = extractField(document, idField);
+        if (id === undefined)
+            throw new Error(`MiniSemanticSearch: document does not have ID field "${idField}"`);
+        if (this.index.has(id))
+            throw new Error(`MiniSemanticSearch: duplicate ID ${id}`);
+        const values = fields.map((f) => texts(extractField(document, f)));
+        const all = values.flat().join(' ');
+        this.index.set(id, {
+            fields: values.map((v) => v.map((s) => this.model.embed(s))),
+            whole: all ? this.model.embed(all) : null,
+            stored: Object.fromEntries(storeFields.map((f) => [f, extractField(document, f)])),
+        });
+    }
+    addAll(documents) {
+        for (const d of documents)
+            this.add(d);
+    }
+    /** Adds documents `chunkSize` at a time, yielding between chunks so a page stays responsive. */
+    async addAllAsync(documents, { chunkSize = 100 } = {}) {
+        for (let i = 0; i < documents.length; i += chunkSize) {
+            this.addAll(documents.slice(i, i + chunkSize));
+            await new Promise((resolve) => setTimeout(resolve, 0));
+        }
+    }
+    /** Removes a document, found by its id. */
+    remove(document) {
+        this.discard(this.options.extractField(document, this.options.idField));
+    }
+    /** Removes the document with this id. */
+    discard(id) {
+        if (!this.index.delete(id))
+            throw new Error(`MiniSemanticSearch: cannot discard document with ID ${id}: it is not in the index`);
+    }
+    discardAll(ids) {
+        for (const id of ids)
+            this.discard(id);
+    }
+    /** Removes the given documents, or every document if none are given. */
+    removeAll(documents) {
+        if (documents === undefined)
+            this.index.clear();
+        else
+            for (const d of documents)
+                this.remove(d);
+    }
+    /** Replaces the document that has the same id. */
+    replace(document) {
+        this.remove(document);
+        this.add(document);
+    }
+    has(id) {
+        return this.index.has(id);
+    }
+    /** The document's `storeFields`, or undefined if it isn't indexed. */
+    getStoredFields(id) {
+        return this.index.get(id)?.stored;
+    }
+    /** Every document, best first, with its id, score and stored fields. */
+    search(query, searchOptions = {}) {
+        const { fields: only, boost = {}, filter, minScore = -Infinity } = { ...this.options.searchOptions, ...searchOptions };
+        const { fields } = this.options;
+        const use = fields.map((f) => !only || only.includes(f));
+        const subset = use.some((u) => !u);
+        const q = this.model.embed(query);
+        const results = [];
+        for (const [id, doc] of this.index) {
+            let sum = 0, weight = 0;
+            doc.fields.forEach((vs, i) => {
+                if (!use[i] || !vs.length)
+                    return;
+                const w = boost[fields[i]] ?? 1;
+                let best = -Infinity;
+                for (const v of vs)
+                    best = Math.max(best, similarity(q, v));
+                sum += w * best;
+                weight += w;
+            });
+            if (!weight)
+                continue;
+            // The whole-document vector covers every field, so it only counts when every field does.
+            const score = subset || !doc.whole ? sum / weight : 0.5 * similarity(q, doc.whole) + 0.5 * (sum / weight);
+            if (score < minScore)
+                continue;
+            const result = { id, score, ...doc.stored }; // stored fields come last, as in MiniSearch
+            if (!filter || filter(result))
+                results.push(result);
+        }
+        return results.sort((a, b) => b.score - a.score);
+    }
+}

package/dist/tokenizer.d.ts

@@ -0,0 +1,6 @@
+/** Drops control characters, spaces out CJK characters, strips accents and lower-cases. */
+export declare function normalize(text: string): string;
+/** Splits normalised text into words and single punctuation characters. */
+export declare function preTokenize(text: string): string[];
+/** Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt. */
+export declare function wordPiece(vocab: Map<string, number>, unk: number, text: string): number[];

package/dist/tokenizer.js

@@ -0,0 +1,76 @@
+// BERT's tokenizer, as Hugging Face's `tokenizers` runs it for Model2Vec models (BertNormalizer with lowercasing,
+// BertPreTokenizer, WordPiece with "##" continuations). Checked token for token against it in the tests.
+const isPunct = (c) => {
+    const cp = c.codePointAt(0);
+    return (cp >= 33 && cp <= 47) || (cp >= 58 && cp <= 64) || (cp >= 91 && cp <= 96) || (cp >= 123 && cp <= 126)
+        || /\p{P}/u.test(c);
+};
+const isCjk = (cp) => (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf)
+    || (cp >= 0x20000 && cp <= 0x2a6df) || (cp >= 0x2a700 && cp <= 0x2b73f) || (cp >= 0x2b740 && cp <= 0x2b81f)
+    || (cp >= 0x2b820 && cp <= 0x2ceaf) || (cp >= 0xf900 && cp <= 0xfaff) || (cp >= 0x2f800 && cp <= 0x2fa1f);
+/** Drops control characters, spaces out CJK characters, strips accents and lower-cases. */
+export function normalize(text) {
+    let out = '';
+    for (const c of text) {
+        const cp = c.codePointAt(0);
+        if (cp === 0 || cp === 0xfffd || (/\p{C}/u.test(c) && !/[\t\n\r]/.test(c)))
+            continue;
+        if (/\s/u.test(c))
+            out += ' ';
+        else if (isCjk(cp))
+            out += ` ${c} `;
+        else
+            out += c;
+    }
+    return out.normalize('NFD').replace(/\p{Mn}/gu, '').toLowerCase();
+}
+/** Splits normalised text into words and single punctuation characters. */
+export function preTokenize(text) {
+    const words = [];
+    for (const chunk of text.split(' ')) {
+        let word = '';
+        for (const c of chunk) {
+            if (isPunct(c)) {
+                if (word)
+                    words.push(word);
+                words.push(c);
+                word = '';
+            }
+            else
+                word += c;
+        }
+        if (word)
+            words.push(word);
+    }
+    return words;
+}
+/** Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt. */
+export function wordPiece(vocab, unk, text) {
+    const ids = [];
+    for (const w of preTokenize(normalize(text))) {
+        const chars = Array.from(w);
+        if (chars.length > 100) {
+            ids.push(unk);
+            continue;
+        }
+        const pieces = [];
+        for (let start = 0; start < chars.length;) {
+            let end = chars.length;
+            let id;
+            for (; end > start; end--) {
+                id = vocab.get((start ? '##' : '') + chars.slice(start, end).join(''));
+                if (id !== undefined)
+                    break;
+            }
+            if (end === start) {
+                pieces.length = 0;
+                pieces.push(unk);
+                break;
+            }
+            pieces.push(id);
+            start = end;
+        }
+        ids.push(...pieces);
+    }
+    return ids;
+}

package/models/README.md

@@ -0,0 +1,13 @@
+# Bundled data
+
+This file is not covered by the package's MIT licence for its code; it keeps its source's terms.
+
+## potion-base-8M-128d.bin
+
+[minishlab/potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) by MinishLab, a
+[Model2Vec](https://github.com/MinishLab/model2vec) model distilled from
+[BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5), released under the MIT licence. Its
+vocabulary is bge-base-en-v1.5's (BERT's, uncased).
+
+Modified: repacked by `mini-semantic-search pack` to its first 128 of 256 dimensions at 4 bits per value (each
+token's 128 values sharing an f16 scale), 2.2 MB instead of 30.5 MB. All 29,528 tokens are kept.

package/package.json

@@ -1,6 +1,63 @@
 {
   "name": "mini-semantic-search",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "0.1.0",
+  "description": "Semantic search with a MiniSearch-style API, using Model2Vec static embeddings. Runs in browsers and Node; no dependencies; 2.2 MB with the model.",
+  "keywords": [
+    "semantic-search",
+    "search",
+    "minisearch",
+    "model2vec",
+    "embeddings",
+    "static-embeddings",
+    "sentence-embeddings",
+    "browser",
+    "offline"
+  ],
+  "license": "MIT",
+  "author": "ShakirasHips",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/ShakirasHips/model2vec-js.git"
+  },
+  "homepage": "https://github.com/ShakirasHips/model2vec-js#readme",
+  "bugs": "https://github.com/ShakirasHips/model2vec-js/issues",
+  "type": "module",
+  "exports": {
+    ".": {
+      "node": {
+        "types": "./dist/node.d.ts",
+        "default": "./dist/node.js"
+      },
+      "default": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.js"
+      }
+    },
+    "./models/*": "./models/*"
+  },
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "mini-semantic-search": "dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "models",
+    "!dist/**/*.test.*"
+  ],
+  "sideEffects": [
+    "./dist/node.js",
+    "./dist/cli.js"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "build": "tsc",
+    "test": "tsc && node --test \"test/*.test.mjs\"",
+    "prepublishOnly": "npm test"
+  },
+  "devDependencies": {
+    "@types/node": "^20.19.0",
+    "typescript": "^5.9.0"
+  }
+}