mini-semantic-search 0.1.0 → 0.2.0

package/README.md

@@ -3,7 +3,7 @@
 Semantic search with [MiniSearch](https://github.com/lucaong/minisearch)'s API. Documents are matched by meaning
 rather than shared words: "stop sending me newsletters" finds "How do I unsubscribe from your emails?", and "the mug
 came smashed" finds "My item arrived damaged or broken". It runs in browsers and Node with no dependencies, no
-server and no API calls, and the package is 2.2 MB with its model (1.8 MB gzipped).
+server and no API calls, and the package is 2.2 MB with its model (2.0 MB gzipped).
 
 ```ts
 import MiniSemanticSearch from 'mini-semantic-search';
@@ -21,15 +21,15 @@ search.addAll([
 
 search.search('the mug came smashed');
 // => [
-//   { id: 2, score: 0.39, title: 'Damaged items' },
-//   { id: 3, score: 0.15, title: 'Shipping times' },
-//   { id: 1, score: 0.08, title: 'Returns' },
+//   { id: 2, score: 0.29, title: 'Damaged items' },
+//   { id: 3, score: 0.16, title: 'Shipping times' },
+//   { id: 1, score: 0.02, title: 'Returns' },
 // ]
 ```
 
 The embeddings come from [Model2Vec](https://github.com/MinishLab/model2vec), a static embedding model. It keeps one
 vector per token, and a text's vector is the average of its tokens' vectors. There is no neural network to run, so
-embedding a document takes about 0.1 ms on a CPU.
+embedding a paragraph takes about 15 µs on a CPU.
 
 ## Install
 
@@ -44,7 +44,7 @@ npm install mini-semantic-search
 
   ```html
   <script type="module">
-    import MiniSemanticSearch from 'https://cdn.jsdelivr.net/npm/mini-semantic-search@0.1.0/dist/index.js';
+    import MiniSemanticSearch from 'https://cdn.jsdelivr.net/npm/mini-semantic-search@0.2.0/dist/index.js';
     const search = await MiniSemanticSearch.create({ fields: ['title', 'text'] });
   </script>
   ```
@@ -100,9 +100,11 @@ depends on your documents' wording. Try your own off-topic questions and put it
 
 **How it differs from MiniSearch.** Results have no `terms`, `queryTerms` or `match`, because nothing is matched
 word for word. There's no `prefix`, `fuzzy`, `combineWith`, `autoSuggest` or query syntax, and no
-`toJSON`/`loadJSON`: indexing costs about 0.1 ms per document, so you can just index again on load.
+`toJSON`/`loadJSON`: indexing costs about 0.05 ms per document, so you can just index again on load.
 
-Indexing takes 0.05–0.2 ms per document. A search over 10,000 documents takes about 8 ms.
+Indexing takes about 0.05 ms per document with three fields. A search over 10,000 documents takes about 4 ms, or
+about 13 ms when its `fields` or `boost` differ from the last search's. (Options used twice running get each
+document's vectors added up into one, so a search is one dot product per document.)
 
 ## Embeddings
 
@@ -110,7 +112,7 @@ Indexing takes 0.05–0.2 ms per document. A search over 10,000 documents takes
 import { StaticModel, similarity } from 'mini-semantic-search';
 
 const model = await StaticModel.load();            // or StaticModel.load(urlOrPathOrBytes)
-const a = model.embed('how do I cancel my order');  // Float32Array(128), unit length
+const a = model.embed('how do I cancel my order');  // Float32Array(256), unit length
 const b = model.embed('stop my purchase before it ships');
 similarity(a, b);                                   // cosine similarity, 1 = same meaning
 model.embedMany(['refund', 'exchange']);           // several at once
@@ -122,14 +124,14 @@ model the vectors match `model2vec` to within 1e-6. The tests check the token id
 against fixtures made with `scripts/make-fixture.py`.
 
 Any Model2Vec model with a lower-casing BERT tokenizer and no token weights can be packed (most `minishlab/potion-*`
-models). A bigger model is more accurate. The full 256 dimensions of potion-base-8M at 4 bits are 4.2 MB:
+models). At 4 bits instead of 2, potion-base-8M is 4.1 MB and scores as the full-precision model does:
 
 ```sh
-npx mini-semantic-search pack ./potion-base-8M ./potion-base-8M.bin --bits 4
+npx mini-semantic-search pack ./potion-base-8M ./potion-base-8M-4bit.bin --bits 4
 ```
 
 ```ts
-const search = await MiniSemanticSearch.create({ fields: ['title', 'text'], model: '/potion-base-8M.bin' });
+const search = await MiniSemanticSearch.create({ fields: ['title', 'text'], model: '/potion-base-8M-4bit.bin' });
 ```
 
 ## Command line
@@ -137,7 +139,7 @@ const search = await MiniSemanticSearch.create({ fields: ['title', 'text'], mode
 ```sh
 npx mini-semantic-search search docs.json "how do I get a refund" --fields title,text [--k 5]
 npx mini-semantic-search embed "some text"                  # the vector as JSON
-npx mini-semantic-search pack <model folder> out.bin [--bits 4|8|32] [--group 64] [--dims N]
+npx mini-semantic-search pack <model folder> out.bin [--bits 2|4|8|32] [--dims N]
 ```
 
 ## How well it works
@@ -147,13 +149,31 @@ test questions), with the right entry ranked first or in the top 3:
 
 | Model | Size | `boost` | First | Top 3 |
 |---|---|---|---|---|
-| potion-base-8M, 256 dims at 4 bits | 4.2 MB | `{ q: 3 }` | 63.0% | 87.4% |
-| **potion-base-8M, 128 dims at 4 bits (bundled)** | **2.2 MB** | none | **59.3%** | **80.0%** |
-| **potion-base-8M, 128 dims at 4 bits (bundled)** | **2.2 MB** | `{ q: 3 }` | **60.7%** | **83.7%** |
+| **potion-base-8M at 2 bits (bundled)** | **2.2 MB** | none | **66.7%** | **83.7%** |
+| **potion-base-8M at 2 bits (bundled)** | **2.2 MB** | `{ q: 3 }` | **65.2%** | **85.9%** |
+| potion-base-8M at 4 bits | 4.1 MB | `{ q: 3 }` | 65.2% | 88.1% |
+| potion-base-8M, full precision | 30.5 MB | `{ q: 3 }` | 64.4% | 88.1% |
+| 0.1's model: 128 of the 256 dimensions at 4 bits | 2.2 MB | `{ q: 3 }` | 60.7% | 83.7% |
 
 The test questions were written with the entries in view, so expect real questions to do somewhat worse. The set
-is small: one question is 0.7%. The bundled model keeps potion-base-8M's first 128 dimensions (its PCA components
-with the most variance), and its vectors are within cosine 0.98 of the full-precision ones cut to 128.
+is small: one question is 0.7%. Two public benchmarks are bigger. In [Banking77](https://huggingface.co/datasets/mteb/banking77),
+3,080 customer questions each count as right when the closest of 10,003 other questions has the same intent (of 77).
+In [SciFact](https://huggingface.co/datasets/mteb/scifact), 300 claims look for the abstracts about them among 5,183
+(nDCG@10). `scripts/eval.mjs` runs both:
+
+| Model | Size | Banking77 | SciFact |
+|---|---|---|---|
+| **potion-base-8M at 2 bits (bundled)** | **2.2 MB** | **88.3%** | **50.1** |
+| potion-base-8M at 4 bits | 4.1 MB | 88.3% | 50.4 |
+| potion-base-8M, full precision | 30.5 MB | 88.5% | 50.6 |
+| 0.1's model: 128 of the 256 dimensions at 4 bits | 2.2 MB | 87.3% | 45.9 |
+
+**How the model is packed.** Each value is one of four levels fitted to the values (Lloyd-Max), times a scale for
+its dimension and one for its token. The scale per dimension matters because Model2Vec's dimensions are PCA
+components whose spread falls about fivefold from the first to the last. At 2 bits a single vector is noisy (cosine
+about 0.95 to the full-precision one), but the noise is spread across 256 dimensions and mostly cancels out when
+vectors are compared, so it loses less than keeping half the dimensions at 4 bits. The model stays packed in memory,
+so it takes about 3 MB of it rather than 30 MB, and loads in about 6 ms.
 
 **What it can't do:**
 - Averaging tokens loses word order, so "can I add a shopper" and "can a shopper add me" look alike.
@@ -169,10 +189,11 @@ npm install
 npm test                       # builds, then runs the tests (Node 21+)
 sh scripts/smoke.sh            # packs the package, installs it in an empty project and uses it from there
 python scripts/make-fixture.py <potion-base-8M folder>   # rebuilds the reference fixture (needs model2vec)
+node scripts/eval.mjs <data folder> <model.bin> ...       # Banking77 and SciFact (see the script for the data)
 ```
 
 The bundled model is rebuilt with
-`node dist/cli.js pack <potion-base-8M folder> models/potion-base-8M-128d.bin --bits 4 --dims 128 --group 128`.
+`node dist/cli.js pack <potion-base-8M folder> models/potion-base-8M.bin --bits 2`.
 
 CI (`.github/workflows/ci.yml`) builds every push and pull request on Node 18, 20, 22 and 24, runs the tests (22
 and 24) and the install check (all four).
@@ -185,8 +206,8 @@ repository `ShakirasHips/model2vec-js`, workflow `release.yml`, environment `npm
 ## Credits and licence
 
 The code is MIT-licensed (`LICENSE`). The bundled model is
-[potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) by MinishLab, MIT, repacked at 128 dimensions and
-4 bits (see `models/README.md`). The API follows [MiniSearch](https://github.com/lucaong/minisearch) by Luca Ongaro.
+[potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) by MinishLab, MIT, repacked at 2 bits (see
+`models/README.md`). The API follows [MiniSearch](https://github.com/lucaong/minisearch) by Luca Ongaro.
 
 This package reimplements Model2Vec's inference and BERT's tokenizer in TypeScript; it contains no code from
 Model2Vec, BERT or MiniSearch.

package/dist/cli.js

@@ -7,10 +7,10 @@ const HELP = `mini-semantic-search: semantic search with Model2Vec static embedd
       The documents closest in meaning to the query. docs.json is an array of objects.
   mini-semantic-search embed "<text>" [--model m.bin]
       The text's vector as JSON.
-  mini-semantic-search pack <model folder> <out.bin> [--bits 4|8|32] [--group 64] [--dims N]
-      Packs a Model2Vec model from Hugging Face (model.safetensors + tokenizer.json).
+  mini-semantic-search pack <model folder> <out.bin> [--bits 2|4|8|32] [--dims N] [--group 64]
+      Packs a Model2Vec model from Hugging Face (model.safetensors + tokenizer.json). --group is for 8 bits.
 
---model defaults to the bundled potion-base-8M (128 dimensions at 4 bits).`;
+--model defaults to the bundled potion-base-8M (256 dimensions at 2 bits).`;
 const argv = process.argv.slice(2);
 const flags = new Map();
 const args = [];
@@ -41,8 +41,8 @@ async function main() {
     }
     else if (cmd === 'pack' && rest.length === 2) {
         const bits = num('bits', 4);
-        if (bits !== 4 && bits !== 8 && bits !== 32)
-            throw new Error('--bits must be 4, 8 or 32');
+        if (bits !== 2 && bits !== 4 && bits !== 8 && bits !== 32)
+            throw new Error('--bits must be 2, 4, 8 or 32');
         const bytes = await packModel(rest[0], { bits, group: num('group', 64), dims: flags.has('dims') ? num('dims', 0) : undefined });
         await writeFile(rest[1], bytes);
         const m = StaticModel.fromBytes(bytes);

package/dist/format.d.ts

@@ -4,13 +4,29 @@ export interface ModelData {
     /** tokens.length × dims values, row by row. */
     vectors: Float32Array;
 }
+/** A version 2 model as stored: value (row, d) = levels[code] × rowScales[row] × colScales[d]. */
+export interface PackedModelData {
+    tokens: string[];
+    dims: number;
+    bits: 2 | 4;
+    levels: Float32Array;
+    colScales: Float32Array;
+    rowScales: Float32Array;
+    /** dims × bits / 8 bytes per row. */
+    codes: Uint8Array;
+}
 export interface EncodeOptions {
-    /** 32, 8 or 4 (default 4). */
-    bits?: 32 | 8 | 4;
-    /** Values sharing one scale (default 64); must divide the number of dimensions. */
+    /** 32, 8, 4 or 2 (default 4). 4 and 2 use a learned table of levels; 8 a scale per group. */
+    bits?: 32 | 8 | 4 | 2;
+    /** At 8 bits, the values sharing one scale (default 64); must divide the number of dimensions. */
     group?: number;
 }
 export declare function fromF16(h: number): number;
 export declare function toF16(x: number): number;
+/** Reads a model file, leaving a version 2 model packed (as `StaticModel` keeps it). */
+export declare function decodePacked(bytes: Uint8Array): ModelData | PackedModelData;
+/** The full table of a packed model. */
+export declare function unpack({ tokens, dims, bits, levels, colScales, rowScales, codes }: PackedModelData): ModelData;
+/** Reads a model file into its full table of vectors. */
 export declare function decodeModel(bytes: Uint8Array): ModelData;
-export declare function encodeModel({ tokens, dims, vectors }: ModelData, { bits, group }?: EncodeOptions): Uint8Array;
+export declare function encodeModel({ tokens, dims, vectors }: ModelData, options?: EncodeOptions): Uint8Array;

package/dist/format.js

@@ -1,10 +1,18 @@
-// The packed model file. Little-endian: magic "FAQM", version, vocabulary size, dimensions, bits (32, 8 or 4),
-// group size and the vocabulary's byte length; then the tokens joined by "\n", padded to 4 bytes; then the
-// vectors. At 32 bits they are plain f32. At 8 bits each value is an int8, and at 4 bits two values share a byte
-// (low nibble first, each nibble − 8); either way every `group` values share one f16 scale, and all the scales
-// come first, padded to 4 bytes.
+// The packed model file. Little-endian. It starts with seven u32s: magic "FAQM", version, vocabulary size, dimensions,
+// bits, group size (version 1 only, else 0) and the vocabulary's byte length; then the tokens joined by "\n", padded
+// to 4 bytes; then the vectors.
+//
+// Version 2 (2 or 4 bits): each value is a code into a learned table of 2^bits levels, scaled by its column's scale
+// and its row's scale. After the vocabulary come the levels (f32), the column scales (one f32 per dimension), the row
+// scales (one f16 per token, padded to 4 bytes) and the codes, each row in dims × bits / 8 bytes, low bits first.
+// Model2Vec's dimensions are PCA components whose spread falls about fivefold from the first to the last, so a scale
+// per column lets every dimension use all the levels, and a table fitted to the values (Lloyd-Max) puts the levels
+// where the values are.
+//
+// Version 1 (32, 8 or 4 bits), still read: at 32 bits plain f32; at 8 bits each value is an int8, and at 4 bits two
+// values share a byte (low nibble first, each nibble − 8); either way every `group` values share one f16 scale, and
+// all the scales come first, padded to 4 bytes.
 const MAGIC = 0x4d514146;
-const VERSION = 1;
 export function fromF16(h) {
     const e = (h >> 10) & 31, m = h & 1023, s = h & 0x8000 ? -1 : 1;
     if (e === 0)
@@ -25,23 +33,43 @@ export function toF16(x) {
     return m === 1024 ? s | ((e + 1) << 10) : s | (e << 10) | m;
 }
 const pad4 = (n) => (n + 3) & ~3;
-export function decodeModel(bytes) {
+/** Reads a model file, leaving a version 2 model packed (as `StaticModel` keeps it). */
+export function decodePacked(bytes) {
     const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
     if (bytes.byteLength < 28 || dv.getUint32(0, true) !== MAGIC)
         throw new Error('not a model2vec-js model file');
-    const header = Array.from({ length: 7 }, (_, i) => dv.getUint32(i * 4, true));
-    const [, version, n, dims, bits, group, vocabLen] = header;
-    if (version !== VERSION)
-        throw new Error(`unsupported model file version ${version}`);
+    const [, version, n, dims, bits, group, vocabLen] = Array.from({ length: 7 }, (_, i) => dv.getUint32(i * 4, true));
+    if (version !== 1 && version !== 2)
+        throw new Error(`unsupported model file version ${version}; a newer mini-semantic-search may read it`);
     let off = 28;
     const tokens = new TextDecoder().decode(bytes.subarray(off, off + vocabLen)).split('\n');
     if (tokens.length !== n)
         throw new Error('model file is corrupt: vocabulary size');
     off += pad4(vocabLen);
     const count = n * dims;
+    if (version === 2) {
+        if (bits !== 2 && bits !== 4)
+            throw new Error(`unsupported bits: ${bits}`);
+        const k = 1 << bits, rowBytes = (dims * bits) / 8;
+        if (bytes.byteLength < off + 4 * (k + dims) + pad4(n * 2) + n * rowBytes)
+            throw new Error('model file is corrupt: too short');
+        const f32 = (len) => {
+            const a = new Float32Array(len);
+            for (let i = 0; i < len; i++, off += 4)
+                a[i] = dv.getFloat32(off, true);
+            return a;
+        };
+        const levels = f32(k), colScales = f32(dims);
+        const rowScales = new Float32Array(n);
+        for (let i = 0; i < n; i++)
+            rowScales[i] = fromF16(dv.getUint16(off + i * 2, true));
+        off += pad4(n * 2);
+        return { tokens, dims, bits, levels, colScales, rowScales, codes: bytes.subarray(off, off + n * rowBytes) };
+    }
     const vectors = new Float32Array(count);
     if (bits === 32) {
-        vectors.set(new Float32Array(bytes.slice(off, off + count * 4).buffer));
+        // Copied byte for byte: a Node Buffer's `slice` is a view, so its `.buffer` can hold much more than this.
+        new Uint8Array(vectors.buffer).set(bytes.subarray(off, off + count * 4));
     }
     else if (bits === 8 || bits === 4) {
         const groups = count / group;
@@ -59,42 +87,107 @@ export function decodeModel(bytes) {
     }
     return { tokens, dims, vectors };
 }
-export function encodeModel({ tokens, dims, vectors }, { bits = 4, group = 64 } = {}) {
-    const n = tokens.length, count = n * dims;
-    if (vectors.length !== count)
-        throw new Error('vectors must hold tokens.length × dims values');
-    if (tokens.some((t) => t.includes('\n')))
-        throw new Error('tokens must not contain newlines');
-    if (bits !== 32 && dims % group)
-        throw new Error(`group (${group}) must divide dims (${dims})`);
-    const vocab = new TextEncoder().encode(tokens.join('\n'));
-    let body;
-    if (bits === 32) {
-        body = new Uint8Array(new Float32Array(vectors).buffer);
+/** The full table of a packed model. */
+export function unpack({ tokens, dims, bits, levels, colScales, rowScales, codes }) {
+    const vectors = new Float32Array(tokens.length * dims), mask = (1 << bits) - 1;
+    for (let i = 0; i < vectors.length; i++) {
+        const bit = i * bits, code = (codes[bit >> 3] >> (bit & 7)) & mask;
+        vectors[i] = levels[code] * rowScales[(i / dims) | 0] * colScales[i % dims];
     }
-    else {
-        const levels = bits === 8 ? 127 : 7, groups = count / group;
-        const scaleBytes = pad4(groups * 2);
-        body = new Uint8Array(scaleBytes + (bits === 8 ? count : Math.ceil(count / 2)));
-        const dv = new DataView(body.buffer);
+    return { tokens, dims, vectors };
+}
+/** Reads a model file into its full table of vectors. */
+export function decodeModel(bytes) {
+    const data = decodePacked(bytes);
+    return 'vectors' in data ? data : unpack(data);
+}
+/** Lloyd-Max: the `k` levels that minimise the squared error of rounding `values` (sorted) to the nearest one. */
+function fitLevels(values, k, iterations = 30) {
+    const levels = Float32Array.from({ length: k }, (_, i) => values[Math.floor(((i + 0.5) / k) * values.length)]);
+    for (let it = 0; it < iterations; it++) {
+        const sum = new Float64Array(k), count = new Float64Array(k);
+        for (let i = 0, j = 0; i < values.length; i++) {
+            const x = values[i];
+            while (j < k - 1 && Math.abs(x - levels[j + 1]) <= Math.abs(x - levels[j]))
+                j++;
+            sum[j] += x;
+            count[j]++;
+        }
+        for (let j = 0; j < k; j++)
+            if (count[j])
+                levels[j] = sum[j] / count[j];
+    }
+    return levels;
+}
+function encodeBody(n, dims, vectors, { bits = 4, group = 64 }) {
+    if (bits === 32)
+        return new Uint8Array(new Float32Array(vectors).buffer);
+    if (bits === 8) {
+        if (dims % group)
+            throw new Error(`group (${group}) must divide dims (${dims})`);
+        const count = n * dims, groups = count / group, scaleBytes = pad4(groups * 2);
+        const body = new Uint8Array(scaleBytes + count), dv = new DataView(body.buffer);
         for (let g = 0; g < groups; g++) {
             let max = 0;
             for (let i = g * group; i < (g + 1) * group; i++)
                 max = Math.max(max, Math.abs(vectors[i]));
-            const h = toF16(max / levels), scale = fromF16(h) || 1;
+            const h = toF16(max / 127), scale = fromF16(h) || 1;
             dv.setUint16(g * 2, h, true);
-            for (let i = g * group; i < (g + 1) * group; i++) {
-                const v = Math.max(-levels, Math.min(levels, Math.round(vectors[i] / scale)));
-                if (bits === 8)
-                    dv.setInt8(scaleBytes + i, v);
-                else
-                    body[scaleBytes + (i >> 1)] |= (v + 8) << ((i & 1) * 4);
-            }
+            for (let i = g * group; i < (g + 1) * group; i++)
+                dv.setInt8(scaleBytes + i, Math.max(-127, Math.min(127, Math.round(vectors[i] / scale))));
         }
+        return body;
+    }
+    if (bits !== 4 && bits !== 2)
+        throw new Error(`unsupported bits: ${bits}`);
+    if ((dims * bits) % 8)
+        throw new Error(`dims (${dims}) × bits (${bits}) must be a whole number of bytes`);
+    const k = 1 << bits, rowBytes = (dims * bits) / 8;
+    // Column scales, then row scales over the column-scaled values, as stored (f32, f16).
+    const col = new Float64Array(dims);
+    for (let i = 0; i < vectors.length; i++)
+        col[i % dims] += vectors[i] ** 2;
+    const colScales = Float32Array.from(col, (s) => Math.sqrt(s / n) || 1);
+    const scaled = Float32Array.from(vectors, (x, i) => x / colScales[i % dims]);
+    const rowHalves = new Uint16Array(n), rowScales = new Float32Array(n);
+    for (let r = 0; r < n; r++) {
+        let s = 0;
+        for (let d = 0; d < dims; d++)
+            s += scaled[r * dims + d] ** 2;
+        rowHalves[r] = toF16(Math.sqrt(s / dims));
+        rowScales[r] = fromF16(rowHalves[r]);
+        const div = rowScales[r] || 1;
+        for (let d = 0; d < dims; d++)
+            scaled[r * dims + d] /= div;
     }
+    const levels = fitLevels(scaled.slice().sort(), k);
+    const bounds = levels.subarray(1).map((l, j) => (levels[j] + l) / 2);
+    const body = new Uint8Array(4 * (k + dims) + pad4(n * 2) + n * rowBytes), dv = new DataView(body.buffer);
+    let off = 0;
+    for (const x of [...levels, ...colScales])
+        dv.setFloat32((off += 4) - 4, x, true);
+    rowHalves.forEach((h, r) => dv.setUint16(off + r * 2, h, true));
+    off += pad4(n * 2);
+    for (let i = 0; i < scaled.length; i++) {
+        let code = 0;
+        while (code < k - 1 && scaled[i] > bounds[code])
+            code++;
+        body[off + ((i * bits) >> 3)] |= code << ((i * bits) & 7);
+    }
+    return body;
+}
+export function encodeModel({ tokens, dims, vectors }, options = {}) {
+    const n = tokens.length, bits = options.bits ?? 4;
+    if (vectors.length !== n * dims)
+        throw new Error('vectors must hold tokens.length × dims values');
+    if (tokens.some((t) => t.includes('\n')))
+        throw new Error('tokens must not contain newlines');
+    const vocab = new TextEncoder().encode(tokens.join('\n'));
+    const body = encodeBody(n, dims, vectors, { ...options, bits });
+    const version = bits === 4 || bits === 2 ? 2 : 1, group = bits === 8 ? options.group ?? 64 : 0;
     const out = new Uint8Array(28 + pad4(vocab.length) + body.length);
     const dv = new DataView(out.buffer);
-    [MAGIC, VERSION, n, dims, bits, group, vocab.length].forEach((v, i) => dv.setUint32(i * 4, v, true));
+    [MAGIC, version, n, dims, bits, group, vocab.length].forEach((v, i) => dv.setUint32(i * 4, v, true));
     out.set(vocab, 28);
     out.set(body, 28 + pad4(vocab.length));
     return out;

package/dist/load.d.ts

@@ -3,7 +3,7 @@ export type Source = string | URL | ArrayBuffer | ArrayBufferView;
 type FileReader = (path: string | URL) => Promise<Uint8Array>;
 /** @internal Used by the Node entry point. */
 export declare function setFileReader(reader: FileReader): void;
-/** The bundled model: potion-base-8M's first 128 dimensions, packed at 4 bits. */
+/** The bundled model: potion-base-8M, all 256 dimensions packed at 2 bits. */
 export declare const DEFAULT_MODEL: URL;
 export declare function loadBytes(src: Source): Promise<Uint8Array>;
 export {};

package/dist/load.js

@@ -5,8 +5,8 @@ let readFile = null;
 export function setFileReader(reader) {
     readFile = reader;
 }
-/** The bundled model: potion-base-8M's first 128 dimensions, packed at 4 bits. */
-export const DEFAULT_MODEL = new URL('../models/potion-base-8M-128d.bin', import.meta.url);
+/** The bundled model: potion-base-8M, all 256 dimensions packed at 2 bits. */
+export const DEFAULT_MODEL = new URL('../models/potion-base-8M.bin', import.meta.url);
 export async function loadBytes(src) {
     if (src instanceof ArrayBuffer)
         return new Uint8Array(src);

package/dist/model.d.ts

@@ -1,24 +1,30 @@
-import { type ModelData } from './format.js';
+import { type ModelData, type PackedModelData } from './format.js';
 import { type Source } from './load.js';
 /**
  * A Model2Vec static embedding model: one vector per token, and a sentence's vector is the mean of its tokens'
  * vectors, normalised to length 1. No network runs, so embedding takes microseconds. Long texts are cut as the
  * Python library cuts them: to `maxTokens` × the median token length in characters, then to `maxTokens` tokens,
  * before unknown tokens are left out.
+ *
+ * A packed model (2 or 4 bits) stays packed in memory: about 2 MB for the bundled one instead of 30 MB.
  */
 export declare class StaticModel {
     readonly tokens: string[];
     readonly dims: number;
-    readonly vectors: Float32Array;
     readonly unk: number;
     /** Tokens read per text; Model2Vec's default is 512. */
     maxTokens: number;
     private readonly vocab;
     private readonly medianTokenLength;
-    constructor({ tokens, dims, vectors }: ModelData);
-    /** Loads a packed model file; by default the bundled potion-base-8M (128 dimensions at 4 bits, 2.2 MB). */
+    private readonly longestPiece;
+    private table;
+    private readonly packed;
+    constructor(data: ModelData | PackedModelData);
+    /** Loads a model file; by default the bundled potion-base-8M (256 dimensions at 2 bits, 2.2 MB). */
     static load(src?: Source): Promise<StaticModel>;
     static fromBytes(bytes: Uint8Array): StaticModel;
+    /** The table of token vectors, tokens.length × dims, row by row (unpacked on first use for a packed model). */
+    get vectors(): Float32Array;
     /** Token ids, as Hugging Face's tokenizer gives them without special tokens (unknown words included as UNK). */
     tokenize(text: string): number[];
     /** Whether `token` (e.g. "voucher" or "##ing") is in the vocabulary. */

package/dist/model.js

@@ -1,4 +1,4 @@
-import { decodeModel } from './format.js';
+import { decodePacked, unpack } from './format.js';
 import { DEFAULT_MODEL, loadBytes } from './load.js';
 import { wordPiece } from './tokenizer.js';
 /**
@@ -6,35 +6,71 @@ import { wordPiece } from './tokenizer.js';
  * vectors, normalised to length 1. No network runs, so embedding takes microseconds. Long texts are cut as the
  * Python library cuts them: to `maxTokens` × the median token length in characters, then to `maxTokens` tokens,
  * before unknown tokens are left out.
+ *
+ * A packed model (2 or 4 bits) stays packed in memory: about 2 MB for the bundled one instead of 30 MB.
  */
 export class StaticModel {
     tokens;
     dims;
-    vectors;
     unk;
     /** Tokens read per text; Model2Vec's default is 512. */
     maxTokens = 512;
     vocab;
     medianTokenLength;
-    constructor({ tokens, dims, vectors }) {
+    longestPiece;
+    table;
+    packed;
+    constructor(data) {
+        const { tokens, dims } = data;
         this.tokens = tokens;
         this.dims = dims;
-        this.vectors = vectors;
         this.vocab = new Map(tokens.map((t, i) => [t, i]));
         this.unk = this.vocab.get('[UNK]') ?? -1;
-        const lengths = tokens.map((t) => Array.from(t).length).sort((a, b) => a - b), mid = lengths.length >> 1;
-        this.medianTokenLength = Math.trunc(lengths.length % 2 ? lengths[mid] : (lengths[mid - 1] + lengths[mid]) / 2);
+        // Token lengths in characters, counted (they are short), for the median and the longest word piece.
+        const counts = [];
+        let longest = 0;
+        for (const t of tokens) {
+            const len = t.length === 1 || !/[\uD800-\uDFFF]/.test(t) ? t.length : Array.from(t).length;
+            counts[len] = (counts[len] ?? 0) + 1;
+            longest = Math.max(longest, len);
+        }
+        const nth = (k) => {
+            for (let len = 0, seen = 0;; len++)
+                if ((seen += counts[len] ?? 0) > k)
+                    return len;
+        };
+        const mid = tokens.length >> 1;
+        this.medianTokenLength = Math.trunc(tokens.length % 2 ? nth(mid) : (nth(mid - 1) + nth(mid)) / 2);
+        this.longestPiece = longest;
+        if ('vectors' in data) {
+            this.table = data.vectors;
+            this.packed = null;
+        }
+        else {
+            // Each code byte holds `perByte` values; `lookup` gives their levels for every possible byte.
+            const perByte = 8 / data.bits, mask = (1 << data.bits) - 1;
+            const lookup = new Float32Array(256 * perByte);
+            for (let b = 0; b < 256; b++)
+                for (let j = 0; j < perByte; j++)
+                    lookup[b * perByte + j] = data.levels[(b >> (j * data.bits)) & mask];
+            this.table = null;
+            this.packed = { ...data, lookup, perByte, rowBytes: (dims * data.bits) / 8 };
+        }
     }
-    /** Loads a packed model file; by default the bundled potion-base-8M (128 dimensions at 4 bits, 2.2 MB). */
+    /** Loads a model file; by default the bundled potion-base-8M (256 dimensions at 2 bits, 2.2 MB). */
     static async load(src = DEFAULT_MODEL) {
         return StaticModel.fromBytes(await loadBytes(src));
     }
     static fromBytes(bytes) {
-        return new StaticModel(decodeModel(bytes));
+        return new StaticModel(decodePacked(bytes));
+    }
+    /** The table of token vectors, tokens.length × dims, row by row (unpacked on first use for a packed model). */
+    get vectors() {
+        return (this.table ??= unpack(this.packed).vectors);
     }
     /** Token ids, as Hugging Face's tokenizer gives them without special tokens (unknown words included as UNK). */
     tokenize(text) {
-        return wordPiece(this.vocab, this.unk, text);
+        return wordPiece(this.vocab, this.unk, text, this.longestPiece);
     }
     /** Whether `token` (e.g. "voucher" or "##ing") is in the vocabulary. */
     has(token) {
@@ -42,16 +78,49 @@ export class StaticModel {
     }
     /** The text's vector: unit length, or all zeros if it has no known tokens. */
     embed(text) {
-        const { dims, vectors } = this;
+        const { dims, packed } = this;
         const v = new Float32Array(dims);
         const chars = this.maxTokens * this.medianTokenLength;
         if (text.length > chars)
             text = Array.from(text).slice(0, chars).join('');
-        for (const id of this.tokenize(text).slice(0, this.maxTokens)) {
-            if (id === this.unk)
-                continue;
-            for (let d = 0, o = id * dims; d < dims; d++)
-                v[d] += vectors[o + d];
+        const ids = this.tokenize(text);
+        const end = Math.min(ids.length, this.maxTokens);
+        if (packed) {
+            const { codes, rowScales, lookup, perByte, rowBytes, colScales } = packed;
+            for (let t = 0; t < end; t++) {
+                const id = ids[t];
+                if (id === this.unk)
+                    continue;
+                const r = rowScales[id];
+                if (perByte === 4) {
+                    for (let p = id * rowBytes, d = 0; d < dims; p++, d += 4) {
+                        const o = codes[p] << 2;
+                        v[d] += lookup[o] * r;
+                        v[d + 1] += lookup[o + 1] * r;
+                        v[d + 2] += lookup[o + 2] * r;
+                        v[d + 3] += lookup[o + 3] * r;
+                    }
+                }
+                else {
+                    for (let p = id * rowBytes, d = 0; d < dims; p++, d += 2) {
+                        const o = codes[p] << 1;
+                        v[d] += lookup[o] * r;
+                        v[d + 1] += lookup[o + 1] * r;
+                    }
+                }
+            }
+            for (let d = 0; d < dims; d++)
+                v[d] *= colScales[d];
+        }
+        else {
+            const table = this.table;
+            for (let t = 0; t < end; t++) {
+                const id = ids[t];
+                if (id === this.unk)
+                    continue;
+                for (let d = 0, o = id * dims; d < dims; d++)
+                    v[d] += table[o + d];
+            }
         }
         let norm = 0;
         for (let d = 0; d < dims; d++)

package/dist/search.d.ts

@@ -42,6 +42,9 @@ export declare class MiniSemanticSearch<T = AnyDocument> {
     readonly model: StaticModel;
     private readonly options;
     private readonly index;
+    private plan;
+    /** The weighting of the last search that had no plan: a plan is made when one is used twice running. */
+    private lastKey;
     /** Loads the model (the bundled one unless `options.model` says otherwise), then makes the index. */
     static create<T = AnyDocument>(options: Options<T>): Promise<MiniSemanticSearch<T>>;
     /** For a model you have loaded already (`StaticModel.load()`); `create` loads it for you. */
@@ -50,7 +53,7 @@ export declare class MiniSemanticSearch<T = AnyDocument> {
     });
     /** How many documents are indexed. */
     get documentCount(): number;
-    /** Adds a document; it can be searched straight away. Embedding takes about 0.1 ms. */
+    /** Adds a document; it can be searched straight away. Embedding takes about 0.05 ms. */
     add(document: T): void;
     addAll(documents: readonly T[]): void;
     /** Adds documents `chunkSize` at a time, yielding between chunks so a page stays responsive. */
@@ -71,5 +74,11 @@ export declare class MiniSemanticSearch<T = AnyDocument> {
     getStoredFields(id: DocumentId): Record<string, unknown> | undefined;
     /** Every document, best first, with its id, score and stored fields. */
     search(query: string, searchOptions?: SearchOptions): SearchResult[];
+    /** The plan for this weighting: the current one, a new one if it was also the last search's, or else none. */
+    private planFor;
+    /** Adds the document's row to the plan. */
+    private place;
+    /** Takes the document's row out of the plan, moving the last row into its place. */
+    private unplace;
 }
 export {};

package/dist/search.js

@@ -1,5 +1,60 @@
 import { DEFAULT_MODEL } from './load.js';
-import { similarity, StaticModel } from './model.js';
+import { StaticModel } from './model.js';
+const pack8 = (v) => {
+    let max = 0;
+    for (let i = 0; i < v.length; i++)
+        max = Math.max(max, Math.abs(v[i]));
+    const s = max / 127 || 1, q = new Int8Array(v.length);
+    for (let i = 0; i < v.length; i++)
+        q[i] = Math.round(v[i] / s);
+    return { q, s };
+};
+/** q · row `o` of m. */
+const dot = (q, m, o) => {
+    let a = 0, b = 0, c = 0, d = 0, i = 0;
+    for (; i + 3 < q.length; i += 4) {
+        a += q[i] * m[o + i];
+        b += q[i + 1] * m[o + i + 1];
+        c += q[i + 2] * m[o + i + 2];
+        d += q[i + 3] * m[o + i + 3];
+    }
+    for (; i < q.length; i++)
+        a += q[i] * m[o + i];
+    return a + b + c + d;
+};
+const dot8 = (q, { q: v, s }) => {
+    let a = 0, b = 0, c = 0, d = 0, i = 0;
+    for (; i + 3 < q.length; i += 4) {
+        a += q[i] * v[i];
+        b += q[i + 1] * v[i + 1];
+        c += q[i + 2] * v[i + 2];
+        d += q[i + 3] * v[i + 3];
+    }
+    for (; i < q.length; i++)
+        a += q[i] * v[i];
+    return (a + b + c + d) * s;
+};
+/**
+ * A document's score as terms, each a weight times the best similarity among some vectors (a single-valued field has
+ * one): half the whole vector's similarity plus half the fields' similarities averaged by weight, or just the fields'
+ * average when only some fields count. Null if no field with a value counts.
+ */
+function terms(doc, { use, weights, subset }) {
+    let weight = 0;
+    for (let i = 0; i < doc.fields.length; i++)
+        if (use[i] && doc.fields[i].length)
+            weight += weights[i];
+    if (!weight)
+        return null;
+    // The whole-document vector covers every field, so it only counts when every field does.
+    const whole = subset || !doc.whole ? 0 : 0.5;
+    const out = whole ? [[whole, [doc.whole]]] : [];
+    for (let i = 0; i < doc.fields.length; i++) {
+        if (use[i] && doc.fields[i].length)
+            out.push([((1 - whole) * weights[i]) / weight, doc.fields[i]]);
+    }
+    return out;
+}
 const texts = (value) => {
     if (value === undefined || value === null)
         return [];
@@ -15,6 +70,9 @@ export class MiniSemanticSearch {
     model;
     options;
     index = new Map();
+    plan = null;
+    /** The weighting of the last search that had no plan: a plan is made when one is used twice running. */
+    lastKey = '';
     /** Loads the model (the bundled one unless `options.model` says otherwise), then makes the index. */
     static async create(options) {
         const { model = DEFAULT_MODEL } = options;
@@ -34,7 +92,7 @@ export class MiniSemanticSearch {
     get documentCount() {
         return this.index.size;
     }
-    /** Adds a document; it can be searched straight away. Embedding takes about 0.1 ms. */
+    /** Adds a document; it can be searched straight away. Embedding takes about 0.05 ms. */
     add(document) {
         const { fields, storeFields, idField, extractField } = this.options;
         const id = extractField(document, idField);
@@ -44,11 +102,16 @@ export class MiniSemanticSearch {
             throw new Error(`MiniSemanticSearch: duplicate ID ${id}`);
         const values = fields.map((f) => texts(extractField(document, f)));
         const all = values.flat().join(' ');
-        this.index.set(id, {
-            fields: values.map((v) => v.map((s) => this.model.embed(s))),
-            whole: all ? this.model.embed(all) : null,
+        const doc = {
+            id,
+            fields: values.map((v) => v.map((s) => pack8(this.model.embed(s)))),
+            whole: all ? pack8(this.model.embed(all)) : null,
             stored: Object.fromEntries(storeFields.map((f) => [f, extractField(document, f)])),
-        });
+            row: -1,
+        };
+        this.index.set(id, doc);
+        if (this.plan)
+            this.place(this.plan, doc);
     }
     addAll(documents) {
         for (const d of documents)
@@ -67,8 +130,11 @@ export class MiniSemanticSearch {
     }
     /** Removes the document with this id. */
     discard(id) {
-        if (!this.index.delete(id))
+        const doc = this.index.get(id);
+        if (!doc)
             throw new Error(`MiniSemanticSearch: cannot discard document with ID ${id}: it is not in the index`);
+        this.unplace(doc);
+        this.index.delete(id);
     }
     discardAll(ids) {
         for (const id of ids)
@@ -76,8 +142,10 @@ export class MiniSemanticSearch {
     }
     /** Removes the given documents, or every document if none are given. */
     removeAll(documents) {
-        if (documents === undefined)
+        if (documents === undefined) {
             this.index.clear();
+            this.plan = null;
+        }
         else
             for (const d of documents)
                 this.remove(d);
@@ -99,31 +167,104 @@ export class MiniSemanticSearch {
         const { fields: only, boost = {}, filter, minScore = -Infinity } = { ...this.options.searchOptions, ...searchOptions };
         const { fields } = this.options;
         const use = fields.map((f) => !only || only.includes(f));
-        const subset = use.some((u) => !u);
+        const weights = fields.map((f, i) => (use[i] ? boost[f] ?? 1 : 0));
+        const weighting = { key: JSON.stringify([use, weights]), use, weights, subset: use.some((u) => !u) };
         const q = this.model.embed(query);
         const results = [];
-        for (const [id, doc] of this.index) {
-            let sum = 0, weight = 0;
-            doc.fields.forEach((vs, i) => {
-                if (!use[i] || !vs.length)
-                    return;
-                const w = boost[fields[i]] ?? 1;
-                let best = -Infinity;
-                for (const v of vs)
-                    best = Math.max(best, similarity(q, v));
-                sum += w * best;
-                weight += w;
-            });
-            if (!weight)
-                continue;
-            // The whole-document vector covers every field, so it only counts when every field does.
-            const score = subset || !doc.whole ? sum / weight : 0.5 * similarity(q, doc.whole) + 0.5 * (sum / weight);
+        const emit = (doc, score) => {
             if (score < minScore)
-                continue;
-            const result = { id, score, ...doc.stored }; // stored fields come last, as in MiniSearch
+                return;
+            const result = { id: doc.id, score, ...doc.stored }; // stored fields come last, as in MiniSearch
             if (!filter || filter(result))
                 results.push(result);
+        };
+        const best = (items) => {
+            let b = -Infinity;
+            for (const v of items)
+                b = Math.max(b, dot8(q, v));
+            return b;
+        };
+        const plan = this.planFor(weighting);
+        if (plan) {
+            const { rows, docs, multi } = plan;
+            for (let r = 0, o = 0; r < docs.length; r++, o += q.length) {
+                let score = dot(q, rows, o);
+                const m = multi[r];
+                if (m)
+                    for (const [w, items] of m)
+                        score += w * best(items);
+                emit(docs[r], score);
+            }
+        }
+        else {
+            for (const doc of this.index.values()) {
+                const t = terms(doc, weighting);
+                if (!t)
+                    continue;
+                let score = 0;
+                for (const [w, items] of t)
+                    score += w * best(items);
+                emit(doc, score);
+            }
         }
         return results.sort((a, b) => b.score - a.score);
     }
+    /** The plan for this weighting: the current one, a new one if it was also the last search's, or else none. */
+    planFor(weighting) {
+        if (this.plan?.key === weighting.key)
+            return this.plan;
+        if (this.lastKey !== weighting.key) {
+            this.lastKey = weighting.key;
+            return null;
+        }
+        const plan = { ...weighting, rows: new Float32Array(Math.max(this.index.size, 16) * this.model.dims), docs: [], multi: [] };
+        this.plan = plan;
+        for (const doc of this.index.values())
+            this.place(plan, doc);
+        return plan;
+    }
+    /** Adds the document's row to the plan. */
+    place(plan, doc) {
+        doc.row = -1;
+        const t = terms(doc, plan);
+        if (!t)
+            return;
+        const dims = this.model.dims, n = plan.docs.length;
+        if ((n + 1) * dims > plan.rows.length) {
+            const grown = new Float32Array(plan.rows.length * 2);
+            grown.set(plan.rows);
+            plan.rows = grown;
+        }
+        const row = plan.rows.subarray(n * dims, (n + 1) * dims).fill(0);
+        let multi = null;
+        for (const [w, items] of t) {
+            if (items.length > 1) {
+                (multi ??= []).push([w, items]);
+                continue;
+            }
+            const { q, s } = items[0], k = s * w;
+            for (let i = 0; i < dims; i++)
+                row[i] += q[i] * k;
+        }
+        plan.docs.push(doc);
+        plan.multi.push(multi);
+        doc.row = n;
+    }
+    /** Takes the document's row out of the plan, moving the last row into its place. */
+    unplace(doc) {
+        const plan = this.plan;
+        if (!plan || doc.row < 0)
+            return;
+        const dims = this.model.dims, last = plan.docs.length - 1;
+        if (doc.row !== last) {
+            plan.rows.copyWithin(doc.row * dims, last * dims, (last + 1) * dims);
+            const moved = plan.docs[last];
+            plan.docs[doc.row] = moved;
+            plan.multi[doc.row] = plan.multi[last];
+            moved.row = doc.row;
+        }
+        plan.docs.pop();
+        plan.multi.pop();
+        doc.row = -1;
+    }
 }

package/dist/tokenizer.d.ts

@@ -2,5 +2,8 @@
 export declare function normalize(text: string): string;
 /** Splits normalised text into words and single punctuation characters. */
 export declare function preTokenize(text: string): string[];
-/** Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt. */
-export declare function wordPiece(vocab: Map<string, number>, unk: number, text: string): number[];
+/**
+ * Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt.
+ * `longest` (the vocabulary's longest token, in characters) only saves looking up pieces too long to be known.
+ */
+export declare function wordPiece(vocab: Map<string, number>, unk: number, text: string, longest?: number): number[];

package/dist/tokenizer.js

@@ -2,14 +2,18 @@
 // BertPreTokenizer, WordPiece with "##" continuations). Checked token for token against it in the tests.
 const isPunct = (c) => {
     const cp = c.codePointAt(0);
-    return (cp >= 33 && cp <= 47) || (cp >= 58 && cp <= 64) || (cp >= 91 && cp <= 96) || (cp >= 123 && cp <= 126)
-        || /\p{P}/u.test(c);
+    if (cp < 128)
+        return (cp >= 33 && cp <= 47) || (cp >= 58 && cp <= 64) || (cp >= 91 && cp <= 96) || (cp >= 123 && cp <= 126);
+    return /\p{P}/u.test(c);
 };
 const isCjk = (cp) => (cp >= 0x4e00 && cp <= 0x9fff) || (cp >= 0x3400 && cp <= 0x4dbf)
     || (cp >= 0x20000 && cp <= 0x2a6df) || (cp >= 0x2a700 && cp <= 0x2b73f) || (cp >= 0x2b740 && cp <= 0x2b81f)
     || (cp >= 0x2b820 && cp <= 0x2ceaf) || (cp >= 0xf900 && cp <= 0xfaff) || (cp >= 0x2f800 && cp <= 0x2fa1f);
 /** Drops control characters, spaces out CJK characters, strips accents and lower-cases. */
 export function normalize(text) {
+    // Printable ASCII, tabs and newlines: nothing to drop, space out or strip.
+    if (/^[\t\n\r\x20-\x7e]*$/.test(text))
+        return text.replace(/[\t\n\r]/g, ' ').toLowerCase();
     let out = '';
     for (const c of text) {
         const cp = c.codePointAt(0);
@@ -44,8 +48,11 @@ export function preTokenize(text) {
     }
     return words;
 }
-/** Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt. */
-export function wordPiece(vocab, unk, text) {
+/**
+ * Token ids for text: each word as the longest known pieces, left to right, or `unk` if it can't be spelt.
+ * `longest` (the vocabulary's longest token, in characters) only saves looking up pieces too long to be known.
+ */
+export function wordPiece(vocab, unk, text, longest = Infinity) {
     const ids = [];
     for (const w of preTokenize(normalize(text))) {
         const chars = Array.from(w);
@@ -53,12 +60,13 @@ export function wordPiece(vocab, unk, text) {
             ids.push(unk);
             continue;
         }
+        const piece = chars.length === w.length ? (s, e) => w.slice(s, e) : (s, e) => chars.slice(s, e).join('');
         const pieces = [];
         for (let start = 0; start < chars.length;) {
-            let end = chars.length;
+            let end = Math.min(chars.length, start + longest);
             let id;
             for (; end > start; end--) {
-                id = vocab.get((start ? '##' : '') + chars.slice(start, end).join(''));
+                id = vocab.get(start ? '##' + piece(start, end) : piece(start, end));
                 if (id !== undefined)
                     break;
             }
@@ -70,7 +78,8 @@ export function wordPiece(vocab, unk, text) {
             pieces.push(id);
             start = end;
         }
-        ids.push(...pieces);
+        for (const id of pieces)
+            ids.push(id);
     }
     return ids;
 }

package/models/README.md

@@ -2,12 +2,13 @@
 
 This file is not covered by the package's MIT licence for its code; it keeps its source's terms.
 
-## potion-base-8M-128d.bin
+## potion-base-8M.bin
 
 [minishlab/potion-base-8M](https://huggingface.co/minishlab/potion-base-8M) by MinishLab, a
 [Model2Vec](https://github.com/MinishLab/model2vec) model distilled from
 [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5), released under the MIT licence. Its
 vocabulary is bge-base-en-v1.5's (BERT's, uncased).
 
-Modified: repacked by `mini-semantic-search pack` to its first 128 of 256 dimensions at 4 bits per value (each
-token's 128 values sharing an f16 scale), 2.2 MB instead of 30.5 MB. All 29,528 tokens are kept.
+Modified: repacked by `mini-semantic-search pack` at 2 bits per value (each value one of four fitted levels, times
+an f32 scale for its dimension and an f16 scale for its token), 2.2 MB instead of 30.5 MB. All 29,528 tokens and all
+256 dimensions are kept.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mini-semantic-search",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Semantic search with a MiniSearch-style API, using Model2Vec static embeddings. Runs in browsers and Node; no dependencies; 2.2 MB with the model.",
   "keywords": [
     "semantic-search",