sqlite-sparse 1.0.0__tar.gz → 1.1.0__tar.gz

{sqlite_sparse-1.0.0/sqlite_sparse.egg-info → sqlite_sparse-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlite-sparse
-Version: 1.0.0
+Version: 1.1.0
 Summary: Semantic search in one SQLite file. No model, no server at query time.
 Author-email: Arbaz Siddiqui <arbaz00@gmail.com>
 License: MIT
@@ -55,27 +55,33 @@ but the keywords were chosen by a transformer. OpenSearch's inference-free varia
 the encoder only on documents; each query token gets one learned weight from a lookup
 table, and retrieval is an exact dot product.
 
-Dense retrieval (vector search) instead runs an embedding model on every query. A learned
-sparse index moves all model work to write time, and the representation is legible, since
-you can see which terms matched and with what weight. The cost is quality against good
-dense models of the same size and much slower indexing. `mini` (23M) averages 0.497
-nDCG@10 on BEIR and mdbr-leaf-ir (23M dense) reports 0.5355 in its symmetric
-configuration. The two do very different amounts of work at query time, so this is
-context, not a controlled comparison. For read-heavy workloads where indexing is
-amortized over many searches, the trade can be favourable.
-
-This library ships OpenSearch's inference-free sparse models, which until now have lived
-inside search clusters (OpenSearch, Elasticsearch, Vespa).
-[sqlite-vec](https://github.com/asg017/sqlite-vec) provides a way to use embeddings in
-SQLite; this does the same for learned sparse, whose posting lists are plain rows in the
-database file. What this project contributes is the embedded
-implementation, the file format with a reference implementation to test it against, and
-the measurements.
-
-The searchable index and the query weights live in the SQLite database. The encoder is
-needed only when inserting documents. So the `.db` can be copied, shipped inside an app,
-opened on any machine with no GPU, and queried with plain SQL from any language that
-has SQLite. Prebuilt binaries cover Linux x86-64 and macOS arm64.
+Dense retrieval (vector search) runs an embedding model on every query. A learned sparse
+index moves all model work to write time, and you can see which terms matched and with
+what weight. The cost is some quality against good dense models of the same size and much
+slower indexing; the numbers are under Benchmarks.
+
+SPLADE encoders are BERT models with their masked-language-model head still attached. BERT
+was trained to fill in blanks: shown `aspirin prevents [MASK]`, that head scores every
+word in the vocabulary as a candidate for the blank. SPLADE points the same head at every
+token of a document and keeps the best score each word gets, so a sentence about heart
+attacks earns a weight for `cardiac` even though the word is not in it. Those per-word
+scores are the sparse vector; the head is the entire trick. llama.cpp runs BERT-family
+models for embeddings only: its converter drops the head (the `cls.predictions` tensors,
+along with the pooler) and its graph stops at the per-token vectors, so `llama-embedding`
+on one of these models returns embeddings and no way to turn them back into words.
+
+sqlite-sparse keeps the head. The converter copies its weights out of the checkpoint into
+a small `.sprs` file next to the GGUF, along with the query weight table. At insert time
+llama.cpp runs the encoder as usual and the extension runs the head over the token vectors
+itself, in C on ggml: a dense layer, GELU, LayerNorm, then a score for every word in the
+vocabulary, keeping the highest score each word received across the tokens and applying
+log(1 + ReLU) so the weights are positive and compressed. That turns an encoder llama.cpp
+can already run into a sparse retriever. The rest is what a search cluster provides and
+SQLite does not: the virtual table, posting lists stored as rows, the query-time
+scatter-add, and the file format with a reference implementation to test it against.
+[sqlite-vec](https://github.com/asg017/sqlite-vec) did this for embeddings in SQLite; this
+does it for learned sparse, which so far has lived inside OpenSearch, Elasticsearch and
+Vespa.
 
 ## Install
 
@@ -87,15 +93,14 @@ Or take the binary from the [releases page](https://github.com/arbazsiddiqui/sql
 and use it from any language.
 
 ```
-tar xzf sparse0-1.0.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
+tar xzf sparse0-1.1.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
 sqlite3 notes.db
 sqlite> .load ./sparse0
 ```
 
 Keep the filename `sparse0.so` / `sparse0.dylib`, since SQLite derives the entry point
-from it. On macOS, the python.org installer's `sqlite3` module cannot load extensions.
-Use Python from Homebrew or conda, or `pip install sqlean.py` and `import sqlean as
-sqlite3`.
+from it. On macOS the python.org `sqlite3` module cannot load extensions; use Homebrew or
+conda Python, or `pip install sqlean.py` and `import sqlean as sqlite3`.
 
 ## Quickstart
 
@@ -106,87 +111,63 @@ db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)                       # loads the sparse0 extension
 sqlite_sparse.register(db, "mini")           # downloads the model on first use
 db.execute("CREATE VIRTUAL TABLE notes USING sparse0(model='mini')")
-db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")
+db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")  # the model runs here
 db.commit()
-db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",
+db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",              # and never here
            ("what prevents cardiac arrest",)).fetchall()
 ```
 
-The model runs at INSERT only; MATCH never loads it. Searching an existing index needs
-no model at all, on any machine.
-
 ```python
+# Another machine, no model downloaded: MATCH only reads the file.
 db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)
-db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the file
+db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the index in the file
 db.execute("SELECT rowid, score FROM temp.notes WHERE notes MATCH 'heart medication' LIMIT 5")
 ```
 
-A database file holds one sparse index; another `sparse0` table in the same file
-attaches to the same index rather than creating a second one.
-
-Because results are rows, semantic search composes with plain SQL. Ask for extra
-candidates with `k`, then filter and join like any other table.
-
 ```sql
+-- Results are rows: ask for k candidates, then filter and join like any other table.
 SELECT n.rowid, n.score, d.title
 FROM notes n JOIN documents d ON d.id = n.rowid
 WHERE n.text MATCH 'heart medication' AND k = 50 AND d.folder = 'work'
 ORDER BY n.score DESC LIMIT 10;
 ```
 
-Indexing is the expensive half, so large corpora are best built once on a GPU machine
-with the Python package (`sqlite-sparse build`, which writes the same file format) and
-the `.db` then shipped to wherever the reads happen.
-
-`LIMIT n` and `AND k = n` both work, and `ORDER BY score DESC` is honoured without a sort
-step. `DELETE FROM notes WHERE rowid = ?` marks a document deleted; run
-`SELECT sparse_compact()` now and then on an index with heavy churn to reclaim its
-postings. Documents longer than `max_seq` tokens (default 512) are truncated at insert, and
-each row in the `docs` table records `ntokens` and `truncated`. The full surface, including
-the Python helpers, is in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
+Indexing is the expensive half. Build a large corpus once on a GPU machine with
+`sqlite-sparse build` and ship the `.db` to wherever the reads happen. Deletes,
+compaction, truncation and the rest of the surface are in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
 
 ## How it works
 
 ![How sqlite-sparse indexes and searches](https://raw.githubusercontent.com/arbazsiddiqui/sqlite-sparse/master/docs/how-it-works.svg)
 
-**INSERT.** The extension tokenizes the text with the registered model's tokenizer and
-runs the encoder from the GGUF file through llama.cpp, one vector per token. It then
-applies the scoring head from the `.sprs` sidecar, which scores every word in the model's
-vocabulary against those vectors; positive scores are kept, compressed with a logarithm,
-and become the document's weighted terms. For the aspirin sentence that is 157 terms
-(`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42, `prevents` 0.18,
-and so on). Each term is appended to that word's posting list, a row in the file listing
-the documents it scored and the weight, stored as one byte. The document's token count
-and whether it was truncated go into the `docs` table.
-
-**MATCH.** The extension tokenizes the query the same way and reads one number per token
-from the query weight table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac`
-6.53, `arrest` 6.87). For each query word it walks that word's posting list and adds
-query weight × stored weight into the running total of every document listed; that
-accumulation is the scatter-add. `prevents` contributes 6.72 × 0.18 and `cardiac`
-6.53 × 0.42 to document 1, total 3.95. The documents touched are sorted and the top k
-returned. Scoring is exact over the stored weights, with no candidate stage and no
-approximate index, and ties break on the lower rowid. Nothing from the GGUF or the
-sidecar is read at query time.
+**INSERT.** The text is tokenized, the encoder runs through llama.cpp, and the head scores
+every vocabulary word against the token vectors. For the aspirin sentence that leaves 157
+weighted terms (`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42,
+`prevents` 0.18, and so on). Each term is appended to that word's posting list, a row in
+the file listing the documents it scored and the weight as one byte.
+
+**MATCH.** The query is tokenized the same way and each token gets its weight from the
+table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac` 6.53, `arrest` 6.87).
+For each query word the extension walks that word's posting list and adds query weight ×
+stored weight into every listed document's total, the scatter-add: `prevents` contributes
+6.72 × 0.18 and `cardiac` 6.53 × 0.42 to document 1, total 3.95. Scoring is exact over the
+stored weights, with no candidate stage or approximate index, and nothing from the GGUF or
+the sidecar is read.
 
 The file layout is in [FORMAT.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/FORMAT.md). The format and the sidecar header carry
 version 1, and files written by any 1.x release stay readable by later 1.x releases.
 
 ## Benchmarks
 
-Three ways to search inside a SQLite file, each in its shipped form, on the same machine.
-FTS5 is SQLite's built-in keyword search ranked by BM25. Dense brute-force is
-[sqlite-vec](https://github.com/asg017/sqlite-vec) int8, which scans every vector, with
-[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on torch
-CPU with 8 threads. Sparse is the `sparse0` extension with `mini` (23M, Q8_0 encoder, u8
-postings).
-
-Every number is end-to-end query latency, which for dense includes encoding the query,
-because that is its real query path. This compares the brute-force vector path inside
-SQLite, not an approximate nearest-neighbour index. The FTS5 query is the disjunction of
-the query's tokens ranked by `bm25()`; a conjunction is faster but misses documents that
-match only some of the terms.
+Three ways to search inside a SQLite file, each in its shipped form, on the same machine:
+FTS5, SQLite's built-in keyword search ranked by BM25; dense brute-force with
+[sqlite-vec](https://github.com/asg017/sqlite-vec) int8 and
+[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on
+torch CPU; and `sparse0` with `mini` (23M, Q8_0 encoder, u8 postings). Latency is end to
+end, so dense includes encoding the query, because that is its real query path. FTS5 runs
+the OR of the query's tokens ranked by `bm25()` (an AND is faster but misses partial
+matches), and the dense lane is the brute-force scan, not an approximate index.
 
 | msmarco, 1M documents | FTS5 BM25 | dense brute-force | sqlite-sparse |
 |---|---|---|---|
@@ -199,7 +180,9 @@ match only some of the terms.
 | model at query time | none | 23M transformer | none |
 | retrieval | lexical | semantic | semantic |
 
-At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively.
+At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively. Measured on a GCE
+`c3-standard-8` (8 vCPU, 4 physical cores); the scripts and raw results are attached to
+each release.
 
 ### The extension does not lose the model's quality
 
@@ -218,14 +201,9 @@ the same documents and queries. The gain ranges from small (SciFact) to large (F
 
 The same model run in torch at fp32 agrees with the extension on 96 to 98 percent of
 top-10 results on every dataset, and storing weights as one byte instead of fp32 changed
-nDCG@10 by less than 0.001.
-
-Venue was a dedicated GCE `c3-standard-8` (8 vCPU, 4 physical cores) running Debian 12,
-with each lane in its own fresh process. 4,000 samples × 5 repetitions per lane (900 × 3
-for dense and 1,000 × 3 for FTS5 at 1M), median of repetition medians. Cold start is the
-second of three fresh-process runs. RAM is peak RSS after 50 warm queries. The corpus is
-the first 100K and 1M passages of MS MARCO with its dev queries. Benchmark scripts and raw
-results are attached to each release.
+nDCG@10 by less than 0.001. For context against dense models of the same size,
+mdbr-leaf-ir (23M) reports 0.5355 BEIR average to `mini`'s 0.497; the two do very
+different amounts of work at query time, so that is context, not a controlled comparison.
 
 ## Models
 
@@ -235,17 +213,16 @@ results are attached to each release.
 | `base` | [doc-v3-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) | 67M | 0.517 | [arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF) |
 | `multilingual` | [multilingual-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1) | 168M | multilingual | [arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF) |
 
-All three are in the [sqlite-sparse models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
-collection, and `sqlite_sparse.register(db, alias)` fetches one into `~/.cache/sqlite-sparse`.
-Each conversion is validated against the original SentenceTransformers implementation
-(encoder hidden states at cosine 0.9997 or better, term weights within 1.3e-3). Weights
-are unmodified from the Apache-2.0 originals by the OpenSearch project.
+All three are in the [sqlite-sparse
+models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
+collection, and `sqlite_sparse.register(db, alias)` fetches one into
+`~/.cache/sqlite-sparse`. Weights are unmodified from the Apache-2.0 originals by the
+OpenSearch project.
 
 ### Bring your own model
 
-Any inference-free OpenSearch-style sparse encoder on Hugging Face works. Two files are
-needed, the encoder as GGUF and a sidecar with the scoring head, query weight table and
-vocabulary.
+Any inference-free OpenSearch-style sparse encoder on Hugging Face works: the encoder as
+GGUF plus a `.sprs` sidecar holding the head and the query weight table.
 
 ```
 git clone --depth 1 https://github.com/ggml-org/llama.cpp
@@ -260,9 +237,29 @@ SELECT sparse_register('mine', 'model_q8.gguf', 'model.sprs');
 CREATE VIRTUAL TABLE notes USING sparse0(model='mine');
 ```
 
-The converter requires the checkpoint to be a BERT-family encoder with a masked-LM head
-and a static query weight table. llama.cpp must support the encoder architecture; it does
-not support GTE (`doc-v3-gte`).
+The checkpoint must be a BERT-family encoder with a masked-LM head and a static query
+weight table, and llama.cpp must support the architecture (it does not support GTE,
+`doc-v3-gte`).
+
+### Bring your own vectors
+
+Any sparse model works if you run it yourself, including SPLADE models that encode the
+query too, and models llama.cpp cannot run. Create the index from the model's vocabulary
+and hand it `{"token": weight}` objects for documents and for queries. Nothing is
+converted; the extension stores and scores, and the file is the same format.
+
+```sql
+CREATE VIRTUAL TABLE notes USING sparse0(vocab='vocab.txt');    -- one token per line, no model
+INSERT INTO notes(rowid, terms) VALUES (1, '{"heart": 0.95, "cardiac": 0.42, "stroke": 0.92}');
+SELECT rowid, score FROM notes WHERE notes.terms MATCH '{"cardiac": 6.53, "arrest": 6.87}' LIMIT 5;
+```
+
+Tokens must be in the vocabulary (an unknown token is an error on insert and ignored in a
+query), weights must be positive, and weights above 6.375 saturate the one-byte storage.
+`terms MATCH` also works on an index one of the shipped models built, so a query encoded
+by your own model can search it. Text queries on a vocabulary-only index are an error,
+since there is no query weight table. In Python: `SparseIndex.create_external(path,
+vocab)`, `add_terms(id, terms)`, `search_terms(terms)`.
 
 ## Development
 
@@ -275,7 +272,10 @@ make test     # installs the Python binding and runs the suite
 `src/` is the extension (`sparse0.c` virtual table, `wordpiece.c` tokenizer on utf8proc, `head.c`
 scoring head on ggml, `scorer.c` scatter-add, `encoder.c` llama.cpp wrapper).
 `bindings/python` is the reference implementation of the file format and the test oracle.
-Component agreement is logged in [`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
+Each shipped conversion is validated against the original SentenceTransformers
+implementation (encoder hidden states at cosine 0.9997 or better, term weights within
+1.3e-3), and component agreement is logged in
+[`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
 
 ## License
 

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0}/README.md

@@ -27,27 +27,33 @@ but the keywords were chosen by a transformer. OpenSearch's inference-free varia
 the encoder only on documents; each query token gets one learned weight from a lookup
 table, and retrieval is an exact dot product.
 
-Dense retrieval (vector search) instead runs an embedding model on every query. A learned
-sparse index moves all model work to write time, and the representation is legible, since
-you can see which terms matched and with what weight. The cost is quality against good
-dense models of the same size and much slower indexing. `mini` (23M) averages 0.497
-nDCG@10 on BEIR and mdbr-leaf-ir (23M dense) reports 0.5355 in its symmetric
-configuration. The two do very different amounts of work at query time, so this is
-context, not a controlled comparison. For read-heavy workloads where indexing is
-amortized over many searches, the trade can be favourable.
-
-This library ships OpenSearch's inference-free sparse models, which until now have lived
-inside search clusters (OpenSearch, Elasticsearch, Vespa).
-[sqlite-vec](https://github.com/asg017/sqlite-vec) provides a way to use embeddings in
-SQLite; this does the same for learned sparse, whose posting lists are plain rows in the
-database file. What this project contributes is the embedded
-implementation, the file format with a reference implementation to test it against, and
-the measurements.
-
-The searchable index and the query weights live in the SQLite database. The encoder is
-needed only when inserting documents. So the `.db` can be copied, shipped inside an app,
-opened on any machine with no GPU, and queried with plain SQL from any language that
-has SQLite. Prebuilt binaries cover Linux x86-64 and macOS arm64.
+Dense retrieval (vector search) runs an embedding model on every query. A learned sparse
+index moves all model work to write time, and you can see which terms matched and with
+what weight. The cost is some quality against good dense models of the same size and much
+slower indexing; the numbers are under Benchmarks.
+
+SPLADE encoders are BERT models with their masked-language-model head still attached. BERT
+was trained to fill in blanks: shown `aspirin prevents [MASK]`, that head scores every
+word in the vocabulary as a candidate for the blank. SPLADE points the same head at every
+token of a document and keeps the best score each word gets, so a sentence about heart
+attacks earns a weight for `cardiac` even though the word is not in it. Those per-word
+scores are the sparse vector; the head is the entire trick. llama.cpp runs BERT-family
+models for embeddings only: its converter drops the head (the `cls.predictions` tensors,
+along with the pooler) and its graph stops at the per-token vectors, so `llama-embedding`
+on one of these models returns embeddings and no way to turn them back into words.
+
+sqlite-sparse keeps the head. The converter copies its weights out of the checkpoint into
+a small `.sprs` file next to the GGUF, along with the query weight table. At insert time
+llama.cpp runs the encoder as usual and the extension runs the head over the token vectors
+itself, in C on ggml: a dense layer, GELU, LayerNorm, then a score for every word in the
+vocabulary, keeping the highest score each word received across the tokens and applying
+log(1 + ReLU) so the weights are positive and compressed. That turns an encoder llama.cpp
+can already run into a sparse retriever. The rest is what a search cluster provides and
+SQLite does not: the virtual table, posting lists stored as rows, the query-time
+scatter-add, and the file format with a reference implementation to test it against.
+[sqlite-vec](https://github.com/asg017/sqlite-vec) did this for embeddings in SQLite; this
+does it for learned sparse, which so far has lived inside OpenSearch, Elasticsearch and
+Vespa.
 
 ## Install
 
@@ -59,15 +65,14 @@ Or take the binary from the [releases page](https://github.com/arbazsiddiqui/sql
 and use it from any language.
 
 ```
-tar xzf sparse0-1.0.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
+tar xzf sparse0-1.1.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
 sqlite3 notes.db
 sqlite> .load ./sparse0
 ```
 
 Keep the filename `sparse0.so` / `sparse0.dylib`, since SQLite derives the entry point
-from it. On macOS, the python.org installer's `sqlite3` module cannot load extensions.
-Use Python from Homebrew or conda, or `pip install sqlean.py` and `import sqlean as
-sqlite3`.
+from it. On macOS the python.org `sqlite3` module cannot load extensions; use Homebrew or
+conda Python, or `pip install sqlean.py` and `import sqlean as sqlite3`.
 
 ## Quickstart
 
@@ -78,87 +83,63 @@ db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)                       # loads the sparse0 extension
 sqlite_sparse.register(db, "mini")           # downloads the model on first use
 db.execute("CREATE VIRTUAL TABLE notes USING sparse0(model='mini')")
-db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")
+db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")  # the model runs here
 db.commit()
-db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",
+db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",              # and never here
            ("what prevents cardiac arrest",)).fetchall()
 ```
 
-The model runs at INSERT only; MATCH never loads it. Searching an existing index needs
-no model at all, on any machine.
-
 ```python
+# Another machine, no model downloaded: MATCH only reads the file.
 db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)
-db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the file
+db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the index in the file
 db.execute("SELECT rowid, score FROM temp.notes WHERE notes MATCH 'heart medication' LIMIT 5")
 ```
 
-A database file holds one sparse index; another `sparse0` table in the same file
-attaches to the same index rather than creating a second one.
-
-Because results are rows, semantic search composes with plain SQL. Ask for extra
-candidates with `k`, then filter and join like any other table.
-
 ```sql
+-- Results are rows: ask for k candidates, then filter and join like any other table.
 SELECT n.rowid, n.score, d.title
 FROM notes n JOIN documents d ON d.id = n.rowid
 WHERE n.text MATCH 'heart medication' AND k = 50 AND d.folder = 'work'
 ORDER BY n.score DESC LIMIT 10;
 ```
 
-Indexing is the expensive half, so large corpora are best built once on a GPU machine
-with the Python package (`sqlite-sparse build`, which writes the same file format) and
-the `.db` then shipped to wherever the reads happen.
-
-`LIMIT n` and `AND k = n` both work, and `ORDER BY score DESC` is honoured without a sort
-step. `DELETE FROM notes WHERE rowid = ?` marks a document deleted; run
-`SELECT sparse_compact()` now and then on an index with heavy churn to reclaim its
-postings. Documents longer than `max_seq` tokens (default 512) are truncated at insert, and
-each row in the `docs` table records `ntokens` and `truncated`. The full surface, including
-the Python helpers, is in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
+Indexing is the expensive half. Build a large corpus once on a GPU machine with
+`sqlite-sparse build` and ship the `.db` to wherever the reads happen. Deletes,
+compaction, truncation and the rest of the surface are in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
 
 ## How it works
 
 ![How sqlite-sparse indexes and searches](https://raw.githubusercontent.com/arbazsiddiqui/sqlite-sparse/master/docs/how-it-works.svg)
 
-**INSERT.** The extension tokenizes the text with the registered model's tokenizer and
-runs the encoder from the GGUF file through llama.cpp, one vector per token. It then
-applies the scoring head from the `.sprs` sidecar, which scores every word in the model's
-vocabulary against those vectors; positive scores are kept, compressed with a logarithm,
-and become the document's weighted terms. For the aspirin sentence that is 157 terms
-(`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42, `prevents` 0.18,
-and so on). Each term is appended to that word's posting list, a row in the file listing
-the documents it scored and the weight, stored as one byte. The document's token count
-and whether it was truncated go into the `docs` table.
-
-**MATCH.** The extension tokenizes the query the same way and reads one number per token
-from the query weight table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac`
-6.53, `arrest` 6.87). For each query word it walks that word's posting list and adds
-query weight × stored weight into the running total of every document listed; that
-accumulation is the scatter-add. `prevents` contributes 6.72 × 0.18 and `cardiac`
-6.53 × 0.42 to document 1, total 3.95. The documents touched are sorted and the top k
-returned. Scoring is exact over the stored weights, with no candidate stage and no
-approximate index, and ties break on the lower rowid. Nothing from the GGUF or the
-sidecar is read at query time.
+**INSERT.** The text is tokenized, the encoder runs through llama.cpp, and the head scores
+every vocabulary word against the token vectors. For the aspirin sentence that leaves 157
+weighted terms (`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42,
+`prevents` 0.18, and so on). Each term is appended to that word's posting list, a row in
+the file listing the documents it scored and the weight as one byte.
+
+**MATCH.** The query is tokenized the same way and each token gets its weight from the
+table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac` 6.53, `arrest` 6.87).
+For each query word the extension walks that word's posting list and adds query weight ×
+stored weight into every listed document's total, the scatter-add: `prevents` contributes
+6.72 × 0.18 and `cardiac` 6.53 × 0.42 to document 1, total 3.95. Scoring is exact over the
+stored weights, with no candidate stage or approximate index, and nothing from the GGUF or
+the sidecar is read.
 
 The file layout is in [FORMAT.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/FORMAT.md). The format and the sidecar header carry
 version 1, and files written by any 1.x release stay readable by later 1.x releases.
 
 ## Benchmarks
 
-Three ways to search inside a SQLite file, each in its shipped form, on the same machine.
-FTS5 is SQLite's built-in keyword search ranked by BM25. Dense brute-force is
-[sqlite-vec](https://github.com/asg017/sqlite-vec) int8, which scans every vector, with
-[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on torch
-CPU with 8 threads. Sparse is the `sparse0` extension with `mini` (23M, Q8_0 encoder, u8
-postings).
-
-Every number is end-to-end query latency, which for dense includes encoding the query,
-because that is its real query path. This compares the brute-force vector path inside
-SQLite, not an approximate nearest-neighbour index. The FTS5 query is the disjunction of
-the query's tokens ranked by `bm25()`; a conjunction is faster but misses documents that
-match only some of the terms.
+Three ways to search inside a SQLite file, each in its shipped form, on the same machine:
+FTS5, SQLite's built-in keyword search ranked by BM25; dense brute-force with
+[sqlite-vec](https://github.com/asg017/sqlite-vec) int8 and
+[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on
+torch CPU; and `sparse0` with `mini` (23M, Q8_0 encoder, u8 postings). Latency is end to
+end, so dense includes encoding the query, because that is its real query path. FTS5 runs
+the OR of the query's tokens ranked by `bm25()` (an AND is faster but misses partial
+matches), and the dense lane is the brute-force scan, not an approximate index.
 
 | msmarco, 1M documents | FTS5 BM25 | dense brute-force | sqlite-sparse |
 |---|---|---|---|
@@ -171,7 +152,9 @@ match only some of the terms.
 | model at query time | none | 23M transformer | none |
 | retrieval | lexical | semantic | semantic |
 
-At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively.
+At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively. Measured on a GCE
+`c3-standard-8` (8 vCPU, 4 physical cores); the scripts and raw results are attached to
+each release.
 
 ### The extension does not lose the model's quality
 
@@ -190,14 +173,9 @@ the same documents and queries. The gain ranges from small (SciFact) to large (F
 
 The same model run in torch at fp32 agrees with the extension on 96 to 98 percent of
 top-10 results on every dataset, and storing weights as one byte instead of fp32 changed
-nDCG@10 by less than 0.001.
-
-Venue was a dedicated GCE `c3-standard-8` (8 vCPU, 4 physical cores) running Debian 12,
-with each lane in its own fresh process. 4,000 samples × 5 repetitions per lane (900 × 3
-for dense and 1,000 × 3 for FTS5 at 1M), median of repetition medians. Cold start is the
-second of three fresh-process runs. RAM is peak RSS after 50 warm queries. The corpus is
-the first 100K and 1M passages of MS MARCO with its dev queries. Benchmark scripts and raw
-results are attached to each release.
+nDCG@10 by less than 0.001. For context against dense models of the same size,
+mdbr-leaf-ir (23M) reports 0.5355 BEIR average to `mini`'s 0.497; the two do very
+different amounts of work at query time, so that is context, not a controlled comparison.
 
 ## Models
 
@@ -207,17 +185,16 @@ results are attached to each release.
 | `base` | [doc-v3-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) | 67M | 0.517 | [arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF) |
 | `multilingual` | [multilingual-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1) | 168M | multilingual | [arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF) |
 
-All three are in the [sqlite-sparse models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
-collection, and `sqlite_sparse.register(db, alias)` fetches one into `~/.cache/sqlite-sparse`.
-Each conversion is validated against the original SentenceTransformers implementation
-(encoder hidden states at cosine 0.9997 or better, term weights within 1.3e-3). Weights
-are unmodified from the Apache-2.0 originals by the OpenSearch project.
+All three are in the [sqlite-sparse
+models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
+collection, and `sqlite_sparse.register(db, alias)` fetches one into
+`~/.cache/sqlite-sparse`. Weights are unmodified from the Apache-2.0 originals by the
+OpenSearch project.
 
 ### Bring your own model
 
-Any inference-free OpenSearch-style sparse encoder on Hugging Face works. Two files are
-needed, the encoder as GGUF and a sidecar with the scoring head, query weight table and
-vocabulary.
+Any inference-free OpenSearch-style sparse encoder on Hugging Face works: the encoder as
+GGUF plus a `.sprs` sidecar holding the head and the query weight table.
 
 ```
 git clone --depth 1 https://github.com/ggml-org/llama.cpp
@@ -232,9 +209,29 @@ SELECT sparse_register('mine', 'model_q8.gguf', 'model.sprs');
 CREATE VIRTUAL TABLE notes USING sparse0(model='mine');
 ```
 
-The converter requires the checkpoint to be a BERT-family encoder with a masked-LM head
-and a static query weight table. llama.cpp must support the encoder architecture; it does
-not support GTE (`doc-v3-gte`).
+The checkpoint must be a BERT-family encoder with a masked-LM head and a static query
+weight table, and llama.cpp must support the architecture (it does not support GTE,
+`doc-v3-gte`).
+
+### Bring your own vectors
+
+Any sparse model works if you run it yourself, including SPLADE models that encode the
+query too, and models llama.cpp cannot run. Create the index from the model's vocabulary
+and hand it `{"token": weight}` objects for documents and for queries. Nothing is
+converted; the extension stores and scores, and the file is the same format.
+
+```sql
+CREATE VIRTUAL TABLE notes USING sparse0(vocab='vocab.txt');    -- one token per line, no model
+INSERT INTO notes(rowid, terms) VALUES (1, '{"heart": 0.95, "cardiac": 0.42, "stroke": 0.92}');
+SELECT rowid, score FROM notes WHERE notes.terms MATCH '{"cardiac": 6.53, "arrest": 6.87}' LIMIT 5;
+```
+
+Tokens must be in the vocabulary (an unknown token is an error on insert and ignored in a
+query), weights must be positive, and weights above 6.375 saturate the one-byte storage.
+`terms MATCH` also works on an index one of the shipped models built, so a query encoded
+by your own model can search it. Text queries on a vocabulary-only index are an error,
+since there is no query weight table. In Python: `SparseIndex.create_external(path,
+vocab)`, `add_terms(id, terms)`, `search_terms(terms)`.
 
 ## Development
 
@@ -247,7 +244,10 @@ make test     # installs the Python binding and runs the suite
 `src/` is the extension (`sparse0.c` virtual table, `wordpiece.c` tokenizer on utf8proc, `head.c`
 scoring head on ggml, `scorer.c` scatter-add, `encoder.c` llama.cpp wrapper).
 `bindings/python` is the reference implementation of the file format and the test oracle.
-Component agreement is logged in [`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
+Each shipped conversion is validated against the original SentenceTransformers
+implementation (encoder hidden states at cosine 0.9997 or better, term weights within
+1.3e-3), and component agreement is logged in
+[`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
 
 ## License
 

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sqlite-sparse"
-version = "1.0.0"
+version = "1.1.0"
 description = "Semantic search in one SQLite file. No model, no server at query time."
 readme = "README.md"
 requires-python = ">=3.9"

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0}/sqlite_sparse/__init__.py

@@ -3,4 +3,4 @@ from .loadable import load, loadable_path
 from .models import MODELS, fetch, register
 
 __all__ = ["SparseIndex", "load", "loadable_path", "MODELS", "fetch", "register"]
-__version__ = "1.0.0"
+__version__ = "1.1.0"

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0}/sqlite_sparse/api.py

@@ -23,6 +23,20 @@ class SparseIndex:
     def create(cls, path, model="mini", **kw):
         return cls(path, model=model, **kw)
 
+    @classmethod
+    def create_external(cls, path, vocab):
+        """An index that takes term vectors from the caller: no model and no query
+        weight table. vocab is the list of token strings, index = term id."""
+        ix = cls.__new__(cls)
+        ix.store = SparseStore(path)
+        ix._enc, ix._model, ix._max_seq, ix._device = None, "external", 256, None
+        if not ix.store.get_meta("format"):
+            vocab = list(vocab)
+            assert len(set(vocab)) == len(vocab), "vocabulary has duplicate tokens"
+            ix.store.init_model("external", vocab, np.zeros(len(vocab)))
+        ix.engine = QueryEngine(ix.store.db)
+        return ix
+
     def encoder(self):
         if self._enc is None:
             from .encoder import TorchEncoder
@@ -51,6 +65,23 @@ class SparseIndex:
         self.engine.reload()
         return total
 
+    def add_terms(self, id, terms, title=""):
+        """Store a document encoded by the caller, {token: weight}. Every token must
+        be in the index vocabulary; non-positive weights are dropped."""
+        v2i = self.engine._v2i
+        unknown = [t for t in terms if t not in v2i]
+        if unknown:
+            raise ValueError(f"terms not in the index vocabulary: {unknown[:5]}")
+        enc = {}
+        for tok, w in terms.items():
+            if w > 0:
+                enc[v2i[tok]] = enc.get(v2i[tok], 0.0) + float(w)
+        self.store.add_encoded([(id, title, None, enc)])
+        self.engine.reload()
+
+    def search_terms(self, terms, k=10):
+        return self.engine.search_terms(terms, k=k)
+
     def delete(self, id):
         self.store.delete(id)
 

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0}/sqlite_sparse/search.py

@@ -106,8 +106,27 @@ class QueryEngine:
                 qw[t] = qw.get(t, 0.0) + w
         return qw
 
+    def encode_terms(self, terms):
+        """{token: weight} -> {term id: weight}. Unknown tokens and non-positive
+        weights are dropped, repeated tokens summed."""
+        qw = {}
+        for tok, w in terms.items():
+            t = self._v2i.get(tok)
+            if t is None or not w > 0:
+                continue
+            qw[t] = qw.get(t, 0.0) + float(w)
+        return qw
+
     def search(self, text, k=10):
-        qw = self.encode_query(text)
+        if not self._qlut:
+            raise ValueError("this index has no query weight table (built from a vocabulary alone); "
+                             "use search_terms with a {token: weight} query")
+        return self._search_ids(self.encode_query(text), k)
+
+    def search_terms(self, terms, k=10):
+        return self._search_ids(self.encode_terms(terms), k)
+
+    def _search_ids(self, qw, k):
         if not qw or not self.ndocs:
             return []
         score = np.zeros(self.ndocs + 1, dtype=np.float64)

{sqlite_sparse-1.0.0 → sqlite_sparse-1.1.0/sqlite_sparse.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlite-sparse
-Version: 1.0.0
+Version: 1.1.0
 Summary: Semantic search in one SQLite file. No model, no server at query time.
 Author-email: Arbaz Siddiqui <arbaz00@gmail.com>
 License: MIT
@@ -55,27 +55,33 @@ but the keywords were chosen by a transformer. OpenSearch's inference-free varia
 the encoder only on documents; each query token gets one learned weight from a lookup
 table, and retrieval is an exact dot product.
 
-Dense retrieval (vector search) instead runs an embedding model on every query. A learned
-sparse index moves all model work to write time, and the representation is legible, since
-you can see which terms matched and with what weight. The cost is quality against good
-dense models of the same size and much slower indexing. `mini` (23M) averages 0.497
-nDCG@10 on BEIR and mdbr-leaf-ir (23M dense) reports 0.5355 in its symmetric
-configuration. The two do very different amounts of work at query time, so this is
-context, not a controlled comparison. For read-heavy workloads where indexing is
-amortized over many searches, the trade can be favourable.
-
-This library ships OpenSearch's inference-free sparse models, which until now have lived
-inside search clusters (OpenSearch, Elasticsearch, Vespa).
-[sqlite-vec](https://github.com/asg017/sqlite-vec) provides a way to use embeddings in
-SQLite; this does the same for learned sparse, whose posting lists are plain rows in the
-database file. What this project contributes is the embedded
-implementation, the file format with a reference implementation to test it against, and
-the measurements.
-
-The searchable index and the query weights live in the SQLite database. The encoder is
-needed only when inserting documents. So the `.db` can be copied, shipped inside an app,
-opened on any machine with no GPU, and queried with plain SQL from any language that
-has SQLite. Prebuilt binaries cover Linux x86-64 and macOS arm64.
+Dense retrieval (vector search) runs an embedding model on every query. A learned sparse
+index moves all model work to write time, and you can see which terms matched and with
+what weight. The cost is some quality against good dense models of the same size and much
+slower indexing; the numbers are under Benchmarks.
+
+SPLADE encoders are BERT models with their masked-language-model head still attached. BERT
+was trained to fill in blanks: shown `aspirin prevents [MASK]`, that head scores every
+word in the vocabulary as a candidate for the blank. SPLADE points the same head at every
+token of a document and keeps the best score each word gets, so a sentence about heart
+attacks earns a weight for `cardiac` even though the word is not in it. Those per-word
+scores are the sparse vector; the head is the entire trick. llama.cpp runs BERT-family
+models for embeddings only: its converter drops the head (the `cls.predictions` tensors,
+along with the pooler) and its graph stops at the per-token vectors, so `llama-embedding`
+on one of these models returns embeddings and no way to turn them back into words.
+
+sqlite-sparse keeps the head. The converter copies its weights out of the checkpoint into
+a small `.sprs` file next to the GGUF, along with the query weight table. At insert time
+llama.cpp runs the encoder as usual and the extension runs the head over the token vectors
+itself, in C on ggml: a dense layer, GELU, LayerNorm, then a score for every word in the
+vocabulary, keeping the highest score each word received across the tokens and applying
+log(1 + ReLU) so the weights are positive and compressed. That turns an encoder llama.cpp
+can already run into a sparse retriever. The rest is what a search cluster provides and
+SQLite does not: the virtual table, posting lists stored as rows, the query-time
+scatter-add, and the file format with a reference implementation to test it against.
+[sqlite-vec](https://github.com/asg017/sqlite-vec) did this for embeddings in SQLite; this
+does it for learned sparse, which so far has lived inside OpenSearch, Elasticsearch and
+Vespa.
 
 ## Install
 
@@ -87,15 +93,14 @@ Or take the binary from the [releases page](https://github.com/arbazsiddiqui/sql
 and use it from any language.
 
 ```
-tar xzf sparse0-1.0.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
+tar xzf sparse0-1.1.0-loadable-linux-x86_64.tar.gz    # or -macos-arm64
 sqlite3 notes.db
 sqlite> .load ./sparse0
 ```
 
 Keep the filename `sparse0.so` / `sparse0.dylib`, since SQLite derives the entry point
-from it. On macOS, the python.org installer's `sqlite3` module cannot load extensions.
-Use Python from Homebrew or conda, or `pip install sqlean.py` and `import sqlean as
-sqlite3`.
+from it. On macOS the python.org `sqlite3` module cannot load extensions; use Homebrew or
+conda Python, or `pip install sqlean.py` and `import sqlean as sqlite3`.
 
 ## Quickstart
 
@@ -106,87 +111,63 @@ db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)                       # loads the sparse0 extension
 sqlite_sparse.register(db, "mini")           # downloads the model on first use
 db.execute("CREATE VIRTUAL TABLE notes USING sparse0(model='mini')")
-db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")
+db.execute("INSERT INTO notes(rowid, text) VALUES (1, 'Aspirin lowers heart attack risk')")  # the model runs here
 db.commit()
-db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",
+db.execute("SELECT rowid, score FROM notes WHERE notes MATCH ? LIMIT 5",              # and never here
            ("what prevents cardiac arrest",)).fetchall()
 ```
 
-The model runs at INSERT only; MATCH never loads it. Searching an existing index needs
-no model at all, on any machine.
-
 ```python
+# Another machine, no model downloaded: MATCH only reads the file.
 db = sqlite3.connect("notes.db")
 sqlite_sparse.load(db)
-db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the file
+db.execute("CREATE VIRTUAL TABLE temp.notes USING sparse0()")     # adopts the index in the file
 db.execute("SELECT rowid, score FROM temp.notes WHERE notes MATCH 'heart medication' LIMIT 5")
 ```
 
-A database file holds one sparse index; another `sparse0` table in the same file
-attaches to the same index rather than creating a second one.
-
-Because results are rows, semantic search composes with plain SQL. Ask for extra
-candidates with `k`, then filter and join like any other table.
-
 ```sql
+-- Results are rows: ask for k candidates, then filter and join like any other table.
 SELECT n.rowid, n.score, d.title
 FROM notes n JOIN documents d ON d.id = n.rowid
 WHERE n.text MATCH 'heart medication' AND k = 50 AND d.folder = 'work'
 ORDER BY n.score DESC LIMIT 10;
 ```
 
-Indexing is the expensive half, so large corpora are best built once on a GPU machine
-with the Python package (`sqlite-sparse build`, which writes the same file format) and
-the `.db` then shipped to wherever the reads happen.
-
-`LIMIT n` and `AND k = n` both work, and `ORDER BY score DESC` is honoured without a sort
-step. `DELETE FROM notes WHERE rowid = ?` marks a document deleted; run
-`SELECT sparse_compact()` now and then on an index with heavy churn to reclaim its
-postings. Documents longer than `max_seq` tokens (default 512) are truncated at insert, and
-each row in the `docs` table records `ntokens` and `truncated`. The full surface, including
-the Python helpers, is in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
+Indexing is the expensive half. Build a large corpus once on a GPU machine with
+`sqlite-sparse build` and ship the `.db` to wherever the reads happen. Deletes,
+compaction, truncation and the rest of the surface are in [docs/api.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/docs/api.md).
 
 ## How it works
 
 ![How sqlite-sparse indexes and searches](https://raw.githubusercontent.com/arbazsiddiqui/sqlite-sparse/master/docs/how-it-works.svg)
 
-**INSERT.** The extension tokenizes the text with the registered model's tokenizer and
-runs the encoder from the GGUF file through llama.cpp, one vector per token. It then
-applies the scoring head from the `.sprs` sidecar, which scores every word in the model's
-vocabulary against those vectors; positive scores are kept, compressed with a logarithm,
-and become the document's weighted terms. For the aspirin sentence that is 157 terms
-(`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42, `prevents` 0.18,
-and so on). Each term is appended to that word's posting list, a row in the file listing
-the documents it scored and the weight, stored as one byte. The document's token count
-and whether it was truncated go into the `docs` table.
-
-**MATCH.** The extension tokenizes the query the same way and reads one number per token
-from the query weight table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac`
-6.53, `arrest` 6.87). For each query word it walks that word's posting list and adds
-query weight × stored weight into the running total of every document listed; that
-accumulation is the scatter-add. `prevents` contributes 6.72 × 0.18 and `cardiac`
-6.53 × 0.42 to document 1, total 3.95. The documents touched are sorted and the top k
-returned. Scoring is exact over the stored weights, with no candidate stage and no
-approximate index, and ties break on the lower rowid. Nothing from the GGUF or the
-sidecar is read at query time.
+**INSERT.** The text is tokenized, the encoder runs through llama.cpp, and the head scores
+every vocabulary word against the token vectors. For the aspirin sentence that leaves 157
+weighted terms (`heart` 0.95, `stroke` 0.92, `risk` 0.78, `reduce` 0.70, `cardiac` 0.42,
+`prevents` 0.18, and so on). Each term is appended to that word's posting list, a row in
+the file listing the documents it scored and the weight as one byte.
+
+**MATCH.** The query is tokenized the same way and each token gets its weight from the
+table stored in the file (`what` 2.77, `prevents` 6.72, `cardiac` 6.53, `arrest` 6.87).
+For each query word the extension walks that word's posting list and adds query weight ×
+stored weight into every listed document's total, the scatter-add: `prevents` contributes
+6.72 × 0.18 and `cardiac` 6.53 × 0.42 to document 1, total 3.95. Scoring is exact over the
+stored weights, with no candidate stage or approximate index, and nothing from the GGUF or
+the sidecar is read.
 
 The file layout is in [FORMAT.md](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/FORMAT.md). The format and the sidecar header carry
 version 1, and files written by any 1.x release stay readable by later 1.x releases.
 
 ## Benchmarks
 
-Three ways to search inside a SQLite file, each in its shipped form, on the same machine.
-FTS5 is SQLite's built-in keyword search ranked by BM25. Dense brute-force is
-[sqlite-vec](https://github.com/asg017/sqlite-vec) int8, which scans every vector, with
-[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on torch
-CPU with 8 threads. Sparse is the `sparse0` extension with `mini` (23M, Q8_0 encoder, u8
-postings).
-
-Every number is end-to-end query latency, which for dense includes encoding the query,
-because that is its real query path. This compares the brute-force vector path inside
-SQLite, not an approximate nearest-neighbour index. The FTS5 query is the disjunction of
-the query's tokens ranked by `bm25()`; a conjunction is faster but misses documents that
-match only some of the terms.
+Three ways to search inside a SQLite file, each in its shipped form, on the same machine:
+FTS5, SQLite's built-in keyword search ranked by BM25; dense brute-force with
+[sqlite-vec](https://github.com/asg017/sqlite-vec) int8 and
+[mdbr-leaf-ir](https://huggingface.co/MongoDB/mdbr-leaf-ir) (23M) encoding queries on
+torch CPU; and `sparse0` with `mini` (23M, Q8_0 encoder, u8 postings). Latency is end to
+end, so dense includes encoding the query, because that is its real query path. FTS5 runs
+the OR of the query's tokens ranked by `bm25()` (an AND is faster but misses partial
+matches), and the dense lane is the brute-force scan, not an approximate index.
 
 | msmarco, 1M documents | FTS5 BM25 | dense brute-force | sqlite-sparse |
 |---|---|---|---|
@@ -199,7 +180,9 @@ match only some of the terms.
 | model at query time | none | 23M transformer | none |
 | retrieval | lexical | semantic | semantic |
 
-At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively.
+At 100K documents the p50s are 54 ms, 82 ms and 0.26 ms respectively. Measured on a GCE
+`c3-standard-8` (8 vCPU, 4 physical cores); the scripts and raw results are attached to
+each release.
 
 ### The extension does not lose the model's quality
 
@@ -218,14 +201,9 @@ the same documents and queries. The gain ranges from small (SciFact) to large (F
 
 The same model run in torch at fp32 agrees with the extension on 96 to 98 percent of
 top-10 results on every dataset, and storing weights as one byte instead of fp32 changed
-nDCG@10 by less than 0.001.
-
-Venue was a dedicated GCE `c3-standard-8` (8 vCPU, 4 physical cores) running Debian 12,
-with each lane in its own fresh process. 4,000 samples × 5 repetitions per lane (900 × 3
-for dense and 1,000 × 3 for FTS5 at 1M), median of repetition medians. Cold start is the
-second of three fresh-process runs. RAM is peak RSS after 50 warm queries. The corpus is
-the first 100K and 1M passages of MS MARCO with its dev queries. Benchmark scripts and raw
-results are attached to each release.
+nDCG@10 by less than 0.001. For context against dense models of the same size,
+mdbr-leaf-ir (23M) reports 0.5355 BEIR average to `mini`'s 0.497; the two do very
+different amounts of work at query time, so that is context, not a controlled comparison.
 
 ## Models
 
@@ -235,17 +213,16 @@ results are attached to each release.
 | `base` | [doc-v3-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) | 67M | 0.517 | [arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-doc-v3-distill-GGUF) |
 | `multilingual` | [multilingual-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1) | 168M | multilingual | [arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF](https://huggingface.co/arbazsiddiqui/opensearch-neural-sparse-multilingual-v1-GGUF) |
 
-All three are in the [sqlite-sparse models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
-collection, and `sqlite_sparse.register(db, alias)` fetches one into `~/.cache/sqlite-sparse`.
-Each conversion is validated against the original SentenceTransformers implementation
-(encoder hidden states at cosine 0.9997 or better, term weights within 1.3e-3). Weights
-are unmodified from the Apache-2.0 originals by the OpenSearch project.
+All three are in the [sqlite-sparse
+models](https://huggingface.co/collections/arbazsiddiqui/sqlite-sparse-models-6a929c8e0cb15b0e8ed47d43)
+collection, and `sqlite_sparse.register(db, alias)` fetches one into
+`~/.cache/sqlite-sparse`. Weights are unmodified from the Apache-2.0 originals by the
+OpenSearch project.
 
 ### Bring your own model
 
-Any inference-free OpenSearch-style sparse encoder on Hugging Face works. Two files are
-needed, the encoder as GGUF and a sidecar with the scoring head, query weight table and
-vocabulary.
+Any inference-free OpenSearch-style sparse encoder on Hugging Face works: the encoder as
+GGUF plus a `.sprs` sidecar holding the head and the query weight table.
 
 ```
 git clone --depth 1 https://github.com/ggml-org/llama.cpp
@@ -260,9 +237,29 @@ SELECT sparse_register('mine', 'model_q8.gguf', 'model.sprs');
 CREATE VIRTUAL TABLE notes USING sparse0(model='mine');
 ```
 
-The converter requires the checkpoint to be a BERT-family encoder with a masked-LM head
-and a static query weight table. llama.cpp must support the encoder architecture; it does
-not support GTE (`doc-v3-gte`).
+The checkpoint must be a BERT-family encoder with a masked-LM head and a static query
+weight table, and llama.cpp must support the architecture (it does not support GTE,
+`doc-v3-gte`).
+
+### Bring your own vectors
+
+Any sparse model works if you run it yourself, including SPLADE models that encode the
+query too, and models llama.cpp cannot run. Create the index from the model's vocabulary
+and hand it `{"token": weight}` objects for documents and for queries. Nothing is
+converted; the extension stores and scores, and the file is the same format.
+
+```sql
+CREATE VIRTUAL TABLE notes USING sparse0(vocab='vocab.txt');    -- one token per line, no model
+INSERT INTO notes(rowid, terms) VALUES (1, '{"heart": 0.95, "cardiac": 0.42, "stroke": 0.92}');
+SELECT rowid, score FROM notes WHERE notes.terms MATCH '{"cardiac": 6.53, "arrest": 6.87}' LIMIT 5;
+```
+
+Tokens must be in the vocabulary (an unknown token is an error on insert and ignored in a
+query), weights must be positive, and weights above 6.375 saturate the one-byte storage.
+`terms MATCH` also works on an index one of the shipped models built, so a query encoded
+by your own model can search it. Text queries on a vocabulary-only index are an error,
+since there is no query weight table. In Python: `SparseIndex.create_external(path,
+vocab)`, `add_terms(id, terms)`, `search_terms(terms)`.
 
 ## Development
 
@@ -275,7 +272,10 @@ make test     # installs the Python binding and runs the suite
 `src/` is the extension (`sparse0.c` virtual table, `wordpiece.c` tokenizer on utf8proc, `head.c`
 scoring head on ggml, `scorer.c` scatter-add, `encoder.c` llama.cpp wrapper).
 `bindings/python` is the reference implementation of the file format and the test oracle.
-Component agreement is logged in [`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
+Each shipped conversion is validated against the original SentenceTransformers
+implementation (encoder hidden states at cosine 0.9997 or better, term weights within
+1.3e-3), and component agreement is logged in
+[`tests/test_differential.md`](https://github.com/arbazsiddiqui/sqlite-sparse/blob/master/tests/test_differential.md).
 
 ## License