zeusdb-vector-database 0.5.0__tar.gz → 0.7.0__tar.gz

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/NOTICE

@@ -6,19 +6,15 @@ This product includes software developed by:
 - The Python Software Foundation (https://www.python.org/psf/)
 - The Rust Project Developers (https://www.rust-lang.org/)
 - The PyO3 Project (https://pyo3.rs/)
-- Jean-pierreBoth (https://crates.io/crates/hnsw_rs/)
-
-The hnsw_rs crate, version 0.3.4, is vendored in source form at vendor/hnsw_rs
-and carries five ZeusDB patches, one to the reverse link layer assignment, one
-seeding level assignment so index builds are reproducible, one guarding the
-neighbour list overflow pop against stranding a point's last inbound link, one
-correcting the filtered search path so it cannot panic or return an empty
-result set, and one correcting the per-layer capacity reservation so a declared
-index size does not commit hundreds of times the memory it needs. See
-vendor/hnsw_rs/ZEUSDB-PATCH.md for the patch record. hnsw_rs is dual licensed
-under MIT or Apache-2.0.
-
-Third-party components may be licensed under their respective terms. See LICENSES/ for full license texts.
+
+Releases up to and including 0.6.0 were built on a vendored copy of a Rust HNSW
+crate by Jean-Pierre Both, dual licensed under MIT or Apache-2.0. From 0.7.0 the
+graph structure, its traversal, its insert, its level stream and its on-disk
+format are ZeusDB's own, and neither that crate nor its source is part of this
+package. ZeusDB's graph reproduces the behaviour of that implementation, and the
+acknowledgement is kept here for that reason.
+
+Third-party components may be licensed under their respective terms.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zeusdb-vector-database
-Version: 0.5.0
+Version: 0.7.0
 Classifier: Programming Language :: Rust
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Requires-Dist: numpy>=2.2.6,<3.0.0
@@ -35,7 +35,7 @@ Project-URL: Repository, https://github.com/zeusdb/zeusdb-vector-database
       <td><strong>Meta</strong></td>
       <td>
         <a href="https://pypi.org/project/zeusdb-vector-database/"><img src="https://img.shields.io/pypi/v/zeusdb-vector-database?label=PyPI&color=blue"></a>&nbsp;
-        <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%7C3.11%7C3.12%7C3.13-blue?logo=python&logoColor=ffdd54"></a>&nbsp;
+        <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%7C3.11%7C3.12%7C3.13%7C3.14-blue?logo=python&logoColor=ffdd54"></a>&nbsp;
         <a href="https://github.com/zeusdb/zeusdb-vector-database/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"></a>&nbsp;
         <a href="https://www.rust-lang.org"><img src="https://img.shields.io/badge/Powered%20by-Rust-black?logo=rust&logoColor=white" alt="Powered by Rust"></a>&nbsp;
         <a href="https://github.com/ZeusDB"><img src="https://github.com/user-attachments/assets/e140d900-1160-4eaa-85c0-2b3507a5f0f5" alt="ZeusDB"></a>&nbsp;
@@ -212,7 +212,7 @@ HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=5, vecto
 | `dim`            | `int`  | `1536`    | Dimensionality of the vectors to be indexed. Each vector must have this length. Must be positive. The default of 1536 matches the output dimensionality of OpenAI's `text-embedding-3-small` and `text-embedding-ada-002` models. |
 | `space`          | `str`  | `"cosine"`| Distance metric used for similarity search. One of `"cosine"`, `"l1"`, `"l2"`. Case-insensitive. |
 | `m`              | `int`  | `16` or `32`, see below | Number of bi-directional connections created for each new node, from 2 to 256. Higher `m` improves recall but increases index size and build time. |
-| `ef_construction`| `int`  | `200`     | Size of the dynamic list used during index construction. Must be positive. Larger values increase indexing time and memory, but improve quality. |
+| `ef_construction`| `int`  | `200`     | Width of the candidate search each insertion runs. Must be positive. It costs build time and buys graph quality, and it changes neither search latency nor the size of the finished index. See below. |
 | `expected_size`  | `int`  | `10000`   | Estimated number of records to be inserted, from 1 to 100,000,000. Used for preallocating internal data structures and for choosing the default `m`. Not a hard limit, see below. |
 | `quantization_config` | `dict` | `None` | Product Quantization configuration for memory-efficient vector compression. See [Product Quantization](#️-product-quantization). |
 
@@ -229,6 +229,24 @@ The upper bound of 100,000,000 exists because the graph reserves one slot per de
 
 **`m` starts at 2, not 1.** Layer assignment samples from a scale of `1 / ln(m)`, which is infinity at `m` of 1, so every point is redispatched uniformly across all 16 layers rather than following the exponential distribution the graph depends on. On 3,000 records of 32 dimensions, recall at 10 measured 0.0220 at `m` of 1 against 0.6880 at 2 and 1.0000 at 16.
 
+**`ef_construction` governs insertion and nothing else.** It is the width of the candidate search each insertion runs, at the new point's own layer and at every layer below it. The descent above that layer runs at a width of 1 and ignores it. The candidates that search returns are the pool the neighbour selection draws from, so `ef_construction` sets the supply and `m` sets how much of it is kept.
+
+**It does not change search latency or index size.** A node holds at most `2 × m` neighbours at layer zero and `m` above it whatever width found them, so the finished graph is the same size at every `ef_construction`. Measured over 50,000 records, mean query latency moved between 0.89 ms and 2.85 ms across a sixteenfold range of `ef_construction` with no trend in it, and resident memory ran backwards on one of the three datasets. What it does change is build time, which is linear in it above 100. On OpenAI embeddings of 1,536 dimensions, 50,000 records built in 62.7 s, 131.2 s, 262.8 s and 464.9 s at `ef_construction` 100, 200, 400 and 800. On GloVe of 100 dimensions the same four are 37.7 s, 75.0 s, 140.4 s and 239.1 s. Below 100 the curve flattens, because the fixed per-insert work stops being dominated by the candidate search.
+
+**Recall stops improving at or near the default.** Recall at 10 over 50,000 records at the default `m` of 32, 500 queries, at the default search width:
+
+| `ef_construction` | GloVe 100d | SIFT 128d | OpenAI 1536d |
+|-------------------|------------|-----------|--------------|
+| 50                | 0.9176     | 0.9982    | 0.9852       |
+| 100               | 0.9400     | 0.9992    | 0.9912       |
+| 200               | 0.9606     | 0.9996    | 0.9962       |
+| 400               | 0.9684     | 0.9998    | 0.9972       |
+| 800               | 0.9714     | 0.9996    | 0.9978       |
+
+SIFT plateaus at 100 and OpenAI at 200. GloVe is the one dataset that keeps climbing, and there the cheaper move is the search width rather than the build. At `ef_construction` 200 GloVe returns 0.9606 at the default width and 0.9984 at `ef_search=500`, so widening the search buys 0.0378 on the graph already built where doubling `ef_construction` buys 0.0078 for 1.9 times the build time. Raise `ef_search` before raising this.
+
+**Keep `ef_construction` above `2 × m`.** The neighbour selection heuristic runs only when the candidate list is longer than the neighbour budget, which is `2 × m` at layer zero, and the candidate list is exactly `ef_construction` long on any index holding more records than that. At or below the budget the graph keeps every candidate the search returned, in distance order, and prunes none of them. `create()` warns when the pair reaches that point. `m` is capped at 256, so `m=100` with the default `ef_construction` is enough to switch the heuristic off. The defaults are clear of it, 200 against a budget of 32 at `m` 16 and 64 at `m` 32.
+
 <br/>
 
 ### 2️⃣ Add Data to the Index
@@ -625,17 +643,21 @@ storage_mode_description: raw_only
 
 `get_stats()` returns a `str` to `str` map. It also carries `dimension`, `space`, `m`, `ef_construction`, `expected_size`, `index_type`, `raw_vectors_stored`, `quantized_codes_stored` and `storage_mode`, plus training and compression fields once quantization is configured.
 
-It is also where the memory figures live, on every index rather than only on a quantized one. `graph_memory_mb` is the HNSW graph, `raw_vectors_memory_mb` is the raw vector store and `total_memory_mb` is the sum of everything below. On a quantized index `quantized_codes_memory_mb` scales with the record count, while `codebook_memory_mb` and `sdc_table_memory_mb` are fixed by `dim`, `subvectors` and `bits` and do not move as records arrive. `raw_vectors_retained` states the mode's policy: `none_once_trained` for `quantized_only`, whose raw vectors are released when training completes, and `all_records` for `quantized_with_raw`.
+It is also where the memory figures live, on every index rather than only on a quantized one. `graph_memory_mb` is the HNSW graph, `raw_vectors_memory_mb` is the raw vector store, `index_bookkeeping_memory_mb` is the hash tables that find a record, and `total_memory_mb` is the sum of everything below. On a quantized index `quantized_codes_memory_mb` scales with the record count, while `codebook_memory_mb` and `sdc_table_memory_mb` are fixed by `dim`, `subvectors` and `bits` and do not move as records arrive. `raw_vectors_retained` states the mode's policy: `none_once_trained` for `quantized_only`, whose raw vectors are released when training completes, and `all_records` for `quantized_with_raw`.
 
 **The graph is usually the largest of those figures, and on a memory optimized index it is nearly all of it.** It owns a second copy of every point on top of its neighbour lists, its sixteen per point layer headers and the counters around them. At 50,000 records of `dim=1536` a trained `quantized_only` index reports 98.1 MB of graph against 2.3 MB of codes and 7.5 MB of fixed tables.
 
-**`total_memory_mb` is not the resident set and it does not claim to be.** It prices the five structures above. The id maps, the metadata map, the hash table slots and the allocator's own headers and fragmentation sit outside it, and together they run at roughly 1,500 bytes per record whatever the dimension. Measured on three loaded indexes of 50,000 real 1,536-dimensional embeddings, one process each:
+`index_bookkeeping_memory_mb` is what the index spends on finding a record rather than on holding one. Five hash tables carry a record, being `id_map`, `rev_map`, the metadata map, the raw vector store and the code store, and two of them carry a second copy of its id. It is proportional to the record count and independent of the dimension, at 265 bytes per record with no quantization, 334 under `quantized_with_raw` and 265 under `quantized_only` at 50,000 records.
+
+**`total_memory_mb` is not the resident set and it does not claim to be.** It prices the six structures above, meaning the bytes the index asked the allocator for. What it leaves out is the allocator itself, being its own block headers, its rounding and its fragmentation, and that is a property of the platform rather than something the index holds. Measured on three loaded indexes of 50,000 real 1,536-dimensional embeddings, one process each, with the interpreter's own resident set subtracted:
 
-| storage mode | `total_memory_mb` | resident | ratio |
-|---|---:|---:|---:|
-| no quantization | 692.4 MB | 805.9 MiB | 1.16x |
-| `quantized_with_raw` | 401.2 MB | 474.8 MiB | 1.18x |
-| `quantized_only` | 107.8 MB | 181.4 MiB | 1.68x |
+| storage mode | `total_memory_mb` | resident | share reported | unpriced |
+|---|---:|---:|---:|---:|
+| no quantization | 705.1 MB | 805.4 MiB | 0.88 | 2,103 B per record |
+| `quantized_with_raw` | 415.9 MB | 473.3 MiB | 0.88 | 1,202 B per record |
+| `quantized_only` | 119.7 MB | 181.0 MiB | 0.66 | 1,285 B per record |
+
+The unpriced share is almost all allocator overhead on the graph, which asks for six small blocks per point. It runs 1.25 times the graph figure with no quantization, where the per point data block is 6,144 bytes and rounding is a small share of it, and 1.63 and 1.67 times under the two quantized modes, where that block is 48 bytes. That is why `quantized_only` reports the smallest share of its resident set.
 
 Size infrastructure from the resident column rather than the reported one. The reported figure is what the index holds; the process holds that plus what the allocator takes to hold it.
 
@@ -1012,13 +1034,22 @@ The relation is curved as well as sublinear. On dbpedia-openai the calibration m
 
 What the calibration asks for at 100,000 records, against what the data needs and against the fixed 2%. Every figure is measured on a built index through the ordinary search path, 1,000 queries at `top_k=10` against exact ground truth. The requirement is the smallest fetch on that same index reaching recall at 10 of 0.99, read off a sweep of the explicit `rerank` argument:
 
-| dataset | calibrated fetch | measured requirement | the old fixed fetch | recall, calibrated | recall, fixed |
-|---|---:|---:|---:|---:|---:|
-| dbpedia-openai | 776 | 750 | 2,000 | 0.9905 | 0.9954 |
-| glove-100 | 7,749 | 4,500 | 2,000 | 0.9962 | 0.9723 |
-| sift-128 | 596 | 450 | 2,000 | 0.9941 | 1.0000 |
+| dataset | records | calibrated fetch | measured requirement | ratio | recall at 10 | mean query |
+|---|---:|---:|---:|---:|---:|---:|
+| dbpedia-openai | 50,000 | 554 | 860 | 0.64 | 0.9883 | 3.33 ms |
+| glove-100 | 50,000 | 6,534 | 3,060 | 2.14 | 0.9982 | 35.79 ms |
+| sift-128 | 50,000 | 465 | 370 | 1.26 | 0.9953 | 1.88 ms |
+| dbpedia-openai | 100,000 | 747 | 620 | 1.20 | 0.9907 | 5.02 ms |
+| glove-100 | 100,000 | 11,439 | 4,620 | 2.48 | 0.9985 | 140.08 ms |
+| sift-128 | 100,000 | 656 | 450 | 1.46 | 0.9968 | 3.19 ms |
+
+**The margin is not the same on the three, and glove-100 carries most of it.** The rule is one measurement per index scaled by two constants that are the same everywhere, being a safety factor of 1.75 and a bias of 0.15 on the fitted exponent. Taking the fitted exponent alone with neither constant, the extrapolation to 100,000 records asks for 4,626 candidates on glove-100 against a requirement of 4,620, which is 1.001 times what that index needs. On the other two it asks for 0.49 and 0.59 times what they need. The constants exist for those two, and on glove-100 they are margin over a fetch that was already right.
 
-The calibration clears the requirement on all three. **It does not land on it and it is not meant to**, since the measurement is taken on a tenth of the records with queries drawn from the corpus, where your queries will not be.
+Neither constant can come down without costing recall elsewhere. The constant that would bring glove-100 to the 1.20 ratio dbpedia-openai has at 100,000 records takes the dbpedia-openai fetch from 747 to 359, and a sweep of that same index reads recall 0.9787 at 360 candidates. dbpedia-openai also sits under 0.99 at 10,000, 25,000 and 50,000 records, at 0.9878, 0.9800 and 0.9883, so it has nothing to give back. What separates the datasets is the ratio between the requirement on a built index and the depth the calibration measures in the code ordering over its training sample, and the training sample cannot measure that ratio because the graph does not exist yet when it runs.
+
+**If query time on glove-like data matters more to you than the last thousandth of recall, name `rerank` yourself.** On glove-100 at 50,000 records a fetch of 3,060 reads recall 0.9900 at 18.14 ms against 0.9982 at 35.79 ms, so half the query time buys back 0.008 of recall.
+
+The two tables below carry fetch figures measured before the codebook seeding this release ships, which moved the calibrated fetch upward on glove-100. Read their ratios as the shape rather than as current figures.
 
 **On data with no resolvable structure no fetch works.** Once the group the codes cannot separate is smaller than `top_k`, the true top ten span groups, the distances between groups differ in the fourth decimal, and nothing reaches them. At 5,000 clusters over 25,000 records, being five records to a cluster, recall at 10 reaches 0.917 at a fetch of half the corpus, and uniform points on the sphere reach 0.859. Measure recall on your own data before you rely on quantization.
 
@@ -1136,7 +1167,7 @@ print("saved:", sorted(os.listdir("my_index.zdb")))
 
 *Output, with the progress lines omitted*
 ```
-saved: ['config.json', 'hnsw_index.hnsw.data', 'hnsw_index.hnsw.graph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
+saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
 ```
 
 <br />
@@ -1206,7 +1237,7 @@ print("saved:", sorted(os.listdir("quantized_index.zdb")))
 quantization active: True
 quantization active after load: True
 storage mode after load: quantized_active
-saved: ['config.json', 'hnsw_index.hnsw.data', 'hnsw_index.hnsw.graph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
+saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
 ```
 
 <br/>
@@ -1224,11 +1255,12 @@ my_index.zdb/
 ├── quantization.json       # PQ configuration (if enabled)
 ├── pq_centroids.bin        # Trained centroids (if PQ trained)
 ├── pq_codes.bin            # Quantized codes (if PQ active)
-├── hnsw_index.hnsw.graph   # HNSW graph structure
-└── hnsw_index.hnsw.data    # HNSW graph payload
+└── hnsw_index.zdbgraph     # HNSW graph structure and payload
 ```
 
-`manifest.json` lists both graph files under `files_included`. The load path restores the saved graph rather than rebuilding it, so both are required to reopen a directory holding records.
+`manifest.json` lists the graph file under `files_included`. The load path restores the saved graph rather than rebuilding it, so the file is required to reopen a directory holding records.
+
+A directory saved by 0.6.0 or earlier holds `hnsw_index.hnsw.graph` and `hnsw_index.hnsw.data` instead. Opening it still works: the graph is rebuilt once from the stored records, and the next `.save()` writes the single file.
 
 <br/>
 

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/README.md

@@ -14,7 +14,7 @@
       <td><strong>Meta</strong></td>
       <td>
         <a href="https://pypi.org/project/zeusdb-vector-database/"><img src="https://img.shields.io/pypi/v/zeusdb-vector-database?label=PyPI&color=blue"></a>&nbsp;
-        <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%7C3.11%7C3.12%7C3.13-blue?logo=python&logoColor=ffdd54"></a>&nbsp;
+        <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.10%7C3.11%7C3.12%7C3.13%7C3.14-blue?logo=python&logoColor=ffdd54"></a>&nbsp;
         <a href="https://github.com/zeusdb/zeusdb-vector-database/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"></a>&nbsp;
         <a href="https://www.rust-lang.org"><img src="https://img.shields.io/badge/Powered%20by-Rust-black?logo=rust&logoColor=white" alt="Powered by Rust"></a>&nbsp;
         <a href="https://github.com/ZeusDB"><img src="https://github.com/user-attachments/assets/e140d900-1160-4eaa-85c0-2b3507a5f0f5" alt="ZeusDB"></a>&nbsp;
@@ -191,7 +191,7 @@ HNSWIndex(dim=8, space=cosine, m=16, ef_construction=200, expected_size=5, vecto
 | `dim`            | `int`  | `1536`    | Dimensionality of the vectors to be indexed. Each vector must have this length. Must be positive. The default of 1536 matches the output dimensionality of OpenAI's `text-embedding-3-small` and `text-embedding-ada-002` models. |
 | `space`          | `str`  | `"cosine"`| Distance metric used for similarity search. One of `"cosine"`, `"l1"`, `"l2"`. Case-insensitive. |
 | `m`              | `int`  | `16` or `32`, see below | Number of bi-directional connections created for each new node, from 2 to 256. Higher `m` improves recall but increases index size and build time. |
-| `ef_construction`| `int`  | `200`     | Size of the dynamic list used during index construction. Must be positive. Larger values increase indexing time and memory, but improve quality. |
+| `ef_construction`| `int`  | `200`     | Width of the candidate search each insertion runs. Must be positive. It costs build time and buys graph quality, and it changes neither search latency nor the size of the finished index. See below. |
 | `expected_size`  | `int`  | `10000`   | Estimated number of records to be inserted, from 1 to 100,000,000. Used for preallocating internal data structures and for choosing the default `m`. Not a hard limit, see below. |
 | `quantization_config` | `dict` | `None` | Product Quantization configuration for memory-efficient vector compression. See [Product Quantization](#️-product-quantization). |
 
@@ -208,6 +208,24 @@ The upper bound of 100,000,000 exists because the graph reserves one slot per de
 
 **`m` starts at 2, not 1.** Layer assignment samples from a scale of `1 / ln(m)`, which is infinity at `m` of 1, so every point is redispatched uniformly across all 16 layers rather than following the exponential distribution the graph depends on. On 3,000 records of 32 dimensions, recall at 10 measured 0.0220 at `m` of 1 against 0.6880 at 2 and 1.0000 at 16.
 
+**`ef_construction` governs insertion and nothing else.** It is the width of the candidate search each insertion runs, at the new point's own layer and at every layer below it. The descent above that layer runs at a width of 1 and ignores it. The candidates that search returns are the pool the neighbour selection draws from, so `ef_construction` sets the supply and `m` sets how much of it is kept.
+
+**It does not change search latency or index size.** A node holds at most `2 × m` neighbours at layer zero and `m` above it whatever width found them, so the finished graph is the same size at every `ef_construction`. Measured over 50,000 records, mean query latency moved between 0.89 ms and 2.85 ms across a sixteenfold range of `ef_construction` with no trend in it, and resident memory ran backwards on one of the three datasets. What it does change is build time, which is linear in it above 100. On OpenAI embeddings of 1,536 dimensions, 50,000 records built in 62.7 s, 131.2 s, 262.8 s and 464.9 s at `ef_construction` 100, 200, 400 and 800. On GloVe of 100 dimensions the same four are 37.7 s, 75.0 s, 140.4 s and 239.1 s. Below 100 the curve flattens, because the fixed per-insert work stops being dominated by the candidate search.
+
+**Recall stops improving at or near the default.** Recall at 10 over 50,000 records at the default `m` of 32, 500 queries, at the default search width:
+
+| `ef_construction` | GloVe 100d | SIFT 128d | OpenAI 1536d |
+|-------------------|------------|-----------|--------------|
+| 50                | 0.9176     | 0.9982    | 0.9852       |
+| 100               | 0.9400     | 0.9992    | 0.9912       |
+| 200               | 0.9606     | 0.9996    | 0.9962       |
+| 400               | 0.9684     | 0.9998    | 0.9972       |
+| 800               | 0.9714     | 0.9996    | 0.9978       |
+
+SIFT plateaus at 100 and OpenAI at 200. GloVe is the one dataset that keeps climbing, and there the cheaper move is the search width rather than the build. At `ef_construction` 200 GloVe returns 0.9606 at the default width and 0.9984 at `ef_search=500`, so widening the search buys 0.0378 on the graph already built where doubling `ef_construction` buys 0.0078 for 1.9 times the build time. Raise `ef_search` before raising this.
+
+**Keep `ef_construction` above `2 × m`.** The neighbour selection heuristic runs only when the candidate list is longer than the neighbour budget, which is `2 × m` at layer zero, and the candidate list is exactly `ef_construction` long on any index holding more records than that. At or below the budget the graph keeps every candidate the search returned, in distance order, and prunes none of them. `create()` warns when the pair reaches that point. `m` is capped at 256, so `m=100` with the default `ef_construction` is enough to switch the heuristic off. The defaults are clear of it, 200 against a budget of 32 at `m` 16 and 64 at `m` 32.
+
 <br/>
 
 ### 2️⃣ Add Data to the Index
@@ -604,17 +622,21 @@ storage_mode_description: raw_only
 
 `get_stats()` returns a `str` to `str` map. It also carries `dimension`, `space`, `m`, `ef_construction`, `expected_size`, `index_type`, `raw_vectors_stored`, `quantized_codes_stored` and `storage_mode`, plus training and compression fields once quantization is configured.
 
-It is also where the memory figures live, on every index rather than only on a quantized one. `graph_memory_mb` is the HNSW graph, `raw_vectors_memory_mb` is the raw vector store and `total_memory_mb` is the sum of everything below. On a quantized index `quantized_codes_memory_mb` scales with the record count, while `codebook_memory_mb` and `sdc_table_memory_mb` are fixed by `dim`, `subvectors` and `bits` and do not move as records arrive. `raw_vectors_retained` states the mode's policy: `none_once_trained` for `quantized_only`, whose raw vectors are released when training completes, and `all_records` for `quantized_with_raw`.
+It is also where the memory figures live, on every index rather than only on a quantized one. `graph_memory_mb` is the HNSW graph, `raw_vectors_memory_mb` is the raw vector store, `index_bookkeeping_memory_mb` is the hash tables that find a record, and `total_memory_mb` is the sum of everything below. On a quantized index `quantized_codes_memory_mb` scales with the record count, while `codebook_memory_mb` and `sdc_table_memory_mb` are fixed by `dim`, `subvectors` and `bits` and do not move as records arrive. `raw_vectors_retained` states the mode's policy: `none_once_trained` for `quantized_only`, whose raw vectors are released when training completes, and `all_records` for `quantized_with_raw`.
 
 **The graph is usually the largest of those figures, and on a memory optimized index it is nearly all of it.** It owns a second copy of every point on top of its neighbour lists, its sixteen per point layer headers and the counters around them. At 50,000 records of `dim=1536` a trained `quantized_only` index reports 98.1 MB of graph against 2.3 MB of codes and 7.5 MB of fixed tables.
 
-**`total_memory_mb` is not the resident set and it does not claim to be.** It prices the five structures above. The id maps, the metadata map, the hash table slots and the allocator's own headers and fragmentation sit outside it, and together they run at roughly 1,500 bytes per record whatever the dimension. Measured on three loaded indexes of 50,000 real 1,536-dimensional embeddings, one process each:
+`index_bookkeeping_memory_mb` is what the index spends on finding a record rather than on holding one. Five hash tables carry a record, being `id_map`, `rev_map`, the metadata map, the raw vector store and the code store, and two of them carry a second copy of its id. It is proportional to the record count and independent of the dimension, at 265 bytes per record with no quantization, 334 under `quantized_with_raw` and 265 under `quantized_only` at 50,000 records.
+
+**`total_memory_mb` is not the resident set and it does not claim to be.** It prices the six structures above, meaning the bytes the index asked the allocator for. What it leaves out is the allocator itself, being its own block headers, its rounding and its fragmentation, and that is a property of the platform rather than something the index holds. Measured on three loaded indexes of 50,000 real 1,536-dimensional embeddings, one process each, with the interpreter's own resident set subtracted:
 
-| storage mode | `total_memory_mb` | resident | ratio |
-|---|---:|---:|---:|
-| no quantization | 692.4 MB | 805.9 MiB | 1.16x |
-| `quantized_with_raw` | 401.2 MB | 474.8 MiB | 1.18x |
-| `quantized_only` | 107.8 MB | 181.4 MiB | 1.68x |
+| storage mode | `total_memory_mb` | resident | share reported | unpriced |
+|---|---:|---:|---:|---:|
+| no quantization | 705.1 MB | 805.4 MiB | 0.88 | 2,103 B per record |
+| `quantized_with_raw` | 415.9 MB | 473.3 MiB | 0.88 | 1,202 B per record |
+| `quantized_only` | 119.7 MB | 181.0 MiB | 0.66 | 1,285 B per record |
+
+The unpriced share is almost all allocator overhead on the graph, which asks for six small blocks per point. It runs 1.25 times the graph figure with no quantization, where the per point data block is 6,144 bytes and rounding is a small share of it, and 1.63 and 1.67 times under the two quantized modes, where that block is 48 bytes. That is why `quantized_only` reports the smallest share of its resident set.
 
 Size infrastructure from the resident column rather than the reported one. The reported figure is what the index holds; the process holds that plus what the allocator takes to hold it.
 
@@ -991,13 +1013,22 @@ The relation is curved as well as sublinear. On dbpedia-openai the calibration m
 
 What the calibration asks for at 100,000 records, against what the data needs and against the fixed 2%. Every figure is measured on a built index through the ordinary search path, 1,000 queries at `top_k=10` against exact ground truth. The requirement is the smallest fetch on that same index reaching recall at 10 of 0.99, read off a sweep of the explicit `rerank` argument:
 
-| dataset | calibrated fetch | measured requirement | the old fixed fetch | recall, calibrated | recall, fixed |
-|---|---:|---:|---:|---:|---:|
-| dbpedia-openai | 776 | 750 | 2,000 | 0.9905 | 0.9954 |
-| glove-100 | 7,749 | 4,500 | 2,000 | 0.9962 | 0.9723 |
-| sift-128 | 596 | 450 | 2,000 | 0.9941 | 1.0000 |
+| dataset | records | calibrated fetch | measured requirement | ratio | recall at 10 | mean query |
+|---|---:|---:|---:|---:|---:|---:|
+| dbpedia-openai | 50,000 | 554 | 860 | 0.64 | 0.9883 | 3.33 ms |
+| glove-100 | 50,000 | 6,534 | 3,060 | 2.14 | 0.9982 | 35.79 ms |
+| sift-128 | 50,000 | 465 | 370 | 1.26 | 0.9953 | 1.88 ms |
+| dbpedia-openai | 100,000 | 747 | 620 | 1.20 | 0.9907 | 5.02 ms |
+| glove-100 | 100,000 | 11,439 | 4,620 | 2.48 | 0.9985 | 140.08 ms |
+| sift-128 | 100,000 | 656 | 450 | 1.46 | 0.9968 | 3.19 ms |
+
+**The margin is not the same on the three, and glove-100 carries most of it.** The rule is one measurement per index scaled by two constants that are the same everywhere, being a safety factor of 1.75 and a bias of 0.15 on the fitted exponent. Taking the fitted exponent alone with neither constant, the extrapolation to 100,000 records asks for 4,626 candidates on glove-100 against a requirement of 4,620, which is 1.001 times what that index needs. On the other two it asks for 0.49 and 0.59 times what they need. The constants exist for those two, and on glove-100 they are margin over a fetch that was already right.
 
-The calibration clears the requirement on all three. **It does not land on it and it is not meant to**, since the measurement is taken on a tenth of the records with queries drawn from the corpus, where your queries will not be.
+Neither constant can come down without costing recall elsewhere. The constant that would bring glove-100 to the 1.20 ratio dbpedia-openai has at 100,000 records takes the dbpedia-openai fetch from 747 to 359, and a sweep of that same index reads recall 0.9787 at 360 candidates. dbpedia-openai also sits under 0.99 at 10,000, 25,000 and 50,000 records, at 0.9878, 0.9800 and 0.9883, so it has nothing to give back. What separates the datasets is the ratio between the requirement on a built index and the depth the calibration measures in the code ordering over its training sample, and the training sample cannot measure that ratio because the graph does not exist yet when it runs.
+
+**If query time on glove-like data matters more to you than the last thousandth of recall, name `rerank` yourself.** On glove-100 at 50,000 records a fetch of 3,060 reads recall 0.9900 at 18.14 ms against 0.9982 at 35.79 ms, so half the query time buys back 0.008 of recall.
+
+The two tables below carry fetch figures measured before the codebook seeding this release ships, which moved the calibrated fetch upward on glove-100. Read their ratios as the shape rather than as current figures.
 
 **On data with no resolvable structure no fetch works.** Once the group the codes cannot separate is smaller than `top_k`, the true top ten span groups, the distances between groups differ in the fourth decimal, and nothing reaches them. At 5,000 clusters over 25,000 records, being five records to a cluster, recall at 10 reaches 0.917 at a fetch of half the corpus, and uniform points on the sphere reach 0.859. Measure recall on your own data before you rely on quantization.
 
@@ -1115,7 +1146,7 @@ print("saved:", sorted(os.listdir("my_index.zdb")))
 
 *Output, with the progress lines omitted*
 ```
-saved: ['config.json', 'hnsw_index.hnsw.data', 'hnsw_index.hnsw.graph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
+saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'vectors.bin']
 ```
 
 <br />
@@ -1185,7 +1216,7 @@ print("saved:", sorted(os.listdir("quantized_index.zdb")))
 quantization active: True
 quantization active after load: True
 storage mode after load: quantized_active
-saved: ['config.json', 'hnsw_index.hnsw.data', 'hnsw_index.hnsw.graph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
+saved: ['config.json', 'hnsw_index.zdbgraph', 'manifest.json', 'mappings.bin', 'metadata.json', 'pq_centroids.bin', 'pq_codes.bin', 'quantization.json', 'vectors.bin']
 ```
 
 <br/>
@@ -1203,11 +1234,12 @@ my_index.zdb/
 ├── quantization.json       # PQ configuration (if enabled)
 ├── pq_centroids.bin        # Trained centroids (if PQ trained)
 ├── pq_codes.bin            # Quantized codes (if PQ active)
-├── hnsw_index.hnsw.graph   # HNSW graph structure
-└── hnsw_index.hnsw.data    # HNSW graph payload
+└── hnsw_index.zdbgraph     # HNSW graph structure and payload
 ```
 
-`manifest.json` lists both graph files under `files_included`. The load path restores the saved graph rather than rebuilding it, so both are required to reopen a directory holding records.
+`manifest.json` lists the graph file under `files_included`. The load path restores the saved graph rather than rebuilding it, so the file is required to reopen a directory holding records.
+
+A directory saved by 0.6.0 or earlier holds `hnsw_index.hnsw.graph` and `hnsw_index.hnsw.data` instead. Opening it still works: the graph is rebuilt once from the stored records, and the next `.save()` writes the single file.
 
 <br/>
 

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/pyproject.toml

@@ -1,13 +1,19 @@
 [project]
 name = "zeusdb-vector-database"
-version = "0.5.0"
+version = "0.7.0"
 description = "Blazing-fast vector DB with real-time similarity search and metadata filtering."
 readme = "README.md"
 authors = [
     { name = "ZeusDB", email = "contact@zeusdb.com" }
 ]
+# The floor and the abi3 target move together. vdb-core/Cargo.toml enables
+# pyo3/abi3-py310, so the wheel is tagged cp310-abi3 and pip on 3.10 selects
+# it. Raising one without the other leaves a version with no wheel that pip
+# can match, which falls back to building the sdist from source.
 requires-python = ">=3.10"
 license = "Apache-2.0"
+# CPython only, and now literally so. A stable ABI wheel cannot be loaded by
+# PyPy or by a free-threaded build, so no wheel is produced for either.
 classifiers = [
     "Programming Language :: Rust",
     "Programming Language :: Python :: Implementation :: CPython",
@@ -21,20 +27,20 @@ requires = ["maturin>=1.9.1,<2.0"]
 build-backend = "maturin"
 
 [tool.maturin]
+# abi3-py310 is deliberately not listed here. maturin has no abi3 key and
+# reads the target from the pyo3 cargo feature, which is declared in
+# vdb-core/Cargo.toml instead, so that cargo build, cargo clippy and cargo
+# test compile the configuration the wheel ships rather than a different one.
 features = ["pyo3/extension-module"]
 python-source = "src"
 manifest-path = "vdb-core/Cargo.toml"
 module-name = "zeusdb_vector_database" # <-- matches Rust crate name
 bindings = "pyo3" # <-- explicitly state to maturin that it's a PyO3 project
-# vendor/** must ship in the sdist, or an sdist build cannot resolve the
-# patched hnsw_rs path dependency declared in vdb-core/Cargo.toml.
 include = [
     "LICENSE",
     "NOTICE",
     "README.md",
     "src/**",
-    { path = "vendor/**/*", format = "sdist" },
-    { path = "LICENSES/*", format = "sdist" },
 ]
 
 [project.optional-dependencies]

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/src/zeusdb_vector_database/__init__.py

@@ -5,7 +5,7 @@
 """
 ZeusDB Vector Database Module
 """
-__version__ = "0.5.0"
+__version__ = "0.7.0"
 
 # STEP 1: Configure logging FIRST, before importing anything that uses Rust.
 from .logging_config import _auto_configure_logging

{zeusdb_vector_database-0.5.0 → zeusdb_vector_database-0.7.0}/src/zeusdb_vector_database/vector_database.py

@@ -69,6 +69,71 @@ class VectorDatabase:
             return 16
         return 32 if expected_size > 25_000 else 16
 
+    @staticmethod
+    def _warn_if_selection_disabled(m: Any, ef_construction: Any) -> None:
+        """Warn where ef_construction switches the neighbour selection heuristic off.
+
+        `select_neighbours` in the vendored crate opens with
+        `if candidates.len() <= nb_neighbours_asked`, and with
+        `extend_candidates` false it copies every candidate into the neighbour
+        list and returns from inside that branch. The budget is `2 * m` at
+        layer zero and `m` above it, both set at `hnsw.rs:1270` to `:1278`.
+
+        The candidate list is exactly `ef_construction` long once the index
+        holds more records than that, which is a property of `search_layer`
+        rather than an assumption. Insertion passes it no filter, so every
+        point it pushes into the candidate heap it also pushes into the result
+        heap, and the result heap is trimmed only above `ef`. While the result
+        heap is short it therefore holds every visited point, which makes the
+        nearest unexplored candidate a member of it and leaves the early
+        return unable to fire. The traversal
+        therefore stops either at `ef` results or with the whole reachable
+        component visited. Measured at m 8 over 2,000 records, mean layer zero
+        out-degree is 16.000 at ef_construction 16 with every node at the cap,
+        and 7.170 at ef_construction 17 with 6.5 percent of nodes at it, so the
+        flip lands exactly on `2 * m` and needs no margin. That measurement is
+        `neighbour_selection_threshold_is_twice_m` in the Rust guard tests.
+
+        The threshold is therefore `ef_construction <= 2 * m` and the warning
+        carries no slack. Slack would claim something the source does not say
+        and would fire on m 64 with the default ef_construction, which measured
+        higher recall on real embeddings than the default m of 32 does.
+
+        A pair the Rust layer is going to reject returns without warning, so
+        that an invalid m or ef_construction produces its real validation error
+        and nothing else. That covers a non-integer of either, an
+        ef_construction below 1, and an m outside the 2 to 256 the graph
+        accepts.
+        """
+        integral = (
+            isinstance(m, int) and not isinstance(m, bool)
+            and isinstance(ef_construction, int)
+            and not isinstance(ef_construction, bool)
+        )
+        if not integral or ef_construction < 1 or not 2 <= m <= 256:
+            return
+
+        budget = 2 * m
+        if ef_construction > budget:
+            return
+
+        # The largest m that leaves the heuristic running at this
+        # ef_construction. Below the floor of 2 there is no such m, so the
+        # message offers the one remedy that exists.
+        largest_m = (ef_construction - 1) // 2
+        remedy = f"Raise ef_construction above {budget}"
+        if largest_m >= 2:
+            remedy += f", or lower m to {largest_m} or below"
+
+        import warnings
+        warnings.warn(
+            f"ef_construction={ef_construction} is not greater than 2*m={budget}, "
+            f"so the neighbour selection heuristic does not run. Layer zero "
+            f"insertion keeps every candidate the construction search returns, in "
+            f"distance order, and prunes none of them. {remedy}, to run it.",
+            UserWarning,
+            stacklevel=3
+        )
 
     def __init__(self):
         """Initialize the vector database factory."""
@@ -93,7 +158,17 @@ class VectorDatabase:
                   expected_size honestly or set m directly. The floor is 2
                   because at 1 the layer scale is infinite and the graph is
                   degenerate.
-                - ef_construction (int): Construction candidate list size (default: 200)
+                - ef_construction (int): Width of the candidate search each
+                  insertion runs (default: 200). It sets the supply the
+                  neighbour selection draws from, and m sets how much of it is
+                  kept, so it changes neither search latency nor the size of
+                  the finished index. Build time is linear in it above 100: at
+                  50,000 records of 1,536 dimensions, 62.7s, 131.2s, 262.8s and
+                  464.9s at 100, 200, 400 and 800. Recall at 10 plateaus at 100
+                  on SIFT and at 200 on OpenAI embeddings, and on GloVe raising
+                  ef_search on the graph already built buys more than raising
+                  this. The neighbour selection heuristic runs only while this
+                  is above 2 * m, and create() warns where it is not.
                 - expected_size (int): Expected number of vectors (default: 10000,
                   max: 100,000,000). Also selects the default m, see above. It is
                   a capacity hint rather than a limit, and an index that grows
@@ -121,10 +196,11 @@ class VectorDatabase:
             Under 'quantized_only' the records collected for training are held at
             full width only until training completes, then released. get_stats()
             reports the codes, the raw vectors, the codebook, the centroid
-            distance table and the graph, and sums them under total_memory_mb.
-            That sum is what the index holds rather than what the process holds,
-            since the id maps, the metadata map and the allocator's own headers
-            sit outside it at roughly 1,500 bytes a record. Recall falls sharply,
+            distance table, the graph and the hash tables that find a record,
+            and sums them under total_memory_mb. That sum is what the index
+            asked the allocator for rather than what the process holds, since
+            the allocator's own headers and its rounding sit outside it at
+            between 1,200 and 2,100 bytes a record. Recall falls sharply,
             not slightly. Only 'quantized_with_raw' can recover it, by reranking
             against the raw vectors it keeps. Training triggers automatically on
             the first .add() call that reaches the training_size threshold, and
@@ -195,6 +271,10 @@ class VectorDatabase:
             kwargs.setdefault("expected_size", 10000)
             # After expected_size, since the graph degree is derived from it.
             kwargs.setdefault("m", self._default_m(kwargs["expected_size"]))
+            # After both, since the pair is what decides it. No default
+            # combination reaches it: the default m is 16 or 32, so the budget
+            # is 32 or 64, and the default ef_construction of 200 clears both.
+            self._warn_if_selection_disabled(kwargs["m"], kwargs["ef_construction"])
 
         # Validate and process quantization config
         if quantization_config is not None:
@@ -359,7 +439,8 @@ class VectorDatabase:
                 "holds 0.69 times an unquantized index at 10,000 records and 0.59 "
                 "times at 100,000. "
                 "get_stats() reports the memory the codes, the raw vectors, the codebook, "
-                "the centroid distance table and the graph hold once records are loaded, "
+                "the centroid distance table, the graph and the index bookkeeping hold "
+                "once records are loaded, "
                 "and sums them under total_memory_mb. This mode is required for rerank "
                 "and for exact vector reconstruction.",
                 UserWarning,