numpy-vector-store 0.5.0__tar.gz → 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/CHANGELOG.md +142 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/MIGRATION.md +16 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/PKG-INFO +110 -16
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/README.md +109 -15
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/ROADMAP.md +68 -13
- numpy_vector_store-0.6.0/benchmarks/README.md +91 -0
- numpy_vector_store-0.6.0/benchmarks/__init__.py +1 -0
- numpy_vector_store-0.6.0/benchmarks/benchmark.py +335 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/src/numpy_vector_store/__init__.py +1 -1
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/src/numpy_vector_store/vector_store.py +122 -46
- numpy_vector_store-0.6.0/tests/fixtures/README.md +36 -0
- numpy_vector_store-0.6.0/tests/fixtures/vector-store-0.4.0-format-v1.npz +0 -0
- numpy_vector_store-0.6.0/tests/test_benchmark.py +119 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/tests/test_vector_store.py +363 -27
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/.github/FUNDING.yml +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/.github/workflows/checks.yml +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/.github/workflows/publish-pypi.yml +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/.github/workflows/publish-testpypi.yml +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/.gitignore +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/LICENSE +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/justfile +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/pyproject.toml +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/src/numpy_vector_store/py.typed +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/tests/__init__.py +0 -0
- {numpy_vector_store-0.5.0 → numpy_vector_store-0.6.0}/uv.lock +0 -0
|
@@ -4,6 +4,148 @@ This changelog records user-visible changes to NumPy Vector Store. Earlier
|
|
|
4
4
|
release notes remain available on the
|
|
5
5
|
[GitHub releases page](https://github.com/tvanreenen/numpy-vector-store/releases).
|
|
6
6
|
|
|
7
|
+
## 0.6.0 - 2026-08-21
|
|
8
|
+
|
|
9
|
+
This release makes the existing API's input and failure contracts predictable
|
|
10
|
+
before 1.0. It closes cases where the same invalid call could behave
|
|
11
|
+
differently on an empty and populated store, where Python booleans could act as
|
|
12
|
+
row indexes or counts, and where loosely coerced configuration could change
|
|
13
|
+
meaning after persistence. It also turns the project's performance claims and
|
|
14
|
+
format-version-1 compatibility promise into reproducible regression evidence.
|
|
15
|
+
|
|
16
|
+
Valid, documented 0.5 usage continues to work unchanged. Version 0.6 does not
|
|
17
|
+
add a new feature family, archive format, runtime dependency, or public
|
|
18
|
+
exception hierarchy.
|
|
19
|
+
|
|
20
|
+
### API at a glance
|
|
21
|
+
|
|
22
|
+
The complete workflow remains centered on `VectorStore` and `VectorHit`:
|
|
23
|
+
|
|
24
|
+
```python
|
|
25
|
+
store = VectorStore(dimensions=1536, normalize=True)
|
|
26
|
+
store.add(vectors, metadata)
|
|
27
|
+
|
|
28
|
+
hits = store.cosine_search(query, top_k=10)
|
|
29
|
+
hits = store.dot_search(query, top_k=10, min_value=0.5)
|
|
30
|
+
hits = store.euclidean_search(query, top_k=10, within_rows=row_indexes)
|
|
31
|
+
|
|
32
|
+
row = store.get(0)
|
|
33
|
+
store.clear()
|
|
34
|
+
|
|
35
|
+
store.save("vectors.npz")
|
|
36
|
+
store.save()
|
|
37
|
+
|
|
38
|
+
loaded = VectorStore.open("vectors.npz")
|
|
39
|
+
loaded.reload()
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The ownership, amortized ingestion, deterministic tie ordering, explicit
|
|
43
|
+
persistence lifecycle, and trusted-local-file boundary established in 0.5 are
|
|
44
|
+
unchanged.
|
|
45
|
+
|
|
46
|
+
### Predictable scalar inputs
|
|
47
|
+
|
|
48
|
+
- Accept Python integers, NumPy integer scalars, and other values implementing
|
|
49
|
+
the standard integer-index protocol for `dimensions`, `top_k`, and
|
|
50
|
+
`get(index)`, then convert them to a Python `int`.
|
|
51
|
+
- Reject Python and NumPy booleans where integer semantics are required.
|
|
52
|
+
`top_k=True` no longer means one, and `get(True)` no longer invokes NumPy
|
|
53
|
+
boolean indexing.
|
|
54
|
+
- Continue requiring positive `dimensions` and `top_k` values. A valid integer
|
|
55
|
+
outside the stored row range still makes `get()` return `None`.
|
|
56
|
+
- Accept Python and NumPy booleans for `normalize`, then retain a canonical
|
|
57
|
+
Python `bool` in memory and persistence.
|
|
58
|
+
- Accept finite Python and NumPy integer or floating-point scalars for
|
|
59
|
+
`min_value` and `max_value`. Reject booleans, strings, complex values,
|
|
60
|
+
arrays, and non-finite thresholds before search.
|
|
61
|
+
|
|
62
|
+
An inappropriate scalar type now raises `TypeError`, while a supported type
|
|
63
|
+
with an invalid value raises `ValueError`. Row bounds continue using
|
|
64
|
+
`IndexError` where row selectors require an existing row. Exact error wording
|
|
65
|
+
remains explanatory rather than a compatibility guarantee.
|
|
66
|
+
|
|
67
|
+
### Search checks independent of store state
|
|
68
|
+
|
|
69
|
+
- Validate `within_rows` before returning an empty result. It must be a
|
|
70
|
+
one-dimensional sequence of unique integer row indexes, without booleans,
|
|
71
|
+
and every index must be within the current store.
|
|
72
|
+
- Reject duplicate filtered indexes instead of returning the same stored row
|
|
73
|
+
more than once.
|
|
74
|
+
- Apply the same zero-query rules before the empty-store shortcut. Cosine
|
|
75
|
+
search always requires a non-zero query. Dot-product and Euclidean searches
|
|
76
|
+
require one when the store normalizes vectors, while raw stores continue to
|
|
77
|
+
accept a zero query for those two metrics.
|
|
78
|
+
- Keep selector validation vectorized for native NumPy integer arrays and avoid
|
|
79
|
+
changing process-global warning filters while validating Python sequences.
|
|
80
|
+
|
|
81
|
+
These changes affect preconditions, not ranking. Valid 0.5 searches retain the
|
|
82
|
+
same metric calculations, thresholds, deterministic tie handling, filtered-row
|
|
83
|
+
semantics, and `VectorHit` results.
|
|
84
|
+
|
|
85
|
+
### Persistence inputs and compatibility boundaries
|
|
86
|
+
|
|
87
|
+
- Accept strings and string-returning path-like objects for `open()`,
|
|
88
|
+
`save(path)`, and bound archive paths. Reject an explicitly empty path with
|
|
89
|
+
`ValueError` and an inappropriate path type, including bytes paths, with
|
|
90
|
+
`TypeError`.
|
|
91
|
+
- Keep `save()` with no argument distinct from `save("")`: omission means
|
|
92
|
+
"save to the current binding," while an empty string is an invalid path.
|
|
93
|
+
- Preserve extension resolution. A path without `.npz` still resolves to the
|
|
94
|
+
same archive for saving, opening, and reloading.
|
|
95
|
+
- Preserve owning exceptions at the persistence boundary. Filesystem failures
|
|
96
|
+
use the relevant `OSError` subclass, malformed archive schemas use
|
|
97
|
+
`ValueError`, and NumPy, pickle, or application metadata-loading failures are
|
|
98
|
+
not hidden inside a package-specific wrapper.
|
|
99
|
+
|
|
100
|
+
Archive format version 1 is unchanged. The compatibility suite now includes an
|
|
101
|
+
archive produced by the published 0.4.0 package on Python 3.11 with NumPy
|
|
102
|
+
1.23.2 and verifies that the current reader recovers its configuration, vectors,
|
|
103
|
+
and metadata. This is a forward-reading promise for the self-describing format,
|
|
104
|
+
not support for older unversioned two-array archives or a promise that an old
|
|
105
|
+
package can read an arbitrary future format. Persistence remains a trusted-file
|
|
106
|
+
feature because opaque metadata uses NumPy's pickle-backed object arrays.
|
|
107
|
+
|
|
108
|
+
### Reproducible performance evidence
|
|
109
|
+
|
|
110
|
+
- Add repository benchmark commands for public exact search and ingestion.
|
|
111
|
+
Each command records the workload, seeded input digests, complete timing
|
|
112
|
+
samples, median, Python and NumPy versions, Git state, timer, thread-related
|
|
113
|
+
environment variables, and NumPy build configuration as JSON.
|
|
114
|
+
- Measure prepared inputs and documented public operations rather than random
|
|
115
|
+
input generation. Search records complete query batches; ingestion records a
|
|
116
|
+
fresh store and every `add()` call.
|
|
117
|
+
- Add structural regression checks for geometric capacity reuse, partial
|
|
118
|
+
top-k selection, and unfiltered access to the stored vector matrix without a
|
|
119
|
+
preliminary full-matrix copy.
|
|
120
|
+
- Keep wall-clock limits out of shared CI, where runner load would make failures
|
|
121
|
+
noisy and machine-specific.
|
|
122
|
+
|
|
123
|
+
The README now reports one aligned grid at 1,000, 10,000, and 100,000 rows for
|
|
124
|
+
384, 1,536, and 3,072 dimensions, measured on a 24 GB Apple M4 Mac mini. It
|
|
125
|
+
also makes the intended scale explicit: this project is for small-to-medium,
|
|
126
|
+
in-process exact search. The 100,000-row measurements are an upper reference,
|
|
127
|
+
not a target for indefinite scaling; routinely million-row workloads generally
|
|
128
|
+
need an indexed or service-backed system.
|
|
129
|
+
|
|
130
|
+
### Runtime compatibility and upgrade notes
|
|
131
|
+
|
|
132
|
+
- Continue supporting Python 3.11 through 3.14 and NumPy 1.23.2 or newer.
|
|
133
|
+
- Continue exercising every supported Python version in CI, including a
|
|
134
|
+
dedicated Python 3.11 job with the minimum NumPy version.
|
|
135
|
+
- Keep the runtime wheel limited to the package. Benchmark tooling and its
|
|
136
|
+
guide are included in the source distribution without adding runtime
|
|
137
|
+
dependencies.
|
|
138
|
+
- Require no archive conversion when upgrading from 0.5; format version 1 is
|
|
139
|
+
still the only supported archive format.
|
|
140
|
+
|
|
141
|
+
Applications using documented 0.5 input types should not need code changes.
|
|
142
|
+
Review call sites that pass booleans where integers are expected, fractional
|
|
143
|
+
result counts, string or boolean thresholds, duplicate or malformed
|
|
144
|
+
`within_rows` values, zero queries that were only attempted against empty
|
|
145
|
+
stores, or explicitly empty persistence paths. Those accidentally accepted or
|
|
146
|
+
state-dependent cases now fail at the public boundary with consistent standard
|
|
147
|
+
exceptions.
|
|
148
|
+
|
|
7
149
|
## 0.5.0 - 2026-08-21
|
|
8
150
|
|
|
9
151
|
This release gives `VectorStore` clear ownership of its configuration and row
|
|
@@ -177,6 +177,11 @@ contents, loads its rows, and binds its path. Applications no longer need to
|
|
|
177
177
|
keep archive configuration separately or risk loading the same vectors with
|
|
178
178
|
different semantics.
|
|
179
179
|
|
|
180
|
+
Paths may be strings or string-valued `os.PathLike` objects such as
|
|
181
|
+
`pathlib.Path`. Empty paths raise `ValueError`, while bytes and other non-path
|
|
182
|
+
values raise `TypeError`. A `None` path is meaningful only for `save()`, where
|
|
183
|
+
it reuses an existing binding.
|
|
184
|
+
|
|
180
185
|
The generic parameter still describes application metadata. It can be kept
|
|
181
186
|
when useful:
|
|
182
187
|
|
|
@@ -284,6 +289,17 @@ configuration safely.
|
|
|
284
289
|
Metadata is stored in a pickle-backed NumPy object array. Archives remain
|
|
285
290
|
trusted input and must not be opened from untrusted or unverifiable sources.
|
|
286
291
|
|
|
292
|
+
Current releases retain forward-reading support for self-describing
|
|
293
|
+
format-version-1 archives written by 0.4. The test suite keeps a binary fixture
|
|
294
|
+
generated by the published 0.4.0 package on Python 3.11 and NumPy 1.23.2. This
|
|
295
|
+
does not restore the unversioned reader or promise that an older release can
|
|
296
|
+
read an arbitrary future format.
|
|
297
|
+
|
|
298
|
+
Read and write failures retain the exception type owned by their source:
|
|
299
|
+
filesystem `OSError` subclasses, schema `ValueError`, and NumPy, pickle, or
|
|
300
|
+
application metadata-loading errors are not collapsed into a package-specific
|
|
301
|
+
exception.
|
|
302
|
+
|
|
287
303
|
Saves use same-directory temporary files and atomic replacement, but the
|
|
288
304
|
library does not add file locking, coordinate concurrent writers, or promise
|
|
289
305
|
power-loss durability across every operating system and filesystem.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: numpy-vector-store
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.6.0
|
|
4
4
|
Summary: A fast, lightweight, and zero-setup in-memory vector store powered by NumPy
|
|
5
5
|
Project-URL: Homepage, https://github.com/tvanreenen/numpy-vector-store
|
|
6
6
|
Project-URL: Repository, https://github.com/tvanreenen/numpy-vector-store
|
|
@@ -39,18 +39,59 @@ offers a simple alternative to heavyweight vector databases when you do not need
|
|
|
39
39
|
network services, indexing infrastructure, ingestion pipelines, or domain-specific
|
|
40
40
|
metadata filtering.
|
|
41
41
|
|
|
42
|
-
##
|
|
42
|
+
## Performance
|
|
43
43
|
|
|
44
|
-
|
|
45
|
-
|
|
44
|
+
`VectorStore` performs exact search over every selected row. The following
|
|
45
|
+
measurements are reference points, not latency guarantees:
|
|
46
46
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
| **Sentence Transformers** | 384 | 1K vectors<br/>1.5MB | 10K vectors<br/>15MB | 100K vectors<br/>147MB | 500K vectors<br/>732MB |
|
|
50
|
-
| **OpenAI Small** | 1536 | 500 vectors<br/>3MB | 5K vectors<br/>29MB | 25K vectors<br/>147MB | 100K vectors<br/>586MB |
|
|
51
|
-
| **OpenAI Large** | 3072 | 200 vectors<br/>2MB | 2.5K vectors<br/>29MB | 5K vectors<br/>59MB | 25K vectors<br/>293MB |
|
|
47
|
+
Each cell shows the median time per cosine query followed by the stored vector
|
|
48
|
+
matrix size:
|
|
52
49
|
|
|
53
|
-
|
|
50
|
+
| Rows | 384 dimensions | 1,536 dimensions | 3,072 dimensions |
|
|
51
|
+
|---:|---:|---:|---:|
|
|
52
|
+
| 1,000 | 0.032 ms · 1.5 MB | 0.049 ms · 6.1 MB | 0.088 ms · 12.3 MB |
|
|
53
|
+
| 10,000 | 0.286 ms · 15.4 MB | 1.065 ms · 61.4 MB | 2.122 ms · 122.9 MB |
|
|
54
|
+
| 100,000 | 2.983 ms · 153.6 MB | 9.877 ms · 614.4 MB | 20.509 ms · 1.23 GB |
|
|
55
|
+
|
|
56
|
+
These benchmarks intentionally stop at 100,000 rows. NumPy Vector Store is
|
|
57
|
+
designed for small-to-medium, in-process exact search; 100,000 rows is an upper
|
|
58
|
+
reference, not a promised limit or a target for continued scaling. The practical
|
|
59
|
+
boundary depends on vector dimensions, metadata, available memory, and latency
|
|
60
|
+
requirements. Workloads that routinely reach millions of vectors generally
|
|
61
|
+
need an indexed or service-backed system.
|
|
62
|
+
|
|
63
|
+
These are unfiltered `top_k=10` searches on a normalized store. Each row divides
|
|
64
|
+
the median duration of seven measured 20-query trials by 20, after two discarded
|
|
65
|
+
warmup trials. The vector matrix size excludes metadata, temporary search
|
|
66
|
+
arrays, and Python process overhead.
|
|
67
|
+
|
|
68
|
+
The measurements were taken from commit `4b23810` on a 24 GB Apple M4 Mac mini
|
|
69
|
+
with macOS 26.6.1, CPython 3.13.5, NumPy 2.3.3, and Accelerate BLAS. Hardware,
|
|
70
|
+
operating system activity, Python and NumPy versions, BLAS implementation, and
|
|
71
|
+
thread settings can all change the result.
|
|
72
|
+
|
|
73
|
+
The repository includes benchmark commands that emit the inputs, environment,
|
|
74
|
+
raw samples, and median as JSON:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
uv run python benchmarks/benchmark.py search \
|
|
78
|
+
--rows 10000 --dimensions 384 --queries 20 --top-k 10 \
|
|
79
|
+
--warmup 2 --repetitions 7 > /tmp/nvs-search.json
|
|
80
|
+
|
|
81
|
+
uv run python benchmarks/benchmark.py ingest \
|
|
82
|
+
--rows 10000 --dimensions 384 --batch-size 1 \
|
|
83
|
+
--warmup 2 --repetitions 7 > /tmp/nvs-ingest.json
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
For the same 10,000-by-384 prepared input, repeated single-row ingestion had a
|
|
87
|
+
median of 78.6 ms, or about 127,000 rows per second. Supplying 1,000 rows per
|
|
88
|
+
`add()` call had a median of 6.63 ms, or about 1.51 million rows per second.
|
|
89
|
+
Both measurements include construction of a fresh normalized store and every
|
|
90
|
+
`add()` call, but exclude input generation.
|
|
91
|
+
|
|
92
|
+
See [the benchmark guide](benchmarks/README.md) for the timed regions, complete
|
|
93
|
+
options, output fields, and interpretation notes. The project keeps structural
|
|
94
|
+
complexity checks in CI, but does not fail shared runners on wall-clock timing.
|
|
54
95
|
|
|
55
96
|
## Installation
|
|
56
97
|
|
|
@@ -93,6 +134,32 @@ Each payload can be a dict, dataclass, tuple, list, string, integer row ID, or
|
|
|
93
134
|
another Python object that fits your application. Tuple and list payloads remain
|
|
94
135
|
single row values rather than being interpreted as additional array dimensions.
|
|
95
136
|
|
|
137
|
+
## Scalar inputs and errors
|
|
138
|
+
|
|
139
|
+
`dimensions`, `top_k`, and the index passed to `get()` use integer semantics.
|
|
140
|
+
Python integers and NumPy integer scalars are accepted and converted to Python
|
|
141
|
+
`int`; booleans are rejected rather than being treated as zero or one.
|
|
142
|
+
`dimensions` and `top_k` must be greater than zero. A valid integer outside the
|
|
143
|
+
stored row range still makes `get()` return `None`.
|
|
144
|
+
|
|
145
|
+
`normalize` accepts Python and NumPy booleans and is stored as a Python `bool`.
|
|
146
|
+
Search thresholds such as `min_value` and `max_value` accept finite Python
|
|
147
|
+
integer or floating-point values and NumPy integer or floating scalars.
|
|
148
|
+
Booleans, strings, complex numbers, and arrays are not threshold scalars and are
|
|
149
|
+
rejected.
|
|
150
|
+
|
|
151
|
+
`within_rows` must be a one-dimensional sequence of unique integer row indexes.
|
|
152
|
+
Python and NumPy integers are accepted; booleans and non-integer values are not.
|
|
153
|
+
Malformed shapes and duplicate indexes raise `ValueError`, while an index
|
|
154
|
+
outside the current store raises `IndexError`. These checks still run when the
|
|
155
|
+
store is empty.
|
|
156
|
+
|
|
157
|
+
For these scalar inputs, an inappropriate type raises `TypeError` and a
|
|
158
|
+
supported type with an invalid value raises `ValueError`. Row selectors outside
|
|
159
|
+
the store raise `IndexError`, and filesystem operations continue to raise the
|
|
160
|
+
relevant `OSError` subclass. Error messages explain the failed argument, but
|
|
161
|
+
their exact wording is not a compatibility guarantee.
|
|
162
|
+
|
|
96
163
|
## State ownership
|
|
97
164
|
|
|
98
165
|
The store owns its configuration and row structure. `dimensions`, `normalize`,
|
|
@@ -169,6 +236,11 @@ search. Because cosine similarity is undefined for zero vectors,
|
|
|
169
236
|
`cosine_search` raises an error when its selected rows include one; use
|
|
170
237
|
`within_rows` to exclude zero rows when needed.
|
|
171
238
|
|
|
239
|
+
Cosine search also requires a non-zero query. Dot-product and Euclidean searches
|
|
240
|
+
require one when `normalize=True`, because they normalize the query before
|
|
241
|
+
comparison. With `normalize=False`, both methods accept a zero query. These
|
|
242
|
+
rules apply even when the store or `within_rows` selection is empty.
|
|
243
|
+
|
|
172
244
|
### Numerical inputs
|
|
173
245
|
|
|
174
246
|
Stored vectors use `float32` to keep the store compact. Vectors and queries must
|
|
@@ -243,6 +315,10 @@ rows = [
|
|
|
243
315
|
hits = store.cosine_search(query, top_k=10, within_rows=rows)
|
|
244
316
|
```
|
|
245
317
|
|
|
318
|
+
Each stored row may appear at most once in `within_rows`; duplicate indexes are
|
|
319
|
+
rejected rather than producing duplicate hits. An empty sequence returns no
|
|
320
|
+
hits, but it does not bypass validation of the query or other search arguments.
|
|
321
|
+
|
|
246
322
|
Searches without `within_rows` compute directly against the stored vector matrix
|
|
247
323
|
and do not make a full copy of it. A filtered search gathers the selected rows
|
|
248
324
|
into a temporary matrix, so its additional memory use scales with the number of
|
|
@@ -328,6 +404,12 @@ current in-memory vectors and metadata unchanged.
|
|
|
328
404
|
The `.npz` suffix may be omitted. An extensionless path such as `"vectors"` is
|
|
329
405
|
resolved to `"vectors.npz"` for saving, opening, and reloading.
|
|
330
406
|
|
|
407
|
+
Persistence paths accept strings and string-valued `os.PathLike` objects such
|
|
408
|
+
as `pathlib.Path`. Passing an empty path raises `ValueError`; passing `None`,
|
|
409
|
+
bytes, or another non-path value to `open()` raises `TypeError`. For `save()`,
|
|
410
|
+
`None` retains its established meaning: reuse the current binding, or raise
|
|
411
|
+
`ValueError` if the store has not been bound yet.
|
|
412
|
+
|
|
331
413
|
Raw-vector configuration is also restored from the archive:
|
|
332
414
|
|
|
333
415
|
```python
|
|
@@ -343,15 +425,27 @@ assert loaded.normalize is False
|
|
|
343
425
|
```
|
|
344
426
|
|
|
345
427
|
Archives written by 0.4 use format version 1 and contain `format_version`,
|
|
346
|
-
`dimensions`, `normalize`, `vectors`, and `metadata`. The
|
|
347
|
-
|
|
348
|
-
|
|
428
|
+
`dimensions`, `normalize`, `vectors`, and `metadata`. The compatibility suite
|
|
429
|
+
opens an archive generated by the published 0.4.0 package on the oldest
|
|
430
|
+
supported Python and NumPy versions. The stored configuration prevents an
|
|
431
|
+
archive from being loaded with different dimensions or normalization semantics.
|
|
432
|
+
|
|
433
|
+
This is a forward-reading promise for self-describing format-version-1
|
|
434
|
+
archives: current releases can open the recorded 0.4 fixture. It does not
|
|
435
|
+
restore the unversioned two-array reader or guarantee that an older package can
|
|
436
|
+
open files written by an arbitrary future archive format.
|
|
349
437
|
|
|
350
438
|
Opening and reloading validate the complete schema, array dtypes and shapes, row
|
|
351
439
|
counts, finite vector values, and zero-norm behavior before changing in-memory
|
|
352
440
|
state. Opaque metadata values remain individual row payloads across persistence
|
|
353
441
|
round trips.
|
|
354
442
|
|
|
443
|
+
Persistence failures keep their owning exception type. Filesystem failures use
|
|
444
|
+
the relevant `OSError` subclass, malformed schemas use `ValueError`, and NumPy,
|
|
445
|
+
pickle, or application metadata-loading exceptions are not wrapped in a package
|
|
446
|
+
exception. Exact message text remains explanatory rather than a compatibility
|
|
447
|
+
guarantee.
|
|
448
|
+
|
|
355
449
|
Each save writes a uniquely named temporary archive in the destination
|
|
356
450
|
directory, closes it, and then replaces the destination with `os.replace`.
|
|
357
451
|
Readers opening the destination path therefore see either the previous complete
|
|
@@ -364,7 +458,7 @@ writers can still replace one another, and the library does not promise that a
|
|
|
364
458
|
successful save has reached durable hardware storage across every operating
|
|
365
459
|
system or power failure.
|
|
366
460
|
|
|
367
|
-
Version 0.
|
|
461
|
+
Version 0.6 reads only self-describing format version 1 archives. Unversioned
|
|
368
462
|
archives containing only `vectors` and `metadata` cannot be opened because they
|
|
369
463
|
do not record dimensions or normalization semantics. Recreate those archives
|
|
370
464
|
from source data, or convert them with NumPy Vector Store 0.4 before upgrading.
|
|
@@ -384,10 +478,10 @@ the API stabilizes. Changes are documented in the [changelog](CHANGELOG.md) and
|
|
|
384
478
|
GitHub release notes. Deprecated APIs will keep warning for at least one point
|
|
385
479
|
release before removal.
|
|
386
480
|
|
|
387
|
-
Version 0.
|
|
481
|
+
Version 0.6 supports Python 3.11 through 3.14 and NumPy 1.23.2 or newer. These
|
|
388
482
|
versions are listed in the package metadata and exercised in CI, including a
|
|
389
483
|
dedicated check against the minimum NumPy version. Python 3.10 remains supported
|
|
390
|
-
by the 0.3 release series but is not supported by 0.4 or
|
|
484
|
+
by the 0.3 release series but is not supported by 0.4 or later.
|
|
391
485
|
|
|
392
486
|
The project generally retains stable CPython versions until their upstream
|
|
393
487
|
end-of-life, adds new versions after its dependencies and CI support them, and
|
|
@@ -16,18 +16,59 @@ offers a simple alternative to heavyweight vector databases when you do not need
|
|
|
16
16
|
network services, indexing infrastructure, ingestion pipelines, or domain-specific
|
|
17
17
|
metadata filtering.
|
|
18
18
|
|
|
19
|
-
##
|
|
19
|
+
## Performance
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
`VectorStore` performs exact search over every selected row. The following
|
|
22
|
+
measurements are reference points, not latency guarantees:
|
|
23
23
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
| **Sentence Transformers** | 384 | 1K vectors<br/>1.5MB | 10K vectors<br/>15MB | 100K vectors<br/>147MB | 500K vectors<br/>732MB |
|
|
27
|
-
| **OpenAI Small** | 1536 | 500 vectors<br/>3MB | 5K vectors<br/>29MB | 25K vectors<br/>147MB | 100K vectors<br/>586MB |
|
|
28
|
-
| **OpenAI Large** | 3072 | 200 vectors<br/>2MB | 2.5K vectors<br/>29MB | 5K vectors<br/>59MB | 25K vectors<br/>293MB |
|
|
24
|
+
Each cell shows the median time per cosine query followed by the stored vector
|
|
25
|
+
matrix size:
|
|
29
26
|
|
|
30
|
-
|
|
27
|
+
| Rows | 384 dimensions | 1,536 dimensions | 3,072 dimensions |
|
|
28
|
+
|---:|---:|---:|---:|
|
|
29
|
+
| 1,000 | 0.032 ms · 1.5 MB | 0.049 ms · 6.1 MB | 0.088 ms · 12.3 MB |
|
|
30
|
+
| 10,000 | 0.286 ms · 15.4 MB | 1.065 ms · 61.4 MB | 2.122 ms · 122.9 MB |
|
|
31
|
+
| 100,000 | 2.983 ms · 153.6 MB | 9.877 ms · 614.4 MB | 20.509 ms · 1.23 GB |
|
|
32
|
+
|
|
33
|
+
These benchmarks intentionally stop at 100,000 rows. NumPy Vector Store is
|
|
34
|
+
designed for small-to-medium, in-process exact search; 100,000 rows is an upper
|
|
35
|
+
reference, not a promised limit or a target for continued scaling. The practical
|
|
36
|
+
boundary depends on vector dimensions, metadata, available memory, and latency
|
|
37
|
+
requirements. Workloads that routinely reach millions of vectors generally
|
|
38
|
+
need an indexed or service-backed system.
|
|
39
|
+
|
|
40
|
+
These are unfiltered `top_k=10` searches on a normalized store. Each row divides
|
|
41
|
+
the median duration of seven measured 20-query trials by 20, after two discarded
|
|
42
|
+
warmup trials. The vector matrix size excludes metadata, temporary search
|
|
43
|
+
arrays, and Python process overhead.
|
|
44
|
+
|
|
45
|
+
The measurements were taken from commit `4b23810` on a 24 GB Apple M4 Mac mini
|
|
46
|
+
with macOS 26.6.1, CPython 3.13.5, NumPy 2.3.3, and Accelerate BLAS. Hardware,
|
|
47
|
+
operating system activity, Python and NumPy versions, BLAS implementation, and
|
|
48
|
+
thread settings can all change the result.
|
|
49
|
+
|
|
50
|
+
The repository includes benchmark commands that emit the inputs, environment,
|
|
51
|
+
raw samples, and median as JSON:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
uv run python benchmarks/benchmark.py search \
|
|
55
|
+
--rows 10000 --dimensions 384 --queries 20 --top-k 10 \
|
|
56
|
+
--warmup 2 --repetitions 7 > /tmp/nvs-search.json
|
|
57
|
+
|
|
58
|
+
uv run python benchmarks/benchmark.py ingest \
|
|
59
|
+
--rows 10000 --dimensions 384 --batch-size 1 \
|
|
60
|
+
--warmup 2 --repetitions 7 > /tmp/nvs-ingest.json
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
For the same 10,000-by-384 prepared input, repeated single-row ingestion had a
|
|
64
|
+
median of 78.6 ms, or about 127,000 rows per second. Supplying 1,000 rows per
|
|
65
|
+
`add()` call had a median of 6.63 ms, or about 1.51 million rows per second.
|
|
66
|
+
Both measurements include construction of a fresh normalized store and every
|
|
67
|
+
`add()` call, but exclude input generation.
|
|
68
|
+
|
|
69
|
+
See [the benchmark guide](benchmarks/README.md) for the timed regions, complete
|
|
70
|
+
options, output fields, and interpretation notes. The project keeps structural
|
|
71
|
+
complexity checks in CI, but does not fail shared runners on wall-clock timing.
|
|
31
72
|
|
|
32
73
|
## Installation
|
|
33
74
|
|
|
@@ -70,6 +111,32 @@ Each payload can be a dict, dataclass, tuple, list, string, integer row ID, or
|
|
|
70
111
|
another Python object that fits your application. Tuple and list payloads remain
|
|
71
112
|
single row values rather than being interpreted as additional array dimensions.
|
|
72
113
|
|
|
114
|
+
## Scalar inputs and errors
|
|
115
|
+
|
|
116
|
+
`dimensions`, `top_k`, and the index passed to `get()` use integer semantics.
|
|
117
|
+
Python integers and NumPy integer scalars are accepted and converted to Python
|
|
118
|
+
`int`; booleans are rejected rather than being treated as zero or one.
|
|
119
|
+
`dimensions` and `top_k` must be greater than zero. A valid integer outside the
|
|
120
|
+
stored row range still makes `get()` return `None`.
|
|
121
|
+
|
|
122
|
+
`normalize` accepts Python and NumPy booleans and is stored as a Python `bool`.
|
|
123
|
+
Search thresholds such as `min_value` and `max_value` accept finite Python
|
|
124
|
+
integer or floating-point values and NumPy integer or floating scalars.
|
|
125
|
+
Booleans, strings, complex numbers, and arrays are not threshold scalars and are
|
|
126
|
+
rejected.
|
|
127
|
+
|
|
128
|
+
`within_rows` must be a one-dimensional sequence of unique integer row indexes.
|
|
129
|
+
Python and NumPy integers are accepted; booleans and non-integer values are not.
|
|
130
|
+
Malformed shapes and duplicate indexes raise `ValueError`, while an index
|
|
131
|
+
outside the current store raises `IndexError`. These checks still run when the
|
|
132
|
+
store is empty.
|
|
133
|
+
|
|
134
|
+
For these scalar inputs, an inappropriate type raises `TypeError` and a
|
|
135
|
+
supported type with an invalid value raises `ValueError`. Row selectors outside
|
|
136
|
+
the store raise `IndexError`, and filesystem operations continue to raise the
|
|
137
|
+
relevant `OSError` subclass. Error messages explain the failed argument, but
|
|
138
|
+
their exact wording is not a compatibility guarantee.
|
|
139
|
+
|
|
73
140
|
## State ownership
|
|
74
141
|
|
|
75
142
|
The store owns its configuration and row structure. `dimensions`, `normalize`,
|
|
@@ -146,6 +213,11 @@ search. Because cosine similarity is undefined for zero vectors,
|
|
|
146
213
|
`cosine_search` raises an error when its selected rows include one; use
|
|
147
214
|
`within_rows` to exclude zero rows when needed.
|
|
148
215
|
|
|
216
|
+
Cosine search also requires a non-zero query. Dot-product and Euclidean searches
|
|
217
|
+
require one when `normalize=True`, because they normalize the query before
|
|
218
|
+
comparison. With `normalize=False`, both methods accept a zero query. These
|
|
219
|
+
rules apply even when the store or `within_rows` selection is empty.
|
|
220
|
+
|
|
149
221
|
### Numerical inputs
|
|
150
222
|
|
|
151
223
|
Stored vectors use `float32` to keep the store compact. Vectors and queries must
|
|
@@ -220,6 +292,10 @@ rows = [
|
|
|
220
292
|
hits = store.cosine_search(query, top_k=10, within_rows=rows)
|
|
221
293
|
```
|
|
222
294
|
|
|
295
|
+
Each stored row may appear at most once in `within_rows`; duplicate indexes are
|
|
296
|
+
rejected rather than producing duplicate hits. An empty sequence returns no
|
|
297
|
+
hits, but it does not bypass validation of the query or other search arguments.
|
|
298
|
+
|
|
223
299
|
Searches without `within_rows` compute directly against the stored vector matrix
|
|
224
300
|
and do not make a full copy of it. A filtered search gathers the selected rows
|
|
225
301
|
into a temporary matrix, so its additional memory use scales with the number of
|
|
@@ -305,6 +381,12 @@ current in-memory vectors and metadata unchanged.
|
|
|
305
381
|
The `.npz` suffix may be omitted. An extensionless path such as `"vectors"` is
|
|
306
382
|
resolved to `"vectors.npz"` for saving, opening, and reloading.
|
|
307
383
|
|
|
384
|
+
Persistence paths accept strings and string-valued `os.PathLike` objects such
|
|
385
|
+
as `pathlib.Path`. Passing an empty path raises `ValueError`; passing `None`,
|
|
386
|
+
bytes, or another non-path value to `open()` raises `TypeError`. For `save()`,
|
|
387
|
+
`None` retains its established meaning: reuse the current binding, or raise
|
|
388
|
+
`ValueError` if the store has not been bound yet.
|
|
389
|
+
|
|
308
390
|
Raw-vector configuration is also restored from the archive:
|
|
309
391
|
|
|
310
392
|
```python
|
|
@@ -320,15 +402,27 @@ assert loaded.normalize is False
|
|
|
320
402
|
```
|
|
321
403
|
|
|
322
404
|
Archives written by 0.4 use format version 1 and contain `format_version`,
|
|
323
|
-
`dimensions`, `normalize`, `vectors`, and `metadata`. The
|
|
324
|
-
|
|
325
|
-
|
|
405
|
+
`dimensions`, `normalize`, `vectors`, and `metadata`. The compatibility suite
|
|
406
|
+
opens an archive generated by the published 0.4.0 package on the oldest
|
|
407
|
+
supported Python and NumPy versions. The stored configuration prevents an
|
|
408
|
+
archive from being loaded with different dimensions or normalization semantics.
|
|
409
|
+
|
|
410
|
+
This is a forward-reading promise for self-describing format-version-1
|
|
411
|
+
archives: current releases can open the recorded 0.4 fixture. It does not
|
|
412
|
+
restore the unversioned two-array reader or guarantee that an older package can
|
|
413
|
+
open files written by an arbitrary future archive format.
|
|
326
414
|
|
|
327
415
|
Opening and reloading validate the complete schema, array dtypes and shapes, row
|
|
328
416
|
counts, finite vector values, and zero-norm behavior before changing in-memory
|
|
329
417
|
state. Opaque metadata values remain individual row payloads across persistence
|
|
330
418
|
round trips.
|
|
331
419
|
|
|
420
|
+
Persistence failures keep their owning exception type. Filesystem failures use
|
|
421
|
+
the relevant `OSError` subclass, malformed schemas use `ValueError`, and NumPy,
|
|
422
|
+
pickle, or application metadata-loading exceptions are not wrapped in a package
|
|
423
|
+
exception. Exact message text remains explanatory rather than a compatibility
|
|
424
|
+
guarantee.
|
|
425
|
+
|
|
332
426
|
Each save writes a uniquely named temporary archive in the destination
|
|
333
427
|
directory, closes it, and then replaces the destination with `os.replace`.
|
|
334
428
|
Readers opening the destination path therefore see either the previous complete
|
|
@@ -341,7 +435,7 @@ writers can still replace one another, and the library does not promise that a
|
|
|
341
435
|
successful save has reached durable hardware storage across every operating
|
|
342
436
|
system or power failure.
|
|
343
437
|
|
|
344
|
-
Version 0.
|
|
438
|
+
Version 0.6 reads only self-describing format version 1 archives. Unversioned
|
|
345
439
|
archives containing only `vectors` and `metadata` cannot be opened because they
|
|
346
440
|
do not record dimensions or normalization semantics. Recreate those archives
|
|
347
441
|
from source data, or convert them with NumPy Vector Store 0.4 before upgrading.
|
|
@@ -361,10 +455,10 @@ the API stabilizes. Changes are documented in the [changelog](CHANGELOG.md) and
|
|
|
361
455
|
GitHub release notes. Deprecated APIs will keep warning for at least one point
|
|
362
456
|
release before removal.
|
|
363
457
|
|
|
364
|
-
Version 0.
|
|
458
|
+
Version 0.6 supports Python 3.11 through 3.14 and NumPy 1.23.2 or newer. These
|
|
365
459
|
versions are listed in the package metadata and exercised in CI, including a
|
|
366
460
|
dedicated check against the minimum NumPy version. Python 3.10 remains supported
|
|
367
|
-
by the 0.3 release series but is not supported by 0.4 or
|
|
461
|
+
by the 0.3 release series but is not supported by 0.4 or later.
|
|
368
462
|
|
|
369
463
|
The project generally retains stable CPython versions until their upstream
|
|
370
464
|
end-of-life, adds new versions after its dependencies and CI support them, and
|
|
@@ -36,10 +36,11 @@ compatibility contract and are exercised in CI. The project generally:
|
|
|
36
36
|
- Drops a Python version only in a minor release and calls out the change in
|
|
37
37
|
release notes.
|
|
38
38
|
|
|
39
|
-
The 0.3 series supports Python 3.10 through 3.14.
|
|
40
|
-
3.11 through 3.14, using the minor-release boundary to remove
|
|
41
|
-
of its upstream end-of-life in October 2026. NumPy 1.23.2 is
|
|
42
|
-
0.4 because it is the earliest NumPy release that
|
|
39
|
+
The 0.3 series supports Python 3.10 through 3.14. Versions 0.4 through 0.6
|
|
40
|
+
support Python 3.11 through 3.14, using the 0.4 minor-release boundary to remove
|
|
41
|
+
Python 3.10 ahead of its upstream end-of-life in October 2026. NumPy 1.23.2 is
|
|
42
|
+
the minimum for 0.4 and later because it is the earliest NumPy release that
|
|
43
|
+
supports Python 3.11.
|
|
43
44
|
|
|
44
45
|
Supported NumPy versions are also part of the runtime contract. The declared
|
|
45
46
|
minimum should be installable on the oldest supported Python version and should
|
|
@@ -393,17 +394,71 @@ State encapsulation and spare capacity do not change serialized data, so format
|
|
|
393
394
|
version 1 remains sufficient. Broader validation and exception consistency stay
|
|
394
395
|
in the 0.6 stabilization milestone.
|
|
395
396
|
|
|
396
|
-
## 0.6.0:
|
|
397
|
+
## 0.6.0: Predictable input contracts and regression guarantees
|
|
397
398
|
|
|
398
|
-
|
|
399
|
-
new feature family.
|
|
400
|
-
|
|
401
|
-
Planned direction:
|
|
399
|
+
Status: complete
|
|
402
400
|
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
-
|
|
401
|
+
Version 0.6 makes the existing API's accepted inputs, failures, performance
|
|
402
|
+
expectations, and versioned-archive compatibility explicit before 1.0. Valid
|
|
403
|
+
documented 0.5 calls remain supported. Inputs that were accepted only through
|
|
404
|
+
Python truth-value coercion, NumPy indexing side effects, or row-count shortcuts
|
|
405
|
+
may be rejected at this minor-release boundary.
|
|
406
|
+
|
|
407
|
+
### Scalar configuration, counts, and indexes
|
|
408
|
+
|
|
409
|
+
- Accept Python and NumPy integer scalars for `dimensions`, `top_k`, and
|
|
410
|
+
`get(index)`, then convert them to Python `int`.
|
|
411
|
+
- Reject Python and NumPy booleans where integer semantics are required. This
|
|
412
|
+
prevents `top_k=True` from meaning one and prevents `get(True)` from invoking
|
|
413
|
+
NumPy boolean indexing instead of retrieving one row.
|
|
414
|
+
- Require positive dimensions and result counts while keeping `get()`'s
|
|
415
|
+
established `None` result for a valid integer outside the stored row range.
|
|
416
|
+
- Accept Python and NumPy booleans for `normalize`, then keep a canonical Python
|
|
417
|
+
`bool` so configuration cannot change meaning across persistence.
|
|
418
|
+
- Accept finite Python and NumPy integer or floating scalar search thresholds,
|
|
419
|
+
excluding booleans, strings, complex numbers, and arrays.
|
|
420
|
+
|
|
421
|
+
Wrong argument types raise `TypeError`, supported types with invalid values
|
|
422
|
+
raise `ValueError`, and row selectors outside the store raise `IndexError`.
|
|
423
|
+
Exact error wording is explanatory rather than a compatibility contract.
|
|
424
|
+
|
|
425
|
+
### State-independent search preconditions
|
|
426
|
+
|
|
427
|
+
- Validate `within_rows` shape, integer values, uniqueness, and bounds before an
|
|
428
|
+
empty store can return no hits.
|
|
429
|
+
- Apply zero-query rules consistently: cosine always requires a non-zero query;
|
|
430
|
+
normalized dot and Euclidean searches also normalize their query; raw dot and
|
|
431
|
+
Euclidean searches continue accepting a zero query.
|
|
432
|
+
- Reject duplicate filtered row indexes rather than returning the same stored
|
|
433
|
+
row more than once.
|
|
434
|
+
|
|
435
|
+
### Versioned persistence compatibility
|
|
436
|
+
|
|
437
|
+
- Distinguish an omitted path from an explicitly empty or incorrectly typed
|
|
438
|
+
path while preserving normal filesystem exception types.
|
|
439
|
+
- Keep current releases able to open format-version-1 archives produced by 0.4
|
|
440
|
+
at the oldest supported Python and NumPy boundary.
|
|
441
|
+
- Preserve schema and deserialization failure context without wrapping every
|
|
442
|
+
NumPy, pickle, or application exception in a package-specific error.
|
|
443
|
+
|
|
444
|
+
This compatibility promise applies to self-describing format-version-1
|
|
445
|
+
archives. It does not restore the unversioned two-array reader, removed 0.4
|
|
446
|
+
entry points, or a guarantee that an older package can open files written by an
|
|
447
|
+
arbitrary newer format.
|
|
448
|
+
|
|
449
|
+
### Reproducible performance evidence
|
|
450
|
+
|
|
451
|
+
- Add deterministic ingestion and search benchmark commands that record their
|
|
452
|
+
environment, input sizes, warmup, repetitions, and summary statistic.
|
|
453
|
+
- Keep structural regression tests for capacity growth, partial top-k
|
|
454
|
+
selection, and avoidable full-matrix copies.
|
|
455
|
+
- Keep wall-clock thresholds out of shared CI, where runner variance would make
|
|
456
|
+
failures noisy rather than diagnostic.
|
|
457
|
+
|
|
458
|
+
Version 0.6 does not add update, delete, builder, streaming, async, locking, or
|
|
459
|
+
metadata-query APIs. It does not introduce a validation dependency, public
|
|
460
|
+
exception hierarchy, archive format version 2, or deep-copy policy for opaque
|
|
461
|
+
metadata.
|
|
407
462
|
|
|
408
463
|
## 1.0.0: Stable contracts
|
|
409
464
|
|