numpy-vector-store 0.4.0__tar.gz → 0.5.0__tar.gz

{numpy_vector_store-0.4.0 → numpy_vector_store-0.5.0}/CHANGELOG.md

@@ -4,6 +4,137 @@ This changelog records user-visible changes to NumPy Vector Store. Earlier
 release notes remain available on the
 [GitHub releases page](https://github.com/tvanreenen/numpy-vector-store/releases).
 
+## 0.5.0 - 2026-08-21
+
+This release gives `VectorStore` clear ownership of its configuration and row
+storage, makes repeated additions scale without recopying the complete store on
+every call, and defines deterministic ordering for equal search values. It also
+finishes the persistence transition announced in 0.4: the explicit
+create/open/save/reload lifecycle is now the only persistence API.
+
+### API at a glance
+
+The core workflow remains small:
+
+```python
+store = VectorStore(dimensions=1536, normalize=True)
+store.add(vectors, metadata)
+
+hits = store.cosine_search(query, top_k=10)
+row = store.get(0)
+
+store.save("vectors.npz")
+store.save()
+
+loaded = VectorStore.open("vectors.npz")
+loaded.reload()
+```
+
+`VectorStore` and `VectorHit` remain the only public classes. Version 0.5 does
+not add a document wrapper, builder, snapshot object, metadata query language,
+or another persistence abstraction.
+
+### Store-owned configuration and rows
+
+- Make `dimensions`, `normalize`, and `file_path` read-only properties. The
+  constructor owns configuration, while `open(path)` and a successful
+  `save(path)` own archive binding changes.
+- Return zero-copy, non-writeable active-row views from `vectors` and
+  `metadata`. Direct item assignment and ordinary attempts to enable writes are
+  rejected.
+- Keep inspection views moment-in-time. Code holding a view across `add()`,
+  `clear()`, or `reload()` must request a new one to inspect current rows.
+- Return an independent `float32` vector copy from `get(index)`, so changing a
+  retrieved vector cannot change normalized storage or a later search.
+- Preserve opaque metadata payloads by reference. The read-only metadata view
+  protects row alignment, not the contents of a caller-owned dict, list,
+  dataclass, or other application object.
+
+The views prevent accidental mutation through the supported API; they are not
+tamper-proof snapshots. Deliberately reaching backing storage through `.base`,
+private attributes, `ctypes`, or similar escape hatches remains unsupported.
+Call `.copy()` when code needs an independently mutable array.
+
+### Amortized repeated additions
+
+- Replace whole-store concatenation on every noninitial `add()` with private
+  contiguous vector and metadata capacity plus an active row count.
+- Reuse spare rows when a new batch fits. When it does not, grow vector and
+  metadata storage together and copy active rows once.
+- Keep spare capacity out of `len()`, inspection, search, `within_rows`,
+  retrieval, and saved archives.
+- Make `clear()` return the store to empty arrays and drop the store's retained
+  capacity. A caller-held older NumPy view may still keep its previous buffer
+  alive until that view is released.
+
+The `add(vectors, metadata)` signature, insertion order, validation,
+normalization, and opaque metadata behavior do not change. Passing a batch is
+still useful when the application already has one, but repeated small
+additions no longer move every earlier row on each call.
+
+### Deterministic search ties
+
+- Continue ordering cosine and dot-product results from larger values to
+  smaller values and Euclidean results from smaller distances to larger ones.
+- Break exact computed-value ties by ascending original store row index.
+- Apply the row-index tie break when choosing which rows cross the `top_k`
+  boundary, not only when ordering an already selected subset.
+- Use original store indexes for filtered searches, so shuffling the same
+  `within_rows` values does not change tied results.
+- Preserve partial top-k selection rather than replacing it with a full-store
+  sort.
+
+Only exactly equal computed values use the row-index tie break. Close but
+unequal values remain ordered by their metric value.
+
+### Final persistence lifecycle
+
+The 0.4 compatibility window is now closed:
+
+- Remove constructor `file_path=`. Create an in-memory store, then call
+  `save(path)` to write and bind it.
+- Remove instance `load()`. Use `VectorStore.open(path)` to construct a store
+  from an archive and `reload()` to refresh a bound store.
+- Remove context-manager persistence. Call `save()` explicitly where the
+  application intends to persist state.
+- Remove the reader for unversioned archives containing only `vectors` and
+  `metadata`.
+
+Archive format version 1 is unchanged. Applications that already use the 0.4
+`open()`, `save(path)`, `save()`, and `reload()` lifecycle need no persistence
+changes. An older unversioned archive must be converted with 0.4 using its
+original dimensions and normalization mode, or recreated from source data,
+before upgrading. See the [persistence migration guide](MIGRATION.md) for the
+side-by-side replacements and conversion procedure.
+
+### Thread safety and persistence boundaries
+
+- Support concurrent search, `get()`, and inspection on one instance only
+  while its state and shared metadata payloads remain unchanged.
+- Require application-level synchronization for every access when any thread
+  may call `add()`, `clear()`, `reload()`, or `save()`, or mutate shared
+  metadata.
+- Keep atomic archive replacement as a destination-visibility guarantee, not a
+  store snapshot, file lock, or multi-writer coordination system.
+
+Separate store instances writing the same path can still replace one another;
+applications with multiple writers must serialize them. Metadata persistence
+continues to use NumPy's pickle-backed object arrays, so archives remain trusted
+input and must not be opened from untrusted or unverifiable sources.
+
+### Runtime compatibility and upgrade notes
+
+- Continue supporting Python 3.11 through 3.14 and NumPy 1.23.2 or newer.
+- Continue exercising every supported Python version in CI, with a dedicated
+  minimum-NumPy job on Python 3.11.
+- Keep archive format version 1 readable and writable without a file migration.
+- Expect `AttributeError` from code that assigns public configuration or row
+  arrays, and different ordering from code that relied on incidental NumPy
+  partition order for exact ties.
+- Expect a migration before upgrading code that still uses constructor
+  `file_path=`, instance `load()`, context-manager persistence, or an
+  unversioned two-array archive.
+
 ## 0.4.0 - 2026-08-09
 
 This release makes persistence explicit, self-describing, and safer to update.

numpy_vector_store-0.5.0/MIGRATION.md

@@ -0,0 +1,289 @@
+# Persistence migration guide
+
+Version 0.5 completes the persistence lifecycle introduced in 0.4. Creating a
+store, opening an archive, saving, and refreshing from disk now have separate,
+explicit operations. Applications that adopted the recommended 0.4 API need no
+further persistence changes.
+
+The 0.4 compatibility paths are no longer present: constructor `file_path=`,
+instance `load()`, context-manager persistence, and the unversioned archive
+reader have been removed.
+
+## API at a glance
+
+```python
+store = VectorStore(dimensions=1536, normalize=True)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+
+store.save()  # Update the bound archive.
+
+loaded = VectorStore.open("vectors.npz")
+loaded.reload()  # Deliberately discard memory and reread the archive.
+```
+
+The persistence changes from 0.4 to 0.5 are:
+
+| Area | 0.4 | 0.5 |
+|---|---|---|
+| Create a store | `VectorStore(dimensions, normalize=...)` | Unchanged |
+| Constructor `file_path=` | Works with `FutureWarning` | Removed |
+| Open an archive | `VectorStore.open(path)` | Unchanged |
+| Save and bind | `save(path)` | Unchanged |
+| Save again | `save()` | Unchanged |
+| Refresh from disk | `reload()` | Unchanged |
+| Instance `load()` | Works with `FutureWarning` | Removed |
+| Context manager | Works with `FutureWarning` | Removed |
+| Unversioned archive | Temporary migration reader | Reader removed |
+| Format version 1 archive | Supported | Supported unchanged |
+| `dimensions`, `normalize`, `file_path` | Writable attributes | Read-only properties |
+| `vectors`, `metadata` | Writable owning arrays | Read-only inspection views |
+| Vector returned by `get()` | View into live storage | Independent `float32` copy |
+| Repeated `add()` calls | Recopy all existing rows | Reuse private spare capacity |
+| Equal search values | Unspecified order | Lower original store row index first |
+| Thread safety | Not formally defined | Concurrent reads only while state is unchanged |
+
+## Store-owned state
+
+Most code that reads configuration or uses NumPy operations for inspection and
+prefiltering remains unchanged. The familiar property names are still present:
+
+```python
+store.dimensions
+store.normalize
+store.file_path
+store.vectors
+store.metadata
+```
+
+The difference is ownership. Configuration cannot be assigned directly, and
+the vector and metadata views reject normal row mutation. Their views describe
+the rows present when the property was requested, so code should request a
+fresh view after `add()`, `clear()`, or `reload()`.
+
+The views are a supported inspection boundary, not tamper-proof snapshots.
+Deliberately mutating their backing storage through `.base`, private attributes,
+`ctypes`, or similar escape hatches is unsupported and can corrupt the store.
+Call `.copy()` when code needs an independently mutable full-array snapshot.
+For metadata, this copies the outer row array while preserving the opaque
+payload objects by reference.
+
+Row retrieval deliberately treats vectors and metadata differently:
+
+```python
+vector, payload = store.get(0)
+
+vector[0] = 10.0           # Independent copy; the store is unchanged.
+payload["reviewed"] = True  # Shared application metadata object.
+```
+
+Vectors have one uniform NumPy representation, so copying a single row gives
+the caller clear ownership at bounded cost. Metadata can be any Python object,
+so the store preserves payload identity instead of imposing a potentially
+expensive or invalid deep-copy policy. Applications that need immutable
+metadata can use frozen application objects or copy payloads themselves.
+
+## Repeated additions
+
+The `add(vectors, metadata)` signature and validation rules do not change. In
+0.4, every noninitial call concatenated the new rows with the complete store.
+In 0.5, the store reserves private row capacity and grows it geometrically when
+needed. Repeated small additions therefore copy existing rows only when the
+current allocation is full.
+
+This does not add a public capacity setting or change row indexes. `len(store)`,
+inspection, search, retrieval, and persistence continue to use only active
+rows. `clear()` discards reserved storage as well as active rows, so it does not
+retain metadata payloads through unused capacity. A caller-held inspection view
+can still keep its previous NumPy buffer and payload references alive until the
+view itself is released.
+
+## Search result ordering
+
+Version 0.5 makes exact metric ties deterministic. Cosine and dot-product
+searches still rank larger values first, and Euclidean search still ranks
+smaller distances first. When computed values are equal, the lower original
+store row index comes first.
+
+This tie break also decides which rows are returned when equal values cross
+the `top_k` boundary. A shuffled `within_rows` input does not change the result:
+the original store indexes, rather than positions in the filtered input, break
+the tie. No call-site change is required, but code that depended on an
+incidental NumPy partition order should update its expectations.
+
+## Thread safety
+
+Version 0.5 defines the existing synchronization boundary without adding
+internal locks. Search, `get()`, and inspection may run concurrently on one
+instance only while store state and shared metadata payloads remain unchanged.
+
+Applications must externally synchronize all access to an instance whenever
+any thread may call `add()`, `clear()`, `reload()`, or `save()`, or mutate a
+metadata payload shared with the store. Saves must not overlap in-memory
+mutation because archive replacement protects the destination path, not the
+consistency of vectors and metadata read from a changing instance.
+
+Atomic replacement also does not coordinate separate store instances writing
+the same path. Applications with multiple writers must serialize them; without
+that coordination, each complete save may replace another and the last
+successful replacement wins.
+
+## Creating and saving a new store
+
+Previously, the destination was supplied while constructing the store:
+
+```python
+store = VectorStore(dimensions=1536, file_path="vectors.npz")
+store.add(vectors, metadata)
+store.save()
+```
+
+Create the in-memory store first, then bind its destination with the first
+save:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Later `save()` calls update the bound archive. Supplying another path performs
+a Save As operation and binds the new destination after the write succeeds.
+Calling `save()` before a store is bound raises `ValueError` rather than
+silently leaving the data unsaved.
+
+## Opening an existing archive
+
+The old API required callers to repeat configuration that should belong to the
+archive:
+
+```python
+store = VectorStore(
+    dimensions=1536,
+    file_path="vectors.npz",
+    normalize=True,
+)
+store.load()
+```
+
+Open a version 1 archive directly:
+
+```python
+store = VectorStore.open("vectors.npz")
+```
+
+`open()` restores `dimensions` and `normalize` from the archive, validates its
+contents, loads its rows, and binds its path. Applications no longer need to
+keep archive configuration separately or risk loading the same vectors with
+different semantics.
+
+The generic parameter still describes application metadata. It can be kept
+when useful:
+
+```python
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ChunkMetadata:
+    source: str
+    chunk_index: int
+
+
+store = VectorStore[ChunkMetadata].open("vectors.npz")
+```
+
+`ChunkMetadata` is an example application type, not a class provided by this
+library.
+
+## Refreshing from disk
+
+Use `reload()` when another process may have changed the bound archive and the
+current in-memory changes should be discarded:
+
+```python
+store = VectorStore.open("vectors.npz")
+
+# Later, after the file may have changed:
+store.reload()
+```
+
+`reload()` always attempts to read. It raises if the store is unbound, the file
+is missing, or the archive is invalid. A failed reload leaves the current
+in-memory vectors and metadata unchanged.
+
+## Replacing context-manager persistence
+
+The earlier context manager saved automatically on exit:
+
+```python
+with VectorStore(dimensions=1536, file_path="vectors.npz") as store:
+    store.add(vectors, metadata)
+```
+
+Use an explicit save after the work succeeds:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Normal Python control flow already prevents the final line from running if
+`add()` raises. The persistence boundary is visible, and readers do not need to
+remember an implicit exit side effect.
+
+There is no replacement autosave context manager. An explicit `save()` keeps
+the persistence boundary visible and lets the application decide whether work
+completed successfully enough to persist.
+
+## Migrating an archive created before 0.4
+
+Older archives contain only `vectors` and `metadata`. They do not record their
+dimensions or whether vectors use normalized or raw semantics, so `open()`
+cannot construct a correct store from them.
+
+Before upgrading to 0.5, use the 0.4 compatibility API once with the archive's
+original configuration:
+
+```python
+legacy = VectorStore(
+    dimensions=1536,
+    file_path="legacy-vectors.npz",
+    normalize=True,
+)
+legacy.load()
+legacy.save()
+```
+
+This code emits transition warnings in 0.4 by design. The final `save()`
+rewrites the archive as format version 1 with `format_version`, `dimensions`,
+`normalize`, `vectors`, and `metadata`. It can then be opened by 0.5:
+
+```python
+store = VectorStore.open("legacy-vectors.npz")
+```
+
+Applications that can recreate archives from source vectors and metadata may
+do that instead. NumPy Vector Store 0.5 cannot perform this conversion because
+the unversioned file does not contain enough information to reconstruct its
+configuration safely.
+
+## Removal schedule
+
+| Transitional behavior | 0.4 | 0.5 |
+|---|---|---|
+| Constructor `file_path=` | Works with `FutureWarning` | Removed |
+| Instance `load()` | Works with `FutureWarning` | Removed |
+| Direct context-manager persistence | Saves only on successful exit and warns | Removed |
+| Unversioned two-array archives | Load with known configuration and warn | Reader removed |
+| `open()`, `save(path)`, `save()`, and `reload()` | Preferred | Supported |
+
+## Persistence boundaries that do not change
+
+Metadata is stored in a pickle-backed NumPy object array. Archives remain
+trusted input and must not be opened from untrusted or unverifiable sources.
+
+Saves use same-directory temporary files and atomic replacement, but the
+library does not add file locking, coordinate concurrent writers, or promise
+power-loss durability across every operating system and filesystem.

{numpy_vector_store-0.4.0 → numpy_vector_store-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: numpy-vector-store
-Version: 0.4.0
+Version: 0.5.0
 Summary: A fast, lightweight, and zero-setup in-memory vector store powered by NumPy
 Project-URL: Homepage, https://github.com/tvanreenen/numpy-vector-store
 Project-URL: Repository, https://github.com/tvanreenen/numpy-vector-store
@@ -93,6 +93,62 @@ Each payload can be a dict, dataclass, tuple, list, string, integer row ID, or
 another Python object that fits your application. Tuple and list payloads remain
 single row values rather than being interpreted as additional array dimensions.
 
+## State ownership
+
+The store owns its configuration and row structure. `dimensions`, `normalize`,
+and `file_path` are readable properties, but callers cannot assign them
+directly. Use the constructor for configuration, `open(path)` to open an
+archive, and `save(path)` to establish or change a binding.
+
+`store.vectors` and `store.metadata` are zero-copy, non-writeable NumPy views
+for inspection and metadata prefiltering. Direct item assignment and ordinary
+attempts to enable writes are rejected. Request a fresh view after `add()`,
+`clear()`, or `reload()` when current rows are required.
+
+These views prevent accidental mutation through the supported API; they are not
+tamper-proof snapshots. NumPy exposes shared buffers, and Python private state
+can be reached deliberately. Mutating backing storage through `.base`, private
+attributes, `ctypes`, or similar escape hatches is unsupported and can corrupt
+store invariants. Use `.copy()` when code needs an independently mutable
+full-array snapshot:
+
+```python
+vectors = store.vectors.copy()
+metadata_rows = store.metadata.copy()
+```
+
+Copying `metadata` isolates the outer row array but still shares the opaque
+payload objects stored inside it.
+
+`get(index)` has a narrower ownership boundary:
+
+```python
+vector, payload = store.get(0)
+
+vector[0] = 10.0           # Independent copy; the store is unchanged.
+payload["reviewed"] = True  # Shared application metadata object.
+```
+
+The returned vector is an independent `float32` copy. The metadata payload is
+the same opaque object supplied to `add()` or restored from the archive; the
+store protects the metadata row structure but does not deep-copy arbitrary
+dicts, lists, dataclasses, or application objects. Applications that require
+immutable payloads can use frozen objects or copy them at their own boundary.
+
+## Repeated additions
+
+`add()` accepts one row or a batch, and preserves insertion order in either
+case. The store keeps private spare capacity so repeated small additions do not
+copy every existing row on every call. When that capacity is full, the vector
+and metadata arrays grow together and the active rows are copied once.
+
+Spare capacity is internal. `len(store)`, inspection views, search, `get()`, and
+saved archives contain only rows that were added. `clear()` releases both the
+active rows and any reserved capacity held by the store. As with any NumPy
+view, an older inspection view keeps its previous buffer alive until that view
+is released. Adding a batch is still preferable when the application already
+has one because it also reduces per-call validation and Python overhead.
+
 ## Normalization
 
 `VectorStore` defaults to `normalize=True`, which scales each stored vector to
@@ -132,7 +188,7 @@ compared reliably.
 | `euclidean_search` | Distance between normalized directions; useful only when direction-normalized distance is intended | True Euclidean distance over original vectors; use for geometric/feature-space nearest neighbors |
 | `get` | Returns normalized vectors | Returns original vectors |
 | `save` | Saves normalized vectors | Saves raw vectors |
-| `load` | Loads and normalizes vectors | Loads vectors exactly as stored |
+| `open` and `reload` | Restore normalized storage semantics | Restore raw vectors exactly as stored |
 
 ## Search Methods
 
@@ -160,6 +216,18 @@ store.add(vectors, metadata)
 hits = store.euclidean_search(query, top_k=10, max_value=1.5)
 ```
 
+### Result ordering
+
+Cosine and dot-product results are ordered from larger values to smaller
+values. Euclidean results are ordered from smaller distances to larger ones.
+When two computed values are exactly equal, the row with the lower original
+store index comes first.
+
+The same rule determines which tied rows cross the `top_k` boundary. It also
+applies to `within_rows`: the original store index breaks a tie, regardless of
+the order in which filtered row indexes were supplied. Values that are close
+but not exactly equal remain ordered by their computed metric value.
+
 ## Prefiltering
 
 The store does not implement a metadata query language. To filter by metadata,
@@ -207,6 +275,24 @@ for hit in hits:
     print(row["title"], hit.value)
 ```
 
+## Thread safety
+
+`VectorStore` does not use internal locks. Multiple threads may call search,
+`get()`, or the inspection properties on the same instance while its rows,
+configuration, binding, and metadata payloads remain unchanged.
+
+If any thread may call `add()`, `clear()`, `reload()`, or `save()`, every access
+to that store must use the same application-level lock or another external
+synchronization mechanism. The same rule applies when application code mutates
+a metadata payload shared with the store. In particular, do not overlap a save
+with an in-memory mutation: an atomic file replacement cannot turn two separate
+array reads into a consistent store snapshot.
+
+Separate store instances writing the same path also need external writer
+coordination. Atomic replacement prevents readers from seeing a partially
+written archive, but concurrent writers can replace one another and the last
+successful replacement wins.
+
 ## Persistence
 
 Create a new store normally, then supply its destination on the first save:
@@ -235,10 +321,9 @@ explicitly want to discard current in-memory changes:
 store.reload()
 ```
 
-Unlike the transitional `load()` method, `reload()` always rereads the bound
-archive and raises if the store is unbound, the file is missing, or the archive
-is invalid. A failed reload leaves the current in-memory vectors and metadata
-unchanged.
+`reload()` always rereads the bound archive and raises if the store is unbound,
+the file is missing, or the archive is invalid. A failed reload leaves the
+current in-memory vectors and metadata unchanged.
 
 The `.npz` suffix may be omitted. An extensionless path such as `"vectors"` is
 resolved to `"vectors.npz"` for saving, opening, and reloading.
@@ -279,30 +364,14 @@ writers can still replace one another, and the library does not promise that a
 successful save has reached durable hardware storage across every operating
 system or power failure.
 
-Older archives containing only `vectors` and `metadata` remain readable in 0.4
-through the configuration-aware legacy API:
-
-```python
-legacy = VectorStore[dict[str, str]](
-    dimensions=1536,
-    file_path="legacy-vectors.npz",
-    normalize=True,
-)
-legacy.load()
-legacy.save()
-```
-
-Loading a legacy archive emits a `FutureWarning`, and saving rewrites it as
-format version 1. `open()` intentionally rejects unversioned archives because
-they do not contain enough configuration to construct a store safely. The
-legacy reader will be removed in 0.5; migrate an archive once with 0.4 or
-recreate it from source data.
+Version 0.5 reads only self-describing format version 1 archives. Unversioned
+archives containing only `vectors` and `metadata` cannot be opened because they
+do not record dimensions or normalization semantics. Recreate those archives
+from source data, or convert them with NumPy Vector Store 0.4 before upgrading.
 
-Constructor `file_path=`, instance `load()`, and direct context-manager
-persistence remain available with `FutureWarning` during the 0.4 transition.
-The explicit lifecycle shown above is the preferred API and will be the only
-persistence API in 0.5. See the [persistence migration guide](MIGRATION.md) for
-side-by-side replacements and the one-time legacy archive conversion.
+Constructor `file_path=`, instance `load()`, and context-manager persistence
+were removed in 0.5. See the [persistence migration guide](MIGRATION.md) for the
+direct replacements and the pre-upgrade procedure for an unversioned archive.
 
 Metadata persistence uses `allow_pickle=True` for flexible Python payloads, so
 only load files generated by your own application or another trusted local
@@ -315,10 +384,10 @@ the API stabilizes. Changes are documented in the [changelog](CHANGELOG.md) and
 GitHub release notes. Deprecated APIs will keep warning for at least one point
 release before removal.
 
-Version 0.4 supports Python 3.11 through 3.14 and NumPy 1.23.2 or newer. These
+Version 0.5 supports Python 3.11 through 3.14 and NumPy 1.23.2 or newer. These
 versions are listed in the package metadata and exercised in CI, including a
 dedicated check against the minimum NumPy version. Python 3.10 remains supported
-by the 0.3 release series but is not supported by 0.4.
+by the 0.3 release series but is not supported by 0.4 or 0.5.
 
 The project generally retains stable CPython versions until their upstream
 end-of-life, adds new versions after its dependencies and CI support them, and