numpy-vector-store 0.3.2__tar.gz → 0.4.0__tar.gz

{numpy_vector_store-0.3.2 → numpy_vector_store-0.4.0}/.github/workflows/checks.yml

@@ -55,7 +55,6 @@ jobs:
       fail-fast: false
       matrix:
         python-version:
-          - "3.10"
           - "3.11"
           - "3.12"
           - "3.13"
@@ -84,7 +83,7 @@ jobs:
         run: uv run --locked pytest
 
   minimum-numpy:
-    name: Test minimum NumPy on Python 3.10
+    name: Test minimum NumPy on Python 3.11
     runs-on: ubuntu-latest
 
     steps:
@@ -94,13 +93,13 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
-          python-version: "3.10"
+          python-version: "3.11"
 
       - name: Set up uv
         uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
         with:
           version: "0.11.32"
-          python-version: "3.10"
+          python-version: "3.11"
           enable-cache: true
 
       - name: Run tests with the minimum NumPy version
@@ -108,8 +107,8 @@ jobs:
           uv run
           --isolated
           --no-project
-          --python 3.10
+          --python 3.11
           --with-editable .
-          --with numpy==1.21.3
+          --with numpy==1.23.2
           --with pytest==8.4.2
           python -m pytest

numpy_vector_store-0.4.0/CHANGELOG.md

@@ -0,0 +1,160 @@
+# Changelog
+
+This changelog records user-visible changes to NumPy Vector Store. Earlier
+release notes remain available on the
+[GitHub releases page](https://github.com/tvanreenen/numpy-vector-store/releases).
+
+## 0.4.0 - 2026-08-09
+
+This release makes persistence explicit, self-describing, and safer to update.
+The earlier archive format stored vectors and metadata but omitted the settings
+needed to interpret those vectors correctly. Version 0.4 records that
+configuration in every new archive and introduces a lifecycle that clearly
+separates creating, opening, saving, and reloading a store.
+
+### Explicit persistence lifecycle
+
+- Add `VectorStore.open(path)` to construct a store from a versioned archive.
+  The archive supplies its own dimensions and normalization mode, so callers no
+  longer need to repeat configuration that may be wrong.
+- Let `save(path)` perform the first save or a Save As operation and bind that
+  destination. Later `save()` calls update the bound archive.
+- Add `reload()` as a deliberate refresh from disk. It always attempts to read
+  the bound archive and leaves current in-memory state unchanged if reading or
+  validation fails.
+- Keep creating a new in-memory store separate from opening one on disk. This
+  makes file access and persistence boundaries visible in application code.
+
+### Versioned, self-describing archives
+
+- Write archive format version 1 with `format_version`, `dimensions`,
+  `normalize`, `vectors`, and `metadata` fields.
+- Validate the complete archive before changing live store state, including
+  field names, scalar configuration, array dtypes and shapes, row counts,
+  finite vector values, and normalized-store zero-vector rules.
+- Reject unsupported format versions and malformed archives clearly rather
+  than inferring missing configuration or partially applying valid fields.
+- Continue preserving each metadata item as one opaque row payload.
+
+### Safer archive replacement
+
+- Write each save to a uniquely named temporary archive in the destination
+  directory, close it, and then replace the destination with `os.replace`.
+- Preserve the previous complete archive when writing or replacement fails and
+  clean up temporary files after failures.
+- Bind a new Save As destination only after its archive has been written
+  successfully.
+
+This provides an atomic visibility boundary: a reader opening the destination
+sees the previous complete archive or the new complete archive instead of a
+partially written file. It does not provide file locking, multi-writer
+coordination, or a universal power-loss durability guarantee.
+
+### Short migration window
+
+- Keep constructor `file_path=`, instance `load()`, and direct context-manager
+  persistence for version 0.4 with `FutureWarning`. They will be removed in
+  0.5.
+- Make the deprecated context manager save only after a successful block. If
+  the block raises, it does not save or suppress the exception.
+- Keep a configuration-aware reader for older archives containing only
+  `vectors` and `metadata`. Loading one warns, and its next save rewrites it as
+  format version 1.
+- Intentionally make `open()` reject an unversioned archive because that file
+  cannot report its original dimensions or normalization semantics.
+- Add a dedicated [persistence migration guide](MIGRATION.md) with side-by-side
+  API replacements and one-time legacy archive conversion instructions.
+
+The legacy API and unversioned archive reader are removed in 0.5. Applications
+should migrate an old archive once with 0.4 or recreate it from source data;
+indefinite compatibility with the incomplete two-array format is not planned.
+
+### Runtime compatibility
+
+- Support Python 3.11 through 3.14. Python 3.10 remains supported by the 0.3
+  release series but is not supported by 0.4.
+- Raise the minimum NumPy version from 1.21.3 to 1.23.2, the earliest release
+  that supports Python 3.11.
+- Exercise every supported Python version in CI and test NumPy 1.23.2 in a
+  dedicated minimum-dependency job.
+
+### Upgrade notes and boundaries
+
+- Search, insertion, retrieval, clearing, normalization, and metadata behavior
+  are unchanged from 0.3.2.
+- Code using `VectorStore.open()`, `save(path)`, `save()`, and `reload()` is on
+  the persistence API intended for 0.5.
+- Code using a transitional entry point continues to work in 0.4 but emits a
+  warning so the required 0.5 migration is visible during testing.
+- Metadata still uses NumPy's pickle-backed object-array loading. Only open
+  archives produced by your application or another trusted source.
+- Mutable public state, repeated-add performance, deterministic tie ordering,
+  and a formal thread-safety contract remain planned for later releases.
+
+## 0.3.2 - 2026-07-27
+
+This reliability and performance patch makes existing vector storage, search,
+metadata, and persistence behavior safer and more predictable. It does not
+intentionally break valid existing usage or change the `.npz` archive format.
+
+### Numerical reliability
+
+- Reject vectors, queries, and search thresholds that contain non-finite values
+  or cannot remain finite when represented as `float32`. Invalid input now
+  fails before it can corrupt stored state or ranking.
+- Calculate norms and raw metric intermediates with `float64` where `float32`
+  could overflow or underflow. Large and very small finite vectors can now be
+  normalized and compared reliably.
+- Allow zero vectors in stores created with `normalize=False`, where they are
+  valid for dot-product and Euclidean search.
+- Raise a clear error if a raw cosine search includes a zero vector, because
+  cosine similarity is undefined for that row.
+- Avoid duplicate full-size `float64` buffers when calculating raw Euclidean
+  distance.
+
+### Persistence
+
+- Resolve a path without an `.npz` suffix to the same archive for both saving
+  and loading. For example, `file_path="vectors"` consistently uses
+  `vectors.npz`.
+- Allow `load()` to be retried when the persistence file did not exist during
+  an earlier attempt.
+- Reset load state in `clear()` so a subsequent explicit `load()` can restore
+  the saved rows.
+- Keep repeated `load()` calls idempotent after a successful load.
+
+### Metadata
+
+- Preserve each item in the outer metadata sequence as one opaque row payload.
+  Tuples and lists are no longer mistaken for extra NumPy array dimensions.
+- Support dictionary, dataclass, tuple, list, string, integer, and other scalar
+  payloads consistently through insertion, search results, saving, and loading.
+- Continue rejecting explicitly multidimensional NumPy metadata arrays rather
+  than silently flattening ambiguous input.
+
+### Search memory use
+
+- Search the stored vector matrix directly when `within_rows` is omitted,
+  avoiding an unnecessary full-matrix copy on every unfiltered query.
+- Preserve original store indexes and metadata when `within_rows` selects a
+  filtered subset.
+- Document that filtered searches allocate a temporary matrix proportional to
+  the selected row count and vector dimensions.
+
+### Compatibility and validation
+
+- Test Python 3.10 through 3.14 in GitHub Actions.
+- Test the minimum supported NumPy version in a dedicated Python 3.10 job.
+- Raise the minimum NumPy requirement from 1.20 to 1.21.3 so it is compatible
+  with the oldest supported Python version.
+- Require linting, formatting, type checking, and the full Python test matrix
+  before publishing to PyPI.
+
+### Upgrade notes
+
+- No public method signatures or persisted field names changed.
+- Existing trusted `.npz` archives with `vectors` and `metadata` remain
+  readable.
+- Environments using NumPy 1.20 must upgrade to NumPy 1.21.3 or newer.
+- Inputs that previously produced `nan`, `inf`, or unreliable rankings now
+  raise `ValueError` instead.

numpy_vector_store-0.4.0/MIGRATION.md

@@ -0,0 +1,169 @@
+# Persistence migration guide
+
+Version 0.4 introduces a persistence lifecycle that separates creating a new
+store, opening an existing archive, saving, and deliberately refreshing from
+disk. It also provides a one-release bridge for applications and archives using
+the earlier API.
+
+The transition is intentionally short. Version 0.4 emits `FutureWarning` for
+constructor `file_path=`, instance `load()`, direct context-manager use, and
+unversioned archives. Version 0.5 removes those entry points and the
+unversioned archive reader.
+
+## Creating and saving a new store
+
+Previously, the destination was supplied while constructing the store:
+
+```python
+store = VectorStore(dimensions=1536, file_path="vectors.npz")
+store.add(vectors, metadata)
+store.save()
+```
+
+Create the in-memory store first, then bind its destination with the first
+save:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Later `save()` calls update the bound archive. Supplying another path performs
+a Save As operation and binds the new destination after the write succeeds.
+Calling `save()` before a store is bound raises `ValueError` rather than
+silently leaving the data unsaved.
+
+## Opening an existing archive
+
+The old API required callers to repeat configuration that should belong to the
+archive:
+
+```python
+store = VectorStore(
+    dimensions=1536,
+    file_path="vectors.npz",
+    normalize=True,
+)
+store.load()
+```
+
+Open a version 1 archive directly:
+
+```python
+store = VectorStore.open("vectors.npz")
+```
+
+`open()` restores `dimensions` and `normalize` from the archive, validates its
+contents, loads its rows, and binds its path. Applications no longer need to
+keep archive configuration separately or risk loading the same vectors with
+different semantics.
+
+The generic parameter still describes application metadata. It can be kept
+when useful:
+
+```python
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ChunkMetadata:
+    source: str
+    chunk_index: int
+
+
+store = VectorStore[ChunkMetadata].open("vectors.npz")
+```
+
+`ChunkMetadata` is an example application type, not a class provided by this
+library.
+
+## Refreshing from disk
+
+Use `reload()` when another process may have changed the bound archive and the
+current in-memory changes should be discarded:
+
+```python
+store = VectorStore.open("vectors.npz")
+
+# Later, after the file may have changed:
+store.reload()
+```
+
+Unlike transitional `load()`, `reload()` always attempts to read. It raises if
+the store is unbound, the file is missing, or the archive is invalid. A failed
+reload leaves the current in-memory vectors and metadata unchanged.
+
+## Replacing context-manager persistence
+
+The earlier context manager saved automatically on exit:
+
+```python
+with VectorStore(dimensions=1536, file_path="vectors.npz") as store:
+    store.add(vectors, metadata)
+```
+
+Use an explicit save after the work succeeds:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Normal Python control flow already prevents the final line from running if
+`add()` raises. The persistence boundary is visible, and readers do not need to
+remember an implicit exit side effect.
+
+During 0.4, the deprecated context manager saves only after normal completion.
+It does not save while an exception is propagating and does not suppress the
+exception. There is no planned replacement autosave context manager.
+
+## Migrating an archive created before 0.4
+
+Older archives contain only `vectors` and `metadata`. They do not record their
+dimensions or whether vectors use normalized or raw semantics, so `open()`
+cannot construct a correct store from them.
+
+Use the 0.4 compatibility API once with the archive's original configuration:
+
+```python
+legacy = VectorStore(
+    dimensions=1536,
+    file_path="legacy-vectors.npz",
+    normalize=True,
+)
+legacy.load()
+legacy.save()
+```
+
+This code emits transition warnings by design. The final `save()` rewrites the
+archive as format version 1 with `format_version`, `dimensions`, `normalize`,
+`vectors`, and `metadata`. It can then use the preferred API:
+
+```python
+store = VectorStore.open("legacy-vectors.npz")
+```
+
+Applications that can recreate archives from source vectors and metadata may
+choose to do that instead. The unversioned reader is removed in 0.5 rather than
+maintained as a long-term compatibility format.
+
+## Removal schedule
+
+| Transitional behavior | 0.4 | 0.5 |
+|---|---|---|
+| Constructor `file_path=` | Works with `FutureWarning` | Removed |
+| Instance `load()` | Works with `FutureWarning` | Removed |
+| Direct context-manager persistence | Saves only on successful exit and warns | Removed |
+| Unversioned two-array archives | Load with known configuration and warn | Reader removed |
+| `open()`, `save(path)`, `save()`, and `reload()` | Preferred | Supported |
+
+## Persistence boundaries that do not change
+
+Metadata is stored in a pickle-backed NumPy object array. Archives remain
+trusted input and must not be opened from untrusted or unverifiable sources.
+
+Saves use same-directory temporary files and atomic replacement, but the
+library does not add file locking, coordinate concurrent writers, or promise
+power-loss durability across every operating system and filesystem.

{numpy_vector_store-0.3.2 → numpy_vector_store-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: numpy-vector-store
-Version: 0.3.2
+Version: 0.4.0
 Summary: A fast, lightweight, and zero-setup in-memory vector store powered by NumPy
 Project-URL: Homepage, https://github.com/tvanreenen/numpy-vector-store
 Project-URL: Repository, https://github.com/tvanreenen/numpy-vector-store
@@ -13,13 +13,12 @@ Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
-Requires-Python: >=3.10
-Requires-Dist: numpy>=1.21.3
+Requires-Python: >=3.11
+Requires-Dist: numpy>=1.23.2
 Description-Content-Type: text/markdown
 
 # NumPy Vector Store
@@ -30,7 +29,7 @@ A fast, lightweight, zero-setup in-memory vector store powered by NumPy.
 - **Fast exact vector search** using vectorized NumPy operations
 - **Simple typed API** returning `VectorHit(index, value, metadata)`
 - **Composable filtering** by passing prefiltered row indexes with `within_rows`
-- **Portable persistence** as trusted local `.npz` files with `vectors` + `metadata`
+- **Portable persistence** as versioned, self-describing trusted local `.npz` files
 - **No framework opinions**: bring your own embeddings, chunking, async, and metadata model
 
 ## Why?
@@ -210,54 +209,102 @@ for hit in hits:
 
 ## Persistence
 
-Pass a `file_path` and call `save()` / `load()` explicitly:
+Create a new store normally, then supply its destination on the first save:
 
 ```python
-store = VectorStore[dict[str, str]](dimensions=1536, file_path="vectors.npz")
+store = VectorStore[dict[str, str]](dimensions=1536)
 store.add(embeddings, metadata)
-store.save()
+store.save("vectors.npz")
+```
+
+`save(path)` writes the archive and binds that path to the store. Later
+`save()` calls update the bound archive. Passing another path performs a Save
+As operation; the new path becomes the binding only after the write succeeds.
+
+Open an existing store directly from its self-describing archive:
 
-loaded = VectorStore[dict[str, str]](dimensions=1536, file_path="vectors.npz")
-loaded.load()
+```python
+store = VectorStore[dict[str, str]].open("vectors.npz")
 ```
 
+`open()` restores `dimensions` and `normalize` from the archive and binds its
+path. Use `reload()` when the file may have changed externally and you
+explicitly want to discard current in-memory changes:
+
+```python
+store.reload()
+```
+
+Unlike the transitional `load()` method, `reload()` always rereads the bound
+archive and raises if the store is unbound, the file is missing, or the archive
+is invalid. A failed reload leaves the current in-memory vectors and metadata
+unchanged.
+
 The `.npz` suffix may be omitted. An extensionless path such as `"vectors"` is
-resolved to `"vectors.npz"` for both saving and loading.
+resolved to `"vectors.npz"` for saving, opening, and reloading.
 
-If you save with `normalize=False`, load with `normalize=False` too:
+Raw-vector configuration is also restored from the archive:
 
 ```python
 store = VectorStore[dict[str, str]](
     dimensions=1536,
-    file_path="raw-vectors.npz",
     normalize=False,
 )
 store.add(raw_vectors, metadata)
-store.save()
+store.save("raw-vectors.npz")
 
-loaded = VectorStore[dict[str, str]](
-    dimensions=1536,
-    file_path="raw-vectors.npz",
-    normalize=False,
-)
-loaded.load()
+loaded = VectorStore[dict[str, str]].open("raw-vectors.npz")
+assert loaded.normalize is False
 ```
 
-Context manager usage auto-saves on exit:
+Archives written by 0.4 use format version 1 and contain `format_version`,
+`dimensions`, `normalize`, `vectors`, and `metadata`. The stored configuration
+prevents an archive from being loaded with different dimensions or normalization
+semantics.
+
+Opening and reloading validate the complete schema, array dtypes and shapes, row
+counts, finite vector values, and zero-norm behavior before changing in-memory
+state. Opaque metadata values remain individual row payloads across persistence
+round trips.
+
+Each save writes a uniquely named temporary archive in the destination
+directory, closes it, and then replaces the destination with `os.replace`.
+Readers opening the destination path therefore see either the previous complete
+archive or the new complete archive rather than a partially written file. If
+writing or replacement fails, the previous destination remains in place and the
+temporary file is removed.
+
+Atomic replacement is not file locking or multi-writer coordination. Concurrent
+writers can still replace one another, and the library does not promise that a
+successful save has reached durable hardware storage across every operating
+system or power failure.
+
+Older archives containing only `vectors` and `metadata` remain readable in 0.4
+through the configuration-aware legacy API:
 
 ```python
-with VectorStore[dict[str, str]](dimensions=1536, file_path="vectors.npz") as store:
-    store.add(embeddings, metadata)
+legacy = VectorStore[dict[str, str]](
+    dimensions=1536,
+    file_path="legacy-vectors.npz",
+    normalize=True,
+)
+legacy.load()
+legacy.save()
 ```
 
-Persistence uses a minimal NumPy `.npz` contract with `vectors` and `metadata`
-arrays. The `.npz` file does not encode the `normalize` setting; choose the same
-setting when loading that you used when saving. Loading validates shape,
-dimensions, row counts, and zero-norm vectors. A `load()` call made before the
-file exists can be retried after the file is created. Repeated calls after a
-successful load do nothing unless `clear()` resets the in-memory store. Opaque
-metadata values remain individual row payloads across save/load round trips.
-Persistence uses `allow_pickle=True` for flexible Python metadata payloads, so
+Loading a legacy archive emits a `FutureWarning`, and saving rewrites it as
+format version 1. `open()` intentionally rejects unversioned archives because
+they do not contain enough configuration to construct a store safely. The
+legacy reader will be removed in 0.5; migrate an archive once with 0.4 or
+recreate it from source data.
+
+Constructor `file_path=`, instance `load()`, and direct context-manager
+persistence remain available with `FutureWarning` during the 0.4 transition.
+The explicit lifecycle shown above is the preferred API and will be the only
+persistence API in 0.5. See the [persistence migration guide](MIGRATION.md) for
+side-by-side replacements and the one-time legacy archive conversion.
+
+Metadata persistence uses `allow_pickle=True` for flexible Python payloads, so
 only load files generated by your own application or another trusted local
 process. Loading untrusted `.npz` files is not a supported security model.
 
@@ -268,14 +315,19 @@ the API stabilizes. Changes are documented in the [changelog](CHANGELOG.md) and
 GitHub release notes. Deprecated APIs will keep warning for at least one point
 release before removal.
 
-Supported Python versions are listed in the package classifiers and exercised
-in CI. The project generally retains stable CPython versions until their
-upstream end-of-life, adds new versions after its dependencies and CI support
-them, and drops versions only in minor releases.
+Version 0.4 supports Python 3.11 through 3.14 and NumPy 1.23.2 or newer. These
+versions are listed in the package metadata and exercised in CI, including a
+dedicated check against the minimum NumPy version. Python 3.10 remains supported
+by the 0.3 release series but is not supported by 0.4.
+
+The project generally retains stable CPython versions until their upstream
+end-of-life, adds new versions after its dependencies and CI support them, and
+drops versions only in minor releases.
 
 See the [changelog](CHANGELOG.md) for release history and the
 [project roadmap](ROADMAP.md) for the planned path to stable API and persistence
-contracts.
+contracts. Persistence users upgrading from the 0.3 API should also read the
+[migration guide](MIGRATION.md).
 
 ## Contributing