numpy-vector-store 0.3.2__tar.gz → 0.5.0__tar.gz

{numpy_vector_store-0.3.2 → numpy_vector_store-0.5.0}/.github/workflows/checks.yml

@@ -55,7 +55,6 @@ jobs:
       fail-fast: false
       matrix:
         python-version:
-          - "3.10"
           - "3.11"
           - "3.12"
           - "3.13"
@@ -84,7 +83,7 @@ jobs:
         run: uv run --locked pytest
 
   minimum-numpy:
-    name: Test minimum NumPy on Python 3.10
+    name: Test minimum NumPy on Python 3.11
     runs-on: ubuntu-latest
 
     steps:
@@ -94,13 +93,13 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v6
         with:
-          python-version: "3.10"
+          python-version: "3.11"
 
       - name: Set up uv
         uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
         with:
           version: "0.11.32"
-          python-version: "3.10"
+          python-version: "3.11"
           enable-cache: true
 
       - name: Run tests with the minimum NumPy version
@@ -108,8 +107,8 @@ jobs:
           uv run
           --isolated
           --no-project
-          --python 3.10
+          --python 3.11
           --with-editable .
-          --with numpy==1.21.3
+          --with numpy==1.23.2
           --with pytest==8.4.2
           python -m pytest

numpy_vector_store-0.5.0/CHANGELOG.md

@@ -0,0 +1,291 @@
+# Changelog
+
+This changelog records user-visible changes to NumPy Vector Store. Earlier
+release notes remain available on the
+[GitHub releases page](https://github.com/tvanreenen/numpy-vector-store/releases).
+
+## 0.5.0 - 2026-08-21
+
+This release gives `VectorStore` clear ownership of its configuration and row
+storage, makes repeated additions scale without recopying the complete store on
+every call, and defines deterministic ordering for equal search values. It also
+finishes the persistence transition announced in 0.4: the explicit
+create/open/save/reload lifecycle is now the only persistence API.
+
+### API at a glance
+
+The core workflow remains small:
+
+```python
+store = VectorStore(dimensions=1536, normalize=True)
+store.add(vectors, metadata)
+
+hits = store.cosine_search(query, top_k=10)
+row = store.get(0)
+
+store.save("vectors.npz")
+store.save()
+
+loaded = VectorStore.open("vectors.npz")
+loaded.reload()
+```
+
+`VectorStore` and `VectorHit` remain the only public classes. Version 0.5 does
+not add a document wrapper, builder, snapshot object, metadata query language,
+or another persistence abstraction.
+
+### Store-owned configuration and rows
+
+- Make `dimensions`, `normalize`, and `file_path` read-only properties. The
+  constructor owns configuration, while `open(path)` and a successful
+  `save(path)` own archive binding changes.
+- Return zero-copy, non-writeable active-row views from `vectors` and
+  `metadata`. Direct item assignment and ordinary attempts to enable writes are
+  rejected.
+- Keep inspection views moment-in-time. Code holding a view across `add()`,
+  `clear()`, or `reload()` must request a new one to inspect current rows.
+- Return an independent `float32` vector copy from `get(index)`, so changing a
+  retrieved vector cannot change normalized storage or a later search.
+- Preserve opaque metadata payloads by reference. The read-only metadata view
+  protects row alignment, not the contents of a caller-owned dict, list,
+  dataclass, or other application object.
+
+The views prevent accidental mutation through the supported API; they are not
+tamper-proof snapshots. Deliberately reaching backing storage through `.base`,
+private attributes, `ctypes`, or similar escape hatches remains unsupported.
+Call `.copy()` when code needs an independently mutable array.
+
+### Amortized repeated additions
+
+- Replace whole-store concatenation on every noninitial `add()` with private
+  contiguous vector and metadata capacity plus an active row count.
+- Reuse spare rows when a new batch fits. When it does not, grow vector and
+  metadata storage together and copy active rows once.
+- Keep spare capacity out of `len()`, inspection, search, `within_rows`,
+  retrieval, and saved archives.
+- Make `clear()` return the store to empty arrays and drop the store's retained
+  capacity. A caller-held older NumPy view may still keep its previous buffer
+  alive until that view is released.
+
+The `add(vectors, metadata)` signature, insertion order, validation,
+normalization, and opaque metadata behavior do not change. Passing a batch is
+still useful when the application already has one, but repeated small
+additions no longer move every earlier row on each call.
+
+### Deterministic search ties
+
+- Continue ordering cosine and dot-product results from larger values to
+  smaller values and Euclidean results from smaller distances to larger ones.
+- Break exact computed-value ties by ascending original store row index.
+- Apply the row-index tie break when choosing which rows cross the `top_k`
+  boundary, not only when ordering an already selected subset.
+- Use original store indexes for filtered searches, so shuffling the same
+  `within_rows` values does not change tied results.
+- Preserve partial top-k selection rather than replacing it with a full-store
+  sort.
+
+Only exactly equal computed values use the row-index tie break. Close but
+unequal values remain ordered by their metric value.
+
+### Final persistence lifecycle
+
+The 0.4 compatibility window is now closed:
+
+- Remove constructor `file_path=`. Create an in-memory store, then call
+  `save(path)` to write and bind it.
+- Remove instance `load()`. Use `VectorStore.open(path)` to construct a store
+  from an archive and `reload()` to refresh a bound store.
+- Remove context-manager persistence. Call `save()` explicitly where the
+  application intends to persist state.
+- Remove the reader for unversioned archives containing only `vectors` and
+  `metadata`.
+
+Archive format version 1 is unchanged. Applications that already use the 0.4
+`open()`, `save(path)`, `save()`, and `reload()` lifecycle need no persistence
+changes. An older unversioned archive must be converted with 0.4 using its
+original dimensions and normalization mode, or recreated from source data,
+before upgrading. See the [persistence migration guide](MIGRATION.md) for the
+side-by-side replacements and conversion procedure.
+
+### Thread safety and persistence boundaries
+
+- Support concurrent search, `get()`, and inspection on one instance only
+  while its state and shared metadata payloads remain unchanged.
+- Require application-level synchronization for every access when any thread
+  may call `add()`, `clear()`, `reload()`, or `save()`, or mutate shared
+  metadata.
+- Keep atomic archive replacement as a destination-visibility guarantee, not a
+  store snapshot, file lock, or multi-writer coordination system.
+
+Separate store instances writing the same path can still replace one another;
+applications with multiple writers must serialize them. Metadata persistence
+continues to use NumPy's pickle-backed object arrays, so archives remain trusted
+input and must not be opened from untrusted or unverifiable sources.
+
+### Runtime compatibility and upgrade notes
+
+- Continue supporting Python 3.11 through 3.14 and NumPy 1.23.2 or newer.
+- Continue exercising every supported Python version in CI, with a dedicated
+  minimum-NumPy job on Python 3.11.
+- Keep archive format version 1 readable and writable without a file migration.
+- Expect `AttributeError` from code that assigns public configuration or row
+  arrays, and different ordering from code that relied on incidental NumPy
+  partition order for exact ties.
+- Expect a migration before upgrading code that still uses constructor
+  `file_path=`, instance `load()`, context-manager persistence, or an
+  unversioned two-array archive.
+
+## 0.4.0 - 2026-08-09
+
+This release makes persistence explicit, self-describing, and safer to update.
+The earlier archive format stored vectors and metadata but omitted the settings
+needed to interpret those vectors correctly. Version 0.4 records that
+configuration in every new archive and introduces a lifecycle that clearly
+separates creating, opening, saving, and reloading a store.
+
+### Explicit persistence lifecycle
+
+- Add `VectorStore.open(path)` to construct a store from a versioned archive.
+  The archive supplies its own dimensions and normalization mode, so callers no
+  longer need to repeat configuration that may be wrong.
+- Let `save(path)` perform the first save or a Save As operation and bind that
+  destination. Later `save()` calls update the bound archive.
+- Add `reload()` as a deliberate refresh from disk. It always attempts to read
+  the bound archive and leaves current in-memory state unchanged if reading or
+  validation fails.
+- Keep creating a new in-memory store separate from opening one on disk. This
+  makes file access and persistence boundaries visible in application code.
+
+### Versioned, self-describing archives
+
+- Write archive format version 1 with `format_version`, `dimensions`,
+  `normalize`, `vectors`, and `metadata` fields.
+- Validate the complete archive before changing live store state, including
+  field names, scalar configuration, array dtypes and shapes, row counts,
+  finite vector values, and normalized-store zero-vector rules.
+- Reject unsupported format versions and malformed archives clearly rather
+  than inferring missing configuration or partially applying valid fields.
+- Continue preserving each metadata item as one opaque row payload.
+
+### Safer archive replacement
+
+- Write each save to a uniquely named temporary archive in the destination
+  directory, close it, and then replace the destination with `os.replace`.
+- Preserve the previous complete archive when writing or replacement fails and
+  clean up temporary files after failures.
+- Bind a new Save As destination only after its archive has been written
+  successfully.
+
+This provides an atomic visibility boundary: a reader opening the destination
+sees the previous complete archive or the new complete archive instead of a
+partially written file. It does not provide file locking, multi-writer
+coordination, or a universal power-loss durability guarantee.
+
+### Short migration window
+
+- Keep constructor `file_path=`, instance `load()`, and direct context-manager
+  persistence for version 0.4 with `FutureWarning`. They will be removed in
+  0.5.
+- Make the deprecated context manager save only after a successful block. If
+  the block raises, it does not save or suppress the exception.
+- Keep a configuration-aware reader for older archives containing only
+  `vectors` and `metadata`. Loading one warns, and its next save rewrites it as
+  format version 1.
+- Intentionally make `open()` reject an unversioned archive because that file
+  cannot report its original dimensions or normalization semantics.
+- Add a dedicated [persistence migration guide](MIGRATION.md) with side-by-side
+  API replacements and one-time legacy archive conversion instructions.
+
+The legacy API and unversioned archive reader are removed in 0.5. Applications
+should migrate an old archive once with 0.4 or recreate it from source data;
+indefinite compatibility with the incomplete two-array format is not planned.
+
+### Runtime compatibility
+
+- Support Python 3.11 through 3.14. Python 3.10 remains supported by the 0.3
+  release series but is not supported by 0.4.
+- Raise the minimum NumPy version from 1.21.3 to 1.23.2, the earliest release
+  that supports Python 3.11.
+- Exercise every supported Python version in CI and test NumPy 1.23.2 in a
+  dedicated minimum-dependency job.
+
+### Upgrade notes and boundaries
+
+- Search, insertion, retrieval, clearing, normalization, and metadata behavior
+  are unchanged from 0.3.2.
+- Code using `VectorStore.open()`, `save(path)`, `save()`, and `reload()` is on
+  the persistence API intended for 0.5.
+- Code using a transitional entry point continues to work in 0.4 but emits a
+  warning so the required 0.5 migration is visible during testing.
+- Metadata still uses NumPy's pickle-backed object-array loading. Only open
+  archives produced by your application or another trusted source.
+- Mutable public state, repeated-add performance, deterministic tie ordering,
+  and a formal thread-safety contract remain planned for later releases.
+
+## 0.3.2 - 2026-07-27
+
+This reliability and performance patch makes existing vector storage, search,
+metadata, and persistence behavior safer and more predictable. It does not
+intentionally break valid existing usage or change the `.npz` archive format.
+
+### Numerical reliability
+
+- Reject vectors, queries, and search thresholds that contain non-finite values
+  or cannot remain finite when represented as `float32`. Invalid input now
+  fails before it can corrupt stored state or ranking.
+- Calculate norms and raw metric intermediates with `float64` where `float32`
+  could overflow or underflow. Large and very small finite vectors can now be
+  normalized and compared reliably.
+- Allow zero vectors in stores created with `normalize=False`, where they are
+  valid for dot-product and Euclidean search.
+- Raise a clear error if a raw cosine search includes a zero vector, because
+  cosine similarity is undefined for that row.
+- Avoid duplicate full-size `float64` buffers when calculating raw Euclidean
+  distance.
+
+### Persistence
+
+- Resolve a path without an `.npz` suffix to the same archive for both saving
+  and loading. For example, `file_path="vectors"` consistently uses
+  `vectors.npz`.
+- Allow `load()` to be retried when the persistence file did not exist during
+  an earlier attempt.
+- Reset load state in `clear()` so a subsequent explicit `load()` can restore
+  the saved rows.
+- Keep repeated `load()` calls idempotent after a successful load.
+
+### Metadata
+
+- Preserve each item in the outer metadata sequence as one opaque row payload.
+  Tuples and lists are no longer mistaken for extra NumPy array dimensions.
+- Support dictionary, dataclass, tuple, list, string, integer, and other scalar
+  payloads consistently through insertion, search results, saving, and loading.
+- Continue rejecting explicitly multidimensional NumPy metadata arrays rather
+  than silently flattening ambiguous input.
+
+### Search memory use
+
+- Search the stored vector matrix directly when `within_rows` is omitted,
+  avoiding an unnecessary full-matrix copy on every unfiltered query.
+- Preserve original store indexes and metadata when `within_rows` selects a
+  filtered subset.
+- Document that filtered searches allocate a temporary matrix proportional to
+  the selected row count and vector dimensions.
+
+### Compatibility and validation
+
+- Test Python 3.10 through 3.14 in GitHub Actions.
+- Test the minimum supported NumPy version in a dedicated Python 3.10 job.
+- Raise the minimum NumPy requirement from 1.20 to 1.21.3 so it is compatible
+  with the oldest supported Python version.
+- Require linting, formatting, type checking, and the full Python test matrix
+  before publishing to PyPI.
+
+### Upgrade notes
+
+- No public method signatures or persisted field names changed.
+- Existing trusted `.npz` archives with `vectors` and `metadata` remain
+  readable.
+- Environments using NumPy 1.20 must upgrade to NumPy 1.21.3 or newer.
+- Inputs that previously produced `nan`, `inf`, or unreliable rankings now
+  raise `ValueError` instead.

numpy_vector_store-0.5.0/MIGRATION.md

@@ -0,0 +1,289 @@
+# Persistence migration guide
+
+Version 0.5 completes the persistence lifecycle introduced in 0.4. Creating a
+store, opening an archive, saving, and refreshing from disk now have separate,
+explicit operations. Applications that adopted the recommended 0.4 API need no
+further persistence changes.
+
+The 0.4 compatibility paths are no longer present: constructor `file_path=`,
+instance `load()`, context-manager persistence, and the unversioned archive
+reader have been removed.
+
+## API at a glance
+
+```python
+store = VectorStore(dimensions=1536, normalize=True)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+
+store.save()  # Update the bound archive.
+
+loaded = VectorStore.open("vectors.npz")
+loaded.reload()  # Deliberately discard memory and reread the archive.
+```
+
+The persistence changes from 0.4 to 0.5 are:
+
+| Area | 0.4 | 0.5 |
+|---|---|---|
+| Create a store | `VectorStore(dimensions, normalize=...)` | Unchanged |
+| Constructor `file_path=` | Works with `FutureWarning` | Removed |
+| Open an archive | `VectorStore.open(path)` | Unchanged |
+| Save and bind | `save(path)` | Unchanged |
+| Save again | `save()` | Unchanged |
+| Refresh from disk | `reload()` | Unchanged |
+| Instance `load()` | Works with `FutureWarning` | Removed |
+| Context manager | Works with `FutureWarning` | Removed |
+| Unversioned archive | Temporary migration reader | Reader removed |
+| Format version 1 archive | Supported | Supported unchanged |
+| `dimensions`, `normalize`, `file_path` | Writable attributes | Read-only properties |
+| `vectors`, `metadata` | Writable owning arrays | Read-only inspection views |
+| Vector returned by `get()` | View into live storage | Independent `float32` copy |
+| Repeated `add()` calls | Recopy all existing rows | Reuse private spare capacity |
+| Equal search values | Unspecified order | Lower original store row index first |
+| Thread safety | Not formally defined | Concurrent reads only while state is unchanged |
+
+## Store-owned state
+
+Most code that reads configuration or uses NumPy operations for inspection and
+prefiltering remains unchanged. The familiar property names are still present:
+
+```python
+store.dimensions
+store.normalize
+store.file_path
+store.vectors
+store.metadata
+```
+
+The difference is ownership. Configuration cannot be assigned directly, and
+the vector and metadata views reject normal row mutation. Their views describe
+the rows present when the property was requested, so code should request a
+fresh view after `add()`, `clear()`, or `reload()`.
+
+The views are a supported inspection boundary, not tamper-proof snapshots.
+Deliberately mutating their backing storage through `.base`, private attributes,
+`ctypes`, or similar escape hatches is unsupported and can corrupt the store.
+Call `.copy()` when code needs an independently mutable full-array snapshot.
+For metadata, this copies the outer row array while preserving the opaque
+payload objects by reference.
+
+Row retrieval deliberately treats vectors and metadata differently:
+
+```python
+vector, payload = store.get(0)
+
+vector[0] = 10.0           # Independent copy; the store is unchanged.
+payload["reviewed"] = True  # Shared application metadata object.
+```
+
+Vectors have one uniform NumPy representation, so copying a single row gives
+the caller clear ownership at bounded cost. Metadata can be any Python object,
+so the store preserves payload identity instead of imposing a potentially
+expensive or invalid deep-copy policy. Applications that need immutable
+metadata can use frozen application objects or copy payloads themselves.
+
+## Repeated additions
+
+The `add(vectors, metadata)` signature and validation rules do not change. In
+0.4, every noninitial call concatenated the new rows with the complete store.
+In 0.5, the store reserves private row capacity and grows it geometrically when
+needed. Repeated small additions therefore copy existing rows only when the
+current allocation is full.
+
+This does not add a public capacity setting or change row indexes. `len(store)`,
+inspection, search, retrieval, and persistence continue to use only active
+rows. `clear()` discards reserved storage as well as active rows, so it does not
+retain metadata payloads through unused capacity. A caller-held inspection view
+can still keep its previous NumPy buffer and payload references alive until the
+view itself is released.
+
+## Search result ordering
+
+Version 0.5 makes exact metric ties deterministic. Cosine and dot-product
+searches still rank larger values first, and Euclidean search still ranks
+smaller distances first. When computed values are equal, the lower original
+store row index comes first.
+
+This tie break also decides which rows are returned when equal values cross
+the `top_k` boundary. A shuffled `within_rows` input does not change the result:
+the original store indexes, rather than positions in the filtered input, break
+the tie. No call-site change is required, but code that depended on an
+incidental NumPy partition order should update its expectations.
+
+## Thread safety
+
+Version 0.5 defines the existing synchronization boundary without adding
+internal locks. Search, `get()`, and inspection may run concurrently on one
+instance only while store state and shared metadata payloads remain unchanged.
+
+Applications must externally synchronize all access to an instance whenever
+any thread may call `add()`, `clear()`, `reload()`, or `save()`, or mutate a
+metadata payload shared with the store. Saves must not overlap in-memory
+mutation because archive replacement protects the destination path, not the
+consistency of vectors and metadata read from a changing instance.
+
+Atomic replacement also does not coordinate separate store instances writing
+the same path. Applications with multiple writers must serialize them; without
+that coordination, each complete save may replace another and the last
+successful replacement wins.
+
+## Creating and saving a new store
+
+Previously, the destination was supplied while constructing the store:
+
+```python
+store = VectorStore(dimensions=1536, file_path="vectors.npz")
+store.add(vectors, metadata)
+store.save()
+```
+
+Create the in-memory store first, then bind its destination with the first
+save:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Later `save()` calls update the bound archive. Supplying another path performs
+a Save As operation and binds the new destination after the write succeeds.
+Calling `save()` before a store is bound raises `ValueError` rather than
+silently leaving the data unsaved.
+
+## Opening an existing archive
+
+The old API required callers to repeat configuration that should belong to the
+archive:
+
+```python
+store = VectorStore(
+    dimensions=1536,
+    file_path="vectors.npz",
+    normalize=True,
+)
+store.load()
+```
+
+Open a version 1 archive directly:
+
+```python
+store = VectorStore.open("vectors.npz")
+```
+
+`open()` restores `dimensions` and `normalize` from the archive, validates its
+contents, loads its rows, and binds its path. Applications no longer need to
+keep archive configuration separately or risk loading the same vectors with
+different semantics.
+
+The generic parameter still describes application metadata. It can be kept
+when useful:
+
+```python
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class ChunkMetadata:
+    source: str
+    chunk_index: int
+
+
+store = VectorStore[ChunkMetadata].open("vectors.npz")
+```
+
+`ChunkMetadata` is an example application type, not a class provided by this
+library.
+
+## Refreshing from disk
+
+Use `reload()` when another process may have changed the bound archive and the
+current in-memory changes should be discarded:
+
+```python
+store = VectorStore.open("vectors.npz")
+
+# Later, after the file may have changed:
+store.reload()
+```
+
+`reload()` always attempts to read. It raises if the store is unbound, the file
+is missing, or the archive is invalid. A failed reload leaves the current
+in-memory vectors and metadata unchanged.
+
+## Replacing context-manager persistence
+
+The earlier context manager saved automatically on exit:
+
+```python
+with VectorStore(dimensions=1536, file_path="vectors.npz") as store:
+    store.add(vectors, metadata)
+```
+
+Use an explicit save after the work succeeds:
+
+```python
+store = VectorStore(dimensions=1536)
+store.add(vectors, metadata)
+store.save("vectors.npz")
+```
+
+Normal Python control flow already prevents the final line from running if
+`add()` raises. The persistence boundary is visible, and readers do not need to
+remember an implicit exit side effect.
+
+There is no replacement autosave context manager. An explicit `save()` keeps
+the persistence boundary visible and lets the application decide whether work
+completed successfully enough to persist.
+
+## Migrating an archive created before 0.4
+
+Older archives contain only `vectors` and `metadata`. They do not record their
+dimensions or whether vectors use normalized or raw semantics, so `open()`
+cannot construct a correct store from them.
+
+Before upgrading to 0.5, use the 0.4 compatibility API once with the archive's
+original configuration:
+
+```python
+legacy = VectorStore(
+    dimensions=1536,
+    file_path="legacy-vectors.npz",
+    normalize=True,
+)
+legacy.load()
+legacy.save()
+```
+
+This code emits transition warnings in 0.4 by design. The final `save()`
+rewrites the archive as format version 1 with `format_version`, `dimensions`,
+`normalize`, `vectors`, and `metadata`. It can then be opened by 0.5:
+
+```python
+store = VectorStore.open("legacy-vectors.npz")
+```
+
+Applications that can recreate archives from source vectors and metadata may
+do that instead. NumPy Vector Store 0.5 cannot perform this conversion because
+the unversioned file does not contain enough information to reconstruct its
+configuration safely.
+
+## Removal schedule
+
+| Transitional behavior | 0.4 | 0.5 |
+|---|---|---|
+| Constructor `file_path=` | Works with `FutureWarning` | Removed |
+| Instance `load()` | Works with `FutureWarning` | Removed |
+| Direct context-manager persistence | Saves only on successful exit and warns | Removed |
+| Unversioned two-array archives | Load with known configuration and warn | Reader removed |
+| `open()`, `save(path)`, `save()`, and `reload()` | Preferred | Supported |
+
+## Persistence boundaries that do not change
+
+Metadata is stored in a pickle-backed NumPy object array. Archives remain
+trusted input and must not be opened from untrusted or unverifiable sources.
+
+Saves use same-directory temporary files and atomic replacement, but the
+library does not add file locking, coordinate concurrent writers, or promise
+power-loss durability across every operating system and filesystem.