vicinity 0.3.4__tar.gz → 0.4.1__tar.gz

{vicinity-0.3.4 → vicinity-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: vicinity
-Version: 0.3.4
+Version: 0.4.1
 Summary: Lightweight Nearest Neighbors with Flexible Backends
 Author-email: Stéphan Tulkens <stephantul@gmail.com>, Thomas van Dongen <thomas123@live.nl>
 License: MIT License
@@ -54,6 +54,10 @@ Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-coverage; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
 Requires-Dist: setuptools; extra == "dev"
+Provides-Extra: huggingface
+Requires-Dist: datasets; extra == "huggingface"
+Provides-Extra: integrations
+Requires-Dist: datasets; extra == "integrations"
 Provides-Extra: hnsw
 Requires-Dist: hnswlib; extra == "hnsw"
 Provides-Extra: pynndescent
@@ -69,7 +73,18 @@ Provides-Extra: usearch
 Requires-Dist: usearch; extra == "usearch"
 Provides-Extra: voyager
 Requires-Dist: voyager; extra == "voyager"
+Provides-Extra: backends
+Requires-Dist: hnswlib; extra == "backends"
+Requires-Dist: pynndescent>=0.5.10; extra == "backends"
+Requires-Dist: numba>=0.59.0; extra == "backends"
+Requires-Dist: llvmlite>=0.42.0; extra == "backends"
+Requires-Dist: numpy>=1.24.0; extra == "backends"
+Requires-Dist: annoy; extra == "backends"
+Requires-Dist: faiss-cpu; extra == "backends"
+Requires-Dist: usearch; extra == "backends"
+Requires-Dist: voyager; extra == "backends"
 Provides-Extra: all
+Requires-Dist: datasets; extra == "all"
 Requires-Dist: hnswlib; extra == "all"
 Requires-Dist: pynndescent>=0.5.10; extra == "all"
 Requires-Dist: numba>=0.59.0; extra == "all"
@@ -97,14 +112,20 @@ Requires-Dist: voyager; extra == "all"
     <a href="https://pypi.org/project/vicinity/"><img src="https://img.shields.io/pypi/v/vicinity?color=%23007ec6&label=pypi%20package" alt="Package version"></a>
     <a href="https://pypi.org/project/vicinity/"><img src="https://img.shields.io/pypi/pyversions/vicinity" alt="Supported Python versions"></a>
     <a href="https://pepy.tech/project/vicinity">
-    <img src="https://static.pepy.tech/badge/vicinity" alt="Downloads">
+      <img src="https://static.pepy.tech/badge/vicinity" alt="Downloads">
     </a>
     <a href="https://app.codecov.io/gh/MinishLab/vicinity">
-    <img src="https://codecov.io/gh/MinishLab/vicinity/graph/badge.svg?token=0MQ2945OZL" alt="Codecov">
+      <img src="https://codecov.io/gh/MinishLab/vicinity/graph/badge.svg?token=0MQ2945OZL" alt="Codecov">
+    </a>
+    <a href="https://discord.gg/4BDPR5nmtK">
+      <img src="https://img.shields.io/badge/Join-Discord-5865F2?logo=discord&logoColor=white" alt="Join Discord">
+    </a>
+    <a href="https://github.com/MinishLab/vicinity/blob/main/LICENSE">
+      <img src="https://img.shields.io/badge/license-MIT-green" alt="License - MIT">
     </a>
-    <a href="https://github.com/MinishLab/vicinity/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License - MIT"></a>
   </h2>
 
+
 [Quickstart](#quickstart) •
 [Main Features](#main-features) •
 [Supported Backends](#supported-backends) •
@@ -112,13 +133,11 @@ Requires-Dist: voyager; extra == "all"
 
 </div>
 
-
 Vicinity is a light-weight, low-dependency vector store. It provides a simple and intuitive interface for nearest neighbor search, with support for different backends and evaluation.
 
 There are many nearest neighbors packages and methods out there. However, we found it difficult to compare them. Every package has its own interface, quirks, and limitations, and learning a new package can be time-consuming. In addition to that, how do you effectively evaluate different packages? How do you know which one is the best for your use case?
 
-
- This is where Vicinity comes in. Instead of learning a new interface for each new package or backend, Vicinity provides a unified interface for all backends. This allows you to easily experiment with different indexing methods and distance metrics and choose the best one for your use case. Vicinity also provides a simple way to evaluate the performance of different backends, allowing you to measure the queries per second and recall.
+This is where Vicinity comes in. Instead of learning a new interface for each new package or backend, Vicinity provides a unified interface for all backends. This allows you to easily experiment with different indexing methods and distance metrics and choose the best one for your use case. Vicinity also provides a simple way to evaluate the performance of different backends, allowing you to measure the queries per second and recall.
 
 ## Quickstart
 
@@ -126,18 +145,18 @@ Install the package with:
 ```bash
 pip install vicinity
 ```
-Optionally, [install any of the supported backends](#installation), or simply install all of them with:
+Optionally, [install specific backends and integrations](#installation), or simply install all of them with:
 ```bash
 pip install vicinity[all]
 ```
 
-
 The following code snippet demonstrates how to use Vicinity for nearest neighbor search:
+
 ```python
 import numpy as np
 from vicinity import Vicinity, Backend, Metric
 
-# Create some dummy data
+# Create some dummy data as strings or other serializable objects
 items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"]
 vectors = np.random.rand(len(items), 128)
 
@@ -164,12 +183,21 @@ results = vicinity.query(query_vectors, k=3)
 ```
 
 Saving and loading a vector store:
+
 ```python
 vicinity.save('my_vector_store')
 vicinity = Vicinity.load('my_vector_store')
 ```
 
+Pushing and loading a vector store from the Hugging Face Hub (note that you can optionally add the model used for generating embeddings to the metadata, e.g. `vicinity.metadata["model"] = "minishlab/potion-base-8M"`):
+
+```python
+vicinity.push_to_hub(repo_id='minishlab/my-vicinity-repo')
+vicinity = Vicinity.load_from_hub(repo_id='minishlab/my-vicinity-repo')
+```
+
 Evaluating a backend:
+
 ```python
 # Use the first 1000 vectors as query vectors
 query_vectors = vectors[:1000]
@@ -182,14 +210,17 @@ qps, recall = vicinity.evaluate(
 ```
 
 ## Main Features
+
 Vicinity provides the following features:
 - Lightweight: Minimal dependencies and fast performance.
 - Flexible Backend Support: Use different backends for vector storage and search.
 - Serialization: Save and load vector stores for persistence.
+- HuggingFace Hub Integration: Push and load vector stores directly to and from the HuggingFace Hub.
 - Evaluation: Easily evaluate the performance of different backends.
 - Easy to Use: Simple and intuitive API.
 
 ## Supported Backends
+
 The following backends are supported:
 - `BASIC`: A simple (exact matching) flat index for vector storage and search.
 - [HNSW](https://github.com/nmslib/hnswlib): Hierarchical Navigable Small World Graph (HNSW) for ANN search using hnswlib.
@@ -208,8 +239,6 @@ The following backends are supported:
   - `ivfpqr`: Inverted file search with product quantizer and refinement.
 - [VOYAGER](https://github.com/spotify/voyager): Voyager is a library for performing fast approximate nearest-neighbor searches on an in-memory collection of vectors.
 
-
-
 NOTE: the ANN backends do not support dynamic deletion. To delete items, you need to recreate the index. Insertion is supported in the following backends: `FAISS`, `HNSW`, and `Usearch`. The `BASIC` backend supports both insertion and deletion.
 
 ### Backend Parameters
@@ -241,14 +270,25 @@ NOTE: the ANN backends do not support dynamic deletion. To delete items, you nee
 |                 | `m`                 | The number of connections between nodes in the tree’s internal data structure.                                                              | `16`                |
 
 ## Installation
+
 The following installation options are available:
+
 ```bash
 # Install the base package
 pip install vicinity
 
-# Install all backends
+# Install all integrations and backends
 pip install vicinity[all]
 
+# Install all integrations
+pip install vicinity[integrations]
+
+# Install specific integrations
+pip install vicinity[huggingface]
+
+# Install all backends
+pip install vicinity[backends]
+
 # Install specific backends
 pip install vicinity[annoy]
 pip install vicinity[faiss]

{vicinity-0.3.4 → vicinity-0.4.1}/README.md

@@ -15,14 +15,20 @@
     <a href="https://pypi.org/project/vicinity/"><img src="https://img.shields.io/pypi/v/vicinity?color=%23007ec6&label=pypi%20package" alt="Package version"></a>
     <a href="https://pypi.org/project/vicinity/"><img src="https://img.shields.io/pypi/pyversions/vicinity" alt="Supported Python versions"></a>
     <a href="https://pepy.tech/project/vicinity">
-    <img src="https://static.pepy.tech/badge/vicinity" alt="Downloads">
+      <img src="https://static.pepy.tech/badge/vicinity" alt="Downloads">
     </a>
     <a href="https://app.codecov.io/gh/MinishLab/vicinity">
-    <img src="https://codecov.io/gh/MinishLab/vicinity/graph/badge.svg?token=0MQ2945OZL" alt="Codecov">
+      <img src="https://codecov.io/gh/MinishLab/vicinity/graph/badge.svg?token=0MQ2945OZL" alt="Codecov">
+    </a>
+    <a href="https://discord.gg/4BDPR5nmtK">
+      <img src="https://img.shields.io/badge/Join-Discord-5865F2?logo=discord&logoColor=white" alt="Join Discord">
+    </a>
+    <a href="https://github.com/MinishLab/vicinity/blob/main/LICENSE">
+      <img src="https://img.shields.io/badge/license-MIT-green" alt="License - MIT">
     </a>
-    <a href="https://github.com/MinishLab/vicinity/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License - MIT"></a>
   </h2>
 
+
 [Quickstart](#quickstart) •
 [Main Features](#main-features) •
 [Supported Backends](#supported-backends) •
@@ -30,13 +36,11 @@
 
 </div>
 
-
 Vicinity is a light-weight, low-dependency vector store. It provides a simple and intuitive interface for nearest neighbor search, with support for different backends and evaluation.
 
 There are many nearest neighbors packages and methods out there. However, we found it difficult to compare them. Every package has its own interface, quirks, and limitations, and learning a new package can be time-consuming. In addition to that, how do you effectively evaluate different packages? How do you know which one is the best for your use case?
 
-
- This is where Vicinity comes in. Instead of learning a new interface for each new package or backend, Vicinity provides a unified interface for all backends. This allows you to easily experiment with different indexing methods and distance metrics and choose the best one for your use case. Vicinity also provides a simple way to evaluate the performance of different backends, allowing you to measure the queries per second and recall.
+This is where Vicinity comes in. Instead of learning a new interface for each new package or backend, Vicinity provides a unified interface for all backends. This allows you to easily experiment with different indexing methods and distance metrics and choose the best one for your use case. Vicinity also provides a simple way to evaluate the performance of different backends, allowing you to measure the queries per second and recall.
 
 ## Quickstart
 
@@ -44,18 +48,18 @@ Install the package with:
 ```bash
 pip install vicinity
 ```
-Optionally, [install any of the supported backends](#installation), or simply install all of them with:
+Optionally, [install specific backends and integrations](#installation), or simply install all of them with:
 ```bash
 pip install vicinity[all]
 ```
 
-
 The following code snippet demonstrates how to use Vicinity for nearest neighbor search:
+
 ```python
 import numpy as np
 from vicinity import Vicinity, Backend, Metric
 
-# Create some dummy data
+# Create some dummy data as strings or other serializable objects
 items = ["triforce", "master sword", "hylian shield", "boomerang", "hookshot"]
 vectors = np.random.rand(len(items), 128)
 
@@ -82,12 +86,21 @@ results = vicinity.query(query_vectors, k=3)
 ```
 
 Saving and loading a vector store:
+
 ```python
 vicinity.save('my_vector_store')
 vicinity = Vicinity.load('my_vector_store')
 ```
 
+Pushing and loading a vector store from the Hugging Face Hub (note that you can optionally add the model used for generating embeddings to the metadata, e.g. `vicinity.metadata["model"] = "minishlab/potion-base-8M"`):
+
+```python
+vicinity.push_to_hub(repo_id='minishlab/my-vicinity-repo')
+vicinity = Vicinity.load_from_hub(repo_id='minishlab/my-vicinity-repo')
+```
+
 Evaluating a backend:
+
 ```python
 # Use the first 1000 vectors as query vectors
 query_vectors = vectors[:1000]
@@ -100,14 +113,17 @@ qps, recall = vicinity.evaluate(
 ```
 
 ## Main Features
+
 Vicinity provides the following features:
 - Lightweight: Minimal dependencies and fast performance.
 - Flexible Backend Support: Use different backends for vector storage and search.
 - Serialization: Save and load vector stores for persistence.
+- HuggingFace Hub Integration: Push and load vector stores directly to and from the HuggingFace Hub.
 - Evaluation: Easily evaluate the performance of different backends.
 - Easy to Use: Simple and intuitive API.
 
 ## Supported Backends
+
 The following backends are supported:
 - `BASIC`: A simple (exact matching) flat index for vector storage and search.
 - [HNSW](https://github.com/nmslib/hnswlib): Hierarchical Navigable Small World Graph (HNSW) for ANN search using hnswlib.
@@ -126,8 +142,6 @@ The following backends are supported:
   - `ivfpqr`: Inverted file search with product quantizer and refinement.
 - [VOYAGER](https://github.com/spotify/voyager): Voyager is a library for performing fast approximate nearest-neighbor searches on an in-memory collection of vectors.
 
-
-
 NOTE: the ANN backends do not support dynamic deletion. To delete items, you need to recreate the index. Insertion is supported in the following backends: `FAISS`, `HNSW`, and `Usearch`. The `BASIC` backend supports both insertion and deletion.
 
 ### Backend Parameters
@@ -159,14 +173,25 @@ NOTE: the ANN backends do not support dynamic deletion. To delete items, you nee
 |                 | `m`                 | The number of connections between nodes in the tree’s internal data structure.                                                              | `16`                |
 
 ## Installation
+
 The following installation options are available:
+
 ```bash
 # Install the base package
 pip install vicinity
 
-# Install all backends
+# Install all integrations and backends
 pip install vicinity[all]
 
+# Install all integrations
+pip install vicinity[integrations]
+
+# Install specific integrations
+pip install vicinity[huggingface]
+
+# Install all backends
+pip install vicinity[backends]
+
 # Install specific backends
 pip install vicinity[annoy]
 pip install vicinity[faiss]

{vicinity-0.3.4 → vicinity-0.4.1}/pyproject.toml

@@ -42,6 +42,14 @@ dev = [
     "ruff",
     "setuptools"
 ]
+
+# Integrations
+huggingface = ["datasets"]
+integrations = [
+    "datasets"
+]
+
+# Backends
 hnsw = ["hnswlib"]
 pynndescent = [
     "pynndescent>=0.5.10",
@@ -53,7 +61,20 @@ annoy = ["annoy"]
 faiss = ["faiss-cpu"]
 usearch = ["usearch"]
 voyager = ["voyager"]
+backends = [
+    "hnswlib",
+    "pynndescent>=0.5.10",
+    "numba>=0.59.0",
+    "llvmlite>=0.42.0",
+    "numpy>=1.24.0",
+    "annoy",
+    "faiss-cpu",
+    "usearch",
+    "voyager"
+]
+
 all = [
+    "datasets",
     "hnswlib",
     "pynndescent>=0.5.10",
     "numba>=0.59.0",

{vicinity-0.3.4 → vicinity-0.4.1}/tests/conftest.py

@@ -24,7 +24,19 @@ _faiss_index_types = [
 @pytest.fixture(scope="session")
 def items() -> list[str]:
     """Fixture providing a list of item names."""
-    return [f"item{i}" for i in range(1, 10001)]
+    return [f"item{i}" if i % 2 == 0 else {"name": f"item{i}", "id": i} for i in range(1, 10001)]
+
+
+@pytest.fixture(scope="session")
+def non_serializable_items() -> list[str]:
+    """Fixture providing a list of non-serializable items."""
+
+    class NonSerializable:
+        def __init__(self, name: str, id: int) -> None:
+            self.name = name
+            self.id = id
+
+    return [NonSerializable(f"item{i}", i) for i in range(1, 10001)]
 
 
 @pytest.fixture(scope="session")

{vicinity-0.3.4 → vicinity-0.4.1}/tests/test_vicinity.py

@@ -4,6 +4,7 @@ from pathlib import Path
 
 import numpy as np
 import pytest
+from orjson import JSONEncodeError
 
 from vicinity import Vicinity
 from vicinity.datatypes import Backend
@@ -162,6 +163,21 @@ def test_vicinity_save_and_load_vector_store(tmp_path: Path, vicinity_instance_w
     assert v.vector_store is not None
 
 
+def test_vicinity_save_and_load_non_serializable_items(
+    tmp_path: Path, non_serializable_items: list[str], vectors: np.ndarray
+) -> None:
+    """
+    Test Vicinity.save and Vicinity.load with non-serializable items.
+
+    :param tmp_path: Temporary directory provided by pytest.
+    :param non_serializable_items: A list of non-serializable items.
+    """
+    vicinity = Vicinity.from_vectors_and_items(vectors=vectors, items=non_serializable_items)
+    save_path = tmp_path / "vicinity_data"
+    with pytest.raises(JSONEncodeError):
+        vicinity.save(save_path)
+
+
 def test_index_vector_store(vicinity_with_basic_backend_and_store: Vicinity, vectors: np.ndarray) -> None:
     """
     Index vectors in the Vicinity instance.
@@ -183,18 +199,17 @@ def test_index_vector_store(vicinity_with_basic_backend_and_store: Vicinity, vec
         vicinity_with_basic_backend_and_store.get_vector_by_index([-1])
 
 
-def test_vicinity_insert_duplicate(vicinity_instance: Vicinity, query_vector: np.ndarray) -> None:
+def test_vicinity_insert_duplicate(items: list[str], vicinity_instance: Vicinity, query_vector: np.ndarray) -> None:
     """
     Test that Vicinity.insert raises ValueError when inserting duplicate items.
 
     :param vicinity_instance: A Vicinity instance.
     :raises ValueError: If inserting items that already exist.
     """
-    new_items = ["item1"]
     new_vector = query_vector
 
     with pytest.raises(ValueError):
-        vicinity_instance.insert(new_items, new_vector[None, :])
+        vicinity_instance.insert(items[0], new_vector[None, :])
 
 
 def test_vicinity_delete_nonexistent(vicinity_instance: Vicinity) -> None:
@@ -281,7 +296,8 @@ def test_vicinity_delete_and_query(vicinity_instance: Vicinity, items: list[str]
         return
 
     # Delete some items from the Vicinity instance
-    items_to_delete = ["item2", "item4", "item6"]
+    non_existing_items_indices = [0, 1, 2]
+    items_to_delete = [items[i] for i in non_existing_items_indices]
     vicinity_instance.delete(items_to_delete)
 
     # Ensure the items are no longer in the items list
@@ -289,14 +305,14 @@ def test_vicinity_delete_and_query(vicinity_instance: Vicinity, items: list[str]
         assert item not in vicinity_instance.items
 
     # Query using a vector of an item that wasn't deleted
-    item3_index = items.index("item3")
-    item3_vector = vectors[item3_index]
+    existing_item_index = 3
+    item3_vector = vectors[existing_item_index]
 
     results = vicinity_instance.query(item3_vector, k=10)
     returned_items = [item for item, _ in results[0]]
 
     # Check that the queried item is in the results
-    assert "item3" in returned_items
+    assert items[existing_item_index] in returned_items
 
 
 def test_vicinity_evaluate(vicinity_instance: Vicinity, vectors: np.ndarray) -> None: