qdrant-haystack 7.0.0__tar.gz → 8.1.0__tar.gz

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/CHANGELOG.md

@@ -1,15 +1,47 @@
 # Changelog
 
+## [integrations/qdrant-v8.0.0] - 2025-02-19
+
+### 🧹 Chores
+
+- Fix linting/isort (#1215)
+- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
+- [**breaking**] Qdrant - remove dataframe support (#1403)
+
+
+## [integrations/qdrant-v7.0.0] - 2024-10-29
+
+### ⚙️ CI
+
+- Adopt uv as installer (#1142)
+
+### 🧹 Chores
+
+- Update ruff linting scripts and settings (#1105)
+
+### 🌀 Miscellaneous
+
+- Refactor!: Qdrant - remove `index` parameter from methods (#1160)
+
 ## [integrations/qdrant-v6.0.0] - 2024-09-13
 
+### 🌀 Miscellaneous
+
+- Remove support for deprecated legacy filters in Qdrant (#1084)
+
 ## [integrations/qdrant-v5.1.0] - 2024-09-12
 
 ### 🚀 Features
 
 - Qdrant - Add group_by and group_size optional parameters to Retrievers (#1054)
 
+
 ## [integrations/qdrant-v5.0.0] - 2024-09-02
 
+### 🌀 Miscellaneous
+
+- Fix!: fix type errors in `QdrantDocumentStore`; rename `ids` (parameter of `delete_documents`) to `document_ids` (#1041)
+
 ## [integrations/qdrant-v4.2.0] - 2024-08-27
 
 ### 🚜 Refactor
@@ -20,12 +52,18 @@
 
 - Do not retry tests in `hatch run test` command (#954)
 
+### 🌀 Miscellaneous
+
+- Chore: Update Qdrant tests for the new `apply_filter_policy` usage (#969)
+- Chore: qdrant - ruff update, don't ruff tests (#989)
+
 ## [integrations/qdrant-v4.1.2] - 2024-07-15
 
 ### 🐛 Bug Fixes
 
 - `qdrant` - Fallback to default filter policy when deserializing retrievers without the init parameter (#902)
 
+
 ## [integrations/qdrant-v4.1.1] - 2024-07-10
 
 ### 🚀 Features
@@ -36,6 +74,10 @@
 
 - Errors in convert_filters_to_qdrant (#870)
 
+### 🌀 Miscellaneous
+
+- Chore: Minor retriever pydoc fix (#884)
+
 ## [integrations/qdrant-v4.1.0] - 2024-07-03
 
 ### 🚀 Features
@@ -43,6 +85,7 @@
 - Add `score_threshold` to Qdrant Retrievers (#860)
 - Qdrant - add support for BM42 (#864)
 
+
 ## [integrations/qdrant-v4.0.0] - 2024-07-02
 
 ### 🚜 Refactor
@@ -50,17 +93,25 @@
 - [**breaking**] Qdrant - remove unused init parameters: `content_field`, `name_field`, `embedding_field`, and `duplicate_documents` (#861)
 - [**breaking**] Qdrant - set `scale_score` default value to `False` (#862)
 
-### ⚙️ Miscellaneous Tasks
+### ⚙️ CI
 
 - Retry tests to reduce flakyness (#836)
+
+### 🧹 Chores
+
 - Update ruff invocation to include check parameter (#853)
 
+### 🌀 Miscellaneous
+
+- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
+
 ## [integrations/qdrant-v3.8.1] - 2024-06-20
 
 ### 📚 Documentation
 
 - Added docstrings for QdrantDocumentStore (#808)
 
+
 ## [integrations/qdrant-v3.8.0] - 2024-06-06
 
 ### 🚀 Features
@@ -79,17 +130,29 @@
 
 - Defer database connection to the first usage (#748)
 
+### 🌀 Miscellaneous
+
+- Qdrant - improve docstrings for retrievers (#687)
+- Chore: change the pydoc renderer class (#718)
+- Allow vanilla qdrant filters (#692)
+
 ## [integrations/qdrant-v3.5.0] - 2024-04-24
 
+### 🌀 Miscellaneous
+
+- Chore: add license classifiers (#680)
+- Qdrant - add hybrid retriever (#675)
+
 ## [integrations/qdrant-v3.4.0] - 2024-04-23
 
-### Qdrant
+### 🌀 Miscellaneous
 
 - Add embedding retrieval example (#666)
+- Rename `QdrantSparseRetriever` to `QdrantSparseEmbeddingRetriever` (#681)
 
 ## [integrations/qdrant-v3.3.1] - 2024-04-12
 
-### Qdrant
+### 🌀 Miscellaneous
 
 - Add migration utility function for Sparse Embedding support (#659)
 
@@ -103,9 +166,7 @@
 
 ### 🐛 Bug Fixes
 
-- Fix haystack-ai pin (#649)
-
-
+- Fix `haystack-ai` pins (#649)
 
 ## [integrations/qdrant-v3.2.0] - 2024-03-27
 
@@ -116,15 +177,9 @@
 
 ### 🐛 Bug Fixes
 
-- Fix linter errors (#282)
-
-
+- Fix: fix linter errors (#282)
 - Fix order of API docs (#447)
-
-This PR will also push the docs to Readme
-- Fixes (#518)
-
-
+- Doc: fixing docstrings for qdrant (#518)
 
 ### 🚜 Refactor
 
@@ -136,13 +191,19 @@ This PR will also push the docs to Readme
 - Small consistency improvements (#536)
 - Disable-class-def (#556)
 
-### ⚙️ Miscellaneous Tasks
+### ⚙️ CI
 
 - Generate API docs for Qdrant (#361)
 
+### 🌀 Miscellaneous
+
+- Make tests show coverage (#566)
+- Remove references to Python 3.7 (#601)
+- Make Document Stores initially skip `SparseEmbedding` (#606)
+
 ## [integrations/qdrant-v3.0.0] - 2024-01-22
 
-### Refact
+### 🌀 Miscellaneous
 
 - [**breaking**] Change import paths (#255)
 
@@ -156,10 +217,12 @@ This PR will also push the docs to Readme
 
 - Fix import paths for beta5 (#237)
 
-
-
 ### 🚜 Refactor
 
 - Use `hatch_vcs` to manage integrations versioning (#103)
 
+### 🌀 Miscellaneous
+
+- Renamed QdrntRetriever to QdrntEmbeddingRetriever (#174)
+
 <!-- generated by git-cliff -->

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: qdrant-haystack
-Version: 7.0.0
+Version: 8.1.0
 Summary: An integration of Qdrant ANN vector database backend with Haystack
 Project-URL: Source, https://github.com/deepset-ai/haystack-core-integrations
 Project-URL: Documentation, https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/README.md
@@ -11,13 +11,13 @@ License-File: LICENSE.txt
 Classifier: Development Status :: 4 - Beta
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Requires-Dist: haystack-ai
 Requires-Dist: qdrant-client>=1.10.0
 Description-Content-Type: text/markdown

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/pyproject.toml

@@ -7,7 +7,7 @@ name = "qdrant-haystack"
 dynamic = ["version"]
 description = 'An integration of Qdrant ANN vector database backend with Haystack'
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 license = "Apache-2.0"
 keywords = []
 authors = [
@@ -18,10 +18,10 @@ classifiers = [
   "License :: OSI Approved :: Apache Software License",
   "Development Status :: 4 - Beta",
   "Programming Language :: Python",
-  "Programming Language :: Python :: 3.8",
   "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: Implementation :: PyPy",
 ]
@@ -56,7 +56,7 @@ cov-retry = ["test-cov-retry", "cov-report"]
 docs = ["pydoc-markdown pydoc/config.yml"]
 
 [[tool.hatch.envs.all.matrix]]
-python = ["3.8", "3.9", "3.10", "3.11"]
+python = [ "3.9", "3.10", "3.11"]
 
 [tool.hatch.envs.lint]
 installer = "uv"

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/src/haystack_integrations/components/retrievers/qdrant/__init__.py

@@ -4,4 +4,4 @@
 
 from .retriever import QdrantEmbeddingRetriever, QdrantHybridRetriever, QdrantSparseEmbeddingRetriever
 
-__all__ = ("QdrantEmbeddingRetriever", "QdrantSparseEmbeddingRetriever", "QdrantHybridRetriever")
+__all__ = ("QdrantEmbeddingRetriever", "QdrantHybridRetriever", "QdrantSparseEmbeddingRetriever")

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/src/haystack_integrations/document_stores/qdrant/converters.py

@@ -22,6 +22,15 @@ def convert_haystack_documents_to_qdrant_points(
     points = []
     for document in documents:
         payload = document.to_dict(flatten=False)
+
+        if payload.pop("dataframe", None):
+            logger.warning(
+                "Document %s has the `dataframe` field set,"
+                "QdrantDocumentStore no longer supports dataframes and this field will be ignored. "
+                "The `dataframe` field will soon be removed from Haystack Document.",
+                document.id,
+            )
+
         if use_sparse_embeddings:
             vector = {}
 
@@ -64,6 +73,14 @@ def convert_qdrant_point_to_haystack_document(point: QdrantPoint, use_sparse_emb
     payload = {**point.payload}
     payload["score"] = point.score if hasattr(point, "score") else None
 
+    if payload.pop("dataframe", None):
+        logger.warning(
+            "Document %s has the `dataframe` field set,"
+            "QdrantDocumentStore no longer supports dataframes and this field will be ignored. "
+            "The `dataframe` field will soon be removed from Haystack Document.",
+            payload["id"],
+        )
+
     if not use_sparse_embeddings:
         payload["embedding"] = point.vector if hasattr(point, "vector") else None
     elif hasattr(point, "vector") and point.vector is not None:

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/src/haystack_integrations/document_stores/qdrant/document_store.py

@@ -48,8 +48,8 @@ def get_batches_from_generator(iterable, n):
 
 class QdrantDocumentStore:
     """
-    QdrantDocumentStore is a Document Store for Qdrant.
-    It can be used with any Qdrant instance: in-memory, disk-persisted, Docker-based,
+    A QdrantDocumentStore implementation that you
+    can use with any Qdrant instance: in-memory, disk-persisted, Docker-based,
     and Qdrant Cloud Cluster deployments.
 
     Usage example by creating an in-memory instance:
@@ -866,10 +866,18 @@ class QdrantDocumentStore:
 
         collection_info = self.client.get_collection(collection_name)
 
-        has_named_vectors = (
-            isinstance(collection_info.config.params.vectors, dict)
-            and DENSE_VECTORS_NAME in collection_info.config.params.vectors
-        )
+        has_named_vectors = isinstance(collection_info.config.params.vectors, dict)
+
+        if has_named_vectors and DENSE_VECTORS_NAME not in collection_info.config.params.vectors:
+            msg = (
+                f"Collection '{collection_name}' already exists in Qdrant, "
+                f"but it has been originally created outside of Haystack and is not supported. "
+                f"If possible, you should create a new Document Store with Haystack. "
+                f"In case you want to migrate the existing collection, see an example script in "
+                f"https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/src/"
+                f"haystack_integrations/document_stores/qdrant/migrate_to_sparse.py."
+            )
+            raise QdrantStoreError(msg)
 
         if self.use_sparse_embeddings and not has_named_vectors:
             msg = (
@@ -882,7 +890,7 @@ class QdrantDocumentStore:
             )
             raise QdrantStoreError(msg)
 
-        elif not self.use_sparse_embeddings and has_named_vectors:
+        if not self.use_sparse_embeddings and has_named_vectors:
             msg = (
                 f"Collection '{collection_name}' already exists in Qdrant, "
                 f"but it has been originally created with sparse embedding vectors."

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/tests/test_converters.py

@@ -1,7 +1,10 @@
 import numpy as np
+from haystack import Document
+from pandas import DataFrame
 from qdrant_client.http import models as rest
 
 from haystack_integrations.document_stores.qdrant.converters import (
+    convert_haystack_documents_to_qdrant_points,
     convert_id,
     convert_qdrant_point_to_haystack_document,
 )
@@ -62,3 +65,44 @@ def test_point_to_document_reverts_proper_structure_from_record_without_sparse()
     assert document.sparse_embedding is None
     assert {"test_field": 1} == document.meta
     assert 0.0 == np.sum(np.array([1.0, 0.0, 0.0, 0.0]) - document.embedding)
+
+
+def test_point_to_document_skips_dataframe():
+
+    point = rest.Record(
+        id="c7c62e8e-02b9-4ec6-9f88-46bd97b628b7",
+        payload={
+            "id": "my-id",
+            "content": "Lorem ipsum",
+            "content_type": "text",
+            "meta": {
+                "test_field": 1,
+            },
+            "dataframe": {"a": [1, 2, 3]},
+        },
+        vector=[1.0, 0.0, 0.0, 0.0],
+    )
+    document = convert_qdrant_point_to_haystack_document(point, use_sparse_embeddings=False)
+    assert "my-id" == document.id
+    assert "Lorem ipsum" == document.content
+    assert "text" == document.content_type
+    assert {"test_field": 1} == document.meta
+    assert 0.0 == np.sum(np.array([1.0, 0.0, 0.0, 0.0]) - document.embedding)
+    assert not hasattr(document, "dataframe") or document.dataframe is None
+
+
+def test_documents_to_points_skips_dataframe():
+    doc = Document(
+        id="my-id",
+        content="Lorem ipsum",
+        embedding=[1.0, 0.0, 0.0, 0.0],
+    )
+
+    doc.dataframe = DataFrame([[1, 2], [3, 4]])
+
+    points = convert_haystack_documents_to_qdrant_points([doc], use_sparse_embeddings=False)
+    assert len(points) == 1
+
+    assert points[0].payload["content"] == "Lorem ipsum"
+    assert points[0].vector == [1.0, 0.0, 0.0, 0.0]
+    assert "dataframe" not in points[0].payload

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/tests/test_document_store.py

@@ -1,5 +1,5 @@
 from typing import List
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
 
 import pytest
 from haystack import Document
@@ -15,6 +15,7 @@ from haystack.testing.document_store import (
 from qdrant_client.http import models as rest
 
 from haystack_integrations.document_stores.qdrant.document_store import (
+    DENSE_VECTORS_NAME,
     SPARSE_VECTORS_NAME,
     QdrantDocumentStore,
     QdrantStoreError,
@@ -151,3 +152,79 @@ class TestQdrantDocumentStore(CountDocumentsTest, WriteDocumentsTest, DeleteDocu
 
             with pytest.raises(QdrantStoreError):
                 document_store._query_hybrid(query_sparse_embedding=sparse_embedding, query_embedding=embedding)
+
+    def test_set_up_collection_with_existing_incompatible_collection(self):
+        document_store = QdrantDocumentStore(location=":memory:", use_sparse_embeddings=True)
+
+        # Mock collection info with named vectors but missing DENSE_VECTORS_NAME
+        mock_collection_info = MagicMock()
+        mock_collection_info.config.params.vectors = {"some_other_vector": MagicMock()}
+
+        with patch.object(document_store.client, "collection_exists", return_value=True), patch.object(
+            document_store.client, "get_collection", return_value=mock_collection_info
+        ):
+
+            with pytest.raises(QdrantStoreError, match="created outside of Haystack"):
+                document_store._set_up_collection("test_collection", 768, False, "cosine", True, False)
+
+    def test_set_up_collection_use_sparse_embeddings_true_without_named_vectors(self):
+        """Test that an error is raised when use_sparse_embeddings is True but collection doesn't have named vectors"""
+        document_store = QdrantDocumentStore(location=":memory:", use_sparse_embeddings=True)
+
+        # Mock collection info without named vectors
+        mock_collection_info = MagicMock()
+        mock_collection_info.config.params.vectors = MagicMock(spec=rest.VectorsConfig)
+
+        with patch.object(document_store.client, "collection_exists", return_value=True), patch.object(
+            document_store.client, "get_collection", return_value=mock_collection_info
+        ):
+
+            with pytest.raises(QdrantStoreError, match="without sparse embedding vectors"):
+                document_store._set_up_collection("test_collection", 768, False, "cosine", True, False)
+
+    def test_set_up_collection_use_sparse_embeddings_false_with_named_vectors(self):
+        """Test that an error is raised when use_sparse_embeddings is False but collection has named vectors"""
+        document_store = QdrantDocumentStore(location=":memory:", use_sparse_embeddings=False)
+
+        # Mock collection info with named vectors
+        mock_collection_info = MagicMock()
+        mock_collection_info.config.params.vectors = {DENSE_VECTORS_NAME: MagicMock()}
+
+        with patch.object(document_store.client, "collection_exists", return_value=True), patch.object(
+            document_store.client, "get_collection", return_value=mock_collection_info
+        ):
+
+            with pytest.raises(QdrantStoreError, match="with sparse embedding vectors"):
+                document_store._set_up_collection("test_collection", 768, False, "cosine", False, False)
+
+    def test_set_up_collection_with_distance_mismatch(self):
+        document_store = QdrantDocumentStore(location=":memory:", use_sparse_embeddings=False, similarity="cosine")
+
+        # Mock collection info with different distance
+        mock_collection_info = MagicMock()
+        mock_collection_info.config.params.vectors = MagicMock()
+        mock_collection_info.config.params.vectors.distance = rest.Distance.DOT
+        mock_collection_info.config.params.vectors.size = 768
+
+        with patch.object(document_store.client, "collection_exists", return_value=True), patch.object(
+            document_store.client, "get_collection", return_value=mock_collection_info
+        ):
+
+            with pytest.raises(ValueError, match="different similarity"):
+                document_store._set_up_collection("test_collection", 768, False, "cosine", False, False)
+
+    def test_set_up_collection_with_dimension_mismatch(self):
+        document_store = QdrantDocumentStore(location=":memory:", use_sparse_embeddings=False, similarity="cosine")
+
+        # Mock collection info with different vector size
+        mock_collection_info = MagicMock()
+        mock_collection_info.config.params.vectors = MagicMock()
+        mock_collection_info.config.params.vectors.distance = rest.Distance.COSINE
+        mock_collection_info.config.params.vectors.size = 512
+
+        with patch.object(document_store.client, "collection_exists", return_value=True), patch.object(
+            document_store.client, "get_collection", return_value=mock_collection_info
+        ):
+
+            with pytest.raises(ValueError, match="different vector size"):
+                document_store._set_up_collection("test_collection", 768, False, "cosine", False, False)

{qdrant_haystack-7.0.0 → qdrant_haystack-8.1.0}/tests/test_filters.py

@@ -208,11 +208,5 @@ class TestQdrantStoreBaseTests(FilterDocumentsTest):
 
     # ======== ========================== ========
 
-    @pytest.mark.skip(reason="Qdrant doesn't support comparision with dataframe")
-    def test_comparison_equal_with_dataframe(self, document_store, filterable_docs): ...
-
-    @pytest.mark.skip(reason="Qdrant doesn't support comparision with dataframe")
-    def test_comparison_not_equal_with_dataframe(self, document_store, filterable_docs): ...
-
     @pytest.mark.skip(reason="Cannot distinguish errors yet")
     def test_missing_top_level_operator_key(self, document_store, filterable_docs): ...