weaviate-haystack 6.2.0__tar.gz → 6.3.0__tar.gz

{weaviate_haystack-6.2.0 → weaviate_haystack-6.3.0}/CHANGELOG.md

@@ -1,181 +1,193 @@
-# Changelog
-
-## [integrations/weaviate-v6.1.0] - 2025-06-20
-
-### 🐛 Bug Fixes
-
-- Weaviate - fix types + add py.typed (#1977)
-
-
-### ⚙️ CI
-
-- Review testing workflows (#1541)
-
-### 🧹 Chores
-
-- Align core-integrations Hatch scripts (#1898)
-- Update md files for new hatch scripts (#1911)
-
-## [integrations/weaviate-v6.0.0] - 2025-03-11
-
-
-### 🧹 Chores
-
-- Remove Python 3.8 support (#1421)
-- Weaviate - pin haystack and remove dataframe checks (#1521)
-
-### 🌀 Miscellaneous
-
-- Docs: Update document store descriptions for deepset Pipeline Builder (#1447)
-
-## [integrations/weaviate-v5.0.0] - 2025-02-17
-
-### 🧹 Chores
-
-- Fix linting/isort (#1215)
-- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
-- [**breaking**] Weaviate - remove dataframe support (#1406)
-
-
-## [integrations/weaviate-v4.0.2] - 2024-11-13
-
-### 🐛 Bug Fixes
-
-- Dependency for weaviate document store (#1186)
-
-
-## [integrations/weaviate-v4.0.1] - 2024-11-11
-
-### 🌀 Miscellaneous
-
-- Fix: Weaviate - skip `_split_overlap` meta field (#1173)
-
-## [integrations/weaviate-v4.0.0] - 2024-10-18
-
-### 🐛 Bug Fixes
-
-- Compatibility with Weaviate 4.9.0 (#1143)
-
-### ⚙️ CI
-
-- Adopt uv as installer (#1142)
-
-### 🧹 Chores
-
-- Weaviate - remove legacy filter support (#1070)
-- Update changelog after removing legacy filters (#1083)
-- Update ruff linting scripts and settings (#1105)
-
-
-## [integrations/weaviate-v2.2.1] - 2024-09-07
-
-### 🚀 Features
-
-- Add filter_policy to weaviate integration (#824)
-
-### 🐛 Bug Fixes
-
-- Weaviate filter error (#811)
-- Fix connection to Weaviate Cloud Service (#624)
-- Pin weaviate-client (#1046)
-- Weaviate - fix connection issues with some WCS URLs (#1058)
-
-### 🧪 Testing
-
-- Do not retry tests in `hatch run test` command (#954)
-
-### ⚙️ CI
-
-- Retry tests to reduce flakyness (#836)
-
-### 🧹 Chores
-
-- Update ruff invocation to include check parameter (#853)
-
-### 🌀 Miscellaneous
-
-- Handle connection to WCS and add tests
-- Revert "Handle connection to WCS and add tests"
-
-This reverts commit f48802b2ce612896fd06a13cf33dffd9f77a8859.
-- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
-- Chore: Minor retriever pydoc fix (#884)
-- Fix: `weaviate` - Fallback to default filter policy when deserializing retrievers without the init parameter  (#903)
-- Fix: Weaviate: Use collections.list_all instead of collections._get_all (#921)
-- Chore: weaviate - ruff update, don't ruff tests (#986)
-
-## [integrations/weaviate-v2.1.0] - 2024-06-10
-
-### 🚀 Features
-
-- Defer the database connection to when it's needed (#802)
-
-### 🐛 Bug Fixes
-
-- Weaviate schema class name conversion which preserves PascalCase (#707)
-
-### 🌀 Miscellaneous
-
-- Chore: add license classifiers (#680)
-- Chore: change the pydoc renderer class (#718)
-
-## [integrations/weaviate-v2.0.0] - 2024-03-25
-
-### 📚 Documentation
-
-- Disable-class-def (#556)
-- Fix docstrings (#586)
-
-### 🌀 Miscellaneous
-
-- Make tests show coverage (#566)
-- Migrate from weaviate python client v3  to v4 (#463)
-- Refactor tests (#574)
-- Remove references to Python 3.7 (#601)
-- Make Document Stores initially skip `SparseEmbedding` (#606)
-- Fix Weaviate retrievers returning no score (#621)
-- Update Weaviate docstrings (#622)
-
-## [integrations/weaviate-v1.0.2] - 2024-02-27
-
-### 🐛 Bug Fixes
-
-- Fix order of API docs (#447)
-- Weaviate: fix auth tests (#488)
-
-### 📚 Documentation
-
-- Update category slug (#442)
-
-### 🌀 Miscellaneous
-
-- Make retrievers return dicts (#491)
-
-## [integrations/weaviate-v1.0.0] - 2024-02-15
-
-### 🚀 Features
-
-- Generate weaviate API docs (#351)
-
-### 🌀 Miscellaneous
-
-- WeaviateDocumentStore initialization and serialization (#187)
-- Move package under haystack_integrations (#214)
-- Add `collection_name` parameter and creation (#215)
-- Support more collection settings when creating a new `WeaviateDocumentStore` (#260)
-- Implement `count_document` for WeaviateDocumentStore (#267)
-- Add methods to convert from Document to Weaviate data object and viceversa (#269)
-- Add filter, write and delete documents in Weaviate (#270)
-- Implement filtering for `WeaviateDocumentStore` (#278)
-- Add `WeaviateBM25Retriever` (#410)
-- Add `WeaviateEmbeddingRetriever` (#412)
-- Update Weaviate docs configs (#414)
-- Update WeaviateDocumentStore authentication to use new Secret class (#425)
-
-## [integrations/weaviate-v0.0.0] - 2024-01-10
-
-### 🌀 Miscellaneous
-
-- Setup everything to start working on the Weaviate integration (#186)
-
-<!-- generated by git-cliff -->
+# Changelog
+
+## [integrations/weaviate-v6.2.0] - 2025-09-18
+
+### 🚀 Features
+
+- Weaviate Hybrid Retrieval (#2276)
+
+### 🧹 Chores
+
+- Remove black (#1985)
+- Standardize readmes - part 2 (#2205)
+
+
+## [integrations/weaviate-v6.1.0] - 2025-06-20
+
+### 🐛 Bug Fixes
+
+- Weaviate - fix types + add py.typed (#1977)
+
+
+### ⚙️ CI
+
+- Review testing workflows (#1541)
+
+### 🧹 Chores
+
+- Align core-integrations Hatch scripts (#1898)
+- Update md files for new hatch scripts (#1911)
+
+## [integrations/weaviate-v6.0.0] - 2025-03-11
+
+
+### 🧹 Chores
+
+- Remove Python 3.8 support (#1421)
+- Weaviate - pin haystack and remove dataframe checks (#1521)
+
+### 🌀 Miscellaneous
+
+- Update document store descriptions (#1447)
+
+## [integrations/weaviate-v5.0.0] - 2025-02-17
+
+### 🧹 Chores
+
+- Fix linting/isort (#1215)
+- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
+- [**breaking**] Weaviate - remove dataframe support (#1406)
+
+
+## [integrations/weaviate-v4.0.2] - 2024-11-13
+
+### 🐛 Bug Fixes
+
+- Dependency for weaviate document store (#1186)
+
+
+## [integrations/weaviate-v4.0.1] - 2024-11-11
+
+### 🌀 Miscellaneous
+
+- Fix: Weaviate - skip `_split_overlap` meta field (#1173)
+
+## [integrations/weaviate-v4.0.0] - 2024-10-18
+
+### 🐛 Bug Fixes
+
+- Compatibility with Weaviate 4.9.0 (#1143)
+
+### ⚙️ CI
+
+- Adopt uv as installer (#1142)
+
+### 🧹 Chores
+
+- Weaviate - remove legacy filter support (#1070)
+- Update changelog after removing legacy filters (#1083)
+- Update ruff linting scripts and settings (#1105)
+
+
+## [integrations/weaviate-v2.2.1] - 2024-09-07
+
+### 🚀 Features
+
+- Add filter_policy to weaviate integration (#824)
+
+### 🐛 Bug Fixes
+
+- Weaviate filter error (#811)
+- Fix connection to Weaviate Cloud Service (#624)
+- Pin weaviate-client (#1046)
+- Weaviate - fix connection issues with some WCS URLs (#1058)
+
+### 🧪 Testing
+
+- Do not retry tests in `hatch run test` command (#954)
+
+### ⚙️ CI
+
+- Retry tests to reduce flakyness (#836)
+
+### 🧹 Chores
+
+- Update ruff invocation to include check parameter (#853)
+
+### 🌀 Miscellaneous
+
+- Handle connection to WCS and add tests
+- Revert "Handle connection to WCS and add tests"
+
+This reverts commit f48802b2ce612896fd06a13cf33dffd9f77a8859.
+- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
+- Chore: Minor retriever pydoc fix (#884)
+- Fix: `weaviate` - Fallback to default filter policy when deserializing retrievers without the init parameter  (#903)
+- Fix: Weaviate: Use collections.list_all instead of collections._get_all (#921)
+- Chore: weaviate - ruff update, don't ruff tests (#986)
+
+## [integrations/weaviate-v2.1.0] - 2024-06-10
+
+### 🚀 Features
+
+- Defer the database connection to when it's needed (#802)
+
+### 🐛 Bug Fixes
+
+- Weaviate schema class name conversion which preserves PascalCase (#707)
+
+### 🌀 Miscellaneous
+
+- Chore: add license classifiers (#680)
+- Chore: change the pydoc renderer class (#718)
+
+## [integrations/weaviate-v2.0.0] - 2024-03-25
+
+### 📚 Documentation
+
+- Disable-class-def (#556)
+- Fix docstrings (#586)
+
+### 🌀 Miscellaneous
+
+- Make tests show coverage (#566)
+- Migrate from weaviate python client v3  to v4 (#463)
+- Refactor tests (#574)
+- Remove references to Python 3.7 (#601)
+- Make Document Stores initially skip `SparseEmbedding` (#606)
+- Fix Weaviate retrievers returning no score (#621)
+- Update Weaviate docstrings (#622)
+
+## [integrations/weaviate-v1.0.2] - 2024-02-27
+
+### 🐛 Bug Fixes
+
+- Fix order of API docs (#447)
+- Weaviate: fix auth tests (#488)
+
+### 📚 Documentation
+
+- Update category slug (#442)
+
+### 🌀 Miscellaneous
+
+- Make retrievers return dicts (#491)
+
+## [integrations/weaviate-v1.0.0] - 2024-02-15
+
+### 🚀 Features
+
+- Generate weaviate API docs (#351)
+
+### 🌀 Miscellaneous
+
+- WeaviateDocumentStore initialization and serialization (#187)
+- Move package under haystack_integrations (#214)
+- Add `collection_name` parameter and creation (#215)
+- Support more collection settings when creating a new `WeaviateDocumentStore` (#260)
+- Implement `count_document` for WeaviateDocumentStore (#267)
+- Add methods to convert from Document to Weaviate data object and viceversa (#269)
+- Add filter, write and delete documents in Weaviate (#270)
+- Implement filtering for `WeaviateDocumentStore` (#278)
+- Add `WeaviateBM25Retriever` (#410)
+- Add `WeaviateEmbeddingRetriever` (#412)
+- Update Weaviate docs configs (#414)
+- Update WeaviateDocumentStore authentication to use new Secret class (#425)
+
+## [integrations/weaviate-v0.0.0] - 2024-01-10
+
+### 🌀 Miscellaneous
+
+- Setup everything to start working on the Weaviate integration (#186)
+
+<!-- generated by git-cliff -->

{weaviate_haystack-6.2.0 → weaviate_haystack-6.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: weaviate-haystack
-Version: 6.2.0
+Version: 6.3.0
 Summary: An integration of Weaviate vector database with Haystack
 Project-URL: Source, https://github.com/deepset-ai/haystack-core-integrations
 Project-URL: Documentation, https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/weaviate/README.md

{weaviate_haystack-6.2.0 → weaviate_haystack-6.3.0}/docker-compose.yml

@@ -8,11 +8,11 @@ services:
     - '8080'
     - --scheme
     - http
-    image: semitechnologies/weaviate:1.24.5
+    image: semitechnologies/weaviate:1.30.17
     ports:
     - 8080:8080
     - 50051:50051
-    restart: on-failure:0
+    restart: 'no'
     environment:
       QUERY_DEFAULTS_LIMIT: 25
       AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'

{weaviate_haystack-6.2.0 → weaviate_haystack-6.3.0}/src/haystack_integrations/document_stores/weaviate/document_store.py

@@ -505,6 +505,55 @@ class WeaviateDocumentStore:
         weaviate_ids = [generate_uuid5(doc_id) for doc_id in document_ids]
         self.collection.data.delete_many(where=weaviate.classes.query.Filter.by_id().contains_any(weaviate_ids))
 
+    def delete_all_documents(self, *, recreate_index: bool = False, batch_size: int = 1000) -> None:
+        """
+        Deletes all documents in a collection.
+
+        If recreate_index is False, it keeps the collection but deletes documents iteratively.
+        If recreate_index is True, the collection is dropped and faithfully recreated.
+        This is recommended for performance reasons.
+
+        :param recreate_index: Use drop and recreate strategy. (recommended for performance)
+        :param batch_size: Only relevant if recreate_index is false. Defines the deletion batch size.
+            Note that this parameter needs to be less or equal to the set `QUERY_MAXIMUM_RESULTS` variable
+            set for the weaviate deployment (default is 10000).
+            Reference: https://docs.weaviate.io/weaviate/manage-objects/delete#delete-all-objects
+        """
+
+        if recreate_index:
+            # get current up-to-date config from server, so we can recreate the collection faithfully
+            cfg = self.client.collections.get(self._collection_settings["class"]).config.get().to_dict()
+            class_name = cfg.get("class", self._collection_settings["class"])
+
+            self.client.collections.delete(class_name)
+            self.client.collections.create_from_dict(cfg)
+
+            self._collection_settings = cfg
+            self._collection = self.client.collections.get(class_name)
+            return
+
+        uuids = []
+        batch_size = max(1, int(batch_size))
+
+        for obj in self.collection.iterator(return_properties=[], include_vector=False):
+            uuids.append(obj.uuid)
+            if len(uuids) >= batch_size:
+                res = self.collection.data.delete_many(where=weaviate.classes.query.Filter.by_id().contains_any(uuids))
+                if res.successful < len(uuids):
+                    logger.warning(
+                        "Not all documents in the batch have been deleted. "
+                        "Make sure to specify a deletion `batch_size` which is less than `QUERY_MAXIMUM_RESULTS`.",
+                    )
+                uuids.clear()
+
+        if uuids:
+            res = self.collection.data.delete_many(where=weaviate.classes.query.Filter.by_id().contains_any(uuids))
+            if res.successful < len(uuids):
+                logger.warning(
+                    "Not all documents have been deleted. "
+                    "Make sure to specify a deletion `batch_size` which is less than `QUERY_MAXIMUM_RESULTS`.",
+                )
+
     def _bm25_retrieval(
         self, query: str, filters: Optional[Dict[str, Any]] = None, top_k: Optional[int] = None
     ) -> List[Document]:

{weaviate_haystack-6.2.0 → weaviate_haystack-6.3.0}/tests/test_document_store.py

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import base64
+import logging
 import os
 from typing import List
 from unittest.mock import MagicMock, patch
@@ -794,3 +795,41 @@ class TestWeaviateDocumentStore(CountDocumentsTest, WriteDocumentsTest, DeleteDo
     def test_connect_to_embedded(self):
         document_store = WeaviateDocumentStore(embedded_options=EmbeddedOptions())
         assert document_store.client
+
+    def test_delete_all_documents(self, document_store):
+        docs = [Document(content="test doc 1"), Document(content="test doc 2")]
+        assert document_store.write_documents(docs) == 2
+        assert document_store.count_documents() == 2
+        document_store.delete_all_documents()
+        assert document_store.count_documents() == 0
+
+    def test_delete_all_documents_recreate(self, document_store):
+        docs = [Document(content="test doc 1"), Document(content="test doc 2")]
+        assert document_store.write_documents(docs) == 2
+        assert document_store.count_documents() == 2
+
+        cls = document_store._collection_settings["class"]
+        collection = document_store.client.collections.get(cls)
+        previous_config = collection.config.get().to_dict()
+
+        document_store.delete_all_documents(recreate_index=True)
+        assert document_store.count_documents() == 0
+
+        new_config = document_store.client.collections.get(cls).config.get().to_dict()
+        assert previous_config == new_config
+
+    def test_delete_all_documents_batch_size(self, document_store):
+        docs = [Document(content=str(i)) for i in range(0, 5)]
+        assert document_store.write_documents(docs) == 5
+        document_store.delete_all_documents(batch_size=2)
+        assert document_store.count_documents() == 0
+
+    def test_delete_all_documents_excessive_batch_size(self, document_store, caplog):
+        """Test that the deletion is not complete if the batch size exceeds the QUERY_MAXIMUM_RESULTS."""
+        # assume QUERY_MAXIMUM_RESULTS == 10000 with standard deployment
+        docs = [Document(content=str(i)) for i in range(0, 10005)]
+        assert document_store.write_documents(docs) == 10005
+        with caplog.at_level(logging.WARNING):
+            document_store.delete_all_documents(batch_size=20000)
+        assert document_store.count_documents() == 5
+        assert "Not all documents have been deleted." in caplog.text