pgvector-haystack 3.0.1__tar.gz → 3.2.0__tar.gz

{pgvector_haystack-3.0.1 → pgvector_haystack-3.2.0}/CHANGELOG.md

@@ -1,135 +1,153 @@
-# Changelog
-
-## [integrations/pgvector-v3.0.0] - 2025-03-11
-
-### 🧹 Chores
-
-- Remove Python 3.8 support (#1421)
-- Use Haystack logging across integrations (#1484)
-- Pgvector - pin haystack and remove dataframe checks (#1518)
-
-## [integrations/pgvector-v2.0.0] - 2025-02-13
-
-### 🧹 Chores
-
-- Pgvector - remove support for dataframe (#1370)
-
-## [integrations/pgvector-v1.3.0] - 2025-02-03
-
-### 🚀 Features
-
-- Pgvector - add like and not like filters (#1341)
-
-### 🧹 Chores
-
-- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
-
-
-## [integrations/pgvector-v1.2.1] - 2025-01-10
-
-### 🐛 Bug Fixes
-
-- PgvectorDocumentStore - use appropriate schema name if dropping index (#1277)
-
-
-## [integrations/pgvector-v1.2.0] - 2024-11-22
-
-### 🚀 Features
-
-- Add `create_extension` parameter to control vector extension creation (#1213)
-
-
-## [integrations/pgvector-v1.1.0] - 2024-11-21
-
-### 🚀 Features
-
-- Add filter_policy to pgvector integration (#820)
-- Add schema support to pgvector document store. (#1095)
-- Pgvector - recreate the connection if it is no longer valid (#1202)
-
-### 🐛 Bug Fixes
-
-- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
-
-### 📚 Documentation
-
-- Explain different connection string formats in the docstring (#1132)
-
-### 🧪 Testing
-
-- Do not retry tests in `hatch run test` command (#954)
-
-### ⚙️ CI
-
-- Retry tests to reduce flakyness (#836)
-- Adopt uv as installer (#1142)
-
-### 🧹 Chores
-
-- Update ruff invocation to include check parameter (#853)
-- PgVector - remove legacy filter support (#1068)
-- Update changelog after removing legacy filters (#1083)
-- Update ruff linting scripts and settings (#1105)
-
-### 🌀 Miscellaneous
-
-- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
-- Chore: Minor retriever pydoc fix (#884)
-- Chore: Update pgvector test for the new `apply_filter_policy` usage (#970)
-- Chore: pgvector ruff update, don't ruff tests (#984)
-
-## [integrations/pgvector-v0.4.0] - 2024-06-20
-
-### 🚀 Features
-
-- Defer the database connection to when it's needed (#773)
-- Add customizable index names for pgvector (#818)
-
-### 🌀 Miscellaneous
-
-- Docs: add missing api references (#728)
-- [deepset-ai/haystack-core-integrations#727] (#738)
-
-## [integrations/pgvector-v0.2.0] - 2024-05-08
-
-### 🚀 Features
-
-- `MongoDBAtlasEmbeddingRetriever` (#427)
-- Implement keyword retrieval for pgvector integration (#644)
-
-### 🐛 Bug Fixes
-
-- Fix order of API docs (#447)
-
-### 📚 Documentation
-
-- Update category slug (#442)
-- Disable-class-def (#556)
-
-### 🌀 Miscellaneous
-
-- Pgvector - review docstrings and API reference (#502)
-- Refactor tests (#574)
-- Remove references to Python 3.7 (#601)
-- Make Document Stores initially skip `SparseEmbedding` (#606)
-- Chore: add license classifiers (#680)
-- Type hints in pgvector document store updated for 3.8 compability (#704)
-- Chore: change the pydoc renderer class (#718)
-
-## [integrations/pgvector-v0.1.0] - 2024-02-14
-
-### 🐛 Bug Fixes
-
-- Pgvector: fix linting (#328)
-
-### 🌀 Miscellaneous
-
-- Pgvector Document Store - minimal implementation (#239)
-- Pgvector - filters (#257)
-- Pgvector - embedding retrieval (#298)
-- Pgvector - Embedding Retriever (#320)
-- Pgvector: generate API docs (#325)
-- Pgvector: add an example (#334)
-- Adopt `Secret` to pgvector (#402)
-
-<!-- generated by git-cliff -->
+# Changelog
+
+## [integrations/pgvector-v3.1.0] - 2025-03-20
+
+### 🚀 Features
+
+- Pgvector - async support (+ refactoring) (#1547)
+
+### ⚙️ CI
+
+- Review testing workflows (#1541)
+
+## [integrations/pgvector-v3.0.1] - 2025-03-12
+
+### 🌀 Miscellaneous
+
+- Fix: pgvector - improve `_create_table_if_not_exists` to be used without admin rights (#1490)
+
+## [integrations/pgvector-v3.0.0] - 2025-03-11
+
+
+### 🧹 Chores
+
+- Remove Python 3.8 support (#1421)
+- Use Haystack logging across integrations (#1484)
+- Pgvector - pin haystack and remove dataframe checks (#1518)
+
+## [integrations/pgvector-v2.0.0] - 2025-02-13
+
+### 🧹 Chores
+
+- Pgvector - remove support for dataframe (#1370)
+
+
+## [integrations/pgvector-v1.3.0] - 2025-02-03
+
+### 🚀 Features
+
+- Pgvector - add like and not like filters (#1341)
+
+### 🧹 Chores
+
+- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
+
+
+## [integrations/pgvector-v1.2.1] - 2025-01-10
+
+### 🐛 Bug Fixes
+
+- PgvectorDocumentStore - use appropriate schema name if dropping index (#1277)
+
+
+## [integrations/pgvector-v1.2.0] - 2024-11-22
+
+### 🚀 Features
+
+- Add `create_extension` parameter to control vector extension creation (#1213)
+
+
+## [integrations/pgvector-v1.1.0] - 2024-11-21
+
+### 🚀 Features
+
+- Add filter_policy to pgvector integration (#820)
+- Add schema support to pgvector document store. (#1095)
+- Pgvector - recreate the connection if it is no longer valid (#1202)
+
+### 🐛 Bug Fixes
+
+- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
+
+### 📚 Documentation
+
+- Explain different connection string formats in the docstring (#1132)
+
+### 🧪 Testing
+
+- Do not retry tests in `hatch run test` command (#954)
+
+### ⚙️ CI
+
+- Retry tests to reduce flakyness (#836)
+- Adopt uv as installer (#1142)
+
+### 🧹 Chores
+
+- Update ruff invocation to include check parameter (#853)
+- PgVector - remove legacy filter support (#1068)
+- Update changelog after removing legacy filters (#1083)
+- Update ruff linting scripts and settings (#1105)
+
+### 🌀 Miscellaneous
+
+- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
+- Chore: Minor retriever pydoc fix (#884)
+- Chore: Update pgvector test for the new `apply_filter_policy` usage (#970)
+- Chore: pgvector ruff update, don't ruff tests (#984)
+
+## [integrations/pgvector-v0.4.0] - 2024-06-20
+
+### 🚀 Features
+
+- Defer the database connection to when it's needed (#773)
+- Add customizable index names for pgvector (#818)
+
+### 🌀 Miscellaneous
+
+- Docs: add missing api references (#728)
+- [deepset-ai/haystack-core-integrations#727] (#738)
+
+## [integrations/pgvector-v0.2.0] - 2024-05-08
+
+### 🚀 Features
+
+- `MongoDBAtlasEmbeddingRetriever` (#427)
+- Implement keyword retrieval for pgvector integration (#644)
+
+### 🐛 Bug Fixes
+
+- Fix order of API docs (#447)
+
+### 📚 Documentation
+
+- Update category slug (#442)
+- Disable-class-def (#556)
+
+### 🌀 Miscellaneous
+
+- Pgvector - review docstrings and API reference (#502)
+- Refactor tests (#574)
+- Remove references to Python 3.7 (#601)
+- Make Document Stores initially skip `SparseEmbedding` (#606)
+- Chore: add license classifiers (#680)
+- Type hints in pgvector document store updated for 3.8 compability (#704)
+- Chore: change the pydoc renderer class (#718)
+
+## [integrations/pgvector-v0.1.0] - 2024-02-14
+
+### 🐛 Bug Fixes
+
+- Pgvector: fix linting (#328)
+
+### 🌀 Miscellaneous
+
+- Pgvector Document Store - minimal implementation (#239)
+- Pgvector - filters (#257)
+- Pgvector - embedding retrieval (#298)
+- Pgvector - Embedding Retriever (#320)
+- Pgvector: generate API docs (#325)
+- Pgvector: add an example (#334)
+- Adopt `Secret` to pgvector (#402)
+
+<!-- generated by git-cliff -->

{pgvector_haystack-3.0.1 → pgvector_haystack-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgvector-haystack
-Version: 3.0.1
+Version: 3.2.0
 Summary: An integration of pgvector (vector search extension for Postgres) with Haystack
 Project-URL: Source, https://github.com/deepset-ai/haystack-core-integrations
 Project-URL: Documentation, https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/pgvector/README.md
@@ -15,6 +15,7 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Requires-Python: >=3.9

{pgvector_haystack-3.0.1 → pgvector_haystack-3.2.0}/pyproject.toml

@@ -19,6 +19,7 @@ classifiers = [
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: Implementation :: PyPy",
 ]
@@ -48,6 +49,7 @@ dependencies = [
   "pytest-rerunfailures",
   "ipython",
   "haystack-pydoc-tools",
+  "pytest-asyncio",
 ]
 [tool.hatch.envs.default.scripts]
 test = "pytest {args:tests}"
@@ -58,9 +60,6 @@ cov = ["test-cov", "cov-report"]
 cov-retry = ["test-cov-retry", "cov-report"]
 docs = ["pydoc-markdown pydoc/config.yml"]
 
-[[tool.hatch.envs.all.matrix]]
-python = [ "3.9", "3.10", "3.11", "3.12"]
-
 [tool.hatch.envs.lint]
 installer = "uv"
 detached = true
@@ -125,6 +124,8 @@ ignore = [
   "PLR0915",
   # ignore function-call-in-default-argument
   "B008",
+  # ignore assertions
+  "S101",
 ]
 unfixable = [
   # Don't touch unused imports

{pgvector_haystack-3.0.1 → pgvector_haystack-3.2.0}/src/haystack_integrations/components/retrievers/pgvector/embedding_retriever.py

@@ -151,7 +151,8 @@ class PgvectorEmbeddingRetriever:
         :param top_k: Maximum number of Documents to return.
         :param vector_function: The similarity function to use when searching for similar embeddings.
 
-        :returns: List of Documents similar to `query_embedding`.
+        :returns: A dictionary with the following keys:
+            - `documents`: List of `Document`s that are similar to `query_embedding`.
         """
         filters = apply_filter_policy(self.filter_policy, self.filters, filters)
         top_k = top_k or self.top_k
@@ -164,3 +165,36 @@ class PgvectorEmbeddingRetriever:
             vector_function=vector_function,
         )
         return {"documents": docs}
+
+    @component.output_types(documents=List[Document])
+    async def run_async(
+        self,
+        query_embedding: List[float],
+        filters: Optional[Dict[str, Any]] = None,
+        top_k: Optional[int] = None,
+        vector_function: Optional[Literal["cosine_similarity", "inner_product", "l2_distance"]] = None,
+    ):
+        """
+        Asynchronously retrieve documents from the `PgvectorDocumentStore`, based on their embeddings.
+
+        :param query_embedding: Embedding of the query.
+        :param filters: Filters applied to the retrieved Documents. The way runtime filters are applied depends on
+                        the `filter_policy` chosen at retriever initialization. See init method docstring for more
+                        details.
+        :param top_k: Maximum number of Documents to return.
+        :param vector_function: The similarity function to use when searching for similar embeddings.
+
+        :returns: A dictionary with the following keys:
+            - `documents`: List of `Document`s that are similar to `query_embedding`.
+        """
+        filters = apply_filter_policy(self.filter_policy, self.filters, filters)
+        top_k = top_k or self.top_k
+        vector_function = vector_function or self.vector_function
+
+        docs = await self.document_store._embedding_retrieval_async(
+            query_embedding=query_embedding,
+            filters=filters,
+            top_k=top_k,
+            vector_function=vector_function,
+        )
+        return {"documents": docs}

{pgvector_haystack-3.0.1 → pgvector_haystack-3.2.0}/src/haystack_integrations/components/retrievers/pgvector/keyword_retriever.py

@@ -136,3 +136,33 @@ class PgvectorKeywordRetriever:
             top_k=top_k,
         )
         return {"documents": docs}
+
+    @component.output_types(documents=List[Document])
+    async def run_async(
+        self,
+        query: str,
+        filters: Optional[Dict[str, Any]] = None,
+        top_k: Optional[int] = None,
+    ):
+        """
+        Asynchronously retrieve documents from the `PgvectorDocumentStore`, based on keywords.
+
+        :param query: String to search in `Document`s' content.
+        :param filters: Filters applied to the retrieved Documents. The way runtime filters are applied depends on
+                        the `filter_policy` chosen at retriever initialization. See init method docstring for more
+                        details.
+        :param top_k: Maximum number of Documents to return.
+
+        :returns: A dictionary with the following keys:
+            - `documents`: List of `Document`s that match the query.
+        """
+        filters = apply_filter_policy(self.filter_policy, self.filters, filters)
+
+        top_k = top_k or self.top_k
+
+        docs = await self.document_store._keyword_retrieval_async(
+            query=query,
+            filters=filters,
+            top_k=top_k,
+        )
+        return {"documents": docs}

pgvector_haystack-3.2.0/src/haystack_integrations/document_stores/pgvector/converters.py

@@ -0,0 +1,69 @@
+from typing import Any, Dict, List
+
+from haystack import logging
+from haystack.dataclasses import ByteStream, Document
+from psycopg.types.json import Jsonb
+
+logger = logging.getLogger(__name__)
+
+
+def _from_haystack_to_pg_documents(documents: List[Document]) -> List[Dict[str, Any]]:
+    """
+    Internal method to convert a list of Haystack Documents to a list of dictionaries that can be used to insert
+    documents into the PgvectorDocumentStore.
+    """
+
+    db_documents = []
+    for document in documents:
+        db_document = {k: v for k, v in document.to_dict(flatten=False).items() if k not in ["score", "blob"]}
+
+        blob = document.blob
+        db_document["blob_data"] = blob.data if blob else None
+        db_document["blob_meta"] = Jsonb(blob.meta) if blob and blob.meta else None
+        db_document["blob_mime_type"] = blob.mime_type if blob and blob.mime_type else None
+        db_document["meta"] = Jsonb(db_document["meta"])
+
+        if "sparse_embedding" in db_document:
+            sparse_embedding = db_document.pop("sparse_embedding", None)
+            if sparse_embedding:
+                logger.warning(
+                    "Document {doc_id} has the `sparse_embedding` field set,"
+                    "but storing sparse embeddings in Pgvector is not currently supported."
+                    "The `sparse_embedding` field will be ignored.",
+                    doc_id=db_document["id"],
+                )
+
+        db_documents.append(db_document)
+
+    return db_documents
+
+
+def _from_pg_to_haystack_documents(documents: List[Dict[str, Any]]) -> List[Document]:
+    """
+    Internal method to convert a list of dictionaries from pgvector to a list of Haystack Documents.
+    """
+
+    haystack_documents = []
+    for document in documents:
+        haystack_dict = dict(document)
+        blob_data = haystack_dict.pop("blob_data")
+        blob_meta = haystack_dict.pop("blob_meta")
+        blob_mime_type = haystack_dict.pop("blob_mime_type")
+
+        # convert the embedding to a list of floats
+        # for strange reasons, halfvec and vector have different methods to convert the embedding to a list
+        if document.get("embedding") is not None:
+            if hasattr(document["embedding"], "tolist"):  # vector
+                haystack_dict["embedding"] = document["embedding"].tolist()
+            else:  # halfvec
+                haystack_dict["embedding"] = document["embedding"].to_list()
+
+        haystack_document = Document.from_dict(haystack_dict)
+
+        if blob_data:
+            blob = ByteStream(data=blob_data, meta=blob_meta, mime_type=blob_mime_type)
+            haystack_document.blob = blob
+
+        haystack_documents.append(haystack_document)
+
+    return haystack_documents