pgvector-haystack 2.0.0__tar.gz → 3.0.1__tar.gz

{pgvector_haystack-2.0.0 → pgvector_haystack-3.0.1}/CHANGELOG.md

@@ -1,121 +1,135 @@
-# Changelog
-
-## [integrations/pgvector-v1.3.0] - 2025-02-03
-
-### 🚀 Features
-
-- Pgvector - add like and not like filters (#1341)
-
-### 🧹 Chores
-
-- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
-
-
-## [integrations/pgvector-v1.2.1] - 2025-01-10
-
-### 🐛 Bug Fixes
-
-- PgvectorDocumentStore - use appropriate schema name if dropping index (#1277)
-
-
-## [integrations/pgvector-v1.2.0] - 2024-11-22
-
-### 🚀 Features
-
-- Add `create_extension` parameter to control vector extension creation (#1213)
-
-
-## [integrations/pgvector-v1.1.0] - 2024-11-21
-
-### 🚀 Features
-
-- Add filter_policy to pgvector integration (#820)
-- Add schema support to pgvector document store. (#1095)
-- Pgvector - recreate the connection if it is no longer valid (#1202)
-
-### 🐛 Bug Fixes
-
-- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
-
-### 📚 Documentation
-
-- Explain different connection string formats in the docstring (#1132)
-
-### 🧪 Testing
-
-- Do not retry tests in `hatch run test` command (#954)
-
-### ⚙️ CI
-
-- Retry tests to reduce flakyness (#836)
-- Adopt uv as installer (#1142)
-
-### 🧹 Chores
-
-- Update ruff invocation to include check parameter (#853)
-- PgVector - remove legacy filter support (#1068)
-- Update changelog after removing legacy filters (#1083)
-- Update ruff linting scripts and settings (#1105)
-
-### 🌀 Miscellaneous
-
-- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
-- Chore: Minor retriever pydoc fix (#884)
-- Chore: Update pgvector test for the new `apply_filter_policy` usage (#970)
-- Chore: pgvector ruff update, don't ruff tests (#984)
-
-## [integrations/pgvector-v0.4.0] - 2024-06-20
-
-### 🚀 Features
-
-- Defer the database connection to when it's needed (#773)
-- Add customizable index names for pgvector (#818)
-
-### 🌀 Miscellaneous
-
-- Docs: add missing api references (#728)
-- [deepset-ai/haystack-core-integrations#727] (#738)
-
-## [integrations/pgvector-v0.2.0] - 2024-05-08
-
-### 🚀 Features
-
-- `MongoDBAtlasEmbeddingRetriever` (#427)
-- Implement keyword retrieval for pgvector integration (#644)
-
-### 🐛 Bug Fixes
-
-- Fix order of API docs (#447)
-
-### 📚 Documentation
-
-- Update category slug (#442)
-- Disable-class-def (#556)
-
-### 🌀 Miscellaneous
-
-- Pgvector - review docstrings and API reference (#502)
-- Refactor tests (#574)
-- Remove references to Python 3.7 (#601)
-- Make Document Stores initially skip `SparseEmbedding` (#606)
-- Chore: add license classifiers (#680)
-- Type hints in pgvector document store updated for 3.8 compability (#704)
-- Chore: change the pydoc renderer class (#718)
-
-## [integrations/pgvector-v0.1.0] - 2024-02-14
-
-### 🐛 Bug Fixes
-
-- Pgvector: fix linting (#328)
-
-### 🌀 Miscellaneous
-
-- Pgvector Document Store - minimal implementation (#239)
-- Pgvector - filters (#257)
-- Pgvector - embedding retrieval (#298)
-- Pgvector - Embedding Retriever (#320)
-- Pgvector: generate API docs (#325)
-- Pgvector: add an example (#334)
-- Adopt `Secret` to pgvector (#402)
-
-<!-- generated by git-cliff -->
+# Changelog
+
+## [integrations/pgvector-v3.0.0] - 2025-03-11
+
+### 🧹 Chores
+
+- Remove Python 3.8 support (#1421)
+- Use Haystack logging across integrations (#1484)
+- Pgvector - pin haystack and remove dataframe checks (#1518)
+
+## [integrations/pgvector-v2.0.0] - 2025-02-13
+
+### 🧹 Chores
+
+- Pgvector - remove support for dataframe (#1370)
+
+## [integrations/pgvector-v1.3.0] - 2025-02-03
+
+### 🚀 Features
+
+- Pgvector - add like and not like filters (#1341)
+
+### 🧹 Chores
+
+- Inherit from `FilterDocumentsTestWithDataframe` in Document Stores (#1290)
+
+
+## [integrations/pgvector-v1.2.1] - 2025-01-10
+
+### 🐛 Bug Fixes
+
+- PgvectorDocumentStore - use appropriate schema name if dropping index (#1277)
+
+
+## [integrations/pgvector-v1.2.0] - 2024-11-22
+
+### 🚀 Features
+
+- Add `create_extension` parameter to control vector extension creation (#1213)
+
+
+## [integrations/pgvector-v1.1.0] - 2024-11-21
+
+### 🚀 Features
+
+- Add filter_policy to pgvector integration (#820)
+- Add schema support to pgvector document store. (#1095)
+- Pgvector - recreate the connection if it is no longer valid (#1202)
+
+### 🐛 Bug Fixes
+
+- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
+
+### 📚 Documentation
+
+- Explain different connection string formats in the docstring (#1132)
+
+### 🧪 Testing
+
+- Do not retry tests in `hatch run test` command (#954)
+
+### ⚙️ CI
+
+- Retry tests to reduce flakyness (#836)
+- Adopt uv as installer (#1142)
+
+### 🧹 Chores
+
+- Update ruff invocation to include check parameter (#853)
+- PgVector - remove legacy filter support (#1068)
+- Update changelog after removing legacy filters (#1083)
+- Update ruff linting scripts and settings (#1105)
+
+### 🌀 Miscellaneous
+
+- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
+- Chore: Minor retriever pydoc fix (#884)
+- Chore: Update pgvector test for the new `apply_filter_policy` usage (#970)
+- Chore: pgvector ruff update, don't ruff tests (#984)
+
+## [integrations/pgvector-v0.4.0] - 2024-06-20
+
+### 🚀 Features
+
+- Defer the database connection to when it's needed (#773)
+- Add customizable index names for pgvector (#818)
+
+### 🌀 Miscellaneous
+
+- Docs: add missing api references (#728)
+- [deepset-ai/haystack-core-integrations#727] (#738)
+
+## [integrations/pgvector-v0.2.0] - 2024-05-08
+
+### 🚀 Features
+
+- `MongoDBAtlasEmbeddingRetriever` (#427)
+- Implement keyword retrieval for pgvector integration (#644)
+
+### 🐛 Bug Fixes
+
+- Fix order of API docs (#447)
+
+### 📚 Documentation
+
+- Update category slug (#442)
+- Disable-class-def (#556)
+
+### 🌀 Miscellaneous
+
+- Pgvector - review docstrings and API reference (#502)
+- Refactor tests (#574)
+- Remove references to Python 3.7 (#601)
+- Make Document Stores initially skip `SparseEmbedding` (#606)
+- Chore: add license classifiers (#680)
+- Type hints in pgvector document store updated for 3.8 compability (#704)
+- Chore: change the pydoc renderer class (#718)
+
+## [integrations/pgvector-v0.1.0] - 2024-02-14
+
+### 🐛 Bug Fixes
+
+- Pgvector: fix linting (#328)
+
+### 🌀 Miscellaneous
+
+- Pgvector Document Store - minimal implementation (#239)
+- Pgvector - filters (#257)
+- Pgvector - embedding retrieval (#298)
+- Pgvector - Embedding Retriever (#320)
+- Pgvector: generate API docs (#325)
+- Pgvector: add an example (#334)
+- Adopt `Secret` to pgvector (#402)
+
+<!-- generated by git-cliff -->

{pgvector_haystack-2.0.0 → pgvector_haystack-3.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgvector-haystack
-Version: 2.0.0
+Version: 3.0.1
 Summary: An integration of pgvector (vector search extension for Postgres) with Haystack
 Project-URL: Source, https://github.com/deepset-ai/haystack-core-integrations
 Project-URL: Documentation, https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/pgvector/README.md
@@ -11,15 +11,14 @@ License-File: LICENSE.txt
 Classifier: Development Status :: 4 - Beta
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
-Requires-Python: >=3.8
-Requires-Dist: haystack-ai
+Requires-Python: >=3.9
+Requires-Dist: haystack-ai>=2.11.0
 Requires-Dist: pgvector
 Requires-Dist: psycopg[binary]
 Description-Content-Type: text/markdown

{pgvector_haystack-2.0.0 → pgvector_haystack-3.0.1}/pyproject.toml

@@ -7,7 +7,7 @@ name = "pgvector-haystack"
 dynamic = ["version"]
 description = "An integration of pgvector (vector search extension for Postgres) with Haystack"
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 license = "Apache-2.0"
 keywords = []
 authors = [{ name = "deepset GmbH", email = "info@deepset.ai" }]
@@ -15,7 +15,6 @@ classifiers = [
   "License :: OSI Approved :: Apache Software License",
   "Development Status :: 4 - Beta",
   "Programming Language :: Python",
-  "Programming Language :: Python :: 3.8",
   "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
   "Programming Language :: Python :: 3.11",
@@ -23,7 +22,7 @@ classifiers = [
   "Programming Language :: Python :: Implementation :: CPython",
   "Programming Language :: Python :: Implementation :: PyPy",
 ]
-dependencies = ["haystack-ai", "pgvector", "psycopg[binary]"]
+dependencies = ["haystack-ai>=2.11.0", "pgvector", "psycopg[binary]"]
 
 [project.urls]
 Source = "https://github.com/deepset-ai/haystack-core-integrations"
@@ -60,7 +59,7 @@ cov-retry = ["test-cov-retry", "cov-report"]
 docs = ["pydoc-markdown pydoc/config.yml"]
 
 [[tool.hatch.envs.all.matrix]]
-python = ["3.8", "3.9", "3.10", "3.11", "3.12"]
+python = [ "3.9", "3.10", "3.11", "3.12"]
 
 [tool.hatch.envs.lint]
 installer = "uv"

{pgvector_haystack-2.0.0 → pgvector_haystack-3.0.1}/src/haystack_integrations/document_stores/pgvector/document_store.py

@@ -1,10 +1,9 @@
 # SPDX-FileCopyrightText: 2023-present deepset GmbH <info@deepset.ai>
 #
 # SPDX-License-Identifier: Apache-2.0
-import logging
 from typing import Any, Dict, List, Literal, Optional
 
-from haystack import default_from_dict, default_to_dict
+from haystack import default_from_dict, default_to_dict, logging
 from haystack.dataclasses.document import ByteStream, Document
 from haystack.document_stores.errors import DocumentStoreError, DuplicateDocumentError
 from haystack.document_stores.types import DuplicatePolicy
@@ -24,7 +23,7 @@ from .filters import _convert_filters_to_where_clause_and_params
 logger = logging.getLogger(__name__)
 
 CREATE_TABLE_STATEMENT = """
-CREATE TABLE IF NOT EXISTS {schema_name}.{table_name} (
+CREATE TABLE {schema_name}.{table_name} (
 id VARCHAR(128) PRIMARY KEY,
 embedding VECTOR({embedding_dimension}),
 content TEXT,
@@ -193,7 +192,7 @@ class PgvectorDocumentStore:
             try:
                 self._connection.close()
             except Error as e:
-                logger.debug("Failed to close connection: %s", str(e))
+                logger.debug("Failed to close connection: {e}", e=str(e))
 
         conn_str = self.connection_string.resolve_value() or ""
         connection = connect(conn_str)
@@ -295,7 +294,7 @@ class PgvectorDocumentStore:
         cursor = cursor or self.cursor
 
         sql_query_str = sql_query.as_string(cursor) if not isinstance(sql_query, str) else sql_query
-        logger.debug("SQL query: %s\nParameters: %s", sql_query_str, params)
+        logger.debug("SQL query: {query}\nParameters: {parameters}", query=sql_query_str, parameters=params)
 
         try:
             result = cursor.execute(sql_query, params)
@@ -311,13 +310,22 @@ class PgvectorDocumentStore:
         Creates the table to store Haystack documents if it doesn't exist yet.
         """
 
-        create_sql = SQL(CREATE_TABLE_STATEMENT).format(
-            schema_name=Identifier(self.schema_name),
-            table_name=Identifier(self.table_name),
-            embedding_dimension=SQLLiteral(self.embedding_dimension),
+        table_exists = bool(
+            self._execute_sql(
+                "SELECT 1 FROM pg_tables WHERE schemaname = %s AND tablename = %s",
+                (self.schema_name, self.table_name),
+                "Could not check if table exists",
+            ).fetchone()
         )
 
-        self._execute_sql(create_sql, error_msg="Could not create table in PgvectorDocumentStore")
+        if not table_exists:
+            create_sql = SQL(CREATE_TABLE_STATEMENT).format(
+                schema_name=Identifier(self.schema_name),
+                table_name=Identifier(self.table_name),
+                embedding_dimension=SQLLiteral(self.embedding_dimension),
+            )
+
+            self._execute_sql(create_sql, error_msg="Could not create table in PgvectorDocumentStore")
 
     def delete_table(self):
         """
@@ -512,7 +520,7 @@ class PgvectorDocumentStore:
         sql_insert += SQL(" RETURNING id")
 
         sql_query_str = sql_insert.as_string(self.cursor) if not isinstance(sql_insert, str) else sql_insert
-        logger.debug("SQL query: %s\nParameters: %s", sql_query_str, db_documents)
+        logger.debug("SQL query: {query}\nParameters: {parameters}", query=sql_query_str, parameters=db_documents)
 
         try:
             self.cursor.executemany(sql_insert, db_documents, returning=True)
@@ -555,24 +563,14 @@ class PgvectorDocumentStore:
             db_document["blob_mime_type"] = blob.mime_type if blob and blob.mime_type else None
             db_document["meta"] = Jsonb(db_document["meta"])
 
-            if "dataframe" in db_document:
-                dataframe = db_document.pop("dataframe", None)
-                if dataframe:
-                    logger.warning(
-                        "Document %s has the `dataframe` field set. "
-                        "PgvectorDocumentStore no longer supports dataframes and this field will be ignored. "
-                        "The `dataframe` field will soon be removed from Haystack Document.",
-                        db_document["id"],
-                    )
-
             if "sparse_embedding" in db_document:
                 sparse_embedding = db_document.pop("sparse_embedding", None)
                 if sparse_embedding:
                     logger.warning(
-                        "Document %s has the `sparse_embedding` field set,"
+                        "Document {doc_id} has the `sparse_embedding` field set,"
                         "but storing sparse embeddings in Pgvector is not currently supported."
                         "The `sparse_embedding` field will be ignored.",
-                        db_document["id"],
+                        doc_id=db_document["id"],
                     )
 
             db_documents.append(db_document)
@@ -591,20 +589,11 @@ class PgvectorDocumentStore:
             blob_data = haystack_dict.pop("blob_data")
             blob_meta = haystack_dict.pop("blob_meta")
             blob_mime_type = haystack_dict.pop("blob_mime_type")
-            dataframe = haystack_dict.pop("dataframe", None)
 
             # convert the embedding to a list of floats
             if document.get("embedding") is not None:
                 haystack_dict["embedding"] = document["embedding"].tolist()
 
-            if dataframe:
-                logger.warning(
-                    "Document %s has the `dataframe` field set. "
-                    "PgvectorDocumentStore no longer supports dataframes and this field will be ignored. "
-                    "The `dataframe` field will soon be removed from Haystack Document.",
-                    haystack_dict["id"],
-                )
-
             haystack_document = Document.from_dict(haystack_dict)
 
             if blob_data:

{pgvector_haystack-2.0.0 → pgvector_haystack-3.0.1}/tests/test_document_store.py

@@ -288,3 +288,50 @@ def test_hnsw_index_recreation():
 
     # verify that oids differ
     assert second_oid != first_oid, "Index was not recreated (OID remained the same)"
+
+    # Clean up: drop the schema after the test
+    with psycopg.connect(connection_string, autocommit=True) as conn:
+        conn.execute(f"DROP SCHEMA IF EXISTS {schema_name} CASCADE")
+
+
+@pytest.mark.integration
+def test_create_table_if_not_exists():
+    def get_table_oid(document_store, schema_name, table_name):
+        sql_get_table_oid = """
+            SELECT c.oid
+            FROM pg_class c
+            JOIN pg_namespace n ON n.oid = c.relnamespace
+            WHERE c.relkind = 'r'
+            AND n.nspname = %s
+            AND c.relname = %s;
+        """
+        return document_store.cursor.execute(sql_get_table_oid, (schema_name, table_name)).fetchone()[0]
+
+    connection_string = "postgresql://postgres:postgres@localhost:5432/postgres"
+    schema_name = "test_schema"
+    table_name = "test_table"
+
+    # Create a new schema
+    with psycopg.connect(connection_string, autocommit=True) as conn:
+        conn.execute(f"CREATE SCHEMA {schema_name}")
+
+    document_store = PgvectorDocumentStore(
+        connection_string=Secret.from_token(connection_string),
+        schema_name=schema_name,
+        table_name=table_name,
+    )
+
+    document_store._create_table_if_not_exists()
+
+    first_table_oid = get_table_oid(document_store, schema_name, table_name)
+    assert first_table_oid is not None, "Table was not created"
+
+    document_store._create_table_if_not_exists()
+
+    second_table_oid = get_table_oid(document_store, schema_name, table_name)
+
+    assert first_table_oid == second_table_oid, "Table was recreated"
+
+    # Clean up: drop the schema after the test
+    with psycopg.connect(connection_string, autocommit=True) as conn:
+        conn.execute(f"DROP SCHEMA IF EXISTS {schema_name} CASCADE")