pgvector-haystack 1.1.0__tar.gz → 1.2.1__tar.gz

pgvector_haystack-1.2.1/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+## [integrations/pgvector-v1.2.0] - 2024-11-22
+
+### 🚀 Features
+
+- Add `create_extension` parameter to control vector extension creation (#1213)
+
+
+## [integrations/pgvector-v1.1.0] - 2024-11-21
+
+### 🚀 Features
+
+- Add filter_policy to pgvector integration (#820)
+- Add schema support to pgvector document store. (#1095)
+- Pgvector - recreate the connection if it is no longer valid (#1202)
+
+### 🐛 Bug Fixes
+
+- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
+
+### 📚 Documentation
+
+- Explain different connection string formats in the docstring (#1132)
+
+### 🧪 Testing
+
+- Do not retry tests in `hatch run test` command (#954)
+
+### ⚙️ CI
+
+- Retry tests to reduce flakyness (#836)
+- Adopt uv as installer (#1142)
+
+### 🧹 Chores
+
+- Update ruff invocation to include check parameter (#853)
+- PgVector - remove legacy filter support (#1068)
+- Update changelog after removing legacy filters (#1083)
+- Update ruff linting scripts and settings (#1105)
+
+### 🌀 Miscellaneous
+
+- Ci: install `pytest-rerunfailures` where needed; add retry config to `test-cov` script (#845)
+- Chore: Minor retriever pydoc fix (#884)
+- Chore: Update pgvector test for the new `apply_filter_policy` usage (#970)
+- Chore: pgvector ruff update, don't ruff tests (#984)
+
+## [integrations/pgvector-v0.4.0] - 2024-06-20
+
+### 🚀 Features
+
+- Defer the database connection to when it's needed (#773)
+- Add customizable index names for pgvector (#818)
+
+### 🌀 Miscellaneous
+
+- Docs: add missing api references (#728)
+- [deepset-ai/haystack-core-integrations#727] (#738)
+
+## [integrations/pgvector-v0.2.0] - 2024-05-08
+
+### 🚀 Features
+
+- `MongoDBAtlasEmbeddingRetriever` (#427)
+- Implement keyword retrieval for pgvector integration (#644)
+
+### 🐛 Bug Fixes
+
+- Fix order of API docs (#447)
+
+### 📚 Documentation
+
+- Update category slug (#442)
+- Disable-class-def (#556)
+
+### 🌀 Miscellaneous
+
+- Pgvector - review docstrings and API reference (#502)
+- Refactor tests (#574)
+- Remove references to Python 3.7 (#601)
+- Make Document Stores initially skip `SparseEmbedding` (#606)
+- Chore: add license classifiers (#680)
+- Type hints in pgvector document store updated for 3.8 compability (#704)
+- Chore: change the pydoc renderer class (#718)
+
+## [integrations/pgvector-v0.1.0] - 2024-02-14
+
+### 🐛 Bug Fixes
+
+- Pgvector: fix linting (#328)
+
+### 🌀 Miscellaneous
+
+- Pgvector Document Store - minimal implementation (#239)
+- Pgvector - filters (#257)
+- Pgvector - embedding retrieval (#298)
+- Pgvector - Embedding Retriever (#320)
+- Pgvector: generate API docs (#325)
+- Pgvector: add an example (#334)
+- Adopt `Secret` to pgvector (#402)
+
+<!-- generated by git-cliff -->

{pgvector_haystack-1.1.0 → pgvector_haystack-1.2.1}/PKG-INFO

@@ -1,12 +1,13 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: pgvector-haystack
-Version: 1.1.0
+Version: 1.2.1
 Summary: An integration of pgvector (vector search extension for Postgres) with Haystack
 Project-URL: Source, https://github.com/deepset-ai/haystack-core-integrations
 Project-URL: Documentation, https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/pgvector/README.md
 Project-URL: Issues, https://github.com/deepset-ai/haystack-core-integrations/issues
 Author-email: deepset GmbH <info@deepset.ai>
-License: Apache-2.0
+License-Expression: Apache-2.0
+License-File: LICENSE.txt
 Classifier: Development Status :: 4 - Beta
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python
@@ -47,7 +48,7 @@ pip install pgvector-haystack
 
 Ensure that you have a PostgreSQL running with the `pgvector` extension. For a quick setup using Docker, run:
 ```
-docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
+docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres pgvector/pgvector:pg17
 ```
 
 then run the tests:

{pgvector_haystack-1.1.0 → pgvector_haystack-1.2.1}/README.md

@@ -22,7 +22,7 @@ pip install pgvector-haystack
 
 Ensure that you have a PostgreSQL running with the `pgvector` extension. For a quick setup using Docker, run:
 ```
-docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
+docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres pgvector/pgvector:pg17
 ```
 
 then run the tests:

{pgvector_haystack-1.1.0 → pgvector_haystack-1.2.1}/src/haystack_integrations/document_stores/pgvector/document_store.py

@@ -78,6 +78,7 @@ class PgvectorDocumentStore:
         self,
         *,
         connection_string: Secret = Secret.from_env_var("PG_CONN_STR"),
+        create_extension: bool = True,
         schema_name: str = "public",
         table_name: str = "haystack_documents",
         language: str = "english",
@@ -102,6 +103,10 @@ class PgvectorDocumentStore:
             e.g.: `PG_CONN_STR="host=HOST port=PORT dbname=DBNAME user=USER password=PASSWORD"`
             See [PostgreSQL Documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING)
             for more details.
+        :param create_extension: Whether to create the pgvector extension if it doesn't exist.
+            Set this to `True` (default) to automatically create the extension if it is missing.
+            Creating the extension may require superuser privileges.
+            If set to `False`, ensure the extension is already installed; otherwise, an error will be raised.
         :param schema_name: The name of the schema the table is created in. The schema must already exist.
         :param table_name: The name of the table to use to store Haystack documents.
         :param language: The language to be used to parse query and document content in keyword retrieval.
@@ -138,6 +143,7 @@ class PgvectorDocumentStore:
         """
 
         self.connection_string = connection_string
+        self.create_extension = create_extension
         self.table_name = table_name
         self.schema_name = schema_name
         self.embedding_dimension = embedding_dimension
@@ -194,7 +200,8 @@ class PgvectorDocumentStore:
         conn_str = self.connection_string.resolve_value() or ""
         connection = connect(conn_str)
         connection.autocommit = True
-        connection.execute("CREATE EXTENSION IF NOT EXISTS vector")
+        if self.create_extension:
+            connection.execute("CREATE EXTENSION IF NOT EXISTS vector")
         register_vector(connection)  # Note: this must be called before creating the cursors.
 
         self._connection = connection
@@ -246,6 +253,7 @@ class PgvectorDocumentStore:
         return default_to_dict(
             self,
             connection_string=self.connection_string.to_dict(),
+            create_extension=self.create_extension,
             schema_name=self.schema_name,
             table_name=self.table_name,
             embedding_dimension=self.embedding_dimension,
@@ -381,7 +389,9 @@ class PgvectorDocumentStore:
             )
             return
 
-        sql_drop_index = SQL("DROP INDEX IF EXISTS {index_name}").format(index_name=Identifier(self.hnsw_index_name))
+        sql_drop_index = SQL("DROP INDEX IF EXISTS {schema_name}.{index_name}").format(
+            schema_name=Identifier(self.schema_name), index_name=Identifier(self.hnsw_index_name)
+        )
         self._execute_sql(sql_drop_index, error_msg="Could not drop HNSW index")
 
         self._create_hnsw_index()

{pgvector_haystack-1.1.0 → pgvector_haystack-1.2.1}/tests/test_document_store.py

@@ -5,6 +5,7 @@
 from unittest.mock import patch
 
 import numpy as np
+import psycopg
 import pytest
 from haystack.dataclasses.document import ByteStream, Document
 from haystack.document_stores.errors import DuplicateDocumentError
@@ -66,6 +67,7 @@ def test_init(monkeypatch):
     monkeypatch.setenv("PG_CONN_STR", "some_connection_string")
 
     document_store = PgvectorDocumentStore(
+        create_extension=True,
         schema_name="my_schema",
         table_name="my_table",
         embedding_dimension=512,
@@ -79,6 +81,7 @@ def test_init(monkeypatch):
         keyword_index_name="my_keyword_index",
     )
 
+    assert document_store.create_extension
     assert document_store.schema_name == "my_schema"
     assert document_store.table_name == "my_table"
     assert document_store.embedding_dimension == 512
@@ -97,6 +100,7 @@ def test_to_dict(monkeypatch):
     monkeypatch.setenv("PG_CONN_STR", "some_connection_string")
 
     document_store = PgvectorDocumentStore(
+        create_extension=False,
         table_name="my_table",
         embedding_dimension=512,
         vector_function="l2_distance",
@@ -113,6 +117,7 @@ def test_to_dict(monkeypatch):
         "type": "haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore",
         "init_parameters": {
             "connection_string": {"env_vars": ["PG_CONN_STR"], "strict": True, "type": "env_var"},
+            "create_extension": False,
             "table_name": "my_table",
             "schema_name": "public",
             "embedding_dimension": 512,
@@ -255,3 +260,47 @@ def test_from_pg_to_haystack_documents():
     assert haystack_docs[2].meta == {"meta_key": "meta_value"}
     assert haystack_docs[2].embedding == [0.7, 0.8, 0.9]
     assert haystack_docs[2].score is None
+
+
+@pytest.mark.integration
+def test_hnsw_index_recreation():
+    def get_index_oid(document_store, schema_name, index_name):
+        sql_get_index_oid = """
+            SELECT c.oid
+            FROM pg_class c
+            JOIN pg_namespace n ON n.oid = c.relnamespace
+            WHERE c.relkind = 'i'
+            AND n.nspname = %s
+            AND c.relname = %s;
+        """
+        return document_store.cursor.execute(sql_get_index_oid, (schema_name, index_name)).fetchone()[0]
+
+    # create a new schema
+    connection_string = "postgresql://postgres:postgres@localhost:5432/postgres"
+    schema_name = "test_schema"
+    with psycopg.connect(connection_string, autocommit=True) as conn:
+        conn.execute(f"CREATE SCHEMA IF NOT EXISTS {schema_name}")
+
+    # create a first document store and trigger the creation of the hnsw index
+    params = {
+        "connection_string": Secret.from_token(connection_string),
+        "schema_name": schema_name,
+        "table_name": "haystack_test_hnsw_index_recreation",
+        "search_strategy": "hnsw",
+    }
+    ds1 = PgvectorDocumentStore(**params)
+    ds1._initialize_table()
+
+    # get the hnsw index oid
+    hnws_index_name = "haystack_hnsw_index"
+    first_oid = get_index_oid(ds1, ds1.schema_name, hnws_index_name)
+
+    # create second document store with recreation enabled
+    ds2 = PgvectorDocumentStore(**params, hnsw_recreate_index_if_exists=True)
+    ds2._initialize_table()
+
+    # get the index oid
+    second_oid = get_index_oid(ds2, ds2.schema_name, hnws_index_name)
+
+    # verify that oids differ
+    assert second_oid != first_oid, "Index was not recreated (OID remained the same)"

{pgvector_haystack-1.1.0 → pgvector_haystack-1.2.1}/tests/test_retrievers.py

@@ -50,6 +50,7 @@ class TestEmbeddingRetriever:
                     "type": "haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore",
                     "init_parameters": {
                         "connection_string": {"env_vars": ["PG_CONN_STR"], "strict": True, "type": "env_var"},
+                        "create_extension": True,
                         "schema_name": "public",
                         "table_name": "haystack",
                         "embedding_dimension": 768,
@@ -82,6 +83,7 @@ class TestEmbeddingRetriever:
                     "type": "haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore",
                     "init_parameters": {
                         "connection_string": {"env_vars": ["PG_CONN_STR"], "strict": True, "type": "env_var"},
+                        "create_extension": False,
                         "table_name": "haystack_test_to_dict",
                         "embedding_dimension": 768,
                         "vector_function": "cosine_similarity",
@@ -106,6 +108,7 @@ class TestEmbeddingRetriever:
 
         assert isinstance(document_store, PgvectorDocumentStore)
         assert isinstance(document_store.connection_string, EnvVarSecret)
+        assert not document_store.create_extension
         assert document_store.table_name == "haystack_test_to_dict"
         assert document_store.embedding_dimension == 768
         assert document_store.vector_function == "cosine_similarity"
@@ -176,6 +179,7 @@ class TestKeywordRetriever:
                     "type": "haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore",
                     "init_parameters": {
                         "connection_string": {"env_vars": ["PG_CONN_STR"], "strict": True, "type": "env_var"},
+                        "create_extension": True,
                         "schema_name": "public",
                         "table_name": "haystack",
                         "embedding_dimension": 768,
@@ -207,6 +211,7 @@ class TestKeywordRetriever:
                     "type": "haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore",
                     "init_parameters": {
                         "connection_string": {"env_vars": ["PG_CONN_STR"], "strict": True, "type": "env_var"},
+                        "create_extension": False,
                         "table_name": "haystack_test_to_dict",
                         "embedding_dimension": 768,
                         "vector_function": "cosine_similarity",
@@ -230,6 +235,7 @@ class TestKeywordRetriever:
 
         assert isinstance(document_store, PgvectorDocumentStore)
         assert isinstance(document_store.connection_string, EnvVarSecret)
+        assert not document_store.create_extension
         assert document_store.table_name == "haystack_test_to_dict"
         assert document_store.embedding_dimension == 768
         assert document_store.vector_function == "cosine_similarity"

pgvector_haystack-1.1.0/CHANGELOG.md

@@ -1,56 +0,0 @@
-# Changelog
-
-## [integrations/pgvector-v1.0.0] - 2024-09-12
-
-### 🚀 Features
-
-- Add filter_policy to pgvector integration (#820)
-
-### 🐛 Bug Fixes
-
-- `PgVector` - Fallback to default filter policy when deserializing retrievers without the init parameter (#900)
-
-### 🧪 Testing
-
-- Do not retry tests in `hatch run test` command (#954)
-
-### ⚙️ Miscellaneous Tasks
-
-- Retry tests to reduce flakyness (#836)
-- Update ruff invocation to include check parameter (#853)
-- PgVector - remove legacy filter support (#1068)
-
-## [integrations/pgvector-v0.4.0] - 2024-06-20
-
-### 🚀 Features
-
-- Defer the database connection to when it's needed (#773)
-- Add customizable index names for pgvector (#818)
-
-## [integrations/pgvector-v0.2.0] - 2024-05-08
-
-### 🚀 Features
-
-- `MongoDBAtlasEmbeddingRetriever` (#427)
-- Implement keyword retrieval for pgvector integration (#644)
-
-### 🐛 Bug Fixes
-
-- Fix order of API docs (#447)
-
-This PR will also push the docs to Readme
-
-### 📚 Documentation
-
-- Update category slug (#442)
-- Disable-class-def (#556)
-
-## [integrations/pgvector-v0.1.0] - 2024-02-14
-
-### 🐛 Bug Fixes
-
-- Fix linting (#328)
-
-
-
-<!-- generated by git-cliff -->