pgvector-template 0.2.2__tar.gz → 0.3.0__tar.gz

pgvector_template-0.3.0/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.1
+Name: pgvector-template
+Version: 0.3.0
+Summary: Template library for flexible PGVector RAG implementations
+Author-email: DL <v49t9zpqd@mozmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/DavidLiuGit/PGVector-Template
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pgvector>=0.2.0
+Requires-Dist: pydantic<3.0,>=2.11
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: typing-extensions>=4.0.0
+Provides-Extra: test
+Requires-Dist: psycopg[binary]>=3.1.0; extra == "test"
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: python-dotenv>=1.0.0; extra == "test"
+Provides-Extra: dev
+Requires-Dist: black>=23.0.0; extra == "dev"
+
+# PGVector-Template
+
+A flexible, production-ready template library for building Retrieval-Augmented Generation (RAG) applications using PostgreSQL with PGVector extensions.
+
+## Overview
+
+PGVector-Template provides a robust foundation for implementing vector-based document storage and retrieval systems. It offers a clean abstraction layer over PostgreSQL's PGVector extension, making it easy to build scalable RAG applications with proper document management, metadata handling, and efficient vector search capabilities.
+
+## Key Features
+
+- **Flexible Document Model**: Abstract base classes for customizable document schemas
+- **Vector Search**: Optimized HNSW indexing for fast similarity search
+- **Metadata Management**: JSON-based flexible metadata with GIN indexing
+- **Collection Support**: Organize documents into logical collections
+- **Chunk Management**: Handle long content (refer to as **corpus**) by chunking it into smaller **documents**. Handle recovering the original corpus given its id
+- **Database Abstraction**: Clean SQLAlchemy-based database layer, with an API to create schemas
+- **Type Safety**: Full Pydantic validation and type hints
+- **Production Ready**: Comprehensive testing and error handling
+
+## Architecture
+
+The library is organized into several key components:
+
+- **Core**: Document models, embedders, search functionality
+- **Database**: Connection management and document database operations
+- **Service**: High-level document service layer
+- **Types**: Shared type definitions and schemas
+
+## Installation
+
+```bash
+pip install pgvector-template
+```
+
+Or add `pgvector-template` to your dependencies
+
+### Prerequisites
+
+- Python 3.11+
+- To execute tests: PostgreSQL with PGVector extension
+
+## Configuration
+
+### Database Setup
+
+1. Install PostgreSQL with PGVector extension
+2. Create your database and enable the vector extension:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+```
+
+3. Set up your connection string in environment variables or pass directly to `DatabaseManager`
+
+### Environment Variables
+
+For integration tests, create a `.env` file
+
+```bash
+cp integ-tests/.env.example integ-tests/.env
+```
+
+Specify envvars directly in the .env file. It is loaded automatically for integ tests.
+
+```bash
+DATABASE_URL=postgresql://user:password@localhost:5432/test_db
+```
+
+## API Reference
+
+### Core Classes
+
+- `BaseDocument`: Abstract document model with vector embedding support
+  - refer to table schema for explanation of the fields
+- `BaseDocumentOptionalProps`: Optional properties for document creation
+- `DatabaseManager`: Database connection and session management
+- `DocumentDatabaseManager`: High-level document operations
+
+### Key Methods
+
+- `BaseDocument.from_props()`: Create document instances from properties
+- `DocumentDatabaseManager.insert_document()`: Store documents
+- `DocumentDatabaseManager.search_similar()`: Vector similarity search
+
+## Testing
+
+Install dependencies (preferably in a virtualenv) before running tests:
+```bash
+pip install -e .[test]
+```
+
+### Unit Tests
+```bash
+python -m unittest
+```
+
+### Integration Tests
+
+Integration tests require a PostgreSQL database with PGVector extension. Set up your test database and configure the connection in `integ-tests/.env`:
+
+```bash
+python -m unittest discover -s integ-tests
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes with tests
+4. Run the test suite
+5. Submit a pull request
+
+### Development Setup
+
+```bash
+pip install -e .[dev,test]
+black .  # Format code
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Links
+
+- [GitHub Repository](https://github.com/DavidLiuGit/PGVector-Template)
+- [PGVector Documentation](https://github.com/pgvector/pgvector)

pgvector_template-0.3.0/README.md

@@ -0,0 +1,127 @@
+# PGVector-Template
+
+A flexible, production-ready template library for building Retrieval-Augmented Generation (RAG) applications using PostgreSQL with PGVector extensions.
+
+## Overview
+
+PGVector-Template provides a robust foundation for implementing vector-based document storage and retrieval systems. It offers a clean abstraction layer over PostgreSQL's PGVector extension, making it easy to build scalable RAG applications with proper document management, metadata handling, and efficient vector search capabilities.
+
+## Key Features
+
+- **Flexible Document Model**: Abstract base classes for customizable document schemas
+- **Vector Search**: Optimized HNSW indexing for fast similarity search
+- **Metadata Management**: JSON-based flexible metadata with GIN indexing
+- **Collection Support**: Organize documents into logical collections
+- **Chunk Management**: Handle long content (refer to as **corpus**) by chunking it into smaller **documents**. Handle recovering the original corpus given its id
+- **Database Abstraction**: Clean SQLAlchemy-based database layer, with an API to create schemas
+- **Type Safety**: Full Pydantic validation and type hints
+- **Production Ready**: Comprehensive testing and error handling
+
+## Architecture
+
+The library is organized into several key components:
+
+- **Core**: Document models, embedders, search functionality
+- **Database**: Connection management and document database operations
+- **Service**: High-level document service layer
+- **Types**: Shared type definitions and schemas
+
+## Installation
+
+```bash
+pip install pgvector-template
+```
+
+Or add `pgvector-template` to your dependencies
+
+### Prerequisites
+
+- Python 3.11+
+- To execute tests: PostgreSQL with PGVector extension
+
+## Configuration
+
+### Database Setup
+
+1. Install PostgreSQL with PGVector extension
+2. Create your database and enable the vector extension:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+```
+
+3. Set up your connection string in environment variables or pass directly to `DatabaseManager`
+
+### Environment Variables
+
+For integration tests, create a `.env` file
+
+```bash
+cp integ-tests/.env.example integ-tests/.env
+```
+
+Specify envvars directly in the .env file. It is loaded automatically for integ tests.
+
+```bash
+DATABASE_URL=postgresql://user:password@localhost:5432/test_db
+```
+
+## API Reference
+
+### Core Classes
+
+- `BaseDocument`: Abstract document model with vector embedding support
+  - refer to table schema for explanation of the fields
+- `BaseDocumentOptionalProps`: Optional properties for document creation
+- `DatabaseManager`: Database connection and session management
+- `DocumentDatabaseManager`: High-level document operations
+
+### Key Methods
+
+- `BaseDocument.from_props()`: Create document instances from properties
+- `DocumentDatabaseManager.insert_document()`: Store documents
+- `DocumentDatabaseManager.search_similar()`: Vector similarity search
+
+## Testing
+
+Install dependencies (preferably in a virtualenv) before running tests:
+```bash
+pip install -e .[test]
+```
+
+### Unit Tests
+```bash
+python -m unittest
+```
+
+### Integration Tests
+
+Integration tests require a PostgreSQL database with PGVector extension. Set up your test database and configure the connection in `integ-tests/.env`:
+
+```bash
+python -m unittest discover -s integ-tests
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes with tests
+4. Run the test suite
+5. Submit a pull request
+
+### Development Setup
+
+```bash
+pip install -e .[dev,test]
+black .  # Format code
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Links
+
+- [GitHub Repository](https://github.com/DavidLiuGit/PGVector-Template)
+- [PGVector Documentation](https://github.com/pgvector/pgvector)

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pgvector_template/core/document.py

@@ -86,7 +86,7 @@ class BaseDocument(Base):
     """Flexible metadata as JSON"""
     origin_url = Column(String(2048), nullable=True)
     """Optional source URL"""
-    language = Column(String(10), default="en")
+    language = Column(String(2), default="en")
     """Language of the content (ISO 639-1 code), e.g., 'en', 'es', 'zh'."""
     score = Column(Float, nullable=True)
     """Optional score assigned during ingestion (e.g., relevance, confidence)."""
@@ -190,6 +190,8 @@ class BaseDocumentMetadata(BaseModel):
     Base metadata structure.
     It is generally expected that every `BaseDocument`'s metadata follows this exact schema,
     without any extraneous properties, or any missing properties, to avoid ambiguity.
+    It is mandatory to include a description with every field/property.
+    May contain nested Pydantic models, but they may not be optional. Instead, set defaults.
     """
 
     document_type: str = Field(

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pgvector_template/core/search.py

@@ -1,59 +1,23 @@
-from dataclasses import dataclass, asdict
-from datetime import datetime
 from logging import getLogger
 from typing import Any, Type, Sequence
 
-from pydantic import BaseModel, Field, model_validator
-from sqlalchemy import text, select, or_, and_
-from sqlalchemy.sql import Select
-from pgvector.sqlalchemy import Vector
+from pydantic import BaseModel, Field
+from sqlalchemy import select, or_, Integer, Float
+from sqlalchemy.orm import Session
+from sqlalchemy.sql import Select, ColumnElement
 
 from pgvector_template.core import (
     BaseEmbeddingProvider,
     BaseDocument,
     BaseDocumentMetadata,
 )
-from sqlalchemy.orm import Session
+from pgvector_template.models.search import SearchQuery, MetadataFilter, RetrievalResult
+from pgvector_template.utils.metadata_filter import validate_metadata_filters
 
 
 logger = getLogger(__name__)
 
 
-class SearchQuery(BaseModel):
-    """Standardized search query structure. At least 1 search criterion is required."""
-
-    text: str | None = None
-    """String to approximate-search (using vector distance) in a semantic search."""
-    keywords: list[str] | None = None
-    """List of keywords to exact-match in a keyword search."""
-    metadata_filters: dict[str, Any] | None = None
-    """Strict metadata filters that must be matched."""
-    date_range: tuple[datetime, datetime] | None = None
-    """Retrieve/limit results based on created_at & updated_at timestamps"""
-    limit: int = Field(
-        ...,
-        ge=1,
-    )
-    """Maximum number of results to return."""
-
-    @model_validator(mode="after")
-    def ensure_criterion(self):
-        if not any([self.text, self.keywords, self.metadata_filters, self.date_range]):
-            raise ValueError("At least one search criterion is required")
-        return self
-
-
-@dataclass
-class RetrievalResult:
-    """Standardized result structure for all retrieval operations"""
-
-    document: BaseDocument
-    score: float
-
-    def to_dict(self) -> dict[str, Any]:
-        return asdict(self)
-
-
 class BaseSearchClientConfig(BaseModel):
     """Config obj for `BaseSearchClient`."""
 
@@ -112,15 +76,15 @@ class BaseSearchClient:
         if query.text:
             db_query = self._apply_semantic_search(db_query, query)
         db_query = self._apply_keyword_search(db_query, query)
-        # if query.metadata_filters:
-        #     db_query = self._apply_metadata_filters(db_query, query)
+        if query.metadata_filters:
+            db_query = self._apply_metadata_filters(db_query, query)
         db_query = db_query.limit(query.limit)
 
         # execute query and return results
         results = self.session.scalars(db_query).all()
         return self._convert_to_retrieval_results(results)
 
-    def _apply_semantic_search(self, query: Select, search_query: SearchQuery) -> Select:
+    def _apply_semantic_search(self, db_query: Select, search_query: SearchQuery) -> Select:
         """Apply semantic (vector) search criteria to the query.
         `embedding_provider` must be provided at instantiation, or an `ValueError` will be raised.
         In PGVector, `<=>` operator is used to compare cosine distance. Lower = more similar.
@@ -133,9 +97,11 @@ class BaseSearchClient:
             Updated SQLAlchemy query with semantic search applied.
         """
         if not search_query.text:
-            return query
+            return db_query
         query_embedding = self.embedding_provider.embed_text(search_query.text)
-        return query.order_by(self.config.document_cls.embedding.cosine_distance(query_embedding))
+        return db_query.order_by(
+            self.config.document_cls.embedding.cosine_distance(query_embedding)
+        )
 
     def _apply_keyword_search(self, db_query: Select, search_query: SearchQuery) -> Select:
         """Apply keyword (full-text) search criteria to the query.
@@ -154,17 +120,81 @@ class BaseSearchClient:
             conditions.append(self.config.document_cls.content.ilike(f"%{keyword}%"))
         return db_query.where(or_(*conditions))
 
-    def _apply_metadata_filters(self, query: Select, search_query: SearchQuery) -> Select:
-        """Apply metadata filters to the query.
+    def _apply_metadata_filters(self, db_query: Select, search_query: SearchQuery) -> Select:
+        """
+        Apply metadata filters to the query. All condtions in `search_query.metadata_filters`,
+        if not None, are ANDed together. Metadata filters are applied against
+        `BaseDocument.document_metadata` JSONB field.
 
         Args:
             query: The base SQLAlchemy query.
-            search_query: The search query containing metadata filters.
+            search_query: The `SearchQuery` containing metadata filters.
 
         Returns:
             Updated SQLAlchemy query with metadata filters applied.
         """
-        raise NotImplementedError
+        if not search_query.metadata_filters:
+            return db_query
+
+        # strictly speaking, this validation step is optional. You may override this method to disable
+        try:
+            validate_metadata_filters(search_query.metadata_filters, self.config.document_metadata_cls)
+        except ValueError as e:
+            logger.warning(f"Metadata filter validation failed: {e}. Query success not guaranteed!")
+
+        conditions = [
+            self._build_metadata_filter_where_condition(filter_obj)
+            for filter_obj in search_query.metadata_filters
+        ]
+        return db_query.where(*conditions) if conditions else db_query
+
+    def _build_metadata_filter_where_condition(
+        self, filter_obj: MetadataFilter
+    ) -> ColumnElement[bool]:
+        """Build SQLAlchemy WHERE condition for a metadata filter."""
+        field_path = filter_obj.field_name.split(".")
+        metadata_col = self.config.document_cls.document_metadata
+
+        # Navigate to field
+        field_ref = metadata_col
+        for part in field_path:
+            field_ref = field_ref[part]
+
+        if filter_obj.condition == "eq":
+            return (
+                field_ref.astext == filter_obj.value
+                if isinstance(filter_obj.value, str)
+                else field_ref == filter_obj.value
+            )
+        elif filter_obj.condition in {"gt", "gte", "lt", "lte"}:
+            if isinstance(filter_obj.value, str):
+                field_text = field_ref.astext
+            else:
+                cast_type = Integer if isinstance(filter_obj.value, int) else Float
+                field_text = field_ref.astext.cast(cast_type)
+
+            if filter_obj.condition == "gt":
+                return field_text > filter_obj.value
+            elif filter_obj.condition == "gte":
+                return field_text >= filter_obj.value
+            elif filter_obj.condition == "lt":
+                return field_text < filter_obj.value
+            else:  # lte
+                return field_text <= filter_obj.value
+        elif filter_obj.condition == "contains":
+            return field_ref.contains([filter_obj.value])
+        elif filter_obj.condition == "in":
+            return field_ref.astext.in_([str(v) for v in filter_obj.value])
+        elif filter_obj.condition == "exists":
+            if len(field_path) == 1:
+                return metadata_col.has_key(field_path[0])
+            else:
+                parent_ref = metadata_col
+                for part in field_path[:-1]:
+                    parent_ref = parent_ref[part]
+                return parent_ref.has_key(field_path[-1])
+        else:
+            raise ValueError(f"Unsupported condition: {filter_obj.condition}")
 
     def _convert_to_retrieval_results(self, results: Sequence[Any]) -> list[RetrievalResult]:
         """Convert database results to RetrievalResult objects.
@@ -178,6 +208,5 @@ class BaseSearchClient:
         """
         retrieval_results = []
         for result in results:
-            doc = result[0] if isinstance(result, tuple) else result
             retrieval_results.append(RetrievalResult(document=result, score=1.0))
         return retrieval_results

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pgvector_template/db/document_db.py

@@ -37,7 +37,7 @@ class DocumentDatabaseManager(DatabaseManager):
         self.schema_name = f"{self.SCHEMA_PREFIX}{schema_suffix}"
         self.document_classes = document_classes
 
-    def setup(self) -> None:
+    def setup(self) -> str:
         """One-step setup: initialize connection, create schema and tables for all document classes"""
         self.initialize()
         self.create_schema(self.schema_name)
@@ -50,6 +50,7 @@ class DocumentDatabaseManager(DatabaseManager):
         self.logger.info(
             f"Document database setup complete for {self.schema_name} with {len(self.document_classes)} tables"
         )
+        return self.schema_name
 
 
 class TempDocumentDatabaseManager(DocumentDatabaseManager):

pgvector_template-0.3.0/pgvector_template/models/search.py

@@ -0,0 +1,67 @@
+from dataclasses import dataclass, asdict
+from datetime import datetime
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+from pgvector_template.core import BaseDocument
+
+
+class MetadataFilter(BaseModel):
+    """
+    An object acting as a filter for an arbitrary `Metadata` dictionary/map/object.
+    """
+    
+    field_name: str
+    """Field path in metadata. Use dot notation for nested fields (e.g., 'publication_info.journal')"""
+    condition: Literal["eq", "gt", "gte", "lt", "lte", "contains", "in", "exists"]
+    """
+    Comparison operator: 
+        - eq=equal
+        - gt/gte=greater than/equal
+        - lt/lte=less than/equal
+        - contains=array contains values (accepts array)
+        - in=value in array
+        - exists=field exists
+    """
+    value: Any
+    """Value to compare against. Type should match field type (str, int, float, bool, list)"""
+
+    model_config = ConfigDict(use_attribute_docstrings=True)
+
+
+class SearchQuery(BaseModel):
+    """Standardized search query structure. At least 1 search criterion is required."""
+
+    text: str | None = None
+    """String to match against using in a semantic search, i.e. using vector distance."""
+    keywords: list[str] = []
+    """List of keywords to **exact-match** in a keyword search."""
+    metadata_filters: list[MetadataFilter] = []
+    """List of metadata filter conditions that must be matched."""
+    date_range: tuple[datetime, datetime] | None = None
+    """Retrieve/limit results based on created_at & updated_at timestamps (i.e. database operations)"""
+    limit: int = Field(
+        ...,
+        ge=1,
+    )
+    """Maximum number of results to return."""
+
+    model_config = ConfigDict(use_attribute_docstrings=True)
+
+    @model_validator(mode="after")
+    def ensure_criterion(self):
+        if not any([self.text, self.keywords, self.metadata_filters, self.date_range]):
+            raise ValueError("At least one search criterion is required")
+        return self
+
+
+@dataclass
+class RetrievalResult:
+    """Standardized result structure for all retrieval operations"""
+
+    document: BaseDocument
+    score: float
+
+    def to_dict(self) -> dict[str, Any]:
+        return asdict(self)

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pgvector_template/service/document_service.py

@@ -50,8 +50,10 @@ class DocumentServiceConfig(BaseModel):
         # iff either config is an instance of their respective base config classes (and not a subclass)
         if type(self.corpus_manager_cfg) is BaseCorpusManagerConfig:
             self.corpus_manager_cfg.document_cls = self.document_cls
+            self.corpus_manager_cfg.document_metadata_cls = self.document_metadata_cls
         if type(self.search_client_cfg) is BaseSearchClientConfig:
             self.search_client_cfg.document_cls = self.document_cls
+            self.search_client_cfg.document_metadata_cls = self.document_metadata_cls
 
         # assign embedding_provider to CorpusManager & SearchClient configs
         if not self.corpus_manager_cfg.embedding_provider:

pgvector_template-0.3.0/pgvector_template/utils/metadata_filter.py

@@ -0,0 +1,81 @@
+from typing import Type
+
+from pgvector_template.core.document import BaseDocumentMetadata
+from pgvector_template.models.search import MetadataFilter
+
+
+def validate_metadata_filters(
+    filter_obj_list: list[MetadataFilter], metadata_cls: Type[BaseDocumentMetadata]
+) -> None:
+    """Validate a `list[MetadataFilter]` against schema and condition compatibility.
+
+    Args:
+        filter_obj_list (list[MetadataFilter]): _description_
+        metadata_cls (Type[BaseDocumentMetadata]): _description_
+    """
+    for metadata_filter_obj in filter_obj_list:
+        validate_metadata_filter(metadata_filter_obj, metadata_cls)
+
+
+def validate_metadata_filter(
+    filter_obj: MetadataFilter, metadata_cls: Type[BaseDocumentMetadata]
+) -> None:
+    """Validate metadata filter against schema and condition compatibility.
+
+    Note: This validates the schema structure but cannot guarantee runtime data conformity.
+    JSONB field access like field_ref[part] will succeed even if the actual data doesn't match the schema.
+
+    Args:
+        filter_obj: The metadata filter to validate
+        metadata_cls: The metadata class to validate against
+
+    Raises:
+        ValueError: If field doesn't exist in schema or condition is incompatible with field type
+    """
+    field_path = filter_obj.field_name.split(".")
+    current_field_info = metadata_cls.model_fields
+
+    # Navigate nested structure
+    for i, part in enumerate(field_path):
+        if part not in current_field_info:
+            raise ValueError(f"Field '{filter_obj.field_name}' not found in metadata schema")
+
+        field_info = current_field_info[part]
+        field_type = field_info.annotation
+
+        # Handle nested models
+        if not field_type:
+            raise ValueError(f"Field '{filter_obj.field_name}' not found in metadata schema")
+        elif hasattr(field_type, "model_fields"):
+            current_field_info = field_type.model_fields
+        elif i < len(field_path) - 1:
+            raise ValueError(
+                f"Cannot navigate into non-model field '{part}' in path '{filter_obj.field_name}'"
+            )
+
+    # Validate condition compatibility with final field type
+    validate_condition_compatibility(field_type, filter_obj.condition)
+
+
+def validate_condition_compatibility(field_type: Type, condition: str) -> None:
+    """Validate that condition is compatible with field type."""
+    # Extract base type from Optional/Union types
+    origin = getattr(field_type, "__origin__", None)
+    if origin is not None:
+        args = getattr(field_type, "__args__", ())
+        if origin is list:
+            field_type = list
+        elif len(args) > 0:
+            field_type = args[0]  # First non-None type
+
+    valid_conditions = {
+        str: {"eq", "gt", "gte", "lt", "lte", "in", "exists"},
+        int: {"eq", "gt", "gte", "lt", "lte", "exists"},
+        float: {"eq", "gt", "gte", "lt", "lte", "exists"},
+        bool: {"eq", "exists"},
+        list: {"contains", "in", "exists"},
+    }
+
+    allowed = valid_conditions.get(field_type, {"eq", "exists"})
+    if condition not in allowed:
+        raise ValueError(f"Condition '{condition}' not valid for field type {field_type.__name__}")

pgvector_template-0.3.0/pgvector_template.egg-info/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.1
+Name: pgvector-template
+Version: 0.3.0
+Summary: Template library for flexible PGVector RAG implementations
+Author-email: DL <v49t9zpqd@mozmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/DavidLiuGit/PGVector-Template
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pgvector>=0.2.0
+Requires-Dist: pydantic<3.0,>=2.11
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: typing-extensions>=4.0.0
+Provides-Extra: test
+Requires-Dist: psycopg[binary]>=3.1.0; extra == "test"
+Requires-Dist: pytest>=7.0.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0.0; extra == "test"
+Requires-Dist: python-dotenv>=1.0.0; extra == "test"
+Provides-Extra: dev
+Requires-Dist: black>=23.0.0; extra == "dev"
+
+# PGVector-Template
+
+A flexible, production-ready template library for building Retrieval-Augmented Generation (RAG) applications using PostgreSQL with PGVector extensions.
+
+## Overview
+
+PGVector-Template provides a robust foundation for implementing vector-based document storage and retrieval systems. It offers a clean abstraction layer over PostgreSQL's PGVector extension, making it easy to build scalable RAG applications with proper document management, metadata handling, and efficient vector search capabilities.
+
+## Key Features
+
+- **Flexible Document Model**: Abstract base classes for customizable document schemas
+- **Vector Search**: Optimized HNSW indexing for fast similarity search
+- **Metadata Management**: JSON-based flexible metadata with GIN indexing
+- **Collection Support**: Organize documents into logical collections
+- **Chunk Management**: Handle long content (refer to as **corpus**) by chunking it into smaller **documents**. Handle recovering the original corpus given its id
+- **Database Abstraction**: Clean SQLAlchemy-based database layer, with an API to create schemas
+- **Type Safety**: Full Pydantic validation and type hints
+- **Production Ready**: Comprehensive testing and error handling
+
+## Architecture
+
+The library is organized into several key components:
+
+- **Core**: Document models, embedders, search functionality
+- **Database**: Connection management and document database operations
+- **Service**: High-level document service layer
+- **Types**: Shared type definitions and schemas
+
+## Installation
+
+```bash
+pip install pgvector-template
+```
+
+Or add `pgvector-template` to your dependencies
+
+### Prerequisites
+
+- Python 3.11+
+- To execute tests: PostgreSQL with PGVector extension
+
+## Configuration
+
+### Database Setup
+
+1. Install PostgreSQL with PGVector extension
+2. Create your database and enable the vector extension:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+```
+
+3. Set up your connection string in environment variables or pass directly to `DatabaseManager`
+
+### Environment Variables
+
+For integration tests, create a `.env` file
+
+```bash
+cp integ-tests/.env.example integ-tests/.env
+```
+
+Specify envvars directly in the .env file. It is loaded automatically for integ tests.
+
+```bash
+DATABASE_URL=postgresql://user:password@localhost:5432/test_db
+```
+
+## API Reference
+
+### Core Classes
+
+- `BaseDocument`: Abstract document model with vector embedding support
+  - refer to table schema for explanation of the fields
+- `BaseDocumentOptionalProps`: Optional properties for document creation
+- `DatabaseManager`: Database connection and session management
+- `DocumentDatabaseManager`: High-level document operations
+
+### Key Methods
+
+- `BaseDocument.from_props()`: Create document instances from properties
+- `DocumentDatabaseManager.insert_document()`: Store documents
+- `DocumentDatabaseManager.search_similar()`: Vector similarity search
+
+## Testing
+
+Install dependencies (preferably in a virtualenv) before running tests:
+```bash
+pip install -e .[test]
+```
+
+### Unit Tests
+```bash
+python -m unittest
+```
+
+### Integration Tests
+
+Integration tests require a PostgreSQL database with PGVector extension. Set up your test database and configure the connection in `integ-tests/.env`:
+
+```bash
+python -m unittest discover -s integ-tests
+```
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes with tests
+4. Run the test suite
+5. Submit a pull request
+
+### Development Setup
+
+```bash
+pip install -e .[dev,test]
+black .  # Format code
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## Links
+
+- [GitHub Repository](https://github.com/DavidLiuGit/PGVector-Template)
+- [PGVector Documentation](https://github.com/pgvector/pgvector)

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pgvector_template.egg-info/SOURCES.txt

@@ -16,6 +16,9 @@ pgvector_template/core/search.py
 pgvector_template/db/__init__.py
 pgvector_template/db/connection.py
 pgvector_template/db/document_db.py
+pgvector_template/models/__init__.py
+pgvector_template/models/search.py
 pgvector_template/service/__init__.py
 pgvector_template/service/document_service.py
-pgvector_template/utils/__init__.py
+pgvector_template/utils/__init__.py
+pgvector_template/utils/metadata_filter.py

{pgvector_template-0.2.2 → pgvector_template-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pgvector-template"
-version = "0.2.2"
+version = "0.3.0"
 description = "Template library for flexible PGVector RAG implementations"
 authors = [{ name="DL", email="v49t9zpqd@mozmail.com" }]
 license = { text = "MIT" }

pgvector_template-0.2.2/PKG-INFO

@@ -1,46 +0,0 @@
-Metadata-Version: 2.1
-Name: pgvector-template
-Version: 0.2.2
-Summary: Template library for flexible PGVector RAG implementations
-Author-email: DL <v49t9zpqd@mozmail.com>
-License: MIT
-Project-URL: Homepage, https://github.com/DavidLiuGit/PGVector-Template
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pgvector>=0.2.0
-Requires-Dist: pydantic<3.0,>=2.11
-Requires-Dist: sqlalchemy>=2.0.0
-Requires-Dist: typing-extensions>=4.0.0
-Provides-Extra: test
-Requires-Dist: psycopg[binary]>=3.1.0; extra == "test"
-Requires-Dist: pytest>=7.0.0; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: python-dotenv>=1.0.0; extra == "test"
-Provides-Extra: dev
-Requires-Dist: black>=23.0.0; extra == "dev"
-
-# PGVector-Template
-
-Template library for flexible PGVector RAG implementations
-
-
-## Testing
-
-Install dependencies (preferably in a virtualenv) before running tests:
-```bash
-pip install -e .[test]
-```
-
-### Unit tests
-```bash
-python -m unittest
-```
-
-### Integration tests
-```bash
-python -m unittest discover -s integ-tests
-```

pgvector_template-0.2.2/README.md

@@ -1,21 +0,0 @@
-# PGVector-Template
-
-Template library for flexible PGVector RAG implementations
-
-
-## Testing
-
-Install dependencies (preferably in a virtualenv) before running tests:
-```bash
-pip install -e .[test]
-```
-
-### Unit tests
-```bash
-python -m unittest
-```
-
-### Integration tests
-```bash
-python -m unittest discover -s integ-tests
-```

pgvector_template-0.2.2/pgvector_template.egg-info/PKG-INFO

@@ -1,46 +0,0 @@
-Metadata-Version: 2.1
-Name: pgvector-template
-Version: 0.2.2
-Summary: Template library for flexible PGVector RAG implementations
-Author-email: DL <v49t9zpqd@mozmail.com>
-License: MIT
-Project-URL: Homepage, https://github.com/DavidLiuGit/PGVector-Template
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: pgvector>=0.2.0
-Requires-Dist: pydantic<3.0,>=2.11
-Requires-Dist: sqlalchemy>=2.0.0
-Requires-Dist: typing-extensions>=4.0.0
-Provides-Extra: test
-Requires-Dist: psycopg[binary]>=3.1.0; extra == "test"
-Requires-Dist: pytest>=7.0.0; extra == "test"
-Requires-Dist: pytest-cov>=4.0.0; extra == "test"
-Requires-Dist: python-dotenv>=1.0.0; extra == "test"
-Provides-Extra: dev
-Requires-Dist: black>=23.0.0; extra == "dev"
-
-# PGVector-Template
-
-Template library for flexible PGVector RAG implementations
-
-
-## Testing
-
-Install dependencies (preferably in a virtualenv) before running tests:
-```bash
-pip install -e .[test]
-```
-
-### Unit tests
-```bash
-python -m unittest
-```
-
-### Integration tests
-```bash
-python -m unittest discover -s integ-tests
-```