ry-pg-utils 1.0.2__tar.gz → 1.0.3__tar.gz

{ry_pg_utils-1.0.2/src/ry_pg_utils.egg-info → ry_pg_utils-1.0.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ry-pg-utils
-Version: 1.0.2
+Version: 1.0.3
 Summary: Utility functions for PostgreSQL
 Author: Ross Yeager
 Author-email: ryeager12@email.com
@@ -31,7 +31,7 @@ Dynamic: summary
 
 # ry-pg-utils
 
-A Python utility library for PostgreSQL database operations with dynamic table creation, connection management, and Protocol Buffer integration.
+A Python utility library for PostgreSQL database operations with dynamic table creation, connection management, Protocol Buffer integration, and PostgreSQL LISTEN/NOTIFY support.
 
 ## Overview
 
@@ -41,16 +41,19 @@ A Python utility library for PostgreSQL database operations with dynamic table c
 - Dynamic table creation from Protocol Buffer message definitions
 - Thread-safe session management
 - Multi-backend support with automatic backend ID tracking
-- Database updater for dynamic configuration
+- PostgreSQL LISTEN/NOTIFY triggers and notifications
+- Database updater for dynamic configuration via Redis
 - Argument parsing for PostgreSQL connection parameters
 
 ## Features
 
-- **Connection Management**: Thread-safe PostgreSQL connection pooling with automatic retry logic
+- **Connection Management**: Thread-safe PostgreSQL connection pooling with automatic retry logic and health checks
 - **Dynamic Tables**: Automatically create and manage database tables from Protocol Buffer message schemas
 - **Multi-Backend Support**: Track data across multiple backend instances with automatic ID tagging
 - **Session Management**: Context managers for safe database session handling
+- **Notification System**: Built-in PostgreSQL LISTEN/NOTIFY support with triggers and callbacks
 - **Configuration System**: Flexible configuration via environment variables and runtime settings
+- **Redis Integration**: Optional Redis-based database configuration updates
 - **Type Safety**: Full type hints and mypy support
 
 ## Installation
@@ -63,8 +66,13 @@ pip install ry-pg-utils
 
 - Python 3.12+
 - PostgreSQL database
-- SQLAlchemy
-- Protocol Buffer support
+- SQLAlchemy 2.0+
+- Protocol Buffer support (protobuf)
+- psycopg2-binary
+- tenacity (for retry logic)
+- python-dotenv (for environment variables)
+- ryutils (logging utilities)
+- ry_redis_bus (optional, for Redis integration)
 
 ## Configuration
 
@@ -105,11 +113,11 @@ pg_config.backend_id = "custom_backend_id"
 | `postgres_db` | str | From env | Database name |
 | `postgres_user` | str | From env | Database username |
 | `postgres_password` | str | From env | Database password |
-| `backend_id` | str | hostname_ip | Unique identifier for this backend instance |
+| `backend_id` | str | POSTGRES_USER or hostname_ip | Unique identifier for this backend instance |
 | `add_backend_to_all` | bool | True | Add backend_id column to all tables |
 | `add_backend_to_tables` | bool | True | Append backend_id to table names |
 | `raise_on_use_before_init` | bool | True | Raise exception if DB used before initialization |
-| `do_publish_db` | bool | False | Enable database publishing features |
+| `do_publish_db` | bool | False | Enable database publishing features (for Redis integration) |
 | `use_local_db_only` | bool | True | Use only local database connections |
 
 ## Quick Start
@@ -118,19 +126,15 @@ pg_config.backend_id = "custom_backend_id"
 
 ```python
 from ry_pg_utils.connect import init_database, ManagedSession
-from ry_pg_utils.postgres_info import PostgresInfo
 
-# Create database connection info
-db_info = PostgresInfo(
+# Initialize the database connection
+init_database(
     db_name="myapp_db",
-    host="localhost",
-    port=5432,
-    user="postgres",
-    password="secret"
+    db_host="localhost",
+    db_port=5432,
+    db_user="postgres",
+    db_password="secret"
 )
-
-# Initialize the database connection
-init_database(db_info, db_name="myapp")
 ```
 
 ### 2. Use Dynamic Tables with Protocol Buffers
@@ -165,12 +169,14 @@ exists = DynamicTableDb.is_in_db(
 
 ```python
 from ry_pg_utils.connect import ManagedSession
+from sqlalchemy import text
 
 # Use context manager for automatic session cleanup
-with ManagedSession(db="myapp") as session:
-    result = session.execute("SELECT * FROM my_table")
-    for row in result:
-        print(row)
+with ManagedSession(db="myapp_db") as session:
+    if session:
+        result = session.execute(text("SELECT * FROM my_table"))
+        for row in result:
+            print(row)
 ```
 
 ## Core Components
@@ -181,21 +187,28 @@ The connection module provides thread-safe database connection and session manag
 
 ```python
 from ry_pg_utils.connect import (
-    init_database,      # Initialize database connection
-    init_engine,        # Initialize SQLAlchemy engine
-    ManagedSession,     # Context manager for sessions
-    get_backend_id,     # Get current backend ID
-    set_backend_id,     # Set backend ID for thread
-    close_engine,       # Close database connection
-    clear_db,           # Clear all connections
+    init_database,           # Initialize database connection
+    init_engine,             # Initialize SQLAlchemy engine
+    ManagedSession,          # Context manager for sessions
+    get_backend_id,          # Get current backend ID
+    set_backend_id,          # Set backend ID for thread
+    get_engine,              # Get engine for database
+    close_engine,            # Close database connection
+    clear_db,                # Clear all connections
+    get_table_name,          # Get table name with backend suffix
+    is_database_initialized, # Check if database is initialized
+    Base,                    # SQLAlchemy declarative base
 )
 ```
 
 **Key Features:**
 - Thread-local backend ID tracking
-- Connection pooling with configurable parameters
-- Automatic connection recovery on failure
+- Connection pooling with configurable parameters (pool_size, max_overflow, pool_recycle)
+- Automatic connection recovery with retry logic (using tenacity)
 - Session scoping for thread safety
+- Automatic backend_id injection before flush operations
+- Pre-ping health checks to validate connections
+- Support for auto-importing model modules
 
 ### `dynamic_table.py` - Dynamic Table Creation
 
@@ -234,7 +247,7 @@ exists = db.inst_is_in_db(
 
 ### `postgres_info.py` - Connection Information
 
-Data class for PostgreSQL connection parameters:
+Class for PostgreSQL connection parameters:
 
 ```python
 from ry_pg_utils.postgres_info import PostgresInfo
@@ -247,8 +260,12 @@ db_info = PostgresInfo(
     password="secret"
 )
 
-# Get connection URI
-uri = db_info.get_uri()  # postgresql://postgres:secret@localhost:5432/mydb
+# Check if valid
+if not db_info.is_null():
+    print(db_info)  # Prints info with password masked
+
+# Create null instance
+null_info = PostgresInfo.null()
 ```
 
 ### `parse_args.py` - Argument Parsing
@@ -268,48 +285,129 @@ args = parser.parse_args()
 
 ### `updater.py` - Database Configuration Updater
 
-Dynamically update database connections based on configuration messages:
+Dynamically update database connections based on configuration messages via Redis:
 
 ```python
-from ry_pg_utils.updater import PostgresDbUpdater
+from ry_pg_utils.updater import DbUpdater
+from ry_redis_bus.helpers import RedisInfo
+from ryutils.verbose import Verbose
 
-updater = PostgresDbUpdater(
+updater = DbUpdater(
     redis_info=redis_info,
-    verbose=verbose,
-    backend_id="my_backend"
+    args=args,  # argparse.Namespace with postgres config
+    backend_id="my_backend",
+    verbose=Verbose(True)
+)
+
+# Initialize and start listening for configuration updates
+updater.init()
+
+# Run the update loop
+updater.run()  # Blocks, continuously checking for updates
+```
+
+### `notify_trigger.py` - PostgreSQL LISTEN/NOTIFY Support
+
+Create database triggers that send notifications on table changes:
+
+```python
+from ry_pg_utils.notify_trigger import (
+    create_notify_trigger,
+    drop_notify_trigger,
+    subscribe_to_notifications,
+    NotificationListener,
+)
+from ry_pg_utils.connect import get_engine
+
+engine = get_engine("myapp_db")
+
+# Create a trigger that notifies on INSERT/UPDATE/DELETE
+create_notify_trigger(
+    engine=engine,
+    table_name="users",
+    channel_name="user_changes",
+    events=["INSERT", "UPDATE", "DELETE"],
+    columns=["id", "username", "email"]  # Optional: only include specific columns
+)
+
+# Subscribe to notifications
+def handle_notification(notification):
+    print(f"Table: {notification['table']}")
+    print(f"Action: {notification['action']}")
+    print(f"Data: {notification['data']}")
+
+with subscribe_to_notifications(
+    engine=engine,
+    channel_name="user_changes",
+    callback=handle_notification,
+    timeout=60.0
+) as notifications:
+    # Notifications are handled in background thread
+    time.sleep(10)
+
+# Or use the NotificationListener class for long-running listeners
+listener = NotificationListener(db_name="myapp_db")
+listener.create_listener(
+    table_name="users",
+    channel_name="user_changes",
+    events=["INSERT", "UPDATE"]
+)
+listener.add_callback("user_changes", handle_notification)
+listener.start()
+
+# ... your application code ...
+
+listener.stop()
+```
+
+### `ipc/channels.py` - Redis Communication Channels
+
+Pre-defined Redis channels for database configuration updates (requires ry_redis_bus):
+
+```python
+from ry_pg_utils.ipc.channels import (
+    DATABASE_CHANNEL,           # DatabaseConfigPb messages
+    DATABASE_CONFIG_CHANNEL,    # DatabaseSettingsPb messages
+    DATABASE_NOTIFY_CHANNEL,    # DatabaseNotificationPb messages
 )
 
-# Start listening for configuration updates
-updater.run()
+# These channels are used by DbUpdater for dynamic database configuration
 ```
 
 ## Advanced Usage
 
 ### Multi-Backend Support
 
-When `add_backend_to_all` is enabled, all tables automatically get a `backend_id` column:
+When `add_backend_to_all` is enabled (default: True), all tables automatically get a `backend_id` column:
 
 ```python
 from ry_pg_utils.connect import set_backend_id, ManagedSession
+from sqlalchemy import text
 
 # Set backend ID for current thread
 set_backend_id("backend_1")
 
 # All subsequent operations will include this backend_id
-with ManagedSession(db="myapp") as session:
-    # Queries automatically filter by backend_id
-    result = session.execute("SELECT * FROM my_table")
+# ManagedSession automatically sets the backend_id if provided
+with ManagedSession(db="myapp_db", backend_id="backend_1") as session:
+    if session:
+        # New/dirty records automatically get backend_id injected before flush
+        result = session.execute(text("SELECT * FROM my_table"))
 ```
 
 ### Custom Table Names
 
-When `add_backend_to_tables` is enabled, table names are automatically suffixed:
+When `add_backend_to_tables` is enabled (default: True), table names are automatically suffixed:
 
 ```python
 from ry_pg_utils.connect import get_table_name
+from ry_pg_utils.config import pg_config
 
 # Returns "events_my_backend" if add_backend_to_tables=True
-table_name = get_table_name("events", backend_id="my_backend")
+table_name = get_table_name("events", backend_id="my_backend", verbose=True)
+
+# Configure globally
+pg_config.add_backend_to_tables = False  # Disable backend suffix
 ```
 
 ### ORM Base Class
@@ -327,27 +425,53 @@ class User(Base):
     name = Column(String(100))
     email = Column(String(200))
 
-# If add_backend_to_all=True, backend_id column is automatically added
+# If add_backend_to_all=True (default), backend_id column is automatically added
+# The backend_id is a String(256) column, nullable=False
+```
+
+### Auto-Importing Models
+
+Use the `models_module` parameter to automatically import all models before table creation:
+
+```python
+from ry_pg_utils.connect import init_database
+
+# This will walk through 'myapp.models' and import all submodules
+# ensuring all model classes are registered with Base.metadata
+init_database(
+    db_name="myapp_db",
+    db_host="localhost",
+    db_port=5432,
+    db_user="postgres",
+    db_password="secret",
+    models_module="myapp.models"  # Dot-separated module path
+)
 ```
 
 ## Error Handling
 
-The library includes robust error handling:
+The library includes robust error handling with automatic retries:
 
 ```python
 from ry_pg_utils.connect import ManagedSession
+from sqlalchemy import text
 
-with ManagedSession(db="myapp") as session:
+with ManagedSession(db="myapp_db") as session:
     if session is None:
-        # Connection failed, handle gracefully
+        # Connection failed after retries, handle gracefully
         print("Failed to establish database connection")
         return
 
     try:
-        session.execute("SELECT * FROM my_table")
+        session.execute(text("SELECT * FROM my_table"))
     except Exception as e:
-        # Session will automatically rollback
+        # Session will automatically rollback on exception
         print(f"Query failed: {e}")
+
+# Retries are built-in:
+# - Session operations retry 3 times on OperationalError
+# - Exponential backoff: min 4s, max 10s
+# - Connection health checks via pool_pre_ping
 ```
 
 ## Type Safety
@@ -378,12 +502,17 @@ pip install -r packages/requirements-dev.txt
 
 ### Running Tests
 
+Tests require a running PostgreSQL instance. Configure test database connection via environment variables or `.env` file.
+
 ```bash
 # Activate virtual environment
 source venv-dev/bin/activate
 
-# Run tests
-python -m pytest test/
+# Run all tests
+make test
+
+# Run specific test module
+make test TESTMODULE=connect_test
 ```
 
 ### Code Quality
@@ -392,16 +521,12 @@ The project uses several tools for code quality:
 
 ```bash
 # Format code
-black ry_pg_utils/
+make format
 
-# Type checking
-mypy ry_pg_utils/
+# Run linting
+make lint_full
 
-# Linting
-pylint ry_pg_utils/
-
-# Import sorting
-isort ry_pg_utils/
+# Type checking is included in lint_full (uses mypy)
 ```
 
 ## Examples
@@ -412,8 +537,8 @@ isort ry_pg_utils/
 import argparse
 from ry_pg_utils.parse_args import add_postrgres_db_args
 from ry_pg_utils.connect import init_database, ManagedSession
-from ry_pg_utils.postgres_info import PostgresInfo
 from ry_pg_utils.dynamic_table import DynamicTableDb
+from sqlalchemy import text
 
 def parse_args():
     parser = argparse.ArgumentParser(description="My Database App")
@@ -424,26 +549,78 @@ def main():
     args = parse_args()
 
     # Initialize database
-    db_info = PostgresInfo(
+    init_database(
         db_name=args.postgres_db,
-        host=args.postgres_host,
-        port=args.postgres_port,
-        user=args.postgres_user,
-        password=args.postgres_password
+        db_host=args.postgres_host,
+        db_port=args.postgres_port,
+        db_user=args.postgres_user,
+        db_password=args.postgres_password
     )
 
-    init_database(db_info, db_name="myapp")
-
     # Use the database
-    with ManagedSession(db="myapp") as session:
+    with ManagedSession(db=args.postgres_db) as session:
         if session:
-            result = session.execute("SELECT version()")
+            result = session.execute(text("SELECT version()"))
             print(f"PostgreSQL version: {result.fetchone()[0]}")
 
 if __name__ == "__main__":
     main()
 ```
 
+### Real-time Notification Example
+
+```python
+from ry_pg_utils.connect import init_database, get_engine, Base
+from ry_pg_utils.notify_trigger import create_notify_trigger, NotificationListener
+from sqlalchemy import Column, Integer, String
+import time
+
+# Define a model
+class Product(Base):
+    __tablename__ = 'products'
+
+    id = Column(Integer, primary_key=True)
+    name = Column(String(100))
+    price = Column(Integer)
+
+# Initialize database
+init_database(
+    db_name="inventory_db",
+    db_host="localhost",
+    db_port=5432,
+    db_user="postgres",
+    db_password="secret"
+)
+
+# Create notification trigger
+engine = get_engine("inventory_db")
+create_notify_trigger(
+    engine=engine,
+    table_name="products",
+    channel_name="product_updates",
+    events=["INSERT", "UPDATE", "DELETE"],
+    columns=["id", "name", "price"]
+)
+
+# Set up listener
+listener = NotificationListener(db_name="inventory_db")
+
+def on_product_change(notification):
+    action = notification['action']
+    data = notification['data']
+    print(f"Product {action}: {data}")
+
+listener.add_callback("product_updates", on_product_change)
+listener.start()
+
+# Your application runs...
+try:
+    while True:
+        time.sleep(1)
+except KeyboardInterrupt:
+    listener.stop()
+```
+
 ## License
 
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
@@ -464,10 +641,19 @@ Ross Yeager - ryeager12@email.com
 
 ## Changelog
 
+### Version 1.0.2 (Current)
+- PostgreSQL LISTEN/NOTIFY support with triggers and notifications
+- NotificationListener class for background notification handling
+- Automatic connection health checks with pool_pre_ping
+- Auto-importing models from specified module paths
+- Enhanced retry logic with tenacity
+- Improved error handling and connection recovery
+
 ### Version 1.0.0
 - Initial release
-- Database connection management
-- Dynamic table creation
-- Multi-backend support
-- Configuration system
+- Database connection management with pooling
+- Dynamic table creation from Protocol Buffers
+- Multi-backend support with automatic ID tagging
+- Configuration system with environment variables
 - Protocol Buffer integration
+- Redis-based database configuration updates