pg-idx-manager 0.1.0__tar.gz

pg_idx_manager-0.1.0/PKG-INFO

@@ -0,0 +1,67 @@
+Metadata-Version: 2.4
+Name: pg_idx_manager
+Version: 0.1.0
+Summary: A lightweight, framework-agnostic developer linter and cleanup tool for PostgreSQL indexes.
+Author-email: Pierpaolo <tua_email@example.com>
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Database :: Database Engines/Servers
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: psycopg2-binary>=2.9.0
+
+# PG Index Manager & Janitor 🚀
+
+A lightweight, framework-agnostic Python library designed for developers to audit database queries, surface runtime performance bottlenecks, and safely maintain PostgreSQL indexes.
+
+Unlike heavy Application Performance Monitoring (APM) tools that require invasive database extensions (like HypoPG) or expensive SaaS subscriptions, this library operates entirely inside your application or via a clean CLI using native PostgreSQL capabilities.
+
+## The Problem It Solves: "ORM Blindness"
+Modern Object-Relational Mappers (ORMs) like Django or SQLAlchemy maximize developer productivity but hide the underlying SQL execution plan. A seemingly innocent line of Python code can silently trigger a **Sequential Scan (Full Table Scan)** across millions of rows, saturating server CPU and disk I/O.
+
+Since Large Language Models (AI) cannot inspect your production database cardinality, table sizes, or live index catalogs, they cannot reliably predict query performance. This library bridges that gap by running live `EXPLAIN ANALYZE` inspections directly on the database engine.
+
+## Key Features
+
+1. **Agnostic Performance Auditing**: Intercepts raw SQL queries, recursively parses the native PostgreSQL execution tree, and catches performance anomalies (**Sequential Scans**) with exact millisecond runtimes.
+2. **Safe Janitor Mode (Interactive CLI)**: Scans PostgreSQL system catalogs (`pg_stat_user_indexes`) to discover dead, unused indexes that are slowing down your `INSERT`/`UPDATE` mutations. It allows you to drop them interactively using `DROP INDEX CONCURRENTLY` without locking tables or risking production application downtime.
+3. **Zero-Dependency Architecture**: Does not require root or superuser privileges on the database server. If you can connect to the database, you can run this library.
+
+## Architectural Architecture: Where does it live?
+Because the core engine requires only a raw query string and a standard database connection, it is completely independent of your web framework. You can integrate it at the lowest layer of your infrastructure:
+
+* **At the Driver Level (`psycopg2`)**: By extending the native database cursor, you can automatically audit **100% of your application queries** (Django, FastAPI, Flask, or raw SQL) before they hit the wire.
+* **At the ORM Engine Level**: Easily hooks into global ORM events (e.g., SQLAlchemy's `before_cursor_execute`) to catch hidden query costs implicitly across all Services and Repositories with zero business-code modification.
+
+---
+
+## Getting Started & Testing Locally 🧪
+
+This repository includes a fully containerized testing environment to observe performance optimization in real-time.
+
+### 1. Spin up the isolated test database
+```bash
+docker compose up -d
+```
+
+### 2. Install the library locally in editable mode
+```bash
+pip install -e .
+```
+
+### 3. Run the Core Agnostic Test
+This script populates the database with **10,000 mock records**, executes an unindexed query, catches the Sequential Scan risk, and launches the persistent interactive CLI:
+```bash
+python tests/run_test.py
+```
+
+### 4. Run the ORM Integration Test
+Observe how the library seamlessly intercepts real-time query metrics generated implicitly by SQLAlchemy ORM models:
+```bash
+python tests/test_orm.py
+```
+
+## License
+MIT License. Free to use and extend.

pg_idx_manager-0.1.0/README.md

@@ -0,0 +1,53 @@
+# PG Index Manager & Janitor 🚀
+
+A lightweight, framework-agnostic Python library designed for developers to audit database queries, surface runtime performance bottlenecks, and safely maintain PostgreSQL indexes.
+
+Unlike heavy Application Performance Monitoring (APM) tools that require invasive database extensions (like HypoPG) or expensive SaaS subscriptions, this library operates entirely inside your application or via a clean CLI using native PostgreSQL capabilities.
+
+## The Problem It Solves: "ORM Blindness"
+Modern Object-Relational Mappers (ORMs) like Django or SQLAlchemy maximize developer productivity but hide the underlying SQL execution plan. A seemingly innocent line of Python code can silently trigger a **Sequential Scan (Full Table Scan)** across millions of rows, saturating server CPU and disk I/O.
+
+Since Large Language Models (AI) cannot inspect your production database cardinality, table sizes, or live index catalogs, they cannot reliably predict query performance. This library bridges that gap by running live `EXPLAIN ANALYZE` inspections directly on the database engine.
+
+## Key Features
+
+1. **Agnostic Performance Auditing**: Intercepts raw SQL queries, recursively parses the native PostgreSQL execution tree, and catches performance anomalies (**Sequential Scans**) with exact millisecond runtimes.
+2. **Safe Janitor Mode (Interactive CLI)**: Scans PostgreSQL system catalogs (`pg_stat_user_indexes`) to discover dead, unused indexes that are slowing down your `INSERT`/`UPDATE` mutations. It allows you to drop them interactively using `DROP INDEX CONCURRENTLY` without locking tables or risking production application downtime.
+3. **Zero-Dependency Architecture**: Does not require root or superuser privileges on the database server. If you can connect to the database, you can run this library.
+
+## Architectural Architecture: Where does it live?
+Because the core engine requires only a raw query string and a standard database connection, it is completely independent of your web framework. You can integrate it at the lowest layer of your infrastructure:
+
+* **At the Driver Level (`psycopg2`)**: By extending the native database cursor, you can automatically audit **100% of your application queries** (Django, FastAPI, Flask, or raw SQL) before they hit the wire.
+* **At the ORM Engine Level**: Easily hooks into global ORM events (e.g., SQLAlchemy's `before_cursor_execute`) to catch hidden query costs implicitly across all Services and Repositories with zero business-code modification.
+
+---
+
+## Getting Started & Testing Locally 🧪
+
+This repository includes a fully containerized testing environment to observe performance optimization in real-time.
+
+### 1. Spin up the isolated test database
+```bash
+docker compose up -d
+```
+
+### 2. Install the library locally in editable mode
+```bash
+pip install -e .
+```
+
+### 3. Run the Core Agnostic Test
+This script populates the database with **10,000 mock records**, executes an unindexed query, catches the Sequential Scan risk, and launches the persistent interactive CLI:
+```bash
+python tests/run_test.py
+```
+
+### 4. Run the ORM Integration Test
+Observe how the library seamlessly intercepts real-time query metrics generated implicitly by SQLAlchemy ORM models:
+```bash
+python tests/test_orm.py
+```
+
+## License
+MIT License. Free to use and extend.

pg_idx_manager-0.1.0/pg_idx_manager/__init__.py

@@ -0,0 +1,5 @@
+from .core import IndexManagerCore
+from .cli import run_cli
+
+__all__ = ["IndexManagerCore", "run_cli"]
+

pg_idx_manager-0.1.0/pg_idx_manager/cli.py

@@ -0,0 +1,103 @@
+import sys
+from .core import IndexManagerCore
+
+def is_safe_query(query: str) -> bool:
+    """Sanity check to ensure the tool only processes read-only SELECT queries."""
+    clean_query = query.strip().upper()
+    if len(clean_query) < 6:
+        return True
+    dangerous_keywords = ["DROP TABLE", "DROP DATABASE", "DELETE FROM", "INSERT INTO", "UPDATE ", "TRUNCATE "]
+    for keyword in dangerous_keywords:
+        if keyword in clean_query:
+            return False
+    return True
+
+def run_cli(connection):
+    """Launches a persistent, continuous loop for interactive DB optimization."""
+    manager = IndexManagerCore(connection, min_table_rows=0)
+    
+    try:
+        while True:
+            print("\n" + "="*40)
+            print("🚀 PERSISTENT PG INDEX MANAGER CLI")
+            print("="*40)
+            print("1. Audit single SQL query efficiency")
+            print("2. Scan and clean up dead indexes (Janitor Mode)")
+            print("Type 'quit' at any prompt to exit.")
+            print("="*40)
+            
+            choice = input("\nSelect option (1/2 or 'quit'): ").strip().lower()
+            
+            if choice == "quit":
+                print("\n👋 Exiting Index Manager. Keep your database clean!")
+                break
+                
+            elif choice == "1":
+                while True:
+                    query = input("\nPaste SQL query to analyze (type 'back' for main menu, 'quit' to exit):\n> ").strip()
+                    
+                    if query.lower() == 'quit':
+                        print("\n👋 Exiting Index Manager. Keep your database clean!")
+                        return
+                    if query.lower() == 'back':
+                        break
+                    if not query:
+                        continue
+                    if not is_safe_query(query):
+                        print("\n❌ SECURITY ERROR: Only read-only SELECT queries are allowed for auditing.")
+                        continue
+                    
+                    try:
+                        anomalies, execution_time = manager.analyze_query(query)
+                        
+                        if not anomalies:
+                            print("\n" + "="*40)
+                            print("✅ SUCCESS: The query is properly indexed and optimized.")
+                            print(f"   Execution Time: {execution_time} ms (Index Scan active)")
+                            print("="*40)
+                            continue
+                            
+                        for am in anomalies:
+                            print("\n" + "="*40)
+                            print(f"⚠️ ANOMALY DETECTED: Seq Scan on table '{am['table']}'")
+                            print(f"   Execution Time: {am.get('execution_time', 0.0)} ms")
+                            
+                            q_filter = am.get('filter')
+                            if q_filter:
+                                print(f"   Offending filter clause: {q_filter}")
+                            else:
+                                print("   Offending filter clause: [Full Table Scan - No filter constraint applied]")
+                            print("="*40)
+                                    
+                    except Exception as e:
+                        connection.rollback()
+                        print(f"\n❌ SQL SYNTAX OR DATABASE ERROR: {e}")
+                        print("Please verify your table names, column names, or SQL syntax.")
+                        continue
+                                
+            elif choice == "2":
+                print("\n🔎 Scanning relational schemas for unused metadata indexes...")
+                unused = manager.get_unused_indexes()
+                if not unused:
+                    print("✅ Perfect! No dead indexes found in the database catalog.")
+                    continue
+                    
+                print(f"⚠️ Identified {len(unused)} unused indexes slowing down write operations:")
+                for idx in unused:
+                    print(f"- '{idx['index']}' on table '{idx['table']}'")
+                    confirm = input(f"  Drop index asynchronously (CONCURRENTLY)? [s/N]: ").lower().strip()
+                    if confirm == 's':
+                        try:
+                            print(f"  Removing index metadata in background...")
+                            manager.drop_index_safely(idx['index'], idx['schema'])
+                            print(f"  🚀 Index '{idx['index']}' dropped successfully!")
+                        except Exception as e:
+                            print(f"  ❌ Runtime execution error: {e}")
+                    else:
+                        print("  Skipped.")
+            else:
+                print("\n❌ Invalid choice. Please enter 1, 2, or 'quit'.")
+
+    except KeyboardInterrupt:
+        print("\n\n🛑 [Control+C] Interrupted by user. Exiting cleanly. Goodbye!")
+        sys.exit(0)

pg_idx_manager-0.1.0/pg_idx_manager/core.py

@@ -0,0 +1,107 @@
+import logging
+
+class IndexManagerCore:
+    """
+    Core engine handling PostgreSQL index auditing and safe background cleanup.
+    Does not require superuser privileges or invasive external extensions.
+    """
+    def __init__(self, connection, min_table_rows=1000):
+        self.conn = connection
+        self.min_table_rows = min_table_rows
+
+    def _get_table_rows(self, cursor, table_name):
+        """Queries PostgreSQL system catalogs to fetch live tuple statistics."""
+        try:
+            cursor.execute(
+                "SELECT n_live_tup FROM pg_stat_user_tables WHERE relname = %s", 
+                (table_name,)
+            )
+            res = cursor.fetchone()
+            # res è una tupla tipo (10000,). Estraiamo il primo elemento res[0]
+            return res[0] if res else 0
+        except Exception:
+            return 0
+
+
+    def parse_explain_plan(self, node, anomalies=None):
+        """Recursively traverses the EXPLAIN JSON tree to detect Sequential Scans."""
+        if anomalies is None:
+            anomalies = []
+
+        if node.get("Node Type") == "Seq Scan":
+            anomalies.append({
+                "table": node.get("Relation Name"),
+                "filter": node.get("Filter")
+            })
+
+        if "Plans" in node:
+            for sub_node in node["Plans"]:
+                self.parse_explain_plan(sub_node, anomalies)
+
+        return anomalies
+
+    def analyze_query(self, query):
+        """
+        Executes an EXPLAIN (ANALYZE, FORMAT JSON) on the target query.
+        Safely unpacks the nested driver structure without bracket syntax bugs.
+        """
+        anomalies_detected = []
+        with self.conn.cursor() as cursor:
+            cursor.execute(f"EXPLAIN (ANALYZE, FORMAT JSON) {query}")
+            raw_result = cursor.fetchone()
+            
+            # Step 1: Extract the list from the psycopg2 tuple
+            if isinstance(raw_result, tuple) and len(raw_result) > 0:
+                raw_result = raw_result[0]
+            
+            # Step 2: Extract the internal dictionary from the list
+            if isinstance(raw_result, list) and len(raw_result) > 0:
+                raw_result = raw_result[0]
+                
+            # Final validation: check if we successfully unpacked into a dictionary
+            if not isinstance(raw_result, dict):
+                raise ValueError("Failed to parse PostgreSQL JSON plan into a dictionary configuration.")
+                
+            plan = raw_result.get("Plan")
+            execution_time_ms = raw_result.get("Execution Time", 0.0)
+            
+            if not plan:
+                raise ValueError("The 'Plan' root node is missing from the database engine response.")
+                
+            raw_anomalies = self.parse_explain_plan(plan)
+            
+            for anomaly in raw_anomalies:
+                table = anomaly["table"]
+                rows = self._get_table_rows(cursor, table)
+                
+                if rows >= self.min_table_rows:
+                    anomaly["table_rows"] = rows
+                    anomaly["execution_time"] = execution_time_ms
+                    anomalies_detected.append(anomaly)
+                    
+        return anomalies_detected, execution_time_ms
+
+
+
+    def get_unused_indexes(self):
+        """Scans pg_stat_user_indexes for dead indexes (idx_scan = 0)."""
+        sql = """
+            SELECT schemaname, relname AS table_name, indexrelname AS index_name
+            FROM pg_stat_user_indexes
+            JOIN pg_index ON pg_stat_user_indexes.indexrelid = pg_index.indexrelid
+            WHERE idx_scan = 0 AND indisunique = FALSE
+            ORDER BY relname ASC;
+        """
+        unused = []
+        with self.conn.cursor() as cursor:
+            cursor.execute(sql)
+            for row in cursor.fetchall():
+                unused.append({"schema": row[0], "table": row[1], "index": row[2]})
+        return unused
+
+    def drop_index_safely(self, index_name, schema="public"):
+        """Drops an index asynchronously without acquiring exclusive locks."""
+        sql = f"DROP INDEX CONCURRENTLY {schema}.{index_name};"
+        with self.conn.cursor() as cursor:
+            cursor.execute(sql)
+            self.conn.commit()

pg_idx_manager-0.1.0/pg_idx_manager.egg-info/PKG-INFO

@@ -0,0 +1,67 @@
+Metadata-Version: 2.4
+Name: pg_idx_manager
+Version: 0.1.0
+Summary: A lightweight, framework-agnostic developer linter and cleanup tool for PostgreSQL indexes.
+Author-email: Pierpaolo <tua_email@example.com>
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Database :: Database Engines/Servers
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: psycopg2-binary>=2.9.0
+
+# PG Index Manager & Janitor 🚀
+
+A lightweight, framework-agnostic Python library designed for developers to audit database queries, surface runtime performance bottlenecks, and safely maintain PostgreSQL indexes.
+
+Unlike heavy Application Performance Monitoring (APM) tools that require invasive database extensions (like HypoPG) or expensive SaaS subscriptions, this library operates entirely inside your application or via a clean CLI using native PostgreSQL capabilities.
+
+## The Problem It Solves: "ORM Blindness"
+Modern Object-Relational Mappers (ORMs) like Django or SQLAlchemy maximize developer productivity but hide the underlying SQL execution plan. A seemingly innocent line of Python code can silently trigger a **Sequential Scan (Full Table Scan)** across millions of rows, saturating server CPU and disk I/O.
+
+Since Large Language Models (AI) cannot inspect your production database cardinality, table sizes, or live index catalogs, they cannot reliably predict query performance. This library bridges that gap by running live `EXPLAIN ANALYZE` inspections directly on the database engine.
+
+## Key Features
+
+1. **Agnostic Performance Auditing**: Intercepts raw SQL queries, recursively parses the native PostgreSQL execution tree, and catches performance anomalies (**Sequential Scans**) with exact millisecond runtimes.
+2. **Safe Janitor Mode (Interactive CLI)**: Scans PostgreSQL system catalogs (`pg_stat_user_indexes`) to discover dead, unused indexes that are slowing down your `INSERT`/`UPDATE` mutations. It allows you to drop them interactively using `DROP INDEX CONCURRENTLY` without locking tables or risking production application downtime.
+3. **Zero-Dependency Architecture**: Does not require root or superuser privileges on the database server. If you can connect to the database, you can run this library.
+
+## Architectural Architecture: Where does it live?
+Because the core engine requires only a raw query string and a standard database connection, it is completely independent of your web framework. You can integrate it at the lowest layer of your infrastructure:
+
+* **At the Driver Level (`psycopg2`)**: By extending the native database cursor, you can automatically audit **100% of your application queries** (Django, FastAPI, Flask, or raw SQL) before they hit the wire.
+* **At the ORM Engine Level**: Easily hooks into global ORM events (e.g., SQLAlchemy's `before_cursor_execute`) to catch hidden query costs implicitly across all Services and Repositories with zero business-code modification.
+
+---
+
+## Getting Started & Testing Locally 🧪
+
+This repository includes a fully containerized testing environment to observe performance optimization in real-time.
+
+### 1. Spin up the isolated test database
+```bash
+docker compose up -d
+```
+
+### 2. Install the library locally in editable mode
+```bash
+pip install -e .
+```
+
+### 3. Run the Core Agnostic Test
+This script populates the database with **10,000 mock records**, executes an unindexed query, catches the Sequential Scan risk, and launches the persistent interactive CLI:
+```bash
+python tests/run_test.py
+```
+
+### 4. Run the ORM Integration Test
+Observe how the library seamlessly intercepts real-time query metrics generated implicitly by SQLAlchemy ORM models:
+```bash
+python tests/test_orm.py
+```
+
+## License
+MIT License. Free to use and extend.

pg_idx_manager-0.1.0/pg_idx_manager.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+README.md
+pyproject.toml
+pg_idx_manager/__init__.py
+pg_idx_manager/cli.py
+pg_idx_manager/core.py
+pg_idx_manager.egg-info/PKG-INFO
+pg_idx_manager.egg-info/SOURCES.txt
+pg_idx_manager.egg-info/dependency_links.txt
+pg_idx_manager.egg-info/requires.txt
+pg_idx_manager.egg-info/top_level.txt
+tests/test_orm.py

pg_idx_manager-0.1.0/pg_idx_manager.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

pg_idx_manager-0.1.0/pg_idx_manager.egg-info/requires.txt

@@ -0,0 +1 @@
+psycopg2-binary>=2.9.0

pg_idx_manager-0.1.0/pg_idx_manager.egg-info/top_level.txt

@@ -0,0 +1 @@
+pg_idx_manager

pg_idx_manager-0.1.0/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=61.0.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pg_idx_manager"
+version = "0.1.0"
+description = "A lightweight, framework-agnostic developer linter and cleanup tool for PostgreSQL indexes."
+readme = "README.md"
+requires-python = ">=3.8"
+authors = [{ name = "Pierpaolo", email = "tua_email@example.com" }]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Intended Audience :: Developers",
+    "Topic :: Database :: Database Engines/Servers",
+]
+dependencies = [
+    "psycopg2-binary>=2.9.0"
+]
+
+[tool.setuptools]
+packages = ["pg_idx_manager"]

pg_idx_manager-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pg_idx_manager-0.1.0/tests/test_orm.py

@@ -0,0 +1,44 @@
+from sqlalchemy import create_engine, Column, Integer, String, Numeric
+from sqlalchemy.orm import declarative_base, sessionmaker
+from pg_index_manager.decorators import audit_index
+
+# 1. Database Connection Configuration
+DATABASE_URL = "postgresql+psycopg2://tester:supersecretpassword@localhost:5432/testing_perf"
+engine = create_engine(DATABASE_URL)
+SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
+Base = declarative_base()
+
+# 2. ORM Model Mapping
+class Order(Base):
+    __tablename__ = "orders"
+    id = Column(Integer, primary_key=True, index=True)
+    customer_name = Column(String)
+    status = Column(String)
+    amount = Column(Numeric)
+
+# 3. Intercepted Function
+@audit_index(conn_param="raw_conn", min_rows=0)
+def execute_backend_task(raw_conn, query):
+    with raw_conn.cursor() as cursor:
+        cursor.execute(query)
+        return cursor.fetchall()
+
+if __name__ == "__main__":
+    db_session = SessionLocal()
+    
+    try:
+        # Recuperiamo la connessione psycopg2 grezza in modo moderno
+        raw_connection = db_session.connection()._dbapi_connection
+
+        # Metti qui la tua query da testare
+        orm_query = db_session.query(Order).filter(Order.customer_name == "pierpaolo")
+        
+        # Compile and bind parameters to get the raw SQL string
+        sql_string = str(orm_query.statement.compile(engine, compile_kwargs={"literal_binds": True}))
+        
+        # Execute: the decorator handles the audit output automatically
+        execute_backend_task(raw_conn=raw_connection, query=sql_string)
+
+    finally:
+        db_session.close()
+