pg_reports 0.3.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: be278e9401573fc93ab6d619427d7356609bf26a03cb69ac6a5dbbc2f7b39409
-  data.tar.gz: b5fae395bbae259280751183344407b69eb2e90da268b380e5bfb97a6ddbf382
+  metadata.gz: c7f6e8be280967b6768f1646c2cdb0fa75b1d4da7d559151faa5f6f73dd4dfdb
+  data.tar.gz: 2121ea4314ec252406728fb6c5e98314b9703921b22d40a0b8483754dafd483c
 SHA512:
-  metadata.gz: f76f4ff3f31c36aae1b21dc080283d6a49d5a41b6b9405a569548eae21df5a8f5c7f0c205a88d712f84a0b4ad166486a82bec9dbed0a0185fc2301ca375ceb6f
-  data.tar.gz: cf31a2c98627e7d0a8aab41fa041682371fa75b2673500f9d65700a246cef7c6f323e6e1bc9238d4793b0e49a21785b0446cb3216de1c8759877c17e593532fc
+  metadata.gz: a3a5d23ca386675cc409a7f8536fcd2600d0a04f1576edd15ae0a939aa78e01226e25e776bbb0ed2c66dccf4b24121329c4263389f7b17a6ccc4cf7897063433
+  data.tar.gz: 411a3c78f3b45f3fa6c1a7a5bfee766cf0dd59bcab0bacf5e9b338971350b15d169413b1a6a99b540ecb850890bd2dafd60560e7d06e9990a023955310eba965

data/CHANGELOG.md

@@ -5,6 +5,146 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.5.0] - 2026-02-07
+
+### Added
+
+- **EXPLAIN ANALYZE Advanced Analyzer** - intelligent query plan analysis with problem detection:
+  - New `ExplainAnalyzer` service class for parsing and analyzing EXPLAIN output
+  - Color-coded node types (🟢 efficient, 🔵 normal, 🟡 potential issues)
+  - Automatic problem detection:
+    - Sequential scans on large tables (cost > 1000, rows > 1000)
+    - High-cost operations (> 10,000)
+    - Sort operations spilling to disk
+    - Slow sorts (> 1 second)
+    - Inaccurate row estimates (> 10x deviation)
+    - Slow execution/planning times
+  - Summary card with overall status (🟢 No issues / 🟡 Warnings / 🔴 Critical)
+  - Problem list with detailed explanations and recommendations
+  - Line-by-line plan annotations with problem indicators
+  - Metric highlighting (cost, rows, time, loops)
+  - Copy to clipboard functionality
+- **Connection Pool Analytics** - comprehensive pool monitoring and diagnostics:
+  - `pool_usage` report - real-time utilization metrics per database:
+    - Active, idle, idle-in-transaction connection breakdown
+    - Utilization percentage with thresholds (70% warning, 85% critical)
+    - Available connection capacity calculation
+  - `pool_wait_times` report - resource wait analysis:
+    - Queries waiting for locks, I/O, or network operations
+    - Wait event type classification (ClientRead, Lock, IO)
+    - Duration tracking with severity thresholds (10s warning, 60s critical)
+  - `pool_saturation` report - health warnings with recommendations:
+    - Overall pool metrics with status indicators (🟢🟡🔴)
+    - Automatic severity assessment per metric
+    - Context-aware recommendations embedded in SQL
+    - Tracks total, active, idle, idle-in-transaction, and waiting connections
+  - `connection_churn` report - lifecycle and churn analysis:
+    - Connection age distribution per application
+    - Short-lived connection detection (< 10 seconds)
+    - Churn rate percentage calculation (50% warning, 75% critical)
+    - Identifies missing or misconfigured connection pooling
+- Complete i18n translations (English and Russian) for all new reports
+- Documentation for each report with usage patterns and nuances
+- **SQL Query Monitoring** - real-time query capture and analysis:
+  - New `QueryMonitor` singleton service for capturing all SQL queries via ActiveSupport::Notifications
+  - Dashboard panel with start/stop controls and live query feed
+  - Query capture features:
+    - SQL text, execution duration (color-coded: green < 10ms, yellow < 100ms, red > 100ms)
+    - Source location (file:line) with click-to-open in IDE
+    - Timestamp and query name
+    - Session-based tracking with unique UUIDs
+  - Smart filtering:
+    - Automatically excludes SCHEMA, CACHE, EXPLAIN queries
+    - Filters DDL statements (CREATE, ALTER, DROP)
+    - Excludes pg_reports' internal queries
+    - Configurable backtrace filtering for source location extraction
+  - UI features:
+    - Collapsible/expandable queries (truncated to 100 chars by default)
+    - Real-time updates via 2-second polling
+    - Reverse chronological order (newest first)
+    - Query counter with session badge
+  - Persistence:
+    - JSON Lines (JSONL) log format in `log/pg_reports.log`
+    - Session markers (session_start/session_end)
+    - In-memory circular buffer (configurable, default 100 queries)
+    - Automatic log loading on dashboard open
+    - Results persist after stopping monitoring
+  - Export capabilities:
+    - Download in TXT, CSV, or JSON formats
+    - Works even after monitoring stopped
+    - Includes all query metadata (timestamp, duration, source, SQL)
+    - Uses hidden iframe for downloads (doesn't interrupt monitoring)
+  - Configuration options:
+    - `query_monitor_log_file` - custom log file path
+    - `query_monitor_max_queries` - buffer size (default: 100)
+    - `query_monitor_backtrace_filter` - Proc for filtering backtrace lines
+  - New routes and controller actions:
+    - `POST /query_monitor/start` - start monitoring
+    - `POST /query_monitor/stop` - stop monitoring
+    - `GET /query_monitor/status` - check monitoring status
+    - `GET /query_monitor/feed` - get live query feed
+    - `GET /query_monitor/history` - load queries from log file
+    - `GET /query_monitor/download` - export queries
+  - Comprehensive test coverage (39 unit tests + 5 integration tests)
+
+### Changed
+
+- **Unified status indicators** - consistent 🟢🟡🔴 emoji usage across all reports:
+  - Replaced ✅ checkmark with 🟢 green circle for "good" status
+  - Replaced ⚠️ warning sign with 🟡 yellow circle for "warning" status
+  - Retained 🔴 red circle for "critical" status
+  - Applied to EXPLAIN analyzer summary and connection pool reports
+- **Simplified database filtering** - all reports now use only current database from project settings:
+  - Removed database selector UI component from dashboard
+  - All SQL queries now filter by `current_database()` function automatically
+  - Current database name displayed in Live Monitoring header
+  - Removed `database` parameter from all query reports
+  - Removed `databases_list` endpoint and related routes
+  - Cleaner, more focused dashboard experience
+- **Optimized gem dependencies** - replaced full Rails framework dependency with specific components:
+  - Now using `activesupport`, `activerecord`, `actionpack`, `railties` instead of `rails`
+  - Removed unnecessary components: actioncable, actionmailer, actiontext, activejob, activestorage
+  - Reduced total dependencies from 97 to 80 gems (-17.5%)
+  - Faster installation and smaller footprint
+
+## [0.4.0] - 2026-01-29
+
+### Added
+
+- **YAML-based Report Configuration System** - declarative report definitions:
+  - Each report now defined in a single YAML file (1 file = 1 report approach)
+  - Migrated all 31 reports from Ruby methods to YAML format
+  - New classes: `ReportDefinition`, `Filter`, `ReportLoader`, `ModuleGenerator`
+  - Support for post-SQL filtering with operators (eq, ne, lt, lte, gt, gte)
+  - Title interpolation with variable substitution using `${variable}` syntax
+  - Enrichment hooks pattern for data transformation
+  - Problem explanations configuration in YAML (field → explanation key mapping)
+- **Filter UI** on report pages for manual parameter input:
+  - Collapsible filter section with form inputs for each parameter
+  - Support for threshold overrides (limit, min_duration_seconds, etc.)
+  - Real-time report refresh with custom filter values
+  - Horizontal layout with descriptions and inputs side-by-side
+  - Dark theme styling for filter inputs and labels
+
+### Changed
+
+- Report definitions now use declarative YAML files instead of repetitive Ruby methods (~800 lines of code eliminated)
+- Modules now use metaprogramming to dynamically generate methods from YAML at load time
+- Dashboard category cards width increased from 280px to 350px for better readability
+- Documentation and filter sections positioned side-by-side in responsive grid layout
+- Filter parameters section positioned below documentation and collapsed by default
+- Reduced vertical spacing between sections for more compact layout
+- Problem field to explanation key mapping moved from hardcoded JavaScript to server-driven YAML configuration
+
+### Fixed
+
+- CSS Grid `align-items: stretch` causing synchronized expansion of documentation and filter blocks
+- Visual "chin" appearing on closed filter details block due to default margins
+- TypeError when report has no limit parameter default value
+- Excessive margins between report sections
+
 ## [0.3.0] - 2026-01-28
 
 ### Added

data/README.md

@@ -1,5 +1,6 @@
 # PgReports
 
+[![Gem Version](https://badge.fury.io/rb/pg_reports.svg)](https://badge.fury.io/rb/pg_reports)
 [![Ruby](https://img.shields.io/badge/Ruby-2.7%2B-red.svg)](https://www.ruby-lang.org/)
 [![Rails](https://img.shields.io/badge/Rails-5.0%2B-red.svg)](https://rubyonrails.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -20,7 +21,9 @@ A comprehensive PostgreSQL monitoring and analysis library for Rails application
 - 📥 **Export** - Download reports in TXT, CSV, or JSON format
 - 🔗 **IDE Integration** - Open source locations in VS Code, Cursor, RubyMine, or IntelliJ (with WSL support)
 - 📌 **Comparison Mode** - Save records to compare before/after optimization
-- 📊 **EXPLAIN ANALYZE** - Run query plans directly from the dashboard
+- 📊 **EXPLAIN ANALYZE** - Advanced query plan analyzer with problem detection and recommendations
+- 🔍 **SQL Query Monitoring** - Real-time monitoring of all executed SQL queries with source location tracking
+- 🔌 **Connection Pool Analytics** - Monitor pool usage, wait times, saturation warnings, and connection churn
 - 🗑️ **Migration Generator** - Generate Rails migrations to drop unused indexes
 
 ## Installation
@@ -194,6 +197,10 @@ config.active_record.query_log_tags = [:controller, :action]
 | `blocking_queries` | Queries blocking others |
 | `locks` | Current database locks |
 | `idle_connections` | Idle connections |
+| `pool_usage` | 🆕 Connection pool utilization analysis |
+| `pool_wait_times` | 🆕 Resource wait time analysis |
+| `pool_saturation` | 🆕 Pool health warnings with recommendations |
+| `connection_churn` | 🆕 Connection lifecycle and churn rate analysis |
 | `kill_connection(pid)` | Terminate a backend process |
 | `cancel_query(pid)` | Cancel a running query |
 
@@ -267,6 +274,7 @@ The dashboard provides:
 
 - 📊 Overview of all report categories with descriptions
 - ⚡ One-click report execution
+- 🔍 Filter parameters - adjust thresholds and limits on the fly
 - 🔍 Expandable rows for full query text
 - 📋 Copy query to clipboard
 - 📥 Download in multiple formats (TXT, CSV, JSON)
@@ -291,6 +299,16 @@ Click on source locations in reports to open the file directly in your IDE. Supp
 
 Use the ⚙️ button to set your default IDE and skip the selection menu.
 
+### Filter Parameters
+
+Each report page includes a collapsible "Параметры фильтрации" (Filter Parameters) section where you can:
+
+1. **Adjust thresholds** - Override default thresholds (e.g., slow query threshold, unused index scans)
+2. **Change limits** - Set the maximum number of results to display
+3. **Real-time refresh** - Reports automatically refresh when you change parameters
+
+Parameters show their current configured values and allow you to experiment with different thresholds without changing your configuration file.
+
 ### Save Records for Comparison
 
 When optimizing queries, you can save records to compare before/after results:
@@ -304,13 +322,85 @@ Records are stored in browser localStorage per report type.
 
 ### EXPLAIN ANALYZE
 
-For query reports, you can run EXPLAIN ANALYZE directly:
+The advanced query analyzer provides intelligent problem detection and recommendations:
 
 1. Expand a row with a query
 2. Click "📊 EXPLAIN ANALYZE"
-3. View the execution plan with timing statistics
+3. View the color-coded execution plan with:
+   - **🟢🟡🔴 Status indicator** - Overall query health assessment
+   - **📈 Key metrics** - Planning/Execution time, Cost, Rows
+   - **⚠️ Detected problems** - Sequential scans, high costs, slow sorts, estimation errors
+   - **💡 Recommendations** - Actionable advice for each issue
+   - **🎨 Colored plan** - Node types color-coded by performance impact:
+     - 🟢 Green: Efficient operations (Index Scan, Hash Join)
+     - 🔵 Blue: Normal operations (Bitmap Scan, HashAggregate)
+     - 🟡 Yellow: Potential issues (Seq Scan, Sort, Materialize)
+   - **Line-by-line annotations** - Problems highlighted on specific plan lines
+
+**Problem Detection:**
+- Sequential scans on large tables (> 1000 rows)
+- High-cost operations (> 10,000 cost units)
+- Sorts spilling to disk
+- Slow sort operations (> 1s)
+- Inaccurate row estimates (> 10x off)
+- Slow execution/planning times
+
+> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements require parameter input before analysis.
+
+### SQL Query Monitoring
+
+Monitor all SQL queries executed in your Rails application in real-time:
+
+1. Visit the dashboard at `/pg_reports`
+2. Click **"▶ Start Monitoring"** button in the SQL Query Monitor panel
+3. Execute operations in your application (web requests, console commands, background jobs)
+4. View captured queries in the dashboard with:
+   - **SQL text** - Formatted with syntax highlighting
+   - **Execution duration** - Color-coded: 🟢 green (< 10ms), 🟡 yellow (< 100ms), 🔴 red (> 100ms)
+   - **Source location** - File:line with click-to-open in IDE
+   - **Timestamp** - When the query was executed
+5. Click **"⏹ Stop Monitoring"** when done
+
+**Features:**
+- Uses Rails' built-in **ActiveSupport::Notifications** (`sql.active_record` events)
+- Global monitoring session (shared by all dashboard users)
+- Automatically filters internal queries (SCHEMA, CACHE, pg_reports' own queries)
+- Keeps last N queries in memory (configurable, default 100)
+- 2-second auto-refresh while monitoring is active
+- Session-based tracking with unique IDs
+- Logged to `log/pg_reports.log` in JSON Lines format
+
+**Configuration:**
+
+```ruby
+PgReports.configure do |config|
+  # Query monitoring
+  config.query_monitor_log_file = Rails.root.join("log", "custom_monitor.log")
+  config.query_monitor_max_queries = 200  # Keep last 200 queries (default: 100)
+
+  # Custom backtrace filtering to show only application code
+  config.query_monitor_backtrace_filter = ->(location) {
+    !location.path.match?(%r{/(gems|ruby|railties)/})
+  }
+end
+```
+
+**Log Format:**
 
-> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements cannot be analyzed directly. Copy the query and replace parameters with actual values.
+The log file uses JSON Lines format (one JSON object per line):
+
+```json
+{"type":"session_start","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:30:00Z"}
+{"type":"query","session_id":"550e8400-e29b-41d4-a716-446655440000","sql":"SELECT * FROM users WHERE id = 1","duration_ms":2.34,"name":"User Load","source_location":{"file":"app/controllers/users_controller.rb","line":15,"method":"show"},"timestamp":"2024-02-07T10:30:01Z"}
+{"type":"session_end","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:35:00Z"}
+```
+
+**Use Cases:**
+- 🐛 Debug N+1 query problems during development
+- 🐌 Identify slow queries in real-time
+- 🔍 Track down source of unexpected queries
+- 📊 Monitor query patterns during feature development
+- 📚 Teaching tool for understanding ActiveRecord behavior
 
 ### Migration Generator
 
@@ -321,6 +411,41 @@ For unused or invalid indexes, generate Rails migrations:
 3. Copy the code or create the file directly
 4. The file opens automatically in your configured IDE
 
+### Connection Pool Analytics
+
+Monitor your connection pool health with specialized reports:
+
+**Pool Usage** - Real-time utilization metrics:
+- Total, active, idle connections per database
+- Pool utilization percentage with 🟢🟡🔴 indicators
+- Idle in transaction connections (resource waste)
+- Available connection capacity
+
+**Wait Times** - Identify resource bottlenecks:
+- Queries waiting for locks, I/O, or network
+- Wait event types (ClientRead, Lock, IO)
+- Wait duration with severity thresholds
+- Helps diagnose contention issues
+
+**Pool Saturation** - Health warnings with actionable recommendations:
+- Overall pool metrics with status indicators
+- Automatic severity assessment (Normal/Elevated/Warning/Critical)
+- Context-aware recommendations for each metric
+- Detects approaching exhaustion, high idle transactions
+
+**Connection Churn** - Lifecycle analysis:
+- Connection age distribution by application
+- Short-lived connection detection (< 10 seconds)
+- Churn rate percentage calculation
+- Identifies missing/misconfigured connection pooling
+
+```ruby
+# Console usage
+PgReports.pool_usage.display
+PgReports.pool_saturation.display
+PgReports.connection_churn.display
+```
+
 ### Authentication
 
 ```ruby