pg_reports 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7a57deebae9eda0df4aceb3cfeb1912e2559a7cd70c3c7f197fb86f06bb689b1
-  data.tar.gz: 9369b5d6f54d8b0f2939f57d429f477aa0a8c2a862add70e2116232f364fe1e7
+  metadata.gz: 05027e6e3278a707d98301a3d4a44dbce9930d26e963359d1afb01b57626df0c
+  data.tar.gz: 0c03b76ef459a04fa4ea732e062fd73020d814c5f11bcc6e3f0a771d17e66976
 SHA512:
-  metadata.gz: 0e6248b26056ffe059105c2399f7a9c8e607e3f34030d4af6f14b324deb76435163b8b2116ff5d7502350410f0a6a20368ac3757b9247d30328da3330a941a86
-  data.tar.gz: 5d67f8d3c5663421b839301c3c4463502b7cbaeb57d3351d808ff00ebb355141b6c7cf676df091f7a2bb52238809cb02b8e9123d16b7a597bbc636e0cf888540
+  metadata.gz: a5e8a8eae451b10265dcbd3a9839f21d7c647d0876e7a41b7ac8e2a2219b87bbfbd45ec621f6ac552aba1c1cb8284005a114ca8758c3bd61e9cf3440ca6838d6
+  data.tar.gz: cf37e2c3458b8f24ca6bc548f7d58dea29af6547c14bf1c6db1468962fa1ec6e2095a9df8923045e5b8e86c6cf8c00e054e90e21cd803c7137d10db96e3f12de

data/CHANGELOG.md

@@ -5,6 +5,136 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.5.1] - 2026-02-09
+
+### Added
+
+- **Query Execution Security** - new configuration to control raw SQL execution from dashboard:
+  - New config option `allow_raw_query_execution` (default: `false`)
+  - Environment variable support: `PG_REPORTS_ALLOW_RAW_QUERY_EXECUTION`
+  - Security documentation in README with examples and best practices
+  - Frontend UI: disabled buttons with tooltips when feature is off
+  - JavaScript validation in `executeQuery()` and `executeExplainAnalyze()` functions
+  - Configuration tests for new security setting
+
+### Changed
+
+- **BREAKING CHANGE**: `execute_query` and `explain_analyze` endpoints now require explicit opt-in
+  - Both endpoints return 403 Forbidden when `allow_raw_query_execution` is `false` (default)
+  - Dashboard "Execute Query" and "EXPLAIN ANALYZE" buttons disabled by default
+  - To restore previous behavior, add to initializer: `config.allow_raw_query_execution = true`
+  - **Migration path**: Users must explicitly enable this feature if they were using Query Analyzer
+
+### Security
+
+- Raw SQL execution from dashboard is now **disabled by default** to prevent unauthorized data access
+- Recommended setup: only enable in development/staging environments
+- Existing safety measures (SELECT/SHOW only, automatic LIMIT) still apply when enabled
+
+## [0.5.0] - 2026-02-07
+
+### Added
+
+- **EXPLAIN ANALYZE Advanced Analyzer** - intelligent query plan analysis with problem detection:
+  - New `ExplainAnalyzer` service class for parsing and analyzing EXPLAIN output
+  - Color-coded node types (🟢 efficient, 🔵 normal, 🟡 potential issues)
+  - Automatic problem detection:
+    - Sequential scans on large tables (cost > 1000, rows > 1000)
+    - High-cost operations (> 10,000)
+    - Sort operations spilling to disk
+    - Slow sorts (> 1 second)
+    - Inaccurate row estimates (> 10x deviation)
+    - Slow execution/planning times
+  - Summary card with overall status (🟢 No issues / 🟡 Warnings / 🔴 Critical)
+  - Problem list with detailed explanations and recommendations
+  - Line-by-line plan annotations with problem indicators
+  - Metric highlighting (cost, rows, time, loops)
+  - Copy to clipboard functionality
+- **Connection Pool Analytics** - comprehensive pool monitoring and diagnostics:
+  - `pool_usage` report - real-time utilization metrics per database:
+    - Active, idle, idle-in-transaction connection breakdown
+    - Utilization percentage with thresholds (70% warning, 85% critical)
+    - Available connection capacity calculation
+  - `pool_wait_times` report - resource wait analysis:
+    - Queries waiting for locks, I/O, or network operations
+    - Wait event type classification (ClientRead, Lock, IO)
+    - Duration tracking with severity thresholds (10s warning, 60s critical)
+  - `pool_saturation` report - health warnings with recommendations:
+    - Overall pool metrics with status indicators (🟢🟡🔴)
+    - Automatic severity assessment per metric
+    - Context-aware recommendations embedded in SQL
+    - Tracks total, active, idle, idle-in-transaction, and waiting connections
+  - `connection_churn` report - lifecycle and churn analysis:
+    - Connection age distribution per application
+    - Short-lived connection detection (< 10 seconds)
+    - Churn rate percentage calculation (50% warning, 75% critical)
+    - Identifies missing or misconfigured connection pooling
+- Complete i18n translations (English and Russian) for all new reports
+- Documentation for each report with usage patterns and nuances
+- **SQL Query Monitoring** - real-time query capture and analysis:
+  - New `QueryMonitor` singleton service for capturing all SQL queries via ActiveSupport::Notifications
+  - Dashboard panel with start/stop controls and live query feed
+  - Query capture features:
+    - SQL text, execution duration (color-coded: green < 10ms, yellow < 100ms, red > 100ms)
+    - Source location (file:line) with click-to-open in IDE
+    - Timestamp and query name
+    - Session-based tracking with unique UUIDs
+  - Smart filtering:
+    - Automatically excludes SCHEMA, CACHE, EXPLAIN queries
+    - Filters DDL statements (CREATE, ALTER, DROP)
+    - Excludes pg_reports' internal queries
+    - Configurable backtrace filtering for source location extraction
+  - UI features:
+    - Collapsible/expandable queries (truncated to 100 chars by default)
+    - Real-time updates via 2-second polling
+    - Reverse chronological order (newest first)
+    - Query counter with session badge
+  - Persistence:
+    - JSON Lines (JSONL) log format in `log/pg_reports.log`
+    - Session markers (session_start/session_end)
+    - In-memory circular buffer (configurable, default 100 queries)
+    - Automatic log loading on dashboard open
+    - Results persist after stopping monitoring
+  - Export capabilities:
+    - Download in TXT, CSV, or JSON formats
+    - Works even after monitoring stopped
+    - Includes all query metadata (timestamp, duration, source, SQL)
+    - Uses hidden iframe for downloads (doesn't interrupt monitoring)
+  - Configuration options:
+    - `query_monitor_log_file` - custom log file path
+    - `query_monitor_max_queries` - buffer size (default: 100)
+    - `query_monitor_backtrace_filter` - Proc for filtering backtrace lines
+  - New routes and controller actions:
+    - `POST /query_monitor/start` - start monitoring
+    - `POST /query_monitor/stop` - stop monitoring
+    - `GET /query_monitor/status` - check monitoring status
+    - `GET /query_monitor/feed` - get live query feed
+    - `GET /query_monitor/history` - load queries from log file
+    - `GET /query_monitor/download` - export queries
+  - Comprehensive test coverage (39 unit tests + 5 integration tests)
+
+### Changed
+
+- **Unified status indicators** - consistent 🟢🟡🔴 emoji usage across all reports:
+  - Replaced ✅ checkmark with 🟢 green circle for "good" status
+  - Replaced ⚠️ warning sign with 🟡 yellow circle for "warning" status
+  - Retained 🔴 red circle for "critical" status
+  - Applied to EXPLAIN analyzer summary and connection pool reports
+- **Simplified database filtering** - all reports now use only current database from project settings:
+  - Removed database selector UI component from dashboard
+  - All SQL queries now filter by `current_database()` function automatically
+  - Current database name displayed in Live Monitoring header
+  - Removed `database` parameter from all query reports
+  - Removed `databases_list` endpoint and related routes
+  - Cleaner, more focused dashboard experience
+- **Optimized gem dependencies** - replaced full Rails framework dependency with specific components:
+  - Now using `activesupport`, `activerecord`, `actionpack`, `railties` instead of `rails`
+  - Removed unnecessary components: actioncable, actionmailer, actiontext, activejob, activestorage
+  - Reduced total dependencies from 97 to 80 gems (-17.5%)
+  - Faster installation and smaller footprint
+
 ## [0.4.0] - 2026-01-29
 
 ### Added

data/README.md

@@ -1,5 +1,6 @@
 # PgReports
 
+[![Gem Version](https://badge.fury.io/rb/pg_reports.svg)](https://badge.fury.io/rb/pg_reports)
 [![Ruby](https://img.shields.io/badge/Ruby-2.7%2B-red.svg)](https://www.ruby-lang.org/)
 [![Rails](https://img.shields.io/badge/Rails-5.0%2B-red.svg)](https://rubyonrails.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -20,7 +21,9 @@ A comprehensive PostgreSQL monitoring and analysis library for Rails application
 - 📥 **Export** - Download reports in TXT, CSV, or JSON format
 - 🔗 **IDE Integration** - Open source locations in VS Code, Cursor, RubyMine, or IntelliJ (with WSL support)
 - 📌 **Comparison Mode** - Save records to compare before/after optimization
-- 📊 **EXPLAIN ANALYZE** - Run query plans directly from the dashboard
+- 📊 **EXPLAIN ANALYZE** - Advanced query plan analyzer with problem detection and recommendations
+- 🔍 **SQL Query Monitoring** - Real-time monitoring of all executed SQL queries with source location tracking
+- 🔌 **Connection Pool Analytics** - Monitor pool usage, wait times, saturation warnings, and connection churn
 - 🗑️ **Migration Generator** - Generate Rails migrations to drop unused indexes
 
 ## Installation
@@ -125,6 +128,47 @@ PgReports.configure do |config|
 end
 ```
 
+### Query Execution Security
+
+⚠️ **Security Warning**: By default, the dashboard **does not allow** executing raw SQL queries via "Execute Query" and "EXPLAIN ANALYZE" buttons. This prevents accidental or malicious query execution in production environments.
+
+To enable query execution (only in secure environments):
+
+```ruby
+PgReports.configure do |config|
+  # Enable query execution from dashboard (default: false)
+  config.allow_raw_query_execution = true
+end
+```
+
+Or via environment variable:
+
+```bash
+export PG_REPORTS_ALLOW_RAW_QUERY_EXECUTION=true
+```
+
+**Recommended setup** (only enable in development/staging):
+
+```ruby
+# config/initializers/pg_reports.rb
+PgReports.configure do |config|
+  # Only allow query execution in development/staging
+  config.allow_raw_query_execution = Rails.env.development? || Rails.env.staging?
+
+  # Combine with authentication for additional security
+  config.dashboard_auth = -> {
+    authenticate_or_request_with_http_basic do |user, pass|
+      user == ENV["PG_REPORTS_USER"] && pass == ENV["PG_REPORTS_PASSWORD"]
+    end
+  }
+end
+```
+
+When disabled:
+- API endpoints `/execute_query` and `/explain_analyze` return 403 Forbidden
+- UI buttons are disabled with explanation tooltips
+- Existing safety measures (SELECT/SHOW only, automatic LIMIT) still apply when enabled
+
 ## Query Source Tracking
 
 PgReports automatically parses query annotations to show **where queries originated**. Works with:
@@ -194,6 +238,10 @@ config.active_record.query_log_tags = [:controller, :action]
 | `blocking_queries` | Queries blocking others |
 | `locks` | Current database locks |
 | `idle_connections` | Idle connections |
+| `pool_usage` | 🆕 Connection pool utilization analysis |
+| `pool_wait_times` | 🆕 Resource wait time analysis |
+| `pool_saturation` | 🆕 Pool health warnings with recommendations |
+| `connection_churn` | 🆕 Connection lifecycle and churn rate analysis |
 | `kill_connection(pid)` | Terminate a backend process |
 | `cancel_query(pid)` | Cancel a running query |
 
@@ -267,6 +315,7 @@ The dashboard provides:
 
 - 📊 Overview of all report categories with descriptions
 - ⚡ One-click report execution
+- 🔍 Filter parameters - adjust thresholds and limits on the fly
 - 🔍 Expandable rows for full query text
 - 📋 Copy query to clipboard
 - 📥 Download in multiple formats (TXT, CSV, JSON)
@@ -291,6 +340,16 @@ Click on source locations in reports to open the file directly in your IDE. Supp
 
 Use the ⚙️ button to set your default IDE and skip the selection menu.
 
+### Filter Parameters
+
+Each report page includes a collapsible "Параметры фильтрации" (Filter Parameters) section where you can:
+
+1. **Adjust thresholds** - Override default thresholds (e.g., slow query threshold, unused index scans)
+2. **Change limits** - Set the maximum number of results to display
+3. **Real-time refresh** - Reports automatically refresh when you change parameters
+
+Parameters show their current configured values and allow you to experiment with different thresholds without changing your configuration file.
+
 ### Save Records for Comparison
 
 When optimizing queries, you can save records to compare before/after results:
@@ -304,13 +363,85 @@ Records are stored in browser localStorage per report type.
 
 ### EXPLAIN ANALYZE
 
-For query reports, you can run EXPLAIN ANALYZE directly:
+The advanced query analyzer provides intelligent problem detection and recommendations:
 
 1. Expand a row with a query
 2. Click "📊 EXPLAIN ANALYZE"
-3. View the execution plan with timing statistics
+3. View the color-coded execution plan with:
+   - **🟢🟡🔴 Status indicator** - Overall query health assessment
+   - **📈 Key metrics** - Planning/Execution time, Cost, Rows
+   - **⚠️ Detected problems** - Sequential scans, high costs, slow sorts, estimation errors
+   - **💡 Recommendations** - Actionable advice for each issue
+   - **🎨 Colored plan** - Node types color-coded by performance impact:
+     - 🟢 Green: Efficient operations (Index Scan, Hash Join)
+     - 🔵 Blue: Normal operations (Bitmap Scan, HashAggregate)
+     - 🟡 Yellow: Potential issues (Seq Scan, Sort, Materialize)
+   - **Line-by-line annotations** - Problems highlighted on specific plan lines
+
+**Problem Detection:**
+- Sequential scans on large tables (> 1000 rows)
+- High-cost operations (> 10,000 cost units)
+- Sorts spilling to disk
+- Slow sort operations (> 1s)
+- Inaccurate row estimates (> 10x off)
+- Slow execution/planning times
+
+> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements require parameter input before analysis.
+
+### SQL Query Monitoring
+
+Monitor all SQL queries executed in your Rails application in real-time:
+
+1. Visit the dashboard at `/pg_reports`
+2. Click **"▶ Start Monitoring"** button in the SQL Query Monitor panel
+3. Execute operations in your application (web requests, console commands, background jobs)
+4. View captured queries in the dashboard with:
+   - **SQL text** - Formatted with syntax highlighting
+   - **Execution duration** - Color-coded: 🟢 green (< 10ms), 🟡 yellow (< 100ms), 🔴 red (> 100ms)
+   - **Source location** - File:line with click-to-open in IDE
+   - **Timestamp** - When the query was executed
+5. Click **"⏹ Stop Monitoring"** when done
+
+**Features:**
+- Uses Rails' built-in **ActiveSupport::Notifications** (`sql.active_record` events)
+- Global monitoring session (shared by all dashboard users)
+- Automatically filters internal queries (SCHEMA, CACHE, pg_reports' own queries)
+- Keeps last N queries in memory (configurable, default 100)
+- 2-second auto-refresh while monitoring is active
+- Session-based tracking with unique IDs
+- Logged to `log/pg_reports.log` in JSON Lines format
+
+**Configuration:**
+
+```ruby
+PgReports.configure do |config|
+  # Query monitoring
+  config.query_monitor_log_file = Rails.root.join("log", "custom_monitor.log")
+  config.query_monitor_max_queries = 200  # Keep last 200 queries (default: 100)
+
+  # Custom backtrace filtering to show only application code
+  config.query_monitor_backtrace_filter = ->(location) {
+    !location.path.match?(%r{/(gems|ruby|railties)/})
+  }
+end
+```
+
+**Log Format:**
 
-> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements cannot be analyzed directly. Copy the query and replace parameters with actual values.
+The log file uses JSON Lines format (one JSON object per line):
+
+```json
+{"type":"session_start","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:30:00Z"}
+{"type":"query","session_id":"550e8400-e29b-41d4-a716-446655440000","sql":"SELECT * FROM users WHERE id = 1","duration_ms":2.34,"name":"User Load","source_location":{"file":"app/controllers/users_controller.rb","line":15,"method":"show"},"timestamp":"2024-02-07T10:30:01Z"}
+{"type":"session_end","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:35:00Z"}
+```
+
+**Use Cases:**
+- 🐛 Debug N+1 query problems during development
+- 🐌 Identify slow queries in real-time
+- 🔍 Track down source of unexpected queries
+- 📊 Monitor query patterns during feature development
+- 📚 Teaching tool for understanding ActiveRecord behavior
 
 ### Migration Generator
 
@@ -321,6 +452,41 @@ For unused or invalid indexes, generate Rails migrations:
 3. Copy the code or create the file directly
 4. The file opens automatically in your configured IDE
 
+### Connection Pool Analytics
+
+Monitor your connection pool health with specialized reports:
+
+**Pool Usage** - Real-time utilization metrics:
+- Total, active, idle connections per database
+- Pool utilization percentage with 🟢🟡🔴 indicators
+- Idle in transaction connections (resource waste)
+- Available connection capacity
+
+**Wait Times** - Identify resource bottlenecks:
+- Queries waiting for locks, I/O, or network
+- Wait event types (ClientRead, Lock, IO)
+- Wait duration with severity thresholds
+- Helps diagnose contention issues
+
+**Pool Saturation** - Health warnings with actionable recommendations:
+- Overall pool metrics with status indicators
+- Automatic severity assessment (Normal/Elevated/Warning/Critical)
+- Context-aware recommendations for each metric
+- Detects approaching exhaustion, high idle transactions
+
+**Connection Churn** - Lifecycle analysis:
+- Connection age distribution by application
+- Short-lived connection detection (< 10 seconds)
+- Churn rate percentage calculation
+- Identifies missing/misconfigured connection pooling
+
+```ruby
+# Console usage
+PgReports.pool_usage.display
+PgReports.pool_saturation.display
+PgReports.connection_churn.display
+```
+
 ### Authentication
 
 ```ruby