pg_reports 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7a57deebae9eda0df4aceb3cfeb1912e2559a7cd70c3c7f197fb86f06bb689b1
-  data.tar.gz: 9369b5d6f54d8b0f2939f57d429f477aa0a8c2a862add70e2116232f364fe1e7
+  metadata.gz: c7f6e8be280967b6768f1646c2cdb0fa75b1d4da7d559151faa5f6f73dd4dfdb
+  data.tar.gz: 2121ea4314ec252406728fb6c5e98314b9703921b22d40a0b8483754dafd483c
 SHA512:
-  metadata.gz: 0e6248b26056ffe059105c2399f7a9c8e607e3f34030d4af6f14b324deb76435163b8b2116ff5d7502350410f0a6a20368ac3757b9247d30328da3330a941a86
-  data.tar.gz: 5d67f8d3c5663421b839301c3c4463502b7cbaeb57d3351d808ff00ebb355141b6c7cf676df091f7a2bb52238809cb02b8e9123d16b7a597bbc636e0cf888540
+  metadata.gz: a3a5d23ca386675cc409a7f8536fcd2600d0a04f1576edd15ae0a939aa78e01226e25e776bbb0ed2c66dccf4b24121329c4263389f7b17a6ccc4cf7897063433
+  data.tar.gz: 411a3c78f3b45f3fa6c1a7a5bfee766cf0dd59bcab0bacf5e9b338971350b15d169413b1a6a99b540ecb850890bd2dafd60560e7d06e9990a023955310eba965

data/CHANGELOG.md

@@ -5,6 +5,110 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.5.0] - 2026-02-07
+
+### Added
+
+- **EXPLAIN ANALYZE Advanced Analyzer** - intelligent query plan analysis with problem detection:
+  - New `ExplainAnalyzer` service class for parsing and analyzing EXPLAIN output
+  - Color-coded node types (🟢 efficient, 🔵 normal, 🟡 potential issues)
+  - Automatic problem detection:
+    - Sequential scans on large tables (cost > 1000, rows > 1000)
+    - High-cost operations (> 10,000)
+    - Sort operations spilling to disk
+    - Slow sorts (> 1 second)
+    - Inaccurate row estimates (> 10x deviation)
+    - Slow execution/planning times
+  - Summary card with overall status (🟢 No issues / 🟡 Warnings / 🔴 Critical)
+  - Problem list with detailed explanations and recommendations
+  - Line-by-line plan annotations with problem indicators
+  - Metric highlighting (cost, rows, time, loops)
+  - Copy to clipboard functionality
+- **Connection Pool Analytics** - comprehensive pool monitoring and diagnostics:
+  - `pool_usage` report - real-time utilization metrics per database:
+    - Active, idle, idle-in-transaction connection breakdown
+    - Utilization percentage with thresholds (70% warning, 85% critical)
+    - Available connection capacity calculation
+  - `pool_wait_times` report - resource wait analysis:
+    - Queries waiting for locks, I/O, or network operations
+    - Wait event type classification (ClientRead, Lock, IO)
+    - Duration tracking with severity thresholds (10s warning, 60s critical)
+  - `pool_saturation` report - health warnings with recommendations:
+    - Overall pool metrics with status indicators (🟢🟡🔴)
+    - Automatic severity assessment per metric
+    - Context-aware recommendations embedded in SQL
+    - Tracks total, active, idle, idle-in-transaction, and waiting connections
+  - `connection_churn` report - lifecycle and churn analysis:
+    - Connection age distribution per application
+    - Short-lived connection detection (< 10 seconds)
+    - Churn rate percentage calculation (50% warning, 75% critical)
+    - Identifies missing or misconfigured connection pooling
+- Complete i18n translations (English and Russian) for all new reports
+- Documentation for each report with usage patterns and nuances
+- **SQL Query Monitoring** - real-time query capture and analysis:
+  - New `QueryMonitor` singleton service for capturing all SQL queries via ActiveSupport::Notifications
+  - Dashboard panel with start/stop controls and live query feed
+  - Query capture features:
+    - SQL text, execution duration (color-coded: green < 10ms, yellow < 100ms, red > 100ms)
+    - Source location (file:line) with click-to-open in IDE
+    - Timestamp and query name
+    - Session-based tracking with unique UUIDs
+  - Smart filtering:
+    - Automatically excludes SCHEMA, CACHE, EXPLAIN queries
+    - Filters DDL statements (CREATE, ALTER, DROP)
+    - Excludes pg_reports' internal queries
+    - Configurable backtrace filtering for source location extraction
+  - UI features:
+    - Collapsible/expandable queries (truncated to 100 chars by default)
+    - Real-time updates via 2-second polling
+    - Reverse chronological order (newest first)
+    - Query counter with session badge
+  - Persistence:
+    - JSON Lines (JSONL) log format in `log/pg_reports.log`
+    - Session markers (session_start/session_end)
+    - In-memory circular buffer (configurable, default 100 queries)
+    - Automatic log loading on dashboard open
+    - Results persist after stopping monitoring
+  - Export capabilities:
+    - Download in TXT, CSV, or JSON formats
+    - Works even after monitoring stopped
+    - Includes all query metadata (timestamp, duration, source, SQL)
+    - Uses hidden iframe for downloads (doesn't interrupt monitoring)
+  - Configuration options:
+    - `query_monitor_log_file` - custom log file path
+    - `query_monitor_max_queries` - buffer size (default: 100)
+    - `query_monitor_backtrace_filter` - Proc for filtering backtrace lines
+  - New routes and controller actions:
+    - `POST /query_monitor/start` - start monitoring
+    - `POST /query_monitor/stop` - stop monitoring
+    - `GET /query_monitor/status` - check monitoring status
+    - `GET /query_monitor/feed` - get live query feed
+    - `GET /query_monitor/history` - load queries from log file
+    - `GET /query_monitor/download` - export queries
+  - Comprehensive test coverage (39 unit tests + 5 integration tests)
+
+### Changed
+
+- **Unified status indicators** - consistent 🟢🟡🔴 emoji usage across all reports:
+  - Replaced ✅ checkmark with 🟢 green circle for "good" status
+  - Replaced ⚠️ warning sign with 🟡 yellow circle for "warning" status
+  - Retained 🔴 red circle for "critical" status
+  - Applied to EXPLAIN analyzer summary and connection pool reports
+- **Simplified database filtering** - all reports now use only current database from project settings:
+  - Removed database selector UI component from dashboard
+  - All SQL queries now filter by `current_database()` function automatically
+  - Current database name displayed in Live Monitoring header
+  - Removed `database` parameter from all query reports
+  - Removed `databases_list` endpoint and related routes
+  - Cleaner, more focused dashboard experience
+- **Optimized gem dependencies** - replaced full Rails framework dependency with specific components:
+  - Now using `activesupport`, `activerecord`, `actionpack`, `railties` instead of `rails`
+  - Removed unnecessary components: actioncable, actionmailer, actiontext, activejob, activestorage
+  - Reduced total dependencies from 97 to 80 gems (-17.5%)
+  - Faster installation and smaller footprint
+
 ## [0.4.0] - 2026-01-29
 
 ### Added

data/README.md

@@ -1,5 +1,6 @@
 # PgReports
 
+[![Gem Version](https://badge.fury.io/rb/pg_reports.svg)](https://badge.fury.io/rb/pg_reports)
 [![Ruby](https://img.shields.io/badge/Ruby-2.7%2B-red.svg)](https://www.ruby-lang.org/)
 [![Rails](https://img.shields.io/badge/Rails-5.0%2B-red.svg)](https://rubyonrails.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -20,7 +21,9 @@ A comprehensive PostgreSQL monitoring and analysis library for Rails application
 - 📥 **Export** - Download reports in TXT, CSV, or JSON format
 - 🔗 **IDE Integration** - Open source locations in VS Code, Cursor, RubyMine, or IntelliJ (with WSL support)
 - 📌 **Comparison Mode** - Save records to compare before/after optimization
-- 📊 **EXPLAIN ANALYZE** - Run query plans directly from the dashboard
+- 📊 **EXPLAIN ANALYZE** - Advanced query plan analyzer with problem detection and recommendations
+- 🔍 **SQL Query Monitoring** - Real-time monitoring of all executed SQL queries with source location tracking
+- 🔌 **Connection Pool Analytics** - Monitor pool usage, wait times, saturation warnings, and connection churn
 - 🗑️ **Migration Generator** - Generate Rails migrations to drop unused indexes
 
 ## Installation
@@ -194,6 +197,10 @@ config.active_record.query_log_tags = [:controller, :action]
 | `blocking_queries` | Queries blocking others |
 | `locks` | Current database locks |
 | `idle_connections` | Idle connections |
+| `pool_usage` | 🆕 Connection pool utilization analysis |
+| `pool_wait_times` | 🆕 Resource wait time analysis |
+| `pool_saturation` | 🆕 Pool health warnings with recommendations |
+| `connection_churn` | 🆕 Connection lifecycle and churn rate analysis |
 | `kill_connection(pid)` | Terminate a backend process |
 | `cancel_query(pid)` | Cancel a running query |
 
@@ -267,6 +274,7 @@ The dashboard provides:
 
 - 📊 Overview of all report categories with descriptions
 - ⚡ One-click report execution
+- 🔍 Filter parameters - adjust thresholds and limits on the fly
 - 🔍 Expandable rows for full query text
 - 📋 Copy query to clipboard
 - 📥 Download in multiple formats (TXT, CSV, JSON)
@@ -291,6 +299,16 @@ Click on source locations in reports to open the file directly in your IDE. Supp
 
 Use the ⚙️ button to set your default IDE and skip the selection menu.
 
+### Filter Parameters
+
+Each report page includes a collapsible "Параметры фильтрации" (Filter Parameters) section where you can:
+
+1. **Adjust thresholds** - Override default thresholds (e.g., slow query threshold, unused index scans)
+2. **Change limits** - Set the maximum number of results to display
+3. **Real-time refresh** - Reports automatically refresh when you change parameters
+
+Parameters show their current configured values and allow you to experiment with different thresholds without changing your configuration file.
+
 ### Save Records for Comparison
 
 When optimizing queries, you can save records to compare before/after results:
@@ -304,13 +322,85 @@ Records are stored in browser localStorage per report type.
 
 ### EXPLAIN ANALYZE
 
-For query reports, you can run EXPLAIN ANALYZE directly:
+The advanced query analyzer provides intelligent problem detection and recommendations:
 
 1. Expand a row with a query
 2. Click "📊 EXPLAIN ANALYZE"
-3. View the execution plan with timing statistics
+3. View the color-coded execution plan with:
+   - **🟢🟡🔴 Status indicator** - Overall query health assessment
+   - **📈 Key metrics** - Planning/Execution time, Cost, Rows
+   - **⚠️ Detected problems** - Sequential scans, high costs, slow sorts, estimation errors
+   - **💡 Recommendations** - Actionable advice for each issue
+   - **🎨 Colored plan** - Node types color-coded by performance impact:
+     - 🟢 Green: Efficient operations (Index Scan, Hash Join)
+     - 🔵 Blue: Normal operations (Bitmap Scan, HashAggregate)
+     - 🟡 Yellow: Potential issues (Seq Scan, Sort, Materialize)
+   - **Line-by-line annotations** - Problems highlighted on specific plan lines
+
+**Problem Detection:**
+- Sequential scans on large tables (> 1000 rows)
+- High-cost operations (> 10,000 cost units)
+- Sorts spilling to disk
+- Slow sort operations (> 1s)
+- Inaccurate row estimates (> 10x off)
+- Slow execution/planning times
+
+> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements require parameter input before analysis.
+
+### SQL Query Monitoring
+
+Monitor all SQL queries executed in your Rails application in real-time:
+
+1. Visit the dashboard at `/pg_reports`
+2. Click **"▶ Start Monitoring"** button in the SQL Query Monitor panel
+3. Execute operations in your application (web requests, console commands, background jobs)
+4. View captured queries in the dashboard with:
+   - **SQL text** - Formatted with syntax highlighting
+   - **Execution duration** - Color-coded: 🟢 green (< 10ms), 🟡 yellow (< 100ms), 🔴 red (> 100ms)
+   - **Source location** - File:line with click-to-open in IDE
+   - **Timestamp** - When the query was executed
+5. Click **"⏹ Stop Monitoring"** when done
+
+**Features:**
+- Uses Rails' built-in **ActiveSupport::Notifications** (`sql.active_record` events)
+- Global monitoring session (shared by all dashboard users)
+- Automatically filters internal queries (SCHEMA, CACHE, pg_reports' own queries)
+- Keeps last N queries in memory (configurable, default 100)
+- 2-second auto-refresh while monitoring is active
+- Session-based tracking with unique IDs
+- Logged to `log/pg_reports.log` in JSON Lines format
+
+**Configuration:**
+
+```ruby
+PgReports.configure do |config|
+  # Query monitoring
+  config.query_monitor_log_file = Rails.root.join("log", "custom_monitor.log")
+  config.query_monitor_max_queries = 200  # Keep last 200 queries (default: 100)
+
+  # Custom backtrace filtering to show only application code
+  config.query_monitor_backtrace_filter = ->(location) {
+    !location.path.match?(%r{/(gems|ruby|railties)/})
+  }
+end
+```
+
+**Log Format:**
 
-> Note: Queries with parameter placeholders ($1, $2) from pg_stat_statements cannot be analyzed directly. Copy the query and replace parameters with actual values.
+The log file uses JSON Lines format (one JSON object per line):
+
+```json
+{"type":"session_start","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:30:00Z"}
+{"type":"query","session_id":"550e8400-e29b-41d4-a716-446655440000","sql":"SELECT * FROM users WHERE id = 1","duration_ms":2.34,"name":"User Load","source_location":{"file":"app/controllers/users_controller.rb","line":15,"method":"show"},"timestamp":"2024-02-07T10:30:01Z"}
+{"type":"session_end","session_id":"550e8400-e29b-41d4-a716-446655440000","timestamp":"2024-02-07T10:35:00Z"}
+```
+
+**Use Cases:**
+- 🐛 Debug N+1 query problems during development
+- 🐌 Identify slow queries in real-time
+- 🔍 Track down source of unexpected queries
+- 📊 Monitor query patterns during feature development
+- 📚 Teaching tool for understanding ActiveRecord behavior
 
 ### Migration Generator
 
@@ -321,6 +411,41 @@ For unused or invalid indexes, generate Rails migrations:
 3. Copy the code or create the file directly
 4. The file opens automatically in your configured IDE
 
+### Connection Pool Analytics
+
+Monitor your connection pool health with specialized reports:
+
+**Pool Usage** - Real-time utilization metrics:
+- Total, active, idle connections per database
+- Pool utilization percentage with 🟢🟡🔴 indicators
+- Idle in transaction connections (resource waste)
+- Available connection capacity
+
+**Wait Times** - Identify resource bottlenecks:
+- Queries waiting for locks, I/O, or network
+- Wait event types (ClientRead, Lock, IO)
+- Wait duration with severity thresholds
+- Helps diagnose contention issues
+
+**Pool Saturation** - Health warnings with actionable recommendations:
+- Overall pool metrics with status indicators
+- Automatic severity assessment (Normal/Elevated/Warning/Critical)
+- Context-aware recommendations for each metric
+- Detects approaching exhaustion, high idle transactions
+
+**Connection Churn** - Lifecycle analysis:
+- Connection age distribution by application
+- Short-lived connection detection (< 10 seconds)
+- Churn rate percentage calculation
+- Identifies missing/misconfigured connection pooling
+
+```ruby
+# Console usage
+PgReports.pool_usage.display
+PgReports.pool_saturation.display
+PgReports.connection_churn.display
+```
+
 ### Authentication
 
 ```ruby

data/app/controllers/pg_reports/dashboard_controller.rb

@@ -9,6 +9,7 @@ module PgReports
 
     def index
       @pg_stat_status = PgReports.pg_stat_statements_status
+      @current_database = PgReports.system.current_database
     end
 
     def enable_pg_stat_statements
@@ -164,22 +165,18 @@ module PgReports
       result = ActiveRecord::Base.connection.execute("EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) #{final_query}")
       explain_output = result.map { |r| r["QUERY PLAN"] }.join("\n")
 
-      # Extract stats from the output
-      stats = {}
-      if (match = explain_output.match(/Planning Time: ([\d.]+) ms/))
-        stats[:planning_time] = match[1].to_f
-      end
-      if (match = explain_output.match(/Execution Time: ([\d.]+) ms/))
-        stats[:execution_time] = match[1].to_f
-      end
-      if (match = explain_output.match(/cost=[\d.]+\.\.([\d.]+)/))
-        stats[:total_cost] = match[1].to_f
-      end
-      if (match = explain_output.match(/rows=(\d+)/))
-        stats[:rows] = match[1].to_i
-      end
+      # Analyze the EXPLAIN output
+      analyzer = ExplainAnalyzer.new(explain_output)
+      analysis = analyzer.to_h
 
-      render json: {success: true, explain: explain_output, stats: stats}
+      render json: {
+        success: true,
+        explain: explain_output,
+        stats: analysis[:stats],
+        annotated_lines: analysis[:annotated_lines],
+        problems: analysis[:problems],
+        summary: analysis[:summary]
+      }
     rescue => e
       render json: {success: false, error: e.message}, status: :unprocessable_entity
     end
@@ -195,7 +192,7 @@ module PgReports
 
       # Security: Only allow SELECT and SHOW queries
       normalized = query.strip.gsub(/\s+/, " ").downcase
-      unless normalized.start_with?("select") || normalized.start_with?("show")
+      unless normalized.start_with?("select", "show")
         render json: {success: false, error: "Only SELECT and SHOW queries are allowed"}, status: :unprocessable_entity
         return
       end
@@ -287,6 +284,123 @@ module PgReports
       render json: {success: false, error: e.message}, status: :unprocessable_entity
     end
 
+    def start_query_monitoring
+      monitor = QueryMonitor.instance
+
+      result = monitor.start
+
+      if result[:success]
+        render json: result
+      else
+        render json: result, status: :unprocessable_entity
+      end
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
+    def stop_query_monitoring
+      monitor = QueryMonitor.instance
+
+      result = monitor.stop
+
+      if result[:success]
+        render json: result
+      else
+        render json: result, status: :unprocessable_entity
+      end
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
+    def query_monitor_status
+      monitor = QueryMonitor.instance
+      status = monitor.status
+
+      render json: {
+        success: true,
+        enabled: status[:enabled],
+        session_id: status[:session_id],
+        query_count: status[:query_count]
+      }
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
+    def query_monitor_feed
+      monitor = QueryMonitor.instance
+
+      unless monitor.enabled
+        render json: {success: false, message: "Monitoring not active"}
+        return
+      end
+
+      limit = params[:limit]&.to_i || 50
+      session_id = params[:session_id]
+
+      queries = monitor.queries(limit: limit, session_id: session_id)
+
+      render json: {
+        success: true,
+        queries: queries,
+        timestamp: Time.current.to_i
+      }
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
+    def load_query_history
+      monitor = QueryMonitor.instance
+
+      limit = params[:limit]&.to_i || 100
+      session_id = params[:session_id]
+
+      queries = monitor.load_from_log(limit: limit, session_id: session_id)
+
+      render json: {
+        success: true,
+        queries: queries,
+        timestamp: Time.current.to_i
+      }
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
+    def download_query_monitor
+      monitor = QueryMonitor.instance
+
+      # Allow download even when monitoring is stopped, as long as there are queries
+      queries = monitor.queries
+      if queries.empty?
+        render json: {success: false, error: "No queries to download"}, status: :unprocessable_entity
+        return
+      end
+
+      format_type = params[:format] || "txt"
+      filename = "query-monitor-#{Time.current.strftime("%Y%m%d-%H%M%S")}"
+
+      case format_type
+      when "csv"
+        csv_data = generate_query_monitor_csv(queries)
+        send_data csv_data,
+          filename: "#{filename}.csv",
+          type: "text/csv; charset=utf-8",
+          disposition: "attachment"
+      when "json"
+        send_data queries.to_json,
+          filename: "#{filename}.json",
+          type: "application/json; charset=utf-8",
+          disposition: "attachment"
+      else
+        text_data = generate_query_monitor_text(queries)
+        send_data text_data,
+          filename: "#{filename}.txt",
+          type: "text/plain; charset=utf-8",
+          disposition: "attachment"
+      end
+    rescue => e
+      render json: {success: false, error: e.message}, status: :unprocessable_entity
+    end
+
     private
 
     def authenticate_dashboard!
@@ -326,7 +440,7 @@ module PgReports
 
       # Also allow threshold overrides (calls_threshold, etc.)
       params.each do |key, value|
-        if key.to_s.end_with?('_threshold') && value.present?
+        if key.to_s.end_with?("_threshold") && value.present?
           result[key.to_sym] = value.to_i
         end
       end
@@ -356,7 +470,7 @@ module PgReports
       result = query.dup
 
       # Sort by param number descending to replace $10 before $1
-      params_hash.keys.map(&:to_i).sort.reverse.each do |num|
+      params_hash.keys.map(&:to_i).sort.reverse_each do |num|
         value = params_hash[num.to_s] || params_hash[num]
         next if value.nil? || value.to_s.empty?
 
@@ -371,17 +485,15 @@ module PgReports
     def quote_param_value(value)
       str = value.to_s
 
-      # Check if it looks like a number
-      if str.match?(/\A-?\d+(\.\d+)?\z/)
-        str
+      # Check if it looks like NULL
+      if str.downcase == "null"
+        "NULL"
       # Check if it looks like a boolean
       elsif str.downcase.in?(["true", "false"])
         str.downcase
-      # Check if it looks like NULL
-      elsif str.downcase == "null"
-        "NULL"
       else
-        # Quote as string, escape single quotes
+        # Quote as string by default - PostgreSQL will handle type casting
+        # This ensures compatibility with both text and numeric columns
         "'#{str.gsub("'", "''")}'"
       end
     end
@@ -397,5 +509,56 @@ module PgReports
         "#{query} LIMIT #{limit}"
       end
     end
+
+    def generate_query_monitor_csv(queries)
+      require "csv"
+
+      CSV.generate do |csv|
+        # Header
+        csv << ["Timestamp", "Duration (ms)", "Query Name", "SQL", "Source File", "Source Line"]
+
+        # Data rows
+        queries.each do |query|
+          csv << [
+            query[:timestamp],
+            query[:duration_ms],
+            query[:name],
+            query[:sql],
+            query.dig(:source_location, :file),
+            query.dig(:source_location, :line)
+          ]
+        end
+      end
+    end
+
+    def generate_query_monitor_text(queries)
+      output = []
+      output << "=" * 80
+      output << "Query Monitor Export"
+      output << "Generated: #{Time.current.strftime("%Y-%m-%d %H:%M:%S")}"
+      output << "Total Queries: #{queries.size}"
+      output << "=" * 80
+      output << ""
+
+      queries.each_with_index do |query, index|
+        output << "Query ##{index + 1}"
+        output << "-" * 80
+        output << "Timestamp:  #{query[:timestamp]}"
+        output << "Duration:   #{query[:duration_ms]}ms"
+        output << "Name:       #{query[:name]}"
+
+        if query[:source_location]
+          output << "Source:     #{query[:source_location][:file]}:#{query[:source_location][:line]}"
+        end
+
+        output << ""
+        output << "SQL:"
+        output << query[:sql]
+        output << ""
+        output << ""
+      end
+
+      output.join("\n")
+    end
   end
 end