pg_insights 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3575be5a5a333d43c44c555f5f736c83e5458788143d92b5ca5ec765e3cbd083
-  data.tar.gz: be967ed421de1a002b52ef7b476fd58a02c866e6cf429ddeef053672831679a5
+  metadata.gz: 8d38a7931c85980d77ac00274e9fd20d0aeeccea9968fcdce682e094c2fa8859
+  data.tar.gz: 46ace0d92f363030da888b4fc3c0b81ba3429d6d46be11ed884e87cf953d982e
 SHA512:
-  metadata.gz: b1756de4bb9c8aa2bceab70f936d79a85d678376917f0976bf2b9fa90ef4cc54fe4683387166ba6b93fe2e72f50d879a7185c89f292a87b60f782e92790c38df
-  data.tar.gz: 17594af5eeeb6ed8812a6a7659635ce5efa032c6f67bb882fd8caf52dc617fe54b2dd2773c6883047be386c736f0826d6224cece06ebe06b0c6e0a5e49a2203c
+  metadata.gz: 72dc38b44223c63b08cf4a3021f53d96dca26c91a00447a92c9434f012add4f2693d02217fcbcc3d5a1a7117584ecab9b5248880a1510220ec24879591ec8b46
+  data.tar.gz: 54bfd327be06ff7b2032fbd937e5bd9f1b1ee32042349d58a34efe4025ffcc650e4d219586c0afd116cb21113ca28b3cb43d07d9e2fe8bbf85f717935d648770

data/README.md

@@ -1,39 +1,113 @@
-# PgInsights
+# 📊 PgInsights
 
-**PostgreSQL performance monitoring for Rails apps**
+> PostgreSQL performance monitoring for Rails apps
+
+<div align="center">
 
 [![Gem Version](https://badge.fury.io/rb/pg_insights.svg)](https://badge.fury.io/rb/pg_insights)
 [![CI](https://github.com/mezbahalam/pg_insights/actions/workflows/ci.yml/badge.svg)](https://github.com/mezbahalam/pg_insights/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
+</div>
+
+---
+
 PgInsights is a Rails engine that gives you a web dashboard for monitoring your PostgreSQL database performance. Think of it as a lightweight alternative to external monitoring tools that lives right inside your Rails app.
 
-## Why I built this
+## 🤔 Why I built this
 
-I got tired of switching between different tools to check database performance. Sometimes you just want to quickly see which indexes aren't being used, or find slow queries without setting up a whole monitoring infrastructure. PgInsights gives you that - a simple dashboard you can access at `/pg_insights` in your Rails app.
+I got tired of switching between different tools to check database performance. Sometimes you just want to quickly see which indexes aren't being used, or find slow queries without setting up a whole monitoring infrastructure. 
 
-## What you get
+PgInsights gives you that - a simple dashboard you can access at `/pg_insights` in your Rails app.
 
-**Health Dashboard**
+## 🎁 What you get
+
+<table>
+<tr>
+<td width="50%">
+
+### 🏥 Health Dashboard
 - Find unused indexes that are wasting space
 - Spot tables that might need indexes (high sequential scans)
 - Identify slow queries (if you have pg_stat_statements enabled)
 - Check for table bloat that needs cleanup
 - Review PostgreSQL configuration settings
 
-**Query Runner**
+</td>
+<td width="50%">
+
+### 🔍 Query Runner  
 - Run your own SELECT queries safely
 - Built-in queries for common performance checks
 - Save queries you use frequently
 - Results displayed as tables or charts
 
-**Smart execution**
+</td>
+</tr>
+<tr>
+<td colspan="2">
+
+### 📈 Timeline & Monitoring
+- **Database snapshots**: Automatic collection of performance metrics and configuration parameters
+- **Parameter change tracking**: Detect when PostgreSQL settings are modified over time
+- **Performance trends**: Monitor cache hit rates, query times, and connection counts
+- **Historical comparisons**: Compare database state between any two time periods
+- **Export capabilities**: Download timeline data as CSV or JSON for further analysis
+- **Configurable retention**: Automatic cleanup of old snapshots based on retention policy
+
+</td>
+</tr>
+</table>
+
+### ⚡️ Smart execution
 - Runs health checks in background jobs if you have them set up
 - Falls back to running directly if you don't
 - Caches results so repeated visits are fast
 - Configurable timeouts to prevent slow queries from hanging
 
-## Installation
+## 📸 Screenshots
+
+<details>
+<summary>👀 Click to see the interface</summary>
+
+### Health Dashboard
+Monitor your PostgreSQL database performance and identify potential issues at a glance.
+
+![Health Dashboard](health_dashboard.png)
+
+### Query Runner
+Run custom queries and visualize results with built-in charting capabilities.
+
+![Query Runner](query_runner.png)
+
+### Timeline Dashboard
+Track database configuration changes and performance trends over time.
+
+![Timeline Dashboard](timeline_dashboard.png)
+
+### Snapshot Details
+View detailed information about individual database snapshots including parameters and metadata.
+
+![Snapshot Details](snapshot_details.png)
+
+### Performance Metrics
+Monitor key performance indicators and database health metrics across time periods.
+
+![Performance Metrics](snapshot_performance.png)
+
+### Snapshot Comparison
+Compare database configurations and performance metrics between different time periods.
+
+![Snapshot Comparison](snapshot_comparison.png)
+
+### Side-by-Side Comparison
+Detailed comparison view showing parameter changes and performance differences.
+
+![Compare Snapshots](compare_snapshots.png)
+
+</details>
+
+## ⏩ Quick Start
 
 Add to your Gemfile:
 
@@ -51,45 +125,47 @@ rails db:migrate
 
 That's it. Visit `/pg_insights` in your browser.
 
-## Configuration
+## ⚙️ Configuration
 
 The engine works out of the box, but you can customize it:
 
 ```ruby
 # config/initializers/pg_insights.rb
 PgInsights.configure do |config|
-  # Run health checks in background (default: true)
+  # === Background Jobs ===
   config.enable_background_jobs = true
+  config.background_job_queue = :pg_insights_health
   
-  # How long to cache results (default: 5 minutes)
-  config.health_cache_expiry = 10.minutes
+  # === Health Check Settings ===
+  config.health_cache_expiry = 5.minutes
+  config.health_check_timeout = 10.seconds
   
-  # Timeout for health check queries (default: 10 seconds)
-  config.health_check_timeout = 15.seconds
-  
-  # Queue name for background jobs (default: :pg_insights_health)
-  config.background_job_queue = :low_priority
+  # === Timeline & Snapshot Settings ===
+  config.enable_snapshots = true                    # Enable timeline feature
+  config.snapshot_frequency = 1.day                 # How often to collect snapshots
+  config.snapshot_retention_days = 90               # How long to keep snapshots
+  config.snapshot_collection_enabled = true         # Master switch for snapshot collection
 end
 ```
 
-## How Background Jobs Work
+## 🔄 How Background Jobs Work
 
-**PgInsights uses on-demand background jobs, not automatic scheduling.**
+> **Note:** PgInsights uses on-demand background jobs, not automatic scheduling.
 
-### When health checks run:
-- ✅ **When you visit the health dashboard `/pg_insights/health`** and cached data is older than `health_cache_expiry` (default: 5 minutes)
-- ✅ **When you click the "Refresh" button** in the dashboard
-- ✅ **When you run** `rails pg_insights:health_check` manually
+**When health checks run:**
+- ✅ When you visit the health dashboard `/pg_insights/health` and cached data is older than `health_cache_expiry` (default: 5 minutes)
+- ✅ When you click the "Refresh" button in the dashboard  
+- ✅ When you run `rails pg_insights:health_check` manually
 - ❌ **NOT automatically** - PgInsights doesn't run background jobs on its own
 
-### How caching works:
+**How caching works:**
 ```
 Visit at 2:00 PM → Runs health checks, caches results for 5 minutes
-Visit at 2:03 PM → Uses cached results (still fresh)
+Visit at 2:03 PM → Uses cached results (still fresh)  
 Visit at 2:06 PM → Data is stale, triggers new background jobs
 ```
 
-### Background job setup (optional but recommended):
+**Background job setup (optional but recommended):**
 
 If your app has background jobs (Sidekiq, Resque, etc.), PgInsights will use them for better performance:
 
@@ -101,7 +177,7 @@ rails pg_insights:status
 **Without background jobs**: Health checks run synchronously when you visit the page (slower but works)  
 **With background jobs**: Health checks run asynchronously (faster, non-blocking)
 
-### Optional: Automatic recurring checks
+**Optional: Automatic recurring checks**
 
 If you want health checks to run automatically (not just on-demand), set up a scheduler:
 
@@ -121,51 +197,70 @@ Sidekiq::Cron::Job.create(
 
 **Note**: Even with automatic scheduling, the jobs are smart - they only run expensive queries if the cached data is actually stale.
 
-## Usage
+## 💻 Usage
 
 Navigate to `/pg_insights` in your app. The interface is pretty straightforward:
 
-- **Main page**: Run queries and see results as tables or charts
-- **Health tab**: Database performance overview
-- **Query examples**: Pre-built queries for common checks
+| Page | What it does |
+|------|-------------|
+| **Query Runner** | Run custom queries and see results as tables or charts |
+| **Health Dashboard** | Real-time database performance overview and issue detection |
+| **Timeline** | Historical tracking of database performance metrics and configuration changes |
 
 All queries are read-only (SELECT statements only) and have timeouts to prevent issues.
 
-## Available rake tasks
+## 🛠️ Available rake tasks
 
 ```bash
-rails pg_insights:status        # Check configuration
-rails pg_insights:health_check  # Run health checks manually
-rails pg_insights:stats         # Show usage statistics
-rails pg_insights:clear_data    # Clear stored data and caches
+# Configuration & Status
+rails pg_insights:status              # Check configuration and background job status
+rails pg_insights:test_jobs           # Test background job functionality
+
+# Health Checks
+rails pg_insights:health_check        # Run health checks manually (synchronous)
+rails pg_insights:stats              # Show usage statistics
+
+# Timeline & Snapshots
+rails pg_insights:collect_snapshot    # Collect a database snapshot immediately
+rails pg_insights:start_snapshots    # Start recurring snapshot collection
+rails pg_insights:snapshot_status    # Check snapshot configuration and status
+rails pg_insights:cleanup_snapshots  # Clean up old snapshots
+
+# Data Management
+rails pg_insights:reset              # Reset all PgInsights data (queries + health checks)
+rails pg_insights:clear_data         # Alias for reset (backward compatibility)
+rails pg_insights:cleanup            # Clean up old health check results (30+ days)
+
+# Development & Testing
+rails pg_insights:seed_timeline      # Generate fake timeline data for testing
+rails pg_insights:sample_data        # Generate sample health check data
 ```
 
-## Safety
+## 🔒 Safety
 
 - Only SELECT queries allowed
 - Query timeouts prevent long-running queries
 - Focuses on public schema by default
 - No modification of your data
 
-## Uninstalling
+## 🗑️ Uninstalling
 
 ```bash
 rails generate pg_insights:clean
-rails db:rollback STEP=2
 # Remove gem from Gemfile
 ```
 
-## Requirements
+## 📋 Requirements
 
 - Rails 6.1+
 - PostgreSQL
 - For slow query detection: pg_stat_statements extension (optional)
 
-## Contributing
+## 🤝 Contributing
 
 Found a bug or have an idea? Open an issue or send a pull request. The codebase is pretty straightforward.
 
-Development setup:
+**Development setup:**
 
 ```bash
 git clone https://github.com/mezbahalam/pg_insights.git
@@ -174,10 +269,14 @@ bundle install
 bundle exec rake spec
 ```
 
-## License
+## 📄 License
 
 MIT License. See [LICENSE](MIT-LICENSE) file.
 
 ---
 
-Built by [Mezbah Alam](https://github.com/mezbahalam). Inspired by pg_hero and other database monitoring tools.
+<div align="center">
+
+Built by [Mezbah Alam](https://github.com/mezbahalam) • Inspired by pg_hero and other database monitoring tools
+
+</div>

data/app/controllers/pg_insights/timeline_controller.rb

@@ -0,0 +1,263 @@
+# frozen_string_literal: true
+
+module PgInsights
+  class TimelineController < ApplicationController
+    layout "pg_insights/application"
+
+    def index
+      unless PgInsights.snapshots_available?
+        flash[:notice] = "Database snapshots are not enabled. Configure PgInsights.enable_snapshots = true to use timeline features."
+        redirect_to health_path
+        return
+      end
+
+      @snapshots = HealthCheckResult.snapshots(90)
+      @parameter_changes = HealthCheckResult.detect_parameter_changes_since(30)
+      @timeline_data = HealthCheckResult.timeline_data(30)
+      @stats = calculate_timeline_stats(@snapshots)
+    end
+
+    def show
+      @snapshot = HealthCheckResult.find(params[:id])
+
+      unless @snapshot.check_type == "database_snapshot"
+        flash[:error] = "Invalid snapshot"
+        redirect_to timeline_path
+        return
+      end
+
+      @previous_snapshot = HealthCheckResult.snapshots
+                                           .where("executed_at < ?", @snapshot.executed_at)
+                                           .first
+
+      if @previous_snapshot
+        @parameter_changes = HealthCheckResult.compare_snapshots(@previous_snapshot, @snapshot)
+        @performance_comparison = compare_performance_metrics(@previous_snapshot, @snapshot)
+      end
+    end
+
+    def compare
+      @date1 = Date.parse(params[:date1]) rescue nil
+      @date2 = Date.parse(params[:date2]) rescue nil
+
+      unless @date1 && @date2
+        flash[:error] = "Invalid dates provided"
+        redirect_to timeline_path
+        return
+      end
+
+      @snapshot1 = HealthCheckResult.find_snapshot_by_date(@date1)
+      @snapshot2 = HealthCheckResult.find_snapshot_by_date(@date2)
+
+      unless @snapshot1 && @snapshot2
+        flash[:error] = "Could not find snapshots for the selected dates"
+        redirect_to timeline_path
+        return
+      end
+
+                  @parameter_changes = HealthCheckResult.compare_snapshots(@snapshot1, @snapshot2)
+      performance_metrics = compare_performance_metrics(@snapshot1, @snapshot2)
+      @performance_comparison = {
+        metrics: performance_metrics.transform_values do |data|
+          data.merge(difference: data[:change])
+        end
+      }
+      @metadata_comparison = compare_metadata(@snapshot1, @snapshot2)
+      @configuration_comparison = @parameter_changes
+
+      @comparison_data = {
+        snapshot1: @snapshot1,
+        snapshot2: @snapshot2,
+        parameters: @parameter_changes,
+        performance: @performance_comparison,
+        metadata: @metadata_comparison
+      }
+    end
+
+    def export
+      format = params[:format]&.downcase || "csv"
+      days = params[:days]&.to_i || 30
+
+      snapshots = HealthCheckResult.snapshots(days + 30)
+
+      case format
+      when "csv"
+        send_data generate_csv_export(snapshots),
+                  filename: "pg_insights_timeline_#{Date.current}.csv",
+                  type: "text/csv",
+                  disposition: "attachment"
+      when "json"
+        send_data generate_json_export(snapshots),
+                  filename: "pg_insights_timeline_#{Date.current}.json",
+                  type: "application/json",
+                  disposition: "attachment"
+      else
+        flash[:error] = "Unsupported export format. Use 'csv' or 'json'."
+        redirect_to timeline_path
+      end
+    end
+
+    def refresh
+      if PgInsights.snapshots_available? && PgInsights.background_jobs_available?
+        if PgInsights::DatabaseSnapshotJob.perform_later
+          render json: { message: "Snapshot collection started" }
+        else
+          render json: { error: "Failed to start snapshot collection" }, status: 422
+        end
+      else
+        begin
+          HealthCheckService.execute_and_cache_check("database_snapshot")
+          render json: { message: "Snapshot collected successfully" }
+        rescue => e
+          render json: { error: "Snapshot collection failed: #{e.message}" }, status: 422
+        end
+      end
+    end
+
+    def status
+      snapshot_status = {
+        enabled: PgInsights.snapshots_available?,
+        frequency: PgInsights.snapshot_frequency,
+        retention_days: PgInsights.snapshot_retention_days,
+        latest_snapshot: HealthCheckResult.latest_snapshot&.executed_at,
+        total_snapshots: HealthCheckResult.snapshots.count,
+        configuration_valid: PgInsights::DatabaseSnapshotJob.validate_configuration
+      }
+
+      render json: snapshot_status
+    end
+
+    private
+
+    def calculate_timeline_stats(snapshots)
+      return {} if snapshots.empty?
+
+      latest = snapshots.first
+      oldest = snapshots.last
+
+      cache_hit_rate = latest.result_data.dig("metrics", "cache_hit_rate")
+      numeric_cache_hit_rate = cache_hit_rate ? cache_hit_rate.to_f : nil
+
+      {
+        total_snapshots: snapshots.count,
+        date_range: {
+          from: oldest.executed_at.to_date,
+          to: latest.executed_at.to_date
+        },
+        latest_cache_hit_rate: numeric_cache_hit_rate,
+        parameter_changes_count: @parameter_changes.sum { |change| change[:changes].count }
+      }
+    end
+
+    def compare_performance_metrics(snapshot1, snapshot2)
+      metrics1 = snapshot1.result_data["metrics"] || {}
+      metrics2 = snapshot2.result_data["metrics"] || {}
+
+      comparison = {}
+
+      %w[cache_hit_rate avg_query_time p95_query_time bloated_tables
+         total_connections active_connections high_seq_scan_tables].each do |metric|
+        val1 = metrics1[metric]
+        val2 = metrics2[metric]
+
+        if val1 && val2
+          num_val1 = val1.to_f
+          num_val2 = val2.to_f
+
+          change = num_val2 - num_val1
+          change_pct = num_val1 != 0 ? (change / num_val1) * 100 : 0
+
+          comparison[metric] = {
+            before: num_val1,
+            after: num_val2,
+            change: change.round(2),
+            change_percent: change_pct.round(2),
+            direction: change > 0 ? "increase" : (change < 0 ? "decrease" : "stable")
+          }
+        end
+      end
+
+      comparison
+    end
+
+    def compare_metadata(snapshot1, snapshot2)
+      meta1 = snapshot1.result_data["metadata"] || {}
+      meta2 = snapshot2.result_data["metadata"] || {}
+
+      {
+        postgres_version_changed: meta1["postgresql_version"] != meta2["postgresql_version"],
+        extensions_added: (meta2["extensions"] || []) - (meta1["extensions"] || []),
+        extensions_removed: (meta1["extensions"] || []) - (meta2["extensions"] || []),
+        database_size: {
+          before: meta1["database_size"],
+          after: meta2["database_size"]
+        },
+        table_count: {
+          before: meta1["table_count"],
+          after: meta2["table_count"]
+        },
+        index_count: {
+          before: meta1["index_count"],
+          after: meta2["index_count"]
+        }
+      }
+    end
+
+    def generate_csv_export(snapshots)
+      require "csv"
+
+      CSV.generate(headers: true) do |csv|
+        csv << [
+          "Date", "Time", "Cache Hit Rate %", "Avg Query Time (ms)", "P95 Query Time (ms)",
+          "Bloated Tables", "Total Connections", "Active Connections", "High Seq Scan Tables",
+          "Database Size", "PostgreSQL Version"
+        ]
+
+        snapshots.each do |snapshot|
+          metrics = snapshot.result_data["metrics"] || {}
+          metadata = snapshot.result_data["metadata"] || {}
+
+          csv << [
+            snapshot.executed_at.strftime("%Y-%m-%d"),
+            snapshot.executed_at.strftime("%H:%M:%S"),
+            metrics["cache_hit_rate"]&.to_f,
+            metrics["avg_query_time"]&.to_f,
+            metrics["p95_query_time"]&.to_f,
+            metrics["bloated_tables"]&.to_i,
+            metrics["total_connections"]&.to_i,
+            metrics["active_connections"]&.to_i,
+            metrics["high_seq_scan_tables"]&.to_i,
+            metadata["database_size"],
+            metadata["postgres_version"]&.split(" ")&.first
+          ]
+        end
+      end
+    end
+
+    def generate_json_export(snapshots)
+      {
+        exported_at: Time.current.iso8601,
+        export_info: {
+          snapshot_count: snapshots.size,
+          date_range: {
+            from: snapshots.last&.executed_at,
+            to: snapshots.first&.executed_at
+          },
+          frequency: PgInsights.snapshot_frequency.to_s,
+          retention_days: PgInsights.snapshot_retention_days
+        },
+        parameter_changes: HealthCheckResult.detect_parameter_changes_since(30),
+        snapshots: snapshots.map do |snapshot|
+          {
+            id: snapshot.id,
+            collected_at: snapshot.executed_at.iso8601,
+            execution_time_ms: snapshot.execution_time_ms,
+            parameters: snapshot.result_data["parameters"],
+            metrics: snapshot.result_data["metrics"],
+            metadata: snapshot.result_data["metadata"]
+          }
+        end
+      }.to_json(indent: 2)
+    end
+  end
+end

data/app/jobs/pg_insights/database_snapshot_job.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+module PgInsights
+  class DatabaseSnapshotJob < ApplicationJob
+    queue_as -> { PgInsights.background_job_queue }
+
+    rescue_from(StandardError) do |exception|
+      Rails.logger.error "PgInsights::DatabaseSnapshotJob failed: #{exception.message}"
+    end
+
+    def perform
+      unless PgInsights.snapshots_available?
+        Rails.logger.warn "PgInsights: Snapshots not available, skipping snapshot collection"
+        return
+      end
+
+      Rails.logger.info "PgInsights: Starting database snapshot collection"
+
+      begin
+        # Use existing infrastructure to collect and store snapshot
+        HealthCheckService.execute_and_cache_check("database_snapshot")
+
+        # Cleanup old snapshots after successful collection
+        cleanup_count = HealthCheckResult.cleanup_old_snapshots
+
+        Rails.logger.info "PgInsights: Database snapshot completed successfully"
+        Rails.logger.info "PgInsights: Cleaned up #{cleanup_count} old snapshots" if cleanup_count > 0
+      rescue => e
+        Rails.logger.error "PgInsights: Database snapshot collection failed: #{e.message}"
+        raise e
+      end
+    end
+
+    def self.schedule_next_snapshot
+      return false unless PgInsights.snapshots_available?
+      return false unless PgInsights.background_jobs_available?
+
+      begin
+        # Schedule the next snapshot based on configured frequency
+        next_run_time = Time.current + PgInsights.snapshot_frequency
+        set(wait_until: next_run_time).perform_later
+
+        Rails.logger.info "PgInsights: Next snapshot scheduled for #{next_run_time}"
+        true
+      rescue => e
+        Rails.logger.warn "PgInsights: Failed to schedule next snapshot: #{e.message}"
+        false
+      end
+    end
+
+    def self.start_recurring_snapshots
+      return false unless PgInsights.snapshots_available?
+      return false unless PgInsights.background_jobs_available?
+
+      begin
+        # Start the recurring snapshot cycle
+        perform_later
+        Rails.logger.info "PgInsights: Recurring snapshots started with frequency: #{PgInsights.snapshot_frequency}"
+        true
+      rescue => e
+        Rails.logger.warn "PgInsights: Failed to start recurring snapshots: #{e.message}"
+        false
+      end
+    end
+
+    def self.validate_configuration
+      issues = []
+
+      issues << "Snapshots are disabled" unless PgInsights.enable_snapshots
+      issues << "Snapshot collection is disabled" unless PgInsights.snapshot_collection_enabled
+      issues << "Background jobs not available" unless PgInsights.background_jobs_available?
+      issues << "HealthCheckResult model not available" unless defined?(HealthCheckResult)
+
+      frequency = PgInsights.snapshot_frequency
+      if frequency.respond_to?(:to_i) && frequency.to_i < 60
+        issues << "Snapshot frequency too low (minimum 1 minute recommended)"
+      end
+
+      if issues.empty?
+        { valid: true, message: "Database snapshots are properly configured" }
+      else
+        { valid: false, issues: issues }
+      end
+    end
+
+    private
+
+    # Override perform to automatically schedule the next run
+    def perform_with_scheduling
+      perform
+
+      # Schedule the next snapshot if this one was successful
+      self.class.schedule_next_snapshot if PgInsights.snapshots_available?
+    rescue => e
+      # Log the error but still try to schedule the next run
+      Rails.logger.error "PgInsights: Snapshot failed but will retry: #{e.message}"
+      self.class.schedule_next_snapshot if PgInsights.snapshots_available?
+      raise e
+    end
+  end
+end