sql_genius 0.9.0

data/docs/guides/ssh-tunnel-connections.md

@@ -0,0 +1,151 @@
+# Connecting to MySQL Through an SSH Tunnel
+
+If your MySQL server is behind a firewall, on a private network, or only accessible via a bastion/jump host, you can use an SSH tunnel to connect SqlGenius to it.
+
+## How it works
+
+An SSH tunnel forwards a local port on your machine to the MySQL port on the remote server. SqlGenius connects to `localhost:<local_port>` and the tunnel transparently routes traffic to the actual database server.
+
+```
+Your Machine (SqlGenius)  →  SSH Tunnel  →  Bastion Host  →  MySQL Server
+localhost:3307                                                   db.internal:3306
+```
+
+## Step 1: Open the SSH tunnel
+
+In a terminal, run:
+
+```bash
+ssh -L 3307:db.internal:3306 user@bastion-host.example.com -N
+```
+
+**Flags explained:**
+- `-L 3307:db.internal:3306` — forward local port `3307` to `db.internal:3306` through the tunnel
+- `user@bastion-host.example.com` — your SSH login on the bastion/jump host
+- `-N` — don't open a shell, just forward the port
+
+**With an SSH key:**
+
+```bash
+ssh -L 3307:db.internal:3306 user@bastion-host.example.com -N -i ~/.ssh/my_key
+```
+
+**Keep it running in the background:**
+
+```bash
+ssh -L 3307:db.internal:3306 user@bastion-host.example.com -N -f
+```
+
+The `-f` flag sends SSH to the background after connecting.
+
+## Step 2: Configure your Rails app
+
+SqlGenius uses your app's `ActiveRecord::Base.connection`, which reads from `database.yml`. Point it at the tunnel:
+
+```yaml
+# config/database.yml
+production:
+  adapter: mysql2
+  host: 127.0.0.1
+  port: 3307
+  username: readonly
+  password: <%= ENV["DB_PASSWORD"] %>
+  database: app_production
+```
+
+No special SqlGenius configuration needed — it automatically uses the same connection as your Rails app.
+
+## Step 3: Verify the connection
+
+Start your Rails server and visit `/sql_genius`. If the tunnel is running, the dashboard loads normally. If not, you'll see a connection error.
+
+## Common issues
+
+### "Connection refused"
+
+The SSH tunnel is not running. Start it first:
+
+```bash
+ssh -L 3307:db.internal:3306 user@bastion -N
+```
+
+### "Access denied"
+
+The tunnel is working but the MySQL credentials are wrong. Verify your username/password can connect to the database directly from the bastion host.
+
+### "Lost connection to MySQL server during query"
+
+The SSH tunnel dropped. This happens if the tunnel is idle for too long. Add keep-alive settings:
+
+```bash
+ssh -L 3307:db.internal:3306 user@bastion -N -o ServerAliveInterval=60 -o ServerAliveCountMax=3
+```
+
+### Port already in use
+
+Another process is using port 3307. Pick a different local port:
+
+```bash
+ssh -L 3308:db.internal:3306 user@bastion -N
+```
+
+Then update your SqlGenius config to use port `3308`.
+
+## Multiple databases through one bastion
+
+You can tunnel to multiple MySQL servers through the same bastion:
+
+```bash
+# Production on local port 3307
+ssh -L 3307:db-prod.internal:3306 user@bastion -N -f
+
+# Staging on local port 3308
+ssh -L 3308:db-staging.internal:3306 user@bastion -N -f
+```
+
+Then create separate SqlGenius profiles for each:
+- **Production**: `127.0.0.1:3307`
+- **Staging**: `127.0.0.1:3308`
+
+## Automating the tunnel
+
+### macOS: Launch Agent
+
+Create `~/Library/LaunchAgents/com.sqlgenius.tunnel.plist`:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>Label</key>
+    <string>com.sqlgenius.tunnel</string>
+    <key>ProgramArguments</key>
+    <array>
+        <string>ssh</string>
+        <string>-L</string>
+        <string>3307:db.internal:3306</string>
+        <string>user@bastion</string>
+        <string>-N</string>
+        <string>-o</string>
+        <string>ServerAliveInterval=60</string>
+    </array>
+    <key>KeepAlive</key>
+    <true/>
+    <key>RunAtLoad</key>
+    <true/>
+</dict>
+</plist>
+```
+
+Load it:
+
+```bash
+launchctl load ~/Library/LaunchAgents/com.sqlgenius.tunnel.plist
+```
+
+The tunnel will start automatically on login and restart if it drops.
+
+## Future: Built-in SSH tunnel support
+
+Built-in SSH tunnel support is planned for a future release. When available, you'll be able to configure the SSH connection directly in the SqlGenius profile form without needing a separate terminal.

data/lib/generators/sql_genius/install/install_generator.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module SqlGenius
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Creates a SqlGenius initializer and mounts the engine in routes."
+
+      def copy_initializer
+        template("initializer.rb", "config/initializers/sql_genius.rb")
+      end
+
+      def mount_engine
+        route('mount SqlGenius::Engine, at: "/sql_genius"')
+      end
+    end
+  end
+end

data/lib/generators/sql_genius/install/templates/initializer.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+SqlGenius.configure do |config|
+  # --- Authentication ---
+  # Lambda that receives the controller instance. Return true to allow access.
+  # Default: allows everyone. Use route constraints for most cases.
+  # config.authenticate = ->(controller) { controller.current_user&.admin? }
+
+  # To use current_user or other app helpers, inherit from ApplicationController:
+  # config.base_controller = "ApplicationController"
+
+  # --- Tables ---
+  # Tables featured at the top of the visual builder dropdown (optional).
+  # config.featured_tables = %w[users posts comments]
+
+  # Tables blocked from querying (defaults: sessions, schema_migrations, ar_internal_metadata).
+  # config.blocked_tables += %w[oauth_tokens api_keys]
+
+  # Column patterns to redact with [REDACTED] in results (case-insensitive substring match).
+  # config.masked_column_patterns = %w[password secret digest token ssn]
+
+  # Default columns checked in the visual builder per table (optional).
+  # config.default_columns = {
+  #   "users" => %w[id name email created_at],
+  #   "posts" => %w[id title user_id published_at]
+  # }
+
+  # --- Query Safety ---
+  # config.max_row_limit = 1000       # Hard cap on rows returned
+  # config.default_row_limit = 25     # Default when no limit specified
+  # config.query_timeout_ms = 30_000  # 30 second timeout
+
+  # --- Slow Query Monitoring ---
+  # Requires Redis. Set to nil to disable.
+  # config.redis_url = ENV["REDIS_URL"]
+  # config.slow_query_threshold_ms = 250
+
+  # --- Audit Logging ---
+  # Set to nil to disable. Logs query executions, rejections, and errors.
+  # config.audit_logger = Logger.new(Rails.root.join("log", "sql_genius.log"))
+
+  # --- AI Features (optional) ---
+  # Supports any OpenAI-compatible API: OpenAI, Azure OpenAI, Ollama, or a custom client.
+  # config.ai_endpoint = "https://api.openai.com/v1/chat/completions"
+  # config.ai_api_key = ENV["OPENAI_API_KEY"]
+  # config.ai_model = "gpt-4o"
+  # config.ai_auth_style = :bearer  # :bearer for OpenAI/Ollama, :api_key for Azure
+
+  # Domain context helps the AI understand your schema and generate better queries.
+  # config.ai_system_context = <<~CONTEXT
+  #   This is an e-commerce database.
+  #   - `users` stores customer accounts.
+  #   - `orders` tracks purchases, linked to users via `user_id`.
+  #   - Soft-deleted records have `deleted_at IS NOT NULL`.
+  # CONTEXT
+end

data/lib/sql_genius/configuration.rb

@@ -0,0 +1,114 @@
+# frozen_string_literal: true
+
+module SqlGenius
+  class Configuration
+    # Tables to feature in the visual builder dropdown (array of strings).
+    # When empty, all non-blocked tables are shown.
+    attr_accessor :featured_tables
+
+    # Tables that must never be queried (auth, sessions, internal Rails tables).
+    attr_accessor :blocked_tables
+
+    # Column name patterns to mask with [REDACTED] in query results.
+    # Matched case-insensitively via String#include?.
+    attr_accessor :masked_column_patterns
+
+    # Default columns to check in the visual builder, keyed by table name.
+    # Example: { "users" => %w[id name email created_at] }
+    attr_accessor :default_columns
+
+    # Maximum rows a single query can return.
+    attr_accessor :max_row_limit
+
+    # Default row limit when none is specified.
+    attr_accessor :default_row_limit
+
+    # Query timeout in milliseconds.
+    attr_accessor :query_timeout_ms
+
+    # Proc that receives the controller instance and returns true if the user
+    # is authorized. Example:
+    #   config.authenticate = ->(controller) { controller.current_user&.admin? }
+    attr_accessor :authenticate
+
+    # AI configuration — set to nil to disable AI features entirely.
+    # Must respond to :call(messages:, response_format:, temperature:)
+    # and return a Hash with "choices" in OpenAI-compatible format,
+    # OR set ai_endpoint + ai_api_key for a direct OpenAI-compatible HTTP API.
+    attr_accessor :ai_client
+    attr_accessor :ai_endpoint
+    attr_accessor :ai_api_key
+
+    # AI model name to pass in the request body (e.g. "gpt-4o", "gpt-3.5-turbo").
+    # Optional — if nil, the API default or deployment model is used.
+    attr_accessor :ai_model
+
+    # AI auth style: :bearer (OpenAI, Ollama Cloud) or :api_key (Azure OpenAI).
+    # Defaults to :api_key for backwards compatibility.
+    attr_accessor :ai_auth_style
+
+    # Custom system prompt prepended to AI suggestions. Use this to describe
+    # your domain, table relationships, and naming conventions.
+    attr_accessor :ai_system_context
+
+    # Slow query threshold in milliseconds. Queries slower than this are logged.
+    attr_accessor :slow_query_threshold_ms
+
+    # Redis URL for slow query storage. Set to nil to disable slow query monitoring.
+    attr_accessor :redis_url
+
+    # Logger instance for audit logging. Defaults to a file logger.
+    # Set to nil to disable audit logging.
+    attr_accessor :audit_logger
+
+    # Base controller class for the engine to inherit from.
+    # Set to "ApplicationController" to get current_user and other app helpers.
+    # Defaults to "ActionController::Base".
+    attr_accessor :base_controller
+
+    # Whether to start the background stats collector on boot.
+    # When enabled, performance_schema is sampled periodically and stored
+    # in an in-memory ring buffer accessible via SqlGenius.stats_history.
+    # Defaults to true.
+    attr_accessor :stats_collection
+
+    # Maximum scan count for an index to still be considered "unused" by the
+    # Unused Indexes dashboard. The default (0) means only indexes that have
+    # never been scanned since the stats source was last reset are flagged.
+    # Raise this to ignore indexes that are technically used but rarely
+    # enough to be worth dropping (e.g. min_unused_index_scans = 50 to require
+    # at least 50 scans before considering an index "useful").
+    attr_accessor :min_unused_index_scans
+
+    def initialize
+      @featured_tables = []
+      @blocked_tables = [
+        "sessions",
+        "ar_internal_metadata",
+        "schema_migrations",
+      ]
+      @masked_column_patterns = ["password", "secret", "digest", "token"]
+      @default_columns = {}
+      @max_row_limit = 1000
+      @default_row_limit = 25
+      @query_timeout_ms = 30_000
+      @authenticate = ->(_controller) { true }
+      @ai_client = nil
+      @ai_endpoint = nil
+      @ai_api_key = nil
+      @ai_model = nil
+      @ai_auth_style = :api_key
+      @ai_system_context = nil
+      @slow_query_threshold_ms = 250
+      @redis_url = nil
+      @audit_logger = nil
+      @base_controller = "ActionController::Base"
+      @stats_collection = true
+      @min_unused_index_scans = 0
+    end
+
+    def ai_enabled?
+      !ai_client.nil? || (!ai_endpoint.nil? && !ai_endpoint.empty? && !ai_api_key.nil? && !ai_api_key.empty?)
+    end
+  end
+end

data/lib/sql_genius/core/ai/client.rb

@@ -0,0 +1,155 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "json"
+require "uri"
+
+module SqlGenius
+  module Core
+    module Ai
+      # HTTP client for OpenAI-compatible chat completion APIs.
+      # Construct with a Core::Ai::Config; call #chat with a messages array.
+      class Client
+        class NotConfigured < Core::Error; end
+        class ApiError < Core::Error; end
+        class TooManyRedirects < Core::Error; end
+
+        MAX_REDIRECTS = 3
+
+        def initialize(config)
+          @config = config
+        end
+
+        def chat(messages:, temperature: 0)
+          if @config.client
+            return @config.client.call(messages: messages, temperature: temperature)
+          end
+
+          raise NotConfigured, "AI is not configured" unless @config.enabled?
+
+          body = if anthropic?
+            build_anthropic_body(messages, temperature)
+          else
+            build_openai_body(messages, temperature)
+          end
+
+          response = post_with_redirects(URI(@config.endpoint), body.to_json)
+          parsed = JSON.parse(response.body)
+
+          if parsed["error"]
+            raise ApiError, "AI API error: #{parsed["error"]["message"] || parsed["error"]}"
+          end
+
+          content = if anthropic?
+            parsed.dig("content", 0, "text")
+          else
+            parsed.dig("choices", 0, "message", "content")
+          end
+          raise ApiError, "No content in AI response" if content.nil?
+
+          parse_json_content(content)
+        end
+
+        private
+
+        def anthropic?
+          @config.auth_style == :x_api_key
+        end
+
+        def build_openai_body(messages, temperature)
+          body = {
+            messages: messages,
+            response_format: { type: "json_object" },
+          }
+          # GPT-5 and o-series ("reasoning") models renamed max_tokens to
+          # max_completion_tokens and only accept temperature=1 (the default).
+          # Older chat models (gpt-4o, gpt-4, gpt-3.5) keep the original names.
+          if openai_reasoning_model?
+            body[:max_completion_tokens] = @config.max_tokens.to_i if @config.max_tokens
+          else
+            body[:max_tokens] = @config.max_tokens.to_i if @config.max_tokens
+            body[:temperature] = temperature
+          end
+          body[:model] = @config.model if @config.model && !@config.model.empty?
+          body
+        end
+
+        # Returns true when the configured model belongs to the OpenAI families
+        # that reject `max_tokens` and `temperature` overrides: gpt-5*, o1*,
+        # o3*, o4*. Matches the bare model name plus common deployment-name
+        # prefixes (Azure deployments are user-named but typically include the
+        # model identifier).
+        OPENAI_REASONING_MODEL_PATTERN = /\b(gpt-5|o1|o3|o4)(-|\b)/i.freeze
+        def openai_reasoning_model?
+          model = @config.model.to_s
+          return false if model.empty?
+
+          model.match?(OPENAI_REASONING_MODEL_PATTERN)
+        end
+
+        def build_anthropic_body(messages, temperature)
+          system_text = messages.select { |m| m[:role] == "system" }.map { |m| m[:content] }.join("\n\n")
+          user_messages = messages.reject { |m| m[:role] == "system" }
+
+          body = {
+            messages: user_messages,
+            max_tokens: (@config.max_tokens || 4096).to_i,
+            temperature: temperature,
+          }
+          body[:system] = system_text unless system_text.empty?
+          body[:model] = @config.model if @config.model && !@config.model.empty?
+          body
+        end
+
+        def parse_json_content(content)
+          JSON.parse(content)
+        rescue JSON::ParserError
+          stripped = content.to_s
+            .gsub(/\A\s*```(?:json)?\s*/i, "")
+            .gsub(/\s*```\s*\z/, "")
+            .strip
+          begin
+            JSON.parse(stripped)
+          rescue JSON::ParserError
+            { "raw" => content.to_s }
+          end
+        end
+
+        def post_with_redirects(uri, body, redirects = 0)
+          raise TooManyRedirects, "Too many redirects" if redirects > MAX_REDIRECTS
+
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = uri.scheme == "https"
+          if http.use_ssl?
+            http.verify_mode = OpenSSL::SSL::VERIFY_PEER
+            cert_file = ENV["SSL_CERT_FILE"] || OpenSSL::X509::DEFAULT_CERT_FILE
+            http.ca_file = cert_file if File.exist?(cert_file)
+          end
+          http.open_timeout = 10
+          http.read_timeout = 60
+
+          request = Net::HTTP::Post.new(uri)
+          request["Content-Type"] = "application/json"
+          case @config.auth_style
+          when :bearer
+            request["Authorization"] = "Bearer #{@config.api_key}"
+          when :x_api_key
+            request["x-api-key"] = @config.api_key
+            request["anthropic-version"] = "2023-06-01"
+          else
+            request["api-key"] = @config.api_key
+          end
+          request.body = body
+
+          response = http.request(request)
+
+          if response.is_a?(Net::HTTPRedirection)
+            post_with_redirects(URI(response["location"]), body, redirects + 1)
+          else
+            response
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sql_genius/core/ai/config.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module SqlGenius
+  module Core
+    module Ai
+      # Keyword-init value object holding all the AI settings a Client
+      # needs. Passed explicitly to every AI service constructor — no
+      # module-level globals.
+      #
+      # Fields:
+      #   client         - optional callable; when set, bypasses HTTP.
+      #                    Signature: #call(messages:, temperature:) -> Hash
+      #   endpoint       - HTTPS URL of the chat completions endpoint
+      #   api_key        - API key (used as Bearer or api-key header)
+      #   model          - model name passed in the request body
+      #   auth_style     - :bearer or :api_key
+      #   system_context - optional domain context string that services
+      #                    append to their system prompts
+      #   domain_context - optional host-app context string interpolated into
+      #                    AI system prompts (e.g. "Rails app, no FKs")
+      Config = Struct.new(
+        :client,
+        :endpoint,
+        :api_key,
+        :model,
+        :auth_style,
+        :system_context,
+        :domain_context,
+        :max_tokens,
+        keyword_init: true,
+      ) do
+        def initialize(**kwargs)
+          super(domain_context: "", max_tokens: 4096, **kwargs)
+          freeze
+        end
+
+        def enabled?
+          return true if client
+          return false if endpoint.nil? || endpoint.to_s.empty?
+          return false if api_key.nil? || api_key.to_s.empty?
+
+          true
+        end
+      end
+    end
+  end
+end

data/lib/sql_genius/core/ai/connection_advisor.rb

@@ -0,0 +1,96 @@
+# frozen_string_literal: true
+
+module SqlGenius
+  module Core
+    module Ai
+      # Diagnoses connection pool health by gathering connection-related
+      # metrics from SHOW GLOBAL STATUS and SHOW GLOBAL VARIABLES, then
+      # asking the LLM to distinguish between pool misconfiguration,
+      # connection leaks, missing pooling, and traffic saturation.
+      class ConnectionAdvisor
+        STATUS_KEYS = [
+          "Threads_connected",
+          "Threads_running",
+          "Max_used_connections",
+          "Aborted_connects",
+          "Aborted_clients",
+          "Connections",
+          "Threads_created",
+        ].freeze
+
+        VARIABLE_KEYS = [
+          "max_connections",
+          "wait_timeout",
+          "interactive_timeout",
+          "thread_cache_size",
+        ].freeze
+
+        def initialize(client, config, connection)
+          @client = client
+          @config = config
+          @connection = connection
+        end
+
+        def call
+          if @connection.server_version.postgresql?
+            raise Core::UnsupportedDialect.for_postgresql("Connection Pressure Advisor")
+          end
+
+          variables = fetch_variables
+          status = fetch_status
+
+          messages = [
+            { role: "system", content: system_prompt },
+            { role: "user",   content: user_prompt(variables, status) },
+          ]
+          @client.chat(messages: messages)
+        end
+
+        private
+
+        def fetch_variables
+          result = @connection.exec_query("SHOW GLOBAL VARIABLES")
+          result.rows
+            .select { |row| VARIABLE_KEYS.include?(row[0]) }
+            .map { |row| [row[0], row[1]] }
+        end
+
+        def fetch_status
+          result = @connection.exec_query("SHOW GLOBAL STATUS")
+          result.rows
+            .select { |row| STATUS_KEYS.include?(row[0]) }
+            .map { |row| [row[0], row[1]] }
+        end
+
+        def system_prompt
+          <<~PROMPT
+            You are a MySQL connection health advisor. Analyze the connection-related variables and status counters below, then diagnose the connection health and provide specific recommendations. Consider:
+            - Connection utilization: Max_used_connections vs max_connections (target: stay below 80%)
+            - Aborted connections: Aborted_connects indicates authentication failures or client errors; Aborted_clients indicates clients disconnecting without proper cleanup
+            - Thread cache efficiency: Threads_created vs Connections (high ratio means thread_cache_size is too small)
+            - Timeout configuration: wait_timeout and interactive_timeout impact how long idle connections persist
+            - Connection leak indicators: high Threads_connected with low Threads_running suggests idle connection accumulation
+            - Traffic saturation: high Threads_running relative to CPU cores suggests query contention
+
+            Distinguish between these root causes:
+            1. Pool misconfiguration (max_connections too low/high, bad timeout values)
+            2. Connection leaks (growing Threads_connected, high Aborted_clients)
+            3. Missing connection pooling (high Connections with short-lived threads)
+            4. Traffic saturation (high Threads_running, query contention)
+            #{@config.domain_context}
+            Respond with JSON: {"diagnosis": "markdown analysis distinguishing between pool misconfiguration, connection leaks, missing pooling, and traffic saturation, with specific variable recommendations and values"}
+          PROMPT
+        end
+
+        def user_prompt(variables, status)
+          lines = ["== Connection Variables =="]
+          variables.each { |name, value| lines << "#{name} = #{value}" }
+          lines << ""
+          lines << "== Connection Status Counters =="
+          status.each { |name, value| lines << "#{name} = #{value}" }
+          lines.join("\n")
+        end
+      end
+    end
+  end
+end