ask-ruby-harness 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: bd5a2b1d1916502e5eb5cc92220a4d06149817a05f2bf8ed00e8327630272881
+  data.tar.gz: 847841b9d7fd758f8fb2c3da5d19e8b51414f3e3f2ee94c8e398e3231abe3b6a
+SHA512:
+  metadata.gz: 604554da7aa7e84bd2d779cd68123645757d185000d62575694e51201ef1086e5419e48d24c70679546098a4352bf9c686184ff22725fec22d0995aafa6e02f2
+  data.tar.gz: 9f4bdf2110e1a2dd490cf59c0d0af545726bec839dc64f59e7cdfe8e6438170750ebe6e2dcc1530cedf51586001b8d7a686897b2c46368eefd60c2a3a25a010f

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+## [0.1.0] — 2026-08-10
+
+### Added
+
+- **Generic harness for any Ruby project** — extracted from
+  `ask-rails-harness`: `RunCommand`, `ReadLog`, `QueryDatabase`, `ReadModel`,
+  `SchemaGraph`, and `RunTests` with a language-agnostic tool contract.
+- **No Rails dependency** — the gem loads in plain Ruby projects; database
+  access connects standalone via `ASK_DATABASE_URL` or `config/database.yml`.
+- **Runner detection for `run_tests`** — `bin/rails test` (Rails apps),
+  `bundle exec rspec` (rspec projects), or `bundle exec rake test` (plain
+  Ruby projects), all with the same structured JSON results.
+- **Minitest JSON reporter + plugin** — `minitest/ask_ruby_harness_plugin.rb`
+  (minitest 5 auto-discovery, minitest 6 RUBYOPT injection), inert unless
+  `ASK_TEST_JSON_PATH` is set.
+- **Environment permissions** — per-environment modes and command
+  allow/deny lists.
+- **Audit logging** — `ask_audit_logs` table (when available) +
+  `audit_log.ask_ruby_harness` notification, with sensitive-param redaction.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Kaka Ruto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,67 @@
+# Ask Ruby Harness
+
+Admin AI copilot for any Ruby project — structured, safe, permission-gated
+access for coding agents. The language edition of the ask-rb harness family
+(`ask-rails-harness` builds on this gem for Rails apps).
+
+## What it gives agents
+
+| Tool | What it does |
+|---|---|
+| `QueryDatabase` | Read-only SQL (non-SELECT rejected everywhere; SELECT-only in production) |
+| `ReadModel` | Inspect an ActiveRecord model's columns, associations, validations |
+| `ReadLog` | Read log files with level/search filtering and rotation support |
+| `RunCommand` | Run shell commands in the project root, gated by permission rules |
+| `SchemaGraph` | Full schema introspection: models, tables, columns, associations |
+| `RunTests` | Structured test results with failure reruns (rails test / rspec / rake test) |
+
+Generic file and search capabilities (read, grep, edit) are provided by the
+agent's native tools; the harness focuses on what only an app-aware layer
+can give an agent: database access, model introspection, logs, commands, and
+tests — all structured, permission-gated, and audited.
+
+## How it works
+
+- **No Rails required.** The harness loads in any Ruby project. Database
+  access connects standalone via `ASK_DATABASE_URL` or `config/database.yml`
+  when the host hasn't already connected.
+- **Structured returns, never terminal dumps.** Tools return data an agent
+  can act on directly — `run_tests` reports counts plus per-test
+  file/line/message.
+- **Minitest JSON reporter.** `lib/minitest/ask_ruby_harness_plugin.rb` is
+  auto-discovered by minitest 5 and injected via `RUBYOPT` on minitest 6
+  (which dropped plugin auto-discovery). Only active when the harness starts
+  the run (`ASK_TEST_JSON_PATH` set); ordinary test runs are untouched.
+- **Audit logging.** Every tool call is recorded in `ask_audit_logs` (when
+  the table exists) and broadcast as the `audit_log.ask_ruby_harness`
+  ActiveSupport notification. Sensitive params are redacted.
+- **Environment permissions.** Per-environment `mode` (full access, read
+  only, ask before changes) and command allow/deny lists.
+
+## Usage
+
+```ruby
+Ask::Ruby::Harness.configure do |config|
+  config.environment :production do |env|
+    env.mode = :read_only
+    env.denied_commands = [/rm/, /dropdb/]
+  end
+end
+
+Ask::Ruby::Harness.discover_tools!
+session = Ask::Ruby::Harness.agent_session(model: "gpt-4o")
+```
+
+The Rails edition (`ask-rails-harness`) mounts an agent at `/ask` and adds
+framework-native tools (routes, engine) on top of this gem.
+
+## Development
+
+```
+bundle install
+bundle exec rake test
+```
+
+## License
+
+MIT — see LICENSE.

data/lib/ask/ruby/harness/audit_log.rb

@@ -0,0 +1,187 @@
+# frozen_string_literal: true
+
+require "json"
+require "active_support/core_ext/string/filters"
+
+module Ask
+  module Ruby
+    module Harness
+      # Append-only audit log for tool executions.
+      #
+      # Every tool call made by an agent is recorded in the +ask_audit_logs+
+      # table (when one is available) with the intent (sanitized params) and
+      # outcome (status, timing), but not the data returned. A
+      # +audit_log.ask_ruby_harness+ ActiveSupport notification fires either
+      # way, so hosts can subscribe without the table.
+      #
+      # Sensitive param values (keys matching +password+, +secret+, +token+,
+      # +api_key+, +key+) are automatically redacted before logging.
+      module AuditLog
+        SENSITIVE_KEYS = /\A(password|secret|token|api_key|key|auth_token|access_token)\z/i
+
+        class << self
+          # Log a tool execution event.
+          #
+          # @param session_id [String] The agent session that triggered this call
+          # @param tool_name [String] Name of the tool that ran
+          # @param params [Hash] The parameters passed to the tool (sanitized automatically)
+          # @param result [Ask::Result, Hash, nil] The result returned by the tool
+          # @param error [StandardError, nil] The exception if the tool raised
+          # @param duration_ms [Integer] Wall-clock time for the tool execution
+          # @param user_context [Hash, nil] Who initiated the session (from config)
+          def log(session_id:, tool_name:, params:, result: nil, error: nil, duration_ms:)
+            now = Time.now.utc
+            entry = {
+              session_id: session_id,
+              tool_name: tool_name,
+              params: sanitize_params(params),
+              result_summary: build_summary(tool_name, result, error),
+              status: determine_status(result, error),
+              error_message: determine_error(result, error),
+              duration_ms: duration_ms,
+              user_context: resolve_user_context,
+              environment: environment_name,
+              recorded_at: now,
+              created_at: now,
+              updated_at: now
+            }
+
+            if table_exists?
+              write_entry(entry)
+            end
+
+            # Fire an ActiveSupport notification so host apps can subscribe
+            ActiveSupport::Notifications.instrument("audit_log.ask_ruby_harness", entry)
+
+            entry
+          end
+
+          private
+
+          def sanitize_params(params)
+            return {} unless params.is_a?(Hash)
+
+            params.each_with_object({}) do |(key, value), sanitized|
+              if SENSITIVE_KEYS.match?(key.to_s)
+                sanitized[key] = "[REDACTED]"
+              else
+                sanitized[key] = value
+              end
+            end
+          end
+
+          def determine_status(result, error)
+            return "error" if error
+            return "rejected" if result.is_a?(Ask::Result) && (result.error? || result.blocked?)
+            "success"
+          end
+
+          def determine_error(result, error)
+            return error.message if error
+            if result.is_a?(Ask::Result)
+              return result.error.to_s if result.error?
+              return result.content.to_s if result.blocked?
+            end
+            nil
+          end
+
+          def extract_data(result)
+            return nil unless result
+
+            if result.is_a?(Ask::Result)
+              result.content.is_a?(Hash) ? result.content : nil
+            elsif result.is_a?(Hash)
+              result
+            else
+              nil
+            end
+          end
+
+          def build_summary(tool_name, result, error)
+            if error
+              return { error: error.class.name }
+            end
+
+            if result.is_a?(Ask::Result)
+              if result.error?
+                return { error: "rejected: #{result.error.to_s.truncate(200)}" }
+              end
+              if result.blocked?
+                return { error: "blocked: #{result.content.to_s.truncate(200)}" }
+              end
+            end
+
+            data = extract_data(result)
+            return {} unless data
+
+            summary = {}
+            summary[:rows] = data[:rows]&.length if data.key?(:rows)
+            summary[:columns] = data[:columns]&.length if data.key?(:columns)
+            summary[:exit_status] = data[:exit_status] if data.key?(:exit_status)
+            summary[:size] = data[:size] if data.key?(:size)
+            summary[:matched_lines] = data[:matched_lines] if data.key?(:matched_lines)
+            summary[:results] = data[:results]&.length if data.key?(:results)
+            summary[:model] = data[:name] if data.key?(:name)
+            summary
+          end
+
+          def resolve_user_context
+            proc = Ask::Ruby::Harness.configuration.current_user
+            return nil unless proc.respond_to?(:call)
+
+            result = proc.call
+            result.is_a?(Hash) ? result : nil
+          rescue StandardError
+            nil
+          end
+
+          def environment_name
+            Ask::Ruby::Harness.env
+          end
+
+          def table_exists?
+            return false unless defined?(ActiveRecord::Base)
+
+            # Only cache the true result — recheck if it was false
+            return @table_exists if @table_exists
+
+            @table_exists = begin
+              conn = ActiveRecord::Base.connection
+              conn.data_source_exists?("ask_audit_logs")
+            rescue StandardError
+              false
+            end
+          end
+
+          def write_entry(entry)
+            # Serialize JSON fields for database storage
+            serialized = entry.dup
+            %i[params result_summary user_context].each do |key|
+              serialized[key] = ::JSON.generate(serialized[key]) if serialized[key].is_a?(Hash)
+            end
+
+            # Use raw SQL to avoid requiring a model class
+            columns = serialized.keys
+            values = columns.map { |col| ActiveRecord::Base.connection.quote(serialized[col]) }
+            ActiveRecord::Base.connection.execute(
+              "INSERT INTO ask_audit_logs (#{columns.join(', ')})
+               VALUES (#{values.join(', ')})"
+            )
+          rescue StandardError => e
+            # Silently fail — audit log should never crash the caller.
+            # ::Rails avoids the bare-`Rails` constant resolving to Ask::Rails;
+            # `&.` guards against Rails.logger returning nil (no app booted).
+            ::Rails.logger&.warn("[ask-ruby-harness] Audit log write failed: #{e.message}") if defined?(::Rails.logger)
+          end
+
+          # Reset cached table check (useful in tests)
+          public
+
+          def reset_table_check!
+            @table_exists = nil
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/ruby/harness/configuration.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+module Ask
+  module Ruby
+    module Harness
+      class Configuration
+        # Keys accepted when building a standalone connection from
+        # config/database.yml (unknown keys would fail AR's config validation).
+        DATABASE_CONFIG_KEYS = %w[adapter database host port username password url encoding pool].freeze
+
+        attr_accessor :default_model, :max_turns, :system_prompt,
+                      :tool_concurrency, :persistence_adapter, :tools,
+                      :current_user, :allowed_commands, :denied_commands,
+                      :max_session_age, :max_sessions
+
+        # @return [Hash{Symbol => EnvironmentPermissions}] per-environment permission rules
+        attr_reader :environments
+
+        def initialize
+          @default_model = "gpt-4o"
+          @max_turns = 25
+          @system_prompt = nil
+          @tool_concurrency = 5
+          @persistence_adapter = nil
+          @tools = []
+          @current_user = nil
+          @allowed_commands = nil
+          @denied_commands = nil
+          @max_session_age = nil
+          @max_sessions = nil
+          @environments = {}
+        end
+
+        # Configure permissions for a specific environment.
+        #
+        #   config.environment :production do |env|
+        #     env.mode = :read_only
+        #     env.allowed_commands = [/^rails routes/]
+        #     env.denied_commands = [/rm/, /dropdb/]
+        #   end
+        #
+        # @param name [Symbol, String] environment name (:production, :development, :staging, etc.)
+        def environment(name)
+          env = EnvironmentPermissions.new
+          yield env
+          @environments[name.to_sym] = env
+        end
+
+        # Resolved allowed commands for the current environment.
+        # Falls back to the global +allowed_commands+ if no per-env config.
+        #
+        # @return [Array<Regexp>, nil]
+        def effective_allowed_commands
+          env = @environments[Ask::Ruby::Harness.env.to_sym]
+          env&.allowed_commands || @allowed_commands
+        end
+
+        # Resolved denied commands for the current environment.
+        # Falls back to the global +denied_commands+ if no per-env config.
+        #
+        # @return [Array<Regexp>, nil]
+        def effective_denied_commands
+          env = @environments[Ask::Ruby::Harness.env.to_sym]
+          env&.denied_commands || @denied_commands
+        end
+
+        # Resolved access mode for the current environment.
+        #
+        # @return [Symbol, nil] +:full_access+, +:read_only+, +:ask_before_changes+, or nil
+        def effective_mode
+          env = @environments[Ask::Ruby::Harness.env.to_sym]
+          env&.mode || nil
+        end
+      end
+    end
+  end
+end

data/lib/ask/ruby/harness/environment_permissions.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Ask
+  module Ruby
+    module Harness
+      # Per-environment permission rules for agent tool access.
+      #
+      # Configure within Ask::Ruby::Harness.configure block:
+      #
+      #   Ask::Ruby::Harness.configure do |config|
+      #     config.environment :production do |env|
+      #       env.mode = :read_only
+      #       env.allowed_commands = [/^rails routes/, /^rails log/]
+      #       env.denied_commands = [/rm/, /dropdb/]
+      #     end
+      #
+      #     config.environment :development do |env|
+      #       env.mode = :full_access
+      #     end
+      #   end
+      #
+      class EnvironmentPermissions
+        # @return [Symbol, nil] Access mode for ask-agent's Permissions extension
+        #   (:full_access, :read_only, :ask_before_changes)
+        attr_accessor :mode
+
+        # @return [Array<Regexp>, nil] Allowed command patterns for RunCommand
+        attr_accessor :allowed_commands
+
+        # @return [Array<Regexp>, nil] Denied command patterns for RunCommand (takes precedence)
+        attr_accessor :denied_commands
+
+        def initialize
+          @mode = nil
+          @allowed_commands = nil
+          @denied_commands = nil
+        end
+      end
+    end
+  end
+end

data/lib/ask/ruby/harness/minitest_json_reporter.rb

@@ -0,0 +1,84 @@
+# frozen_string_literal: true
+
+require "minitest"
+require "json"
+
+module Ask
+  module Ruby
+    module Harness
+      # Writes a machine-readable JSON summary of a minitest run.
+      #
+      # Registered by the ask_ruby_harness minitest plugin when the
+      # ASK_TEST_JSON_PATH env var is set (see the run_tests tool). Produces
+      # stable structured results — test name, klass, source file/line, status,
+      # and a message head — so the harness never has to parse terminal output.
+      #
+      # The reporter is inert unless ASK_TEST_JSON_PATH is set, so ordinary
+      # `rake test` / `bin/rails test` runs in projects that ship the harness
+      # are unaffected.
+      class MinitestJsonReporter < Minitest::AbstractReporter
+        def initialize(path = ENV["ASK_TEST_JSON_PATH"])
+          super()
+          @path = path
+          @results = []
+        end
+
+        def record(result)
+          @results << result
+        end
+
+        def report
+          return if @path.nil? || @path.to_s.empty?
+          File.write(@path, JSON.pretty_generate(build_report))
+        end
+
+        # Never influences the process exit status — pass/fail is decided by
+        # minitest's own summary reporter.
+        def passed?
+          true
+        end
+
+        private
+
+        def build_report
+          tests = @results.map { |result| test_entry(result) }
+          {
+            "framework" => "minitest",
+            "run" => tests.size,
+            "failures" => tests.count { |t| t["status"] == "failed" },
+            "errors" => tests.count { |t| t["status"] == "error" },
+            "skips" => tests.count { |t| t["status"] == "skipped" },
+            "tests" => tests
+          }
+        end
+
+        def test_entry(result)
+          file, line = result.source_location
+          failure = result.failure
+          {
+            "name" => result.name,
+            "klass" => result.klass,
+            "file" => file,
+            "line" => line,
+            "time" => result.time,
+            "status" => status_of(result),
+            "message" => failure ? message_of(failure) : nil
+          }
+        end
+
+        def status_of(result)
+          return "skipped" if result.skipped?
+          return "error" if result.error?
+          return "passed" if result.passed?
+          "failed"
+        end
+
+        # First lines of the failure message — enough for an agent to act on
+        # without dumping full backtraces into the report.
+        def message_of(failure)
+          failure.message.to_s.lines.first(3).map(&:strip).reject(&:empty?).join("\n")
+        end
+      end
+    end
+  end
+end

data/lib/ask/ruby/harness/tool.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Ask
+  module Ruby
+    module Harness
+      # Base class for harness tools. Provides the app root and audit-logged
+      # execution with session correlation. Framework editions subclass this
+      # to pin the app root (ask-rails-harness overrides app_root to
+      # ::Rails.root via the railtie).
+      class Tool < Ask::Tool
+        def app_root
+          Ask::Ruby::Harness.app_root
+        end
+
+        # Override call to add audit logging around every tool execution.
+        # Logs the intent (sanitized params) and outcome (status, timing),
+        # but not the returned data.
+        def call(args = {}, abort_controller = nil)
+          start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          result = super
+          duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round
+
+          AuditLog.log(
+            session_id: Thread.current[:ask_session_id],
+            tool_name: name,
+            params: args,
+            result: result,
+            duration_ms: duration_ms
+          )
+
+          result
+        end
+
+        # Allow the session to set its ID for audit log correlation.
+        # Called by the agent loop before executing a tool.
+        def self.session_id=(id)
+          Thread.current[:ask_session_id] = id
+        end
+
+        def self.session_id
+          Thread.current[:ask_session_id]
+        end
+      end
+    end
+  end
+end

data/lib/ask/ruby/harness/tools/query_database.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+require "time"
+
+module Ask
+  module Ruby
+    module Harness
+      module Tools
+        class QueryDatabase < Ask::Ruby::Harness::Tool
+          description "Run a read-only SQL query against the application database. " \
+                       "Returns columns and rows. Only SELECT queries are allowed in production."
+
+          param :sql,   type: :string, desc: "SQL query (SELECT only in production)", required: true
+          param :limit, type: :integer, desc: "Max rows to return (default 50)", required: false
+
+          WRITE_STATEMENTS = /\A\s*(INSERT|UPDATE|DELETE|DROP|TRUNCATE|ALTER|CREATE|GRANT|REVOKE)\b/i
+
+          def execute(sql:, limit: 50)
+            sql = sql.strip
+
+            if WRITE_STATEMENTS.match?(sql)
+              return Ask::Result.failure(
+                "Only SELECT queries are allowed. Write statements (#{sql.match(WRITE_STATEMENTS)[1]}) are rejected in all environments."
+              )
+            end
+
+            if Ask::Ruby::Harness.env == "production" && !sql.match?(/\A\s*SELECT\b/i)
+              return Ask::Result.failure(
+                "Only SELECT queries are allowed in the production environment."
+              )
+            end
+
+            unless Ask::Ruby::Harness.database_connected?
+              Ask::Ruby::Harness.connect_database!
+            end
+            unless Ask::Ruby::Harness.database_connected?
+              return Ask::Result.failure(
+                "Database not connected. Set ASK_DATABASE_URL or provide a config/database.yml."
+              )
+            end
+
+            pool = ActiveRecord::Base.connection_pool
+            pool.with_connection do |conn|
+              limited_sql = sql.match?(/\bLIMIT\b/i) ? sql : "#{sql.chomp(';')} LIMIT #{limit.to_i}"
+              result = conn.exec_query(limited_sql)
+              columns = result.columns
+              rows = result.rows.first(limit.to_i).map { |row| build_row(row, columns) }
+              {
+                columns: columns,
+                rows: rows,
+                count: rows.size,
+                truncated: result.rows.size > limit.to_i
+              }
+            end
+          rescue ActiveRecord::StatementInvalid => e
+            Ask::Result.failure("SQL error: #{e.message}")
+          rescue ActiveRecord::ConnectionNotEstablished => e
+            Ask::Result.failure(
+              "Database not connected: #{e.message}. Set ASK_DATABASE_URL or config/database.yml."
+            )
+          end
+
+          private
+
+          def build_row(row, columns)
+            columns.each_with_index.each_with_object({}) do |(col, i), hash|
+              value = row[i]
+              hash[col] = sanitize_value(value)
+            end
+          end
+
+          def sanitize_value(value)
+            return "[BINARY DATA]" if binary_value?(value)
+            return value.iso8601 if value.respond_to?(:iso8601)
+            value
+          end
+
+          def binary_value?(value)
+            value.is_a?(String) && value.encoding == Encoding::ASCII_8BIT && value.bytesize > 0
+          rescue
+            false
+          end
+        end
+      end
+    end
+  end
+end