activeagent 1.9.4 → 1.9.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 715323643179a060b32d624ba906586778645e818c0436d4e4efe9ea6585847e
-  data.tar.gz: 3efe81e8a8d5357baf0f46adcfa5ce2fedd96234bb8ed603ffae63bac17bec58
+  metadata.gz: 21ef2f8df6b6cb8ff4d37cfa6586cad1a993714a0ca70a39c3354451c131680a
+  data.tar.gz: aa2b6cc445855ff76b6a22a10373887257be2e3fcfb9932360bace049b8ad1ba
 SHA512:
-  metadata.gz: 4a0ca1235fb0e8290ad261b61d77e38b4ee87372158384644612d54e81742f9f743968c8b8fa2c4e5cbebd57b62cdba7b0b0efbede6b3ad78176a7d7cca8dc37
-  data.tar.gz: 8fc1c553e0a008287280492dc401efccf1fdd675417985ef8acd8df3cb86ebe263a7cf6066b1740c8c387c1eb6de92ffffe428c5024b33a3ddf0464e729202a3
+  metadata.gz: 1650882a747d581893f75316cdd5731c4b67cb8596cc6d359b4ae9d5b7dd49daf2bfa255089ebd72eadb96064a5bb16fccc07c8613e92256f08f4f9586e91530
+  data.tar.gz: 0647b570ee151fd2fa231f39b9db0a5545ce5c8b3a8a23b33d4c08c25dc14fa17e9edb3c9a5718083c89bb2d4948260aa082be4de54edc91ddac76bb0680551b

data/CHANGELOG.md

@@ -5,6 +5,121 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.9.5] - 2026-10-28
+
+The `activeagent` command line: the gem's third surface beside the dashboard
+and its API, working with a dashboard over HTTP from a terminal or a CI job,
+and the facade tools it needed. A run, a scenario catalog and a trace each
+move between installs as one file. The framework and the dashboard engine
+move to 1.9.5 together.
+
+Upgrading: `rails generate action_agent:install` and `rails db:migrate` add
+the GitHub App webhook deliveries table (below). The CLI needs `actionagent`
+1.9.5 on the dashboard for `catalogs export`, `catalogs import --preview`,
+`agents versions` and `traces export` (the facade's new tools), and 1.9.3
+for `evals report` (the report export); an older dashboard answers those
+commands with exit code 5.
+
+### Added
+
+- **Evaluation runs published while they run** (`activeagent`, `actionagent`).
+  `ActiveAgent::Evals::Publisher#progress` sends the results so far as a
+  `"status": "running"` envelope, and the collector stores them as a running
+  run that the Evaluations page follows live; each later delivery under the
+  run_id replaces the results until `#call` completes the run. Only the first
+  delivery is admitted and counted. A progress delivery after completion
+  changes nothing. The collector lists `"progress"` in the `capabilities` of
+  its readiness answer, and `#progress` sends nothing to a collector that does
+  not (`#supports_progress?`), since an older one would store the partial
+  report as complete.
+- **Imported evaluation runs keep their dates** (`actionagent`). The
+  collector dates a run by its report's `metadata.finished`, which an export
+  carries, so an exported run imported into another install keeps its place
+  in the run history instead of reading as just run. A report naming no
+  finish time is dated on arrival, and a malformed one is refused.
+- **`activeagent` command line** (`activeagent`, `actionagent`; refs #619).
+  The gem ships an `activeagent` executable (stdlib `OptionParser` and
+  `Net::HTTP`, no new dependencies) that works with a dashboard over HTTP:
+  `login`/`logout` (named profiles in `~/.config/activeagent/credentials.yml`,
+  `0600`; `ACTIVEAGENT_URL`, `ACTIVEAGENT_API_KEY` and
+  `ACTIVEAGENT_INGEST_KEY` override them for CI), `status`,
+  `evals list|runs|run|report|import`, `catalogs list|import|export`
+  (`import --preview` shows what a document would change, `--mode merge`
+  keeps what it leaves out), `agents list|versions` and
+  `traces list|get|export|import`, with `--json` output and distinct exit
+  codes. `evals run --min-pass-rate 0.8` waits for the run and exits 4
+  below the threshold, for CI gating. Reads and runs go through the MCP
+  facade's dashboard tools with a dashboard API key; the imports and
+  `evals report` use the endpoints the framework's own reporters post to,
+  with the ingest key. A run, a catalog and a trace each move between
+  installs as one file: `evals report RUN --format json` writes the
+  version-1 envelope `evals import` posts (`--format html|md` the report
+  page or its Markdown, for a pull request comment), `catalogs export KEY`
+  the canonical document (YAML, or `--format json`) that `catalogs import`
+  reads, and `traces export TRACE` the payload `traces import` sends to
+  trace ingest. See `docs/framework/cli.md`.
+
+- **Four more dashboard tools on the MCP facade** (`actionagent`), the
+  API-key routes the CLI needed: `catalogs_export` (a catalog's canonical
+  document as YAML or JSON, whole or one product or set), `catalogs_preview`
+  (what `catalogs_import` would change, writing nothing), `agents_versions`
+  (an agent's versions with their diffs, as `GET /api/agents/:id/versions`)
+  and `traces_export` (one trace, whole and untruncated, as the payload
+  `POST <mount>/api/traces` takes, so a trace moves between installs).
+  `catalogs_import` takes `mode: replace|merge` as the API does. Every run
+  summary (`GET /api/evaluations`, the facade's `evaluations_get`,
+  `evaluations_run` and `evaluation_runs_get`) now carries `run_id`, the
+  handle `GET <mount>/api/evaluation_reports/:run_id` takes.
+
+- **GitHub App webhooks** (`actionagent`). `POST <mount>/api/github_app/webhooks`
+  receives the App's deliveries, verified with `X-Hub-Signature-256` under
+  `ActionAgent.github_app_webhook_secret` (falling back to
+  `GITHUB_APP_WEBHOOK_SECRET`; unset, the endpoint answers 404), and applies
+  them in `ActionAgent::GithubWebhookJob`: an uninstall marks the linked
+  installation removed, suspend and unsuspend set and clear `suspended_at`,
+  accepted permissions are recorded, a repository removed from the
+  installation leaves the owner's selection, and a revoked App authorization
+  deletes the user's stored connection when connections authorize through the
+  App. Each `X-GitHub-Delivery` is recorded for seven days, so a redelivery
+  is a no-op. Ships a new migration,
+  `create_active_agent_github_webhook_deliveries`; run
+  `rails generate action_agent:install` and `rails db:migrate` to add it.
+
+### Fixed
+
+- **Sandbox sign-in recognises Claude Code 2.1's authorization page**
+  (`actionagent`). "Sign in with your Claude subscription" on a local sandbox
+  stayed at "Preparing Claude's authorization page": the CLI now sends
+  sign-ins to `claude.com/cai/oauth/authorize` and prints the link as a
+  terminal hyperlink whose text repeats the URL, which the flow neither
+  accepted nor cut at the hyperlink's end. Both are read now, and
+  `claude.ai` and `platform.claude.com` still are.
+- **Projects accept GitHub App installations** (`actionagent`). A dashboard
+  that runs only the GitHub App could link an installation but not create a
+  project ("GitHub is not connected"). The New Project checklist now counts a
+  usable installation as connected, `GET /api/projects/repositories` lists
+  the installations' repositories together with the OAuth connection's, and
+  the preflight, secret discovery and creation reach a repository through the
+  installation first, as its checkout does, with a token minted for reading
+  that one repository. Picking a repository the installation has not
+  selected selects it there. A removed or suspended installation answers
+  `installation_unavailable`.
+- **Report a missing Members permission when linking an organization's
+  GitHub App installation** (`actionagent`). GitHub answers the membership
+  check with 403 while the App lacks the organization permission Members
+  (read); the callback now reports `members_permission` instead of
+  `not_admin`. The dashboard guide documents the permission, and that the
+  dashboard's callback URL must be the App's first.
+
+- **Boot on Ruby 3.4 and 4.0 with bootsnap 1.24.6 or newer** (`actionagent`).
+  `ProviderKey#owner_record` unwrapped delegators with
+  `... while defined?(::Delegator) && ...`, which Prism miscompiles when the
+  source is compiled from a string ("argument stack underflow"); bootsnap
+  compiles files that way on those Rubies as its Ruby bug #22023 workaround,
+  so a host on bootsnap 1.26 (and local runs on Ruby 3.4.9 or 4.0.1) failed
+  to boot the engine. The `defined?` check now sits outside the loop, and a
+  test compiles every shipped Ruby file from a string.
+
 ## [1.9.4] - 2026-10-25
 
 ### Added

data/README.md

@@ -181,7 +181,12 @@ telemetry:
 The generator mounts the engine at `/activeagents` — open it and every
 generation appears as a trace. See
 [docs/framework/dashboard.md](docs/framework/dashboard.md) for
-authentication, remote ingestion, and multi-tenant mode. The hosted
+authentication, remote ingestion, and multi-tenant mode. The gem's
+`activeagent` command line works with that dashboard from a terminal or a
+CI job — `activeagent evals run "Support orders" --min-pass-rate 0.8` fails
+a build below the threshold, and runs, scenario catalogs and traces move
+between installs as JSON files; see
+[docs/framework/cli.md](docs/framework/cli.md). The hosted
 platform at [activeagents.ai](https://activeagents.ai) runs this same
 engine multi-tenant, adding what a hosted product has to have — accounts,
 plans, billing, quotas and managed sandboxes; every workspace starts with
@@ -198,6 +203,7 @@ a free low-volume trial.
 - **Agent-as-Tool Delegation**: Hand work to sub-agents with declared schemas, cost/latency budgets, and swappable backends
 - **Context Management**: Maintain conversation history across interactions
 - **Structured Output**: Define JSON schemas for predictable responses
+- **Command Line**: `activeagent` runs evaluations and gates CI on a pass rate from a terminal, and moves runs, scenario catalogs and traces between dashboards as JSON
 
 ## Examples
 

data/exe/activeagent

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# The activeagent CLI. Loads only lib/active_agent/cli, not Rails or the
+# framework. See docs/framework/cli.md.
+require_relative "../lib/active_agent/cli"
+
+exit ActiveAgent::CLI.start(ARGV)

data/lib/active_agent/cli/client.rb

@@ -0,0 +1,227 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+require "openssl"
+require "uri"
+require "zlib"
+require_relative "../evals/publisher"
+
+module ActiveAgent
+  class CLI
+    # HTTP to one dashboard mount, with the conventions of
+    # ActiveAgent::Evals::Publisher: blocking, bounded timeouts, no redirects
+    # followed with a bearer key, and errors that never quote a response body
+    # beyond its sanitized +error+ string, with the keys filtered out.
+    #
+    # The dashboard authenticates two kinds of key, and this client sends
+    # each where it belongs:
+    #
+    # - +api_key+, a dashboard API key (Settings -> API Keys, "aa_..."), for
+    #   the MCP facade at <mount>/mcp, whose dashboard tools back most
+    #   commands.
+    # - +ingest_key+, the install's ingest key (ActionAgent.ingest_api_key,
+    #   or the tenant's telemetry key), for the endpoints the framework's
+    #   reporters post to: the report collector at
+    #   <mount>/api/evaluation_reports and trace ingest at <mount>/api/traces.
+    #   Defaults to +api_key+.
+    class Client
+      DETAIL_LIMIT = 200
+      JSONRPC_INVALID_PARAMS = -32602
+
+      attr_reader :url
+
+      def initialize(url:, api_key: nil, ingest_key: nil, timeout: 30, open_timeout: 10)
+        @uri = self.class.parse_url(url)
+        @url = @uri.to_s
+        @api_key = api_key.to_s.strip
+        @ingest_key = ingest_key.to_s.strip.empty? ? @api_key : ingest_key.to_s.strip
+        @timeout = timeout
+        @open_timeout = open_timeout
+        @request_id = 0
+      end
+
+      # The dashboard mount as a URI, without a trailing slash. HTTPS is
+      # required except on loopback hosts, as for the publisher.
+      def self.parse_url(url)
+        uri = URI.parse(url.to_s.strip.sub(%r{/+\z}, ""))
+        unless uri.is_a?(URI::HTTP) && uri.host && !uri.userinfo && !uri.query && !uri.fragment
+          raise Error.new("The dashboard URL must be an HTTP(S) URL without credentials, query or fragment, " \
+                          "such as https://ops.example.com/activeagents", exit_code: EXIT_USAGE)
+        end
+        unless uri.scheme == "https" || %w[localhost 127.0.0.1 ::1].include?(uri.hostname)
+          raise Error.new("The dashboard URL requires HTTPS except on loopback hosts", exit_code: EXIT_USAGE)
+        end
+        uri
+      rescue URI::InvalidURIError
+        raise Error.new("The dashboard URL is not a valid URL", exit_code: EXIT_USAGE)
+      end
+
+      def mcp_endpoint
+        "#{url}/mcp"
+      end
+
+      def reports_endpoint
+        "#{url}/api/evaluation_reports"
+      end
+
+      def traces_endpoint
+        "#{url}/api/traces"
+      end
+
+      # A Publisher for the mount's collector, authenticated with the ingest key.
+      def publisher
+        require_key!(@ingest_key, "an ingest key")
+        ActiveAgent::Evals::Publisher.new(api_key: @ingest_key, endpoint: reports_endpoint, timeout: @timeout, open_timeout: @open_timeout)
+      rescue ArgumentError => e
+        raise Error.new(e.message, exit_code: EXIT_CONFIG)
+      end
+
+      # Sends one JSON-RPC request to the MCP facade and returns its result.
+      def mcp(method, params = {})
+        require_key!(@api_key, "a dashboard API key")
+        @request_id += 1
+        body = JSON.generate(jsonrpc: "2.0", id: @request_id, method: method, params: params)
+        response = exchange(:post, URI(mcp_endpoint), @api_key, body: body)
+
+        case response.code
+        when "200" then nil
+        when "401" then raise refused("the dashboard refused the API key; create one under Settings -> API Keys", response, @api_key, 401)
+        when "404" then raise refused("nothing at #{mcp_endpoint} answers MCP; check the dashboard URL (the engine's mount)", response, @api_key, 404)
+        else raise refused("the dashboard's MCP endpoint answered HTTP #{response.code}", response, @api_key, response.code.to_i)
+        end
+
+        payload = parse_json(response.body)
+        if (error = payload["error"])
+          raise Error.new("#{method} failed: #{sanitize(error["message"], @api_key) || "error #{error["code"]}"}", status: error["code"])
+        end
+        payload["result"]
+      end
+
+      # Calls one of the dashboard's MCP tools and returns its structured
+      # result. A tool's own refusal (isError) raises with its message; a
+      # dashboard tool the facade does not list means the install turned
+      # them off (ActionAgent.mcp_dashboard_tools = false) or predates them.
+      def tool(name, arguments = {})
+        result = mcp("tools/call", name: name, arguments: arguments.compact)
+        structured = result.is_a?(Hash) ? result["structuredContent"] : nil
+        if result.is_a?(Hash) && result["isError"]
+          message = structured.is_a?(Hash) && structured["error"] || Array(result["content"]).filter_map { |part| part["text"] if part.is_a?(Hash) }.join(" ")
+          raise Error.new("#{name}: #{sanitize(message, @api_key) || "the tool refused the call"}")
+        end
+        structured.is_a?(Hash) ? structured : {}
+      rescue Error => e
+        raise unless e.status == JSONRPC_INVALID_PARAMS && e.message.include?("Unknown tool")
+
+        raise Error.new("This dashboard does not offer the #{name} tool over MCP: it predates the dashboard tools, or " \
+                        "the install sets ActionAgent.mcp_dashboard_tools = false", exit_code: EXIT_UNAVAILABLE)
+      end
+
+      # Rails' own 404 body, from a dashboard whose routes have no such path.
+      GENERIC_NOT_FOUND = /\Anot found\z/i
+
+      # GETs +path+ under the mount with the ingest key and returns the
+      # response, raising on anything but 200. A 404 is one of two things:
+      # the endpoint's own refusal, a JSON error naming what is missing,
+      # raised as +missing+ with that detail; or nothing at the path at all
+      # (an older engine), raised as +unavailable+ with EXIT_UNAVAILABLE.
+      def get(path, accept:, query: nil, missing: nil, unavailable: nil)
+        require_key!(@ingest_key, "an ingest key")
+        uri = URI("#{url}#{path}")
+        uri.query = query if query
+        response = exchange(:get, uri, @ingest_key, accept: accept)
+        return response if response.code == "200"
+
+        case response.code
+        when "404"
+          detail = error_detail(response, @ingest_key)
+          raise Error.new("#{missing} (#{detail})", status: 404) if missing && detail && !detail.match?(GENERIC_NOT_FOUND)
+
+          raise Error.new(unavailable || "Nothing at #{path} answers; check the dashboard URL", status: 404, exit_code: EXIT_UNAVAILABLE)
+        when "401"
+          raise refused("the dashboard refused the ingest key; use the mount's ingest_api_key (--ingest-key)", response, @ingest_key, 401)
+        else
+          raise refused("GET #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i)
+        end
+      end
+
+      # POSTs +body+, a JSON string, to +path+ under the mount with the
+      # ingest key and returns the response, raising on anything but a 2xx.
+      def post(path, body)
+        require_key!(@ingest_key, "an ingest key")
+        response = exchange(:post, URI("#{url}#{path}"), @ingest_key, body: body)
+        return response if response.code.start_with?("2")
+
+        if response.code == "401"
+          raise refused("the dashboard refused the ingest key; use the mount's ingest_api_key (--ingest-key)", response, @ingest_key, 401)
+        end
+        raise refused("POST #{path} answered HTTP #{response.code}", response, @ingest_key, response.code.to_i)
+      end
+
+      private
+
+      def require_key!(key, label)
+        return unless key.empty?
+
+        raise Error.new("No #{label}: run `activeagent login`, or set ACTIVEAGENT_API_KEY", exit_code: EXIT_CONFIG)
+      end
+
+      def exchange(verb, uri, key, body: nil, accept: "application/json")
+        http = Net::HTTP.new(uri.hostname, uri.port)
+        http.use_ssl = uri.scheme == "https"
+        http.open_timeout = @open_timeout
+        http.read_timeout = @timeout
+        http.write_timeout = @timeout
+        request = (verb == :post ? Net::HTTP::Post : Net::HTTP::Get).new(uri.request_uri)
+        request["Authorization"] = "Bearer #{key}"
+        request["Accept"] = accept
+        request["User-Agent"] = "activeagent-cli/#{ActiveAgent::CLI.version}"
+        if body
+          request["Content-Type"] = "application/json"
+          request.body = body
+        end
+        http.request(request)
+      rescue Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Zlib::Error => e
+        # These messages can quote the response's status line, headers or body.
+        raise Error.new("The dashboard returned a malformed response (#{e.class})"), cause: nil
+      rescue IOError, SocketError, SystemCallError, Timeout::Error, OpenSSL::SSL::SSLError => e
+        raise Error.new("Could not reach #{uri.scheme}://#{uri.host} (#{e.class}); check the URL and that the dashboard is up")
+      end
+
+      def parse_json(body)
+        parsed = JSON.parse(body.to_s)
+        raise Error, "The dashboard returned JSON that is not an object" unless parsed.is_a?(Hash)
+
+        parsed
+      rescue JSON::ParserError
+        # The parser's message quotes the body.
+        raise Error.new("The dashboard returned invalid JSON"), cause: nil
+      end
+
+      def refused(message, response, key, status, exit_code: EXIT_FAILURE)
+        detail = error_detail(response, key)
+        Error.new(detail ? "#{message} (#{detail})" : message, status: status, exit_code: exit_code)
+      end
+
+      # The sanitized `error` string of a JSON error body, or nil.
+      def error_detail(response, key)
+        parsed = JSON.parse(response.body.to_s)
+        sanitize(parsed["error"], key) if parsed.is_a?(Hash)
+      rescue JSON::ParserError, EncodingError
+        nil
+      end
+
+      # Control characters and runs of whitespace become one space, the keys
+      # become [FILTERED], and the result is cut to DETAIL_LIMIT characters.
+      def sanitize(message, key)
+        return unless message.is_a?(String)
+
+        detail = message.scrub.gsub(/[[:space:]\p{C}]+/, " ").strip
+        [ key, @api_key, @ingest_key ].uniq.reject(&:empty?).each { |secret| detail = detail.gsub(secret, "[FILTERED]") }
+        return if detail.empty?
+
+        detail.length > DETAIL_LIMIT ? "#{detail[0, DETAIL_LIMIT - 1].rstrip}…" : detail
+      end
+    end
+  end
+end

data/lib/active_agent/cli/credentials.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "yaml"
+
+module ActiveAgent
+  class CLI
+    # Named dashboard profiles in ~/.config/activeagent/credentials.yml
+    # ($XDG_CONFIG_HOME when set), written 0600 inside a 0700 directory:
+    #
+    #   default:
+    #     url: https://ops.example.com/activeagents
+    #     api_key: aa_...
+    #     ingest_key: ...   # only when it differs from api_key
+    #
+    # A profile's values are what +login+ stored; the environment
+    # (ACTIVEAGENT_URL, ACTIVEAGENT_API_KEY, ACTIVEAGENT_INGEST_KEY) overrides
+    # them in CLI#profile, so CI never needs the file.
+    class Credentials
+      FIELDS = %w[url api_key ingest_key].freeze
+
+      attr_reader :path
+
+      def self.default_path(env = ENV)
+        base = env["XDG_CONFIG_HOME"].to_s.empty? ? File.join(Dir.home, ".config") : env["XDG_CONFIG_HOME"]
+        File.join(base, "activeagent", "credentials.yml")
+      end
+
+      def initialize(path)
+        @path = path
+      end
+
+      # Returns the profile's stored values, or an empty hash when the file or
+      # the profile does not exist.
+      def [](name)
+        profiles.fetch(name.to_s, {})
+      end
+
+      def names
+        profiles.keys
+      end
+
+      def save(name, values)
+        data = profiles.merge(name.to_s => values.transform_keys(&:to_s).slice(*FIELDS).compact)
+        FileUtils.mkdir_p(File.dirname(path), mode: 0o700)
+        # Written beside the target and renamed over it, so a reader never
+        # sees half a file, and created 0600 so the key is never readable by
+        # others, even for a moment.
+        temp = "#{path}.#{Process.pid}.tmp"
+        File.open(temp, File::WRONLY | File::CREAT | File::TRUNC, 0o600) { |file| file.write(YAML.dump(data)) }
+        File.chmod(0o600, temp)
+        File.rename(temp, path)
+      ensure
+        FileUtils.rm_f(temp) if temp && File.exist?(temp)
+      end
+
+      def delete(name)
+        data = profiles
+        return false unless data.delete(name.to_s)
+
+        data.empty? ? File.delete(path) : File.open(path, "w", 0o600) { |file| file.write(YAML.dump(data)) }
+        true
+      end
+
+      private
+
+      def profiles
+        return {} unless File.exist?(path)
+
+        data = YAML.safe_load_file(path) || {}
+        raise Error.new("#{path} is not a profile map; fix or remove it", exit_code: EXIT_CONFIG) unless data.is_a?(Hash)
+
+        data.select { |_, values| values.is_a?(Hash) }
+      rescue Psych::Exception => e
+        raise Error.new("#{path} is not valid YAML (#{e.class}); fix or remove it", exit_code: EXIT_CONFIG)
+      end
+    end
+  end
+end