ruby_llm-resilience 0.6.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1006814fe7e55a8b61950aab8f70db4722183e6dedc28e9e00940ea408621af4
+  data.tar.gz: 07c2b081ae0277e41fc8aa90ea0545ccd3e467984639e045b103fc91099c0be4
+SHA512:
+  metadata.gz: c932fdf3b844f0fef419a61ec71b47d6d9ee823c772fce83294b5d716c2423589c61d7b9e86c80c0f06f505ad30b19de57cf1774d5eab278be255bb42b471df3
+  data.tar.gz: fbc659c0e221d00511b19ed3e37b5117e2274629b4ae6d684371a9238337bbab1ba7c24358d13cad61e35ae2adc9e61c974b36587cc0e2243255e7dd4018c1de

data/CHANGELOG.md

@@ -0,0 +1,100 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.6.0] - 2026-07-14
+
+Rails DX:
+
+- `rails g resilience:install` — writes a fully-commented initializer
+  covering every configuration seam
+- The dashboard engine now loads automatically under Rails (no Gemfile
+  `require:` needed; mounting stays opt-in). Non-Rails usage unchanged —
+  the core still has zero dependencies
+- RubyGems Trusted Publishing release workflow (.github/workflows/release.yml)
+
+## [0.5.0] - 2026-07-14
+
+- Dashboard table: click-to-sort columns (text/numeric aware) and a
+  service/state filter box — dependency-free vanilla JS, with sort/filter
+  state persisted in localStorage so it survives the 10s auto-refresh.
+
+## [0.4.0] - 2026-07-14
+
+Dashboard fleshed out:
+
+- **Fallback routes column** — model chains from `fallback_models`, grouped
+  by breaker service via the namer (`Resilience.fallback_routes` is public
+  API for custom dashboards)
+- Failures shown against the effective per-service threshold (`1 / 5`);
+  cooldown column with per-service override markers
+- **Configuration panel** — store, defaults, fallback-map size, and whether
+  each telemetry hook (`on_error`/`on_status`/`on_fallback`) is configured
+  or still a no-op
+- `Breaker.dashboard_status` rows now include `failure_threshold`,
+  `cooldown_seconds`, `overridden`
+- README: telemetry recipes (alert on the handled error, graph the gauge,
+  trend the counter)
+
+## [0.3.3] - 2026-07-14
+
+- New `config.dashboard_services`: default service list for the engine
+  dashboard (nil = per-process registry). Apps with a known static fleet
+  get a complete dashboard from the first request. (Third
+  production-adoption catch.)
+
+## [0.3.2] - 2026-07-14
+
+- Fix dashboard template lookup in apps using the ruby_llm gem: its Railtie
+  registers `acronym "RubyLLM"`, which changed the engine controller's
+  derived controller_path and broke view resolution. controller_path is now
+  pinned. (Second production-adoption catch.)
+
+## [0.3.1] - 2026-07-14
+
+- Breaker registry now uses shared constants instead of class-ivars, making
+  Breaker safely subclassable (apps can add service lists/aliases in a
+  subclass). Found while dogfooding — the shim subclass crashed on v0.3.0.
+
+## [0.3.0] - 2026-07-13
+
+Fallback routing is now fully user-configurable:
+
+- **Multi-hop maps**: `fallback_models` values accept an array of hops
+  (`"gemini-3.5-flash" => ["claude-sonnet-4-6", "claude-opus-4-7"]`) as well
+  as a single model. One deliberate hop remains the documented default.
+- **Per-call override**: `run_with_model_fallback(model, fallback: ...)` —
+  `:map` (default), `false`/`nil` (breaker only, no routing), a model
+  string, or an array chain. Expresses variant→control patterns in one call.
+- **`on_fallback` callback**: fires on every chain advance with
+  `from:`, `to:` (service + model) and the triggering `error:` — including
+  skipped-open steps. Fallbacks are now observable, not silent.
+
+## [0.2.0] - 2026-07-13
+
+- **Mountable dashboard engine** (`require "ruby_llm/resilience/engine"`,
+  `mount RubyLLM::Resilience::Engine => "/resilience"`): live state pills,
+  failure counts, probe countdowns, metadata, reset buttons, auto-refresh.
+  Deny-by-default auth via `config.dashboard_auth` — every request 404s
+  until explicitly allowed. Core remains zero-dependency; the engine loads
+  only under Rails.
+- **Per-service overrides**: `config.services = { "api:x" =>
+  { failure_threshold:, cooldown_seconds:, failures_window_seconds: } }`.
+- **Service metadata**: `config.service_metadata` — dashboard descriptions
+  as config, not hardcoded controllers; included in `dashboard_status`.
+
+## [0.1.0] - 2026-07-13
+
+Initial release. Design proven across 450k+ production LLM calls.
+
+- `RubyLLM::Resilience.run(service)` — circuit-broken single calls
+- `run_with_model_fallback(model)` — one tier-hop via `fallback_models`
+- `run_with_fallback(*steps)` — skip-open chains, `BreakerTripped` on
+  exhaustion when any breaker was open
+- `Breaker` — CLOSED→OPEN→HALF_OPEN state machine, atomic single-probe
+  SETNX lock, atomic trip transition, fail-open on store outage
+- `allow_request?` (mutating gate) split from pure `open?`/`state` reads
+- Five injection seams: `cache_store`, `on_error`, `on_status`,
+  `provider_resolver`, `service_namer`
+- Zero runtime dependencies; lazy resolution of RubyLLM/Faraday error
+  classes; built-in thread-safe `MemoryStore`

data/LICENSE.txt

@@ -0,0 +1,22 @@
+
+MIT License
+
+Copyright (c) 2026 Daniel St Paul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,226 @@
+# ruby_llm-resilience
+
+**RubyLLM gives you every provider. This gives you what happens when one of
+them goes down.**
+
+Circuit breakers and fallback chains for LLM apps — battle-tested behind
+**450,000+ production LLM calls**.
+
+```ruby
+# Guard any call
+RubyLLM::Resilience.run("api:openai:embeddings") { RubyLLM.embed(text) }
+
+# Fallback routing when a model's tier is struggling — fully configurable:
+RubyLLM::Resilience.run_with_model_fallback("claude-haiku-4-5") do |model|
+  RubyLLM.chat(model: model).ask(prompt)   # hops per your fallback_models map
+end
+# fallback: false          → no routing, breaker only
+# fallback: "gpt-5.5"      → explicit per-call hop, ignoring the map
+# fallback: ["a", "b"]     → explicit multi-hop chain
+
+# Custom cross-provider chain
+RubyLLM::Resilience.run_with_fallback(
+  { service: "api:openai:moderation", call: -> { RubyLLM.moderate(content) } },
+  { service: "api:anthropic:haiku",   call: -> { my_anthropic_moderator.call(content) } }
+)
+```
+
+Zero runtime dependencies. Works with any cache store that speaks five
+methods. Fails open when your store blips — the breaker never becomes the
+outage.
+
+## Install
+
+```ruby
+gem "ruby_llm-resilience"
+```
+
+```bash
+rails g resilience:install   # writes a fully-commented initializer
+```
+
+Outside Rails, or by hand:
+
+```ruby
+# config/initializers/resilience.rb
+RubyLLM::Resilience.configure do |c|
+  # REQUIRED for multi-process apps — the default MemoryStore is per-process.
+  c.cache_store = ActiveSupport::Cache::RedisCacheStore.new(url: ENV["REDIS_URL"])
+
+  c.fallback_models = {
+    "claude-haiku-4-5"  => "claude-sonnet-4-6",
+    "claude-sonnet-4-6" => "claude-opus-4-7",
+    "claude-opus-4-7"   => "claude-sonnet-4-6",
+    "gemini-3.5-flash"  => "claude-sonnet-4-6"   # cross-provider safety net
+  }
+
+  c.on_error  = ->(error, ctx) { Rails.error.report(error, handled: true, context: ctx) }
+  c.on_status = ->(service, state) {
+    Appsignal.set_gauge("circuit_breaker.state", state == :open ? 1 : 0, service: service)
+  }
+end
+```
+
+## How it works
+
+State machine per service: `CLOSED → OPEN → HALF_OPEN → CLOSED`.
+
+- **Trip**: `failure_threshold` consecutive server-side failures (default 5)
+  open the breaker for `cooldown_seconds` (default 120).
+- **Probe**: after cooldown, an atomic SETNX lock lets **exactly one** caller
+  across all your processes probe the provider. Success closes; failure
+  re-opens immediately.
+- **Chains**: steps whose breaker is open are *skipped*, first success wins,
+  and if the chain exhausts with any step skipped-open you get
+  `BreakerTripped` — so you can distinguish "the providers are down" (fail
+  closed, show the friendly banner) from "this request failed."
+
+## Design notes — what 450k production calls taught us
+
+These are the decisions that differ from a textbook breaker, learned in
+production:
+
+- **4xx never trips.** Client errors are your bug, not their outage. Only
+  rate limits, 5xx, overloads and transport timeouts count toward the
+  threshold — and the failure count *survives* interleaved 4xx errors.
+- **Tier-hop fallbacks, one hop only.** Provider capacity incidents are
+  tier-correlated: when Sonnet is overloaded, every Sonnet version is.
+  Same-tier version hops are theatre. Hopping tier (haiku→sonnet) puts you
+  on a separate breaker and a separate capacity pool. And chains don't
+  chase transitively — one deliberate escape per model.
+- **Tier-level breaker names.** `api:anthropic:sonnet`, not
+  `api:anthropic:claude-sonnet-4-6` — a version rollover shouldn't fragment
+  your health state across two breakers.
+- **`BreakerTripped` on exhaustion, not the last error.** Callers must be
+  able to fail *closed* on persistent provider failure while failing *open*
+  on one-off errors.
+- **Fail-open on store outage.** If Redis is down, every breaker reports
+  closed and records nothing. The breaker must never take the app down.
+- **`allow_request?` vs `open?`.** The gate that consumes the half-open
+  probe slot is separate from the pure reads — so your dashboard can poll
+  `state`/`open?` forever without stealing probes.
+- **`ModelNotFoundError` advances chains.** New-model rollout windows leave
+  registries briefly stale; degrade to the fallback model instead of
+  surfacing "unknown model" to users.
+- **Deterministic failover, not smart routing.** OpenRouter-style routing
+  optimizes continuously on fleet-wide signal a single app doesn't have —
+  and prompts are model-specific, so silently serving a Sonnet-tuned prompt
+  on GPT is an unlogged quality regression, not an optimization. Here the
+  map is explicit, hops are emergencies, and every hop fires `on_fallback`
+  so your telemetry sees exactly what routed where and why. (RubyLLM speaks
+  OpenRouter as a provider, so the two compose if you want both.)
+
+## The five seams
+
+Everything app-specific is injectable:
+
+| Seam | Default | Wire it to |
+|---|---|---|
+| `cache_store` | in-process `MemoryStore` | Redis (any object with `read`/`write(expires_in:, unless_exist:)`/`increment(expires_in:)`/`delete`/`delete_multi`) |
+| `on_error` | no-op | `Rails.error.report`, Sentry, Honeybadger |
+| `on_status` | no-op | AppSignal/Datadog gauge (fires on every success and every trip — gauge semantics) |
+| `provider_resolver` | RubyLLM's model registry, rescued | your own model→provider logic |
+| `service_namer` | `TierNamer` (per-provider tiers) | `->(model) { "api:#{model}" }` for per-model breakers |
+
+Not LLM-specific, by the way: we run the same breakers around OCR, Slack
+webhooks, CAPTCHA verification and email-validation APIs. Configure custom
+`trippable_errors`/`fallback_errors` and guard anything.
+
+## Per-service configuration
+
+One cooldown doesn't fit all: a cheap fast-recovery moderation endpoint and
+an expensive batch endpoint shouldn't share settings.
+
+```ruby
+RubyLLM::Resilience.configure do |c|
+  c.failure_threshold = 5      # global defaults...
+  c.cooldown_seconds  = 120
+
+  c.services = {               # ...with per-service overrides
+    "api:openai:moderation" => { failure_threshold: 2, cooldown_seconds: 30 },
+    "api:anthropic:batches" => { cooldown_seconds: 600 }
+  }
+
+  c.service_metadata = {       # app knowledge for the dashboard — config, not code
+    "api:anthropic:sonnet" => { description: "Coaching + generation", consumers: "Chat, Practice" }
+  }
+end
+```
+
+## Telemetry recipes
+
+Three hooks, three different jobs — use all three:
+
+| Hook | Fires | Wire it to | Use it for |
+|---|---|---|---|
+| `on_error` | every **trip** (with the `BreakerTripped` exception) and store outages | `Rails.error.report(error, handled: true, context: ctx)` → AppSignal/Sentry as a **handled error** | Alerting. Trips are rare and high-signal; handled errors group by class and dedupe. |
+| `on_status` | every success (`:closed`, gauge-idempotent) and every trip (`:open`) | `Appsignal.set_gauge("circuit_breaker.state", state == :open ? 1 : 0, service:)` | State-over-time graphs; alert on a gauge stuck at 1. |
+| `on_fallback` | every **hop** a chain takes past a failed/skipped step | `Appsignal.increment_counter("llm.fallback", 1, from:, to:, error:)` | Volume/cost trends — how often you're degrading to a pricier model. Not an alert; a dashboard line. |
+
+Rule of thumb: **alert on the handled error, graph the gauge, trend the
+counter.** The counter isn't an alternative to error reporting — a trip is
+one event, but a tripped 5-minute window can be thousands of hops; you want
+the former in your inbox and the latter on a chart.
+
+## The dashboard
+
+An optional mountable engine ships with the gem (the core stays
+zero-dependency — the engine only loads when you ask for it under Rails):
+
+```ruby
+# config/application.rb
+require "ruby_llm/resilience/engine"
+
+# config/routes.rb
+mount RubyLLM::Resilience::Engine => "/resilience"
+
+# config/initializers/resilience.rb — REQUIRED: the dashboard denies by
+# default; every request 404s until you explicitly allow it:
+RubyLLM::Resilience.configure do |c|
+  c.dashboard_auth = ->(controller) {
+    controller.head :not_found unless controller.current_user&.admin?
+  }
+end
+```
+
+One page: live state pills per service, failures shown against their
+*effective* threshold (`1 / 5`), probe countdowns, per-service cooldowns
+(with override markers), **fallback routes** derived from your model map
+(`claude-haiku-4-5 → claude-sonnet-4-6`), your `service_metadata`
+descriptions, reset buttons, and a configuration panel showing your store,
+defaults, and which telemetry hooks are wired vs still no-ops.
+Auto-refreshes — safely, because **all dashboard reads are pure**: polling
+never consumes half-open probe slots (that's why `allow_request?` and
+`open?` are separate methods).
+
+Prefer your own admin panel? The data API is public:
+
+```ruby
+RubyLLM::Resilience::Breaker.dashboard_status
+# => [{ service: "api:anthropic:sonnet", state: :closed, failure_count: 0,
+#       seconds_until_probe: nil, metadata: { description: "..." } }, ...]
+
+RubyLLM::Resilience::Breaker.new("api:anthropic:sonnet").reset!  # console escape hatch
+```
+
+## Vs. circuitbox / stoplight / semian / breaker_machines / ruby_llm-agents
+
+- **[semian](https://github.com/Shopify/semian)** intercepts at the driver
+  level — superb for MySQL/Redis, structurally wrong for LLM SDK calls.
+- **circuitbox / stoplight** are excellent *generic single breakers*. What
+  they don't do is the part this gem exists for: skip-open **fallback
+  chains** with fail-closed exhaustion semantics, model→tier naming, and an
+  LLM-tuned error taxonomy.
+- **[breaker_machines](https://github.com/seuros/breaker_machines)** is a
+  modern generic breaker with per-class fallbacks and a rich DSL — the
+  closest generic cousin. It still isn't LLM-aware: no model→tier naming,
+  no multi-step skip-open chains, no trippable-vs-fallback error split.
+- **ruby_llm-agents** bundles a breaker inside a full agent framework; this
+  is the extractable primitive with zero dependencies.
+
+Why not build the chain layer on stoplight? Zero hard deps, a 5-method
+store contract, and a ~200-line breaker meant vendoring beat depending.
+
+## License
+
+MIT.

data/app/controllers/ruby_llm/resilience/breakers_controller.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module Resilience
+    class BreakersController < ActionController::Base
+      # ruby_llm's Railtie registers `acronym "RubyLLM"`, which changes this
+      # controller's DERIVED controller_path to "rubyllm/resilience/breakers"
+      # in host apps — breaking template lookup (views ship under
+      # app/views/ruby_llm/...). Pin the path so lookup is deterministic
+      # regardless of the host's inflections.
+      def self.controller_path
+        "ruby_llm/resilience/breakers"
+      end
+
+      protect_from_forgery with: :exception
+      layout "ruby_llm/resilience/application"
+
+      # Deny-by-default: the configured hook must explicitly allow the
+      # request (by not rendering). The default hook 404s everything.
+      before_action :authenticate_dashboard!
+
+      def index
+        services = params[:services].presence&.split(",") ||
+                   Resilience.config.dashboard_services
+        @statuses = Breaker.dashboard_status(services: services)
+        @fallback_routes = Resilience.fallback_routes
+        @config = Resilience.config
+      end
+
+      # "configured" when the app replaced the default no-op lambda (defaults
+      # are all defined in configuration.rb — source_location tells them apart).
+      helper_method def hook_status(hook)
+        file = hook.respond_to?(:source_location) ? hook.source_location&.first : nil
+        file&.end_with?("configuration.rb") ? "default" : "configured"
+      end
+
+      def reset
+        Breaker.new(params[:id]).reset!
+        redirect_to root_path, notice: "Reset #{params[:id]}"
+      end
+
+      private
+
+      def authenticate_dashboard!
+        Resilience.config.dashboard_auth.call(self)
+      end
+    end
+  end
+end

data/app/views/layouts/ruby_llm/resilience/application.html.erb

@@ -0,0 +1,60 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <title>Circuit Breakers</title>
+  <%= csrf_meta_tags %>
+  <%= yield :head %>
+  <style>
+    :root { --bg:#fff; --ink:#1a2330; --muted:#6b7686; --line:#e5e9f0;
+            --ok:#16803c; --ok-bg:#e8f6ed; --warn:#92610c; --warn-bg:#fdf3df;
+            --bad:#b42318; --bad-bg:#fdeceb; }
+    @media (prefers-color-scheme: dark) {
+      :root { --bg:#0e1116; --ink:#e7ecf3; --muted:#8b96a5; --line:#232a33;
+              --ok-bg:#10301c; --warn-bg:#33270e; --bad-bg:#3a1512; }
+    }
+    body { margin:0; padding:2rem; background:var(--bg); color:var(--ink);
+           font:14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; }
+    h1 { font-size:18px; margin:0 0 4px; }
+    .sub { color:var(--muted); font-size:12px; margin:0 0 20px; }
+    table { border-collapse:collapse; width:100%; max-width:960px; }
+    th { text-align:left; font-size:11px; text-transform:uppercase; letter-spacing:.06em;
+         color:var(--muted); padding:8px 12px; border-bottom:1px solid var(--line); }
+    td { padding:9px 12px; border-bottom:1px solid var(--line); vertical-align:top; }
+    code { font-size:13px; }
+    .pill { display:inline-block; padding:2px 9px; border-radius:999px; font-size:12px; font-weight:600; }
+    .pill.closed    { color:var(--ok);   background:var(--ok-bg); }
+    .pill.half_open { color:var(--warn); background:var(--warn-bg); }
+    .pill.open      { color:var(--bad);  background:var(--bad-bg); }
+    .meta { color:var(--muted); font-size:12px; }
+    .notice { padding:8px 12px; margin:0 0 16px; background:var(--ok-bg); color:var(--ok);
+              border-radius:6px; max-width:960px; font-size:13px; }
+    form { display:inline; }
+    button { font:inherit; font-size:12px; padding:3px 10px; border:1px solid var(--line);
+             border-radius:6px; background:transparent; color:var(--ink); cursor:pointer; }
+    button:hover { border-color:var(--muted); }
+    .empty { color:var(--muted); padding:24px 12px; }
+    .routes div { margin:1px 0; }
+    .routes code { font-size:12px; }
+    .panel { margin-top:28px; max-width:960px; }
+    .panel h2 { font-size:13px; text-transform:uppercase; letter-spacing:.06em;
+                color:var(--muted); margin:0 0 8px; }
+    table.config th { width:140px; font-size:12px; text-transform:none;
+                      letter-spacing:0; vertical-align:top; }
+    .hook { display:inline-block; padding:1px 7px; border-radius:999px; font-size:11px; }
+    .hook.configured { color:var(--ok); background:var(--ok-bg); }
+    .hook.default { color:var(--muted); background:var(--line); }
+    #breaker-filter { display:block; margin:0 0 12px; padding:6px 10px; width:280px;
+                      font:inherit; font-size:13px; color:var(--ink); background:var(--bg);
+                      border:1px solid var(--line); border-radius:6px; }
+    #breaker-filter:focus { outline:none; border-color:var(--muted); }
+    th[data-sort] { cursor:pointer; user-select:none; }
+    th[data-sort]:hover { color:var(--ink); }
+    th.sorted-asc::after  { content:" ▲"; font-size:9px; }
+    th.sorted-desc::after { content:" ▼"; font-size:9px; }
+  </style>
+</head>
+<body>
+  <% if notice.present? %><div class="notice"><%= notice %></div><% end %>
+  <%= yield %>
+</body>
+</html>

data/app/views/ruby_llm/resilience/breakers/index.html.erb

@@ -0,0 +1,150 @@
+<% content_for :head do %>
+  <meta http-equiv="refresh" content="10">
+<% end %>
+
+<h1>Circuit Breakers</h1>
+<p class="sub">
+  All reads on this page are pure — polling never consumes probe slots.
+  Auto-refreshes every 10s.
+</p>
+
+<% if @statuses.empty? %>
+  <p class="empty">
+    No breakers registered yet in this process. Breakers appear here after
+    their first use, configure <code>dashboard_services</code> for a static
+    fleet, or pass an explicit list with <code>?services=api:a,api:b</code>.
+  </p>
+<% else %>
+  <input id="breaker-filter" type="search" placeholder="Filter by service or state…"
+         autocomplete="off" aria-label="Filter breakers">
+  <table id="breakers" data-sortable>
+    <thead>
+      <tr>
+        <th data-sort="text">Service</th><th data-sort="text">State</th>
+        <th data-sort="num">Failures</th><th data-sort="num">Probe in</th>
+        <th data-sort="num">Cooldown</th><th>Fallback routes</th><th>About</th><th></th>
+      </tr>
+    </thead>
+    <tbody>
+    <% @statuses.each do |status| %>
+      <tr data-filter="<%= status[:service] %> <%= status[:state] %>">
+        <td><code><%= status[:service] %></code></td>
+        <td data-value="<%= status[:state] %>"><span class="pill <%= status[:state] %>"><%= status[:state].to_s.tr("_", "-") %></span></td>
+        <td data-value="<%= status[:failure_count] %>">
+          <%= status[:failure_count] %> / <%= status[:failure_threshold] %>
+        </td>
+        <td data-value="<%= status[:seconds_until_probe] || -1 %>"><%= status[:seconds_until_probe] ? "#{status[:seconds_until_probe]}s" : "—" %></td>
+        <td data-value="<%= status[:cooldown_seconds] %>">
+          <%= status[:cooldown_seconds] %>s<% if status[:overridden] %> <span class="meta" title="Per-service override via config.services">•</span><% end %>
+        </td>
+        <td class="routes">
+          <% routes = @fallback_routes[status[:service]] %>
+          <% if routes.present? %>
+            <% routes.each do |route| %>
+              <div><code><%= route %></code></div>
+            <% end %>
+          <% else %>
+            <span class="meta">— none configured</span>
+          <% end %>
+        </td>
+        <td class="meta">
+          <% if status[:metadata].any? %>
+            <%= status[:metadata].map { |k, v| "#{k}: #{v}" }.join(" · ") %>
+          <% end %>
+        </td>
+        <td>
+          <%= button_to "Reset", reset_breaker_path(id: status[:service]),
+                        method: :post,
+                        data: { turbo_confirm: "Force-close #{status[:service]}?" } %>
+        </td>
+      </tr>
+    <% end %>
+    </tbody>
+  </table>
+
+  <script>
+    // Sort + filter, no dependencies. State persists in localStorage so it
+    // survives the 10s auto-refresh.
+    (function () {
+      var table = document.getElementById("breakers");
+      var filter = document.getElementById("breaker-filter");
+      if (!table || !filter) return;
+      var KEY = "rlr-dashboard-v1";
+      var state = {};
+      try { state = JSON.parse(localStorage.getItem(KEY)) || {}; } catch (e) {}
+
+      function rows() { return Array.prototype.slice.call(table.tBodies[0].rows); }
+
+      function applyFilter() {
+        var q = filter.value.trim().toLowerCase();
+        rows().forEach(function (row) {
+          var hay = (row.getAttribute("data-filter") || "").toLowerCase();
+          row.style.display = !q || hay.indexOf(q) !== -1 ? "" : "none";
+        });
+      }
+
+      function cellValue(row, idx, kind) {
+        var cell = row.cells[idx];
+        var v = cell.getAttribute("data-value");
+        if (v === null) v = cell.textContent.trim();
+        return kind === "num" ? parseFloat(v) : v.toLowerCase();
+      }
+
+      function applySort() {
+        if (state.col == null) return;
+        var ths = table.tHead.rows[0].cells;
+        var kind = ths[state.col] && ths[state.col].getAttribute("data-sort");
+        if (!kind) return;
+        var dir = state.dir === "desc" ? -1 : 1;
+        rows().sort(function (a, b) {
+          var av = cellValue(a, state.col, kind), bv = cellValue(b, state.col, kind);
+          return av < bv ? -dir : av > bv ? dir : 0;
+        }).forEach(function (row) { table.tBodies[0].appendChild(row); });
+        Array.prototype.forEach.call(ths, function (th, i) {
+          th.classList.toggle("sorted-asc", i === state.col && dir === 1);
+          th.classList.toggle("sorted-desc", i === state.col && dir === -1);
+        });
+      }
+
+      function save() { try { localStorage.setItem(KEY, JSON.stringify(state)); } catch (e) {} }
+
+      Array.prototype.forEach.call(table.tHead.rows[0].cells, function (th, i) {
+        if (!th.getAttribute("data-sort")) return;
+        th.addEventListener("click", function () {
+          state.dir = state.col === i && state.dir !== "desc" ? "desc" : "asc";
+          state.col = i;
+          save(); applySort();
+        });
+      });
+      filter.addEventListener("input", function () {
+        state.q = filter.value;
+        save(); applyFilter();
+      });
+
+      if (state.q) filter.value = state.q;
+      applyFilter(); applySort();
+    })();
+  </script>
+<% end %>
+
+<div class="panel">
+  <h2>Configuration</h2>
+  <table class="config">
+    <tr><th>Store</th><td><code><%= @config.cache_store.class.name %></code></td></tr>
+    <tr><th>Defaults</th><td><%= @config.failure_threshold %> failures / <%= @config.cooldown_seconds %>s cooldown / <%= @config.failures_window_seconds %>s window</td></tr>
+    <tr>
+      <th>Telemetry hooks</th>
+      <td>
+        <code>on_error</code> <span class="hook <%= hook_status(@config.on_error) %>"><%= hook_status(@config.on_error) %></span> ·
+        <code>on_status</code> <span class="hook <%= hook_status(@config.on_status) %>"><%= hook_status(@config.on_status) %></span> ·
+        <code>on_fallback</code> <span class="hook <%= hook_status(@config.on_fallback) %>"><%= hook_status(@config.on_fallback) %></span>
+      </td>
+    </tr>
+    <tr><th>Fallback map</th><td><%= @config.fallback_models.size %> model route<%= "s" unless @config.fallback_models.size == 1 %> · <%= @config.services.size %> per-service override<%= "s" unless @config.services.size == 1 %></td></tr>
+  </table>
+  <p class="meta">
+    Trips fire <code>on_error</code> (wire to your error tracker as a handled
+    error), state changes fire the <code>on_status</code> gauge, fallback hops
+    fire <code>on_fallback</code> — see the README's telemetry recipes.
+  </p>
+</div>

data/config/routes.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+RubyLLM::Resilience::Engine.routes.draw do
+  root to: "breakers#index"
+  # Service names contain colons ("api:anthropic:sonnet") — allow anything
+  # but a slash in the segment.
+  post "breakers/:id/reset", to: "breakers#reset", as: :reset_breaker,
+                             constraints: { id: %r{[^/]+} }
+end

data/lib/generators/resilience/install_generator.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module RubyLLM
+  module Resilience
+    # rails g resilience:install
+    #
+    # Writes a fully-commented initializer covering every configuration seam.
+    # (Namespaced inside RubyLLM::Resilience — the top-level ::Resilience
+    # constant is reserved for the opt-in shorthand alias.)
+    class InstallGenerator < Rails::Generators::Base
+      namespace "resilience:install"
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Creates config/initializers/resilience.rb with a commented example configuration"
+
+      def create_initializer
+        template "initializer.rb", "config/initializers/resilience.rb"
+      end
+
+      def show_next_steps
+        say <<~NEXT, :green
+
+          ruby_llm-resilience installed.
+
+          Next steps:
+            1. Review config/initializers/resilience.rb — the defaults work,
+               but multi-process apps should configure a shared cache_store.
+            2. Optional dashboard: mount it behind YOUR auth in config/routes.rb:
+                 mount RubyLLM::Resilience::Engine => "/resilience"
+               (it 404s everything until dashboard_auth is configured — see the
+               initializer.)
+            3. Guard a call:
+                 RubyLLM::Resilience.run("api:openai:embeddings") { RubyLLM.embed(text) }
+        NEXT
+      end
+    end
+  end
+end