sidekiq-fusebox 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: edd6428d9481a00060ac2bae28d6028d10c6339202c1e90ca47ae3a72e71bf53
+  data.tar.gz: 19e43a6860a8150defa625048e39ec41b685cdbae9a4532a4413c2f221a697e6
+SHA512:
+  metadata.gz: 6564ada11b64db6068f7e07189f7858bbde43e279e5d3c5e08222706ee17fedef867d5172e3516f409d359aaa357fbc8938c34e2a8dc266dd3ae16af0cb3c61f
+  data.tar.gz: c14f4d2d72d36e5ac8bf58986a09ad36c71da93b4fb7b8a6662d4341b6eb0ac2f1bfd57a6a276253e8f10b311386a90730e5294717d13a774bb3ee489f15f681

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+## [Unreleased]
+
+- Add a Sidekiq Web UI tab ("Fusebox") listing every known target's circuit
+  state, with a manual "Force close" action.
+- Add `Circuit.known_targets`, `Circuit.snapshot_all`, `Circuit#snapshot`,
+  and `Circuit#force_close!` to support it.
+
+## [0.1.0] - 2026-09-22
+
+- Initial release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 TODO: Write your name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,129 @@
+# Sidekiq::Fusebox
+
+A fuse box has one fuse per circuit — overload the microwave and its fuse
+blows, but the fridge keeps running. `sidekiq-fusebox` brings that to
+Sidekiq: an independent circuit breaker **per target**, so one flaky
+downstream service fails fast without starving jobs bound for unrelated
+targets.
+
+Point it at any Sidekiq worker that talks to multiple external
+systems — ERPs, payment gateways, webhooks, third-party APIs. When Brightpearl
+starts timing out, only jobs targeting Brightpearl trip; jobs for NetSuite,
+SAP, or anything else keep processing normally on the same worker threads.
+
+## Why not just retry?
+
+Sidekiq's built-in retries handle *transient* failures well. They don't
+handle *sustained* outages: if a downstream service is down for 20 minutes,
+every job aimed at it will still make a full-timeout API call, fail, and
+retry — burning worker threads the whole time, and starving unrelated jobs
+that share the same process. Fusebox adds a fast-fail layer in front of that:
+once a target has failed enough times, jobs for that target raise
+immediately, with **no external call**, until the target proves itself
+healthy again.
+
+## Installation
+
+```bash
+bundle add sidekiq-fusebox
+```
+
+## Usage
+
+Register the middleware and configure how to extract a "target" from each
+job:
+
+```ruby
+Sidekiq.configure_server do |config|
+  config.server_middleware do |chain|
+    chain.add Sidekiq::Fusebox::Middleware
+  end
+end
+
+Sidekiq::Fusebox.configure do |c|
+  # Required: how to derive the circuit's key from a job.
+  # Defaults to the worker class name if you don't set this.
+  c.extract_target = ->(_worker, job_payload) { job_payload["args"].first["integration"] }
+
+  # Only these exceptions count as a failure for the circuit. Narrow this to
+  # your integration's actual error classes — a broad StandardError rescue
+  # will also count your own bugs (e.g. a NoMethodError) as "the target is
+  # down", which is not what you want.
+  c.trip_on = [Faraday::TimeoutError, Faraday::ConnectionFailed]
+
+  c.failure_threshold = 5     # failures before the circuit opens
+  c.failure_window     = 60   # seconds; failures older than this don't count
+  c.cooldown_period     = 30  # seconds the circuit stays fully open
+  c.probe_lock_ttl       = 30 # safety TTL on the single half-open probe
+
+  c.on_open   = ->(target) { Rails.logger.warn("fusebox: circuit open for #{target}") }
+  c.on_close  = ->(target) { Rails.logger.info("fusebox: circuit closed for #{target}") }
+  c.on_reject = ->(target, _job) { StatsD.increment("fusebox.rejected", tags: ["target:#{target}"]) }
+end
+```
+
+```ruby
+class IntegrationSyncWorker
+  include Sidekiq::Job
+
+  def perform(order_id, integration:)
+    Integrations.for(integration).sync_order(order_id)
+  end
+end
+```
+
+## How it works
+
+Each target moves through three states, tracked in Redis:
+
+- **closed** — healthy. Jobs run normally.
+- **open** — `failure_threshold` failures happened within `failure_window`.
+  Every job for this target raises `Sidekiq::Fusebox::CircuitOpenError`
+  immediately, without being called, for `cooldown_period` seconds.
+- **half-open** — the cooldown has elapsed. Exactly one job is let through as
+  a probe (via a Redis `SET NX` lock, so concurrent workers don't all hammer
+  the target at once). If it succeeds, the circuit closes. If it fails, the
+  circuit reopens for a fresh `cooldown_period`.
+
+Because rejection raises `CircuitOpenError`, Sidekiq's own retry mechanism
+schedules the retry with its normal backoff — Fusebox doesn't re-implement
+scheduling. **Trade-off:** a rejection still consumes one of the job's
+configured retry attempts. If you run targets with tight `retry:` counts,
+account for open-circuit rejections when choosing that number, or rescue
+`CircuitOpenError` specifically in a `sidekiq_retry_in` block to control its
+backoff independently of your other failure types.
+
+## Web UI
+
+Fusebox adds a "Fusebox" tab to the Sidekiq Web UI listing every target
+that's ever failed, its current state, failure count, and (while open) how
+long until the next half-open probe — with a "Force close" button for manual
+overrides. It's not loaded by `require "sidekiq/fusebox"`, since it pulls in
+`sidekiq/web`; require it wherever you mount `Sidekiq::Web`:
+
+```ruby
+# config.ru, or config/routes.rb in Rails
+require "sidekiq/web"
+require "sidekiq/fusebox/web"
+
+run Sidekiq::Web # or: mount Sidekiq::Web => "/sidekiq"
+```
+
+## Development
+
+```bash
+bin/setup
+bundle exec rake     # runs rspec + rubocop
+```
+
+The spec suite stubs `Sidekiq.redis` with an in-memory fake
+(`spec/support/fake_redis.rb`), so it doesn't need a real Redis server.
+
+## Contributing
+
+Bug reports and pull requests are welcome at
+https://github.com/bhawsartanmay/sidekiq-fusebox.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/sidekiq/fusebox/circuit.rb

@@ -0,0 +1,129 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Fusebox
+    # A single per-target circuit, backed by three Redis keys namespaced
+    # under a hash tag so they always land on the same Redis Cluster slot,
+    # plus a global set that remembers every target that has ever failed
+    # (used by the Web UI to know what to list).
+    #
+    #   fusebox:{target}:failures    - rolling failure counter
+    #   fusebox:{target}:opened_at   - float timestamp, present only while open/half-open
+    #   fusebox:{target}:probe_lock  - claims the single half-open probe attempt
+    #   fusebox:targets              - set of every target that has ever failed
+    class Circuit
+      TARGETS_KEY = "fusebox:targets"
+
+      Snapshot = Struct.new(:target, :state, :failure_count, :opened_at, :cooldown_remaining, :probing,
+                            keyword_init: true)
+
+      class << self
+        def known_targets(config)
+          with_redis(config) { |conn| conn.call("SMEMBERS", TARGETS_KEY) }.sort
+        end
+
+        def snapshot_all(config)
+          known_targets(config).map { |target| new(target, config).snapshot }
+        end
+
+        def with_redis(config, &block)
+          config.redis_pool ? config.redis_pool.with(&block) : Sidekiq.redis(&block)
+        end
+      end
+
+      def initialize(target, config)
+        @target = target
+        @config = config
+      end
+
+      # true  -> caller may proceed (circuit closed, or this job won the probe)
+      # false -> caller must not proceed (circuit open, or another job already probing)
+      def allow_request?
+        case state
+        when :closed then true
+        when :open then false
+        when :half_open then acquire_probe_lock
+        end
+      end
+
+      def record_success
+        removed = redis { |conn| conn.call("DEL", failures_key, opened_at_key, probe_lock_key) }
+        @config.on_close&.call(@target) if removed.to_i.positive?
+      end
+      alias force_close! record_success
+
+      def record_failure
+        redis { |conn| conn.call("SADD", TARGETS_KEY, @target) }
+        count = redis { |conn| conn.call("INCR", failures_key) }
+        redis { |conn| conn.call("EXPIRE", failures_key, @config.failure_window) }
+        return if count < @config.failure_threshold
+
+        open_circuit!
+      end
+
+      def state
+        derive_state(redis { |conn| conn.call("GET", opened_at_key) })
+      end
+
+      # A read-only view of this circuit for display purposes (e.g. the Web UI).
+      def snapshot
+        opened_at = redis { |conn| conn.call("GET", opened_at_key) }
+        current_state = derive_state(opened_at)
+
+        Snapshot.new(
+          target: @target,
+          state: current_state,
+          failure_count: failure_count,
+          opened_at: opened_at ? Time.at(opened_at.to_f) : nil,
+          cooldown_remaining: cooldown_remaining(current_state, opened_at),
+          probing: probing?
+        )
+      end
+
+      private
+
+      def derive_state(opened_at)
+        return :closed if opened_at.nil?
+
+        (Time.now.to_f - opened_at.to_f) < @config.cooldown_period ? :open : :half_open
+      end
+
+      def cooldown_remaining(current_state, opened_at)
+        return 0 unless current_state == :open
+
+        elapsed = Time.now.to_f - opened_at.to_f
+        (@config.cooldown_period - elapsed).round
+      end
+
+      def failure_count
+        (redis { |conn| conn.call("GET", failures_key) } || 0).to_i
+      end
+
+      def probing?
+        !redis { |conn| conn.call("GET", probe_lock_key) }.nil?
+      end
+
+      # The PX TTL is a safety net so the key never lingers forever if
+      # record_success is never reached (worker killed mid-job, deploy, etc).
+      def open_circuit!
+        redis do |conn|
+          conn.call("SET", opened_at_key, Time.now.to_f.to_s, "PX", millis(@config.cooldown_period * 2))
+          conn.call("DEL", probe_lock_key)
+        end
+        @config.on_open&.call(@target)
+      end
+
+      def acquire_probe_lock
+        redis { |conn| conn.call("SET", probe_lock_key, "1", "NX", "PX", millis(@config.probe_lock_ttl)) } == "OK"
+      end
+
+      def millis(seconds) = (seconds * 1000).round
+
+      def redis(&block) = self.class.with_redis(@config, &block)
+
+      def failures_key = "fusebox:{#{@target}}:failures"
+      def opened_at_key = "fusebox:{#{@target}}:opened_at"
+      def probe_lock_key = "fusebox:{#{@target}}:probe_lock"
+    end
+  end
+end

data/lib/sidekiq/fusebox/configuration.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Fusebox
+    # Holds the global defaults for every circuit. Override via
+    # Sidekiq::Fusebox.configure.
+    class Configuration
+      attr_accessor :failure_threshold, :failure_window, :cooldown_period,
+                    :probe_lock_ttl, :extract_target, :trip_on,
+                    :on_open, :on_close, :on_reject, :redis_pool
+
+      def initialize
+        @failure_threshold = 5
+        @failure_window = 60
+        @cooldown_period = 30
+        @probe_lock_ttl = 30
+        @trip_on = [StandardError]
+        @extract_target = ->(_worker, job_payload) { job_payload["class"] }
+        @on_open = nil
+        @on_close = nil
+        @on_reject = nil
+        @redis_pool = nil
+      end
+    end
+  end
+end

data/lib/sidekiq/fusebox/middleware.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Fusebox
+    # Sidekiq server middleware: skips the job (fails fast, no external call)
+    # while the target's circuit is open, and feeds success/failure back into
+    # that target's circuit otherwise.
+    class Middleware
+      include Sidekiq::ServerMiddleware if defined?(Sidekiq::ServerMiddleware)
+
+      def call(job_instance, job_payload, _queue, &block)
+        config = Sidekiq::Fusebox.configuration
+        target = config.extract_target.call(job_instance, job_payload)
+        return block.call if target.nil?
+
+        circuit = Circuit.new(target, config)
+        reject!(config, target, job_payload) unless circuit.allow_request?
+
+        run(circuit, config, &block)
+      end
+
+      private
+
+      def reject!(config, target, job_payload)
+        config.on_reject&.call(target, job_payload)
+        raise CircuitOpenError, "sidekiq-fusebox: circuit open for target=#{target.inspect}"
+      end
+
+      def run(circuit, config)
+        result = yield
+        circuit.record_success
+        result
+      rescue *config.trip_on => e
+        circuit.record_failure
+        raise e
+      end
+    end
+  end
+end

data/lib/sidekiq/fusebox/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Fusebox
+    VERSION = "0.1.0"
+  end
+end

data/lib/sidekiq/fusebox/web/views/index.html.erb

@@ -0,0 +1,48 @@
+<section>
+  <header>
+    <h1>Fusebox Circuits</h1>
+  </header>
+
+  <div class="table_container">
+    <table>
+      <thead>
+        <tr>
+          <th>Target</th>
+          <th>State</th>
+          <th>Failures</th>
+          <th>Opened At</th>
+          <th>Cooldown Left</th>
+          <th>Probe</th>
+          <th>Actions</th>
+        </tr>
+      </thead>
+      <tbody>
+        <% if @targets.empty? %>
+          <tr>
+            <td colspan="7">No targets have failed yet.</td>
+          </tr>
+        <% end %>
+        <% @targets.each do |t| %>
+          <tr>
+            <td><%= h t.target %></td>
+            <td>
+              <% label_class = {open: "label-danger", half_open: "label-warning", closed: "label-success"}.fetch(t.state) %>
+              <span class="label <%= label_class %>"><%= t.state.to_s.upcase.tr("_", "-") %></span>
+            </td>
+            <td class="num"><%= t.failure_count %></td>
+            <td><%= t.opened_at ? relative_time(t.opened_at) : "-" %></td>
+            <td class="num"><%= t.state == :open ? "#{t.cooldown_remaining}s" : "-" %></td>
+            <td><%= t.probing ? "probing now" : "-" %></td>
+            <td>
+              <form action="<%= root_path %>fusebox/<%= CGI.escape(t.target) %>/close" method="post">
+                <%= csrf_tag %>
+                <input class="btn btn-danger" type="submit" value="Force close"
+                  data-confirm="Force-close the circuit for <%= h t.target %>?">
+              </form>
+            </td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+</section>

data/lib/sidekiq/fusebox/web.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+require "sidekiq/web"
+require "sidekiq/fusebox"
+
+module Sidekiq
+  module Fusebox
+    # Adds a "Fusebox" tab to the Sidekiq Web UI listing every known target's
+    # circuit state, with a manual "Force close" action for ops overrides.
+    #
+    # Not required by "sidekiq/fusebox" itself. Wherever you mount Sidekiq::Web
+    # (typically config.ru, or config/routes.rb in Rails):
+    #
+    #   require "sidekiq/web"
+    #   require "sidekiq/fusebox/web"
+    #
+    module Web
+      VIEWS = File.expand_path("web/views", __dir__)
+
+      def self.registered(app)
+        app.get "/fusebox" do
+          @targets = Sidekiq::Fusebox::Circuit.snapshot_all(Sidekiq::Fusebox.configuration)
+          erb(File.read(File.join(VIEWS, "index.html.erb")))
+        end
+
+        app.post "/fusebox/:target/close" do
+          target = route_params(:target)
+          Sidekiq::Fusebox::Circuit.new(target, Sidekiq::Fusebox.configuration).force_close!
+          redirect "#{root_path}fusebox"
+        end
+      end
+    end
+  end
+end
+
+if Sidekiq::Web.respond_to?(:configure)
+  Sidekiq::Web.configure do |cfg|
+    cfg.register(Sidekiq::Fusebox::Web, name: "fusebox", tab: "Fusebox", index: "fusebox")
+  end
+else
+  # Older Sidekiq (< 8.0) had no Config#register; wire the tab up directly.
+  Sidekiq::Fusebox::Web.registered(Sidekiq::Web::Application)
+  Sidekiq::Web.tabs["Fusebox"] = "fusebox"
+end

data/lib/sidekiq/fusebox.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+
+require_relative "fusebox/version"
+require_relative "fusebox/configuration"
+require_relative "fusebox/circuit"
+require_relative "fusebox/middleware"
+
+module Sidekiq
+  # A Sidekiq server middleware that trips an independent circuit breaker per
+  # target (extracted from job args), so one flaky downstream service fails
+  # fast without starving jobs bound for unrelated targets. See README.md.
+  module Fusebox
+    class Error < StandardError; end
+    class CircuitOpenError < Error; end
+
+    class << self
+      def configuration
+        @configuration ||= Configuration.new
+      end
+
+      def configure
+        yield(configuration)
+      end
+    end
+  end
+end

data/sig/sidekiq/fusebox.rbs

@@ -0,0 +1,6 @@
+module Sidekiq
+  module Fusebox
+    VERSION: String
+    # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+  end
+end

metadata

@@ -0,0 +1,78 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-fusebox
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Tanmay Bhawsar
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9'
+description: A Sidekiq server middleware that trips an independent circuit breaker
+  per target (extracted from job args, e.g. an integration or hostname), so one flaky
+  downstream service fails fast without starving jobs bound for unrelated targets.
+email:
+- tanmay.bhawsar@joshsoftware.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/sidekiq/fusebox.rb
+- lib/sidekiq/fusebox/circuit.rb
+- lib/sidekiq/fusebox/configuration.rb
+- lib/sidekiq/fusebox/middleware.rb
+- lib/sidekiq/fusebox/version.rb
+- lib/sidekiq/fusebox/web.rb
+- lib/sidekiq/fusebox/web/views/index.html.erb
+- sig/sidekiq/fusebox.rbs
+homepage: https://github.com/bhawsartanmay/sidekiq-fusebox
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/bhawsartanmay/sidekiq-fusebox
+  source_code_uri: https://github.com/bhawsartanmay/sidekiq-fusebox.git
+  changelog_uri: https://github.com/bhawsartanmay/sidekiq-fusebox/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/bhawsartanmay/sidekiq-fusebox/issues
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.10
+specification_version: 4
+summary: Per-target circuit breaker middleware for Sidekiq.
+test_files: []