cogworker 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2a2d6b5721f03a71d8a74d376311c047175b71f950d51f227c60598dad74dce2
-  data.tar.gz: d6f10624306849c6e32e8d7fa0211db6c2b947d78d53fec10b79039bfec57645
+  metadata.gz: f0b88ae2031cced0f33c8de66892dbd952e7aba6ed64ad1fba49719bb335698d
+  data.tar.gz: 6cd9c81f131107a8b04bb170dbedb29898899d248b900bfcf4fcbeebf69463bf
 SHA512:
-  metadata.gz: 97347577fddf8a44cc0c5df6b6aafd5a60c285f88f58f7c09193f8a1031d5bf9dc0381e2708a9d591ff81dde7781b31be6a72f7d7567b9f2a0a8b74fb0de2658
-  data.tar.gz: 84ea3a607e75d78b23dab9919c33595ade3ac2382c2c253896a6e3b4df7664d8369fbd1b81a387eaaa9ae95fe3867dd8739ce7d757a0e576047f51321e78cd5e
+  metadata.gz: 867b3c3cc51e6ee56f2fa8e7433b52598b366873ba9609ac9cfa81904bb1e9f0fdb0c35b6816bee0a7605c6536c0a88749e77f972e8d611c7fd3df726772b5fd
+  data.tar.gz: 1bf378d2c4e52a6034d47c2239a21539ae722e3877c015585ef446cfff6c0456c893c071e088b9c857759ac04849a7a481f804b816dafea85dc5e0fce46fe59a

data/README.md

@@ -266,8 +266,99 @@ real worker would receive Strings instead of Symbols.
 ### Retries and Dead
 
 A failed job moves to the Retry set with a growing delay
-(`count**4 + 15 + jitter` seconds). When it runs out of attempts, it moves to
-Dead. You can put it back on its queue by hand from the Web UI.
+(`count**4 + 15 + jitter` seconds, `count` being the retry about to happen,
+1 for the first). `retry:` sets the budget: `true` or unset gives 25
+retries, an integer that many, `false` or `0` none. When it runs out of
+retries, it moves to Dead. You can put it back on its queue by hand from the
+Web UI.
+
+#### Custom delay: `cogworker_retry_in`
+
+```ruby
+class SendEventJob
+  include Cogworker::Worker
+  cogworker_options retry: 10
+
+  cogworker_retry_in do |count, exception, job|
+    next :kill if exception.is_a?(PermanentError)
+
+    60 * (2**count)
+  end
+end
+```
+
+`count` is the number of retries made before this failure (0 on the first
+one), `exception` what the attempt raised, `job` a copy of the job hash (the
+block may take fewer parameters). It returns:
+
+| Value | Effect |
+|---|---|
+| Integer / Float ≥ 0 | retry after that many seconds, no jitter added; 0 means on the next poll |
+| `nil` | the default delay |
+| `:kill` | straight to Dead, retries left or not; the death hooks run |
+| `:discard` | dropped: neither Retry nor Dead, no death hooks; counted as failed, its unique locks released |
+| anything else | the default delay, with a warning in the log |
+
+If the block raises, the default delay is used and the error is logged. It
+isn't called for a failure that is the last one anyway (no retry left, or
+`retry: false`), nor for a job whose class can't be loaded. It's asked as
+soon as `perform` raises, before any server middleware sees the error — so
+it also runs when a middleware then swallows that error; keep it free of
+side effects. A delay is never capped (one over a year is logged as a
+likely mistake), and the job's `until_executed` lock is extended by it, so
+the lock never runs out before the retry — however long that is. If
+recording the failure itself fails (Redis trouble), the job is retried
+after a fixed `30 s × n` instead, whatever the hook said. The hook is
+inherited, so a base class of your own (`class ApplicationJob`) is the place
+for an app-wide policy. The Web UI's retry timeline shows `:kill` and
+`:discard` as such.
+
+#### Death hooks: `cogworker_retries_exhausted` and `death_handlers`
+
+```ruby
+class SendEventJob
+  include Cogworker::Worker
+
+  cogworker_retries_exhausted do |job, exception|
+    Metrics.increment('events.lost')
+  end
+end
+
+Cogworker.configure_server do |config|
+  config.death_handlers << ->(job, exception) { ErrorTracker.notify(exception, job) }
+end
+```
+
+When a job lands in Dead for good, its class's `cogworker_retries_exhausted`
+runs first, then every `death_handlers` entry in order. `job` is the entry
+as stored in Dead (with `error_class`, `error_message`, `retry_count`,
+`failed_at`); each hook gets its own copy, so changing it changes nothing.
+
+| How the job died | Class hook | `death_handlers` |
+|---|---|---|
+| out of retries, or `retry: false` / `0` | yes | yes |
+| `cogworker_retry_in` returned `:kill` | yes | yes |
+| its process kept dying while running it (`config.max_orphanings`) | yes, with a `Cogworker::JobOrphanedError` | yes |
+| its failure was filed later, by a reconcile pass | yes, with a `Cogworker::JobFailedError` carrying the original message (the original class is in `job['error_class']`) | yes |
+| its class can't be loaded (renamed, removed) | no | yes |
+| `cogworker_retry_in` returned `:discard` | no | no |
+| an unparseable payload | no | no |
+| deleted or retried from the Web UI | no | no |
+
+The hooks run only after the job is safely in Dead, at most once per death:
+on the reconcile paths, only the process that actually filed it runs them.
+They run synchronously, on the processor thread (once the job is
+acknowledged, before the thread takes the next one) or on the reconcile
+thread, with no timeout of their own — **keep them quick**. One that raises
+is logged and doesn't stop the others, nor the job's ack or lock release.
+
+A hook can be skipped or cut short, never repeated: a process that crashes
+right after the write never runs it, and on shutdown a hook still running
+when the drain timeout (25 s) is up may be cut off as the process exits.
+The job itself is safe either way — already in Dead and acknowledged, so it
+won't run again. `Testing.inline!` raises the job's error to the caller as before
+and runs no hooks; test a hook directly through
+`SomeJob.cogworker_retry_in_block` / `SomeJob.cogworker_retries_exhausted_block`.
 
 ### Unique jobs
 

data/lib/cogworker/config.rb

@@ -9,7 +9,7 @@ module Cogworker
     FETCH_MODES = %i[reliable basic].freeze
 
     attr_reader :server_chain, :client_chain, :periodic_manager, :redis_options, :fetch, :fetch_idle_max_interval,
-                :orphan_threshold, :max_orphanings, :periodic_replace
+                :orphan_threshold, :max_orphanings, :periodic_replace, :death_handlers
     attr_accessor :concurrency, :queues, :periodic_catch_up, :unique_lock_ttl
 
     def initialize
@@ -57,6 +57,10 @@ module Cogworker
       # that crashed is counted, so keep this above the number of crashes an
       # innocent job might plausibly sit through.
       @max_orphanings = 3
+      # Callables `(job, exception)` run, after the job class's own
+      # `cogworker_retries_exhausted`, whenever a job lands in dead for good
+      # (DeathNotifier). `config.death_handlers << ->(job, e) { ... }`.
+      @death_handlers = []
       register_default_middleware
     end
 

data/lib/cogworker/death_notifier.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Cogworker
+  # Runs the death hooks for a job that has just been written to dead for
+  # good: the job class's own `cogworker_retries_exhausted`, then every
+  # `config.death_handlers` entry, in order. Called only by whoever's write
+  # actually put it there (so at most once per death), after that write —
+  # and, from a processor, only once the job is acknowledged, so a slow
+  # hook can't leave it on the in-progress list to be requeued and re-run.
+  #
+  # Each hook gets its own copy of the job (as stored in dead) and runs on
+  # its own: one that raises is logged and never stops the others, nor the
+  # ack / lock releases / reconcile pass around the call. Synchronous, with
+  # no timeout — a hook should be quick.
+  module DeathNotifier
+    module_function
+
+    def notify(job, exception)
+      hook = job_class(job['class'])&.cogworker_retries_exhausted_block
+      run('cogworker_retries_exhausted', job) { |copy| JobUtil.call_hook(hook, copy, exception) } if hook
+      Cogworker.config.death_handlers.each do |handler|
+        run('death handler', job) { |copy| JobUtil.call_hook(handler, copy, exception) }
+      end
+    end
+
+    # A dead entry filed from its stored record (no exception object left).
+    def notify_failed(payload)
+      job = parse(payload)
+      notify(job, JobFailedError.new(job['error_message'].to_s)) if job
+    end
+
+    def notify_orphaned(job)
+      notify(job, JobOrphanedError.new("orphaned more than #{Cogworker.config.max_orphanings} times"))
+    end
+
+    def run(what, job)
+      yield JobUtil.deep_copy(job)
+    rescue Exception => e # rubocop:disable Lint/RescueException -- user code, isolated like `perform`
+      Cogworker.logger.error do
+        "#{what} failed for jid=#{job['jid']} (ignored): #{e.class}: #{JobUtil.error_message(e)}"
+      end
+    end
+
+    # nil for a class that can't be loaded (renamed, removed, never defined)
+    # or doesn't use the job DSL: only `death_handlers` run then.
+    def job_class(name)
+      return nil unless name.is_a?(String) && !name.empty?
+
+      klass = Object.const_get(name)
+      klass.respond_to?(:cogworker_retries_exhausted_block) ? klass : nil
+    rescue StandardError
+      nil
+    end
+
+    def parse(payload)
+      job = JSON.parse(payload)
+      job.is_a?(Hash) ? job : nil
+    rescue JSON::ParserError, TypeError
+      nil
+    end
+  end
+end

data/lib/cogworker/failure_decision.rb

@@ -0,0 +1,131 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # What becomes of a failed attempt: `retry` (after `delay` seconds — nil
+  # for the default backoff), `dead` or `discard`. Made once per failure:
+  # by the job class's `cogworker_retry_in` hook where it has one, otherwise
+  # by the retry budget and the default backoff.
+  #
+  # `Processor` makes it inside the middleware chain, the moment `perform`
+  # raises, and (when a hook took part) stamps it on the job as
+  # `failure_outcome` so `JobUtil.terminal_failure?` — and with it Status,
+  # UniqueJobs and Periodic middleware — gives the same answer as the
+  # retry/dead write. The stamp never reaches a stored entry.
+  class FailureDecision
+    OUTCOMES = %w[retry dead discard].freeze
+    # Used as given, never capped — but a delay past this is most likely a
+    # unit mix-up (ms for s), and it keeps an `until_executed` lock that long.
+    LONG_DELAY = 365 * 24 * 60 * 60
+
+    attr_reader :outcome, :delay
+
+    class << self
+      # `for`, stamped on the job as `failure_outcome` when the class's hook
+      # took part (see the class comment). nil if deciding itself failed.
+      def decide!(job, error, klass)
+        decision = self.for(job, error, klass)
+        job['failure_outcome'] = decision.outcome if decision.hooked?
+        decision
+      rescue StandardError => e
+        Cogworker.logger.error { "couldn't decide the fate of jid=#{job['jid']}: #{e.class}: #{e.message}" }
+        nil
+      end
+
+      # `klass` — the job's class, or nil when it couldn't be resolved (no
+      # hook then). The hook isn't asked about a failure that is the last
+      # one anyway: there's no retry to time.
+      def for(job, error, klass)
+        count = job['retry_count'].to_i
+        return new('dead') if count + 1 > JobUtil.max_retries(job)
+
+        hook = klass.respond_to?(:cogworker_retry_in_block) ? klass.cogworker_retry_in_block : nil
+        return new('retry') unless hook
+
+        from_hook(hook, job, error, count)
+      end
+
+      # Grows with the retry count, with jitter to avoid a thundering herd
+      # of retries all landing on the same second.
+      def default_delay(count)
+        (count**4) + 15 + (rand(30) * (count + 1))
+      end
+
+      private
+
+      def from_hook(hook, job, error, count)
+        value = JobUtil.call_hook(hook, count, error, JobUtil.deep_copy(job))
+        case value
+        when nil then new('retry', hooked: true)
+        when :kill then new('dead', killed: true, hooked: true)
+        when :discard then new('discard', hooked: true)
+        else
+          if valid_delay?(value)
+            warn_long_delay(job, value) if value > LONG_DELAY
+            return new('retry', value, hooked: true)
+          end
+
+          Cogworker.logger.warn do
+            "cogworker_retry_in of #{job['class']} returned #{value.inspect} for jid=#{job['jid']}; " \
+              'using the default backoff'
+          end
+          new('retry', hooked: true)
+        end
+      rescue Exception => e # rubocop:disable Lint/RescueException -- user code, isolated like `perform`
+        Cogworker.logger.error do
+          "cogworker_retry_in of #{job['class']} raised for jid=#{job['jid']} (using the default backoff): " \
+            "#{e.class}: #{JobUtil.error_message(e)}"
+        end
+        new('retry', hooked: true)
+      end
+
+      def warn_long_delay(job, value)
+        Cogworker.logger.warn do
+          "cogworker_retry_in of #{job['class']} returned #{value}s (over a year) for jid=#{job['jid']}; " \
+            'using it as given'
+        end
+      end
+
+      def valid_delay?(value)
+        (value.is_a?(Integer) || (value.is_a?(Float) && value.finite?)) && value >= 0
+      end
+    end
+
+    def initialize(outcome, delay = nil, killed: false, hooked: false)
+      @outcome = outcome
+      @delay = delay
+      @killed = killed
+      @hooked = hooked
+    end
+
+    def retry?
+      outcome == 'retry'
+    end
+
+    def dead?
+      outcome == 'dead'
+    end
+
+    def discard?
+      outcome == 'discard'
+    end
+
+    # Whether the job class's hook took part (only then is the decision
+    # stamped on the job: without one, `terminal_failure?` already agrees).
+    def hooked?
+      @hooked
+    end
+
+    # Ended before its retries ran out: `:kill`, or `:discard`.
+    def cut_short?
+      @killed || discard?
+    end
+
+    # The outcome as the attempts log (and the Web UI's timeline) shows it.
+    def attempt_outcome
+      return 'retrying' if retry?
+      return 'discarded' if discard?
+
+      @killed ? 'killed' : 'dead'
+    end
+  end
+end

data/lib/cogworker/job.rb

@@ -30,6 +30,32 @@ module Cogworker
                                     end
       end
 
+      # `cogworker_retry_in { |count, exception, job| ... }` — the pause
+      # (seconds) before the next attempt, or `:kill` (straight to dead) /
+      # `:discard` (dropped), or `nil` for the default backoff. See
+      # `FailureDecision`. Kept on the class, never in the job payload.
+      def cogworker_retry_in(&block)
+        raise ArgumentError, 'cogworker_retry_in needs a block' unless block
+
+        @cogworker_retry_in_block = block
+      end
+
+      # `cogworker_retries_exhausted { |job, exception| ... }` — called once
+      # the job is in dead for good (see `DeathNotifier`).
+      def cogworker_retries_exhausted(&block)
+        raise ArgumentError, 'cogworker_retries_exhausted needs a block' unless block
+
+        @cogworker_retries_exhausted_block = block
+      end
+
+      def cogworker_retry_in_block
+        inherited_hook(:@cogworker_retry_in_block, :cogworker_retry_in_block)
+      end
+
+      def cogworker_retries_exhausted_block
+        inherited_hook(:@cogworker_retries_exhausted_block, :cogworker_retries_exhausted_block)
+      end
+
       def perform_async(*args)
         Client.push(job_payload('args' => args))
       end
@@ -61,6 +87,14 @@ module Cogworker
 
       private
 
+      # Looked up at call time (not copied at definition), so a subclass
+      # sees a hook its parent defines later too.
+      def inherited_hook(ivar, reader)
+        return instance_variable_get(ivar) if instance_variable_defined?(ivar)
+
+        superclass.respond_to?(reader) ? superclass.public_send(reader) : nil
+      end
+
       def job_payload(extra)
         cogworker_options_hash.transform_keys(&:to_s).merge('class' => name).merge(extra)
       end

data/lib/cogworker/job_failed_error.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # Handed to death hooks (DeathNotifier) for a failure filed in dead later,
+  # from its stored record, when the original exception object is long gone.
+  # The message is the original one; its class is in `job['error_class']`.
+  class JobFailedError < StandardError
+  end
+end

data/lib/cogworker/job_orphaned_error.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # Handed to death hooks (DeathNotifier) for a job buried because its
+  # process kept dying while running it — there's no exception of its own.
+  class JobOrphanedError < StandardError
+  end
+end

data/lib/cogworker/job_util.rb

@@ -61,10 +61,46 @@ module Cogworker
     # `Periodic::ReleaseMiddleware` (releases the running-lock only on the
     # terminal attempt) — previously three separate, subtly different copies
     # of this same check, two of which had the `max_retries` bug above.
+    #
+    # A job class's `cogworker_retry_in` can end a job early (`:kill`,
+    # `:discard`): `Processor` stamps that decision on the job as
+    # `failure_outcome` (FailureDecision) before the middleware sees the
+    # failure, and it wins here when present.
     def terminal_failure?(job)
+      outcome = job['failure_outcome']
+      return outcome != 'retry' if FailureDecision::OUTCOMES.include?(outcome)
+
       job['retry_count'].to_i >= max_retries(job)
     end
 
+    # A copy of a job hash to hand to user code, so changes it makes can't
+    # leak back. Never raises: unlike a JSON round trip, it copes with
+    # whatever middleware may have put in the hash (the job being on its
+    # way to `interrupt` precisely because that doesn't serialize).
+    def deep_copy(value)
+      case value
+      when Hash then value.to_h { |k, v| [deep_copy(k), deep_copy(v)] }
+      when Array then value.map { |v| deep_copy(v) }
+      when String then value.dup
+      else value
+      end
+    end
+
+    # Calls a user-supplied hook (a block, lambda, Method or any `#call`
+    # object) with as many of `args` as it takes: a block with fewer
+    # parameters ignores the rest anyway, a lambda or method would raise.
+    # Counted from `parameters` (required + optional), not `arity`, which
+    # is negative as soon as there's an optional one.
+    def call_hook(hook, *args)
+      callable = hook.is_a?(Proc) || hook.is_a?(Method) ? hook : hook.method(:call)
+      return hook.call(*args) if callable.is_a?(Proc) && !callable.lambda?
+
+      params = callable.parameters
+      return hook.call(*args) if params.any? { |type, _| type == :rest }
+
+      hook.call(*args.first(params.count { |type, _| %i[req opt].include?(type) }))
+    end
+
     # The job Hash a raw `schedule`/`retry`/`dead` entry would requeue as —
     # or `nil` if it can't be: not JSON, not an object, or no String
     # `queue` to push it onto (e.g. an `unparseable_job` wrapper, whose

data/lib/cogworker/processor.rb

@@ -90,6 +90,8 @@ module Cogworker
       @busy = true
       @ran = nil
       @performed = false
+      @decision = nil
+      @death = nil
       disposition = nil
       begin
         disposition = execute(work)
@@ -104,6 +106,10 @@ module Cogworker
         # can't re-queue a job that already ran.
         settle(work, disposition || @ran || :give_back)
         @manager.job_finished(work.raw_job)
+        # Only now, with the job acknowledged (or its ack remembered): a
+        # slow hook used to hold it on the in-progress list, where a
+        # shutdown or orphan recovery requeued it and ran it again.
+        DeathNotifier.notify(*@death) if @death
       end
     end
 
@@ -139,9 +145,10 @@ module Cogworker
       payload.merge!('interrupted_count' => count, 'retry_count' => attempt,
                      'error_class' => error.class.name, 'error_message' => JobUtil.error_message(error),
                      'failed_at' => Time.now.to_f)
-      # No retries left (`retry: false`/`0`, or the last one used up) means
-      # no extra attempt either — straight to dead, like any terminal failure.
-      out_of_retries = attempt > JobUtil.max_retries(payload)
+      # No retries left (`retry: false`/`0`, or the last one used up, or the
+      # job's `cogworker_retry_in` ended it) means no extra attempt either —
+      # straight to dead, like any terminal failure.
+      out_of_retries = attempt > JobUtil.max_retries(payload) || @decision&.cut_short?
       delay = INTERRUPT_DELAY * count
       set, score = if out_of_retries || count > MAX_INTERRUPTS
                      [RedisKeys::DEAD, Time.now.to_f]
@@ -150,25 +157,15 @@ module Cogworker
                    end
       encoded = JSON.generate(payload)
       begin
-        fetcher.interrupt(work, set, score, encoded)
+        written = fetcher.interrupt(work, set, score, encoded) != 0 # (0: no longer ours to file)
       rescue StandardError
         # Remembered with its payload: this process's reconcile (and its
         # shutdown) keep trying to file it — never re-run it as a stray.
         @manager.settle_later(work.raw_job, [set, score, encoded]) if fetcher.is_a?(ReliableFetch)
         raise
       end
-      if set == RedisKeys::RETRY
-        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
-      else
-        best_effort('lock release') { release_locks(payload) }
-      end
-      reason = if set == RedisKeys::RETRY
-                 "retrying in #{delay}s (interruption #{count} of at most #{MAX_INTERRUPTS})"
-               elsif out_of_retries
-                 'moved to dead: no retries left'
-               else
-                 "moved to dead: interrupted more than #{MAX_INTERRUPTS} times"
-               end
+      after_interrupt(job, payload, error, set, delay, written)
+      reason = interrupt_reason(set, out_of_retries, count, delay)
       Cogworker.logger.error { "couldn't record the failure of jid=#{payload['jid']} (#{error.class}); #{reason}" }
     rescue StandardError => e
       # Whatever failed (even before anything was written), remember the job
@@ -181,6 +178,21 @@ module Cogworker
       end
     end
 
+    def after_interrupt(job, payload, error, set, delay, written)
+      if set == RedisKeys::RETRY
+        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
+      else
+        best_effort('lock release') { release_locks(payload) }
+        @death = [payload, error] if written # run by `process_one`, once settled
+      end
+    end
+
+    def interrupt_reason(set, out_of_retries, count, delay)
+      return "retrying in #{delay}s (interruption #{count} of at most #{MAX_INTERRUPTS})" if set == RedisKeys::RETRY
+
+      out_of_retries ? 'moved to dead: no retries left' : "moved to dead: interrupted more than #{MAX_INTERRUPTS} times"
+    end
+
     def remember_for_dead(work, error)
       # The original job's own fields where readable (so filing it releases
       # its locks); just enough to show in Dead otherwise.
@@ -272,10 +284,18 @@ module Cogworker
         nil
       end
 
+      @job_class = worker&.class
       Cogworker.config.server_chain.invoke(worker, job, work.queue) do
         raise resolution_error if resolution_error
 
-        worker.perform(*job['args'])
+        begin
+          worker.perform(*job['args'])
+        rescue Exception => e # rubocop:disable Lint/RescueException
+          # Retry, dead or discard — decided here, while the middleware is
+          # still to see the failure (FailureDecision.decide!).
+          @decision = FailureDecision.decide!(job, e, @job_class)
+          raise
+        end
       end
       nil
     rescue Exception => e # rubocop:disable Lint/RescueException
@@ -349,22 +369,26 @@ module Cogworker
     # must happen; everything after it is best-effort. False if that write
     # failed (`finish_failure` then interrupts the job instead).
     def route_failure(job, error)
+      @decided_late = @decision.nil? # raised by middleware, not `perform`: decided after the chain
+      @decision ||= FailureDecision.decide!(job, error, @job_class)
+      decision = @decision || FailureDecision.new(JobUtil.terminal_failure?(job) ? 'dead' : 'retry')
+      job.delete('failure_outcome')
       job['error_class'] = error.class.name
       job['error_message'] = JobUtil.error_message(error)
       job['failed_at'] ||= Time.now.to_f
-
-      max_retries = JobUtil.max_retries(job)
       new_count = job['retry_count'].to_i + 1
       job['retry_count'] = new_count
-      retrying = new_count <= max_retries
-      delay = retry_delay(new_count) if retrying
+      if decision.discard? # neither retry nor dead, no death hooks: the job just ends, counted as failed
+        after_failure_recorded(job, error, decision, nil)
+        return true
+      end
 
+      delay = decision.delay || retry_delay(new_count) if decision.retry?
       begin
         payload = JSON.generate(job)
         Cogworker.config.redis do |c|
-          if retrying
-            c.zadd(RedisKeys::RETRY, Time.now.to_f + delay,
-                   payload)
+          if decision.retry?
+            c.zadd(RedisKeys::RETRY, Time.now.to_f + delay, payload)
           else
             c.zadd(RedisKeys::DEAD, Time.now.to_f, payload)
           end
@@ -374,11 +398,25 @@ module Cogworker
         return false
       end
 
-      best_effort('lock extension') { extend_locks_for_retry(job, delay) } if retrying
+      after_failure_recorded(job, error, decision, delay)
+      @death = [JSON.parse(payload), error] if decision.dead? # run by `process_one`, once settled
+      true
+    end
+
+    def after_failure_recorded(job, error, decision, delay)
+      if decision.retry?
+        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
+      elsif decision.cut_short?
+        # Normally the release middleware's, on the terminal attempt — but
+        # a decision made after the chain (a middleware raised) never
+        # reached it. Owner-checked, so releasing twice is harmless. Same
+        # for the status the Status middleware wrote then: 'retrying'.
+        best_effort('lock release') { release_locks(job) }
+        best_effort('status') { Status.mark_failed(job, error) } if @decided_late
+      end
       best_effort('attempts log') do
-        Attempts.record(job['jid'], attempt: new_count, error: error, outcome: retrying ? 'retrying' : 'dead')
+        Attempts.record(job['jid'], attempt: job['retry_count'], error: error, outcome: decision.attempt_outcome)
       end
-      true
     end
 
     # A job waiting out its retry backoff still holds its `until_executed`
@@ -404,10 +442,9 @@ module Cogworker
       Cogworker.config.redis { |c| JobUtil.release_terminal_locks(c, job) }
     end
 
-    # Grows with the retry count, with jitter to avoid a thundering herd of
-    # retries all landing on the same second.
+    # The default backoff, for a job whose class has no say in it.
     def retry_delay(count)
-      (count**4) + 15 + (rand(30) * (count + 1))
+      FailureDecision.default_delay(count)
     end
 
     def register_in_workers(queue, job)

data/lib/cogworker/reliable_fetch.rb

@@ -404,20 +404,27 @@ module Cogworker
       # `config.max_orphanings` times) in `dead`, with an error saying why — whatever
       # its `retry` setting, since each of those runs ended with its process
       # gone rather than an error to retry on. Parked entries survive a
-      # failure here and are picked up on the next pass.
+      # failure here and are picked up on the next pass. Death hooks run
+      # only for the entries this call itself moved (the script's 1), so
+      # processes reconciling side by side don't run them twice.
       def bury_repeat_orphans
+        buried_jobs = []
         Cogworker.config.redis do |c|
           c.lrange(RedisKeys::REPEAT_ORPHANS, 0, -1).each do |raw|
-            job = parse_hash(raw) || JobUtil.unparseable_job(raw, queue: nil, error: JSON::ParserError.new('unreadable'))
+            readable = parse_hash(raw)
+            job = readable || JobUtil.unparseable_job(raw, queue: nil, error: JSON::ParserError.new('unreadable'))
             job.merge!(repeat_orphan_error)
             buried = LuaScript.run(c, BURY_ORPHAN_SCRIPT, keys: [RedisKeys::REPEAT_ORPHANS, RedisKeys::DEAD],
                                                           argv: [raw, Time.now.to_f, JSON.generate(job)])
             next unless buried == 1
 
+            buried_jobs << job if readable # it's in dead now, whatever the bookkeeping below does
             Cogworker.logger.error { "jid=#{job['jid']} orphaned too often, moved to dead" }
             buried_as_dead(c, job)
           end
         end
+      ensure
+        buried_jobs.each { |job| DeathNotifier.notify_orphaned(job) }
       end
 
       def recover_identity(identity)
@@ -472,18 +479,20 @@ module Cogworker
       def settle_pending(identity, pending)
         key = RedisKeys.in_progress(identity)
         pending.each do |raw, how|
-          Cogworker.config.redis do |c|
+          buried = Cogworker.config.redis do |c|
             if how == :ack
               jid = parse_hash(raw)&.fetch('jid', nil)
               LuaScript.run(c, ACK_SCRIPT, keys: [key, RedisKeys::ORPHANINGS_PREFIX + jid.to_s], argv: [raw])
+              nil
             else
               set, score, payload = how
               filed = LuaScript.run(c, INTERRUPT_SCRIPT, keys: [key, set, orphanings_key(raw)],
                                                          argv: [raw, score, payload])
-              released_if_dead(c, set, payload) if filed == 1
+              filed == 1 ? filed_in_dead(c, set, payload) : nil
             end
           end
           yield raw if block_given?
+          DeathNotifier.notify_failed(buried) if buried
         rescue StandardError => e
           Cogworker.logger.error do
             "still couldn't settle jid=#{parse_hash(raw)&.fetch('jid', nil)}: #{e.class}: #{e.message}"
@@ -516,7 +525,9 @@ module Cogworker
       # Files what shutting-down processes handed over in UNSETTLED, head
       # first, stopping at the first that still can't be filed (retried on
       # the next pass).
+      # Death hooks run for what this call filed in dead (the script's 1).
       def file_unsettled
+        buried = []
         Cogworker.config.redis do |c|
           c.lrange(RedisKeys::UNSETTLED, 0, -1).each do |record|
             entry = parse_hash(record)
@@ -525,9 +536,11 @@ module Cogworker
             filed = LuaScript.run(c, FILE_UNSETTLED_SCRIPT,
                                   keys: [RedisKeys::UNSETTLED, orphanings_key(entry['payload'])],
                                   argv: [record, entry['set'], entry['score'], entry['payload']])
-            released_if_dead(c, entry['set'], entry['payload']) if filed == 1
+            buried << filed_in_dead(c, entry['set'], entry['payload']) if filed == 1
           end
         end
+      ensure
+        buried.compact.each { |payload| DeathNotifier.notify_failed(payload) }
       end
 
       # Jobs REQUEUE_SCRIPT had to file in dead raw (its parking list
@@ -538,13 +551,19 @@ module Cogworker
       # its error, its job's locks held and its failure uncounted.
       def complete_raw_burials(raws)
         raws.each do |raw|
-          job = (parse_hash(raw) || {}).merge(repeat_orphan_error)
-          Cogworker.config.redis do |c|
-            swapped = LuaScript.run(c, SWAP_DEAD_ENTRY_SCRIPT, keys: [RedisKeys::DEAD],
-                                                               argv: [raw, Time.now.to_f, JSON.generate(job)])
-            buried_as_dead(c, job) if swapped == 1
+          readable = parse_hash(raw)
+          job = (readable || {}).merge(repeat_orphan_error)
+          swapped = Cogworker.config.redis do |c|
+            LuaScript.run(c, SWAP_DEAD_ENTRY_SCRIPT, keys: [RedisKeys::DEAD],
+                                                     argv: [raw, Time.now.to_f, JSON.generate(job)])
           end
           @raw_burials_mutex.synchronize { @pending_raw_burials.delete(raw) }
+          next unless swapped == 1
+
+          # The swap is done: the rest must not be retried (it would find
+          # nothing to swap and skip the hooks), so it's best-effort.
+          BestEffort.call('Orphan recovery') { Cogworker.config.redis { |c| buried_as_dead(c, job) } }
+          DeathNotifier.notify_orphaned(job) if readable
         rescue StandardError => e
           @raw_burials_mutex.synchronize { @pending_raw_burials << raw unless @pending_raw_burials.include?(raw) }
           RedisErrors.report('Orphan recovery', e) { "a repeat orphan's dead entry not completed (will retry)" }
@@ -565,11 +584,17 @@ module Cogworker
 
       # `set` must be one of ours to file into — not any key a damaged record
       # names (another type's, which would fail every pass at that record).
-      # Filing a failure in dead outside the middleware chain: release its
-      # locks, as the chain would for a terminal failure.
-      def released_if_dead(conn, set, payload)
+      # Just filed by this process: if that was a (readable) job into dead,
+      # release its locks, as the chain would for a terminal failure, and
+      # return the payload — its death hooks are due. The release is
+      # best-effort: the job is in dead either way, and a raise here used
+      # to cost it its hooks (the next pass's script finds nothing to file).
+      def filed_in_dead(conn, set, payload)
         job = parse_hash(payload)
-        JobUtil.release_terminal_locks(conn, job) if set == RedisKeys::DEAD && job
+        return nil unless set == RedisKeys::DEAD && job
+
+        BestEffort.call('Lock release') { JobUtil.release_terminal_locks(conn, job) }
+        payload
       end
 
       def valid_unsettled?(entry)

data/lib/cogworker/status.rb

@@ -40,5 +40,17 @@ module Cogworker
     def get(jid)
       Storage.read(jid)
     end
+
+    # Corrects a status already written as 'retrying' to 'failed' — for a
+    # job whose `cogworker_retry_in` ended it (`:kill`/`:discard`) only
+    # after the middleware had seen the failure (a middleware raised, not
+    # `perform`). A no-op for a job nothing tracks the status of.
+    def mark_failed(job, error)
+      return unless Storage.read(job['jid'])
+
+      Storage.write(job['jid'], default_expiration || DEFAULT_EXPIRATION,
+                    'status' => 'failed', 'update_time' => Time.now.to_f,
+                    'error_class' => error.class.name, 'error_message' => JobUtil.error_message(error))
+    end
   end
 end

data/lib/cogworker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cogworker
-  VERSION = '0.6.0'
+  VERSION = '0.7.0'
 end

data/lib/cogworker/web/routes/jobs.rb

@@ -34,6 +34,12 @@ module Cogworker
         # (still capped at 25, but the same 25 either way) is one click
         # away via the History link below, not lost.
         ATTEMPTS_DISPLAY_LIMIT = 5
+        # How the retry timeline reads each `Attempts` outcome; `killed`/
+        # `discarded` are the job class's `cogworker_retry_in` ending it early.
+        ATTEMPT_OUTCOME_LABELS = {
+          'dead' => 'moved to dead set', 'killed' => 'killed by retry_in, moved to dead set',
+          'discarded' => 'discarded by retry_in'
+        }.freeze
         TAG_CLASS = { 'Enqueued' => 'tag-neutral', 'Running' => 'tag-accent', 'Scheduled' => 'tag-accent-2',
                       'Retrying' => 'tag-warning', 'Dead' => 'tag-danger' }.freeze
 
@@ -378,14 +384,14 @@ module Cogworker
 
           shown = attempts.first(ATTEMPTS_DISPLAY_LIMIT)
           items = shown.map do |a|
-            dot = a['outcome'] == 'dead' ? 'var(--color-danger)' : 'var(--color-warning)'
+            dot = a['outcome'] == 'retrying' ? 'var(--color-warning)' : 'var(--color-danger)'
             <<~HTML
               <div style="display: grid; grid-template-columns: 18px minmax(0, 1fr); gap: 10px;">
                 <div style="display: flex; flex-direction: column; align-items: center;">
                   <span style="width: 9px; height: 9px; border-radius: 50%; background: #{dot}; margin-top: 5px;"></span>
                 </div>
                 <div style="padding-bottom: 12px;">
-                  <div style="font-size: 13px;">Attempt #{a['attempt']} · #{a['outcome'] == 'dead' ? 'moved to dead set' : 'failed'}</div>
+                  <div style="font-size: 13px;">Attempt #{a['attempt']} · #{ATTEMPT_OUTCOME_LABELS.fetch(a['outcome'], 'failed')}</div>
                   <div style="font-size: 12px; color: var(--color-neutral-500);">#{Layout.h(Time.at(a['failed_at']).utc.strftime(Web.time_format))} · #{Layout.h(a['error_class'])}: #{Layout.h(a['error_message'])}</div>
                 </div>
               </div>

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cogworker
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - vdonec
@@ -216,12 +216,16 @@ files:
 - lib/cogworker/component.rb
 - lib/cogworker/config.rb
 - lib/cogworker/config_loader.rb
+- lib/cogworker/death_notifier.rb
 - lib/cogworker/deferred_releases.rb
+- lib/cogworker/failure_decision.rb
 - lib/cogworker/heartbeat.rb
 - lib/cogworker/history.rb
 - lib/cogworker/history/middleware.rb
 - lib/cogworker/history/storage.rb
 - lib/cogworker/job.rb
+- lib/cogworker/job_failed_error.rb
+- lib/cogworker/job_orphaned_error.rb
 - lib/cogworker/job_record.rb
 - lib/cogworker/job_util.rb
 - lib/cogworker/key_guard.rb