cogworker 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2a2d6b5721f03a71d8a74d376311c047175b71f950d51f227c60598dad74dce2
-  data.tar.gz: d6f10624306849c6e32e8d7fa0211db6c2b947d78d53fec10b79039bfec57645
+  metadata.gz: 472527fd7492d45c642f017b2cdc788c0741ebe7226b42b416aee90431b61194
+  data.tar.gz: '078375dda087dfe1064ed79e35b92b4fb2e4dbf017e68fb681b939d96a45fdea'
 SHA512:
-  metadata.gz: 97347577fddf8a44cc0c5df6b6aafd5a60c285f88f58f7c09193f8a1031d5bf9dc0381e2708a9d591ff81dde7781b31be6a72f7d7567b9f2a0a8b74fb0de2658
-  data.tar.gz: 84ea3a607e75d78b23dab9919c33595ade3ac2382c2c253896a6e3b4df7664d8369fbd1b81a387eaaa9ae95fe3867dd8739ce7d757a0e576047f51321e78cd5e
+  metadata.gz: 4e66fb20f141b58faa5c13eaa1e9df971d890a8bc5a77c845f80fbdcc621beacf2b03c5d9e1dbae68a9b4981a2e322a4147d1e366677e607cb069b846d2d905e
+  data.tar.gz: f3297307cd6201b8421a0acd54217d5a79cdaa1f93fed083ccbd7bd492cb9867340ffc52b787847058a0ce030ed5b17bd1b9d389fbdbc6ef0c64195f83f3422a

data/README.md

@@ -266,8 +266,99 @@ real worker would receive Strings instead of Symbols.
 ### Retries and Dead
 
 A failed job moves to the Retry set with a growing delay
-(`count**4 + 15 + jitter` seconds). When it runs out of attempts, it moves to
-Dead. You can put it back on its queue by hand from the Web UI.
+(`count**4 + 15 + jitter` seconds, `count` being the retry about to happen,
+1 for the first). `retry:` sets the budget: `true` or unset gives 25
+retries, an integer that many, `false` or `0` none. When it runs out of
+retries, it moves to Dead. You can put it back on its queue by hand from the
+Web UI.
+
+#### Custom delay: `cogworker_retry_in`
+
+```ruby
+class SendEventJob
+  include Cogworker::Worker
+  cogworker_options retry: 10
+
+  cogworker_retry_in do |count, exception, job|
+    next :kill if exception.is_a?(PermanentError)
+
+    60 * (2**count)
+  end
+end
+```
+
+`count` is the number of retries made before this failure (0 on the first
+one), `exception` what the attempt raised, `job` a copy of the job hash (the
+block may take fewer parameters). It returns:
+
+| Value | Effect |
+|---|---|
+| Integer / Float ≥ 0 | retry after that many seconds, no jitter added; 0 means on the next poll |
+| `nil` | the default delay |
+| `:kill` | straight to Dead, retries left or not; the death hooks run |
+| `:discard` | dropped: neither Retry nor Dead, no death hooks; counted as failed, its unique locks released |
+| anything else | the default delay, with a warning in the log |
+
+If the block raises, the default delay is used and the error is logged. It
+isn't called for a failure that is the last one anyway (no retry left, or
+`retry: false`), nor for a job whose class can't be loaded. It's asked as
+soon as `perform` raises, before any server middleware sees the error — so
+it also runs when a middleware then swallows that error; keep it free of
+side effects. A delay is never capped (one over a year is logged as a
+likely mistake), and the job's `until_executed` lock is extended by it, so
+the lock never runs out before the retry — however long that is. If
+recording the failure itself fails (Redis trouble), the job is retried
+after a fixed `30 s × n` instead, whatever the hook said. The hook is
+inherited, so a base class of your own (`class ApplicationJob`) is the place
+for an app-wide policy. The Web UI's retry timeline shows `:kill` and
+`:discard` as such.
+
+#### Death hooks: `cogworker_retries_exhausted` and `death_handlers`
+
+```ruby
+class SendEventJob
+  include Cogworker::Worker
+
+  cogworker_retries_exhausted do |job, exception|
+    Metrics.increment('events.lost')
+  end
+end
+
+Cogworker.configure_server do |config|
+  config.death_handlers << ->(job, exception) { ErrorTracker.notify(exception, job) }
+end
+```
+
+When a job lands in Dead for good, its class's `cogworker_retries_exhausted`
+runs first, then every `death_handlers` entry in order. `job` is the entry
+as stored in Dead (with `error_class`, `error_message`, `retry_count`,
+`failed_at`); each hook gets its own copy, so changing it changes nothing.
+
+| How the job died | Class hook | `death_handlers` |
+|---|---|---|
+| out of retries, or `retry: false` / `0` | yes | yes |
+| `cogworker_retry_in` returned `:kill` | yes | yes |
+| its process kept dying while running it (`config.max_orphanings`) | yes, with a `Cogworker::JobOrphanedError` | yes |
+| its failure was filed later, by a reconcile pass | yes, with a `Cogworker::JobFailedError` carrying the original message (the original class is in `job['error_class']`) | yes |
+| its class can't be loaded (renamed, removed) | no | yes |
+| `cogworker_retry_in` returned `:discard` | no | no |
+| an unparseable payload | no | no |
+| deleted or retried from the Web UI | no | no |
+
+The hooks run only after the job is safely in Dead, at most once per death:
+on the reconcile paths, only the process that actually filed it runs them.
+They run synchronously, on the processor thread (once the job is
+acknowledged, before the thread takes the next one) or on the reconcile
+thread, with no timeout of their own — **keep them quick**. One that raises
+is logged and doesn't stop the others, nor the job's ack or lock release.
+
+A hook can be skipped or cut short, never repeated: a process that crashes
+right after the write never runs it, and on shutdown a hook still running
+when the drain timeout (25 s) is up may be cut off as the process exits.
+The job itself is safe either way — already in Dead and acknowledged, so it
+won't run again. `Testing.inline!` raises the job's error to the caller as before
+and runs no hooks; test a hook directly through
+`SomeJob.cogworker_retry_in_block` / `SomeJob.cogworker_retries_exhausted_block`.
 
 ### Unique jobs
 

data/lib/cogworker/config.rb

@@ -9,7 +9,7 @@ module Cogworker
     FETCH_MODES = %i[reliable basic].freeze
 
     attr_reader :server_chain, :client_chain, :periodic_manager, :redis_options, :fetch, :fetch_idle_max_interval,
-                :orphan_threshold, :max_orphanings, :periodic_replace
+                :orphan_threshold, :max_orphanings, :periodic_replace, :death_handlers
     attr_accessor :concurrency, :queues, :periodic_catch_up, :unique_lock_ttl
 
     def initialize
@@ -57,6 +57,10 @@ module Cogworker
       # that crashed is counted, so keep this above the number of crashes an
       # innocent job might plausibly sit through.
       @max_orphanings = 3
+      # Callables `(job, exception)` run, after the job class's own
+      # `cogworker_retries_exhausted`, whenever a job lands in dead for good
+      # (DeathNotifier). `config.death_handlers << ->(job, e) { ... }`.
+      @death_handlers = []
       register_default_middleware
     end
 

data/lib/cogworker/death_notifier.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module Cogworker
+  # Runs the death hooks for a job that has just been written to dead for
+  # good: the job class's own `cogworker_retries_exhausted`, then every
+  # `config.death_handlers` entry, in order. Called only by whoever's write
+  # actually put it there (so at most once per death), after that write —
+  # and, from a processor, only once the job is acknowledged, so a slow
+  # hook can't leave it on the in-progress list to be requeued and re-run.
+  #
+  # Each hook gets its own copy of the job (as stored in dead) and runs on
+  # its own: one that raises is logged and never stops the others, nor the
+  # ack / lock releases / reconcile pass around the call. Synchronous, with
+  # no timeout — a hook should be quick.
+  module DeathNotifier
+    module_function
+
+    def notify(job, exception)
+      hook = job_class(job['class'])&.cogworker_retries_exhausted_block
+      run('cogworker_retries_exhausted', job) { |copy| JobUtil.call_hook(hook, copy, exception) } if hook
+      Cogworker.config.death_handlers.each do |handler|
+        run('death handler', job) { |copy| JobUtil.call_hook(handler, copy, exception) }
+      end
+    end
+
+    # A dead entry filed from its stored record (no exception object left).
+    def notify_failed(payload)
+      job = parse(payload)
+      notify(job, JobFailedError.new(job['error_message'].to_s)) if job
+    end
+
+    def notify_orphaned(job)
+      notify(job, JobOrphanedError.new("orphaned more than #{Cogworker.config.max_orphanings} times"))
+    end
+
+    def run(what, job)
+      yield JobUtil.deep_copy(job)
+    rescue Exception => e # rubocop:disable Lint/RescueException -- user code, isolated like `perform`
+      Cogworker.logger.error do
+        "#{what} failed for jid=#{job['jid']} (ignored): #{e.class}: #{JobUtil.error_message(e)}"
+      end
+    end
+
+    # nil for a class that can't be loaded (renamed, removed, never defined)
+    # or doesn't use the job DSL: only `death_handlers` run then.
+    def job_class(name)
+      return nil unless name.is_a?(String) && !name.empty?
+
+      klass = Object.const_get(name)
+      klass.respond_to?(:cogworker_retries_exhausted_block) ? klass : nil
+    rescue StandardError
+      nil
+    end
+
+    def parse(payload)
+      job = JSON.parse(payload)
+      job.is_a?(Hash) ? job : nil
+    rescue JSON::ParserError, TypeError
+      nil
+    end
+  end
+end

data/lib/cogworker/failure_decision.rb

@@ -0,0 +1,131 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # What becomes of a failed attempt: `retry` (after `delay` seconds — nil
+  # for the default backoff), `dead` or `discard`. Made once per failure:
+  # by the job class's `cogworker_retry_in` hook where it has one, otherwise
+  # by the retry budget and the default backoff.
+  #
+  # `Processor` makes it inside the middleware chain, the moment `perform`
+  # raises, and (when a hook took part) stamps it on the job as
+  # `failure_outcome` so `JobUtil.terminal_failure?` — and with it Status,
+  # UniqueJobs and Periodic middleware — gives the same answer as the
+  # retry/dead write. The stamp never reaches a stored entry.
+  class FailureDecision
+    OUTCOMES = %w[retry dead discard].freeze
+    # Used as given, never capped — but a delay past this is most likely a
+    # unit mix-up (ms for s), and it keeps an `until_executed` lock that long.
+    LONG_DELAY = 365 * 24 * 60 * 60
+
+    attr_reader :outcome, :delay
+
+    class << self
+      # `for`, stamped on the job as `failure_outcome` when the class's hook
+      # took part (see the class comment). nil if deciding itself failed.
+      def decide!(job, error, klass)
+        decision = self.for(job, error, klass)
+        job['failure_outcome'] = decision.outcome if decision.hooked?
+        decision
+      rescue StandardError => e
+        Cogworker.logger.error { "couldn't decide the fate of jid=#{job['jid']}: #{e.class}: #{e.message}" }
+        nil
+      end
+
+      # `klass` — the job's class, or nil when it couldn't be resolved (no
+      # hook then). The hook isn't asked about a failure that is the last
+      # one anyway: there's no retry to time.
+      def for(job, error, klass)
+        count = job['retry_count'].to_i
+        return new('dead') if count + 1 > JobUtil.max_retries(job)
+
+        hook = klass.respond_to?(:cogworker_retry_in_block) ? klass.cogworker_retry_in_block : nil
+        return new('retry') unless hook
+
+        from_hook(hook, job, error, count)
+      end
+
+      # Grows with the retry count, with jitter to avoid a thundering herd
+      # of retries all landing on the same second.
+      def default_delay(count)
+        (count**4) + 15 + (rand(30) * (count + 1))
+      end
+
+      private
+
+      def from_hook(hook, job, error, count)
+        value = JobUtil.call_hook(hook, count, error, JobUtil.deep_copy(job))
+        case value
+        when nil then new('retry', hooked: true)
+        when :kill then new('dead', killed: true, hooked: true)
+        when :discard then new('discard', hooked: true)
+        else
+          if valid_delay?(value)
+            warn_long_delay(job, value) if value > LONG_DELAY
+            return new('retry', value, hooked: true)
+          end
+
+          Cogworker.logger.warn do
+            "cogworker_retry_in of #{job['class']} returned #{value.inspect} for jid=#{job['jid']}; " \
+              'using the default backoff'
+          end
+          new('retry', hooked: true)
+        end
+      rescue Exception => e # rubocop:disable Lint/RescueException -- user code, isolated like `perform`
+        Cogworker.logger.error do
+          "cogworker_retry_in of #{job['class']} raised for jid=#{job['jid']} (using the default backoff): " \
+            "#{e.class}: #{JobUtil.error_message(e)}"
+        end
+        new('retry', hooked: true)
+      end
+
+      def warn_long_delay(job, value)
+        Cogworker.logger.warn do
+          "cogworker_retry_in of #{job['class']} returned #{value}s (over a year) for jid=#{job['jid']}; " \
+            'using it as given'
+        end
+      end
+
+      def valid_delay?(value)
+        (value.is_a?(Integer) || (value.is_a?(Float) && value.finite?)) && value >= 0
+      end
+    end
+
+    def initialize(outcome, delay = nil, killed: false, hooked: false)
+      @outcome = outcome
+      @delay = delay
+      @killed = killed
+      @hooked = hooked
+    end
+
+    def retry?
+      outcome == 'retry'
+    end
+
+    def dead?
+      outcome == 'dead'
+    end
+
+    def discard?
+      outcome == 'discard'
+    end
+
+    # Whether the job class's hook took part (only then is the decision
+    # stamped on the job: without one, `terminal_failure?` already agrees).
+    def hooked?
+      @hooked
+    end
+
+    # Ended before its retries ran out: `:kill`, or `:discard`.
+    def cut_short?
+      @killed || discard?
+    end
+
+    # The outcome as the attempts log (and the Web UI's timeline) shows it.
+    def attempt_outcome
+      return 'retrying' if retry?
+      return 'discarded' if discard?
+
+      @killed ? 'killed' : 'dead'
+    end
+  end
+end

data/lib/cogworker/heartbeat.rb

@@ -75,7 +75,8 @@ module Cogworker
 
     def cleanup_presence!
       Cogworker.config.redis do |c|
-        c.del(RedisKeys.process(Cogworker.identity), RedisKeys.workers(Cogworker.identity))
+        c.del(RedisKeys.process(Cogworker.identity), RedisKeys.workers(Cogworker.identity),
+              RedisKeys.memory_history(Cogworker.identity))
         c.srem?(RedisKeys::PROCESSES, Cogworker.identity)
         # Clean shutdown: Manager#stop! already requeued this process's
         # in-progress list, so there's nothing left for recovery to find.
@@ -132,6 +133,7 @@ module Cogworker
                'quiet', @manager.quiet?.to_s)
         c.expire(RedisKeys.process(identity), TTL)
         write_registries(c, identity, reliable)
+        secondary('memory history') { MemoryHistory.record(c, identity, info['rss_kb']) }
         touch_running_periodic_locks(c, identity)
         # Not via #secondary: Redis failing here, after the presence key went
         # through, doesn't make this beat a failed one.

data/lib/cogworker/job.rb

@@ -30,6 +30,32 @@ module Cogworker
                                     end
       end
 
+      # `cogworker_retry_in { |count, exception, job| ... }` — the pause
+      # (seconds) before the next attempt, or `:kill` (straight to dead) /
+      # `:discard` (dropped), or `nil` for the default backoff. See
+      # `FailureDecision`. Kept on the class, never in the job payload.
+      def cogworker_retry_in(&block)
+        raise ArgumentError, 'cogworker_retry_in needs a block' unless block
+
+        @cogworker_retry_in_block = block
+      end
+
+      # `cogworker_retries_exhausted { |job, exception| ... }` — called once
+      # the job is in dead for good (see `DeathNotifier`).
+      def cogworker_retries_exhausted(&block)
+        raise ArgumentError, 'cogworker_retries_exhausted needs a block' unless block
+
+        @cogworker_retries_exhausted_block = block
+      end
+
+      def cogworker_retry_in_block
+        inherited_hook(:@cogworker_retry_in_block, :cogworker_retry_in_block)
+      end
+
+      def cogworker_retries_exhausted_block
+        inherited_hook(:@cogworker_retries_exhausted_block, :cogworker_retries_exhausted_block)
+      end
+
       def perform_async(*args)
         Client.push(job_payload('args' => args))
       end
@@ -61,6 +87,14 @@ module Cogworker
 
       private
 
+      # Looked up at call time (not copied at definition), so a subclass
+      # sees a hook its parent defines later too.
+      def inherited_hook(ivar, reader)
+        return instance_variable_get(ivar) if instance_variable_defined?(ivar)
+
+        superclass.respond_to?(reader) ? superclass.public_send(reader) : nil
+      end
+
       def job_payload(extra)
         cogworker_options_hash.transform_keys(&:to_s).merge('class' => name).merge(extra)
       end

data/lib/cogworker/job_failed_error.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # Handed to death hooks (DeathNotifier) for a failure filed in dead later,
+  # from its stored record, when the original exception object is long gone.
+  # The message is the original one; its class is in `job['error_class']`.
+  class JobFailedError < StandardError
+  end
+end

data/lib/cogworker/job_orphaned_error.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # Handed to death hooks (DeathNotifier) for a job buried because its
+  # process kept dying while running it — there's no exception of its own.
+  class JobOrphanedError < StandardError
+  end
+end

data/lib/cogworker/job_util.rb

@@ -61,10 +61,46 @@ module Cogworker
     # `Periodic::ReleaseMiddleware` (releases the running-lock only on the
     # terminal attempt) — previously three separate, subtly different copies
     # of this same check, two of which had the `max_retries` bug above.
+    #
+    # A job class's `cogworker_retry_in` can end a job early (`:kill`,
+    # `:discard`): `Processor` stamps that decision on the job as
+    # `failure_outcome` (FailureDecision) before the middleware sees the
+    # failure, and it wins here when present.
     def terminal_failure?(job)
+      outcome = job['failure_outcome']
+      return outcome != 'retry' if FailureDecision::OUTCOMES.include?(outcome)
+
       job['retry_count'].to_i >= max_retries(job)
     end
 
+    # A copy of a job hash to hand to user code, so changes it makes can't
+    # leak back. Never raises: unlike a JSON round trip, it copes with
+    # whatever middleware may have put in the hash (the job being on its
+    # way to `interrupt` precisely because that doesn't serialize).
+    def deep_copy(value)
+      case value
+      when Hash then value.to_h { |k, v| [deep_copy(k), deep_copy(v)] }
+      when Array then value.map { |v| deep_copy(v) }
+      when String then value.dup
+      else value
+      end
+    end
+
+    # Calls a user-supplied hook (a block, lambda, Method or any `#call`
+    # object) with as many of `args` as it takes: a block with fewer
+    # parameters ignores the rest anyway, a lambda or method would raise.
+    # Counted from `parameters` (required + optional), not `arity`, which
+    # is negative as soon as there's an optional one.
+    def call_hook(hook, *args)
+      callable = hook.is_a?(Proc) || hook.is_a?(Method) ? hook : hook.method(:call)
+      return hook.call(*args) if callable.is_a?(Proc) && !callable.lambda?
+
+      params = callable.parameters
+      return hook.call(*args) if params.any? { |type, _| type == :rest }
+
+      hook.call(*args.first(params.count { |type, _| %i[req opt].include?(type) }))
+    end
+
     # The job Hash a raw `schedule`/`retry`/`dead` entry would requeue as —
     # or `nil` if it can't be: not JSON, not an object, or no String
     # `queue` to push it onto (e.g. an `unparseable_job` wrapper, whose

data/lib/cogworker/key_guard.rb

@@ -65,6 +65,7 @@ module Cogworker
       readable_members(RedisKeys::PROCESSES).each do |id|
         types[RedisKeys.process(id)] = 'hash'
         types[RedisKeys.workers(id)] = 'hash'
+        types[RedisKeys.memory_history(id)] = 'zset'
       end
       readable_fields(RedisKeys::PERIODIC_SCHEDULE).each do |pjid|
         types[RedisKeys.periodic_running(pjid)] = 'string'

data/lib/cogworker/memory_history.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module Cogworker
+  # Per-process RSS for the last 24 hours, in SLOTS fixed SAMPLE_SECONDS
+  # slots, feeding the memory sparkline on the Workers tab's process cards.
+  # Written by Heartbeat on every beat; a slot keeps the *highest* reading
+  # any beat in it took (a peak is what matters for memory, and a short
+  # spike between two slot boundaries would otherwise vanish). One ZSET per
+  # identity, scored by slot start, members "<slot_start>:<rss_kb>" — so a
+  # slot's sample is unique by score, not by value. Old slots are trimmed
+  # on every write, and the key itself expires a little after the window,
+  # so a process that's gone leaves nothing behind for long.
+  module MemoryHistory
+    SAMPLE_SECONDS = 900
+    SLOTS = 96 # 24h of 15-minute slots
+    WINDOW = SLOTS * SAMPLE_SECONDS
+    KEY_TTL = WINDOW + 3600
+
+    # KEYS[1] = the identity's ZSET; ARGV = slot start, rss_kb, window, TTL.
+    RECORD_SCRIPT = <<~LUA
+      local slot = tonumber(ARGV[1])
+      local kb = tonumber(ARGV[2])
+      for _, member in ipairs(redis.call("ZRANGEBYSCORE", KEYS[1], slot, slot)) do
+        local seen = tonumber(string.match(member, ":(%d+)$"))
+        if seen and seen >= kb then kb = nil break end
+      end
+      if kb then
+        redis.call("ZREMRANGEBYSCORE", KEYS[1], slot, slot)
+        redis.call("ZADD", KEYS[1], slot, slot .. ":" .. kb)
+      end
+      redis.call("ZREMRANGEBYSCORE", KEYS[1], "-inf", "(" .. (slot - tonumber(ARGV[3])))
+      redis.call("EXPIRE", KEYS[1], ARGV[4])
+      return 1
+    LUA
+
+    module_function
+
+    def record(conn, identity, rss_kb, at: Time.now.to_i)
+      return unless rss_kb
+
+      LuaScript.run(conn, RECORD_SCRIPT, keys: [RedisKeys.memory_history(identity)],
+                                         argv: [slot_for(at), rss_kb.to_i, WINDOW, KEY_TTL])
+    end
+
+    # Per identity, exactly SLOTS `[slot_start, rss_kb_or_nil]` pairs,
+    # oldest first, the last one being the current (still filling) slot —
+    # a slot no beat wrote to is `nil`, so the chart always spans the full
+    # 24h whatever data exists. One pipelined read for every identity.
+    def series(identities, now: Time.now.to_i)
+      return {} if identities.empty?
+
+      first = slot_for(now) - ((SLOTS - 1) * SAMPLE_SECONDS)
+      raw = Cogworker.config.redis do |c|
+        c.pipelined do |pipe|
+          identities.each { |id| pipe.zrangebyscore(RedisKeys.memory_history(id), first, '+inf') }
+        end
+      end
+      identities.zip(raw).to_h do |id, members|
+        values = Array(members).to_h { |m| m.split(':', 2).map(&:to_i) }
+        [id, Array.new(SLOTS) { |i| [first + (i * SAMPLE_SECONDS), values[first + (i * SAMPLE_SECONDS)]] }]
+      end
+    end
+
+    def slot_for(time)
+      time.to_i - (time.to_i % SAMPLE_SECONDS)
+    end
+  end
+end

data/lib/cogworker/processor.rb

@@ -90,6 +90,8 @@ module Cogworker
       @busy = true
       @ran = nil
       @performed = false
+      @decision = nil
+      @death = nil
       disposition = nil
       begin
         disposition = execute(work)
@@ -104,6 +106,10 @@ module Cogworker
         # can't re-queue a job that already ran.
         settle(work, disposition || @ran || :give_back)
         @manager.job_finished(work.raw_job)
+        # Only now, with the job acknowledged (or its ack remembered): a
+        # slow hook used to hold it on the in-progress list, where a
+        # shutdown or orphan recovery requeued it and ran it again.
+        DeathNotifier.notify(*@death) if @death
       end
     end
 
@@ -139,9 +145,10 @@ module Cogworker
       payload.merge!('interrupted_count' => count, 'retry_count' => attempt,
                      'error_class' => error.class.name, 'error_message' => JobUtil.error_message(error),
                      'failed_at' => Time.now.to_f)
-      # No retries left (`retry: false`/`0`, or the last one used up) means
-      # no extra attempt either — straight to dead, like any terminal failure.
-      out_of_retries = attempt > JobUtil.max_retries(payload)
+      # No retries left (`retry: false`/`0`, or the last one used up, or the
+      # job's `cogworker_retry_in` ended it) means no extra attempt either —
+      # straight to dead, like any terminal failure.
+      out_of_retries = attempt > JobUtil.max_retries(payload) || @decision&.cut_short?
       delay = INTERRUPT_DELAY * count
       set, score = if out_of_retries || count > MAX_INTERRUPTS
                      [RedisKeys::DEAD, Time.now.to_f]
@@ -150,25 +157,15 @@ module Cogworker
                    end
       encoded = JSON.generate(payload)
       begin
-        fetcher.interrupt(work, set, score, encoded)
+        written = fetcher.interrupt(work, set, score, encoded) != 0 # (0: no longer ours to file)
       rescue StandardError
         # Remembered with its payload: this process's reconcile (and its
         # shutdown) keep trying to file it — never re-run it as a stray.
         @manager.settle_later(work.raw_job, [set, score, encoded]) if fetcher.is_a?(ReliableFetch)
         raise
       end
-      if set == RedisKeys::RETRY
-        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
-      else
-        best_effort('lock release') { release_locks(payload) }
-      end
-      reason = if set == RedisKeys::RETRY
-                 "retrying in #{delay}s (interruption #{count} of at most #{MAX_INTERRUPTS})"
-               elsif out_of_retries
-                 'moved to dead: no retries left'
-               else
-                 "moved to dead: interrupted more than #{MAX_INTERRUPTS} times"
-               end
+      after_interrupt(job, payload, error, set, delay, written)
+      reason = interrupt_reason(set, out_of_retries, count, delay)
       Cogworker.logger.error { "couldn't record the failure of jid=#{payload['jid']} (#{error.class}); #{reason}" }
     rescue StandardError => e
       # Whatever failed (even before anything was written), remember the job
@@ -181,6 +178,21 @@ module Cogworker
       end
     end
 
+    def after_interrupt(job, payload, error, set, delay, written)
+      if set == RedisKeys::RETRY
+        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
+      else
+        best_effort('lock release') { release_locks(payload) }
+        @death = [payload, error] if written # run by `process_one`, once settled
+      end
+    end
+
+    def interrupt_reason(set, out_of_retries, count, delay)
+      return "retrying in #{delay}s (interruption #{count} of at most #{MAX_INTERRUPTS})" if set == RedisKeys::RETRY
+
+      out_of_retries ? 'moved to dead: no retries left' : "moved to dead: interrupted more than #{MAX_INTERRUPTS} times"
+    end
+
     def remember_for_dead(work, error)
       # The original job's own fields where readable (so filing it releases
       # its locks); just enough to show in Dead otherwise.
@@ -272,10 +284,18 @@ module Cogworker
         nil
       end
 
+      @job_class = worker&.class
       Cogworker.config.server_chain.invoke(worker, job, work.queue) do
         raise resolution_error if resolution_error
 
-        worker.perform(*job['args'])
+        begin
+          worker.perform(*job['args'])
+        rescue Exception => e # rubocop:disable Lint/RescueException
+          # Retry, dead or discard — decided here, while the middleware is
+          # still to see the failure (FailureDecision.decide!).
+          @decision = FailureDecision.decide!(job, e, @job_class)
+          raise
+        end
       end
       nil
     rescue Exception => e # rubocop:disable Lint/RescueException
@@ -349,22 +369,26 @@ module Cogworker
     # must happen; everything after it is best-effort. False if that write
     # failed (`finish_failure` then interrupts the job instead).
     def route_failure(job, error)
+      @decided_late = @decision.nil? # raised by middleware, not `perform`: decided after the chain
+      @decision ||= FailureDecision.decide!(job, error, @job_class)
+      decision = @decision || FailureDecision.new(JobUtil.terminal_failure?(job) ? 'dead' : 'retry')
+      job.delete('failure_outcome')
       job['error_class'] = error.class.name
       job['error_message'] = JobUtil.error_message(error)
       job['failed_at'] ||= Time.now.to_f
-
-      max_retries = JobUtil.max_retries(job)
       new_count = job['retry_count'].to_i + 1
       job['retry_count'] = new_count
-      retrying = new_count <= max_retries
-      delay = retry_delay(new_count) if retrying
+      if decision.discard? # neither retry nor dead, no death hooks: the job just ends, counted as failed
+        after_failure_recorded(job, error, decision, nil)
+        return true
+      end
 
+      delay = decision.delay || retry_delay(new_count) if decision.retry?
       begin
         payload = JSON.generate(job)
         Cogworker.config.redis do |c|
-          if retrying
-            c.zadd(RedisKeys::RETRY, Time.now.to_f + delay,
-                   payload)
+          if decision.retry?
+            c.zadd(RedisKeys::RETRY, Time.now.to_f + delay, payload)
           else
             c.zadd(RedisKeys::DEAD, Time.now.to_f, payload)
           end
@@ -374,11 +398,25 @@ module Cogworker
         return false
       end
 
-      best_effort('lock extension') { extend_locks_for_retry(job, delay) } if retrying
+      after_failure_recorded(job, error, decision, delay)
+      @death = [JSON.parse(payload), error] if decision.dead? # run by `process_one`, once settled
+      true
+    end
+
+    def after_failure_recorded(job, error, decision, delay)
+      if decision.retry?
+        best_effort('lock extension') { extend_locks_for_retry(job, delay) }
+      elsif decision.cut_short?
+        # Normally the release middleware's, on the terminal attempt — but
+        # a decision made after the chain (a middleware raised) never
+        # reached it. Owner-checked, so releasing twice is harmless. Same
+        # for the status the Status middleware wrote then: 'retrying'.
+        best_effort('lock release') { release_locks(job) }
+        best_effort('status') { Status.mark_failed(job, error) } if @decided_late
+      end
       best_effort('attempts log') do
-        Attempts.record(job['jid'], attempt: new_count, error: error, outcome: retrying ? 'retrying' : 'dead')
+        Attempts.record(job['jid'], attempt: job['retry_count'], error: error, outcome: decision.attempt_outcome)
       end
-      true
     end
 
     # A job waiting out its retry backoff still holds its `until_executed`
@@ -404,10 +442,9 @@ module Cogworker
       Cogworker.config.redis { |c| JobUtil.release_terminal_locks(c, job) }
     end
 
-    # Grows with the retry count, with jitter to avoid a thundering herd of
-    # retries all landing on the same second.
+    # The default backoff, for a job whose class has no say in it.
     def retry_delay(count)
-      (count**4) + 15 + (rand(30) * (count + 1))
+      FailureDecision.default_delay(count)
     end
 
     def register_in_workers(queue, job)

data/lib/cogworker/redis_keys.rb

@@ -47,6 +47,7 @@ module Cogworker
     def job_attempts(jid) = "cogworker:job_attempts:#{jid}"
     def process(identity) = "cogworker:process:#{identity}"
     def workers(identity) = "cogworker:workers:#{identity}"
+    def memory_history(identity) = "cogworker:memory:#{identity}"
     def in_progress(identity) = "#{IN_PROGRESS_PREFIX}#{identity}"
     def signal(identity) = "cogworker:signal:#{identity}"
     def periodic_running(pjid) = "periodic:running:#{pjid}"

data/lib/cogworker/reliable_fetch.rb

@@ -404,20 +404,27 @@ module Cogworker
       # `config.max_orphanings` times) in `dead`, with an error saying why — whatever
       # its `retry` setting, since each of those runs ended with its process
       # gone rather than an error to retry on. Parked entries survive a
-      # failure here and are picked up on the next pass.
+      # failure here and are picked up on the next pass. Death hooks run
+      # only for the entries this call itself moved (the script's 1), so
+      # processes reconciling side by side don't run them twice.
       def bury_repeat_orphans
+        buried_jobs = []
         Cogworker.config.redis do |c|
           c.lrange(RedisKeys::REPEAT_ORPHANS, 0, -1).each do |raw|
-            job = parse_hash(raw) || JobUtil.unparseable_job(raw, queue: nil, error: JSON::ParserError.new('unreadable'))
+            readable = parse_hash(raw)
+            job = readable || JobUtil.unparseable_job(raw, queue: nil, error: JSON::ParserError.new('unreadable'))
             job.merge!(repeat_orphan_error)
             buried = LuaScript.run(c, BURY_ORPHAN_SCRIPT, keys: [RedisKeys::REPEAT_ORPHANS, RedisKeys::DEAD],
                                                           argv: [raw, Time.now.to_f, JSON.generate(job)])
             next unless buried == 1
 
+            buried_jobs << job if readable # it's in dead now, whatever the bookkeeping below does
             Cogworker.logger.error { "jid=#{job['jid']} orphaned too often, moved to dead" }
             buried_as_dead(c, job)
           end
         end
+      ensure
+        buried_jobs.each { |job| DeathNotifier.notify_orphaned(job) }
       end
 
       def recover_identity(identity)
@@ -472,18 +479,20 @@ module Cogworker
       def settle_pending(identity, pending)
         key = RedisKeys.in_progress(identity)
         pending.each do |raw, how|
-          Cogworker.config.redis do |c|
+          buried = Cogworker.config.redis do |c|
             if how == :ack
               jid = parse_hash(raw)&.fetch('jid', nil)
               LuaScript.run(c, ACK_SCRIPT, keys: [key, RedisKeys::ORPHANINGS_PREFIX + jid.to_s], argv: [raw])
+              nil
             else
               set, score, payload = how
               filed = LuaScript.run(c, INTERRUPT_SCRIPT, keys: [key, set, orphanings_key(raw)],
                                                          argv: [raw, score, payload])
-              released_if_dead(c, set, payload) if filed == 1
+              filed == 1 ? filed_in_dead(c, set, payload) : nil
             end
           end
           yield raw if block_given?
+          DeathNotifier.notify_failed(buried) if buried
         rescue StandardError => e
           Cogworker.logger.error do
             "still couldn't settle jid=#{parse_hash(raw)&.fetch('jid', nil)}: #{e.class}: #{e.message}"
@@ -516,7 +525,9 @@ module Cogworker
       # Files what shutting-down processes handed over in UNSETTLED, head
       # first, stopping at the first that still can't be filed (retried on
       # the next pass).
+      # Death hooks run for what this call filed in dead (the script's 1).
       def file_unsettled
+        buried = []
         Cogworker.config.redis do |c|
           c.lrange(RedisKeys::UNSETTLED, 0, -1).each do |record|
             entry = parse_hash(record)
@@ -525,9 +536,11 @@ module Cogworker
             filed = LuaScript.run(c, FILE_UNSETTLED_SCRIPT,
                                   keys: [RedisKeys::UNSETTLED, orphanings_key(entry['payload'])],
                                   argv: [record, entry['set'], entry['score'], entry['payload']])
-            released_if_dead(c, entry['set'], entry['payload']) if filed == 1
+            buried << filed_in_dead(c, entry['set'], entry['payload']) if filed == 1
           end
         end
+      ensure
+        buried.compact.each { |payload| DeathNotifier.notify_failed(payload) }
       end
 
       # Jobs REQUEUE_SCRIPT had to file in dead raw (its parking list
@@ -538,13 +551,19 @@ module Cogworker
       # its error, its job's locks held and its failure uncounted.
       def complete_raw_burials(raws)
         raws.each do |raw|
-          job = (parse_hash(raw) || {}).merge(repeat_orphan_error)
-          Cogworker.config.redis do |c|
-            swapped = LuaScript.run(c, SWAP_DEAD_ENTRY_SCRIPT, keys: [RedisKeys::DEAD],
-                                                               argv: [raw, Time.now.to_f, JSON.generate(job)])
-            buried_as_dead(c, job) if swapped == 1
+          readable = parse_hash(raw)
+          job = (readable || {}).merge(repeat_orphan_error)
+          swapped = Cogworker.config.redis do |c|
+            LuaScript.run(c, SWAP_DEAD_ENTRY_SCRIPT, keys: [RedisKeys::DEAD],
+                                                     argv: [raw, Time.now.to_f, JSON.generate(job)])
           end
           @raw_burials_mutex.synchronize { @pending_raw_burials.delete(raw) }
+          next unless swapped == 1
+
+          # The swap is done: the rest must not be retried (it would find
+          # nothing to swap and skip the hooks), so it's best-effort.
+          BestEffort.call('Orphan recovery') { Cogworker.config.redis { |c| buried_as_dead(c, job) } }
+          DeathNotifier.notify_orphaned(job) if readable
         rescue StandardError => e
           @raw_burials_mutex.synchronize { @pending_raw_burials << raw unless @pending_raw_burials.include?(raw) }
           RedisErrors.report('Orphan recovery', e) { "a repeat orphan's dead entry not completed (will retry)" }
@@ -565,11 +584,17 @@ module Cogworker
 
       # `set` must be one of ours to file into — not any key a damaged record
       # names (another type's, which would fail every pass at that record).
-      # Filing a failure in dead outside the middleware chain: release its
-      # locks, as the chain would for a terminal failure.
-      def released_if_dead(conn, set, payload)
+      # Just filed by this process: if that was a (readable) job into dead,
+      # release its locks, as the chain would for a terminal failure, and
+      # return the payload — its death hooks are due. The release is
+      # best-effort: the job is in dead either way, and a raise here used
+      # to cost it its hooks (the next pass's script finds nothing to file).
+      def filed_in_dead(conn, set, payload)
         job = parse_hash(payload)
-        JobUtil.release_terminal_locks(conn, job) if set == RedisKeys::DEAD && job
+        return nil unless set == RedisKeys::DEAD && job
+
+        BestEffort.call('Lock release') { JobUtil.release_terminal_locks(conn, job) }
+        payload
       end
 
       def valid_unsettled?(entry)

data/lib/cogworker/status.rb

@@ -40,5 +40,17 @@ module Cogworker
     def get(jid)
       Storage.read(jid)
     end
+
+    # Corrects a status already written as 'retrying' to 'failed' — for a
+    # job whose `cogworker_retry_in` ended it (`:kill`/`:discard`) only
+    # after the middleware had seen the failure (a middleware raised, not
+    # `perform`). A no-op for a job nothing tracks the status of.
+    def mark_failed(job, error)
+      return unless Storage.read(job['jid'])
+
+      Storage.write(job['jid'], default_expiration || DEFAULT_EXPIRATION,
+                    'status' => 'failed', 'update_time' => Time.now.to_f,
+                    'error_class' => error.class.name, 'error_message' => JobUtil.error_message(error))
+    end
   end
 end

data/lib/cogworker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Cogworker
-  VERSION = '0.6.0'
+  VERSION = '0.8.0'
 end

data/lib/cogworker/web/routes/jobs.rb

@@ -34,6 +34,12 @@ module Cogworker
         # (still capped at 25, but the same 25 either way) is one click
         # away via the History link below, not lost.
         ATTEMPTS_DISPLAY_LIMIT = 5
+        # How the retry timeline reads each `Attempts` outcome; `killed`/
+        # `discarded` are the job class's `cogworker_retry_in` ending it early.
+        ATTEMPT_OUTCOME_LABELS = {
+          'dead' => 'moved to dead set', 'killed' => 'killed by retry_in, moved to dead set',
+          'discarded' => 'discarded by retry_in'
+        }.freeze
         TAG_CLASS = { 'Enqueued' => 'tag-neutral', 'Running' => 'tag-accent', 'Scheduled' => 'tag-accent-2',
                       'Retrying' => 'tag-warning', 'Dead' => 'tag-danger' }.freeze
 
@@ -378,14 +384,14 @@ module Cogworker
 
           shown = attempts.first(ATTEMPTS_DISPLAY_LIMIT)
           items = shown.map do |a|
-            dot = a['outcome'] == 'dead' ? 'var(--color-danger)' : 'var(--color-warning)'
+            dot = a['outcome'] == 'retrying' ? 'var(--color-warning)' : 'var(--color-danger)'
             <<~HTML
               <div style="display: grid; grid-template-columns: 18px minmax(0, 1fr); gap: 10px;">
                 <div style="display: flex; flex-direction: column; align-items: center;">
                   <span style="width: 9px; height: 9px; border-radius: 50%; background: #{dot}; margin-top: 5px;"></span>
                 </div>
                 <div style="padding-bottom: 12px;">
-                  <div style="font-size: 13px;">Attempt #{a['attempt']} · #{a['outcome'] == 'dead' ? 'moved to dead set' : 'failed'}</div>
+                  <div style="font-size: 13px;">Attempt #{a['attempt']} · #{ATTEMPT_OUTCOME_LABELS.fetch(a['outcome'], 'failed')}</div>
                   <div style="font-size: 12px; color: var(--color-neutral-500);">#{Layout.h(Time.at(a['failed_at']).utc.strftime(Web.time_format))} · #{Layout.h(a['error_class'])}: #{Layout.h(a['error_message'])}</div>
                 </div>
               </div>

data/lib/cogworker/web/routes/overview.rb

@@ -53,6 +53,13 @@ module Cogworker
           'connected_clients' => 'Connections', 'used_memory_human' => 'Memory Usage',
           'used_memory_peak_human' => 'Peak Memory Usage'
         }.freeze
+        # Runtime dependency gem name => card label, for the "Cogworker"
+        # section — read from `Gem.loaded_specs`, so it shows what this
+        # process actually loaded, not the gemspec's version constraint.
+        GEM_INFO_DEPENDENCIES = {
+          'redis' => 'redis gem', 'connection_pool' => 'connection_pool', 'rack' => 'Rack',
+          'zeitwerk' => 'Zeitwerk', 'fugit' => 'Fugit'
+        }.freeze
         # Period switcher options for the "Runs per day" chart —
         # `params['period']` (a plain query string, read fresh on every
         # request) selects one of these by key. Ordered as displayed,
@@ -95,6 +102,9 @@ module Cogworker
                 # part of CONTENT_ID's own poll_div.
                 page_body += Layout.poll_div(REDIS_CONTENT_ID, request.script_name, 'overview/redis',
                                              Overview.redis_section)
+                # Static (versions don't change while the process runs), so
+                # it needs no poll of its own.
+                page_body += Overview.gem_info_section
               end
               extra_head = layout == 'a' ? Overview.chart_head(request.script_name) : ''
               Layout.wrap('Overview', page_body, script_name: request.script_name, extra_head: extra_head)
@@ -261,17 +271,32 @@ module Cogworker
         # it pinned below the charts rather than above them.
         def redis_section
           info = Cogworker::Stats.new.redis_info
-          cards = REDIS_INFO_FIELDS.map do |field, label|
+          info_section('Redis', REDIS_INFO_FIELDS.to_h { |field, label| [label, info[field]] })
+        end
+
+        # The basics about the gem itself and what it's running on — the
+        # last block on the page, below the Redis grid.
+        def gem_info_section
+          spec = Gem.loaded_specs['cogworker']
+          values = { 'Version' => Cogworker::VERSION, 'Ruby' => "#{RUBY_VERSION} (#{RUBY_PLATFORM})" }
+          GEM_INFO_DEPENDENCIES.each { |name, label| values[label] = Gem.loaded_specs[name]&.version&.to_s }
+          values['License'] = spec&.license
+          info_section('Cogworker', values)
+        end
+
+        # A titled grid of label/value cards; a nil value renders as "n/a".
+        def info_section(title, values)
+          cards = values.map do |label, value|
             <<~HTML
               <div class="card elev-sm" style="gap: 4px;">
                 <span class="card-kicker">#{Layout.h(label)}</span>
-                <span style="font-size: 22px; font-family: var(--font-heading);">#{Layout.h(info[field] || 'n/a')}</span>
+                <span style="font-size: 15px; font-family: var(--font-heading);">#{Layout.h(value || 'n/a')}</span>
               </div>
             HTML
           end.join
           <<~HTML
             <section style="background: var(--color-surface); border-radius: var(--radius-md); box-shadow: var(--shadow-sm); padding: 16px 18px; display: flex; flex-direction: column; gap: 12px;">
-              <h4 style="margin: 0; font-family: var(--font-heading); font-weight: var(--font-heading-weight); font-size: 17px;">Redis</h4>
+              <h4 style="margin: 0; font-family: var(--font-heading); font-weight: var(--font-heading-weight); font-size: 17px;">#{Layout.h(title)}</h4>
               <div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); gap: 12px;">#{cards}</div>
             </section>
           HTML

data/lib/cogworker/web/routes/workers.rb

@@ -13,6 +13,7 @@ module Cogworker
       # honest place to look at what's actually running right now.
       module Workers
         CONTENT_ID = 'workers-content'
+        SPARKLINE_HEIGHT_PX = 32
 
         module_function
 
@@ -22,7 +23,8 @@ module Cogworker
             if hx_request?
               content
             else
-              Layout.wrap('Workers', Layout.poll_div(CONTENT_ID, request.script_name, 'workers', content),
+              Layout.wrap('Workers', Layout.poll_div(CONTENT_ID, request.script_name, 'workers', content) +
+                                     Workers.sparkline_script,
                           script_name: request.script_name)
             end
           end
@@ -90,7 +92,9 @@ module Cogworker
           work_set = Cogworker::WorkSet.new.to_a
           busy_counts = work_set.each_with_object(Hash.new(0)) { |(identity, *), h| h[identity] += 1 }
 
-          cards = process_cards(processes, busy_counts, quiet_path, resume_path, stop_path)
+          memory = Cogworker::MemoryHistory.series(processes.map(&:identity))
+
+          cards = process_cards(processes, busy_counts, memory, quiet_path, resume_path, stop_path)
 
           work_rows = work_set.map do |identity, tid, work|
             [Layout.h(identity), Layout.h(tid), Layout.h(work.job['jid']), Layout.h(work.queue),
@@ -134,19 +138,19 @@ module Cogworker
           HTML
         end
 
-        def process_cards(processes, busy_counts, quiet_path, resume_path, stop_path)
+        def process_cards(processes, busy_counts, memory, quiet_path, resume_path, stop_path)
           if processes.empty?
             return %(<p class="text-muted" style="font-size: 13px; font-style: italic;">No worker processes ) +
                    'are reporting in right now.</p>'
           end
 
           items = processes.map do |p|
-            process_card(p, busy_counts[p.identity], quiet_path, resume_path, stop_path)
+            process_card(p, busy_counts[p.identity], memory[p.identity], quiet_path, resume_path, stop_path)
           end.join
           %(<div style="display: grid; grid-template-columns: repeat(auto-fill, minmax(320px, 1fr)); gap: 12px;">#{items}</div>)
         end
 
-        def process_card(process, busy, quiet_path, resume_path, stop_path)
+        def process_card(process, busy, memory_samples, quiet_path, resume_path, stop_path)
           concurrency = process['concurrency'].to_i
           pct = concurrency.positive? ? ((busy.to_f / concurrency) * 100).round : 0
           quiet = [true, 'true'].include?(process['quiet'])
@@ -177,6 +181,7 @@ module Cogworker
                 <span>#{memory_cell(process['rss_kb'])}</span>
                 <span>#{beat_label(process.identity)}</span>
               </div>
+              #{memory_sparkline(process.identity, memory_samples)}
               <div style="height: 4px; border-radius: 2px; background: var(--color-neutral-800); overflow: hidden;">
                 <div style="height: 100%; border-radius: 2px; background: var(--color-accent); width: #{pct}%;"></div>
               </div>
@@ -199,6 +204,152 @@ module Cogworker
           end
         end
 
+        # The last 24h of this process's RSS (MemoryHistory), always the
+        # full 24h wide — one MemoryHistory slot per x position, broken
+        # wherever a slot has no sample (before the process started, or
+        # while Redis was away). An inline SVG rather than a Chart.js chart:
+        # these cards are re-rendered by every htmx poll, and a Chart.js
+        # instance per swap leaks (see CLAUDE.md's Overview charts).
+        # Stretched to the card's width with `preserveAspectRatio="none"`,
+        # so it draws only lines and fills (`non-scaling-stroke` keeps the
+        # line's width) — no point markers, which that stretch would
+        # distort; the hover marker and tooltip are HTML, positioned by
+        # `sparkline_script` from the data attributes.
+        def memory_sparkline(identity, slots)
+          slots ||= Cogworker::MemoryHistory.series([identity])[identity]
+          values = slots.map(&:last)
+          low, high = sparkline_range(values.compact)
+          <<~HTML
+            <div class="cw-spark" data-cw-spark-id="#{Layout.h(identity)}" data-cw-spark="#{Layout.h(JSON.generate(values))}"
+                 data-cw-spark-start="#{slots.first.first}" data-cw-spark-step="#{Cogworker::MemoryHistory::SAMPLE_SECONDS}"
+                 data-cw-spark-min="#{low}" data-cw-spark-max="#{high}"
+                 style="position: relative; height: #{SPARKLINE_HEIGHT_PX}px; cursor: crosshair; background: var(--color-bg); border-radius: 3px;">
+              <span class="cw-spark-label" style="position: absolute; top: 3px; left: 6px; font-size: 10px; line-height: 1; color: var(--color-neutral-500); pointer-events: none;">Memory · 24h</span>
+              <svg viewBox="0 0 100 100" preserveAspectRatio="none" aria-label="Memory, last 24 hours"
+                   style="display: block; width: 100%; height: 100%; overflow: visible;">
+                #{sparkline_segments(values, low, high)}
+              </svg>
+            </div>
+          HTML
+        end
+
+        # One line + fill per run of consecutive slots that have a sample,
+        # each point at its slot's center; a lone sample gets a short
+        # horizontal stroke across its own slot, so it's still visible.
+        def sparkline_segments(values, low, high)
+          width = 100.0 / values.size
+          runs = values.each_with_index.chunk_while { |(a, _), (b, _)| a.nil? == b.nil? }.reject { |run| run[0][0].nil? }
+          runs.map do |run|
+            points = run.map { |kb, i| [(i + 0.5) * width, (1 - ((kb - low) / (high - low))) * 100] }
+            sparkline_segment(points, width)
+          end.join
+        end
+
+        def sparkline_segment(points, width)
+          if points.one?
+            x, y = points.first
+            points = [[x - (width / 2), y], [x + (width / 2), y]]
+          end
+          coords = points.map { |x, y| format('%<x>.2f,%<y>.2f', x: x, y: y) }.join(' ')
+          area = format('%<x0>.2f,100 %<coords>s %<x1>.2f,100', x0: points.first[0], coords: coords, x1: points.last[0])
+          <<~HTML
+            <polygon points="#{area}" style="fill: var(--color-accent); opacity: 0.15;"></polygon>
+            <polyline points="#{coords}" vector-effect="non-scaling-stroke" style="fill: none; stroke: var(--color-accent); stroke-width: 1.5;"></polyline>
+          HTML
+        end
+
+        # A little headroom above and below, so the line never hugs the
+        # card's edges; a flat series gets a range around its one value.
+        def sparkline_range(values)
+          return [0.0, 1.0] if values.empty?
+
+          low = values.min.to_f
+          high = values.max.to_f
+          pad = [(high - low) * 0.1, 1024].max
+          [(low - pad).clamp(0, nil), high + pad]
+        end
+
+        # Hover readout for every `.cw-spark` on the page: the sample
+        # nearest the pointer, as a marker line + dot and a tooltip with
+        # its value and (local) time, hiding the "Memory" label meanwhile.
+        # Delegated from `document`, so it keeps working on cards htmx
+        # swaps in; a poll that replaces the card under the pointer
+        # re-shows the readout on the new one.
+        def sparkline_script
+          <<~HTML
+            <script>
+              (function () {
+                var hover = null;
+                function part(el, cls, css) {
+                  var node = el.querySelector('.' + cls);
+                  if (!node) {
+                    node = document.createElement('div');
+                    node.className = cls;
+                    node.style.cssText = 'position: absolute; pointer-events: none; ' + css;
+                    el.appendChild(node);
+                  }
+                  return node;
+                }
+                function label(el, visible) {
+                  var node = el.querySelector('.cw-spark-label');
+                  if (node) node.style.visibility = visible ? '' : 'hidden';
+                }
+                function hide(el) {
+                  el.querySelectorAll('.cw-spark-line, .cw-spark-dot, .cw-spark-tip').forEach(function (n) { n.remove(); });
+                  label(el, true);
+                }
+                function show(el, clientX) {
+                  var values = JSON.parse(el.getAttribute('data-cw-spark'));
+                  var start = parseInt(el.getAttribute('data-cw-spark-start'), 10);
+                  var step = parseInt(el.getAttribute('data-cw-spark-step'), 10);
+                  var lo = parseFloat(el.getAttribute('data-cw-spark-min'));
+                  var hi = parseFloat(el.getAttribute('data-cw-spark-max'));
+                  var rect = el.getBoundingClientRect();
+                  label(el, false);
+                  var frac = Math.min(Math.max((clientX - rect.left) / rect.width, 0), 1);
+                  var i = Math.min(Math.floor(frac * values.length), values.length - 1);
+                  var x = (i + 0.5) / values.length * rect.width;
+                  var kb = values[i];
+                  part(el, 'cw-spark-line', 'top: 0; bottom: 0; width: 1px; background: var(--color-neutral-500);').style.left = x + 'px';
+                  var dot = part(el, 'cw-spark-dot', 'width: 6px; height: 6px; border-radius: 50%; margin: -3px 0 0 -3px; background: var(--color-accent);');
+                  dot.style.display = kb === null ? 'none' : '';
+                  dot.style.left = x + 'px';
+                  if (kb !== null) dot.style.top = (1 - (kb - lo) / (hi - lo)) * rect.height + 'px';
+                  var tip = part(el, 'cw-spark-tip', 'bottom: calc(100% + 4px); z-index: 10; white-space: nowrap; font-size: 11px; ' +
+                    'padding: 3px 6px; border-radius: 4px; background: var(--color-surface); color: var(--color-text); ' +
+                    'box-shadow: var(--shadow-md); border: 1px solid var(--color-neutral-800);');
+                  var at = new Date((start + i * step) * 1000);
+                  var when = window.cogworkerFormatTime ? window.cogworkerFormatTime(at) : at.toLocaleString();
+                  tip.textContent = (kb === null ? 'no data' : (kb / 1024).toFixed(1) + 'M') + ' · ' + when;
+                  tip.style.left = x + 'px';
+                  tip.style.transform = 'translateX(-' + (frac * 100) + '%)';
+                }
+                function find(id) {
+                  var all = document.querySelectorAll('.cw-spark');
+                  for (var i = 0; i < all.length; i++) if (all[i].getAttribute('data-cw-spark-id') === id) return all[i];
+                  return null;
+                }
+                document.addEventListener('mousemove', function (evt) {
+                  var el = evt.target.closest && evt.target.closest('.cw-spark');
+                  if (hover && (!el || el.getAttribute('data-cw-spark-id') !== hover.id)) {
+                    var old = find(hover.id);
+                    if (old) hide(old);
+                    hover = null;
+                  }
+                  if (!el) return;
+                  hover = { id: el.getAttribute('data-cw-spark-id'), x: evt.clientX };
+                  show(el, evt.clientX);
+                });
+                document.addEventListener('htmx:afterSwap', function () {
+                  if (!hover) return;
+                  var el = find(hover.id);
+                  if (el) show(el, hover.x);
+                });
+              })();
+            </script>
+          HTML
+        end
+
         # `rss_kb` is absent on any process whose heartbeat predates this
         # field (or that never measured cleanly — see
         # `Heartbeat#current_rss_kb`) — render "n/a" rather than "0M".

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cogworker
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.8.0
 platform: ruby
 authors:
 - vdonec
@@ -216,12 +216,16 @@ files:
 - lib/cogworker/component.rb
 - lib/cogworker/config.rb
 - lib/cogworker/config_loader.rb
+- lib/cogworker/death_notifier.rb
 - lib/cogworker/deferred_releases.rb
+- lib/cogworker/failure_decision.rb
 - lib/cogworker/heartbeat.rb
 - lib/cogworker/history.rb
 - lib/cogworker/history/middleware.rb
 - lib/cogworker/history/storage.rb
 - lib/cogworker/job.rb
+- lib/cogworker/job_failed_error.rb
+- lib/cogworker/job_orphaned_error.rb
 - lib/cogworker/job_record.rb
 - lib/cogworker/job_util.rb
 - lib/cogworker/key_guard.rb
@@ -229,6 +233,7 @@ files:
 - lib/cogworker/logging.rb
 - lib/cogworker/lua_script.rb
 - lib/cogworker/manager.rb
+- lib/cogworker/memory_history.rb
 - lib/cogworker/middleware/chain.rb
 - lib/cogworker/owned_key.rb
 - lib/cogworker/periodic/claim.lua