cronwatch 0.9.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6df945edfb266ca737de6343276860554a4d3d740331091e1167d00fe93888a4
-  data.tar.gz: ec33c3140fb5f453de2cd0263b74a894b590408394f9f1aade76f1be2485e190
+  metadata.gz: cb5e3c0550792cfd9ba9d34f828cf099051f30b5b52cfd780373efe581e665e2
+  data.tar.gz: ba21834f9a3d3b8b641e26fa23990f34f12f2190b7ad5bbf7d9355b0bdc0685f
 SHA512:
-  metadata.gz: ba15e440f49e390715b9668a24c0ee2b1c0cf35dac907a461555d18181fc95572b0ba3fee50b00237893733ca5b864b9f35b6a4e30b0f0bee64e6eefa1f3c5b0
-  data.tar.gz: 71e579e3ebba4a279cfef5de79d611157dcc2fc8ad176fac77cc737e7e0750c9ecd3cfefed383b2ac51f2f7aef5a5a24a24a472b6b739e90c1bd007a5814f0f5
+  metadata.gz: bf853e3e1f3ac6db1f74178033a1bd92a0863a182285bfe944e872a8463f9a261319b8c5c27d5c2a2acd31d54a4f328a22b6a80d1d722170414aef18785e2dc4
+  data.tar.gz: f23c2439e7b9d0fb9f9f818cc49313c15d0e35e42dcffb74aaf246a327d4318b8951293060e269cf1e01224f5986ac25090ce9ef92c4e688f0d4e7feb8614976

data/README.md

@@ -74,7 +74,7 @@ bin/rails generate cronwatch:install    # a migration and config/initializers/cr
 bin/rails db:migrate
 ```
 
-The generator takes `--prefix` (table prefix, default `cronwatch_`) and `--database` (the database whose migrations directory gets the migration). The initializer it writes sets the ActiveRecord store, which Rails needs so web, worker and check processes see the same runs:
+The generator takes `--prefix` (table prefix, default `cronwatch_`) and `--database` (the database, from `config/database.yml`, whose migrations directory gets the migration; it also writes `app/models/cronwatch_record.rb`, an abstract class that `connects_to` that database, and names it as the store's `connection_class`, so the store reads the tables where the migration made them). The initializer it writes sets the ActiveRecord store, which Rails needs so web, worker and check processes see the same runs:
 
 ```ruby
 # config/initializers/cronwatch.rb
@@ -124,7 +124,7 @@ The same `cronwatch` as for ActiveJob, for classes that include `Sidekiq::Job` (
 cronwatch schedule: :from_scheduler, grace: "15m"   # ActiveJob or Sidekiq
 ```
 
-reads the class's one entry in Solid Queue's `config/recurring.yml` (the section for `Rails.env`) or sidekiq-cron's `config/schedule.yml`, so the schedule is written once. Fugit phrases such as `every day at 3am` or `every hour at minute 12` become the cron expression Fugit makes of them (`0 3 * * *`, `12 * * * *`), in the zone the scheduler reads them in. Each conversion is checked against Fugit's own next run times; anything CronWatch would not expect at exactly those times is refused at boot rather than approximated: every other week (`%`), days counted back from the month's end other than the last, random times (`~`), offsets instead of IANA zones, and a time that daylight saving skips (Fugit skips that run; CronWatch would report it missed). A class with no entry, or with two, also stops the boot.
+reads the class's one entry in Solid Queue's `config/recurring.yml` (the section for `Rails.env`) or sidekiq-cron's `config/schedule.yml`, so the schedule is written once. Fugit phrases such as `every day at 3am` or `every hour at minute 12` become the cron expression Fugit makes of them (`0 3 * * *`, `12 * * * *`), in the zone the scheduler reads them in. Each conversion is checked against Fugit's own next run times; anything CronWatch would not expect at exactly those times is refused at boot rather than approximated: every other week (`%`), days counted back from the month's end other than the last, random times (`~`), offsets instead of IANA zones, and a time that daylight saving skips (Fugit skips that run; CronWatch would report it missed). A class with no entry, or with two, also stops the boot; one scheduled only in another environment's section (Rails 8's `recurring.yml` has only `production:`) is declared without a schedule, and a process started with `SOLID_QUEUE_SKIP_RECURRING` still reads the entry.
 
 `Cronwatch.declare_from_scheduler!(grace: "10m")` in the initializer watches every enabled entry once the app has booted: classes that call `cronwatch` declare themselves, other classes are named as `cronwatch` would name them and their runs are recorded, and Solid Queue `command:` tasks are named after their key. `except:` leaves keys out. `Cronwatch::Scheduler.sources` sets where to read, when the defaults (Solid Queue's file when Solid Queue is loaded, sidekiq-cron's when sidekiq-cron is) are not right.
 
@@ -168,7 +168,7 @@ Rails.application.routes.draw do
 end
 ```
 
-Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. Without a token, while the environment (`Rails.env`, else `RAILS_ENV` or `RACK_ENV`) is `development` or `test`, it makes a token of its own and prints a sign-in link to standard output on its first request (open it once; unless `origin:` is set or the request's host is loopback, the link leaves the host out, since a client chooses it); anywhere else it answers 503. To rely on the app's own sign in, mount it behind that (Devise's `authenticate` block, or a routing constraint) and pass `token: nil`. The URLs, JSON shapes, headers and CSRF rules are the SDK's, so the MCP server reads it unchanged. The request's origin, which the CSRF check compares against, is what Rack reports and so already follows `X-Forwarded-Host` and `X-Forwarded-Proto`; pass `origin: "https://app.example.com"` to pin it instead. `GET /cronwatch/api/check` with a bearer (the token or `CRON_SECRET`) runs the check, for an outside cron.
+Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. Without a token, while the environment (`Rails.env`, else `RAILS_ENV` or `RACK_ENV`) is `development` or `test`, it makes a token of its own and prints a sign-in link to standard output on its first request (open it once; unless `origin:` is set or the request's host is loopback, the link leaves the host out, since a client chooses it); anywhere else it answers 503. Puma, Unicorn, Thin and rackup set `RACK_ENV` to `development` when nothing names an environment, production included, so under one of them `RACK_ENV` alone does not count: `Rails.env`, `RAILS_ENV` or `APP_ENV` must say `development` or `test` (or `RACK_ENV` say `test`). To rely on the app's own sign in, mount it behind that (Devise's `authenticate` block, or a routing constraint) and pass `token: nil`. The URLs, JSON shapes, headers and CSRF rules are the SDK's, so the MCP server reads it unchanged. The request's origin, which the CSRF check compares against, is what Rack reports and so already follows `X-Forwarded-Host` and `X-Forwarded-Proto`; pass `origin: "https://app.example.com"` to pin it instead. `GET /cronwatch/api/check` with a bearer (the token or `CRON_SECRET`) runs the check, for an outside cron.
 
 There is no `handler()` as in the TypeScript SDK: for a job triggered over HTTP, wrap the controller action's body in `CW.job(...).run` (declared once, at boot) and check the bearer in the controller.
 

data/lib/cronwatch/alerts/discord.rb

@@ -9,6 +9,10 @@ module Cronwatch
         missed: 0xb7791f, failed: 0xc62828, stuck: 0xc62828, slow: 0xb7791f, over_budget: 0xb7791f, recovered: 0x1f8a4c,
       }.freeze
 
+      # The longest embed description Discord takes. The title (under 256)
+      # and it stay well inside the embed's 6000.
+      DESCRIPTION_MAX = 4096
+
       attr_reader :name
 
       def initialize(webhook_url:, link: nil, http: HTTP.default)
@@ -24,8 +28,7 @@ module Cronwatch
         url = @link&.call(alert)
         embed = { "title" => alert.title }
         embed["url"] = url if url && !url.empty?
-        triage = alert.triage && !alert.triage.empty? ? "\n**Triage:** #{Discord.escape_markdown(JS.head16(alert.triage, 1000))}" : ""
-        embed["description"] = "```\n#{Discord.code_block_safe(JS.head16(alert.message, 3800))}\n```#{triage}"
+        embed["description"] = Discord.embed_description(alert)
         embed["color"] = COLOR[alert.type]
         embed["timestamp"] = JS.iso(alert.at)
         payload = {
@@ -38,6 +41,16 @@ module Cronwatch
         raise "Discord webhook answered #{response.status}: #{JS.head16(response.body.to_s, 200)}" unless response.ok?
       end
 
+      # The message in a code block, then the triage. Each part has its own
+      # cap, and escaping can grow both, so the whole is held to
+      # DESCRIPTION_MAX (in UTF-16 units) by cutting the message's block,
+      # never the triage: Discord refuses a longer one on every retry.
+      def self.embed_description(alert)
+        triage = alert.triage && !alert.triage.empty? ? "\n**Triage:** #{escape_markdown(JS.head16(alert.triage, 1000))}" : ""
+        room = DESCRIPTION_MAX - "```\n".length - "\n```".length - JS.length16(triage)
+        "```\n#{JS.head16(code_block_safe(JS.head16(alert.message, 3800)), room)}\n```#{triage}"
+      end
+
       # Breaks up ``` so text inside a code block cannot close it.
       def self.code_block_safe(text)
         text.gsub("```", "`\u200b`\u200b`")

data/lib/cronwatch/client.rb

@@ -25,7 +25,7 @@ module Cronwatch
     CHANNEL_TIMEOUT_MS = 15_000
     PRUNE_INTERVAL_MS = 60 * 60_000
     # Undelivered alerts kept per job for retry; the oldest go first.
-    MAX_UNDELIVERED = 20
+    MAX_UNDELIVERED = Evaluate::MAX_UNDELIVERED
     # Wall-clock time one check spends retrying undelivered alerts, across
     # every job. Once it is spent the rest wait for the next check.
     RETRY_BUDGET_MS = 20_000
@@ -198,7 +198,7 @@ module Cronwatch
                     duration_ms: nil, error: nil, output: nil, metrics: {}, trigger: trigger)
       recorded = false
       begin
-        sync(definition)
+        sync(definition, confirm: true)
         @store.insert_run(run.dup)
         recorded = true
       rescue StandardError => e
@@ -232,7 +232,7 @@ module Cronwatch
       run.duration_ms = Evaluate.run_duration(started_at, finished_at)
       run.metrics = recorder.metrics
       returned = result.is_a?(String) ? Output.utf8(result) : nil
-      run.output = recorder.output || (returned && Output.cap(returned))
+      run.output = recorder.output || returned
 
       conclude(definition, run, result, error, threw, recorder.expect_text || returned, failure: failure)
       begin
@@ -316,9 +316,10 @@ module Cronwatch
     # alone and reported. A finished run is judged as if it had been wrapped
     # here (expect, failures, duration, budgets) and its output and error are
     # redacted the same way; one finishing after a check marked it timeout is
-    # judged only when it succeeded, as RunHandle#finish does. `evaluate:
-    # false` stores it without judging it, for history imported on first
-    # sight. Returns the alerts it sent.
+    # judged only when it succeeded, as RunHandle#finish does. A metric that
+    # is not a finite number raises ArgumentError before anything is written,
+    # as JobContext#metric does. `evaluate: false` stores it without judging
+    # it, for history imported on first sight. Returns the alerts it sent.
     #
     # `run` is a Cronwatch::Run, or a hash of its fields (camelCase or snake_case keys).
     def record_run(run, evaluate: true)
@@ -329,6 +330,16 @@ module Cronwatch
       if input.id.to_s.include?("\0")
         raise ArgumentError, "record_run: run ids cannot contain a NUL character (job \"#{input.job}\")"
       end
+      # Refused as JobContext#metric refuses them: a store keeps NaN and Infinity as null.
+      unless input.metrics.nil? || input.metrics.is_a?(Hash)
+        raise ArgumentError, "record_run: metrics must be a Hash of numbers (job \"#{input.job}\", run \"#{input.id}\")"
+      end
+
+      (input.metrics || {}).each do |metric, value|
+        next if value.is_a?(Numeric) && value.real? && JS.finite?(value)
+
+        raise ArgumentError, "record_run: metric \"#{metric}\" must be a finite number (job \"#{input.job}\", run \"#{input.id}\")"
+      end
 
       sync(declared)
       run = input.dup
@@ -343,8 +354,8 @@ module Cronwatch
           run.error = unmet
         end
       end
-      run.output = Output.strip_nul(@redact.call(Output.cap(run.output))) unless run.output.nil?
-      run.error = Output.strip_nul(@redact.call(Output.cap(run.error))) unless run.error.nil?
+      run.output = Output.redact_and_cap(run.output, @redact) unless run.output.nil?
+      run.error = Output.redact_and_cap(run.error, @redact) unless run.error.nil?
       definition = Serialize.to_stored(declared)
 
       stored = @store.get_run(run.id)
@@ -414,16 +425,18 @@ module Cronwatch
     def jobs_with_runs(limit = 20)
       after_fork_check
       ensure_ready
-      defined_jobs.each { |definition| sync(definition) }
+      jobs = stored_jobs
       at = now
       count = clamp_limit(limit, 20, 0)
-      @store.list_jobs.map { |stored| snapshot(stored, at, count) }
+      jobs.map { |stored| snapshot(stored, at, count) }
     end
 
+    # A job's summary, or nil for one the store does not have. One declared
+    # here and forgotten elsewhere is written again, as stored_jobs does.
     def job_summary(name)
       ensure_ready
       definition = @registry.synchronize { @definitions[name] }
-      sync(definition) if definition
+      sync(definition, confirm: true) if definition
       stored = @store.get_job(name)
       return nil unless stored
 
@@ -441,7 +454,8 @@ module Cronwatch
       @store.get_run(id)
     end
 
-    # Stop alerts for a job for a while. State keeps updating underneath.
+    # Stop alerts for a job for a while. State keeps updating underneath. The
+    # end is a whole millisecond, held at 2**53 - 1 (see Evaluate.silence_end).
     #   silence("nightly-report", for: "2h")   # or silence("nightly-report", "2h")
     def silence(name, duration = nil, **options)
       unknown = options.keys - SILENCE_OPTIONS
@@ -450,14 +464,17 @@ module Cronwatch
 
       duration = options[:for] if duration.nil?
       ms = Duration.parse(duration, "silence duration")
-      patch_state(name) { |state| state.silenced_until = now + ms }
+      patch_state(name) { |state| state.silenced_until = Evaluate.silence_end(now, ms) }
     end
 
     def unsilence(name)
       patch_state(name) { |state| state.silenced_until = nil }
     end
 
-    # Remove a job and its runs from the store. A job still declared in code comes back on its next run.
+    # Remove a job and its runs from the store. A job still declared in code
+    # comes back: here on its next run, and in any other process that
+    # declares it on its next run there, or at that process's next check or
+    # dashboard read.
     def forget(name)
       ensure_ready
       @registry.synchronize do
@@ -477,6 +494,8 @@ module Cronwatch
       after_fork_check
       ms = [5_000, Duration.parse(every, "check interval")].max
       @ticker_lock.synchronize do
+        # One whose thread ended (killed from outside) is replaced.
+        @ticker = nil if @ticker && !@ticker.alive?
         if @ticker
           unless @ticker_ms == ms
             report(ArgumentError.new("start(#{every.inspect}) ignored: already checking every #{Duration.format(@ticker_ms)}; " \
@@ -493,7 +512,10 @@ module Cronwatch
         end
         @ticker = Ticker.new(ms / 1000.0, @first_tick_s) do
           check
-        rescue StandardError => e
+        # Anything, a source's LoadError or a SystemStackError included, is
+        # reported and the next tick comes as usual: the thread must not end
+        # silently and leave missed and stuck unchecked in this process.
+        rescue Exception => e # rubocop:disable Lint/RescueException
           report(e, "check")
         end
       end
@@ -501,18 +523,23 @@ module Cronwatch
     end
 
     def stop
-      after_fork_check
-      ticker = @ticker_lock.synchronize do
-        current = @ticker
-        @ticker = nil
-        current
-      end
-      ticker&.stop
+      stop_ticker
       nil
     end
 
+    # Stop the interval, wait for a check already under way (a failed one is
+    # reported by whoever started it), then close the store. Runs being
+    # recorded are not waited for.
     def close
-      stop
+      stop_ticker&.join
+      flight = @check_lock.synchronize { @checking }
+      if flight && !flight.owner.equal?(Thread.current)
+        begin
+          flight.value
+        rescue StandardError
+          nil
+        end
+      end
       @store.close if @store.respond_to?(:close)
       nil
     end
@@ -534,6 +561,18 @@ module Cronwatch
 
     private
 
+    # Stops the interval thread and returns it, or nil when there was none.
+    def stop_ticker
+      after_fork_check
+      ticker = @ticker_lock.synchronize do
+        current = @ticker
+        @ticker = nil
+        current
+      end
+      ticker&.stop
+      ticker
+    end
+
     # Locks, the check in flight, the interval thread and the channel and
     # triage threads belong to one process. A forked child (Puma, Unicorn,
     # Sidekiq) starts with fresh ones, so start and check work there.
@@ -653,11 +692,22 @@ module Cronwatch
     # its own. The writes of one name take turns (the name's lock in
     # @syncing, held across the write), so one still under way cannot land
     # after a later one; and a name declared again while its write was under
-    # way is still to be written.
-    def sync(definition)
+    # way is still to be written. A name is marked as written only while
+    # that same declaration stands, so a forget that lands during the write
+    # (deleting the row after it) leaves the name to be written again, as
+    # does one forgotten before it.
+    #
+    # With `confirm`, as a run starts, a name already written is read back:
+    # another process may have forgotten the job since, and a job still
+    # declared here comes back on its next run.
+    def sync(definition, confirm: false)
       ensure_ready
       name = definition.name
-      return if @registry.synchronize { @synced.include?(name) }
+      if @registry.synchronize { @synced.include?(name) }
+        return if !confirm || @store.get_job(name)
+
+        @registry.synchronize { @synced.delete(name) }
+      end
 
       after_fork_check
       turn = @registry.synchronize { @syncing[name] ||= Mutex.new }
@@ -666,8 +716,28 @@ module Cronwatch
         next if standing.nil?
 
         @store.upsert_job(Serialize.to_stored(standing), now)
-        @registry.synchronize { @synced << name if @definitions.fetch(name, standing).equal?(standing) }
+        @registry.synchronize { @synced << name if @definitions[name].equal?(standing) }
+      end
+    end
+
+    # Every stored job, once each declaration has been written. A job
+    # declared here that the store no longer has was forgotten by another
+    # process after this one wrote it: it is written again, as its next run
+    # would, so it is checked and shown while any process still declares it.
+    def stored_jobs
+      defined_jobs.each { |definition| sync(definition) }
+      jobs = @store.list_jobs
+      listed = jobs.to_set(&:name)
+      missing = defined_jobs.reject { |definition| listed.include?(definition.name) }
+      return jobs if missing.empty?
+
+      missing.each do |definition|
+        name = definition.name
+        # Not one forgotten here meanwhile.
+        standing = @registry.synchronize { @definitions[name].equal?(definition) && @synced.delete(name) }
+        sync(definition) if standing
       end
+      @store.list_jobs
     end
 
     # Runs the block while holding the job's lock, so two runs (or a run and a
@@ -743,11 +813,6 @@ module Cronwatch
       end
     end
 
-    # Identifies an alert across retries.
-    def alert_key(alert)
-      "#{alert.type}|#{alert.at}|#{alert.run&.id}"
-    end
-
     # record_run for a run already stored.
     def record_over(definition, stored, run, evaluate)
       if stored.job != run.job
@@ -795,11 +860,12 @@ module Cronwatch
     end
 
     # Sets a finished run's status and error from how it ended, then redacts
-    # its output and error. Shared by execute and RunHandle#finish.
+    # its output and error and caps them, in that order. Shared by execute
+    # and RunHandle#finish.
     def conclude(definition, run, result, error, threw, expect_text, failure: nil)
       if threw
         run.status = :failed
-        run.error = Output.error_message(error)
+        run.error = Output.describe_error(error)
       elsif (problem = failure&.call(result))
         run.status = :failed
         run.error = Output.utf8(problem.to_s)
@@ -813,9 +879,10 @@ module Cronwatch
         end
       end
       # Redacted after the expect check, so a rule can still match what was
-      # logged. NULs go last, so not even a custom redact can store one.
-      run.output = Output.strip_nul(@redact.call(run.output)) unless run.output.nil?
-      run.error = Output.strip_nul(@redact.call(run.error)) unless run.error.nil?
+      # logged, and before the cap, so the cut cannot keep half a secret. NULs
+      # go last, so not even a custom redact can store one.
+      run.output = Output.redact_and_cap(run.output, @redact) unless run.output.nil?
+      run.error = Output.redact_and_cap(run.error, @redact) unless run.error.nil?
     end
 
     # Writes a finished run and evaluates it. `recorded` says whether its
@@ -868,7 +935,7 @@ module Cronwatch
                     duration_ms: nil, error: nil, output: nil, metrics: {}, trigger: trigger)
       recorded = false
       begin
-        sync(definition)
+        sync(definition, confirm: true)
         @store.insert_run(run.dup)
         recorded = true
       rescue StandardError => e
@@ -955,13 +1022,14 @@ module Cronwatch
       failed, result, error = RunHandle.read_outcome(outcome)
       finished_at = now
       returned = result.is_a?(String) ? Output.utf8(result) : nil
-      added = recorder.output || (returned && Output.cap(returned))
+      added = recorder.output || returned
       run = from.dup
       run.status = :running
       run.finished_at = finished_at
       run.duration_ms = Evaluate.run_duration(from.started_at, finished_at)
       run.error = nil
-      run.output = join_output(from.output, added)
+      # Capped by conclude, after it is redacted.
+      run.output = join_lines(from.output, added)
       run.metrics = (from.metrics || {}).merge(recorder.metrics)
       expect_text = join_lines(head, join_lines(from.output, recorder.expect_text || returned))
       conclude(definition, run, result, error, failed, expect_text)
@@ -998,7 +1066,7 @@ module Cronwatch
       end
 
       updated = stored.dup
-      updated.output = join_output(stored.output, Output.strip_nul(@redact.call(lines))) unless lines.nil?
+      updated.output = join_output(stored.output, Output.redact_and_cap(lines, @redact)) unless lines.nil?
       updated.metrics = (stored.metrics || {}).merge(metrics)
       write_run_if(updated, [:running])
     rescue StandardError => e
@@ -1035,21 +1103,45 @@ module Cronwatch
     end
 
     # Evaluate a finished run (ok, failed, or timed out by a check), already
-    # written, against the job's state and send what that produces. Never raises.
+    # written, against the job's state and send what that produces. The
+    # alerts are written with that state (see outbox). Never raises.
     def finish_run(definition, run, at)
-      drafts = nil
+      held = nil
       begin
         past = nil
-        _, drafts = update_state(run.job) do |previous|
+        _, held = update_state(run.job) do |previous|
           past ||= history(run)
-          settled = Evaluate.apply_silence(previous, Evaluate.on_run_finish(definition, run, previous, past, at), at)
-          [settled.state, settled.alerts]
+          outbox(Evaluate.apply_silence(previous, Evaluate.on_run_finish(definition, run, previous, past, at), at), definition, at)
         end
       rescue StandardError => e
         report(e, "evaluating #{run.job}")
         return []
       end
-      dispatch(drafts, definition, at)
+      report_dropped(run.job, held.dropped)
+      dispatch(run.job, held.alerts, at)
+    end
+
+    # Alerts written with the state that opened their conditions, and how
+    # many older ones the queue let go.
+    Outbox = Struct.new(:alerts, :dropped)
+
+    # An evaluation as it is written: its drafts composed into alerts and held
+    # in the same state (Evaluate.hold_alerts), so the write that opens a
+    # condition also keeps its alerts, and a process that stops before
+    # sending them does not lose them. Called inside update_state, so it only
+    # computes. Returns `[state, Outbox]`.
+    def outbox(settled, definition, at)
+      alerts = settled.alerts.map { |draft| Format.compose_alert(draft, definition, at) }
+      held = Evaluate.hold_alerts(settled.state, alerts, now + Evaluate::SEND_LEASE_MS, @defer_delivery)
+      [held.state, Outbox.new(alerts, held.dropped)]
+    end
+
+    # Reports alerts let go because a job's queue was full.
+    def report_dropped(name, dropped)
+      return unless dropped.positive?
+
+      report(RuntimeError.new("#{dropped} undelivered alert#{dropped == 1 ? "" : "s"} for #{name} dropped: " \
+                              "only the newest #{Evaluate::MAX_UNDELIVERED} are kept for retry"), "alert queue for #{name}")
     end
 
     # The runs before `run`, newest first, with up to BASELINE_WINDOW
@@ -1078,10 +1170,18 @@ module Cronwatch
 
       # Runs that never reported back. One that cannot be judged (its job's
       # stored timeout no longer parses, say) is reported and skipped.
-      @store.running_runs.each do |run|
-        declared = @registry.synchronize { @definitions[run.job] }
-        definition = declared ? Serialize.to_stored(declared) : @store.get_job(run.job)&.definition
-        next if definition.nil? || !Evaluate.stuck?(definition, run, at)
+      @store.running_runs.each do |listed|
+        declared = @registry.synchronize { @definitions[listed.job] }
+        definition = declared ? Serialize.to_stored(declared) : @store.get_job(listed.job)&.definition
+        next if definition.nil?
+
+        evaluable(listed.job, definition)
+        next unless Evaluate.stuck?(definition, listed, at)
+
+        # Read again just before the write: lines and metrics flushed since the
+        # list was read (while earlier stuck runs were sent, say) are kept.
+        run = @store.get_run(listed.id)
+        next if run.nil? || run.status != :running || run.job != listed.job
 
         timeout = Evaluate.timeout_ms(definition)
         run.status = :timeout
@@ -1093,7 +1193,7 @@ module Cronwatch
 
         alerts.concat(finish_run(definition, run, at))
       rescue StandardError => e
-        report(e, "checking #{run.job}")
+        report(e, "checking #{listed.job}")
       end
 
       # Each job on its own: one that cannot be evaluated (a stored schedule
@@ -1102,17 +1202,22 @@ module Cronwatch
       # the others.
       jobs = []
       retries = RetryBudget.new(0)
-      @store.list_jobs.each do |stored|
+      stored_jobs.each do |stored|
+        evaluable(stored.name, stored.definition)
         recent = @store.list_runs(stored.name, Evaluate::BASELINE_WINDOW)
         next_expected_at = nil
-        state, drafts = update_state(stored.name) do |previous|
+        state, held = update_state(stored.name) do |previous|
           evaluation = Evaluate.on_check(stored.definition, stored, recent.first, previous, at)
           next_expected_at = evaluation.next_expected_at
           settled = Evaluate.apply_silence(previous, evaluation, at)
-          [settled.state, settled.alerts]
+          # Alerts a process stopped sending part way go back to the retry queue.
+          released = Evaluate.release_sending(settled.state, now)
+          following, out = outbox(Evaluate::Evaluation.new(state: released.state, alerts: settled.alerts), stored.definition, at)
+          [following, Outbox.new(out.alerts, released.dropped + out.dropped)]
         end
+        report_dropped(stored.name, held.dropped)
         alerts.concat(retry_undelivered(stored.name, state, at, retries))
-        alerts.concat(dispatch(drafts, stored.definition, at))
+        alerts.concat(dispatch(stored.name, held.alerts, at))
         jobs << Evaluate.summarize(stored, recent, state, next_expected_at, at)
       rescue StandardError => e
         report(e, "checking #{stored.name}")
@@ -1139,6 +1244,7 @@ module Cronwatch
       recent = []
       begin
         recent = @store.list_runs(stored.name, [count, Evaluate::BASELINE_WINDOW].max)
+        evaluable(stored.name, stored.definition)
         state = read_state(stored.name)
         next_expected_at = Evaluate.on_check(stored.definition, stored, recent.first, state, at).next_expected_at
         JobWithRuns.new(job: Evaluate.summarize(stored, recent, state, next_expected_at, at), runs: recent.first(count))
@@ -1148,6 +1254,13 @@ module Cronwatch
       end
     end
 
+    # Raises for a stored definition that was not a JSON object (see
+    # JobDefinition.from_h), so that one job is reported and shown as
+    # failing, and the others are checked as usual.
+    def evaluable(name, definition)
+      raise ArgumentError, "job \"#{name}\": its stored definition is not a JSON object" if definition.unreadable?
+    end
+
     # The summary of a job whose evaluation failed, from whatever can still be read.
     def unevaluable(stored, at)
       recent = begin
@@ -1174,27 +1287,25 @@ module Cronwatch
       state
     end
 
-    # Compose, triage and send each draft. The state was saved before this
-    # (update_state), so a slow channel holds up nothing else; afterwards only
-    # the delivery fields are written back, onto a fresh read of the state.
-    def dispatch(drafts, definition, at)
-      return [] if drafts.empty?
+    # Triage and send each alert the outbox holds (see outbox). The state,
+    # with the alerts in it, was saved before this, so a slow channel holds up
+    # nothing else; afterwards only the delivery fields are written back,
+    # onto a fresh read of the state, and the alerts leave `sending`. Triage
+    # is made here, never stored with the held alert: the write that opens a
+    # condition cannot wait for it, and a retry triages an alert that has
+    # none. With deliver: :check the alerts were queued for a check elsewhere
+    # instead.
+    def dispatch(name, alerts, at)
+      return alerts if alerts.empty? || @defer_delivery
 
-      composed = []
       delivered = []
       failed = []
-      drafts.each do |draft|
-        alert = Format.compose_alert(draft, definition, at)
-        if @defer_delivery
-          failed << alert
-        else
-          add_triage(alert, @triage_timeout_ms) if @triage && alert.type != :recovered
-          (deliver(alert) ? delivered : failed) << alert
-        end
-        composed << alert
+      alerts.each do |alert|
+        add_triage(alert, @triage_timeout_ms) if @triage && alert.type != :recovered
+        (deliver(alert) ? delivered : failed) << alert
       end
-      record_delivery(definition.name, delivered, failed, [], at)
-      composed
+      record_delivery(name, delivered, failed, [], at)
+      alerts
     end
 
     # The wall-clock milliseconds one check has spent retrying, across its jobs.
@@ -1230,24 +1341,16 @@ module Cronwatch
     end
 
     # Mark delivered alerts done, drop stale ones, and keep failed ones for the
-    # next check. A failed alert replaces its stored copy, so a triage made on
-    # this attempt is kept. last_alert_at moves only on a delivery.
+    # next check, taking them all out of `sending` (Evaluate.record_sent). A
+    # failed alert replaces its stored copy, so a triage made on this attempt
+    # is kept. last_alert_at moves only on a delivery. When this write fails,
+    # alerts still in `sending` are retried once their lease runs out.
     def record_delivery(name, delivered, failed, dropped, at)
       _, trimmed = update_state(name) do |previous|
-        state = Evaluate.normalize_state(previous, name)
-        done = (delivered + dropped).map { |a| alert_key(a) }.to_set
-        retried = failed.to_h { |a| [alert_key(a), a] }
-        kept = state.undelivered.reject { |a| done.include?(alert_key(a)) }.map { |a| retried.fetch(alert_key(a), a) }
-        known = kept.map { |a| alert_key(a) }.to_set
-        kept.concat(failed.reject { |a| known.include?(alert_key(a)) })
-        state.undelivered = kept.last(MAX_UNDELIVERED)
-        state.last_alert_at = at if delivered.any?
-        [state, [0, kept.length - MAX_UNDELIVERED].max]
-      end
-      if trimmed.positive?
-        report(RuntimeError.new("#{trimmed} undelivered alert#{trimmed == 1 ? "" : "s"} for #{name} dropped: " \
-                                "only the newest #{MAX_UNDELIVERED} are kept for retry"), "alert queue for #{name}")
+        held = Evaluate.record_sent(Evaluate.normalize_state(previous, name), delivered, failed, dropped, at)
+        [held.state, held.dropped]
       end
+      report_dropped(name, trimmed)
     rescue StandardError => e
       report(e, "recording alert delivery for #{name}")
     end
@@ -1272,14 +1375,20 @@ module Cronwatch
           Thread.current.report_on_exception = false
           send_to(channel, alert)
           outcomes[i] = true
-        rescue StandardError, ScriptError => e
+        # Anything the channel raised, SystemStackError and exit included, is
+        # a failed send: raised again through join, it would replace the
+        # job's own error and lose the alert.
+        rescue Exception => e # rubocop:disable Lint/RescueException
           outcomes[i] = e
         end
       end
       deadline = AbortSignal.monotonic + (@channel_timeout_ms / 1000.0)
       results = threads.each_with_index.map do |thread, i|
         next outcomes[i] if thread.nil?
-        next outcomes[i] if thread.join([deadline - AbortSignal.monotonic, 0].max)
+        if thread.join([deadline - AbortSignal.monotonic, 0].max)
+          # A thread killed from outside leaves no outcome: not sent.
+          next outcomes[i] || RuntimeError.new("the channel's thread ended without sending")
+        end
 
         @sending_lock.synchronize { @abandoned[i] = thread }
         TimeoutError.new("timed out after #{@channel_timeout_ms}ms")
@@ -1334,21 +1443,28 @@ module Cronwatch
       thread = Thread.new do
         Thread.current.report_on_exception = false
         outcome = [:ok, @triage.call(context)]
-      rescue StandardError, ScriptError => e
+      # Anything triage raised counts as no diagnosis, as a channel's does.
+      rescue Exception => e # rubocop:disable Lint/RescueException
         outcome = [:error, e]
       end
       unless thread.join(timeout_ms / 1000.0)
         @sending_lock.synchronize { @abandoned[:triage] = thread }
         raise TimeoutError, "timed out after #{timeout_ms}ms"
       end
-      raise outcome[1] if outcome[0] == :error
+      outcome ||= [:error, RuntimeError.new("the triage thread ended without an answer")]
+      return no_triage(alert, signal, outcome[1]) if outcome[0] == :error
 
       diagnosis = outcome[1]
       alert.triage_result = diagnosis.is_a?(String) && !diagnosis.empty? ? Output.utf8(diagnosis) : nil
     rescue StandardError => e
+      no_triage(alert, signal, e)
+    end
+
+    # A triage that gave nothing: the alert goes without one, and why is reported.
+    def no_triage(alert, signal, error)
       signal&.abort!
       alert.triage_result = nil
-      report(e, "triage for #{alert.job}")
+      report(error, "triage for #{alert.job}")
     end
 
     # A whole number in range, or the fallback for anything that is not a number.