cronwatch 0.12.5 → 0.13.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a99bb19743d769cd5e3c5c105c83410ad2882378e3d05707f9b607523a2372d4
-  data.tar.gz: 895572e99a4849f4ab93f072289236d6a1e21c45fc5d4b57c11d449469a687d1
+  metadata.gz: 2550285307bd17b80e7aace7fe3ed098853b83258f1e9b9030d666739b415020
+  data.tar.gz: 4392543773b56b11d1be70af9b67f0077ead1fff1efd0477b4a33733daf7337b
 SHA512:
-  metadata.gz: 0f93f594c2a75ff6b7f86aa30ae487f9ae9828e31c0fa3fb21e32a732b69232bd0b442f6d0be1fb2188f545e098d7f1d59179d3ed44627bdc46b259071384a2b
-  data.tar.gz: c994fff472654f78cd23de0f3be5575cb784d0f0cb16e705ed76cd5bf4eb130ef5513dc47a94b48cbf1c4c123463a04e104ae7b4e70178eae3df93d42b73a46b
+  metadata.gz: 9d9eb6e74fd3c89fbe884c6c8d0801a2b82a47860fb00d88e181a5e91846d67cb1c072cbbbe8c16dfda467c6d09b7d382f71fee0072638130003552c6c6f0596
+  data.tar.gz: 1971c1a964d90823ce29c32d8661fa12ec19460557ff3650c842479cdf166556ca550ca24deab7aeaf613aa7c51ab510a5c76a8883353123ed56657849ea69af

data/README.md

@@ -117,7 +117,7 @@ class HardWorker
 end
 ```
 
-The same `cronwatch` as for ActiveJob, for classes that include `Sidekiq::Job` (or `Sidekiq::Worker`) directly. `Cronwatch::Sidekiq::ServerMiddleware` records each `perform` with the trigger `"sidekiq"` and raises the job's error on to Sidekiq, so retries and error handlers behave as before. In Rails the Railtie adds it to the Sidekiq server; elsewhere add it in `Sidekiq.configure_server` and call `Cronwatch::Sidekiq.ready!` once `Cronwatch.configure` has run. An ActiveJob class on Sidekiq's adapter keeps `Cronwatch::ActiveJob`; the middleware passes ActiveJob's wrapper through, so it is recorded once.
+The same `cronwatch` as for ActiveJob, for classes that include `Sidekiq::Job` (or `Sidekiq::Worker`) directly. `Cronwatch::Sidekiq::ServerMiddleware` records each `perform` with the trigger `"sidekiq"` and raises the job's error on to Sidekiq, so retries and error handlers behave as before. In Rails the Railtie adds it to the Sidekiq server; elsewhere add it in `Sidekiq.configure_server` and call `Cronwatch::Sidekiq.ready!` once `Cronwatch.configure` has run. An ActiveJob class on Sidekiq's adapter keeps `Cronwatch::ActiveJob`; the middleware passes ActiveJob's wrapper through, so it is recorded once. A job stopped by Sidekiq's shutdown (`Sidekiq::Shutdown`, raised into a job still running when a deploy outlasts Sidekiq's timeout) goes back on the queue and runs again after the restart, so its run is recorded as failed (`Interrupted: Sidekiq::Shutdown`) but not judged: it opens no failed alert and does not count toward `failures_before_alert`. The same holds for an ActiveJob class on Sidekiq's adapter.
 
 ### Schedules from the scheduler's config
 
@@ -169,7 +169,7 @@ Rails.application.routes.draw do
 end
 ```
 
-Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. Without a token, while the environment (the first of `CRONWATCH_ENV`, `APP_ENV`, then `Rails.env` (when Rails is loaded), `RAILS_ENV`, or `RACK_ENV` that holds more than spaces, trimmed and lowercased) is `development`, `dev`, `local`, `test`, or `testing`, it makes a token of its own and prints a sign-in link to standard output on its first request (open it once; unless `origin:` is set or the request's host is loopback, the link leaves the host out, since a client chooses it); anywhere else it answers 503. Puma, Unicorn, Thin, and rackup set `RACK_ENV` to `development` when nothing names an environment, production included, so under one of them `RACK_ENV=development` alone does not count: `CRONWATCH_ENV`, `APP_ENV`, `Rails.env`, or `RAILS_ENV` must say development (or `RACK_ENV` say `test`, `dev`, or `local`). To rely on the app's own sign in, mount it behind that (Devise's `authenticate` block, or a routing constraint) and pass `token: nil`. The URLs, JSON shapes, headers, and CSRF rules are the SDK's, so the MCP server reads it unchanged. The request's origin, which the CSRF check compares against, is what Rack reports and so already follows `X-Forwarded-Host` and `X-Forwarded-Proto`; pass `origin: "https://app.example.com"` to pin it instead. `GET /cronwatch/api/check` with a bearer (the token or `CRON_SECRET`) runs the check, for an outside cron.
+Set `CRONWATCH_TOKEN` to a long random string and open `/cronwatch?token=<it>` once; the browser keeps a cookie. The token and `CRON_SECRET` are trimmed of the whitespace around them, so one read from a file with a trailing newline still works. Without a token, while the environment (the first of `CRONWATCH_ENV`, `APP_ENV`, then `Rails.env` (when Rails is loaded), `RAILS_ENV`, or `RACK_ENV` that holds more than spaces, trimmed and lowercased) is `development`, `dev`, or `local` (not `test` or `testing`: a test run or CI needs a token, as production does), it makes a token of its own and prints a sign-in link to standard output on its first request (open it once; unless `origin:` is set or the request's host is loopback, the link leaves the host out, since a client chooses it); anywhere else it answers 503. Puma, Unicorn, Thin, and rackup set `RACK_ENV` to `development` when nothing names an environment, production included, so under one of them `RACK_ENV=development` alone does not count: `CRONWATCH_ENV`, `APP_ENV`, `Rails.env`, or `RAILS_ENV` must say development (or `RACK_ENV` say `dev` or `local`). To rely on the app's own sign in, mount it behind that (Devise's `authenticate` block, or a routing constraint) and pass `token: nil`. The URLs, JSON shapes, headers, and CSRF rules are the SDK's, so the MCP server reads it unchanged. The request's origin, which the CSRF check compares against, is the request's own scheme and `Host`. Behind a proxy, pass `trust_proxy: true` to read the first `X-Forwarded-Proto` and `X-Forwarded-Host` instead (only when the proxy sets or overwrites both, since a client can send them too), or `origin: "https://app.example.com"` to pin it. Under Rails `trust_proxy` defaults to true, as Rails itself trusts those headers; anywhere else it defaults to false, as in the SDK. `GET /cronwatch/api/check` with a bearer (the token or `CRON_SECRET`) runs the check, for an outside cron.
 
 There is no `handler()` as in the TypeScript SDK: for a job triggered over HTTP, wrap the controller action's body in `CW.job(...).run` (declared once, at boot) and check the bearer in the controller.
 
@@ -180,7 +180,7 @@ In any other Rack app:
 map("/cronwatch") { run CW.routes }
 ```
 
-`client.routes(**options)` (`token:`, `base_path:`, `origin:`) is the one way to build the dashboard, as `routes()` is in every language.
+`client.routes(**options)` (`token:`, `base_path:`, `origin:`, `trust_proxy:`) is the one way to build the dashboard, as `routes()` is in every language.
 
 ### Public and internal
 

data/lib/cronwatch/alerts/discord.rb

@@ -40,7 +40,7 @@ module Cronwatch
           "embeds" => [embed],
         }
         response = @http.post(@webhook_url, JS.json(payload), { "content-type" => "application/json" })
-        raise "Discord webhook answered #{response.status}: #{JS.head16(response.body.to_s, 200)}" unless response.ok?
+        raise "Discord webhook answered #{response.status}: #{Provider.webhook_error_body(response, @webhook_url)}" unless response.ok?
       end
 
       # The message in a code block, then the triage. Each part has its own

data/lib/cronwatch/alerts/provider.rb

@@ -59,6 +59,24 @@ module Cronwatch
         JS.head16(head, ERROR_BODY_MAX)
       end
 
+      # The parts of a webhook URL that are its credential, to cut from an
+      # error body that echoes the request: its path and query together, then
+      # the last segment of its path (Slack's and Discord's token) on its own.
+      def url_secrets(url)
+        uri = URI.parse(url.to_s)
+        path = uri.path.nil? || uri.path.empty? ? "/" : uri.path
+        last = path.split("/", -1).last.to_s
+        ["#{path}#{"?#{uri.query}" if uri.query && !uri.query.empty?}", last].reject { |part| part.empty? || part == "/" }
+      rescue URI::Error
+        []
+      end
+
+      # The start of a webhook's error answer, with the URL's credential cut
+      # out in case a proxy echoes the request: Slack's and Discord's errors.
+      def webhook_error_body(response, url)
+        error_body(Output.utf8(response.body.to_s).delete_prefix("\u{FEFF}"), url_secrets(url))
+      end
+
       # A credential with the spaces and newlines a paste leaves around it
       # taken off, as it goes in a header. Anything not a String is "".
       def trimmed(value)

data/lib/cronwatch/alerts/slack.rb

@@ -42,7 +42,7 @@ module Cronwatch
           "blocks" => blocks,
         }
         response = @http.post(@webhook_url, JS.json(payload), { "content-type" => "application/json" })
-        raise "Slack webhook answered #{response.status}: #{JS.head16(response.body.to_s, 200)}" unless response.ok?
+        raise "Slack webhook answered #{response.status}: #{Provider.webhook_error_body(response, @webhook_url)}" unless response.ok?
       end
 
       # Slack's three control characters. Escaping < and > also stops <!channel> and <url|links>.

data/lib/cronwatch/alerts/webhook.rb

@@ -37,8 +37,15 @@ module Cronwatch
         OpenSSL::HMAC.hexdigest("SHA256", secret.to_s, body.to_s)
       end
 
-      def call(alert)
+      # `context` is the client's channel context: an empty-string secret (an
+      # unset variable read as "", say) signs nothing, and each send reports
+      # that through its on_error.
+      def call(alert, context = nil)
         body = JS.json({ "schema" => SCHEMA }.merge(alert.to_h))
+        if @secret == "" && context.respond_to?(:on_error)
+          context.on_error(RuntimeError.new("webhook: secret is an empty string, so this request is sent unsigned; " \
+                                            "set the secret, or leave it out to send unsigned on purpose"))
+        end
         headers = { "content-type" => "application/json", "user-agent" => "cronwatch" }
         # A pasted Authorization value often carries a stray space or newline, which a header would refuse.
         @headers.each { |name, value| headers[name.to_s] = value.is_a?(String) ? JS.trim(value) : value }

data/lib/cronwatch/client.rb

@@ -81,8 +81,8 @@ module Cronwatch
     # alerts:      where alerts go: objects with #call(alert) and #name. Defaults to the console.
     # triage:      a callable taking a TriageContext and returning a short diagnosis, added to every alert but recoveries.
     # cron_secret: a second bearer Cronwatch::Web accepts for /api/check, for an outside cron. Defaults to
-    #              ENV["CRON_SECRET"]; an empty string or one of only whitespace, given here or in the
-    #              variable, counts as unset. Pass nil for none. Anything but a String or nil (false, a
+    #              ENV["CRON_SECRET"], trimmed of the whitespace around it; an empty string or one of
+    #              only whitespace, given here or in the variable, counts as unset. Pass nil for none. Anything but a String or nil (false, a
     #              number, a Symbol) raises TypeError.
     # retention:   how long finished runs are kept. Default "30d".
     # defaults:    grace, timeout, timezone, and failures_before_alert applied to every job unless it sets its own.
@@ -137,6 +137,9 @@ module Cronwatch
       @clock = now || -> { Process.clock_gettime(Process::CLOCK_REALTIME, :millisecond) }
       @on_error = on_error || method(:default_on_error)
       @definitions = {}
+      # Declarations forgotten here (forget), kept so that run and resume_run
+      # by name declare the job again as it was written, not bare.
+      @forgotten = {}
       @synced = Set.new
       @registry = Mutex.new
       @ready = false
@@ -166,11 +169,29 @@ module Cronwatch
       validate_definition(definition)
       @registry.synchronize do
         @definitions[name] = definition
+        @forgotten.delete(name)
         @synced.delete(name)
       end
       JobHandle.new(self, definition)
     end
 
+    # The declaration of `name`: the standing one, else one forgotten here,
+    # declared again as it was.
+    def declared(name)
+      @registry.synchronize do
+        standing = @definitions[name]
+        next standing if standing
+
+        forgotten = @forgotten.delete(name)
+        if forgotten
+          @definitions[name] = forgotten
+          @synced.delete(name)
+        end
+        forgotten
+      end
+    end
+    private :declared
+
     # Run a job by name without keeping a handle, declaring it on first use
     # (or again, when options are given).
     #
@@ -184,8 +205,8 @@ module Cronwatch
         return get_run(name_or_id)
       end
 
-      declared = @registry.synchronize { @definitions[name_or_id.to_s] }
-      handle = options.any? || declared.nil? ? job(name_or_id, **options) : JobHandle.new(self, declared)
+      definition = options.any? ? nil : declared(name_or_id.to_s)
+      handle = definition.nil? ? job(name_or_id, **options) : JobHandle.new(self, definition)
       handle.run(&block)
     end
 
@@ -205,8 +226,14 @@ module Cronwatch
     # error message, or nil when the result is fine: an HTTP handler uses it to
     # count a 500 response as a failed run.
     #
+    # `requeued_on_shutdown` is for a queue that puts a job back when its
+    # worker shuts down (Sidekiq raises Sidekiq::Shutdown into the job and
+    # runs it again after the restart): a run stopped that way is recorded as
+    # failed but not judged, so a deploy opens no failed alert and does not
+    # count toward failures_before_alert.
+    #
     # @api private For integrations (JobHandle#run, ActiveJob, Sidekiq), not apps.
-    def execute(definition, trigger, failure: nil)
+    def execute(definition, trigger, failure: nil, requeued_on_shutdown: false)
       after_fork_check
       name = definition.name
       started_at = now
@@ -224,7 +251,7 @@ module Cronwatch
       # just before the block runs. The result is the same.
       if recorded
         begin
-          update_state(name) { |before| [Evaluate.on_run_start(before), nil] }
+          update_state(name) { |before| [Evaluate.hold_run(Evaluate.on_run_start(before), run.id), nil] }
         rescue StandardError => e
           report(e, "starting #{name}")
         end
@@ -252,7 +279,8 @@ module Cronwatch
 
       conclude(definition, run, result, error, threw, recorder.expect_text || returned, failure: failure)
       begin
-        ignored = record_finish(definition, run, recorded, finished_at)
+        judge = !(requeued_on_shutdown && threw && Client.shutdown?(error))
+        ignored = record_finish(definition, run, recorded, finished_at, judge: judge)
         report(RuntimeError.new("run #{run.id} of #{name} #{ignored}; ignored"), "finishing #{name}") if ignored
       rescue StandardError => e
         report(e, "recording #{name}")
@@ -267,7 +295,7 @@ module Cronwatch
     # job must be declared in this process, or it raises ArgumentError.
     def resume_run(name, run_id)
       name = name.to_s if name.is_a?(Symbol)
-      definition = @registry.synchronize { @definitions[name] }
+      definition = declared(name)
       raise ArgumentError, "resume_run: job \"#{name}\" is not declared; call job first" unless definition
 
       resume_handle(definition, run_id)
@@ -397,7 +425,7 @@ module Cronwatch
       end
       return [] unless evaluate
 
-      update_state(run.job) { |before| [Evaluate.on_run_start(before), nil] }
+      update_state(run.job) { |before| [Evaluate.hold_run(Evaluate.on_run_start(before), run.id), nil] }
       return [] if run.status == :running
 
       finish_run(definition, run, now)
@@ -496,13 +524,15 @@ module Cronwatch
     end
 
     # Remove a job and its runs from the store. A job still declared in code
-    # comes back: here on its next run, and in any other process that
-    # declares it on its next run there, or at that process's next check or
-    # dashboard read.
+    # comes back, with no history: here on its next run, declared as it was
+    # written (run and resume_run by name included), and in any other process
+    # that declares it on its next run there, or at that process's next check
+    # or dashboard read.
     def forget(name)
       ensure_ready
       @registry.synchronize do
-        @definitions.delete(name)
+        definition = @definitions.delete(name)
+        @forgotten[name] = definition if definition
         @synced.delete(name)
       end
       @store.delete_job(name)
@@ -886,11 +916,24 @@ module Cronwatch
         report(RuntimeError.new("run #{run.id} of #{run.job} #{ignored}; ignored"), "recording #{run.job}")
         return []
       end
-      return [] if !evaluate || (late && run.status != :ok)
+      unless evaluate
+        # Stored unjudged on purpose: a check must not evaluate it as one whose evaluation was lost.
+        release_unjudged(run)
+        return []
+      end
+      return [] if late && run.status != :ok
 
       finish_run(definition, run, now)
     end
 
+    # Takes a run stored without judging out of the job's pending runs, so a
+    # check does not evaluate it as one whose evaluation was lost.
+    def release_unjudged(run)
+      update_state(run.job) { |current| [Evaluate.release_run(current, run.id), nil] }
+    rescue StandardError => e
+      report(e, "recording #{run.job}")
+    end
+
     # A conditional write (the store's update_run_if), or for a store
     # without one, a read then a plain write, which is safe only while one
     # process at a time finishes a given run.
@@ -926,7 +969,7 @@ module Cronwatch
       if threw
         run.status = :failed
         run.error = Output.describe_error(error)
-      elsif (problem = failure&.call(result))
+      elsif (problem = failure_of(failure, result))
         run.status = :failed
         run.error = Output.utf8(problem.to_s)
       else
@@ -945,18 +988,26 @@ module Cronwatch
       run.error = Output.redact_and_cap(run.error, @redact) unless run.error.nil?
     end
 
+    # What the failure: callable makes of the result, or nil. One that raises
+    # fails the run with its error described, so the run is still recorded.
+    def failure_of(failure, result)
+      failure&.call(result)
+    rescue StandardError => e
+      Output.describe_error(e)
+    end
+
     # Writes a finished run and evaluates it. `recorded` says whether its
     # start was written; if not, it is inserted now. Returns why nothing was
     # recorded (another process finished the run first, say), or nil. Raises
     # when the store does, so a handle can be finished again. Shared by
     # execute and RunHandle#finish.
-    def record_finish(definition, run, recorded, finished_at)
+    def record_finish(definition, run, recorded, finished_at, judge: true)
       unless recorded
         # The start was never written; the store may be back by now.
         sync(definition)
         begin
           @store.insert_run(run)
-          finish_run(Serialize.to_stored(definition), run, finished_at)
+          finish_run(Serialize.to_stored(definition), run, finished_at) if judge
           return nil
         rescue StandardError => e
           # Another process may have recorded a run with this id meanwhile.
@@ -972,10 +1023,22 @@ module Cronwatch
       late, ignored = claim_finish(run)
       return ignored if ignored
 
-      finish_run(Serialize.to_stored(definition), run, finished_at) if !late || run.status == :ok
+      if !judge
+        release_unjudged(run)
+      elsif !late || run.status == :ok
+        finish_run(Serialize.to_stored(definition), run, finished_at)
+      end
       nil
     end
 
+    # A job's worker shutting down, which its queue answers by running the job
+    # again: Sidekiq::Shutdown, when Sidekiq is loaded.
+    #
+    # @api private
+    def self.shutdown?(error)
+      defined?(::Sidekiq::Shutdown) && error.is_a?(::Sidekiq::Shutdown) ? true : false
+    end
+
     # The start of execute without the block: the run is inserted and missed
     # and stuck close (on_run_start). A store that fails is reported and the
     # handle inserts the finished run instead, as execute does.
@@ -1011,7 +1074,7 @@ module Cronwatch
       end
       if recorded
         begin
-          update_state(name) { |before| [Evaluate.on_run_start(before), nil] }
+          update_state(name) { |before| [Evaluate.hold_run(Evaluate.on_run_start(before), run.id), nil] }
         rescue StandardError => e
           report(e, "starting #{name}")
         end
@@ -1162,6 +1225,30 @@ module Cronwatch
       joined && Output.cap(joined)
     end
 
+    # Runs the job's state still holds as pending (Evaluate.hold_run) whose
+    # finish was written but never evaluated: the process that finished them
+    # stopped, or its state write failed, between the two writes. Each one
+    # finished more than SEND_LEASE_MS ago is evaluated now, as its finish
+    # would have been; one still running is left to the stuck check, and one
+    # the store no longer has (given back, or pruned) is let go. The wait
+    # leaves a finish under way in another process to write its own evaluation.
+    def evaluate_pending(stored, recent, state, at)
+      alerts = []
+      (state.pending_runs || []).each do |id|
+        run = recent.find { |r| r.id == id } || @store.get_run(id)
+        if run.nil? || run.job != stored.name
+          update_state(stored.name) { |current| [Evaluate.release_run(current, id), nil] }
+          next
+        end
+        next if run.status == :running || !run.finished_at.is_a?(Numeric) || run.finished_at > at - Evaluate::SEND_LEASE_MS
+
+        alerts.concat(finish_run(stored.definition, run, at))
+      rescue StandardError => e
+        report(e, "evaluating run #{id} of #{stored.name}")
+      end
+      alerts
+    end
+
     # Evaluate a finished run (ok, failed, or timed out by a check), already
     # written, against the job's state and send what that produces. The
     # alerts are written with that state (see outbox). Never raises.
@@ -1171,7 +1258,8 @@ module Cronwatch
         past = nil
         _, held = update_state(run.job) do |previous|
           past ||= history(run)
-          outbox(Evaluate.apply_silence(previous, Evaluate.on_run_finish(definition, run, previous, past, at), at), definition, at)
+          before = Evaluate.release_run(previous, run.id)
+          outbox(Evaluate.apply_silence(before, Evaluate.on_run_finish(definition, run, before, past, at), at), definition, at)
         end
       rescue StandardError => e
         report(e, "evaluating #{run.job}")
@@ -1280,6 +1368,7 @@ module Cronwatch
         report_dropped(stored.name, held.dropped)
         alerts.concat(retry_undelivered(stored.name, state, at, retries))
         alerts.concat(dispatch(stored.name, held.alerts, at))
+        alerts.concat(evaluate_pending(stored, recent, state, at))
         jobs << Evaluate.summarize(stored, recent, state, next_expected_at, at)
       rescue StandardError => e
         report(e, "checking #{stored.name}")
@@ -1376,22 +1465,36 @@ module Cronwatch
     RetryBudget = Struct.new(:spent_ms)
 
     # Send the alerts that no channel accepted last time, once each, oldest
-    # first. `state` is the job's state as this check left it: an alert that
-    # no longer describes it (Evaluate.stale_alert?) is dropped instead.
-    # Retries across a check share RETRY_BUDGET_MS of wall-clock time; once it
-    # is spent the rest stay queued for the next check.
+    # first. They are claimed first (Evaluate.claim_undelivered): moved into
+    # `sending` with a lease, in one conditional write, so when several
+    # processes check at once only the one whose write lands sends them. An
+    # alert that no longer describes the job (Evaluate.stale_alert?) is
+    # dropped instead. Retries across a check share RETRY_BUDGET_MS of
+    # wall-clock time; claimed alerts it has no time left for go back to the
+    # queue for the next check.
     def retry_undelivered(name, state, at, budget)
-      pending = state.undelivered || []
-      return [] if pending.empty? || Evaluate.silenced?(state, at) || @defer_delivery
+      return [] if (state.undelivered || []).empty? || @defer_delivery
+
+      begin
+        _, claimed = update_state(name) do |previous|
+          claim = Evaluate.claim_undelivered(previous, at, now + Evaluate::SEND_LEASE_MS)
+          [claim.state, claim.claimed]
+        end
+      rescue StandardError => e
+        report(e, "claiming undelivered alerts for #{name}")
+        return []
+      end
+      return [] if claimed.empty?
 
       delivered = []
       failed = []
-      dropped = pending.select { |alert| Evaluate.stale_alert?(alert, state) }
-      pending.each do |alert|
-        next if dropped.any? { |d| d.equal?(alert) }
-
+      claimed.each do |alert|
         left = @retry_budget_ms - budget.spent_ms
-        break if left <= 0
+        if left <= 0
+          # Not tried: back to the queue as it was.
+          failed << alert
+          next
+        end
 
         started = AbortSignal.monotonic
         # An alert queued by a process that delivers at check time was never
@@ -1400,7 +1503,7 @@ module Cronwatch
         (deliver(alert) ? delivered : failed) << alert
         budget.spent_ms += [0, ((AbortSignal.monotonic - started) * 1000).round].max
       end
-      record_delivery(name, delivered, failed, dropped, at)
+      record_delivery(name, delivered, failed, [], at)
       delivered
     end
 
@@ -1493,12 +1596,16 @@ module Cronwatch
 
     # Sets the alert's triage to the diagnosis, or to nil (JSON null) when
     # there is none (it raised, timed out, or answered nil or ""), so it is
-    # tried once per alert. While a triage that timed out is still going,
-    # alerts go out without one rather than start another beside it.
+    # tried once per alert. While a triage that timed out on its full timeout
+    # is still going, alerts go out without one rather than start another
+    # beside it, and stay untried so a later retry can triage them. A timeout
+    # cut short by the retry budget says nothing about Claude, so it does not
+    # hold back the triages after it.
     def add_triage(alert, timeout_ms)
       signal = AbortSignal.new
       if @sending_lock.synchronize { @abandoned[:triage]&.alive? }
-        raise TimeoutError, "skipped: an earlier triage timed out and is still running"
+        report(TimeoutError.new("skipped: an earlier triage timed out and is still running"), "triage for #{alert.job}")
+        return
       end
 
       recent = @store.list_runs(alert.job, 5)
@@ -1512,7 +1619,7 @@ module Cronwatch
         outcome = [:error, e]
       end
       unless thread.join(timeout_ms / 1000.0)
-        @sending_lock.synchronize { @abandoned[:triage] = thread }
+        @sending_lock.synchronize { @abandoned[:triage] = thread } if timeout_ms >= @triage_timeout_ms
         raise TimeoutError, "timed out after #{timeout_ms}ms"
       end
       outcome ||= [:error, RuntimeError.new("the triage thread ended without an answer")]

data/lib/cronwatch/environment.rb

@@ -4,19 +4,17 @@ module Cronwatch
   # The app's environment, read as every CronWatch library reads it: the
   # first of CRONWATCH_ENV, APP_ENV, then the app's own (Rails.env when Rails
   # is loaded, else RAILS_ENV, then RACK_ENV) that holds more than spaces,
-  # trimmed and lowercased, with "prod" read as "production" and "dev",
-  # "local", "test", and "testing" as "development". None set is neither,
-  # which is the safe reading. The SDK reads NODE_ENV where this reads the
-  # app's own.
+  # trimmed and lowercased, with "prod" read as "production" and "dev" and
+  # "local" as "development". "test" and "testing" are not development: a
+  # test run or CI needs CRON_SECRET or a token, as production does. None set
+  # is neither, which is the safe reading. The SDK reads NODE_ENV where this
+  # reads the app's own.
   #
   # @api private
   module Environment
     # The variables read before the app's own, in order.
     SHARED = %w[CRONWATCH_ENV APP_ENV].freeze
-    ALIASES = {
-      "prod" => "production", "dev" => "development", "local" => "development", "test" => "development",
-      "testing" => "development",
-    }.freeze
+    ALIASES = { "prod" => "production", "dev" => "development", "local" => "development" }.freeze
     private_constant :SHARED, :ALIASES
 
     module_function
@@ -59,20 +57,21 @@ module Cronwatch
       JS.trim(value.dup.force_encoding(Encoding::UTF_8).scrub).empty?
     end
 
-    # A secret from the environment (CRONWATCH_TOKEN, CRON_SECRET): nil when
-    # the variable is unset, empty, or only whitespace, so a blank value counts
-    # as not set and the routes fail closed. Any other value is used as it
-    # is, untrimmed.
+    # A secret from the environment (CRONWATCH_TOKEN, CRON_SECRET), trimmed as
+    # JavaScript's String.prototype.trim trims (JS::WHITESPACE): a value read
+    # from a file often ends in a newline that no header can carry, since
+    # HTTP strips the whitespace around a header's value. nil when the
+    # variable is unset, empty, or only whitespace, so a blank value counts as
+    # not set and the routes fail closed.
     def secret(variable)
-      value = ENV.fetch(variable, nil)
-      present?(value) ? value : nil
+      trimmed(ENV.fetch(variable, nil))
     end
 
-    # A token or secret passed in code: a String, or nil. A String that is
-    # empty or only whitespace counts as not given (nil here; the caller
-    # tells that from nil the opt-out). Anything else (false, a number, a
-    # Symbol) raises TypeError naming the option, so it never becomes a
-    # password.
+    # A token or secret passed in code: a String, or nil. A String is
+    # trimmed, as secret trims, and one that is empty or only whitespace
+    # counts as not given (nil here; the caller tells that from nil the
+    # opt-out). Anything else (false, a number, a Symbol) raises TypeError
+    # naming the option, so it never becomes a password.
     def secret_option(value, what)
       return nil if value.nil?
       unless value.is_a?(String)
@@ -80,7 +79,19 @@ module Cronwatch
         raise TypeError, "#{what} must be a String, or nil to opt out, not #{kind}"
       end
 
-      blank?(value) ? nil : value
+      trimmed(value)
+    end
+
+    # A secret trimmed as JavaScript trims, or nil when nothing is left. Bytes
+    # that are not UTF-8 are kept as they are.
+    def trimmed(value)
+      return nil unless present?(value)
+
+      begin
+        JS.trim(value)
+      rescue ArgumentError
+        value
+      end
     end
 
     # The framework's own name for the environment, as it sets it: Rails.env
@@ -94,7 +105,7 @@ module Cronwatch
       [ENV.fetch("RAILS_ENV", nil), ENV.fetch("RACK_ENV", nil)].find { |value| value && !value.empty? }
     end
 
-    # Development (including test): Cronwatch::Web without a token serves only then.
+    # Development (not test): Cronwatch::Web without a token serves only then.
     def development?
       environment == "development"
     end

data/lib/cronwatch/evaluate.rb

@@ -83,8 +83,9 @@ module Cronwatch
     end
 
     # A stored state with every field present, or a fresh one. State written
-    # by an older version lacks the newer fields. `sending` and `under_floor`
-    # are the exceptions: each is there only while it holds something.
+    # by an older version lacks the newer fields. `sending`, `under_floor`, and
+    # `pending_runs` are the exceptions: each is there only while it holds
+    # something.
     #
     # Read leniently, since a foreign, hand-edited, or damaged row must affect
     # only its own job, and the next write puts it right: a state that is not
@@ -93,14 +94,15 @@ module Cronwatch
     # `last_alert_at` that are not numbers read as nil; `pending_recovery`
     # keeps only its conditions (strings as stored), and `undelivered` only
     # its entries that are objects (a list of neither shape reads as []),
-    # and `under_floor` only its strings (absent when none). Unknown fields
-    # are kept as written.
+    # and `under_floor` and `pending_runs` only their strings (each absent
+    # when none). Unknown fields are kept as written.
     def normalize_state(state, job)
       state = JobState.from_h(state)
       return empty_state(job) if state.nil?
 
       sending = state.sending
       under_floor = state.under_floor.is_a?(Array) ? state.under_floor.grep(String) : []
+      pending_runs = state.pending_runs.is_a?(Array) ? state.pending_runs.grep(String) : []
       open = state.open.is_a?(Hash) ? state.open.select { |_, at| json_number?(at) } : {}
       pending = state.pending_recovery.is_a?(Array) ? state.pending_recovery.grep(Symbol) : []
       undelivered = state.undelivered.is_a?(Array) ? state.undelivered.grep(Alert) : []
@@ -115,6 +117,7 @@ module Cronwatch
         version: state.version,
         sending: sending.is_a?(Array) && !sending.empty? ? sending.dup : nil,
         under_floor: under_floor.empty? ? nil : under_floor,
+        pending_runs: pending_runs.empty? ? nil : pending_runs,
         # Fields a newer release wrote, carried through every write.
         extra: state.extra,
       )
@@ -207,6 +210,61 @@ module Cronwatch
       queue_undelivered(following, failed)
     end
 
+    # A check's claim on the undelivered queue, written before it sends, so
+    # two processes checking at once never retry the same alert: the queued
+    # alerts that still describe the job move into `sending`, oldest first,
+    # each with its lease ending at `until_at`, as many as `sending` has room
+    # for under MAX_UNDELIVERED (the rest wait for the next check); stale
+    # ones (stale_alert?) leave the queue. While the job is silenced only
+    # recoveries are claimed. The process whose write lands sends `claimed`.
+    Claim = Struct.new(:state, :claimed, :stale, keyword_init: true) do
+      def to_h = { "state" => state.to_h, "claimed" => claimed.map(&:to_h), "stale" => stale.map(&:to_h) }
+    end
+
+    def claim_undelivered(state, now, until_at)
+      following = clone_state(state)
+      queue = following.undelivered
+      stale = queue.select { |alert| stale_alert?(alert, following) }
+      silenced = silenced?(following, now)
+      room = [0, MAX_UNDELIVERED - (following.sending || []).length].max
+      fresh = queue.reject { |alert| stale.any? { |old| old.equal?(alert) } }
+      claimed = fresh.select { |alert| !silenced || alert.type == :recovered }.first(room)
+      return Claim.new(state: state, claimed: claimed, stale: stale) if stale.empty? && claimed.empty?
+
+      following.undelivered = fresh.reject { |alert| claimed.any? { |sent| sent.equal?(alert) } }
+      unless claimed.empty?
+        following.sending = (following.sending || []) + claimed.map { |alert| { "until" => until_at, "alert" => alert } }
+      end
+      Claim.new(state: following, claimed: claimed, stale: stale)
+    end
+
+    # Runs held in `pending_runs` per job; past it the oldest go.
+    MAX_PENDING_RUNS = 100
+
+    # A run just started, held in `pending_runs` until its finish is
+    # evaluated (release_run), so a check can evaluate it if the process that
+    # finished it stops before writing the evaluation. Written with
+    # on_run_start's state. Only the newest MAX_PENDING_RUNS are kept.
+    def hold_run(state, run_id)
+      following = clone_state(state)
+      held = (following.pending_runs || []).reject { |id| id == run_id }
+      held << run_id
+      following.pending_runs = held.last(MAX_PENDING_RUNS)
+      following
+    end
+
+    # A run taken out of `pending_runs`, as its evaluation is written or once
+    # it is gone from the store.
+    def release_run(state, run_id)
+      held = state.pending_runs.is_a?(Array) ? state.pending_runs : []
+      return state unless held.include?(run_id)
+
+      following = clone_state(state)
+      left = held.reject { |id| id == run_id }
+      following.pending_runs = left.empty? ? nil : left
+      following
+    end
+
     # The alert of an entry of `sending`, or nil when it has none, or one
     # that did not parse as an alert (which could not be sent either).
     def sending_alert(entry)
@@ -504,11 +562,14 @@ module Cronwatch
     end
 
     # An evaluation as it is saved and sent: while the job was silenced when it
-    # began, nothing opens and nothing is sent.
+    # began, nothing opens and no new alert is sent, but a recovery is. A
+    # condition can only close during a silence if it opened (and alerted)
+    # before it, so its recovery closes an incident someone was told about.
     def apply_silence(previous, evaluation, now)
       return evaluation unless silenced?(previous, now)
 
-      Evaluation.new(state: mute_opens(previous, evaluation.state), alerts: [])
+      Evaluation.new(state: mute_opens(previous, evaluation.state),
+                     alerts: evaluation.alerts.select { |alert| alert.type == :recovered })
     end
 
     # Whether an alert waiting to be retried no longer describes the job, so it

data/lib/cronwatch/http.rb

@@ -49,6 +49,10 @@ module Cronwatch
         headers.each { |k, v| request[k] = HTTP.trim_header(v) }
         request.body = body
         http = connection(uri)
+        # Net::HTTP opens the socket under Timeout.timeout(open_timeout), or
+        # TCPSocket's own open_timeout where Ruby has one, and both cover the
+        # name lookup: it can be interrupted from Ruby 3.3. On Ruby 3.2 a lookup
+        # that hangs is cut off only by the client's channel deadline.
         http.open_timeout = HTTP.remaining(deadline)
         http.start do |conn|
           remaining = HTTP.remaining(deadline)
@@ -141,12 +145,19 @@ module Cronwatch
       value.to_s.gsub(/\A[ \t\r\n]+|[ \t\r\n]+\z/, "")
     end
 
-    # Compares two secrets without stopping at the first differing character.
-    def constant_time_equal?(a, b)
-      return false unless a.bytesize == b.bytesize
-
-      diff = 0
-      a.bytes.zip(b.bytes) { |x, y| diff |= x ^ y }
+    # Compares what a request `sent` with the `secret` without stopping at the
+    # first differing byte, and in time that depends only on the length of
+    # what was sent, so neither the secret's bytes nor its length can be
+    # timed. Every byte of `sent` is compared, against the secret's byte at
+    # the same place (repeating it past its end), and a length that differs
+    # counts as a difference. The SDK's constantTimeEqual, over bytes.
+    def constant_time_equal?(sent, secret)
+      expected = secret.b
+      return sent.bytesize.zero? if expected.empty?
+
+      diff = sent.bytesize ^ expected.bytesize
+      size = expected.bytesize
+      sent.b.each_byte.with_index { |byte, i| diff |= byte ^ expected.getbyte(i % size) }
       diff.zero?
     end
   end

data/lib/cronwatch/monitored.rb

@@ -166,7 +166,9 @@ module Cronwatch
         return yield unless handle
 
         holder = job.respond_to?(:cronwatch_with, true) ? job : Discard
-        outcome = client.execute(handle.definition, trigger) do |context|
+        # Sidekiq puts a job stopped by its shutdown back on the queue, so that
+        # run is recorded but not judged (see Client#execute).
+        outcome = client.execute(handle.definition, trigger, requeued_on_shutdown: true) do |context|
           holder.__send__(:cronwatch_with, context) { yield }
         end
         raise outcome.error if outcome.threw

data/lib/cronwatch/sidekiq.rb

@@ -30,7 +30,9 @@ module Cronwatch
   # "sidekiq"; `cronwatch` in the job is the run's context, for log and
   # metric. A perform that raises is recorded as failed and then raises as
   # before, so Sidekiq's retries, death handlers, and error handlers see it
-  # unchanged.
+  # unchanged. A perform stopped by Sidekiq::Shutdown is recorded but not
+  # judged: Sidekiq runs the job again after the restart, so a deploy opens
+  # no failed alert.
   #
   #   class NightlyReportJob
   #     include Sidekiq::Job

data/lib/cronwatch/types.rb

@@ -386,7 +386,7 @@ module Cronwatch
   # when there are none. An open condition this version does not know stays
   # in `open` the same way, under its own name.
   JobState = Struct.new(:job, :open, :consecutive_failures, :silenced_until, :last_alert_at, :pending_recovery,
-                        :undelivered, :version, :sending, :under_floor, :extra, keyword_init: true) do
+                        :undelivered, :version, :sending, :under_floor, :pending_runs, :extra, keyword_init: true) do
     include Serializable
 
     def self.from_h(hash)
@@ -398,6 +398,7 @@ module Cronwatch
       undelivered = Naming.fetch(hash, "undelivered")
       sending = Naming.fetch(hash, "sending")
       under_floor = Naming.fetch(hash, "underFloor")
+      pending_runs = Naming.fetch(hash, "pendingRuns")
       extra = hash.each_with_object({}) { |(k, v), out| out[k.to_s] = v unless JobState::KNOWN.include?(k.to_s) }
       new(
         extra: extra.empty? ? nil : extra,
@@ -411,6 +412,7 @@ module Cronwatch
         version: Naming.fetch(hash, "version"),
         sending: sending.is_a?(Array) ? sending.map { |entry| JobState.sending_entry(entry) } : sending,
         under_floor: under_floor,
+        pending_runs: pending_runs,
       )
     end
 
@@ -436,8 +438,8 @@ module Cronwatch
 
     # pendingRecovery, undelivered, and version are left out when unset, as in
     # state written before they existed. The version comes after them, where
-    # the SDK's spread of a normalized state puts it, then `sending` and
-    # `underFloor`, each only while it holds an entry.
+    # the SDK's spread of a normalized state puts it, then `sending`,
+    # `underFloor`, and `pendingRuns`, each only while it holds an entry.
     def to_h
       out = {
         "job" => job, "open" => open.is_a?(Hash) ? open.transform_keys(&:to_s) : (open || {}),
@@ -457,6 +459,7 @@ module Cronwatch
         end
       end
       out["underFloor"] = under_floor.dup if under_floor.is_a?(Array) && !under_floor.empty?
+      out["pendingRuns"] = pending_runs.dup if pending_runs.is_a?(Array) && !pending_runs.empty?
       extra&.each { |key, value| out[key] = value unless out.key?(key) || JobState::KNOWN.include?(key) }
       out
     end
@@ -464,7 +467,8 @@ module Cronwatch
 
   # The fields this version reads from a stored state; the rest are kept in `extra`.
   # @api private
-  JobState::KNOWN = %w[job open consecutiveFailures silencedUntil lastAlertAt pendingRecovery undelivered version sending underFloor].freeze
+  JobState::KNOWN = %w[job open consecutiveFailures silencedUntil lastAlertAt pendingRecovery undelivered version sending underFloor
+                         pendingRuns].freeze
 
   JobStats = Struct.new(:runs, :ok_rate, :p50_ms, :p95_ms, keyword_init: true) do
     include Serializable

data/lib/cronwatch/version.rb

@@ -2,5 +2,5 @@
 
 module Cronwatch
   # Kept in step with @cronwatch/sdk.
-  VERSION = "0.12.5"
+  VERSION = "0.13.1"
 end

data/lib/cronwatch/web/app.rb

@@ -3,6 +3,7 @@
 require "base64"
 require "digest"
 require "securerandom"
+require "stringio"
 require "uri"
 
 module Cronwatch
@@ -24,11 +25,12 @@ module Cronwatch
   # token:     required to reach anything. Send it as `Authorization: Bearer <token>`,
   #            or open the dashboard once with `?token=<token>` (or sign in with the
   #            form, which posts it to <base>/signin) and a cookie is set.
-  #            Defaults to ENV["CRONWATCH_TOKEN"]; an empty string or one of only
+  #            Defaults to ENV["CRONWATCH_TOKEN"]. It is trimmed of the whitespace
+  #            around it; an empty string or one of only
   #            whitespace, given here or in the variable, counts as unset, and
   #            anything but a String or nil (false, a number, a Symbol) raises TypeError.
-  #            With no token while Cronwatch::Environment is development or
-  #            test (Environment.stated_development?: under Puma, Unicorn,
+  #            With no token while Cronwatch::Environment is development
+  #            (Environment.stated_development?: under Puma, Unicorn,
   #            Thin, or rackup, which set RACK_ENV=development by default,
   #            RACK_ENV alone does not count), the app makes a random one and
   #            prints a sign-in link to stdout on its first request; with no
@@ -39,18 +41,24 @@ module Cronwatch
   #            mount point (SCRIPT_NAME), which is right under Rails' `mount`.
   # origin:    the public origin the dashboard is served from, such as
   #            "https://app.example.com". By default the request's own origin
-  #            is used, as Rack reads it (Rack::Request#base_url, which follows
-  #            X-Forwarded-Proto and X-Forwarded-Host the way Rails does). Set
-  #            this to pin it, for an app behind a proxy that does not set
-  #            those headers or one that should not trust them. Used in place
-  #            of the request's origin for the cross-site check on writes, the
-  #            sign-in cookie's Secure flag, the Referer the redirect back after
-  #            a form follows, and the development sign-in line. Read as
+  #            is used: rack.url_scheme and the Host header, or the forwarded
+  #            ones under trust_proxy. Set this to pin it. Used in place of the
+  #            request's origin for the cross-site check on writes, the sign-in
+  #            cookie's Secure flag, the Referer the redirect back after a form
+  #            follows, and the development sign-in line. Read as
   #            `new URL(value).origin` reads it (Web::Origin) and normalised to
   #            scheme://host[:port]; anything that is not an absolute http or
   #            https URL, or has a port outside 1 to 65535, raises
-  #            ArgumentError here. "" counts as unset. Behind more than one
-  #            proxy set it: Rack takes the last of several forwarded values.
+  #            ArgumentError here. "" counts as unset. Takes precedence over
+  #            trust_proxy.
+  # trust_proxy: take the public origin from X-Forwarded-Proto and
+  #            X-Forwarded-Host (the first value of each, falling back to the
+  #            request's own scheme or host for whichever is missing) when a
+  #            request carries either, as the SDK's trustProxy does. Only for
+  #            an app whose proxy sets or overwrites both headers: a client can
+  #            send them too. Defaults to true when Rails is loaded, since Rails
+  #            itself trusts them (ActionDispatch::Request#base_url), and to
+  #            false otherwise, as in the SDK.
   class Web
     # Tells "token not given" (read CRONWATCH_TOKEN) from "token: nil" (open on purpose).
     UNSET = Object.new.freeze
@@ -103,9 +111,10 @@ module Cronwatch
       app
     end
 
-    def initialize(client = nil, token: UNSET, base_path: nil, origin: nil)
+    def initialize(client = nil, token: UNSET, base_path: nil, origin: nil, trust_proxy: UNSET)
       @client = client
       @origin = Web.configured_origin(origin)
+      @trust_proxy = trust_proxy.equal?(UNSET) ? Web.rails? : trust_proxy == true
       @opted_out = token.nil?
       # A blank token, given or read, counts as unset; one given that is not a String raises.
       given = token.equal?(UNSET) ? nil : Environment.secret_option(token, "routes: token")
@@ -117,10 +126,26 @@ module Cronwatch
       # the server log.
       @generated = @token.nil? && !@opted_out && Environment.stated_development?
       @token = Web.development_token if @generated
+      Web.warn_short_token(@token) unless @generated
       @announced = false
       @announce_lock = Mutex.new
     end
 
+    # Under this many characters a token could be guessed one request at a
+    # time, since nothing slows down wrong guesses.
+    SHORT_TOKEN = 32
+    private_constant :SHORT_TOKEN
+
+    # Warns, once per app, when the dashboard's token is short enough to guess.
+    #
+    # @api private
+    def self.warn_short_token(token)
+      return if token.nil? || token.length >= SHORT_TOKEN
+
+      warn "[cronwatch] the dashboard token is #{token.length} characters; use #{SHORT_TOKEN} or more random characters, " \
+           "such as the output of `ruby -rsecurerandom -e 'puts SecureRandom.urlsafe_base64(32)'`"
+    end
+
     # A token for one app in development, when none is configured: 32 random
     # bytes, base64url (43 characters).
     def self.development_token
@@ -168,6 +193,11 @@ module Cronwatch
       Origin.parse(value)
     end
 
+    # Whether the app runs under Rails, which trusts forwarded headers by default.
+    def self.rails?
+      defined?(::Rails::Railtie) ? true : false
+    end
+
     # The client given, or Cronwatch.client when none was.
     def client
       @client || Cronwatch.client
@@ -230,7 +260,9 @@ module Cronwatch
       # The sign-in form posts the token here, in the body, so it stays out of
       # the URL and every access log. Cross-site posts were refused above.
       if @token && method == "POST" && path == "/signin"
-        sent = read_body(request)["token"]
+        body = read_body(request)
+        # A body too large is no token at all.
+        sent = body.equal?(TOO_LARGE) ? nil : body["token"]
         return sign_in_page(base) if sent.nil? || !HTTP.constant_time_equal?(sent, @token)
 
         return redirect(sign_in_return(request.header("referer"), public_origin(request), base),
@@ -260,7 +292,10 @@ module Cronwatch
           # Move the token from the URL into a cookie, so it is not left in the
           # browser's history. The request line that carried it may still be in
           # an access log, which is why the sign-in form posts instead.
-          return redirect(request.pathname + request.search_without("token"), "set-cookie" => sign_in_cookie(request, base))
+          # One leading slash: "//host/..." would be a protocol-relative Location,
+          # off this site, and a browser reads a backslash there as a slash.
+          return redirect(request.pathname.sub(%r{\A[/\\]+}, "/") + request.search_without("token"),
+                          "set-cookie" => sign_in_cookie(request, base))
         end
       end
 
@@ -301,8 +336,11 @@ module Cronwatch
         return html(HTML.message_page("No such job", "#{name} is not in the store.", base), 404) unless cw.job_summary(name)
 
         if action == "silence"
+          body = read_body(request)
+          return html(HTML.message_page("Not silenced", "The request was too large.", base), 413) if body.equal?(TOO_LARGE)
+
           begin
-            duration = silence_duration(read_body(request)["for"])
+            duration = silence_duration(body["for"])
           rescue ArgumentError => e
             return html(HTML.message_page("Not silenced", e.message, base), 400)
           end
@@ -346,6 +384,8 @@ module Cronwatch
 
         if rest[2] == "silence"
           body = read_body(request)
+          return api({ ok: false, error: "Request body too large" }, 413) if body.equal?(TOO_LARGE)
+
           begin
             duration = silence_duration(body.key?("for") ? body["for"] : request.query("for"))
           rescue ArgumentError => e
@@ -399,7 +439,8 @@ module Cronwatch
       end
       return unless first
 
-      shown = @origin || (Web.loopback_origin?(request.origin) ? request.origin : nil)
+      own = request_origin(request)
+      shown = @origin || (Web.loopback_origin?(own) ? own : nil)
       $stdout.puts(Web.development_sign_in_line(shown, base, @token))
       $stdout.flush
     end
@@ -477,17 +518,26 @@ module Cronwatch
       !site.nil? && site != "same-origin" && site != "none"
     end
 
+    # A request body past Request::MAX_BODY, which read_body answers with.
+    TOO_LARGE = Object.new.freeze
+    private_constant :TOO_LARGE
+
     # The form fields or JSON object of a request, each value as String(value)
-    # gives it in JavaScript. The body is read as the SDK's request.json()
-    # reads it: bytes that are not UTF-8 become U+FFFD, and a leading byte
-    # order mark is dropped.
+    # gives it in JavaScript, or TOO_LARGE for a body past Request::MAX_BODY.
+    # The body is read as the SDK's request.json() reads it: bytes that are
+    # not UTF-8 become U+FFFD, and a leading byte order mark is dropped.
     def read_body(request)
       type = request.header("content-type") || ""
-      if type.include?("application/json")
+      json = type.include?("application/json")
+      form = type.include?("application/x-www-form-urlencoded") || type.include?("multipart/form-data")
+      return {} unless json || form
+      return TOO_LARGE if request.too_large?
+
+      if json
         data = JSON.parse(Output.utf8(request.body).delete_prefix(""))
         return data.to_h { |k, v| [k.to_s, js_string(v)] } if data.is_a?(Hash)
         return data.each_with_index.to_h { |v, i| [i.to_s, js_string(v)] } if data.is_a?(Array)
-      elsif type.include?("application/x-www-form-urlencoded") || type.include?("multipart/form-data")
+      else
         return request.form.transform_values { |v| form_value(v) }
       end
       {}
@@ -547,7 +597,12 @@ module Cronwatch
 
     # The origin a browser sees: the `origin` option when set, otherwise the request's own.
     def public_origin(request)
-      @origin || request.origin
+      @origin || request_origin(request)
+    end
+
+    # The request's origin: the forwarded one under trust_proxy, else its own.
+    def request_origin(request)
+      @trust_proxy ? request.forwarded_origin : request.origin
     end
 
     def api(body, status = 200, headers = {})
@@ -607,11 +662,35 @@ module Cronwatch
         @env[key]
       end
 
-      # scheme://host[:port], the page's origin, lowercased as a browser
-      # writes it in Origin and Referer (Rack's base_url keeps the case of
-      # the Host or X-Forwarded-Host it read).
+      # scheme://host[:port], the page's origin as the request URL gives it
+      # (rack.url_scheme and the Host header, never a forwarded header),
+      # normalised as a browser writes it in Origin and Referer.
       def origin
-        @rack.base_url.downcase
+        scheme = @env["rack.url_scheme"].to_s.downcase
+        scheme = "http" unless %w[http https].include?(scheme)
+        host = @env["HTTP_HOST"].to_s
+        host = [@env["SERVER_NAME"], @env["SERVER_PORT"]].compact.join(":") if host.empty?
+        Origin.bare("#{scheme}://#{host}") || "#{scheme}://#{host.downcase}"
+      end
+
+      # With trust_proxy: the first X-Forwarded-Proto and X-Forwarded-Host,
+      # each falling back to the request's own, when either is present and
+      # well formed; otherwise the request's own origin. The SDK's forwardedOrigin.
+      def forwarded_origin
+        own = origin
+        proto = Request.first_value(header("x-forwarded-proto"))&.downcase
+        host = Request.first_value(header("x-forwarded-host"))
+        return own if proto.nil? && host.nil?
+        return own unless proto.nil? || %w[http https].include?(proto)
+
+        scheme, authority = own.split("://", 2)
+        Origin.bare("#{proto || scheme}://#{host || authority}") || own
+      end
+
+      # The first of a header's comma-separated values, trimmed, or nil.
+      def self.first_value(value)
+        first = value&.split(",", 2)&.first&.strip
+        first.nil? || first.empty? ? nil : first
       end
 
       # URLSearchParams#get: the first value, or nil.
@@ -627,12 +706,24 @@ module Cronwatch
         pairs.empty? ? "" : "?#{URI.encode_www_form(pairs)}"
       end
 
+      # The most of a request body the routes read (1 MiB), as every port's.
+      MAX_BODY = 1 << 20
+
+      # Whether the body is past MAX_BODY: by its Content-Length when that
+      # says more, else by reading one byte past the cap at most, so a client
+      # cannot make the server hold a body of any size. The forms and JSON the
+      # routes take are a few bytes. What was read is left to be read again.
+      def too_large?
+        @too_large = measure_too_large if @too_large.nil?
+        @too_large
+      end
+
       def body
         input = @env["rack.input"]
         return "" unless input
 
         input.rewind if input.respond_to?(:rewind)
-        text = input.read.to_s
+        text = input.read(MAX_BODY + 1).to_s
         input.rewind if input.respond_to?(:rewind)
         text.force_encoding(Encoding::UTF_8)
       end
@@ -645,6 +736,30 @@ module Cronwatch
         fields.each_with_object({}) { |(k, v), out| out[Output.utf8(k)] = v }
       end
 
+      private
+
+      def measure_too_large
+        length = @env["CONTENT_LENGTH"].to_s
+        return true if length.match?(/\A\d+\z/) && length.to_i > MAX_BODY
+
+        input = @env["rack.input"]
+        return false unless input
+
+        input.rewind if input.respond_to?(:rewind)
+        read = input.read(MAX_BODY + 1).to_s
+        return true if read.bytesize > MAX_BODY
+
+        # An input that cannot be rewound is handed on as what was read.
+        if input.respond_to?(:rewind)
+          input.rewind
+        else
+          @env["rack.input"] = StringIO.new(read)
+        end
+        false
+      end
+
+      public
+
       # application/x-www-form-urlencoded parsing as URLSearchParams does it:
       # "+" is a space, and a bad escape is kept as written rather than refused.
       def self.parse_query(text)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cronwatch
 version: !ruby/object:Gem::Version
-  version: 0.12.5
+  version: 0.13.1
 platform: ruby
 authors:
 - Jon C. Phillips