railwatch 0.5.1 → 0.6.1

data/docs/troubleshooting.md

@@ -8,20 +8,48 @@ below are keyed to those lines.
 bin/rails railwatch:doctor
 ```
 
-The task exits non-zero only when **token** or **ingest reachable**
-fails. Everything else is informational. A `✗` there means a feature
+The first lines depend on the transport. An embedded install checks its
+databases, writer and dashboard; a cloud install checks its token and
+ingest host. The task exits non-zero only on the lines marked fatal
+below. Everything else is informational: a `✗` there means a feature
 isn't wired, not that the install is broken.
 
+Embedded (`transport = :local`):
+
+| Doctor line | What a `✗` means |
+|---|---|
+| `export` | Only shown with `export_enabled` on. Export is on and cannot work: no token, no URL, a plain-HTTP URL, or an unsupported policy. Fatal. |
+| `export destination` | The export queue is blocked or deferred; the line gives the reason. `bin/rails railwatch:export:rebind` clears a credential block. |
+| `railwatch database`, `railwatch_telemetry database` | The database is missing from `config/database.yml` for this environment. Fatal. Re-run the install generator. |
+| `railwatch migrations`, `railwatch_telemetry migrations` | Migrations are pending. Fatal. Run `bin/rails db:prepare`. |
+| `telemetry disk` | The telemetry database is not in incremental auto-vacuum, so pruning never shrinks the file. See [Embedded mode](embedded.md#giving-the-disk-back). |
+| `maintenance` | No maintenance tick in the last ten minutes. Expected when the app is stopped: the clock runs in the app's processes, not in rake. |
+| `writer process` | The writer socket is not answering, `plugin :railwatch` is missing from `config/puma.rb`, or the socket path is over Linux's 108-byte limit. Expected when the app is stopped. |
+| `last write` | Shown when the writer answers: nothing written in five minutes. With the app serving traffic, the writer is stuck or workers are not reaching it. |
+| `dashboard access` | HTTP Basic is on with no credentials outside development, so every page is 401; or Basic is off and nothing else is declared. Run `bin/rails railwatch:authentication:configure`, or see [Embedded mode](embedded.md#authentication). |
+| `json compatibility` | The installed `json` gem cannot decode on this Rails; see [below](#binjobs-dies-in-a-loop-with-wrong-number-of-arguments-given-2-expected-1). |
+| `recurring.yml` | `config/recurring.yml` still lists `Railwatch::*` jobs from a pre-release. Remove them. |
+
+Cloud (`transport = :http`):
+
 | Doctor line | What a `✗` means |
 |---|---|
 | `token` | `RAILWATCH_TOKEN` is unset or empty. Fatal: nothing is recorded at all. |
+| `token storage` | A plaintext token is in a file Git tracks. Fatal. Move it to credentials or a secret manager. |
 | `ingest url` | `ingest_url` isn't a parseable HTTP(S) URL. |
+| `ingest transport security` | The ingest URL is plain HTTP on a non-loopback host without `RAILWATCH_ALLOW_HTTP=true`. |
 | `ingest reachable` | `GET {ingest_url}/ingest/ping` didn't return success. Fatal. The ping carries the token, so a missing or wrong token fails this line too; fix `token` first. |
+
+Both:
+
+| Doctor line | What a `✗` means |
+|---|---|
 | `request middleware` | `Railwatch::Middleware::Request` isn't in the stack, so requests aren't executions. |
 | `engine mounted` | `mount Railwatch::Engine, at: "/railwatch"` is missing from `config/routes.rb`; the browser beacon has nowhere to post. |
 | `deploy` | `config.deploy` is unset — records ship, charts get no deploy markers. |
 | `sample rates` | Never fails; it prints the effective rate per execution kind. |
 | `ignored record types` | Never fails; it prints what `c.ignore` is dropping. |
+| `interactive sessions` | Never fails; it prints whether consoles are captured and the runner scratch paths. |
 | `kamal post-deploy hook` | `.kamal/hooks/post-deploy` is missing or doesn't mention Railwatch. Only matters if you deploy with Kamal. |
 | `browser client` | `app/frontend/lib/railwatch.ts` isn't there. Only matters for Inertia visit timing. |
 | `browser client imported` | The client exists but nothing calls `startRailwatch()` — no `startRailwatch` found in `app/frontend/entrypoints`. Visits won't report. |
@@ -33,30 +61,35 @@ isn't wired, not that the install is broken.
 **Symptom.** The environment's pages stay empty however much traffic the
 app takes.
 
-Work down this list. The first five are the same root cause seen from
+Work down this list. Most of it is the same root cause seen from
 different angles: Railwatch decided not to record.
 
-**The token is missing or blank.** `Railwatch.enabled?` is
-`config.enabled && token.present?`. With no token the engine's
-`railwatch.subscribe` initializer returns early, so no subscribers and no
-patches are installed at all. This is by design, so the gem is inert in
-development. Fix: set `RAILWATCH_TOKEN`, restart, and re-run
-`railwatch:doctor`. The `token` line prints the first 6 characters and
-the length, which is enough to spot a truncated or quoted value.
+**Embedded: the writer is not writing.** Run `railwatch:doctor` with the
+app serving traffic. `writer process` and `last write` say whether the
+writer is up and when it last wrote; the migrations lines catch a
+database that was never prepared.
+
+**Cloud: the token is missing or blank.** With `transport = :http`,
+`Railwatch.enabled?` is `config.enabled && token.present?`. With no token
+the engine's `railwatch.subscribe` initializer returns early, so no
+subscribers and no patches are installed at all. Fix: set
+`RAILWATCH_TOKEN`, restart, and re-run `railwatch:doctor`. The `token`
+line prints the first 6 characters and the length, which is enough to
+spot a truncated or quoted value.
 
-**The token is wrong.** A 401 from the ingest marks the transport
+**Cloud: the token is wrong.** A 401 from the ingest marks the transport
 permanently unauthorized: no further flush is attempted for the lifetime
 of that process. Fixing the env var isn't enough. Restart the process.
 `railwatch:doctor`'s `ingest reachable` line catches this before you
 deploy.
 
-**`RAILWATCH_INGEST_URL` points somewhere else.** Records go where you sent
-them. `railwatch:status` prints the URL it is actually using. Compare it
-against the platform you're looking at. Self-hosting: see
+**Cloud: `RAILWATCH_INGEST_URL` points somewhere else.** Records go where
+you sent them. `railwatch:status` prints the URL it is actually using.
+Compare it against the platform you're looking at. Self-hosting: see
 [`self-hosting.md`](self-hosting.md).
 
 **`config.enabled` is false.** `RAILWATCH_ENABLED=0` (or `false`/`no`/`off`)
-turns everything off with a valid token present.
+turns everything off, embedded or not.
 
 **Sample rates are at zero.** `c.sample = { requests: 0.0 }` means no
 request records. So does the per-route `railwatch_never_sample` macro on
@@ -70,11 +103,11 @@ rather than a broken install.
 built. The `ignored record types` doctor line prints the list. Ignoring
 `:queries` also drops `n_plus_one`, since both key off `:queries`.
 
-**You're looking at the test environment.** Requiring `railwatch/rspec`
-(or `railwatch/minitest`) swaps the reporter's transport for an in-memory
-one. A suite records normally but never sends anything over the
-network. Independently: the health sampler, the session flusher, and the
-profiler all refuse to start when `Rails.env.test?`.
+**You're looking at the test environment.** The spec helpers swap the
+reporter's transport for an in-memory one, so a suite records normally
+but never sends or stores anything. Independently: the health sampler,
+the session flusher, and the profiler all refuse to start when
+`Rails.env.test?`.
 
 Still nothing? Set `RAILWATCH_DEBUG=1` and restart. Internal diagnostics go
 to stderr prefixed `[railwatch]`. They never go to `Rails.logger`, so they
@@ -215,14 +248,18 @@ can't be made until it ends.
   (10,000). Past that, records are dropped and counted. The count is
   added to the reporter's drop counter so the loss is visible on the
   platform rather than silent.
-- `c.buffer_size` (default 10,000, the same as `MAX_RECORDS`) caps the
-  process-wide queue between the app and the reporter thread.
-  Oldest-dropped-first, also counted. Do not set it below `MAX_RECORDS`.
-  An execution's tree is written to the queue in one go when it ends, so
-  a tree larger than the queue loses its own first records. Typically
+- The process-wide queue between the app and the reporter thread is
+  bounded by `c.buffer_bytes` (default 16 MiB) and by `c.buffer_size`
+  (default 10,000, the same as `MAX_RECORDS`). Oldest-dropped-first, also
+  counted. The byte ceiling is the one that fills: on a realistic mix of
+  records, 16 MiB holds about 5,000 of them, so raising `buffer_size`
+  changes nothing. Do not set it below `MAX_RECORDS`, though: an
+  execution's tree is written to the queue in one go when it ends, so a
+  tree larger than the queue loses its own first records. Typically
   those are the outgoing requests a long job made before it started
   writing. Keeping far more executions than before means far more
-  records arriving at this queue. Raise it, or lower what you keep.
+  records arriving at this queue. Raise `buffer_bytes`, or lower what
+  you keep.
 - `c.profile_slow_ms` compounds it. It profiles every tail-buffering
   execution from its first line and throws away the fast ones, so the
   profiler's stack table is held alongside the record buffer.
@@ -292,7 +329,13 @@ hook below.
 
 **Cause and fix**, in the order the hook itself checks:
 
-- **`RAILWATCH_TOKEN` isn't exported to the hook.** The first thing
+- **Embedded: `RAILWATCH_TRANSPORT=local` isn't exported to the hook.**
+  The hook reads the deployer's environment, not your initializer. With
+  that variable set it runs `bin/rails railwatch:deploy[$KAMAL_VERSION]`
+  in the primary container, which writes the marker to the embedded
+  database. Without it, the hook treats the install as a cloud one and
+  exits at the next check, since an embedded install has no token.
+- **Cloud: `RAILWATCH_TOKEN` isn't exported to the hook.** The next thing
   `.kamal/hooks/post-deploy` does is `[ -z "$RAILWATCH_TOKEN" ] && exit 0`.
   The hook runs on the deployer machine, in your shell, not in a
   container. So a token that only exists in `.kamal/secrets` for the
@@ -313,8 +356,8 @@ it exits 0 regardless.
 
 ## Log search finds less than it should
 
-**Symptom.** On a Postgres-backed platform install, log search matches
-fewer lines and highlights nothing.
+**Symptom.** On a self-hosted platform backed by Postgres, log search
+matches fewer lines and highlights nothing.
 
 **Cause.** Full-text search uses SQLite's FTS5 (`logs_fts`). The
 platform checks for both a SQLite adapter *and* the `logs_fts` table.

data/lib/generators/railwatch/install/install_generator.rb

@@ -9,10 +9,11 @@ module Railwatch
       source_root File.expand_path("templates", __dir__)
 
       desc "Creates config/initializers/railwatch.rb, a Kamal post-deploy hook, the browser client, and wires the test helpers. " \
-           "With --local, also the two SQLite databases the in-app dashboard needs."
+           "By default telemetry stays in this app, in two SQLite databases, with the dashboard at /railwatch. " \
+           "--cloud (or any token or URL option) sends it to Railwatch Cloud instead."
 
-      class_option :local, type: :boolean, default: false,
-                           desc: "Keep telemetry in this app and serve the dashboard at /railwatch: no token, no cloud."
+      class_option :cloud, type: :boolean, default: false,
+                           desc: "Send telemetry to Railwatch Cloud instead of keeping it in this app. Implied by --prompt-token, --token-stdin, --url and --kamal-secrets."
       class_option :prompt_token, type: :boolean, default: false,
                                   desc: "Prompt for the ingest token without echoing it."
       class_option :token_stdin, type: :boolean, default: false,
@@ -67,9 +68,9 @@ module Railwatch
       # database is, so an app on PostgreSQL or MySQL needs the adapter gem
       # added before those files can be created.
       def ensure_sqlite3_gem
-        return unless options[:local]
+        return unless local?
         return if Gem.loaded_specs.key?("sqlite3")
-        return say("--local needs the sqlite3 gem for its two databases; add `gem \"sqlite3\"` and re-run.", :yellow) unless File.exist?("Gemfile")
+        return say("Embedded mode needs the sqlite3 gem for its two databases; add `gem \"sqlite3\"` and re-run.", :yellow) unless File.exist?("Gemfile")
 
         contents = File.read("Gemfile")
         unless contents.match?(/^\s*gem ["']sqlite3["']/)
@@ -89,9 +90,9 @@ module Railwatch
       # creates the tables now and migrates them after every gem update.
       # Nothing is copied into the app.
       def configure_local_databases
-        return unless options[:local]
+        return unless local?
 
-        return say("--local: no config/database.yml found; add railwatch and railwatch_telemetry databases yourself (docs/embedded.md).", :yellow) unless File.exist?("config/database.yml")
+        return say("No config/database.yml found; add railwatch and railwatch_telemetry databases yourself (docs/embedded.md).", :yellow) unless File.exist?("config/database.yml")
 
         contents = File.read("config/database.yml")
         updated = self.class.database_yml_with_railwatch(contents)
@@ -103,8 +104,8 @@ module Railwatch
       # The writer process: one per Puma master, forked by the gem's Puma
       # plugin, so batches are mapped and written outside the web workers.
       def configure_local_writer
-        return unless options[:local]
-        return say("--local: no config/puma.rb found; add `plugin :railwatch` to your Puma config yourself (docs/embedded.md).", :yellow) unless File.exist?("config/puma.rb")
+        return unless local?
+        return say("No config/puma.rb found; add `plugin :railwatch` to your Puma config yourself (docs/embedded.md).", :yellow) unless File.exist?("config/puma.rb")
 
         contents = File.read("config/puma.rb")
         updated = self.class.puma_rb_with_railwatch(contents)
@@ -166,7 +167,7 @@ module Railwatch
       # A token lands in .env only when Git confirms the file is ignored.
       # URLs are not secret and can still be written to a tracked dotenv file.
       def write_env
-        return if options[:local]
+        return if local?
 
         token = resolved_token
         vars = { TOKEN_VAR => token, URL_VAR => options[:url] }.compact
@@ -219,7 +220,7 @@ module Railwatch
       # the same prepare a deploy runs, for both databases only. The host's
       # own databases are not touched, and a schema file is never written.
       def prepare_local_databases
-        return unless options[:local]
+        return unless local?
         return unless File.exist?("config/database.yml")
         return if @needs_bundle
         return unless defined?(Rails) && Rails.respond_to?(:application) && Rails.application
@@ -242,21 +243,26 @@ module Railwatch
       end
 
       def show_next_steps
-        if options[:local]
+        if local?
           say <<~STEPS, :green
 
             Next steps
-              1. Set the dashboard's HTTP Basic credentials (it is closed until
-                 you do):  bin/rails railwatch:authentication:configure
+              1. Restart the app and open /railwatch. In development it is open.
+              2. Before production, give it a password (it is closed there until
+                 you do):  RAILS_ENV=production bin/rails railwatch:authentication:configure
                  Using your own admin auth instead? See docs/embedded.md,
                  Authentication (base_controller_class or a routes constraint).
-              2. Restart the app and open /railwatch.
               3. #{@prepared ? "Nothing else to run. Both databases were created just now and" : "Create the two databases:  bin/rails db:prepare\n     Then"}
                  `bin/rails db:prepare` (which a deploy already runs) migrates
                  them after every gem update. With `plugin :railwatch` in
                  config/puma.rb Puma forks one Railwatch writer process that
                  writes every batch and runs the maintenance clock, so no web
                  process ever holds the telemetry database. No job worker.
+              4. Optional: mirror to Railwatch Cloud for alerts delivered even
+                 when this app is down, MCP for your AI assistant, and every
+                 app in one place. Get a token (bin/rails railwatch:token), set
+                 RAILWATCH_TOKEN, then c.export_enabled = true in the
+                 initializer (docs/embedded.md).
           STEPS
           return
         end
@@ -287,7 +293,7 @@ module Railwatch
         # This process read its configuration before the initializer was
         # written, so in local mode the doctor would report an http transport
         # with no token. The databases it would check were prepared above.
-        return if options[:local]
+        return if local?
         return unless defined?(Rails) && Rails.respond_to?(:application) && Rails.application
 
         say "\nbin/rails railwatch:doctor", :green
@@ -455,6 +461,16 @@ module Railwatch
 
       private
 
+      # Embedded unless the invocation asks for the cloud: --cloud itself, or
+      # an option that only means something there. A RAILWATCH_TOKEN already
+      # in the environment is not asking -- it is picked up when one of these
+      # is given, never used to choose the mode.
+      CLOUD_OPTIONS = %i[cloud prompt_token token_stdin url kamal_secrets].freeze
+
+      def local?
+        CLOUD_OPTIONS.none? { |name| options[name] }
+      end
+
       def resolved_token
         @resolved_token ||= begin
           value = if options[:prompt_token]

data/lib/generators/railwatch/install/templates/initializer.rb.tt

@@ -3,19 +3,19 @@
 # Railwatch: first-class monitoring for Rails. Every option here can also be
 # set by the RAILWATCH_* env var named in the comment.
 Railwatch.configure do |c|
-<% if options[:local] -%>
+<% if local? -%>
   # Telemetry stays in this app's own railwatch_telemetry database and the
-  # dashboard is served at /railwatch. No token, no cloud. Put the mount
-  # behind your own authentication; this only names who is looking.
+  # dashboard is served at /railwatch. No token, no cloud. Who can open the
+  # dashboard is the Access block below.
   c.transport = :local                              # RAILWATCH_TRANSPORT
   c.ignored_request_paths += ["/railwatch", %r{\A/railwatch/}]
   # c.issue_prefix = "APP"                          # RAILWATCH_ISSUE_PREFIX; issue keys like APP-12
   # c.repository_url = "https://github.com/you/app" # RAILWATCH_REPOSITORY_URL; source links from stack traces
   # c.retention_days = 7                            # RAILWATCH_RETENTION_DAYS; PruneTelemetryJob keeps this much
   # Access. The dashboard shows every query, log line and exception this app
-  # records, so pick one of these. Out of the box it is HTTP Basic, on and
-  # closed until credentials exist:
-  #   bin/rails railwatch:authentication:configure   (writes Rails credentials)
+  # records, so pick one of these. Out of the box it is HTTP Basic: open in
+  # development, closed everywhere else until credentials exist:
+  #   RAILS_ENV=production bin/rails railwatch:authentication:configure
   # or RAILWATCH_HTTP_BASIC_AUTH_USER / _PASSWORD.
   #
   # Using your own admin auth instead? Turn Basic off and say which, so the

data/lib/puma/plugin/railwatch.rb

@@ -21,6 +21,11 @@ Puma::Plugin.create do
   attr_reader :log_writer, :writer_pid
 
   POLL = 2
+  # How often a stopping writer is checked for, and how long a KILL is given
+  # to take before the pid is abandoned (a process stuck in disk I/O cannot
+  # die until the I/O returns, and Puma's exit should not wait for that).
+  REAP_POLL = 0.05
+  KILL_REAP = 1
 
   def start(launcher)
     @log_writer = launcher.log_writer
@@ -151,19 +156,59 @@ Puma::Plugin.create do
     log "Railwatch writer shutdown failed (#{e.class}: #{e.message})"
   end
 
-  # TERM closes the writer's listener and it drains what it is holding; the
-  # wait also reaps it, so a cluster master never leaves a zombie behind.
+  # TERM closes the writer's listener; it finishes what it is holding and
+  # exits on its own. Waited for with a deadline, not Process.wait, which
+  # has none: a writer wedged in a SQLite write or on a full disk would hold
+  # Puma's exit open for as long as it stayed wedged. KILL past the deadline.
+  # Reaped either way, so a cluster master never leaves a zombie behind.
   def stop_writer
     return unless @writer_pid
 
     Process.kill(:TERM, @writer_pid)
-    Process.wait(@writer_pid)
+    return if reaped_within?(stop_timeout)
+
+    log "Railwatch writer (pid #{@writer_pid}) did not exit within #{stop_timeout}s of TERM; killing it"
+    Process.kill(:KILL, @writer_pid)
+    log "Railwatch writer (pid #{@writer_pid}) did not exit on KILL; leaving it" unless reaped_within?(KILL_REAP)
   rescue Errno::ECHILD, Errno::ESRCH
     nil
   ensure
     @writer_pid = nil
   end
 
+  # shutdown_timeout: the same allowance this process gives its own
+  # reporter, and deliberately NOT the writer's full theoretical exit time
+  # (a sequential SHUTDOWN_DRAIN join per worker thread, its maintenance
+  # join, then its own reporter shutdown -- 13s at the defaults). Waiting
+  # that long would buy nothing: a writer killed mid-batch loses no data,
+  # because the transaction rolls back and the worker retries the batch by
+  # id against the next writer (Writer#serve says so). Exit time spent
+  # waiting for the drain is spent for nothing. An idle writer is gone in
+  # well under a second either way.
+  #
+  # This runs from at_exit, inside the container's TERM-to-KILL grace. Under
+  # Kamal that is Docker's 10s default for a proxied role that has not set
+  # `stop_timeout`, or whatever `stop_timeout` says when it has; an app that
+  # wants the writer given longer can raise RAILWATCH_SHUTDOWN_TIMEOUT to
+  # match its grace, and this bound rises with it.
+  def stop_timeout
+    [ ::Railwatch.config.shutdown_timeout.to_f, 0.0 ].max
+  end
+
+  # Non-blocking waits on a short poll; Process.wait has no timeout and a
+  # child that never exits would hold it forever. ECHILD (the cluster's
+  # wait2(-1) reaped it first) propagates to stop_writer, which reads it as
+  # gone.
+  def reaped_within?(seconds)
+    deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds
+    loop do
+      return true if Process.waitpid(@writer_pid, Process::WNOHANG)
+      return false if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
+
+      sleep REAP_POLL
+    end
+  end
+
   def log(message)
     log_writer.log(message)
   end

data/lib/railwatch/configuration.rb

@@ -81,7 +81,7 @@ module Railwatch
                   :max_view_renders_per_execution, :ignored_cache_key_prefixes,
                   :beacon_enabled, :beacon_rate_limit, :beacon_global_rate_limit, :beacon_allowed_origins,
                   :debug, :capture_default_vendor_commands,
-                  :capture_default_vendor_cache_keys, :on_unrecoverable,
+                  :capture_default_vendor_cache_keys, :on_unrecoverable, :warn_on_data_loss,
                   :capture_framework_events,
                   :tail_sample_slow_ms, :failure_context, :propagate_traces, :trace_propagation_hosts,
                   :health_interval, :capture_query_explain, :explain_threshold_ms,
@@ -195,6 +195,12 @@ module Railwatch
       @capture_default_vendor_cache_keys = env_bool("RAILWATCH_CAPTURE_DEFAULT_VENDOR_CACHE_KEYS", false)
       @capture_framework_events = env_bool("RAILWATCH_CAPTURE_FRAMEWORK_EVENTS", false)
       @on_unrecoverable = nil
+      # Off, because a gem printing into an application's own output is the
+      # gem changing that application's behaviour, and this one stays
+      # additive. Turn it on and a batch lost for good says so in one stderr
+      # line; leave it off and the loss shows behind RAILWATCH_DEBUG, or
+      # wherever on_unrecoverable routes it.
+      @warn_on_data_loss = env_bool("RAILWATCH_WARN_ON_DATA_LOSS", true)
       @beacon_enabled = env_bool("RAILWATCH_BEACON", true)
       # The beacon is unauthenticated and forces Railwatch.keep! for browser
       # errors, so without a ceiling anyone can spend an app's event quota
@@ -372,12 +378,21 @@ module Railwatch
       http_basic_auth_user.to_s.strip != "" && http_basic_auth_password.to_s.strip != ""
     end
 
+    # Development with Basic on and nothing configured is open, so a first
+    # run is `rails g railwatch:install` and a page, not a password step
+    # first. Rails already shows full error pages there. Every other
+    # environment stays closed until credentials exist.
+    def http_basic_auth_waived?
+      http_basic_auth_enabled && !http_basic_auth_configured? && defined?(Rails) && Rails.env.development?
+    end
+
     # Whether the request carries the configured HTTP Basic credentials.
     # False when Basic is on and nothing is configured (closed), true when
     # Basic is off (the host's base controller or routes constraint is the
     # gate then). Shared by the dashboard controller and the live channel.
     def http_basic_auth_ok?(request)
       return true unless http_basic_auth_enabled
+      return true if http_basic_auth_waived?
       return false unless http_basic_auth_configured?
 
       ActionController::HttpAuthentication::Basic.authenticate(request) do |user, password|

data/lib/railwatch/engine.rb

@@ -91,16 +91,27 @@ module Railwatch
       config.http_basic_auth_password ||= app.credentials.dig(:railwatch, :http_basic_auth_password)
     end
 
-    # Two things worth one line in the log at boot, because both are
+    # Three things worth one line in the log at boot, because all are
     # invisible until something is already wrong: a Rails/json pair that
-    # cannot decode, and an embedded dashboard with nothing declared in
-    # front of it.
+    # cannot decode, an embedded dashboard with nothing declared in front of
+    # it, and one that is closed to everyone because Basic has no
+    # credentials outside development.
     initializer "railwatch.warnings", after: :load_config_initializers do
       config.after_initialize do
         next unless Railwatch.enabled?
 
         Rails.logger.warn("[railwatch] #{Railwatch::JsonCompat.advice}") if Railwatch::JsonCompat.broken?
 
+        if Railwatch.config.local? && Railwatch.config.dashboard_gate == :basic &&
+           !Railwatch.config.http_basic_auth_configured? && !Rails.env.local?
+          Rails.logger.warn(
+            "[railwatch] the dashboard is closed: HTTP Basic is on and no credentials are configured for " \
+            "#{Rails.env}, so every request to it is 401. Run `RAILS_ENV=#{Rails.env} bin/rails " \
+            "railwatch:authentication:configure`, or gate it with your own auth and set " \
+            "`c.http_basic_auth_enabled = false` (docs/embedded.md)."
+          )
+        end
+
         if Railwatch.config.local? && Railwatch.config.dashboard_gate == :undeclared && !Rails.env.local?
           Rails.logger.warn(
             "[railwatch] the dashboard at the engine's mount has no gate this gem can see: HTTP Basic is off and " \

data/lib/railwatch/reporter.rb

@@ -100,10 +100,17 @@ module Railwatch
 
     def flush
       ensure_process!
-      @flush_mutex.synchronize do
+      deferred = nil
+      result = @flush_mutex.synchronize do
+        @deferred_notifications = []
         update_backpressure
         deliver_buffer
+      ensure
+        deferred = @deferred_notifications
+        @deferred_notifications = nil
       end
+      deferred&.each { |error| Railwatch.notify_unrecoverable(error) }
+      result
     end
 
     def ensure_thread
@@ -278,6 +285,19 @@ module Railwatch
       end
     end
 
+    # Everything reachable from deliver_buffer runs inside @flush_mutex, so
+    # the loss it reports cannot be handed to the application there. Ruby's
+    # Mutex is not reentrant: the documented callback is Rails.error.report,
+    # whose subscriber records the error as an exception and can end up asking
+    # this same reporter to flush -- "ThreadError: deadlock; recursive
+    # locking", rescued by notify_unrecoverable and so a callback cut off
+    # halfway, reporting nothing. Collected here and dispatched by flush once
+    # the lock is released. retain already does this for @mutex; @flush_mutex
+    # is the outer one it still sat inside.
+    def defer_notification(error)
+      @deferred_notifications ? @deferred_notifications << error : Railwatch.notify_unrecoverable(error)
+    end
+
     def deliver_buffer
       # A 401 was reported once, when the transport first saw it; after
       # that the token is wrong until the process restarts, and repeating
@@ -337,7 +357,7 @@ module Railwatch
     rescue StandardError => e
       result = Transport::Http::Result.new(ok: false, error: "#{e.class}: #{e.message}")
       batch&.records&.any? ? retain(batch, result) : delivery_succeeded
-      Railwatch.notify_unrecoverable(e)
+      defer_notification(e)
       result
     ensure
       in_flight(0, 0, 0, 0)
@@ -350,7 +370,7 @@ module Railwatch
     end
 
     def retain(batch, result)
-      @mutex.synchronize do
+      gave_up = @mutex.synchronize do
         @in_flight_records = 0
         @in_flight_dropped = 0
         @in_flight_bytes = 0
@@ -362,17 +382,7 @@ module Railwatch
           @retry_attempt = 0
           @retry_at = nil
           Railwatch.debug { "gave up on a batch of #{batch.records.size} records after #{MAX_RETRY_ATTEMPTS} retries (#{result.error || result.status}); dropped and counted" }
-          # Losing a batch is not a debug-level event: with an ingest (or an
-          # embedded writer) that never comes back this is the only place the
-          # loss is ever reported, and the dropped counter it leaves behind
-          # rides on the NEXT successful delivery, which may never happen.
-          Railwatch.notify_unrecoverable(
-            DeliveryError.new("Railwatch dropped #{batch.records.size} records after #{MAX_RETRY_ATTEMPTS} failed delivery attempts: " \
-                              "#{result.error || result.status}",
-                              status: result.status, records: batch.records.size, bytes: batch.bytes,
-                              dropped: batch.dropped, dropped_bytes: batch.dropped_bytes)
-          )
-          next
+          next true
         end
         @retry_batch = batch
         delay = retry_delay(@retry_attempt)
@@ -382,7 +392,32 @@ module Railwatch
           "retained #{batch.records.size} records after retryable delivery failure " \
             "(#{result.error || result.status}); retry #{@retry_attempt} in #{delay.round(3)}s"
         end
+        false
       end
+      return unless gave_up
+
+      # Losing a batch is not a debug-level event: with an ingest (or an
+      # embedded writer) that never comes back this is the only place the
+      # loss is ever reported, and the dropped counter it leaves behind
+      # rides on the NEXT successful delivery, which may never happen.
+      #
+      # Reported here, after the lock is released, never inside it. The
+      # callback is the app's: the documented one is Rails.error.report,
+      # whose subscriber records the error as an exception and so writes
+      # straight back into this reporter -- which needs @mutex to arm the
+      # thread or ask for a flush, and under the lock that was
+      # "ThreadError: deadlock; recursive locking" and a callback cut off
+      # halfway. A slow callback under the lock was worse: shutdown's own
+      # @mutex.synchronize and every request thread's write_now sat behind
+      # it for as long as it took. notify_unsent and delivery_rejected
+      # already call out unlocked; this was the one that did not.
+      defer_notification(
+        DeliveryError.new("Railwatch dropped #{batch.records.size} #{batch.records.size == 1 ? "record" : "records"} " \
+                          "after #{MAX_RETRY_ATTEMPTS + 1} failed delivery attempts: " \
+                          "#{result.error || result.status}",
+                          status: result.status, records: batch.records.size, bytes: batch.bytes,
+                          dropped: batch.dropped, dropped_bytes: batch.dropped_bytes)
+      )
     end
 
     def discard_unauthorized
@@ -406,7 +441,7 @@ module Railwatch
     def delivery_rejected(batch, result)
       delivery_succeeded
       detail = result.error.to_s.empty? ? "HTTP #{result.status}" : result.error
-      Railwatch.notify_unrecoverable(
+      defer_notification(
         DeliveryError.new("Railwatch ingest permanently rejected #{batch.records.size} records: #{detail}",
                           status: result.status, records: batch.records.size, bytes: batch.bytes,
                           dropped: batch.dropped, dropped_bytes: batch.dropped_bytes)
@@ -550,7 +585,8 @@ module Railwatch
       return unless should_notify
 
       Railwatch.notify_unrecoverable(
-        DeliveryError.new("Railwatch #{reason} with #{records} unsent records retained in memory (#{bytes} bytes)",
+        DeliveryError.new("Railwatch #{reason} with #{records} unsent #{records == 1 ? "record" : "records"} " \
+                          "retained in memory (#{bytes} bytes)",
                           records: records, bytes: bytes, dropped: dropped, dropped_bytes: dropped_bytes)
       )
     end

data/lib/railwatch/transport/http.rb

@@ -8,9 +8,10 @@ require "json"
 
 module Railwatch
   module Transport
-    # POSTs gzip NDJSON batches to the platform. Each call retries one raised
-    # error or 5xx response, then returns a classified, non-raising result;
-    # Reporter owns retention and backoff between calls. A 401 marks the
+    # POSTs gzip NDJSON batches to the platform. Each call makes exactly one
+    # attempt and returns a classified, non-raising result; Reporter owns
+    # retention, the retry ladder, and backoff between calls, so a retry here
+    # would multiply into its schedule rather than add to it. A 401 marks the
     # transport unauthorized so no further requests are made.
     class Http
       # The request headers a delivery carries besides the body. The receiver
@@ -87,17 +88,11 @@ module Railwatch
           dropped += over_cap
           dropped_bytes += over_cap_bytes
         end
-        attempt = 0
         begin
-          attempt += 1
           result = parse(post(body, dropped, dropped_bytes, backpressure_factor, batch_id), expected_count: sent)
-          if attempt < 2 && (500..599).cover?(result.status)
-            result = parse(post(body, dropped, dropped_bytes, backpressure_factor, batch_id), expected_count: sent)
-          end
           apply_status_policy(result)
           result
         rescue StandardError => e
-          retry if attempt < 2
           Result.new(ok: false, error: "#{e.class}: #{e.message}")
         end
       end

data/lib/railwatch/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railwatch
-  VERSION = "0.5.1"
+  VERSION = "0.6.1"
 end