railwatch 0.1.3 → 0.2.0.pre1

data/docs/embedded.md

@@ -0,0 +1,342 @@
+# Embedded mode: the dashboard inside your app
+
+Railwatch can keep every record in your own application and serve the
+full dashboard at `/railwatch`, with no token and no cloud. The gem's
+reporter, buffer and sampling are the same; the only difference is
+where a batch ends up. In embedded mode it is written straight into a
+SQLite database your app owns, and the dashboard reads it back from
+there.
+
+Use it when one server runs the app. Telemetry lands in a file next to
+your other SQLite databases, so several servers would each see only
+their own slice. For more than one server, or for a team that wants one
+place for many apps, point the gem at Railwatch Cloud instead
+([Getting started](getting-started.md)); the two are switchable with
+one setting.
+
+## Install
+
+```sh
+bundle add railwatch
+bin/rails generate railwatch:install --local
+```
+
+Restart the app and open `/railwatch`. Then `bin/rails railwatch:doctor`
+checks the wiring. The generator creates and migrates both databases
+itself; `bin/rails db:prepare`, which a deploy already runs, migrates
+them after every gem update.
+
+The engine needs Active Job (its grouping and scan jobs are Active Job
+classes even though embedded mode calls them directly) and loads it
+itself. Action Cable is optional: with it the dashboard updates live,
+without it (`rails new --minimal`) the pages refresh on navigation.
+
+## Your application's own database
+
+Embedded mode does not care what your application runs on. The two
+databases it adds are SQLite files either way, so the generated entries
+name `adapter: sqlite3` themselves rather than inheriting your default
+block, and they need no `&default` anchor to exist.
+
+On a PostgreSQL or MySQL app that means the install is two commands
+rather than one, because SQLite's adapter gem will not be in your bundle:
+
+```sh
+bin/rails generate railwatch:install --local   # adds gem "sqlite3", writes the config
+bundle install
+bin/rails db:prepare                           # creates the two SQLite files
+```
+
+Verified end to end on both. On a PostgreSQL app and on a MySQL app, the
+application's own four databases stay where they were, Railwatch's two are
+files under `storage/`, and neither server gains a single Railwatch table.
+
+What `--local` writes, on top of the usual install:
+
+- `config/initializers/railwatch.rb` with `c.transport = :local` and the
+  dashboard's own paths excluded from request capture.
+- Two databases in every environment of `config/database.yml`:
+  `railwatch` (issues, comments, saved views, thresholds, deploys:
+  small, permanent) and `railwatch_telemetry` (everything the app
+  reports: written continuously, pruned nightly). A flat
+  `development:` entry is nested under `primary:` first, since named
+  databases need that form. Each entry's `migrations_paths` points into
+  the gem, so `db:prepare` builds the tables from the gem's own
+  migrations and nothing is copied into `db/`.
+- `mount Railwatch::Engine, at: "/railwatch"`, as always.
+
+Nothing touches your primary database.
+
+## Upgrading
+
+The engine's tables migrate the way Active Storage's do: the migrations
+live in the gem and each database's `migrations_paths` points at them.
+After `bundle update railwatch`, run `bin/rails db:prepare` (or
+`db:migrate`) and whatever is new applies; a deploy that already runs
+one of those needs nothing extra. `bin/rails railwatch:doctor` reports
+pending migrations for both databases. Tables in the `railwatch` file
+are prefixed `railwatch_`; the telemetry file's tables are the hosted
+platform's schema and are unprefixed.
+
+## What you get
+
+Every page of Railwatch Cloud: requests, jobs, scheduled tasks,
+commands, queries, spans, exceptions, logs, cache, mail, notifications,
+broadcasts, outgoing requests, LLM calls, storage, views, transactions,
+deprecations, processes, releases, deploys, users, tenants, visits,
+thresholds; issues with comments, assignment, merge and split; saved
+views; alert rules and the alert log. Live updates arrive over Action
+Cable when the app has it. The dashboard is a prebuilt bundle shipped
+inside the gem, so the app needs no Node, no Vite and no asset pipeline
+integration.
+
+Not in embedded mode: accounts and members (the operator is whoever your
+app lets through), integrations (alerts are recorded, not delivered),
+and the MCP server.
+
+## Authentication
+
+The dashboard shows every query, log line and exception your app
+produced, so it works the way Mission Control Jobs does: **HTTP Basic
+authentication is on and closed by default**. With no credentials
+configured every dashboard request is 401, and `railwatch:doctor` says
+so. Set them with
+
+```sh
+bin/rails railwatch:authentication:configure
+RAILS_ENV=production bin/rails railwatch:authentication:configure
+```
+
+which writes them to that environment's Rails credentials:
+
+```yml
+railwatch:
+  http_basic_auth_user: ops
+  http_basic_auth_password: secret
+```
+
+`RAILWATCH_HTTP_BASIC_AUTH_USER` and `RAILWATCH_HTTP_BASIC_AUTH_PASSWORD`,
+or `c.http_basic_auth_user =` / `c.http_basic_auth_password =` in the
+initializer, do the same. The live-update channel checks the same
+credentials (the browser sends them on the WebSocket handshake).
+
+### Your own authentication
+
+Two ways, both from Mission Control's playbook. Either lets an admin of
+your app in with no second password. Turn Basic off when you use one,
+or both gates apply.
+
+A base controller. Every dashboard controller inherits from it, so its
+`before_action` runs first:
+
+```ruby
+c.http_basic_auth_enabled = false
+c.base_controller_class = "AdminController"   # requires an admin, or redirects to sign-in
+```
+
+Your controller's code runs inside the engine, whose route helpers take
+precedence; reach your app's with `main_app.root_path`.
+
+Or a routes constraint, which keeps the engine out of it entirely (for
+example with the sessions Rails' authentication generator creates):
+
+```ruby
+# config/routes.rb
+constraints ->(request) { Session.find_by(id: request.cookie_jar.signed[:session_id])&.user&.admin? } do
+  mount Railwatch::Engine, at: "/railwatch"
+end
+```
+
+Requests that fail the constraint never reach the engine. A constraint is
+invisible to the gem, though, so say that the engine's own gate is off on
+purpose:
+
+```ruby
+c.http_basic_auth_enabled = false
+c.dashboard_open = true   # "something in front of the mount gates this"
+```
+
+### Deliberately public
+
+A dashboard on a private network or behind a VPN can be open, and saying
+so is a setting rather than an omission:
+
+```ruby
+c.http_basic_auth_enabled = false
+c.dashboard_open = true
+```
+
+With Basic off and none of `base_controller_class`, `dashboard_user` or
+`dashboard_open` set, the gem cannot tell a deliberate choice from a
+forgotten one. It serves the dashboard (a routes constraint it cannot see
+is a legitimate answer) but logs a warning at every boot outside
+development, `railwatch:doctor` reports the gate as undeclared, and live
+updates are refused. Declaring any of the three settles it.
+
+### Live updates and `/cable`
+
+Action Cable runs on your application's own `/cable` endpoint, not under
+the engine's mount, so a routes constraint around `/railwatch` does not
+cover it and a base controller cannot reach it. The live-update channel
+therefore follows what you declared: HTTP Basic credentials are checked
+on the WebSocket handshake, a `dashboard_user` resolver is consulted,
+`dashboard_open` and `base_controller_class` are taken at their word, and
+an undeclared gate is refused. The channel carries an ingest ping (a
+timestamp and per-type counts) and never telemetry records.
+
+### Naming the operator
+
+Comments, saved views and issue activity record who did them. Give the
+initializer a resolver and the dashboard shows that person instead of a
+single "Operator":
+
+```ruby
+c.dashboard_user = ->(request) do
+  user = User.find_by(id: request.session[:user_id])
+  user && { id: user.id, name: user.name, email: user.email }
+end
+```
+
+## The writer process
+
+Puma forks one Railwatch writer from its master when `config/puma.rb`
+carries the plugin (`--local` adds it):
+
+```ruby
+plugin :railwatch if defined?(Railwatch)
+```
+
+Every web worker keeps its reporter thread, but instead of writing
+SQLite it hands each batch to the writer over a Unix socket
+(`tmp/sockets/railwatch-writer.sock`, `RAILWATCH_WRITER_SOCKET`). The
+socket's directory is created mode 0700 and the socket 0600, so only
+the app's own user can reach it; keep it that way if you move it. Linux
+caps the whole path at 108 bytes, so an app checked out deep in the
+filesystem should point this at a directory of its own under
+`/run/user/$UID` or `/tmp` (not a bare file in `/tmp`). Single and
+cluster mode alike: a default Rails 8 app runs Puma with no workers, and
+its batches come off its request threads just the same. The
+writer maps the records, writes both databases, folds the rollups,
+groups exceptions into issues and runs the maintenance clock below. It
+is the only process that ever holds the telemetry database's write lock,
+and its Ruby interpreter is its own, so none of that work is ever
+interleaved with a request. Same shape as Solid Queue's
+`solid_queue_mode :fork`: it exits when Puma does, and Puma restarts it
+if it dies. While it is down the reporter keeps batches in memory, with
+the same byte ceiling and backoff as the HTTP transport, and every
+batch carries an id the writer records inside the write transaction, so
+a batch delivered twice is written once.
+
+The writer judges its own health. A single batch write that runs past
+sixty seconds is a stuck writer, not a slow one (a lock that never
+clears, a lost connection), and the process exits so Puma restarts it;
+the batch is retained on the worker and written by the new writer. The
+doctor reports both whether the socket answers and when the last batch
+was actually written, since a process that is alive and a process that
+is doing its job are different questions.
+
+Whether a writer is expected decides what a missing one means. Puma
+workers under the plugin expect one: a socket that is absent or not
+answering is a writer that is starting or restarting, and they retain
+batches and retry for as long as it takes. A process with no plugin
+(`bin/rails runner`, a Solid Queue worker, a `rails server` without it,
+the test suite) expects none, says so once under `RAILWATCH_DEBUG`, and
+writes its batches in-process instead. That fallback is provisional: the
+socket is tried again every 30 seconds, so a process that started before
+the writer did hands the work back as soon as one is listening. What
+changes is which process pays for the write, not whether the write
+happens.
+
+Retention while a writer is away is bounded, not infinite. A writer that
+is restarting is back in seconds; one that is missing for a minute (an
+unwritable socket directory, a fork that keeps failing) is treated as
+absent and the worker writes its own batches again, still re-checking, so
+the records are kept rather than retained to the reporter's retry cap. A
+batch that does exhaust that ladder is counted as dropped and reported
+through `Railwatch.on_unrecoverable`, so loss is never silent. A Puma
+phased restart stops the writer and starts a fresh one once the new
+workers are up.
+
+## Maintenance
+
+Railwatch needs no job worker and nothing in `config/recurring.yml`.
+The work that keeps the dashboard current happens in two places:
+
+- **As each batch lands.** The writer writes the batch, folds its rows
+  into the hour's rollups, and groups any exceptions into issues, all
+  before it picks up the next batch. Counts, percentiles and the issues
+  list move with every batch.
+- **On Railwatch's own clock.** The writer runs a `railwatch-maintenance`
+  thread that wakes every 30 seconds. Without a writer, every web and
+  worker process runs one, and one process at a time runs each task,
+  claimed through a lease row in the `railwatch` database, so a Puma
+  cluster and a Solid Queue worker on the same server do not all prune
+  at once.
+
+| Task | Cadence | What it does |
+| --- | --- | --- |
+| drain follow-ups | every minute | Finishes the exception grouping of any batch whose process died right after the batch committed |
+| release health | every minute | Hourly crash-free aggregates for the current and previous hour |
+| rollup reconcile | hourly | Recomputes the previous hour's rollups from raw rows, for records that arrived after their hour closed |
+| performance scan | every 5 minutes | Threshold breaches become issues |
+| anomaly scan | every 5 minutes | Anomaly rules, when any are enabled |
+| scheduled tasks | every 10 minutes | Missed and late scheduled tasks |
+| auto-resolve | daily | Resolves issues quiet for 14 days |
+| prune | daily | Deletes telemetry older than `retention_days`, then `ANALYZE` |
+
+Because the clock lives in the web process, it keeps running when the
+job worker is down, which is exactly when "scheduled task X missed its
+run" needs to be raised. With Solid Queue, the issue also says which of
+three things happened: the scheduler never enqueued the run, it was
+enqueued but no worker is running, or a worker is alive and it is
+waiting behind a backlog.
+
+Every batch is written exactly once. The reporter gives each batch an
+id before its first attempt and the write records it in the same
+transaction as the rows, so a write that fails (the file locked by a
+backup, say) is retried with backoff and a retry of a batch that did
+commit is a no-op. `bin/rails railwatch:doctor` reports the last
+tick. If you installed a pre-release that added `Railwatch::*` entries to
+`config/recurring.yml`, remove them; the doctor says so too.
+
+## Settings
+
+```ruby
+Railwatch.configure do |c|
+  c.transport = :local            # RAILWATCH_TRANSPORT=local
+  c.issue_prefix = "SHOP"         # RAILWATCH_ISSUE_PREFIX; keys like SHOP-12
+  c.repository_url = "https://github.com/you/shop"  # RAILWATCH_REPOSITORY_URL
+  c.retention_days = 7            # RAILWATCH_RETENTION_DAYS
+  c.http_basic_auth_enabled = true    # RAILWATCH_HTTP_BASIC_AUTH_ENABLED; credentials from Rails credentials or env
+  c.base_controller_class = "ActionController::Base"  # RAILWATCH_BASE_CONTROLLER_CLASS
+  c.dashboard_open = false            # RAILWATCH_DASHBOARD_OPEN; "yes, public, on purpose"
+  c.dashboard_user = ->(request) { ... }
+end
+```
+
+Every other option (sampling, redaction, ignored record types) applies
+unchanged. The default samples every execution; set `c.sample` lower on
+a busy app.
+
+## Deploys
+
+`bin/rails railwatch:deploy` records the marker in the app's own
+database instead of posting it, and the Kamal post-deploy hook does the
+same when `RAILWATCH_TRANSPORT=local` is set on the deployer.
+
+## Storage and overhead
+
+Telemetry is written by the reporter thread in batches, never on a
+request. The cost on the request path is the same instrumentation the
+cloud transport pays; the write itself adds under a millisecond at the
+99th percentile at a hundred requests per second on a single Puma. The
+telemetry database grows with traffic and sampling and is pruned to
+`retention_days`; the `railwatch` database stays small. Both are plain
+SQLite files in `storage/`, so Litestream or a volume snapshot covers
+them.
+
+## Switching to the cloud later
+
+Set a token and drop `c.transport = :local` (or set
+`RAILWATCH_TRANSPORT=http`). The local databases can stay; the dashboard
+at `/railwatch` keeps reading what is there until it is pruned.

data/docs/getting-started.md

@@ -4,6 +4,11 @@ Five minutes from `bundle add` to a request on the dashboard, on a
 Rails 8 app. Everything below is the gem's own generator and rake tasks;
 nothing else has to be wired by hand.
 
+Running one server and want the dashboard inside the app itself, with
+no token and no cloud? That is `bin/rails generate railwatch:install
+--local`; see [Embedded mode](embedded.md). The rest of this page is
+the cloud install.
+
 ## 1. Add the gem
 
 ```sh

data/docs/records.md

@@ -512,7 +512,13 @@ not a dependency — an app without it never emits these. Requires RubyLLM
 
 The model call also appears as an `outgoing_request`, since it is an HTTP
 call like any other. The two are different grains on purpose: the
-`outgoing_request` is the HTTP truth, the `llm_call` is what it cost.
+`outgoing_request` is the HTTP truth, the `llm_call` is what it cost. That
+difference is useful: RubyLLM retries through Faraday, so one `llm_call`
+with several `outgoing_request` rows against it in the same execution is a
+call that was retried. Over a window, `outgoing_requests - llm_calls` to the
+same provider host is the number of *extra attempts*, not a rate -- the
+share of calls that were retried needs counting the calls with more than one
+request against them, which the execution id supports.
 
 **Token counts and cost differ by RubyLLM version.** 1.16 reports token
 counts and no cost at all. 2.0 reports both, from its usage ledger, and
@@ -556,6 +562,15 @@ so cost is always complete.
 | `workflow_step_id` | Step identifier within the workflow (2.0+). |
 | `workflow_step_name` | Step name (2.0+). |
 | `workflow_step_parent_id` | Enclosing step, for nested steps — what reconstructs the tree (2.0+). |
+| `finish_reason` | Why the model stopped: `stop`, `max_tokens`, `tool_calls`, `content_filter`, or whatever the provider spelled it. `max_tokens` means the answer was cut off -- without this a truncated extraction reads exactly like a complete one. |
+| `provider_request_id` | The provider's own id for the request, read from the response headers (`request-id`, `x-request-id`, `x-amzn-requestid`). The only key that joins this record to the provider's side of it, and what a support ticket asks for. |
+| `tools` | Comma-separated names of the tools the model could reach, first 50. `tool_count` says how many; retracing needs which. |
+| `cost_reported` | Whether the provider priced the call itself, or the amount is an estimate from the model registry. Null on gems or operations that report no cost. |
+| `attachments` | How many files the last user turn carried. Absent when it carried none. Only the last turn is measured: earlier turns were counted by the calls that sent them. |
+| `attachment_types` | What they were, by category and count, e.g. `imagex2,pdf`. Categories are RubyLLM's: image, pdf, audio, video, text, document, unknown. On a document-reading call the attachments are most of the input tokens, so without this an expensive scan is indistinguishable from an expensive prompt. |
+| `attachment_names` | Filenames, only when `config.capture_llm_content` is on. A filename like `ACME_invoice_88231.pdf` is business data, not metadata, so it follows the same switch as prompts. |
+| `params` | JSON of the settings that produced the answer, so a surprising one can be reproduced: `temperature`, `max_output_tokens`, `tool_choice`, `tool_call_limit`, `thinking`, `caching`, `citations`, whether a `schema` was used, plus the per-operation ones (`dimensions`, `task_type`, `size`, `count`, `voice`, `format`, `language`, `pages`, `document_count`, `top_n`), `server_tools` and the provider's `server_tool_use` counters. `provider_options` is included, filtered twice: through the app's own parameter filter, and again against the credential-name matcher that catches `X-Api-Key` on a header -- an `api_key` passed per call sails straight through a password-shaped filter. For `operation: "tool"` this holds the tool result's class instead. |
+| `tool_call_id` | The provider's id for a tool invocation, for joining a tool call to the assistant turn that asked for it. |
 | `prompt` | Last user turn, only when `config.capture_llm_content` is on (off by default). Capped at 4 KiB of bytes. |
 | `completion` | The reply, same condition and cap. For `operation: "tool"` these two hold the tool's arguments and result instead. |
 

data/lib/generators/railwatch/install/install_generator.rb

@@ -8,8 +8,11 @@ module Railwatch
     class InstallGenerator < Rails::Generators::Base
       source_root File.expand_path("templates", __dir__)
 
-      desc "Creates config/initializers/railwatch.rb, a Kamal post-deploy hook, the browser client, and wires the test helpers."
+      desc "Creates config/initializers/railwatch.rb, a Kamal post-deploy hook, the browser client, and wires the test helpers. " \
+           "With --local, also the two SQLite databases the in-app dashboard needs."
 
+      class_option :local, type: :boolean, default: false,
+                           desc: "Keep telemetry in this app and serve the dashboard at /railwatch: no token, no cloud."
       class_option :token, type: :string,
                            desc: "Deprecated: token in process arguments. Prefer --prompt-token, --token-stdin, or RAILWATCH_TOKEN."
       class_option :prompt_token, type: :boolean, default: false,
@@ -43,6 +46,75 @@ module Railwatch
         template "initializer.rb", "config/initializers/railwatch.rb"
       end
 
+      # Not a gemspec dependency: the breakage is the host application's
+      # either way (its own sessions are failing), so this offers the pin in
+      # the app's Gemfile, where the app can drop it the day Rails ships the
+      # fix, rather than constraining every bundle that installs this gem.
+      def pin_json_when_it_cannot_decode
+        return unless Railwatch::JsonCompat.broken?
+        return say("#{Railwatch::JsonCompat.advice} (no Gemfile here to add it to)", :yellow) unless File.exist?("Gemfile")
+
+        contents = File.read("Gemfile")
+        return say_status(:identical, "Gemfile (json pin)", :blue) if contents.match?(/^\s*gem ["']json["']/)
+
+        append_to_file "Gemfile", "#{contents.end_with?("\n") ? "" : "\n"}\n" \
+                                  "# #{Railwatch::JsonCompat::ISSUE}: Rails cannot decode with json 3 yet. Remove when it can.\n" \
+                                  "#{Railwatch::JsonCompat::PIN}\n"
+        @needs_bundle = true
+        say "#{Railwatch::JsonCompat.advice} Added the pin to your Gemfile; run `bundle install`, then " \
+            "`bin/rails db:prepare`.", :yellow
+      end
+
+      # Embedded mode stores telemetry in SQLite files whatever the app's own
+      # database is, so an app on PostgreSQL or MySQL needs the adapter gem
+      # added before those files can be created.
+      def ensure_sqlite3_gem
+        return unless options[:local]
+        return if Gem.loaded_specs.key?("sqlite3")
+        return say("--local needs the sqlite3 gem for its two databases; add `gem \"sqlite3\"` and re-run.", :yellow) unless File.exist?("Gemfile")
+
+        contents = File.read("Gemfile")
+        unless contents.match?(/^\s*gem ["']sqlite3["']/)
+          append_to_file "Gemfile", %(#{contents.end_with?("\n") ? "" : "\n"}\n# Railwatch (embedded) keeps its telemetry in two SQLite files of its own.\ngem "sqlite3"\n)
+        end
+        @needs_bundle = true
+        say "Added `gem \"sqlite3\"` to the Gemfile: Railwatch's two databases are SQLite files whatever this app's " \
+            "own database is. Run `bundle install`, then `bin/rails db:prepare` to create them.", :yellow
+      end
+
+
+      # Two databases of its own, never the app's primary: `railwatch` for
+      # what people author (issues, comments, saved views, thresholds) and
+      # `railwatch_telemetry` for what the app reports, which is written
+      # continuously and pruned. Their migrations live in the gem; the
+      # database entries point migrations_paths at them, so db:prepare
+      # creates the tables now and migrates them after every gem update.
+      # Nothing is copied into the app.
+      def configure_local_databases
+        return unless options[:local]
+
+        return say("--local: no config/database.yml found; add railwatch and railwatch_telemetry databases yourself (docs/embedded.md).", :yellow) unless File.exist?("config/database.yml")
+
+        contents = File.read("config/database.yml")
+        updated = self.class.database_yml_with_railwatch(contents)
+        return say_status(:identical, "config/database.yml", :blue) if updated == contents
+
+        create_file "config/database.yml", updated, force: true
+      end
+
+      # The writer process: one per Puma master, forked by the gem's Puma
+      # plugin, so batches are mapped and written outside the web workers.
+      def configure_local_writer
+        return unless options[:local]
+        return say("--local: no config/puma.rb found; add `plugin :railwatch` to your Puma config yourself (docs/embedded.md).", :yellow) unless File.exist?("config/puma.rb")
+
+        contents = File.read("config/puma.rb")
+        updated = self.class.puma_rb_with_railwatch(contents)
+        return say_status(:identical, "config/puma.rb", :blue) if updated == contents
+
+        create_file "config/puma.rb", updated, force: true
+      end
+
       def create_kamal_hook
         return unless File.exist?("config/deploy.yml")
         template "post-deploy", ".kamal/hooks/post-deploy"
@@ -92,6 +164,8 @@ module Railwatch
       # A token lands in .env only when Git confirms the file is ignored.
       # URLs are not secret and can still be written to a tracked dotenv file.
       def write_env
+        return if options[:local]
+
         token = resolved_token
         if options[:token]
           say("--token exposes #{Railwatch::SecretSafety.token_preview(options[:token])} in process arguments; " \
@@ -142,7 +216,53 @@ module Railwatch
         create_file "config/deploy.yml", updated, force: true
       end
 
+      # The two databases exist and are migrated when the generator returns:
+      # config/database.yml was just rewritten, so this re-reads it and runs
+      # the same prepare a deploy runs, for both databases only. The host's
+      # own databases are not touched, and a schema file is never written.
+      def prepare_local_databases
+        return unless options[:local]
+        return unless File.exist?("config/database.yml")
+        return if @needs_bundle
+        return unless defined?(Rails) && Rails.respond_to?(:application) && Rails.application
+
+        say "\nbin/rails db:prepare (railwatch, railwatch_telemetry)", :green
+        require "active_record"
+        ActiveRecord::Base.configurations = Rails.application.config.database_configuration
+        %w[railwatch railwatch_telemetry].each do |name|
+          db_config = ActiveRecord::Base.configurations.configs_for(env_name: Rails.env, name: name)
+          next say("  #{name}: not in config/database.yml for #{Rails.env}", :yellow) unless db_config
+
+          ActiveRecord::Tasks::DatabaseTasks.with_temporary_pool_for_each(env: Rails.env, name: name) do
+            ActiveRecord::Tasks::DatabaseTasks.migrate
+          end
+          say_status :prepared, "#{name} (#{db_config.database})", :green
+        end
+        @prepared = true
+      rescue StandardError => e
+        say "Could not prepare the Railwatch databases here (#{e.class}: #{e.message}). Run `bin/rails db:prepare` yourself.", :yellow
+      end
+
       def show_next_steps
+        if options[:local]
+          say <<~STEPS, :green
+
+            Next steps
+              1. Set the dashboard's HTTP Basic credentials (it is closed until
+                 you do):  bin/rails railwatch:authentication:configure
+                 Using your own admin auth instead? See docs/embedded.md,
+                 Authentication (base_controller_class or a routes constraint).
+              2. Restart the app and open /railwatch.
+              3. #{@prepared ? "Nothing else to run. Both databases were created just now and" : "Create the two databases:  bin/rails db:prepare\n     Then"}
+                 `bin/rails db:prepare` (which a deploy already runs) migrates
+                 them after every gem update. With `plugin :railwatch` in
+                 config/puma.rb Puma forks one Railwatch writer process that
+                 writes every batch and runs the maintenance clock, so no web
+                 process ever holds the telemetry database. No job worker.
+          STEPS
+          return
+        end
+
         say <<~STEPS, :green
 
           Next steps
@@ -166,6 +286,10 @@ module Railwatch
       # note below says so rather than letting a ✗ look like a broken install.
       def run_doctor
         return unless options[:doctor]
+        # This process read its configuration before the initializer was
+        # written, so in local mode the doctor would report an http transport
+        # with no token. The databases it would check were prepared above.
+        return if options[:local]
         return unless defined?(Rails) && Rails.respond_to?(:application) && Rails.application
 
         say "\nbin/rails railwatch:doctor", :green
@@ -199,6 +323,103 @@ module Railwatch
         insert_lines(lines, block_end(lines, secret_start), "    - #{name}\n")
       end
 
+      # Spelled out rather than `<<: *default`, on purpose. The host's default
+      # block carries ITS adapter: on a PostgreSQL or MySQL app inheriting it
+      # would ask that server for a database called
+      # "storage/production_railwatch.sqlite3". These two are always SQLite
+      # files the app owns, whatever the app's own database is, so they name
+      # their adapter themselves -- which also means the file needs no
+      # `&default` anchor to exist at all.
+      RAILWATCH_DATABASES = <<~YAML
+        railwatch:
+          adapter: sqlite3
+          database: storage/%<env>s_railwatch.sqlite3
+          pool: <%%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %%>
+          timeout: 5000
+          migrations_paths: <%%= Railwatch.migrations_path(:railwatch) %%>
+          schema_dump: false
+        railwatch_telemetry:
+          adapter: sqlite3
+          database: storage/%<env>s_railwatch_telemetry.sqlite3
+          pool: <%%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %%>
+          timeout: 5000
+          migrations_paths: <%%= Railwatch.migrations_path(:railwatch_telemetry) %%>
+          schema_dump: false
+          pragmas:
+            journal_mode: wal
+            synchronous: normal
+            mmap_size: 134217728
+            cache_size: -65536
+            temp_store: memory
+      YAML
+
+      # Adds the railwatch and railwatch_telemetry databases to every
+      # environment in config/database.yml. A flat environment
+      # (`development:` straight to `<<: *default`) becomes a `primary:`
+      # entry first, since named databases need the nested form. Text
+      # insertion rather than a YAML round trip, for the same reason as
+      # deploy_yml_with_secret: the comments are most of the file.
+      def self.database_yml_with_railwatch(contents)
+        lines = contents.lines
+        # Rails 8.1's non-Docker template leaves every production `database:`
+        # commented out ("path/to/persistent/storage/..."), so db:prepare
+        # cannot run in production at all until the host fills them in. The
+        # storage/ paths are what the Docker template writes and what
+        # config/deploy.yml mounts; use them.
+        lines = lines.map do |line|
+          line.sub(%r{\A(\s+)# database: path/to/persistent/storage/(\S+)$}, '\1database: storage/\2')
+        end
+        environments_in(lines).each do |env|
+          start = lines.index { |line| line.match?(/\A#{env}:\s*(#.*)?$/) }
+          next unless start
+
+          stop = block_end(lines, start)
+          block = lines[(start + 1)...stop]
+          next if block.any? { |line| line.match?(/\A\s+railwatch_telemetry:\s*$/) }
+
+          nested = block.any? { |line| line.match?(/\A  [a-z_]+:\s*$/) }
+          unless nested
+            lines[(start + 1)...stop] = block.map { |line| line.strip.empty? ? line : "  #{line}" }
+            lines.insert(start + 1, "  primary:\n")
+            stop += 1
+          end
+          entries = format(RAILWATCH_DATABASES, env: env).lines.map { |line| "  #{line}" }
+          lines = insert_lines(lines, stop, entries.join).lines
+        end
+        lines.join
+      end
+
+      # Unconditional: `bundle exec puma` evaluates this file before it loads
+      # the Rack app, so `if defined?(Railwatch)` would be false there and the
+      # writer would never start. The plugin itself is inert when Railwatch is
+      # absent, off, or not in embedded mode.
+      PUMA_PLUGIN_LINES = <<~RUBY
+
+        # Railwatch (embedded): fork one writer process from Puma so telemetry
+        # is mapped and written outside the processes serving requests.
+        plugin :railwatch
+      RUBY
+
+      # Appends the plugin line once. Puma's config is plain Ruby evaluated top
+      # to bottom, so the end of the file is always a valid place for it.
+      def self.puma_rb_with_railwatch(contents)
+        return contents if contents.include?("plugin :railwatch")
+
+        "#{contents.sub(/\n*\z/, "\n")}#{PUMA_PLUGIN_LINES}"
+      end
+
+      # Every environment the file defines, not a fixed three: an app with a
+      # `staging` (or `review`, or `qa`) environment needs the databases
+      # there too, and hardcoding names silently left it without them. A
+      # top-level key with a block under it, minus YAML's own anchors.
+      NON_ENVIRONMENT_KEYS = %w[default shared].freeze
+
+      def self.environments_in(lines)
+        lines.filter_map { |line| line[/\A([a-z_][a-z0-9_]*):\s*(?:&\S+\s*)?(?:#.*)?$/, 1] }
+             .reject { |name| NON_ENVIRONMENT_KEYS.include?(name) }
+             .uniq
+      end
+
       # Index of the first line after the block opened at `start`: the next
       # line indented no more deeply than it, ignoring blanks and comments,
       # then backed up over any trailing blank lines so an insertion lands