railwatch 0.1.0

data/README.md

@@ -0,0 +1,226 @@
+# Railwatch
+
+First-class monitoring for Rails. One gem instruments requests, jobs,
+scheduled tasks, commands, queries, exceptions, cache, mail, broadcasts,
+outgoing HTTP, storage, views, and logs, links them into one trace per
+execution, and ships them to Railwatch Cloud for about half a millisecond
+per request plus tens of microseconds per query, with zero writes to your
+database.
+
+## Install
+
+```sh
+bundle add railwatch            # 1. add the public gem
+bin/rails generate railwatch:install --prompt-token  # 2. hidden token input plus app wiring
+bin/rails railwatch:doctor                    # 3. check every piece is wired up after restart
+```
+
+The gem, its Ruby namespace, and its require path share one name:
+`railwatch`, `Railwatch::*`, `require "railwatch"`. For a self-hosted
+deployment or an unreleased revision, use the Git source instead:
+
+```ruby
+gem "railwatch", github: "Rebulk/railwatch"
+```
+
+The generator writes `config/initializers/railwatch.rb`, mounts the beacon
+engine, and — where the app already has them — adds a Kamal `post-deploy`
+hook, the Inertia browser client with its `startRailwatch()` call, and
+`require "railwatch/rspec"` in `spec/rails_helper.rb`. `railwatch:doctor` prints
+a ✓/✗ checklist of all of it and exits non-zero if the token is missing or
+the ingest host is unreachable.
+
+Pass `--prompt-token` (and `--url=` when self-hosting) to read the token
+without echo or process-argument exposure. The generator writes it to `.env`
+only when Git confirms that file is ignored; otherwise it points you to Rails
+credentials or your deployment secret manager without printing the value. Add
+`--kamal-secrets` to wire `RAILWATCH_TOKEN` through `.kamal/secrets` and
+`config/deploy.yml`. It finishes by running `railwatch:doctor` for you.
+`bin/rails railwatch:token` says where to get a token; `bin/rails railwatch:mcp`
+prints ready-to-paste MCP client configuration.
+
+## Documentation
+
+- [Getting started](docs/getting-started.md) — five-minute install for a
+  Rails 8 app, the three optional lines, and deploying with Kamal, Docker,
+  Heroku, Render, or none of them.
+- [Configuration](docs/configuration.md) — every option and `RAILWATCH_*`
+  variable, field by field.
+- [Record types](docs/records.md) — every record Railwatch ships and every
+  attribute on it, sourced from the code that builds it.
+- [Testing](docs/testing.md) — the RSpec and Minitest matchers, and a CI
+  performance gate.
+- [AI assistants and MCP](docs/ai-and-mcp.md) — connecting Claude Code,
+  Cursor, VS Code, or Zed to your production data.
+- [Replacing Sentry](docs/replacing-sentry.md) — a step-by-step migration,
+  option by option and call site by call site.
+- [Coming from Laravel Nightwatch](docs/replacing-nightwatch.md) — the
+  record-type mapping and the sampling model, for Laravel people.
+- [Self-hosting](docs/self-hosting.md) — pointing the gem at your own
+  Railwatch Cloud.
+- [Troubleshooting](docs/troubleshooting.md) — every failure mode, paired
+  with the `railwatch:doctor` line it shows up as.
+- [FAQ](docs/faq.md) — overhead numbers, retention, PII posture, SQLite.
+- [Security](docs/security.md) — transport, capture defaults, the browser
+  beacon, and application responsibilities.
+
+AI coding agents working on an app that uses Railwatch: [`llms.txt`](llms.txt)
+and [`AGENTS.md`](AGENTS.md).
+
+Configuration lives in `config/initializers/railwatch.rb`; every option has
+a `RAILWATCH_*` environment variable. Sampling is decided once per execution:
+a sampled-in request ships its whole tree, a sampled-out one ships nothing
+except unhandled exceptions.
+
+```ruby
+Railwatch.configure do |c|
+  c.sample = { requests: 0.1, jobs: 1.0 }
+  c.user { |u| { id: u.id, name: u.name, email: u.email } }
+end
+
+class ReportsController < ApplicationController
+  railwatch_sample 0.01, only: :index
+end
+
+Railwatch.ignore { ExpensiveSync.run }
+Railwatch.context(tenant: org.slug, plan: org.plan)
+```
+
+Time any block of your own code as a `span` on the surrounding request,
+job, or command — the block's value is returned untouched:
+
+```ruby
+Railwatch.span("pdf.render", template: "invoice", pages: 12) { renderer.call }
+```
+
+When a span isn't enough to say where the time went, Railwatch can attach a
+real stack profile to an execution. Add `gem "vernier"` (Ruby ≥ 3.2) or
+`gem "stackprof"` to the Gemfile and set `c.profile_sample = 0.01` to
+profile 1% of executions, or `c.profile_slow_ms = 500` alongside
+`c.tail_sample_slow_ms` to profile the slow ones. The collapsed stacks
+ship as their own `profile` record, gzipped, and the request or job it
+belongs to is marked `profiled`. Off by default, and one Float comparison
+per execution while it stays off.
+
+Sampling can also be decided at the *end* of an execution instead of the
+start: set `c.tail_sample_slow_ms = 500` (or call `Railwatch.keep!`) and a
+head-sampled-out request that turns out to be slow, or to have raised,
+ships its whole tree anyway. Outgoing HTTP carries a W3C `traceparent`,
+and an inbound one is adopted, so a trace spans services.
+
+Inertia apps get real page-visit timing by calling `startRailwatch()` from the
+generated `app/frontend/lib/railwatch.ts`. Server-side rendering is timed
+automatically wherever `inertia_rails` SSR is already enabled — no extra
+configuration needed.
+
+Outgoing HTTP made through Faraday is instrumented by adding
+`Railwatch::Faraday` to the connection's middleware stack (`Net::HTTP` is
+already covered globally, with no setup); any other client can be wrapped
+with `Railwatch.instrument_outgoing`:
+
+```ruby
+Faraday.new(url) { |f| f.use Railwatch::Faraday }
+Railwatch.instrument_outgoing(:get, url) { http_client.get(url) }
+```
+
+## Testing
+
+Production browser errors can be resolved to original source files using
+[private source-map uploads](docs/source-maps.md).
+
+The same instrumentation runs in your test suite, so a spec can hold a hot
+path to a query budget and CI can fail the pull request that regresses it:
+
+```ruby
+expect { get "/widgets" }.to have_railwatch_queries(at_most: 6)
+expect { get "/widgets" }.not_to have_railwatch_n_plus_one
+```
+
+Failures list the offending SQL. Set-up, every matcher (RSpec and Minitest),
+and a CI performance-gate recipe are in [`docs/testing.md`](docs/testing.md).
+
+---
+
+Every attribute, the full public facade, sampling, redaction/rejection,
+transport/buffering behavior, the overhead gate, and the Kamal deploy hook
+are documented field-by-field in [`docs/configuration.md`](docs/configuration.md).
+Every record type Railwatch ships — `request`, `job_attempt`, `query`,
+`exception`, and the rest — is documented field-by-field, sourced directly
+from the code that builds it, in [`docs/records.md`](docs/records.md).
+
+## Replacing Sentry
+
+Railwatch subscribes to `Rails.error` on install
+(`Rails.error.subscribe`), so any existing `Rails.error.report` or
+`Rails.error.handle` call — which is how Sentry's own Rails integration
+is normally wired in — is captured with no code changes. An unhandled
+exception bypasses the execution buffer: it is enqueued immediately and
+wakes the in-memory reporter without doing network I/O on the application
+thread. Delivery is still asynchronous and memory-only, so a hard kill,
+OOM, or process exit after the shutdown deadline can lose it.
+
+What differs from a dedicated error tracker: exceptions aren't reported in
+isolation — each one is linked (`execution_id`/`trace_id`) to the request,
+job, or command it happened inside, alongside every query, cache read,
+outgoing request, and log line from that same execution. There's no
+separate error-tracking SDK/config to maintain — `severity`, `handled`,
+and `context` all come from the same `Railwatch.configure` block and
+`Railwatch.context` calls used for everything else the gem instruments.
+
+### Coming from Sentry
+
+| Sentry | Railwatch |
+|---|---|
+| `dsn:` | `RAILWATCH_TOKEN` (+ `RAILWATCH_INGEST_URL` for a self-hosted platform). |
+| `environment:` | `config.environment` — defaults to `Rails.env`, set it only to report under a different name. |
+| `release:` | `config.deploy` — auto-detected from the deploy platform or Git checkout. Stamped on every record. |
+| `traces_sample_rate:` / `profiles_sample_rate:` | `config.sample`, a rate per execution kind (`requests`, `jobs`, `commands`, `scheduled_tasks`, `exceptions`), decided once per execution rather than per event. Per-route: `railwatch_sample 0.01, only: :index`. |
+| `excluded_exceptions:` | `config.ignored_exceptions` — same default list, plus every named ancestor is matched, not just the exact class. |
+| `before_send:` / `before_send_transaction:` | `Railwatch.before_ingest { \|batch\| ... }` for the whole outgoing batch; `Railwatch.redact_queries`/`redact_logs`/... to scrub one record type in place; `Railwatch.reject_queries`/`reject_logs`/... to drop records by predicate. |
+| `fingerprint` / grouping rules | `Railwatch.fingerprint { \|error, default\| ... }` globally, `def railwatch_fingerprint` on your own error class, or `Railwatch.report(error, fingerprint: [...])` per call. The literal `:default` splices in the parts Railwatch would have hashed, like Sentry's `{{ default }}`. |
+| `include_local_variables:` | `config.capture_exception_locals`. |
+| `config.rails.active_job_report_on_retry_error` | `config.capture_job_retry_errors` (off by default). |
+| `send_default_pii:` | Deliberately split: `config.capture_request_payload` for params, `config.capture_job_arguments` for job arguments, `config.capture_response_body_on_error` for what a failing upstream sent back, `config.redact_headers`/`redact_params` for what's scrubbed, and the `Railwatch.user { ... }` block for who. There is no single "send everything" switch. |
+| Breadcrumbs | Not a separate concept — every query, cache read, outgoing request, log line, and view render is already a first-class record linked to its execution by `execution_id`/`trace_id`. The execution *is* the breadcrumb trail, and it's queryable. |
+| `Sentry.capture_message` | `Railwatch.report(error, ...)` for an exception; plain `Rails.logger` for a message — log lines at or above `config.log_level` become `log` records automatically. |
+| `Sentry.set_user` | `Railwatch.user { ... }` (a resolver block, evaluated per execution). |
+| `Sentry.set_tags` / `set_context` / `set_extras` | `Railwatch.context(key: value)` — serialized onto the parent record and every exception. |
+| `Sentry.with_child_span` | `Railwatch.span("name") { ... }`. |
+| `Sentry.add_attachment` | `Railwatch.attach("payload.json", data)` — a String, `Pathname`, or IO, gzipped on the wire and capped at `config.max_attachment_bytes`. `exception:` files it against that error's issue, and `Railwatch.report(error, attachments: { "payload.json" => data })` captures and attaches in one call. |
+| `Sentry.capture_check_in` (cron monitoring) | Automatic: Solid Queue recurring tasks become `scheduled_task` records with `task_key`, `schedule`, and `drift`. Nothing to instrument. |
+| `config.rails.report_rescued_exceptions` | `config.capture_rescued_exceptions` (on by default). |
+| Rack `X-Request-Start` queue time | Automatic: `queue_time` on every `request` record. |
+| `auto_session_tracking:` (release health) | Automatic: `session` records from the browser client and the request middleware, keyed on `config.deploy` as the release. `config.track_sessions` turns both off. |
+
+These mappings cover the Rails-server migration path. Browser Replay,
+native/mobile SDKs, some direct worker and scheduler entry points, and
+Sentry's broader managed integration catalog are not equivalent today.
+Use the supported-workload matrix in
+[`docs/replacing-sentry.md`](docs/replacing-sentry.md) before removing
+Sentry from an application that depends on those capabilities.
+
+`config.deploy` checks `RAILWATCH_DEPLOY`, `KAMAL_VERSION`, common Git and
+platform environment variables, `REVISION`, then `.git/HEAD`. Full 40-character
+SHAs are consistently shortened to 12 characters. Set
+`RAILWATCH_DETECT_DEPLOY=false` (or `c.detect_deploy = false`) to keep only the
+explicit `RAILWATCH_DEPLOY` and Kamal defaults.
+
+To report an exception manually (the `Rails.error.report`-equivalent):
+
+```ruby
+Railwatch.report(error, handled: true, context: { order_id: order.id })
+```
+
+`severity` defaults to `:warning` when `handled: true`, `:error`
+otherwise. See the `exception` section of
+[`docs/records.md`](docs/records.md) for the full field list, and
+[`docs/configuration.md`](docs/configuration.md) for redaction
+(`Railwatch.redact_exceptions`), `capture_exception_source`, and
+`Railwatch.on_unrecoverable` (Railwatch watching its own internal failures).
+
+## Development
+
+```sh
+bundle install
+bundle exec rspec
+```

data/app/controllers/railwatch/beacon_controller.rb

@@ -0,0 +1,254 @@
+# frozen_string_literal: true
+
+module Railwatch
+  # Receives Inertia visit timings and JavaScript errors from the browser
+  # client (app/frontend/lib/railwatch.ts). Mounted at /railwatch/beacon.
+  class BeaconController < ActionController::API
+    # Core Web Vitals ceilings. Anything beyond these is a broken clock or a
+    # forged beacon, not a page load, so it is clamped rather than stored.
+    MAX_METRIC_MS = 120_000
+    MAX_CLS = 100.0
+
+    MAX_REQUEST_BYTES = 256 * 1024
+    MAX_VISITS = 50
+    MAX_ERRORS = 50
+    MAX_ERROR_MESSAGE = 1024
+    MAX_ERROR_STACK = 8192
+    MAX_BREADCRUMBS = 20
+    MAX_CRUMB_TEXT = 500
+    MAX_CONTEXT_KEYS = 20
+    MAX_CONTEXT_VALUE = 4096
+    CRUMB_KINDS = %w[console click navigate].freeze
+
+    RATE_LIMIT_WINDOW = 60 # seconds
+
+    before_action :limit_payload, :throttle, if: -> { Railwatch.config.beacon_enabled }
+
+    def create
+      return head :no_content unless Railwatch.config.beacon_enabled
+
+      visits = params[:visits].is_a?(Array) ? params[:visits].first(MAX_VISITS).filter_map { |visit| object(visit) } : []
+      user_id = Subscribers::Users.resolve_beacon_id(request)
+      tenant = beacon_tenant(params[:tenant])
+      record_session(params[:session], visits, user_id, tenant)
+      record_errors(params[:errors], user_id, tenant)
+      visits.each do |v|
+        Railwatch.record(:visit,
+          group: Record.group_hash(safe_string(v["component"], 255)),
+          timestamp: safe_float(v["started_at"]) / 1000.0,
+          component: safe_string(v["component"], 255),
+          url: safe_string(v["url"], 2048),
+          method: safe_string(v["method"], 10),
+          duration: (safe_float(v["duration_ms"]) * 1000).round,
+          status: safe_string(v["status"], 20),
+          partial: v["partial"] ? true : false,
+          only: array(v["only"]).first(50).map { |value| safe_string(value, 255) },
+          props_bytes: safe_integer(v["props_bytes"]),
+          lcp: metric_ms(v["lcp"]),
+          cls: cumulative_layout_shift(v["cls"]),
+          inp: metric_ms(v["inp"]),
+          ttfb: metric_ms(v["ttfb"]),
+          user: user_id,
+          tenant: tenant,
+          user_agent: safe_string(request.user_agent, 256))
+      end
+      head :no_content
+    rescue StandardError => error
+      # The endpoint is unauthenticated and its payload is untrusted. A bad
+      # shape must not turn into an exception in the customer application.
+      Railwatch.debug { "discarded invalid beacon payload: #{error.class}: #{error.message}" }
+      head :no_content
+    end
+
+    private
+
+    # The beacon takes no credential and keeps every browser error it is
+    # sent, so a client that is not the page -- a script, a bored visitor
+    # with curl -- could otherwise fill the app's quota with junk issues.
+    # Same shape as Rails' rate_limit (a counter per client IP in the app's
+    # cache store), read from config at request time so an initializer can
+    # raise or disable it. A store that cannot count (NullStore) fails open:
+    # the beacon keeps working, just unthrottled.
+    def throttle
+      limit = Railwatch.config.beacon_rate_limit.to_i
+      return unless limit.positive?
+
+      count = cache_store.increment("railwatch:beacon:#{request.remote_ip}", 1, expires_in: RATE_LIMIT_WINDOW)
+      return unless count && count > limit
+
+      response.set_header("Retry-After", RATE_LIMIT_WINDOW.to_s)
+      head :too_many_requests
+    end
+
+    def limit_payload
+      head :content_too_large if request.raw_post.bytesize > MAX_REQUEST_BYTES
+    end
+
+    # The browser half of release health: one `session` record per beacon
+    # flush, never more, whatever the flush carried. The first one the client
+    # sends has no duration yet and opens the session; every later flush
+    # beats it along; the pagehide flush closes it with `ended`.
+    def record_session(session, visits, user_id, tenant)
+      session = object(session)
+      return unless session
+
+      id = safe_string(session["id"], 64)
+      return if id.empty?
+
+      duration_ms = session["duration_ms"]
+      Railwatch.record(:session,
+        group: Record.group_hash(id),
+        id: id,
+        source: "browser",
+        status: duration_ms.nil? ? "started" : "ok",
+        started_at: safe_float(session["started_at"]) / 1000.0,
+        duration: duration_ms.nil? ? nil : (safe_float(duration_ms) * 1000).round,
+        visits: visits.size,
+        errors: visits.count { |v| v["status"] == "error" },
+        ended: session["ended"] ? true : false,
+        user: user_id,
+        tenant: tenant)
+    end
+
+    # One exception record per JavaScript error the page threw, filed the
+    # same way the Ruby side files one: the browser's stack parsed into
+    # frames, and a default fingerprint of class, top in-app frame, and the
+    # message with its variable data removed, so a browser issue groups and
+    # regresses like any other. Nothing here may raise on a payload the app
+    # did not write: an entry that is not a hash, or carries no error name,
+    # is dropped rather than answered with a 500.
+    def record_errors(errors, user_id, tenant)
+      errors = errors.is_a?(Array) ? errors.first(MAX_ERRORS) : []
+      return if errors.empty?
+
+      # A browser error is unhandled by definition, so it has to survive the
+      # beacon request's own head sampling decision the way an unhandled
+      # Ruby exception does.
+      Railwatch.keep!
+      origin = request.base_url
+      session_id = session_id_from(params[:session])
+      errors.each do |error|
+        error = object(error)
+        next unless error
+        name = safe_string(error["name"], 255)
+        next if name.empty?
+        record_error(error, name, origin, session_id, user_id, tenant)
+      end
+    end
+
+    # The tenant the browser says it was looking at. The beacon does not go
+    # through whatever path or subdomain the app scopes tenants by, so an
+    # app that scopes them at all resolves nothing here -- the client's hint
+    # (startRailwatch({ tenant })) fills that gap, and never overrides a
+    # tenant the server did resolve for itself.
+    def beacon_tenant(hint)
+      Context.current_tenant || (hint.is_a?(String) ? safe_string(hint, 255).presence : nil)
+    end
+
+    # The tab's session id, so a browser error can be lined up with the
+    # session it crashed. Shaped like every other read off this payload:
+    # anything but a hash carrying an id yields no id at all.
+    def session_id_from(session)
+      session = object(session)
+      session ? safe_string(session["id"], 64) : ""
+    end
+
+    def record_error(error, name, origin, session_id, user_id, tenant)
+      message = safe_string(error["message"], MAX_ERROR_MESSAGE)
+      frames = Backtrace.js_frames(safe_string(error["stack"], MAX_ERROR_STACK), origin: origin)
+      top = Subscribers::Exceptions.top_frame(frames)
+      parts = Subscribers::Exceptions.cap_fingerprint(
+        [ name, top[:file], top[:line], Subscribers::Exceptions.normalize_message(message) ])
+      Railwatch.record(:exception,
+        group: Record.group_hash(*parts),
+        timestamp: safe_float(error["at"]).positive? ? safe_float(error["at"]) / 1000.0 : nil,
+        class: name,
+        message: message,
+        handled: false,
+        severity: "error",
+        source: "browser",
+        file: top[:file],
+        line: top[:line],
+        frames: frames,
+        context: Context.serialized_with(error_context(error["context"]).merge(browser: {
+          url: safe_string(error["url"], 2048).presence,
+          component: safe_string(error["component"], 255).presence,
+          visit: safe_string(error["visit"], 2048).presence,
+          session: session_id.presence,
+          user_agent: safe_string(request.user_agent, 256).presence,
+          breadcrumbs: breadcrumbs(error["breadcrumbs"]).presence
+        }.compact)),
+        fingerprint: parts,
+        fingerprint_source: "default",
+        user: user_id,
+        tenant: tenant)
+    end
+
+    # What the user did in the run-up to the crash, kept only where each
+    # entry is the shape the client writes. A garbled or forged trail is
+    # dropped entry by entry rather than failing the beacon.
+    def breadcrumbs(raw)
+      return [] unless raw.is_a?(Array)
+      raw.first(MAX_BREADCRUMBS).filter_map do |crumb|
+        crumb = object(crumb)
+        next unless crumb
+        kind = safe_string(crumb["kind"], 20)
+        next unless CRUMB_KINDS.include?(kind)
+        text = safe_string(crumb["text"], MAX_CRUMB_TEXT)
+        next if text.empty?
+        { at: safe_float(crumb["at"]), kind: kind, text: text }
+      end
+    end
+
+    # Whatever the app handed Railwatch's reportError -- a React error
+    # boundary's component stack, most often. Flattened to strings so a
+    # cyclic or enormous object cannot ride in on it.
+    def error_context(raw)
+      raw = object(raw)
+      return {} unless raw
+      raw.first(MAX_CONTEXT_KEYS).to_h do |key, value|
+        [ safe_string(key, 64), safe_string(value, MAX_CONTEXT_VALUE) ]
+      end
+    end
+
+    # Web vitals only ride along on the initial-load visit, and only from
+    # browsers that support the entry type behind them, so every one of these
+    # is nil far more often than not.
+    def metric_ms(value)
+      return nil if value.nil? || value == ""
+      safe_float(value).round.clamp(0, MAX_METRIC_MS)
+    end
+
+    def cumulative_layout_shift(value)
+      return nil if value.nil? || value == ""
+      safe_float(value).clamp(0.0, MAX_CLS).round(4)
+    end
+
+    def object(value)
+      value = value.to_unsafe_h if value.respond_to?(:to_unsafe_h)
+      value if value.is_a?(Hash)
+    end
+
+    def array(value)
+      value.is_a?(Array) ? value : []
+    end
+
+    def safe_string(value, length)
+      value.to_s.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)[0, length]
+    rescue StandardError
+      ""
+    end
+
+    def safe_float(value)
+      Float(value, exception: false).to_f
+    rescue StandardError
+      0.0
+    end
+
+    def safe_integer(value)
+      Integer(value, exception: false).to_i
+    rescue StandardError
+      0
+    end
+  end
+end

data/config/routes.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+Railwatch::Engine.routes.draw do
+  post "beacon", to: "beacon#create"
+end