railwatch 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/AGENTS.md +122 -0
- data/CHANGELOG.md +462 -0
- data/MIT-LICENSE +20 -0
- data/README.md +226 -0
- data/app/controllers/railwatch/beacon_controller.rb +254 -0
- data/config/routes.rb +5 -0
- data/docs/ai-and-mcp.md +227 -0
- data/docs/configuration.md +931 -0
- data/docs/faq.md +230 -0
- data/docs/getting-started.md +279 -0
- data/docs/records.md +834 -0
- data/docs/replacing-nightwatch.md +216 -0
- data/docs/replacing-sentry.md +573 -0
- data/docs/security.md +94 -0
- data/docs/self-hosting.md +60 -0
- data/docs/source-maps.md +60 -0
- data/docs/testing.md +175 -0
- data/docs/troubleshooting.md +319 -0
- data/lib/generators/railwatch/install/install_generator.rb +280 -0
- data/lib/generators/railwatch/install/templates/initializer.rb +54 -0
- data/lib/generators/railwatch/install/templates/post-deploy +98 -0
- data/lib/generators/railwatch/install/templates/railwatch.ts +658 -0
- data/lib/railwatch/attachments.rb +83 -0
- data/lib/railwatch/backtrace.rb +158 -0
- data/lib/railwatch/buffer.rb +122 -0
- data/lib/railwatch/clock.rb +25 -0
- data/lib/railwatch/configuration.rb +334 -0
- data/lib/railwatch/console.rb +48 -0
- data/lib/railwatch/context.rb +125 -0
- data/lib/railwatch/controller_helpers.rb +21 -0
- data/lib/railwatch/current.rb +32 -0
- data/lib/railwatch/engine.rb +144 -0
- data/lib/railwatch/execution.rb +367 -0
- data/lib/railwatch/faraday.rb +73 -0
- data/lib/railwatch/health.rb +188 -0
- data/lib/railwatch/job_tracing.rb +49 -0
- data/lib/railwatch/middleware/request.rb +289 -0
- data/lib/railwatch/minitest.rb +43 -0
- data/lib/railwatch/patches/inertia.rb +34 -0
- data/lib/railwatch/patches/net_http.rb +102 -0
- data/lib/railwatch/patches/rake_task.rb +88 -0
- data/lib/railwatch/patches/runner_command.rb +120 -0
- data/lib/railwatch/patches.rb +43 -0
- data/lib/railwatch/profiler.rb +270 -0
- data/lib/railwatch/record.rb +119 -0
- data/lib/railwatch/redactor.rb +67 -0
- data/lib/railwatch/release_detector.rb +97 -0
- data/lib/railwatch/reporter.rb +539 -0
- data/lib/railwatch/rspec.rb +139 -0
- data/lib/railwatch/sampler.rb +17 -0
- data/lib/railwatch/secret_safety.rb +62 -0
- data/lib/railwatch/sessions.rb +162 -0
- data/lib/railwatch/source_maps.rb +59 -0
- data/lib/railwatch/spec_helper.rb +147 -0
- data/lib/railwatch/sql_normalizer.rb +398 -0
- data/lib/railwatch/subscribers/base.rb +54 -0
- data/lib/railwatch/subscribers/broadcasts.rb +107 -0
- data/lib/railwatch/subscribers/cache.rb +107 -0
- data/lib/railwatch/subscribers/deprecations.rb +26 -0
- data/lib/railwatch/subscribers/exceptions.rb +304 -0
- data/lib/railwatch/subscribers/jobs.rb +282 -0
- data/lib/railwatch/subscribers/logs.rb +137 -0
- data/lib/railwatch/subscribers/mail.rb +42 -0
- data/lib/railwatch/subscribers/notifications.rb +36 -0
- data/lib/railwatch/subscribers/process_info.rb +98 -0
- data/lib/railwatch/subscribers/queries.rb +183 -0
- data/lib/railwatch/subscribers/requests.rb +94 -0
- data/lib/railwatch/subscribers/storage.rb +35 -0
- data/lib/railwatch/subscribers/users.rb +159 -0
- data/lib/railwatch/subscribers/views.rb +54 -0
- data/lib/railwatch/subscribers.rb +34 -0
- data/lib/railwatch/transport/http.rb +208 -0
- data/lib/railwatch/version.rb +5 -0
- data/lib/railwatch.rb +550 -0
- data/lib/tasks/railwatch_tasks.rake +289 -0
- data/llms.txt +38 -0
- metadata +157 -0
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Self-hosting
|
|
2
|
+
|
|
3
|
+
Railwatch Cloud is a Rails app you can run yourself. The gem doesn't care
|
|
4
|
+
which install it talks to — point it at yours and everything works the
|
|
5
|
+
same.
|
|
6
|
+
|
|
7
|
+
## Point the gem at your platform
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
# config/initializers/railwatch.rb
|
|
11
|
+
Railwatch.configure do |c|
|
|
12
|
+
c.ingest_url = "https://telemetry.example.com" # RAILWATCH_INGEST_URL
|
|
13
|
+
c.token = ENV["RAILWATCH_TOKEN"]
|
|
14
|
+
end
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
`ingest_url` defaults to `https://railwatch.rebulk.com`, so this is the one
|
|
18
|
+
setting a self-hosted install always needs; everything the gem sends —
|
|
19
|
+
records, ping, deploys — hangs off that host. Pass `--url=` to the
|
|
20
|
+
installer to have it written for you:
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
bin/rails generate railwatch:install --url=https://telemetry.example.com
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Getting a token
|
|
27
|
+
|
|
28
|
+
On your install: sign up, create an application, then create an
|
|
29
|
+
environment inside it (`production`, `staging` — one token each). The
|
|
30
|
+
token is shown once, right after you create the environment; a lost one
|
|
31
|
+
is rotated from the environment's settings, not recovered.
|
|
32
|
+
|
|
33
|
+
```sh
|
|
34
|
+
bin/rails railwatch:token # prints the URL to create/copy a token
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Check the connection
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
bin/rails railwatch:status # pings {ingest_url}/ingest/ping with your token
|
|
41
|
+
bin/rails railwatch:doctor # the full checklist: token, URL, reachability, wiring
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
`railwatch:status` aborts if the token is unset or the host is unreachable,
|
|
45
|
+
so it works as a post-deploy smoke test.
|
|
46
|
+
[`troubleshooting.md`](troubleshooting.md) covers each doctor line.
|
|
47
|
+
|
|
48
|
+
## MCP
|
|
49
|
+
|
|
50
|
+
Your install serves its own MCP endpoint at `<ingest host>/mcp` —
|
|
51
|
+
`https://telemetry.example.com/mcp` for the example above. It authenticates
|
|
52
|
+
with a per-user API token generated from Settings → Profile, the same
|
|
53
|
+
token the public JSON API at `/api/v1` uses. See
|
|
54
|
+
[`ai-and-mcp.md`](ai-and-mcp.md) for every client's configuration block.
|
|
55
|
+
|
|
56
|
+
## Running the platform
|
|
57
|
+
|
|
58
|
+
Railwatch Cloud's backend is not part of this gem repository. Licensed
|
|
59
|
+
self-hosted customers receive separate platform deployment, backup, retention,
|
|
60
|
+
quota, and restore documentation from Rebulk.
|
data/docs/source-maps.md
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Production browser source maps
|
|
2
|
+
|
|
3
|
+
Railwatch can resolve minified browser errors to their original source files,
|
|
4
|
+
lines, function names and embedded code. Maps stay in the environment's
|
|
5
|
+
private telemetry database; Railwatch never downloads a `sourceMappingURL`.
|
|
6
|
+
|
|
7
|
+
Enable hidden source maps in Vite:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
export default defineConfig({
|
|
11
|
+
// Keep your existing plugins and options.
|
|
12
|
+
build: { sourcemap: "hidden" },
|
|
13
|
+
})
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
After building, upload the maps **before publishing the image or assets**:
|
|
17
|
+
|
|
18
|
+
```sh
|
|
19
|
+
RAILWATCH_DEPLOY="$RELEASE_SHA" bin/rails 'railwatch:sourcemaps[public,true]'
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
The task uses the application's `RAILWATCH_TOKEN` and `RAILWATCH_INGEST_URL`.
|
|
23
|
+
Use the same deploy value the running application reports. The first argument
|
|
24
|
+
is the public URL root: `public/vite/assets/index-abc.js.map` becomes
|
|
25
|
+
`vite/assets/index-abc.js`, matching `/vite/assets/index-abc.js` in a browser
|
|
26
|
+
stack. The optional second argument `true` deletes each map only after the
|
|
27
|
+
server acknowledges its upload. Without it, files are retained. A failed
|
|
28
|
+
upload fails the task and leaves that file on disk. Hidden maps still exist
|
|
29
|
+
on disk, so the deletion step belongs before publishing public assets.
|
|
30
|
+
|
|
31
|
+
The equivalent environment options are `RAILWATCH_SOURCEMAPS_DIR=public` and
|
|
32
|
+
`RAILWATCH_SOURCEMAPS_DELETE=true`. The generated Kamal post-deploy hook also
|
|
33
|
+
accepts `--sourcemaps` or `RAILWATCH_SOURCEMAPS=true`, using local artifacts on
|
|
34
|
+
the deployer and `KAMAL_VERSION` as the deploy. That hook reports failures
|
|
35
|
+
without failing the deployment. Build-time upload is preferable because it
|
|
36
|
+
removes maps before assets become public.
|
|
37
|
+
|
|
38
|
+
Uploads accept flat Source Map v3 files (Vite's output), up to 10 MiB each,
|
|
39
|
+
500,000 mapping segments and 200,000 generated lines. Indexed `sections`
|
|
40
|
+
maps and remote references are unsupported. Embedded `sourcesContent`
|
|
41
|
+
provides snippets; it is optional. No local source files are opened by the
|
|
42
|
+
server. Maps for the current deploy and releases referenced by retained
|
|
43
|
+
requests, jobs, exceptions or sessions stay available. Maps for inactive
|
|
44
|
+
releases expire once their last update is older than the environment's raw
|
|
45
|
+
telemetry retention; replacing a map renews that window.
|
|
46
|
+
|
|
47
|
+
New default browser issue fingerprints use the resolved original location
|
|
48
|
+
when a map is already available. Custom fingerprints are preserved. Late
|
|
49
|
+
uploads improve existing stack displays, Copy for AI, API issue details and
|
|
50
|
+
MCP `get_issue`, while existing occurrences keep their original grouping.
|
|
51
|
+
Missing maps or columns fall back to the raw stack. Upgrade the gem for
|
|
52
|
+
new browser occurrences to include columns; older gem versions discarded
|
|
53
|
+
them and cannot resolve minified locations accurately.
|
|
54
|
+
|
|
55
|
+
For a custom release uploader, POST the raw `.map` bytes to
|
|
56
|
+
`/ingest/sourcemaps` with `Content-Type: application/octet-stream`,
|
|
57
|
+
`Authorization: Bearer lt_...`, `X-Railwatch-Deploy`, and
|
|
58
|
+
`X-Railwatch-Filename` (the generated JavaScript URL path without a leading
|
|
59
|
+
slash). A successful response is HTTP 201 with
|
|
60
|
+
`{"ok":true,"filename":"vite/assets/index-abc.js","bytes":1234}`.
|
data/docs/testing.md
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# Testing with Railwatch
|
|
2
|
+
|
|
3
|
+
Railwatch already watches every query, N+1, span, exception, and outgoing
|
|
4
|
+
request your app makes. The same instrumentation works in your test suite,
|
|
5
|
+
which means a spec can assert on them — and CI can fail a pull request that
|
|
6
|
+
adds an N+1 or doubles a page's query count.
|
|
7
|
+
|
|
8
|
+
## Set-up
|
|
9
|
+
|
|
10
|
+
RSpec — add one line to `spec/rails_helper.rb` (the install generator adds it
|
|
11
|
+
for you):
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
require "rspec/rails"
|
|
15
|
+
require "railwatch/rspec"
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
That requires `railwatch/spec_helper`, includes `Railwatch::SpecHelper` into every
|
|
19
|
+
example group, and defines the matchers below.
|
|
20
|
+
|
|
21
|
+
Minitest — the same thing in `test/test_helper.rb`:
|
|
22
|
+
|
|
23
|
+
```ruby
|
|
24
|
+
require "rails/test_help"
|
|
25
|
+
require "railwatch/minitest"
|
|
26
|
+
|
|
27
|
+
class ActiveSupport::TestCase
|
|
28
|
+
include Railwatch::Minitest
|
|
29
|
+
end
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Railwatch must be *enabled* in the test environment or every block would look
|
|
33
|
+
empty. `config.enabled?` is true when `config.enabled` is set and a token is
|
|
34
|
+
present, so set any non-blank `RAILWATCH_TOKEN` for the test env — records go to
|
|
35
|
+
an in-memory transport, never over the network. If Railwatch is disabled, the
|
|
36
|
+
matchers raise `Railwatch::SpecHelper::Disabled` rather than quietly passing.
|
|
37
|
+
|
|
38
|
+
Sampling is forced on for the block, so a fractional `c.sample` in the app's
|
|
39
|
+
test config can't turn an assertion into one that never fires either.
|
|
40
|
+
|
|
41
|
+
## Matchers
|
|
42
|
+
|
|
43
|
+
All of them are block matchers.
|
|
44
|
+
|
|
45
|
+
### `have_railwatch_queries`
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
expect { OrderSummary.new(order).to_h }.to have_railwatch_queries(at_most: 5)
|
|
49
|
+
expect { user.reload }.to have_railwatch_queries(exactly: 1)
|
|
50
|
+
expect { Report.generate }.to have_railwatch_queries(at_least: 1)
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Exactly one of `at_most:`, `exactly:`, `at_least:` — passing two (or none)
|
|
54
|
+
raises `ArgumentError`. On failure the message lists every statement, each
|
|
55
|
+
truncated to 120 characters, so CI output says what to go and fix:
|
|
56
|
+
|
|
57
|
+
```
|
|
58
|
+
expected the block to run at most 1 database queries, but it ran 3:
|
|
59
|
+
1. SELECT COUNT(*) FROM "widgets"
|
|
60
|
+
2. SELECT "widgets".* FROM "widgets" WHERE "widgets"."id" = ?
|
|
61
|
+
3. SELECT "gadgets".* FROM "gadgets" WHERE "gadgets"."id" = ?
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Cached queries don't count — they never become `query` records.
|
|
65
|
+
|
|
66
|
+
### `have_railwatch_n_plus_one`
|
|
67
|
+
|
|
68
|
+
```ruby
|
|
69
|
+
expect { get "/widgets" }.not_to have_railwatch_n_plus_one
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Matches when the block trips Railwatch's own N+1 detector: the same normalized
|
|
73
|
+
query shape repeated `config.n_plus_one_threshold` times (default 5) inside one
|
|
74
|
+
execution. The negated failure message names the shape, the repeat count, and
|
|
75
|
+
the app-code line that issued it.
|
|
76
|
+
|
|
77
|
+
### `record_railwatch_span`
|
|
78
|
+
|
|
79
|
+
```ruby
|
|
80
|
+
expect { Checkout.new(cart).total }.to record_railwatch_span("checkout.total")
|
|
81
|
+
expect { Checkout.new(cart).total }.to record_railwatch_span(nil) # any span
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### `record_railwatch_exception` / `record_railwatch_exceptions`
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
expect { importer.run }.to record_railwatch_exception(ArgumentError)
|
|
88
|
+
expect { importer.run }.not_to record_railwatch_exceptions
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
These see anything that reaches `Rails.error` — `Rails.error.handle`,
|
|
92
|
+
`Rails.error.report`, `Railwatch.report`, and unhandled exceptions a request
|
|
93
|
+
spec's middleware catches. A block that raises out of the matcher still
|
|
94
|
+
raises; nothing is swallowed.
|
|
95
|
+
|
|
96
|
+
### `have_railwatch_outgoing_requests`
|
|
97
|
+
|
|
98
|
+
```ruby
|
|
99
|
+
expect { SyncCustomers.run }.to have_railwatch_outgoing_requests(at_most: 1)
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Same bounds as `have_railwatch_queries`. Failures list the method and URL of
|
|
103
|
+
every request the block made.
|
|
104
|
+
|
|
105
|
+
## Minitest assertions
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
assert_railwatch_queries(at_most: 5) { OrderSummary.new(order).to_h }
|
|
109
|
+
refute_railwatch_n_plus_one { get widgets_url }
|
|
110
|
+
assert_railwatch_span("checkout.total") { Checkout.new(cart).total }
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
`assert_railwatch_queries` takes `exactly:`/`at_most:`/`at_least:` too, and
|
|
114
|
+
produces the same statement listing on failure.
|
|
115
|
+
|
|
116
|
+
## Where the matchers work
|
|
117
|
+
|
|
118
|
+
Anywhere. A request spec's `get "/widgets"` opens and closes its own
|
|
119
|
+
execution, so its whole tree — queries, N+1s, outgoing HTTP — is visible by
|
|
120
|
+
the time the block returns:
|
|
121
|
+
|
|
122
|
+
```ruby
|
|
123
|
+
expect { get "/widgets" }.to have_railwatch_queries(at_most: 6)
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
A model or service spec has nothing executing, so the block is wrapped in an
|
|
127
|
+
execution for the duration of the assertion and closed afterwards. No parent
|
|
128
|
+
`command` record is written for it. A block running *inside* an execution you
|
|
129
|
+
opened yourself has its records read straight off that execution's buffer.
|
|
130
|
+
|
|
131
|
+
Under the hood every matcher calls `Railwatch::SpecHelper#railwatch_capture`,
|
|
132
|
+
which is public — use it directly for anything the matchers don't cover:
|
|
133
|
+
|
|
134
|
+
```ruby
|
|
135
|
+
records = railwatch_capture { get "/widgets" }
|
|
136
|
+
expect(records.select { |r| r[:t] == "cache_event" }.size).to eq(2)
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`railwatch_records(type = nil)` is still there for assertions about the whole
|
|
140
|
+
example rather than one block.
|
|
141
|
+
|
|
142
|
+
## CI performance gate
|
|
143
|
+
|
|
144
|
+
Put the budget for a hot path in a spec and let it fail the build when
|
|
145
|
+
someone regresses it. The point is that the number is checked in, so raising
|
|
146
|
+
it is a reviewed decision rather than an accident:
|
|
147
|
+
|
|
148
|
+
```ruby
|
|
149
|
+
# spec/performance/widgets_spec.rb
|
|
150
|
+
RSpec.describe "performance budgets", type: :request do
|
|
151
|
+
before { create_list(:widget, 25) }
|
|
152
|
+
|
|
153
|
+
it "renders the widget index within its query budget" do
|
|
154
|
+
expect { get "/widgets" }.to have_railwatch_queries(at_most: 6)
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
it "renders the widget index without an N+1" do
|
|
158
|
+
expect { get "/widgets" }.not_to have_railwatch_n_plus_one
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
it "renders the widget index without calling out to anyone" do
|
|
162
|
+
expect { get "/widgets" }.to have_railwatch_outgoing_requests(exactly: 0)
|
|
163
|
+
end
|
|
164
|
+
end
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Two ways to run it: tag these examples and run them as their own CI step
|
|
168
|
+
(`bundle exec rspec --tag performance`) so a budget failure is obvious in the
|
|
169
|
+
job list, or leave them in the main suite so any pull request that adds a
|
|
170
|
+
query fails immediately. Either way the failure message names the statements,
|
|
171
|
+
so the fix is usually an `includes` one line away.
|
|
172
|
+
|
|
173
|
+
Seed enough rows in `before` that an N+1 actually crosses
|
|
174
|
+
`config.n_plus_one_threshold` — with three records, a five-query threshold
|
|
175
|
+
never fires and the gate passes on code that would melt in production.
|
|
@@ -0,0 +1,319 @@
|
|
|
1
|
+
# Troubleshooting
|
|
2
|
+
|
|
3
|
+
Start with `bin/rails railwatch:doctor`. It checks every piece of the
|
|
4
|
+
install in one pass and prints a `✓`/`✗` line per piece; the sections
|
|
5
|
+
below are keyed to those lines.
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
bin/rails railwatch:doctor
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The task exits non-zero only when **token** or **ingest reachable**
|
|
12
|
+
fails. Everything else is informational — a `✗` there means a feature
|
|
13
|
+
isn't wired, not that the install is broken.
|
|
14
|
+
|
|
15
|
+
| Doctor line | What a `✗` means |
|
|
16
|
+
|---|---|
|
|
17
|
+
| `token` | `RAILWATCH_TOKEN` is unset or empty. Fatal: nothing is recorded at all. |
|
|
18
|
+
| `ingest url` | `ingest_url` isn't a parseable HTTP(S) URL. |
|
|
19
|
+
| `ingest reachable` | `GET {ingest_url}/ingest/ping` didn't return success. Fatal. |
|
|
20
|
+
| `request middleware` | `Railwatch::Middleware::Request` isn't in the stack, so requests aren't executions. |
|
|
21
|
+
| `engine mounted` | `mount Railwatch::Engine, at: "/railwatch"` is missing from `config/routes.rb`; the browser beacon has nowhere to post. |
|
|
22
|
+
| `deploy` | `config.deploy` is unset — records ship, charts get no deploy markers. |
|
|
23
|
+
| `sample rates` | Never fails; it prints the effective rate per execution kind. |
|
|
24
|
+
| `ignored record types` | Never fails; it prints what `c.ignore` is dropping. |
|
|
25
|
+
| `kamal post-deploy hook` | `.kamal/hooks/post-deploy` is missing or doesn't mention Railwatch. Only matters if you deploy with Kamal. |
|
|
26
|
+
| `browser client` | `app/frontend/lib/railwatch.ts` isn't there. Only matters for Inertia visit timing. |
|
|
27
|
+
| `browser client imported` | The client exists but nothing calls `startRailwatch()` — no `startRailwatch` found in `app/frontend/entrypoints`. Visits won't report. |
|
|
28
|
+
| `profiler backend` | Neither `vernier` nor `stackprof` is installed, so `Railwatch::Profiler.available?` is false and the profiling settings are inert. |
|
|
29
|
+
| `test matchers` | Neither `spec/rails_helper.rb` requires `railwatch/rspec` nor `test/test_helper.rb` requires `railwatch/minitest`. |
|
|
30
|
+
|
|
31
|
+
## No records at all
|
|
32
|
+
|
|
33
|
+
**Symptom.** The environment's pages stay empty however much traffic the
|
|
34
|
+
app takes.
|
|
35
|
+
|
|
36
|
+
Work down this list; the first five are the same root cause seen from
|
|
37
|
+
different angles — Railwatch decided not to record.
|
|
38
|
+
|
|
39
|
+
**The token is missing or blank.** `Railwatch.enabled?` is
|
|
40
|
+
`config.enabled && token.present?`. With no token the engine's
|
|
41
|
+
`railwatch.subscribe` initializer returns early, so no subscribers and no
|
|
42
|
+
patches are installed at all — this is by design, so the gem is inert in
|
|
43
|
+
development. Fix: set `RAILWATCH_TOKEN`, restart, re-run `railwatch:doctor`
|
|
44
|
+
(the `token` line prints the first 6 characters and the length, which is
|
|
45
|
+
enough to spot a truncated or quoted value).
|
|
46
|
+
|
|
47
|
+
**The token is wrong.** A 401 from the ingest marks the transport
|
|
48
|
+
permanently unauthorized: no further flush is attempted for the lifetime
|
|
49
|
+
of that process. Fixing the env var isn't enough — restart the process.
|
|
50
|
+
`railwatch:doctor`'s `ingest reachable` line catches this before you
|
|
51
|
+
deploy.
|
|
52
|
+
|
|
53
|
+
**`RAILWATCH_INGEST_URL` points somewhere else.** Records go where you sent
|
|
54
|
+
them. `railwatch:status` prints the URL it is actually using; compare it
|
|
55
|
+
against the platform you're looking at. Self-hosting: see
|
|
56
|
+
[`self-hosting.md`](self-hosting.md).
|
|
57
|
+
|
|
58
|
+
**`config.enabled` is false.** `RAILWATCH_ENABLED=0` (or `false`/`no`/`off`)
|
|
59
|
+
turns everything off with a valid token present.
|
|
60
|
+
|
|
61
|
+
**Sample rates are at zero.** `c.sample = { requests: 0.0 }` — or the
|
|
62
|
+
per-route `railwatch_never_sample` macro on a controller — means no
|
|
63
|
+
request records. The `sample rates` doctor line prints the effective
|
|
64
|
+
values. Note that an unhandled exception still ships from a sampled-out
|
|
65
|
+
execution, so "exceptions arrive but nothing else does" is the
|
|
66
|
+
signature of a low sample rate rather than a broken install.
|
|
67
|
+
|
|
68
|
+
**The record type is ignored.** `c.ignore` drops a type before it is
|
|
69
|
+
built. The `ignored record types` doctor line prints the list. Ignoring
|
|
70
|
+
`:queries` also drops `n_plus_one`, since both key off `:queries`.
|
|
71
|
+
|
|
72
|
+
**You're looking at the test environment.** Requiring `railwatch/rspec`
|
|
73
|
+
(or `railwatch/minitest`) swaps the reporter's transport for an in-memory
|
|
74
|
+
one, so a suite records normally but never sends anything over the
|
|
75
|
+
network. Independently: the health sampler, the session flusher, and the
|
|
76
|
+
profiler all refuse to start when `Rails.env.test?`.
|
|
77
|
+
|
|
78
|
+
Still nothing? Set `RAILWATCH_DEBUG=1` and restart. Internal diagnostics go
|
|
79
|
+
to stderr prefixed `[railwatch]` (never to `Rails.logger`, so they can't
|
|
80
|
+
become `log` records about themselves). `Railwatch.on_unrecoverable { |e|
|
|
81
|
+
... }` gets the same failures as a callback.
|
|
82
|
+
|
|
83
|
+
## Doubled scheduled_task records
|
|
84
|
+
|
|
85
|
+
**Symptom.** Every recurring task shows twice on the Scheduled tasks
|
|
86
|
+
page, at the same minute, usually with different `drift`.
|
|
87
|
+
|
|
88
|
+
**Cause.** Two Solid Queue supervisors are running against the same
|
|
89
|
+
queue database — a stale `bin/jobs` left over from a previous `bin/dev`,
|
|
90
|
+
`SOLID_QUEUE_IN_PUMA=true` while a dedicated job role is also booted, or
|
|
91
|
+
two containers of the job role. Each supervisor has its own recurring
|
|
92
|
+
scheduler and its own workers, so the job really is performed twice.
|
|
93
|
+
Railwatch doesn't dedupe: it records one `scheduled_task` per
|
|
94
|
+
`perform.active_job`, in whichever process performed it. The doubled
|
|
95
|
+
rows are a true report of a doubled run.
|
|
96
|
+
|
|
97
|
+
**Fix.** Run one supervisor. `SolidQueue::Process.where(kind:
|
|
98
|
+
"Supervisor")` tells you how many think they're alive; `bin/kamal app
|
|
99
|
+
logs -r job` tells you which containers are booting one. The same
|
|
100
|
+
duplication also doubles the work itself, so this is worth fixing
|
|
101
|
+
regardless of what the dashboard says.
|
|
102
|
+
|
|
103
|
+
## Outgoing requests missing in specs
|
|
104
|
+
|
|
105
|
+
**Symptom.** `have_railwatch_outgoing_requests` never sees anything, and no
|
|
106
|
+
`outgoing_request` records appear from the test suite. Production is
|
|
107
|
+
fine.
|
|
108
|
+
|
|
109
|
+
**Cause.** WebMock replaces `::Net::HTTP` with a subclass whose
|
|
110
|
+
`#request` short-circuits before calling `super`, so Railwatch's prepend on
|
|
111
|
+
the real class never runs.
|
|
112
|
+
|
|
113
|
+
**Fix.** Re-prepend the patch onto the replacement, once, before the
|
|
114
|
+
suite. This is exactly what the gem's own suite does
|
|
115
|
+
(`spec/spec_helper.rb`), and app suites using WebMock should do the
|
|
116
|
+
same:
|
|
117
|
+
|
|
118
|
+
```ruby
|
|
119
|
+
# WebMock replaces ::Net::HTTP with a subclass whose #request short-circuits
|
|
120
|
+
# before calling super, so Railwatch's prepend on the real class never runs
|
|
121
|
+
# under WebMock. Re-prepend on the replacement so outgoing requests are still
|
|
122
|
+
# observed in this suite. Apps using WebMock in their own tests would do the same.
|
|
123
|
+
RSpec.configure do |config|
|
|
124
|
+
config.before(:suite) { Net::HTTP.prepend(Railwatch::Patches::NetHttp) }
|
|
125
|
+
end
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## Puma cluster mode: threads after fork
|
|
129
|
+
|
|
130
|
+
**Symptom.** You expect to have to re-arm something in
|
|
131
|
+
`on_worker_boot`, or the Processes page shows fewer processes than you
|
|
132
|
+
have workers.
|
|
133
|
+
|
|
134
|
+
**What actually happens.** Ruby routes `fork`, `Process.fork`, and
|
|
135
|
+
`Kernel#fork` through `Process._fork`, and Railwatch registers one callback
|
|
136
|
+
with Rails' own `ActiveSupport::ForkTracker` (the same hook Active Record
|
|
137
|
+
uses to reset its connection pools). Before the child returns from
|
|
138
|
+
`fork`, it replaces the inherited reporter buffer, drop accounting,
|
|
139
|
+
transport policy state, mutexes, condition variables, dead threads, and the
|
|
140
|
+
profiler's process-global state (a parent's in-flight profile would
|
|
141
|
+
otherwise leave the child permanently unable to profile).
|
|
142
|
+
The parent's half-finished session map is discarded too. The child then
|
|
143
|
+
emits its own `process` record and starts fresh health/session threads for
|
|
144
|
+
its role. Parent records remain owned by and delivered from the parent;
|
|
145
|
+
they can never be replayed by every child. **No `on_worker_boot`
|
|
146
|
+
configuration is needed**, in Puma cluster mode or in Solid Queue's
|
|
147
|
+
forked workers.
|
|
148
|
+
|
|
149
|
+
The synchronization objects are replaced without locking them. That is
|
|
150
|
+
deliberate: if another parent thread owned a mutex at the instant of
|
|
151
|
+
`fork`, Ruby preserves the locked mutex in the child but not the thread
|
|
152
|
+
that could unlock it.
|
|
153
|
+
|
|
154
|
+
**When a process legitimately reports nothing.** `Health.start!` returns
|
|
155
|
+
early unless the process's role is `web` or `worker`, and
|
|
156
|
+
`Sessions.start!` only runs for `web`. Role detection is
|
|
157
|
+
`Railwatch::Subscribers::ProcessInfo.role`, in this order: `worker` when
|
|
158
|
+
Solid Queue is loaded and `$PROGRAM_NAME` includes `"jobs"` (or the
|
|
159
|
+
command starts with `solid_queue:`), then `console`, then `command` when
|
|
160
|
+
`$PROGRAM_NAME` ends in `rake`, then `web` when Puma is defined, else
|
|
161
|
+
`process`. The worker check comes first deliberately — Puma is loaded in
|
|
162
|
+
a job container too. A console or a rake task ships no health records by
|
|
163
|
+
design, and both modules also return early in the `test` env.
|
|
164
|
+
|
|
165
|
+
## Memory growth with tail sampling on
|
|
166
|
+
|
|
167
|
+
**Symptom.** RSS climbs after enabling `c.tail_sample_slow_ms`, or the
|
|
168
|
+
`peak_memory` on parent records rises across the board.
|
|
169
|
+
|
|
170
|
+
**Cause.** That is the trade-off, not a leak. With head sampling only, a
|
|
171
|
+
sampled-out execution builds and buffers nothing. With tail sampling on,
|
|
172
|
+
*every* execution buffers its child records — queries, cache events,
|
|
173
|
+
logs, view renders — for its whole lifetime, because the keep-or-discard
|
|
174
|
+
decision can't be made until it ends.
|
|
175
|
+
|
|
176
|
+
**What to check.**
|
|
177
|
+
|
|
178
|
+
- Per execution, the buffer is capped at `Execution::MAX_RECORDS`
|
|
179
|
+
(10,000). Past that, records are dropped and counted, and the count is
|
|
180
|
+
added to the reporter's drop counter so the loss is visible on the
|
|
181
|
+
platform rather than silent.
|
|
182
|
+
- `c.buffer_size` (default 10,000, the same as `MAX_RECORDS`) caps the
|
|
183
|
+
process-wide queue between the app and the reporter thread.
|
|
184
|
+
Oldest-dropped-first, also counted. Do not set it below `MAX_RECORDS`:
|
|
185
|
+
an execution's tree is written to the queue in one go when it ends, so
|
|
186
|
+
a tree larger than the queue loses its own first records -- typically
|
|
187
|
+
the outgoing requests a long job made before it started writing.
|
|
188
|
+
Keeping far more executions than before means far more records
|
|
189
|
+
arriving at this queue; raise it, or lower what you keep.
|
|
190
|
+
- `c.profile_slow_ms` compounds it: it profiles every tail-buffering
|
|
191
|
+
execution from its first line and throws away the fast ones, so the
|
|
192
|
+
profiler's stack table is held alongside the record buffer.
|
|
193
|
+
- `c.failure_context` buffers sampled-out executions too, but a ring of
|
|
194
|
+
that many records each rather than all of them. If RSS climbed after
|
|
195
|
+
setting it, lower the count: it is a per-execution bound, so the
|
|
196
|
+
process-wide cost is that many records times the executions running
|
|
197
|
+
concurrently.
|
|
198
|
+
|
|
199
|
+
**Fix.** Lower `tail_sample_slow_ms` so fewer executions qualify to be
|
|
200
|
+
buffered, drop the highest-volume child types for tail-kept traffic with
|
|
201
|
+
`c.ignore`, or use `Railwatch.keep!` on the specific paths you care about
|
|
202
|
+
instead of a global threshold.
|
|
203
|
+
|
|
204
|
+
## Profiles never appear
|
|
205
|
+
|
|
206
|
+
**Symptom.** `c.profile_sample` is set but no `profile` records ship and
|
|
207
|
+
no request is marked `profiled`.
|
|
208
|
+
|
|
209
|
+
**Cause.** No profiler backend is installed. Railwatch doesn't vendor one:
|
|
210
|
+
`Railwatch::Profiler.available?` is false unless `vernier` or `stackprof`
|
|
211
|
+
loads, and every profiling setting is inert while it is. The doctor's
|
|
212
|
+
`profiler backend` line reports this.
|
|
213
|
+
|
|
214
|
+
**Fix.** Add `gem "vernier"` (Ruby ≥ 3.2, preferred) or
|
|
215
|
+
`gem "stackprof"`. Two other reasons a profile can be absent even with a
|
|
216
|
+
backend: `c.profiler` pinned to a name that doesn't load (profiling
|
|
217
|
+
stays off rather than falling back), and the `test` env, where profiling
|
|
218
|
+
is skipped unless `profile_sample` is explicitly non-zero. Both backends
|
|
219
|
+
are process-global, so an execution that starts while another one is
|
|
220
|
+
being profiled is simply not profiled — expected, not a bug.
|
|
221
|
+
|
|
222
|
+
## No deploy marker on the charts
|
|
223
|
+
|
|
224
|
+
**Symptom.** Charts have no vertical deploy lines; the Releases page
|
|
225
|
+
groups everything under one blank release.
|
|
226
|
+
|
|
227
|
+
**Cause.** `config.deploy` is unset. The doctor's `deploy` line says so,
|
|
228
|
+
and when it is set, names the environment variable, `REVISION`, Git checkout,
|
|
229
|
+
or initializer it came from.
|
|
230
|
+
|
|
231
|
+
**Fix.** Set one of them; they are read in this order:
|
|
232
|
+
|
|
233
|
+
1. `RAILWATCH_DEPLOY` — the explicit override on any platform.
|
|
234
|
+
2. `KAMAL_VERSION`.
|
|
235
|
+
3. `GIT_REV`, `GIT_SHA`, `SOURCE_VERSION`, `HEROKU_SLUG_COMMIT`,
|
|
236
|
+
`RENDER_GIT_COMMIT`, the tag from `FLY_IMAGE_REF`,
|
|
237
|
+
`VERCEL_GIT_COMMIT_SHA`, `CI_COMMIT_SHA`, or `GITHUB_SHA`.
|
|
238
|
+
4. A Capistrano `REVISION` file.
|
|
239
|
+
5. `.git/HEAD`, resolved through a loose ref or `packed-refs` without a Git
|
|
240
|
+
subprocess.
|
|
241
|
+
|
|
242
|
+
Full 40-character SHAs are shortened to 12 characters. Or assign `deploy` in
|
|
243
|
+
the initializer. Set `detect_deploy`/`RAILWATCH_DETECT_DEPLOY` to false to ignore
|
|
244
|
+
steps 3–5. The value is stamped on every record, so a change only affects
|
|
245
|
+
records shipped after the restart. Note that
|
|
246
|
+
`config.deploy` and the deploy *marker* are two different things: the
|
|
247
|
+
marker (with its commit list) comes from `railwatch:deploy` or the Kamal
|
|
248
|
+
hook below.
|
|
249
|
+
|
|
250
|
+
## The Kamal hook doesn't fire
|
|
251
|
+
|
|
252
|
+
**Symptom.** Deploys happen; the Deploys page doesn't grow.
|
|
253
|
+
|
|
254
|
+
**Cause and fix**, in the order the hook itself checks:
|
|
255
|
+
|
|
256
|
+
- **`RAILWATCH_TOKEN` isn't exported to the hook.** The first thing
|
|
257
|
+
`.kamal/hooks/post-deploy` does is `[ -z "$RAILWATCH_TOKEN" ] && exit 0`.
|
|
258
|
+
The hook runs on the deployer machine, in your shell — not in a
|
|
259
|
+
container — so a token that only exists in `.kamal/secrets` for the
|
|
260
|
+
*app* isn't necessarily in the deployer's environment. Export it there
|
|
261
|
+
(or source the same secret store your CI uses).
|
|
262
|
+
- **`curl` or `ruby` isn't on the deployer, or `RAILWATCH_INGEST_URL`
|
|
263
|
+
isn't set.** Then the hook falls back to
|
|
264
|
+
`bin/kamal app exec --primary --reuse "bin/rails railwatch:deploy[$KAMAL_VERSION]"`,
|
|
265
|
+
which records the same deploy **minus the commit list** — a container
|
|
266
|
+
has the code but not the git history. If your deploys show up without
|
|
267
|
+
commits, this is the path you're on.
|
|
268
|
+
- **The hook isn't there.** The install generator only writes it when
|
|
269
|
+
`config/deploy.yml` already exists. Re-run
|
|
270
|
+
`bin/rails generate railwatch:install` after adopting Kamal.
|
|
271
|
+
|
|
272
|
+
The hook never fails a deploy: every network call ends in `|| true`, and
|
|
273
|
+
it exits 0 regardless.
|
|
274
|
+
|
|
275
|
+
## Log search finds less than it should
|
|
276
|
+
|
|
277
|
+
**Symptom.** On a Postgres-backed platform install, log search matches
|
|
278
|
+
fewer lines and highlights nothing.
|
|
279
|
+
|
|
280
|
+
**Cause.** Full-text search uses SQLite's FTS5 (`logs_fts`). The
|
|
281
|
+
platform checks for both — a SQLite adapter *and* the `logs_fts` table —
|
|
282
|
+
and when either is absent falls back to `message LIKE '%...%'`. That
|
|
283
|
+
fallback is a plain substring match: no phrase or negation syntax, and no
|
|
284
|
+
snippet highlighting.
|
|
285
|
+
|
|
286
|
+
**Fix.** Nothing on the gem side; this is a property of the platform's
|
|
287
|
+
telemetry store. SQLite is the default and first-class target for
|
|
288
|
+
per-environment telemetry precisely because of features like this. A
|
|
289
|
+
telemetry database created before the FTS index existed also falls back. A
|
|
290
|
+
self-hosted platform operator should rebuild the documented search index.
|
|
291
|
+
|
|
292
|
+
## The Tenants page is empty
|
|
293
|
+
|
|
294
|
+
**Symptom.** Every other page has data; Tenants shows nothing.
|
|
295
|
+
|
|
296
|
+
**Cause.** `tenant` is a column on every telemetry row, filled from
|
|
297
|
+
`Railwatch::Context.current_tenant`, and a tenant only exists as a GROUP BY
|
|
298
|
+
over those rows. If nothing ever sets it, every row has a null tenant and
|
|
299
|
+
there is nothing to group.
|
|
300
|
+
|
|
301
|
+
**Fix.** Apps on `activerecord-tenanted` get it free — Railwatch reads
|
|
302
|
+
`ActiveRecord::Base.current_tenant` / `TenantRecord.current_tenant` with
|
|
303
|
+
no configuration. Everyone else sets it explicitly, as early in the
|
|
304
|
+
request as the tenant is known:
|
|
305
|
+
|
|
306
|
+
```ruby
|
|
307
|
+
Railwatch.context(tenant: org.slug)
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
Set it in the same `before_action` that resolves the tenant, so every
|
|
311
|
+
record in the execution carries it. Context set after a record is built
|
|
312
|
+
does not retroactively apply to it.
|
|
313
|
+
|
|
314
|
+
## See also
|
|
315
|
+
|
|
316
|
+
- [`configuration.md`](configuration.md) — every option and its default.
|
|
317
|
+
- [`records.md`](records.md) — what each record type contains.
|
|
318
|
+
- [`faq.md`](faq.md) — overhead, retention, PII, and what happens when
|
|
319
|
+
the platform is unreachable.
|