dead_bro 0.2.9 → 0.2.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/FEATURES.md +1 -6
- data/README.md +82 -0
- data/lib/dead_bro/configuration.rb +45 -3
- data/lib/dead_bro/elasticsearch_subscriber.rb +141 -0
- data/lib/dead_bro/http_instrumentation.rb +108 -15
- data/lib/dead_bro/job_subscriber.rb +1 -1
- data/lib/dead_bro/railtie.rb +4 -0
- data/lib/dead_bro/sql_subscriber.rb +2 -3
- data/lib/dead_bro/sql_tracking_middleware.rb +7 -1
- data/lib/dead_bro/subscriber.rb +4 -1
- data/lib/dead_bro/version.rb +1 -1
- data/lib/dead_bro.rb +56 -37
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 56f6a72ccfa1a5e362c74eddace39a1ffeba97d06bab551f38d9091311609770
|
|
4
|
+
data.tar.gz: 20ee34708c5eed8ac00af8d462953422ba7a24e3b9701e56e31eea8ba35752f0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 932ce6601969ac9f95ba0ce327b93a5412c167a4c7fe367d0c14bd05c5c751b69bb3da48eaf829e73bcf6ca10c66fd7678ff0caca4eba7e0ec42a3b4bae72e54
|
|
7
|
+
data.tar.gz: 032ee4230c5b043a7ef8cf176963500eb60157927ad18f18c8e657f2439b31ab8b3526e7dd90036f79ec5fc7e978b55de540fcad14925f1fb33ffb6308379964
|
data/FEATURES.md
CHANGED
|
@@ -284,12 +284,7 @@ A comprehensive feature list for comparing ApmBro with other APM (Application Pe
|
|
|
284
284
|
## Deployment & Environment
|
|
285
285
|
|
|
286
286
|
### Deploy Tracking
|
|
287
|
-
- **Deploy ID Resolution**: Multiple sources for deploy identification
|
|
288
|
-
- Explicit configuration
|
|
289
|
-
- Rails settings/credentials
|
|
290
|
-
- Environment variables (APM_BRO_DEPLOY_ID, GIT_REV)
|
|
291
|
-
- Heroku (HEROKU_SLUG_COMMIT)
|
|
292
|
-
- Process-stable UUID fallback
|
|
287
|
+
- **Deploy ID Resolution**: Multiple sources for deploy identification (`Configuration#deploy_id=` wins when set, then ENV in `Configuration::DEPLOY_REVISION_ENV_KEYS` order—including `DEAD_BRO_DEPLOY_ID`, git/CI vars, `DD_VERSION`, etc.), otherwise a **per-process UUID** (fine for single dyno/process; unusable alone for fleets like ECS replicas)
|
|
293
288
|
- **Revision Tracking**: Includes deploy/revision ID in all metric payloads
|
|
294
289
|
|
|
295
290
|
### Environment Support
|
data/README.md
CHANGED
|
@@ -30,6 +30,25 @@ end
|
|
|
30
30
|
|
|
31
31
|
That is enough to start shipping metrics. Create or copy the key from your DeadBro account, then wire it into `DEAD_BRO_API_KEY` (or assign `config.api_key` directly).
|
|
32
32
|
|
|
33
|
+
### Deploy / release tracking (ECS, Kubernetes, autoscaling groups)
|
|
34
|
+
|
|
35
|
+
DeadBro sends a **`revision`** string on every payload so charts and deploys group metrics by release. If you do **not** configure a stable value, the gem generates a UUID **once per Ruby process**. With multiple containers or EC2 instances, each replica then looks like a separate deployment.
|
|
36
|
+
|
|
37
|
+
**Use one shared identifier for every task in the same rollout**, for example:
|
|
38
|
+
|
|
39
|
+
- Set `DEAD_BRO_DEPLOY_ID` (or `dead_bro_DEPLOY_ID`) in the task definition / pod spec to your **git SHA**, **image digest**, or CD **release id** injected at build or deploy time.
|
|
40
|
+
- Or set it in Ruby (this wins over environment variables when non-blank):
|
|
41
|
+
|
|
42
|
+
```ruby
|
|
43
|
+
DeadBro.configure do |config|
|
|
44
|
+
config.deploy_id = ENV.fetch("GIT_SHA") { raise "Set GIT_SHA in the task definition" }
|
|
45
|
+
end
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The gem also checks these environment variables in order (first non-empty wins): `DEAD_BRO_DEPLOY_ID`, `dead_bro_DEPLOY_ID`, `GIT_REV`, `GIT_COMMIT`, `GIT_COMMIT_SHA`, `GIT_SHA`, `CODEBUILD_RESOLVED_SOURCE_REVISION`, `HEROKU_SLUG_COMMIT`, `RENDER_GIT_COMMIT`, `DD_VERSION`, `APP_REVISION`, `RELEASE_VERSION`, `SOURCE_VERSION`.
|
|
49
|
+
|
|
50
|
+
For **Amazon ECS**, add one of these variables in your task definition from CodeBuild, GitHub Actions, or your pipeline so all tasks in the service share the same value for that image version.
|
|
51
|
+
|
|
33
52
|
### Dashboard configuration
|
|
34
53
|
|
|
35
54
|
Use the DeadBro UI to turn features on or off, set sample rates, define controller/job inclusions and exclusions, tune slow-query EXPLAIN, enable queue and system metrics, and adjust related limits. After you deploy the initializer above, those choices take effect when the gem receives them from the API (typically on the next successful metric or heartbeat response).
|
|
@@ -388,6 +407,69 @@ The control plane job sends a single JSON payload roughly shaped like:
|
|
|
388
407
|
Not all fields will be present in all environments; unsupported or unavailable metrics may be `null` or omitted, and any hard failures are captured in `error_class` / `error_message` fields per section.
|
|
389
408
|
|
|
390
409
|
|
|
410
|
+
## One-Off Analysis with `DeadBro.analyze`
|
|
411
|
+
|
|
412
|
+
Use `DeadBro.analyze` to profile any block of code inline — useful in the Rails console, rake tasks, or debug sessions. It tracks execution time, SQL queries, and memory usage without sending data to the DeadBro backend.
|
|
413
|
+
|
|
414
|
+
```ruby
|
|
415
|
+
result = DeadBro.analyze("load active users") do
|
|
416
|
+
User.where(active: true).includes(:profile).to_a
|
|
417
|
+
end
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
Returns a `DeadBro::AnalysisResult` struct.
|
|
421
|
+
|
|
422
|
+
### Return Value
|
|
423
|
+
|
|
424
|
+
`DeadBro::AnalysisResult` exposes:
|
|
425
|
+
|
|
426
|
+
| Member | Type | Description |
|
|
427
|
+
|---|---|---|
|
|
428
|
+
| `label` | String | Label passed to the block |
|
|
429
|
+
| `total_time_ms` | Float | Wall time of the block |
|
|
430
|
+
| `sql_count` | Integer | Number of SQL queries executed |
|
|
431
|
+
| `sql_time_ms` | Float | Total SQL execution time |
|
|
432
|
+
| `sql_queries` | Array | Per-query breakdown (see below) |
|
|
433
|
+
| `memory_before_mb` | Float | RSS before block |
|
|
434
|
+
| `memory_after_mb` | Float | RSS after block |
|
|
435
|
+
| `memory_delta_mb` | Float | Memory change |
|
|
436
|
+
| `memory_details` | Hash | GC stats, new objects, heap pages |
|
|
437
|
+
|
|
438
|
+
**`sql_queries` is intentionally excluded from `inspect`/`to_s`** to avoid flooding the console with a long array. Access it explicitly when needed:
|
|
439
|
+
|
|
440
|
+
```ruby
|
|
441
|
+
result # => #<DeadBro::AnalysisResult label="load active users" total_time_ms=42.3 ...>
|
|
442
|
+
result.sql_queries # => [{sql: "SELECT ...", query_type: "SELECT", count: 3, total_time_ms: 12.1}, ...]
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
### Options
|
|
446
|
+
|
|
447
|
+
```ruby
|
|
448
|
+
DeadBro.analyze("my block", verbose: true) do
|
|
449
|
+
# verbose: true lowers the Rails log level to DEBUG and enables
|
|
450
|
+
# ActiveRecord.verbose_query_logs for the duration of the block
|
|
451
|
+
MyService.call
|
|
452
|
+
end
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
### Helper methods
|
|
456
|
+
|
|
457
|
+
```ruby
|
|
458
|
+
result.most_queries # top 5 query patterns by execution count
|
|
459
|
+
result.longest_queries # top 5 query patterns by total_time_ms
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
Both return an array of query hashes (same shape as `sql_queries`):
|
|
463
|
+
|
|
464
|
+
- `sql` — normalized SQL (literals replaced, whitespace collapsed)
|
|
465
|
+
- `query_type` — e.g. `"SELECT"`, `"INSERT"`, `"UPDATE"`
|
|
466
|
+
- `count` — how many times this pattern ran
|
|
467
|
+
- `total_time_ms` — combined time across all occurrences
|
|
468
|
+
|
|
469
|
+
### `sql_queries` fields
|
|
470
|
+
|
|
471
|
+
`result.sql_queries` returns the full list. Each entry is a hash with the same fields as above.
|
|
472
|
+
|
|
391
473
|
## Development
|
|
392
474
|
|
|
393
475
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
|
@@ -7,7 +7,7 @@ module DeadBro
|
|
|
7
7
|
# returns it in a response; local configure() values apply until the next remote update.
|
|
8
8
|
attr_accessor :api_key, :open_timeout, :read_timeout, :enabled, :ruby_dev,
|
|
9
9
|
:circuit_breaker_enabled, :circuit_breaker_failure_threshold, :circuit_breaker_recovery_timeout,
|
|
10
|
-
:circuit_breaker_retry_timeout, :
|
|
10
|
+
:circuit_breaker_retry_timeout, :disk_paths, :interfaces_ignore
|
|
11
11
|
|
|
12
12
|
# Remote-managed settings (overwritten by backend JSON `settings` on successful API responses)
|
|
13
13
|
attr_accessor :memory_tracking_enabled, :allocation_tracking_enabled,
|
|
@@ -30,6 +30,24 @@ module DeadBro
|
|
|
30
30
|
|
|
31
31
|
HEARTBEAT_INTERVAL = 60 # seconds
|
|
32
32
|
|
|
33
|
+
# First non-empty ENV value wins for release/revision payloads and deploy grouping on the server.
|
|
34
|
+
# Order is roughly: DeadBro-native → common CI/hosting → observability tooling.
|
|
35
|
+
DEPLOY_REVISION_ENV_KEYS = %w[
|
|
36
|
+
DEAD_BRO_DEPLOY_ID
|
|
37
|
+
dead_bro_DEPLOY_ID
|
|
38
|
+
GIT_REV
|
|
39
|
+
GIT_COMMIT
|
|
40
|
+
GIT_COMMIT_SHA
|
|
41
|
+
GIT_SHA
|
|
42
|
+
CODEBUILD_RESOLVED_SOURCE_REVISION
|
|
43
|
+
HEROKU_SLUG_COMMIT
|
|
44
|
+
RENDER_GIT_COMMIT
|
|
45
|
+
DD_VERSION
|
|
46
|
+
APP_REVISION
|
|
47
|
+
RELEASE_VERSION
|
|
48
|
+
SOURCE_VERSION
|
|
49
|
+
].freeze
|
|
50
|
+
|
|
33
51
|
REMOTE_SETTING_KEYS = %w[
|
|
34
52
|
enabled sample_rate memory_tracking_enabled allocation_tracking_enabled
|
|
35
53
|
explain_analyze_enabled slow_query_threshold_ms max_sql_queries_to_send max_logs_to_send
|
|
@@ -47,7 +65,7 @@ module DeadBro
|
|
|
47
65
|
@circuit_breaker_failure_threshold = 3
|
|
48
66
|
@circuit_breaker_recovery_timeout = 60
|
|
49
67
|
@circuit_breaker_retry_timeout = 300
|
|
50
|
-
@
|
|
68
|
+
@explicit_deploy_revision = nil
|
|
51
69
|
@disk_paths = ["/"]
|
|
52
70
|
@interfaces_ignore = %w[lo lo0 docker0]
|
|
53
71
|
|
|
@@ -74,6 +92,17 @@ module DeadBro
|
|
|
74
92
|
@settings_mutex = Mutex.new
|
|
75
93
|
end
|
|
76
94
|
|
|
95
|
+
# Current release revision sent as `revision` on all API payloads — same semantics as `#resolve_deploy_id`.
|
|
96
|
+
def deploy_id
|
|
97
|
+
resolve_deploy_id
|
|
98
|
+
end
|
|
99
|
+
|
|
100
|
+
# Overrides ENV-based resolution when set to a non-empty string (or clears override when nil/blank).
|
|
101
|
+
def deploy_id=(value)
|
|
102
|
+
s = value&.respond_to?(:to_s) ? value.to_s.strip : ""
|
|
103
|
+
@explicit_deploy_revision = s.empty? ? nil : s
|
|
104
|
+
end
|
|
105
|
+
|
|
77
106
|
def excluded_controllers=(value)
|
|
78
107
|
@excluded_controllers = Array(value).map(&:to_s)
|
|
79
108
|
@compiled_excluded_controllers = compile_patterns(@excluded_controllers)
|
|
@@ -124,7 +153,20 @@ module DeadBro
|
|
|
124
153
|
end
|
|
125
154
|
|
|
126
155
|
def resolve_deploy_id
|
|
127
|
-
|
|
156
|
+
explicit = @explicit_deploy_revision&.to_s&.strip
|
|
157
|
+
return explicit unless explicit.nil? || explicit.empty?
|
|
158
|
+
|
|
159
|
+
DEPLOY_REVISION_ENV_KEYS.each do |key|
|
|
160
|
+
v = ENV[key]
|
|
161
|
+
next unless v.respond_to?(:to_s)
|
|
162
|
+
|
|
163
|
+
stripped = v.to_s.strip
|
|
164
|
+
next if stripped.empty?
|
|
165
|
+
|
|
166
|
+
return stripped
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
DeadBro.process_deploy_id
|
|
128
170
|
end
|
|
129
171
|
|
|
130
172
|
def excluded_controller?(controller_name, action_name = nil)
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module DeadBro
|
|
4
|
+
class ElasticsearchSubscriber
|
|
5
|
+
THREAD_LOCAL_KEY = :dead_bro_elasticsearch_events
|
|
6
|
+
MAX_TRACKED_EVENTS = 500
|
|
7
|
+
|
|
8
|
+
# Install gem-based notification subscriber (request.elasticsearch / request.elastic_transport).
|
|
9
|
+
# The Net::HTTP path is handled by HttpInstrumentation, which calls .record directly.
|
|
10
|
+
def self.subscribe!
|
|
11
|
+
install_notifications_subscription!
|
|
12
|
+
rescue
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
# Called by HttpInstrumentation when it detects a Net::HTTP request to an ES host.
|
|
16
|
+
def self.record(method:, path:, status:, duration_ms:)
|
|
17
|
+
events = Thread.current[THREAD_LOCAL_KEY]
|
|
18
|
+
return unless events
|
|
19
|
+
return unless should_continue_tracking?
|
|
20
|
+
|
|
21
|
+
events << build_event(method, path, status, duration_ms)
|
|
22
|
+
rescue
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def self.start_request_tracking
|
|
26
|
+
Thread.current[THREAD_LOCAL_KEY] = []
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def self.stop_request_tracking
|
|
30
|
+
events = Thread.current[THREAD_LOCAL_KEY]
|
|
31
|
+
Thread.current[THREAD_LOCAL_KEY] = nil
|
|
32
|
+
events || []
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def self.should_continue_tracking?
|
|
36
|
+
events = Thread.current[THREAD_LOCAL_KEY]
|
|
37
|
+
return false unless events
|
|
38
|
+
return false if events.length >= MAX_TRACKED_EVENTS
|
|
39
|
+
|
|
40
|
+
start_time = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
|
|
41
|
+
if start_time
|
|
42
|
+
elapsed_seconds = Time.now - start_time
|
|
43
|
+
return false if elapsed_seconds >= DeadBro::MAX_TRACKING_DURATION_SECONDS
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
true
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def self.extract_operation(method, path)
|
|
50
|
+
return "unknown" if path.nil?
|
|
51
|
+
|
|
52
|
+
clean = path.to_s.split("?").first.to_s
|
|
53
|
+
m = method.to_s.upcase
|
|
54
|
+
|
|
55
|
+
if clean =~ /_search\z/i
|
|
56
|
+
"search"
|
|
57
|
+
elsif clean =~ /_msearch\z/i
|
|
58
|
+
"msearch"
|
|
59
|
+
elsif clean =~ /_bulk\z/i
|
|
60
|
+
"bulk"
|
|
61
|
+
elsif clean =~ /_doc\/[^\/]+\/_update\z/i
|
|
62
|
+
"update"
|
|
63
|
+
elsif clean =~ /_update\/[^\/]+\z/i
|
|
64
|
+
"update"
|
|
65
|
+
elsif clean =~ /_delete_by_query\z/i
|
|
66
|
+
"delete_by_query"
|
|
67
|
+
elsif clean =~ /_count\z/i
|
|
68
|
+
"count"
|
|
69
|
+
elsif clean =~ /_mapping\z/i
|
|
70
|
+
m == "GET" ? "get_mapping" : "put_mapping"
|
|
71
|
+
elsif clean =~ /_doc\/[^\/]+\z/i
|
|
72
|
+
case m
|
|
73
|
+
when "GET" then "get"
|
|
74
|
+
when "DELETE" then "delete"
|
|
75
|
+
when "POST", "PUT" then "index"
|
|
76
|
+
else "doc"
|
|
77
|
+
end
|
|
78
|
+
elsif clean =~ /_doc\z/i
|
|
79
|
+
"index"
|
|
80
|
+
elsif clean =~ /\A\/_cluster\//i
|
|
81
|
+
"cluster"
|
|
82
|
+
elsif clean =~ /\A\/_cat\//i
|
|
83
|
+
"cat"
|
|
84
|
+
elsif clean =~ /\A\/[^\/]+\z/
|
|
85
|
+
case m
|
|
86
|
+
when "PUT" then "create_index"
|
|
87
|
+
when "DELETE" then "delete_index"
|
|
88
|
+
when "HEAD" then "index_exists"
|
|
89
|
+
when "GET" then "get_index"
|
|
90
|
+
else "index_op"
|
|
91
|
+
end
|
|
92
|
+
else
|
|
93
|
+
m.downcase
|
|
94
|
+
end
|
|
95
|
+
rescue
|
|
96
|
+
"unknown"
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def self.sanitize_path(path)
|
|
100
|
+
return "" if path.nil?
|
|
101
|
+
path.to_s
|
|
102
|
+
.gsub(/\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i, "/{id}")
|
|
103
|
+
.gsub(/\/\d+(?=\/|\z)/, "/{id}")
|
|
104
|
+
rescue
|
|
105
|
+
path.to_s
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
class << self
|
|
109
|
+
private
|
|
110
|
+
|
|
111
|
+
def install_notifications_subscription!
|
|
112
|
+
return unless defined?(::ActiveSupport::Notifications)
|
|
113
|
+
|
|
114
|
+
%w[request.elasticsearch request.elastic_transport].each do |event_name|
|
|
115
|
+
::ActiveSupport::Notifications.subscribe(event_name) do |_name, started, finished, _id, payload|
|
|
116
|
+
events = Thread.current[THREAD_LOCAL_KEY]
|
|
117
|
+
next unless events
|
|
118
|
+
next unless should_continue_tracking?
|
|
119
|
+
|
|
120
|
+
duration_ms = ((finished - started) * 1000.0).round(2)
|
|
121
|
+
method = payload[:method].to_s.upcase
|
|
122
|
+
path = payload[:path].to_s
|
|
123
|
+
events << build_event(method, path, payload[:status], duration_ms)
|
|
124
|
+
rescue
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
rescue
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
def build_event(method, path, status, duration_ms)
|
|
131
|
+
{
|
|
132
|
+
method: method.to_s.upcase,
|
|
133
|
+
path: sanitize_path(path),
|
|
134
|
+
operation: extract_operation(method, path),
|
|
135
|
+
status: status,
|
|
136
|
+
duration_ms: duration_ms
|
|
137
|
+
}
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
end
|
|
@@ -12,6 +12,7 @@ module DeadBro
|
|
|
12
12
|
def self.install!(client: Client.new)
|
|
13
13
|
install_net_http!(client)
|
|
14
14
|
install_typhoeus!(client) if defined?(::Typhoeus)
|
|
15
|
+
install_faraday!(client) if defined?(::Faraday)
|
|
15
16
|
rescue
|
|
16
17
|
# Never raise from instrumentation install
|
|
17
18
|
end
|
|
@@ -38,23 +39,37 @@ module DeadBro
|
|
|
38
39
|
nil
|
|
39
40
|
end
|
|
40
41
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
42
|
+
host = (uri && uri.host) || @address
|
|
43
|
+
port = (uri && uri.port) || @port
|
|
44
|
+
is_es_host = DeadBro::HttpInstrumentation.elasticsearch_host?(host, port)
|
|
45
|
+
# Skip localhost/internal only for non-ES hosts. ES on localhost (e.g. port 9200)
|
|
46
|
+
# must still be tracked; only skip the deadbro backend itself.
|
|
47
|
+
skip_instrumentation = !is_es_host && uri && (uri.to_s.include?("localhost") || uri.to_s.include?("aberatii.com"))
|
|
44
48
|
|
|
45
|
-
|
|
49
|
+
if is_es_host
|
|
50
|
+
# Route to elasticsearch subscriber instead of http_outgoing
|
|
51
|
+
if Thread.current[DeadBro::ElasticsearchSubscriber::THREAD_LOCAL_KEY]
|
|
52
|
+
path = (uri && uri.path) || req.path
|
|
53
|
+
DeadBro::ElasticsearchSubscriber.record(
|
|
54
|
+
method: req.method,
|
|
55
|
+
path: path,
|
|
56
|
+
status: response && response.code.to_i,
|
|
57
|
+
duration_ms: duration_ms
|
|
58
|
+
)
|
|
59
|
+
end
|
|
60
|
+
elsif !skip_instrumentation
|
|
61
|
+
lib = DeadBro::HttpInstrumentation.typesense_host?(host, port) ? "typesense" : "net_http"
|
|
46
62
|
payload = {
|
|
47
|
-
library:
|
|
63
|
+
library: lib,
|
|
48
64
|
method: req.method,
|
|
49
65
|
url: uri && uri.to_s,
|
|
50
|
-
host:
|
|
66
|
+
host: host,
|
|
51
67
|
path: (uri && uri.path) || req.path,
|
|
52
68
|
status: response && response.code.to_i,
|
|
53
69
|
duration_ms: duration_ms,
|
|
54
70
|
exception: error && error.class.name
|
|
55
71
|
}
|
|
56
|
-
|
|
57
|
-
if Thread.current[THREAD_LOCAL_KEY] && should_continue_tracking?
|
|
72
|
+
if Thread.current[THREAD_LOCAL_KEY] && DeadBro::HttpInstrumentation.should_continue_tracking?
|
|
58
73
|
Thread.current[THREAD_LOCAL_KEY] << payload
|
|
59
74
|
end
|
|
60
75
|
end
|
|
@@ -85,8 +100,6 @@ module DeadBro
|
|
|
85
100
|
(respond_to?(:base_url) ? base_url : nil)
|
|
86
101
|
end
|
|
87
102
|
|
|
88
|
-
# Skip instrumentation for our own APM endpoint to prevent infinite loops,
|
|
89
|
-
# but do NOT alter the original method's return value/control flow.
|
|
90
103
|
skip_instrumentation = req_url && (req_url.include?("localhost:3100/apm/v1/metrics") || req_url.include?("deadbro.aberatii.com/apm/v1/metrics"))
|
|
91
104
|
|
|
92
105
|
unless skip_instrumentation
|
|
@@ -97,8 +110,7 @@ module DeadBro
|
|
|
97
110
|
status: response && response.code,
|
|
98
111
|
duration_ms: duration_ms
|
|
99
112
|
}
|
|
100
|
-
|
|
101
|
-
if Thread.current[THREAD_LOCAL_KEY] && should_continue_tracking?
|
|
113
|
+
if Thread.current[THREAD_LOCAL_KEY] && DeadBro::HttpInstrumentation.should_continue_tracking?
|
|
102
114
|
Thread.current[THREAD_LOCAL_KEY] << payload
|
|
103
115
|
end
|
|
104
116
|
end
|
|
@@ -112,15 +124,96 @@ module DeadBro
|
|
|
112
124
|
rescue
|
|
113
125
|
end
|
|
114
126
|
|
|
115
|
-
|
|
127
|
+
def self.install_faraday!(client)
|
|
128
|
+
return unless defined?(::Faraday)
|
|
129
|
+
|
|
130
|
+
unless defined?(::DeadBro::FaradayMiddleware)
|
|
131
|
+
middleware_klass = Class.new(::Faraday::Middleware) do
|
|
132
|
+
def call(env)
|
|
133
|
+
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
134
|
+
response = nil
|
|
135
|
+
begin
|
|
136
|
+
response = @app.call(env)
|
|
137
|
+
ensure
|
|
138
|
+
finish_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
139
|
+
duration_ms = ((finish_time - start_time) * 1000.0).round(2)
|
|
140
|
+
begin
|
|
141
|
+
url = env.url
|
|
142
|
+
host = url.host.to_s
|
|
143
|
+
port = url.port
|
|
144
|
+
url_str = url.to_s
|
|
145
|
+
|
|
146
|
+
is_es_host = DeadBro::HttpInstrumentation.elasticsearch_host?(host, port)
|
|
147
|
+
skip = !is_es_host && (url_str.include?("localhost") || url_str.include?("aberatii.com"))
|
|
148
|
+
|
|
149
|
+
if is_es_host
|
|
150
|
+
if Thread.current[DeadBro::ElasticsearchSubscriber::THREAD_LOCAL_KEY]
|
|
151
|
+
DeadBro::ElasticsearchSubscriber.record(
|
|
152
|
+
method: env.method.to_s.upcase,
|
|
153
|
+
path: url.path,
|
|
154
|
+
status: response&.status,
|
|
155
|
+
duration_ms: duration_ms
|
|
156
|
+
)
|
|
157
|
+
end
|
|
158
|
+
elsif !skip
|
|
159
|
+
lib = DeadBro::HttpInstrumentation.typesense_host?(host, port) ? "typesense" : "faraday"
|
|
160
|
+
payload = {
|
|
161
|
+
library: lib,
|
|
162
|
+
method: env.method.to_s.upcase,
|
|
163
|
+
url: url_str,
|
|
164
|
+
host: host,
|
|
165
|
+
path: url.path,
|
|
166
|
+
status: response&.status,
|
|
167
|
+
duration_ms: duration_ms
|
|
168
|
+
}
|
|
169
|
+
key = DeadBro::HttpInstrumentation::THREAD_LOCAL_KEY
|
|
170
|
+
if Thread.current[key] && DeadBro::HttpInstrumentation.should_continue_tracking?
|
|
171
|
+
Thread.current[key] << payload
|
|
172
|
+
end
|
|
173
|
+
end
|
|
174
|
+
rescue
|
|
175
|
+
end
|
|
176
|
+
end
|
|
177
|
+
end
|
|
178
|
+
end
|
|
179
|
+
::DeadBro.const_set(:FaradayMiddleware, middleware_klass)
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
unless defined?(::DeadBro::FaradayInstrumentation)
|
|
183
|
+
instrumentation_mod = Module.new do
|
|
184
|
+
define_method(:initialize) do |url = nil, options = {}, &block|
|
|
185
|
+
super(url, options, &block)
|
|
186
|
+
unless builder.handlers.map(&:klass).include?(::DeadBro::FaradayMiddleware)
|
|
187
|
+
builder.use(::DeadBro::FaradayMiddleware)
|
|
188
|
+
end
|
|
189
|
+
rescue
|
|
190
|
+
end
|
|
191
|
+
end
|
|
192
|
+
::DeadBro.const_set(:FaradayInstrumentation, instrumentation_mod)
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
::Faraday::Connection.prepend(::DeadBro::FaradayInstrumentation) unless ::Faraday::Connection.ancestors.include?(::DeadBro::FaradayInstrumentation)
|
|
196
|
+
rescue
|
|
197
|
+
end
|
|
198
|
+
|
|
199
|
+
def self.elasticsearch_host?(host, port)
|
|
200
|
+
return false if host.nil?
|
|
201
|
+
return true if port == 9200
|
|
202
|
+
h = host.to_s
|
|
203
|
+
h.end_with?(".elastic.co") || h.end_with?(".es.amazonaws.com") || h.include?("elasticsearch")
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
def self.typesense_host?(host, port)
|
|
207
|
+
return false if host.nil?
|
|
208
|
+
port == 8108 || host.to_s.end_with?(".typesense.io")
|
|
209
|
+
end
|
|
210
|
+
|
|
116
211
|
def self.should_continue_tracking?
|
|
117
212
|
events = Thread.current[THREAD_LOCAL_KEY]
|
|
118
213
|
return false unless events
|
|
119
214
|
|
|
120
|
-
# Check count limit
|
|
121
215
|
return false if events.length >= MAX_TRACKED_EVENTS
|
|
122
216
|
|
|
123
|
-
# Check time limit
|
|
124
217
|
start_time = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
|
|
125
218
|
if start_time
|
|
126
219
|
elapsed_seconds = Time.now - start_time
|
|
@@ -215,7 +215,7 @@ module DeadBro
|
|
|
215
215
|
(arg.length > 200) ? arg[0, 200] + "..." : arg
|
|
216
216
|
when Hash
|
|
217
217
|
# Filter sensitive keys and limit size
|
|
218
|
-
filtered = arg.
|
|
218
|
+
filtered = arg.reject { |k, _| %w[password token secret key].include?(k.to_s) }
|
|
219
219
|
(filtered.keys.size > 20) ? filtered.first(20).to_h : filtered
|
|
220
220
|
when Array
|
|
221
221
|
arg.first(5)
|
data/lib/dead_bro/railtie.rb
CHANGED
|
@@ -32,6 +32,10 @@ if defined?(Rails) && defined?(Rails::Railtie)
|
|
|
32
32
|
require "dead_bro/redis_subscriber"
|
|
33
33
|
DeadBro::RedisSubscriber.subscribe!
|
|
34
34
|
|
|
35
|
+
# Install Elasticsearch / OpenSearch tracking
|
|
36
|
+
require "dead_bro/elasticsearch_subscriber"
|
|
37
|
+
DeadBro::ElasticsearchSubscriber.subscribe!
|
|
38
|
+
|
|
35
39
|
# Install view rendering tracking
|
|
36
40
|
require "dead_bro/view_rendering_subscriber"
|
|
37
41
|
DeadBro::ViewRenderingSubscriber.subscribe!(client: shared_client)
|
|
@@ -151,9 +151,8 @@ module DeadBro
|
|
|
151
151
|
# storm from spawning unbounded background threads.
|
|
152
152
|
MAX_PENDING_EXPLAINS = 20
|
|
153
153
|
# Overall wall-clock we're willing to block the request thread for pending
|
|
154
|
-
# EXPLAINs.
|
|
155
|
-
|
|
156
|
-
EXPLAIN_WAIT_TIMEOUT_SECONDS = 1.0
|
|
154
|
+
# EXPLAINs. If the plan isn't ready by then, skip it rather than stall the request.
|
|
155
|
+
EXPLAIN_WAIT_TIMEOUT_SECONDS = 5.0
|
|
157
156
|
|
|
158
157
|
def self.wait_for_pending_explains(timeout_seconds)
|
|
159
158
|
pending = Thread.current[THREAD_LOCAL_EXPLAIN_PENDING_KEY]
|
|
@@ -40,6 +40,11 @@ module DeadBro
|
|
|
40
40
|
DeadBro::MemoryTrackingSubscriber.start_request_tracking
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
+
# Start Elasticsearch tracking for this request
|
|
44
|
+
if defined?(DeadBro::ElasticsearchSubscriber)
|
|
45
|
+
DeadBro::ElasticsearchSubscriber.start_request_tracking
|
|
46
|
+
end
|
|
47
|
+
|
|
43
48
|
# Start outgoing HTTP accumulation for this request
|
|
44
49
|
Thread.current[:dead_bro_http_events] = []
|
|
45
50
|
|
|
@@ -74,7 +79,8 @@ module DeadBro
|
|
|
74
79
|
Thread.current[:dead_bro_lightweight_memory] = nil
|
|
75
80
|
end
|
|
76
81
|
|
|
77
|
-
# Clean up HTTP events and tracking start time
|
|
82
|
+
# Clean up HTTP events, ES events, and tracking start time
|
|
83
|
+
Thread.current[:dead_bro_elasticsearch_events] = nil
|
|
78
84
|
Thread.current[:dead_bro_http_events] = nil
|
|
79
85
|
Thread.current[DeadBro::TRACKING_START_TIME_KEY] = nil
|
|
80
86
|
end
|
data/lib/dead_bro/subscriber.rb
CHANGED
|
@@ -46,9 +46,10 @@ module DeadBro
|
|
|
46
46
|
# Stop SQL tracking and get collected queries (this was started by the request)
|
|
47
47
|
sql_queries = DeadBro::SqlSubscriber.stop_request_tracking
|
|
48
48
|
|
|
49
|
-
# Stop cache and
|
|
49
|
+
# Stop cache, redis, and elasticsearch tracking
|
|
50
50
|
cache_events = defined?(DeadBro::CacheSubscriber) ? DeadBro::CacheSubscriber.stop_request_tracking : []
|
|
51
51
|
redis_events = defined?(DeadBro::RedisSubscriber) ? DeadBro::RedisSubscriber.stop_request_tracking : []
|
|
52
|
+
elasticsearch_events = defined?(DeadBro::ElasticsearchSubscriber) ? DeadBro::ElasticsearchSubscriber.stop_request_tracking : []
|
|
52
53
|
|
|
53
54
|
# Stop view rendering tracking and get collected view events
|
|
54
55
|
view_events = DeadBro::ViewRenderingSubscriber.stop_request_tracking
|
|
@@ -152,6 +153,7 @@ module DeadBro
|
|
|
152
153
|
http_outgoing: Thread.current[:dead_bro_http_events] || [],
|
|
153
154
|
cache_events: cache_events,
|
|
154
155
|
redis_events: redis_events,
|
|
156
|
+
elasticsearch_events: elasticsearch_events,
|
|
155
157
|
cache_hits: cache_hits(data),
|
|
156
158
|
cache_misses: cache_misses(data),
|
|
157
159
|
view_events: view_events,
|
|
@@ -171,6 +173,7 @@ module DeadBro
|
|
|
171
173
|
DeadBro::SqlSubscriber.stop_request_tracking if defined?(DeadBro::SqlSubscriber)
|
|
172
174
|
DeadBro::CacheSubscriber.stop_request_tracking if defined?(DeadBro::CacheSubscriber)
|
|
173
175
|
DeadBro::RedisSubscriber.stop_request_tracking if defined?(DeadBro::RedisSubscriber)
|
|
176
|
+
DeadBro::ElasticsearchSubscriber.stop_request_tracking if defined?(DeadBro::ElasticsearchSubscriber)
|
|
174
177
|
DeadBro::ViewRenderingSubscriber.stop_request_tracking if defined?(DeadBro::ViewRenderingSubscriber)
|
|
175
178
|
DeadBro::LightweightMemoryTracker.stop_request_tracking if defined?(DeadBro::LightweightMemoryTracker)
|
|
176
179
|
if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
|
data/lib/dead_bro/version.rb
CHANGED
data/lib/dead_bro.rb
CHANGED
|
@@ -13,6 +13,7 @@ module DeadBro
|
|
|
13
13
|
autoload :SqlTrackingMiddleware, "dead_bro/sql_tracking_middleware"
|
|
14
14
|
autoload :CacheSubscriber, "dead_bro/cache_subscriber"
|
|
15
15
|
autoload :RedisSubscriber, "dead_bro/redis_subscriber"
|
|
16
|
+
autoload :ElasticsearchSubscriber, "dead_bro/elasticsearch_subscriber"
|
|
16
17
|
autoload :ViewRenderingSubscriber, "dead_bro/view_rendering_subscriber"
|
|
17
18
|
autoload :MemoryTrackingSubscriber, "dead_bro/memory_tracking_subscriber"
|
|
18
19
|
autoload :MemoryLeakDetector, "dead_bro/memory_leak_detector"
|
|
@@ -30,6 +31,57 @@ module DeadBro
|
|
|
30
31
|
|
|
31
32
|
class Error < StandardError; end
|
|
32
33
|
|
|
34
|
+
# Returned by DeadBro.analyze. sql_queries is intentionally omitted from
|
|
35
|
+
# inspect/to_s/pretty_print to avoid bloating console output.
|
|
36
|
+
# Access it explicitly via .sql_queries.
|
|
37
|
+
class AnalysisResult
|
|
38
|
+
attr_reader :label, :total_time_ms, :sql_count, :sql_time_ms,
|
|
39
|
+
:sql_queries, :memory_before_mb, :memory_after_mb,
|
|
40
|
+
:memory_delta_mb, :memory_details, :verbose
|
|
41
|
+
|
|
42
|
+
def initialize(label:, total_time_ms:, sql_count:, sql_time_ms:, sql_queries:,
|
|
43
|
+
memory_before_mb:, memory_after_mb:, memory_delta_mb:,
|
|
44
|
+
memory_details:, verbose:)
|
|
45
|
+
@label = label
|
|
46
|
+
@total_time_ms = total_time_ms
|
|
47
|
+
@sql_count = sql_count
|
|
48
|
+
@sql_time_ms = sql_time_ms
|
|
49
|
+
@sql_queries = sql_queries
|
|
50
|
+
@memory_before_mb = memory_before_mb
|
|
51
|
+
@memory_after_mb = memory_after_mb
|
|
52
|
+
@memory_delta_mb = memory_delta_mb
|
|
53
|
+
@memory_details = memory_details
|
|
54
|
+
@verbose = verbose
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def inspect
|
|
58
|
+
"#<DeadBro::AnalysisResult label=#{label.inspect} total_time_ms=#{total_time_ms} " \
|
|
59
|
+
"sql_count=#{sql_count} sql_time_ms=#{sql_time_ms} " \
|
|
60
|
+
"memory_before_mb=#{memory_before_mb} memory_after_mb=#{memory_after_mb} " \
|
|
61
|
+
"memory_delta_mb=#{memory_delta_mb}>"
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
alias_method :to_s, :inspect
|
|
65
|
+
|
|
66
|
+
def most_queries
|
|
67
|
+
sql_queries.max_by(5) { |q| q[:count] }
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def longest_queries
|
|
71
|
+
sql_queries.max_by(5) { |q| q[:total_time_ms] }
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def pretty_print(pp)
|
|
75
|
+
pp.text(inspect)
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
def [](key)
|
|
79
|
+
public_send(key)
|
|
80
|
+
rescue NoMethodError
|
|
81
|
+
nil
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
|
|
33
85
|
def self.configure
|
|
34
86
|
yield configuration
|
|
35
87
|
end
|
|
@@ -102,7 +154,8 @@ module DeadBro
|
|
|
102
154
|
# - memory before/after and delta
|
|
103
155
|
# - when detailed memory tracking is enabled, GC and allocation stats
|
|
104
156
|
#
|
|
105
|
-
# The return value
|
|
157
|
+
# The return value is a DeadBro::AnalysisResult struct. sql_queries is omitted from
|
|
158
|
+
# inspect/to_s but accessible via .sql_queries. Other members:
|
|
106
159
|
# - :label
|
|
107
160
|
# - :total_time_ms
|
|
108
161
|
# - :sql_count
|
|
@@ -249,8 +302,6 @@ module DeadBro
|
|
|
249
302
|
entry[:type] ||= q[:query_type]
|
|
250
303
|
end
|
|
251
304
|
|
|
252
|
-
top_query_signatures = query_signatures.sort_by { |_, data| -data[:count] }.first(3)
|
|
253
|
-
|
|
254
305
|
# Capture post-block memory state — always, regardless of config.
|
|
255
306
|
gc_after = begin; GC.stat; rescue; {}; end
|
|
256
307
|
memory_after_mb = begin; DeadBro::MemoryHelpers.rss_mb; rescue; memory_before_mb; end
|
|
@@ -287,38 +338,6 @@ module DeadBro
|
|
|
287
338
|
large_objects: large_objects
|
|
288
339
|
)
|
|
289
340
|
|
|
290
|
-
sql_queries_segment = ""
|
|
291
|
-
unless top_query_signatures.empty?
|
|
292
|
-
formatted_queries = top_query_signatures.map do |sig, data|
|
|
293
|
-
type = data[:type] || "SQL"
|
|
294
|
-
count = data[:count]
|
|
295
|
-
total_ms = data[:total_time_ms].round(2)
|
|
296
|
-
"#{type} #{sig} (#{count}x, #{total_ms}ms)"
|
|
297
|
-
end
|
|
298
|
-
sql_queries_segment = ", sql_top_queries=[#{formatted_queries.join(" | ")}]"
|
|
299
|
-
end
|
|
300
|
-
|
|
301
|
-
warnings = detailed_memory_summary[:warnings]
|
|
302
|
-
warnings_segment = warnings.any? ? ", warnings=[#{warnings.join(", ")}]" : ""
|
|
303
|
-
summary = "Analysis for #{label} - total_time=#{total_time_ms}ms, " \
|
|
304
|
-
"sql_queries=#{sql_count}, sql_time=#{sql_time_ms}ms, " \
|
|
305
|
-
"memory_before=#{memory_before_mb.round(2)}MB, " \
|
|
306
|
-
"memory_after=#{memory_after_mb.round(2)}MB, " \
|
|
307
|
-
"memory_delta=#{memory_delta_mb}MB, " \
|
|
308
|
-
"gc_collections=+#{detailed_memory_summary[:gc_collections]}, " \
|
|
309
|
-
"heap_pages_added=+#{detailed_memory_summary[:heap_pages_added]}, " \
|
|
310
|
-
"new_objects=+#{detailed_memory_summary[:new_objects]}" \
|
|
311
|
-
"#{sql_queries_segment}#{warnings_segment}"
|
|
312
|
-
|
|
313
|
-
begin
|
|
314
|
-
DeadBro.logger.info(summary)
|
|
315
|
-
rescue
|
|
316
|
-
begin
|
|
317
|
-
$stdout.puts("[DeadBro] #{summary}")
|
|
318
|
-
rescue
|
|
319
|
-
end
|
|
320
|
-
end
|
|
321
|
-
|
|
322
341
|
# Build structured result hash to return to the caller
|
|
323
342
|
sql_queries_detail = query_signatures.map do |sig, data|
|
|
324
343
|
{
|
|
@@ -329,7 +348,7 @@ module DeadBro
|
|
|
329
348
|
}
|
|
330
349
|
end
|
|
331
350
|
|
|
332
|
-
analysis_result =
|
|
351
|
+
analysis_result = AnalysisResult.new(
|
|
333
352
|
label: label,
|
|
334
353
|
total_time_ms: total_time_ms,
|
|
335
354
|
sql_count: sql_count,
|
|
@@ -340,7 +359,7 @@ module DeadBro
|
|
|
340
359
|
memory_delta_mb: memory_delta_mb,
|
|
341
360
|
memory_details: detailed_memory_summary,
|
|
342
361
|
verbose: verbose
|
|
343
|
-
|
|
362
|
+
)
|
|
344
363
|
end
|
|
345
364
|
|
|
346
365
|
raise error if error
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dead_bro
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.2.
|
|
4
|
+
version: 0.2.12
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Emanuel Comsa
|
|
@@ -34,6 +34,7 @@ files:
|
|
|
34
34
|
- lib/dead_bro/collectors/system.rb
|
|
35
35
|
- lib/dead_bro/configuration.rb
|
|
36
36
|
- lib/dead_bro/dispatcher.rb
|
|
37
|
+
- lib/dead_bro/elasticsearch_subscriber.rb
|
|
37
38
|
- lib/dead_bro/error_middleware.rb
|
|
38
39
|
- lib/dead_bro/http_instrumentation.rb
|
|
39
40
|
- lib/dead_bro/job_sql_tracking_middleware.rb
|
|
@@ -64,7 +65,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
64
65
|
requirements:
|
|
65
66
|
- - ">="
|
|
66
67
|
- !ruby/object:Gem::Version
|
|
67
|
-
version:
|
|
68
|
+
version: 2.7.0
|
|
68
69
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
69
70
|
requirements:
|
|
70
71
|
- - ">="
|