lens-rails 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +85 -0
- data/lib/lens/rails/backoff_loop.rb +39 -0
- data/lib/lens/rails/configuration.rb +34 -0
- data/lib/lens/rails/log_exporter.rb +122 -0
- data/lib/lens/rails/metrics_exporter.rb +153 -0
- data/lib/lens/rails/railtie.rb +82 -0
- data/lib/lens/rails/requests_exporter.rb +253 -0
- data/lib/lens/rails/version.rb +5 -0
- data/lib/lens/rails.rb +36 -0
- data/lib/lens-rails.rb +1 -0
- metadata +50 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: f3feb70c30ca8f978b2de8669531cb66b22e2d4d019b711e1ed4aaf744743006
|
|
4
|
+
data.tar.gz: 44e46c8d29db7456d1b622f97d5c1df6256e142994b32d82059bfb8ca23b33d2
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 0b8ac809f6c4383c19ccac2bcf44870051d936eb9fe7d93f73660f57455493412c16bf6db964d2f074014ce9f7bb9b06d4d7ed13656d83c94d752cee26781e4c
|
|
7
|
+
data.tar.gz: ef5d900a3f546f8428d96bf679f11577871a93c83b335e315d1550c3b5ca0c7ecb353215b461858ea5467eb5a0a61ad3284687edeea6b76946d590f8bb75b939
|
data/README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# lens-rails
|
|
2
|
+
|
|
3
|
+
Zero-config APM integration for [Lens](https://github.com/gustech/lens). Drop this gem into any Rails app and logs, metrics, and request traces start flowing to your Lens instance automatically — no OpenTelemetry SDK, no protobuf, no additional processes.
|
|
4
|
+
|
|
5
|
+
## Installation
|
|
6
|
+
|
|
7
|
+
Add to your Gemfile:
|
|
8
|
+
|
|
9
|
+
```ruby
|
|
10
|
+
gem 'lens-rails'
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Set two environment variables (e.g. in `.env` or your secrets manager):
|
|
14
|
+
|
|
15
|
+
```
|
|
16
|
+
LENS_URL=https://lens.example.com
|
|
17
|
+
LENS_APP_TOKEN=your-app-token
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
That's it. No initializer required.
|
|
21
|
+
|
|
22
|
+
## What gets collected automatically
|
|
23
|
+
|
|
24
|
+
| Signal | Source | Detail |
|
|
25
|
+
|---|---|---|
|
|
26
|
+
| **Logs** | `Rails.logger` | Every log line with severity, body, controller, action, and request ID |
|
|
27
|
+
| **Metrics** | Rack middleware | Request count, error count, mean/p50/p95 duration — flushed every 30 s |
|
|
28
|
+
| **HTTP requests** | `process_action.action_controller` | Controller, action, method, path, status, duration, SQL summary, operation waterfall |
|
|
29
|
+
| **Background jobs** | `perform.active_job` | Job class, queue, duration, SQL summary, operation waterfall |
|
|
30
|
+
|
|
31
|
+
Each HTTP request and job record includes a flat **operation waterfall**: one entry per SQL query (name + SQL text + duration + offset from request start) and per view render.
|
|
32
|
+
|
|
33
|
+
## Configuration
|
|
34
|
+
|
|
35
|
+
To override defaults, add an initializer:
|
|
36
|
+
|
|
37
|
+
```ruby
|
|
38
|
+
# config/initializers/lens.rb
|
|
39
|
+
Lens::Rails.configure do |config|
|
|
40
|
+
config.service_name = "my-app" # defaults to Rails app module name
|
|
41
|
+
config.url = "https://..." # overrides LENS_URL
|
|
42
|
+
config.app_token = "tok_..." # overrides LENS_APP_TOKEN
|
|
43
|
+
|
|
44
|
+
# Only record SQL queries that took at least this long (ms). Default: 0.0 (all queries).
|
|
45
|
+
config.sql_threshold_ms = 5.0
|
|
46
|
+
|
|
47
|
+
# Drop SQL queries whose text matches any of these patterns.
|
|
48
|
+
# Useful for suppressing framework noise from the waterfall.
|
|
49
|
+
config.sql_ignore = [
|
|
50
|
+
/solid_queue/i,
|
|
51
|
+
/\ASELECT.*schema_migrations/i,
|
|
52
|
+
]
|
|
53
|
+
|
|
54
|
+
# Minimum severity to export. Default: Logger::INFO (drops DEBUG-level SQL noise).
|
|
55
|
+
# Set to Logger::WARN to export only warnings and above.
|
|
56
|
+
config.min_log_severity = Logger::INFO
|
|
57
|
+
|
|
58
|
+
# Ring-buffer size for log records (drop-oldest when full). Default: 10_000.
|
|
59
|
+
config.max_log_buffer = 10_000
|
|
60
|
+
|
|
61
|
+
# HTTP timeout for each exporter flush call (seconds). Default: 2.
|
|
62
|
+
config.exporter_open_timeout = 2
|
|
63
|
+
config.exporter_read_timeout = 2
|
|
64
|
+
|
|
65
|
+
# Maximum time to wait for a final flush on process shutdown (seconds). Default: 5.
|
|
66
|
+
config.shutdown_timeout = 5
|
|
67
|
+
end
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## How it works
|
|
71
|
+
|
|
72
|
+
The gem wires up three components via a Railtie:
|
|
73
|
+
|
|
74
|
+
- **`LogExporter`** — broadcasts into `Rails.logger` and batches records to `POST /v1/logs` every 5 s.
|
|
75
|
+
- **`MetricsExporter`** — Rack middleware that tracks request durations via reservoir sampling and flushes to `POST /v1/metrics` every 30 s.
|
|
76
|
+
- **`RequestsExporter`** — subscribes to `ActiveSupport::Notifications` (`process_action`, `sql.active_record`, `render_template`, `perform.active_job`) and flushes request and job records to `POST /v1/requests` every 30 s.
|
|
77
|
+
|
|
78
|
+
All payloads are gzip-compressed JSON. Background flush threads restart automatically after a Puma or Unicorn fork.
|
|
79
|
+
|
|
80
|
+
SolidQueue dispatcher jobs are filtered out of the job waterfall automatically.
|
|
81
|
+
|
|
82
|
+
## Requirements
|
|
83
|
+
|
|
84
|
+
- Ruby >= 3.1
|
|
85
|
+
- Rails >= 7.1
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Lens
|
|
4
|
+
module Rails
|
|
5
|
+
# Supervised sleep/call loop for flush threads.
|
|
6
|
+
#
|
|
7
|
+
# - Rescues StandardError (not Exception — we must not swallow Interrupt /
|
|
8
|
+
# SystemExit, which would defeat the bounded at_exit flush).
|
|
9
|
+
# - On failure, doubles the sleep interval up to `max`. Resets to `base` on
|
|
10
|
+
# the next successful call.
|
|
11
|
+
# - Sets a thread name and enables report_on_exception so a truly fatal
|
|
12
|
+
# error is visible instead of silently killing telemetry.
|
|
13
|
+
class BackoffLoop
|
|
14
|
+
def initialize(base:, max: 60, name: "lens-loop")
|
|
15
|
+
@base = base
|
|
16
|
+
@max = max
|
|
17
|
+
@name = name
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def start(&block)
|
|
21
|
+
Thread.new do
|
|
22
|
+
Thread.current.name = @name
|
|
23
|
+
Thread.current.report_on_exception = true
|
|
24
|
+
current = @base
|
|
25
|
+
loop do
|
|
26
|
+
sleep(current)
|
|
27
|
+
begin
|
|
28
|
+
block.call
|
|
29
|
+
current = @base
|
|
30
|
+
rescue => e
|
|
31
|
+
warn "#{@name}: #{e.class}: #{e.message}"
|
|
32
|
+
current = [current * 2, @max].min
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
module Lens
|
|
2
|
+
module Rails
|
|
3
|
+
class Configuration
|
|
4
|
+
attr_accessor :url, :app_token, :service_name,
|
|
5
|
+
:max_log_buffer, :min_log_severity,
|
|
6
|
+
:sql_threshold_ms, :sql_ignore,
|
|
7
|
+
:exporter_open_timeout, :exporter_read_timeout,
|
|
8
|
+
:shutdown_timeout
|
|
9
|
+
|
|
10
|
+
def initialize
|
|
11
|
+
@url = ENV["LENS_URL"]
|
|
12
|
+
@app_token = ENV["LENS_APP_TOKEN"]
|
|
13
|
+
@service_name = ENV.fetch("LENS_SERVICE_NAME") { default_service_name }
|
|
14
|
+
@max_log_buffer = 10_000
|
|
15
|
+
@min_log_severity = ::Logger::INFO
|
|
16
|
+
@sql_threshold_ms = 0.0
|
|
17
|
+
@sql_ignore = []
|
|
18
|
+
@exporter_open_timeout = 2
|
|
19
|
+
@exporter_read_timeout = 2
|
|
20
|
+
@shutdown_timeout = 5
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def configured?
|
|
24
|
+
!url.to_s.empty? && !app_token.to_s.empty?
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def default_service_name
|
|
30
|
+
defined?(::Rails) ? ::Rails.application.class.module_parent_name.underscore : "app"
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "json"
|
|
3
|
+
require "zlib"
|
|
4
|
+
require "lens/rails/backoff_loop"
|
|
5
|
+
|
|
6
|
+
module Lens
|
|
7
|
+
module Rails
|
|
8
|
+
# Logger sink that batches log records and ships them to Lens via POST /v1/logs.
|
|
9
|
+
# Wired into Rails.logger via broadcast in the Railtie.
|
|
10
|
+
class LogExporter < ::Logger
|
|
11
|
+
SEVERITY_TEXT = {
|
|
12
|
+
::Logger::DEBUG => "DEBUG",
|
|
13
|
+
::Logger::INFO => "INFO",
|
|
14
|
+
::Logger::WARN => "WARN",
|
|
15
|
+
::Logger::ERROR => "ERROR",
|
|
16
|
+
::Logger::FATAL => "FATAL",
|
|
17
|
+
::Logger::UNKNOWN => "UNKNOWN"
|
|
18
|
+
}.freeze
|
|
19
|
+
|
|
20
|
+
DROP_WARN_INTERVAL = 60.0
|
|
21
|
+
|
|
22
|
+
def initialize(url:, token:, service_name:,
|
|
23
|
+
max_buffer: 10_000,
|
|
24
|
+
min_severity: ::Logger::INFO,
|
|
25
|
+
open_timeout: 2,
|
|
26
|
+
read_timeout: 2)
|
|
27
|
+
super(File::NULL)
|
|
28
|
+
@token = token
|
|
29
|
+
@service_name = service_name
|
|
30
|
+
@uri = URI("#{url}/v1/logs")
|
|
31
|
+
@max_buffer = max_buffer
|
|
32
|
+
@open_timeout = open_timeout
|
|
33
|
+
@read_timeout = read_timeout
|
|
34
|
+
@min_severity = min_severity
|
|
35
|
+
@buffer = []
|
|
36
|
+
@dropped = 0
|
|
37
|
+
@last_drop_warn_at = 0.0
|
|
38
|
+
@mutex = Mutex.new
|
|
39
|
+
restart_flush_thread
|
|
40
|
+
Lens::Rails.register_flushable(self)
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def add(severity, message = nil, progname = nil, &block)
|
|
44
|
+
return true if Thread.current[:lens_in_export]
|
|
45
|
+
|
|
46
|
+
severity ||= ::Logger::UNKNOWN
|
|
47
|
+
return true if severity < @min_severity
|
|
48
|
+
|
|
49
|
+
message = block ? block.call : progname if message.nil?
|
|
50
|
+
return true if message.nil?
|
|
51
|
+
|
|
52
|
+
record = {
|
|
53
|
+
timestamp: Time.now.iso8601(3),
|
|
54
|
+
severity: SEVERITY_TEXT[severity] || "UNKNOWN",
|
|
55
|
+
body: message.to_s.gsub(/\e\[[0-9;]*m/, ""),
|
|
56
|
+
controller: Thread.current[:lens_controller],
|
|
57
|
+
action: Thread.current[:lens_action],
|
|
58
|
+
request_id: Thread.current[:lens_request_id]
|
|
59
|
+
}.compact
|
|
60
|
+
|
|
61
|
+
dropped_now = 0
|
|
62
|
+
@mutex.synchronize do
|
|
63
|
+
if @buffer.size >= @max_buffer
|
|
64
|
+
@buffer.shift
|
|
65
|
+
@dropped += 1
|
|
66
|
+
dropped_now = @dropped
|
|
67
|
+
end
|
|
68
|
+
@buffer << record
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
maybe_warn_dropped(dropped_now) if dropped_now.positive?
|
|
72
|
+
true
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def flush
|
|
76
|
+
records = @mutex.synchronize { @buffer.tap { @buffer = [] } }
|
|
77
|
+
return if records.empty?
|
|
78
|
+
|
|
79
|
+
body = JSON.generate(logs: records, service: @service_name, version: Lens::Rails::VERSION)
|
|
80
|
+
|
|
81
|
+
req = Net::HTTP::Post.new(@uri)
|
|
82
|
+
req["Content-Type"] = "application/octet-stream"
|
|
83
|
+
req["Authorization"] = "Bearer #{@token}"
|
|
84
|
+
req.body = Zlib.gzip(body)
|
|
85
|
+
|
|
86
|
+
Thread.current[:lens_in_export] = true
|
|
87
|
+
begin
|
|
88
|
+
Net::HTTP.start(@uri.host, @uri.port,
|
|
89
|
+
use_ssl: @uri.scheme == "https",
|
|
90
|
+
open_timeout: @open_timeout,
|
|
91
|
+
read_timeout: @read_timeout) { |http| http.request(req) }
|
|
92
|
+
ensure
|
|
93
|
+
Thread.current[:lens_in_export] = false
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
def shutdown(timeout:)
|
|
98
|
+
Thread.new { flush }.join(timeout)
|
|
99
|
+
rescue
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def restart_flush_thread
|
|
103
|
+
BackoffLoop.new(base: 5, max: 60, name: "lens-log-flush").start { flush }
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
private
|
|
107
|
+
|
|
108
|
+
def maybe_warn_dropped(total_dropped)
|
|
109
|
+
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
110
|
+
should_warn = @mutex.synchronize do
|
|
111
|
+
if now - @last_drop_warn_at >= DROP_WARN_INTERVAL
|
|
112
|
+
@last_drop_warn_at = now
|
|
113
|
+
true
|
|
114
|
+
else
|
|
115
|
+
false
|
|
116
|
+
end
|
|
117
|
+
end
|
|
118
|
+
Kernel.warn "Lens log buffer full: #{total_dropped} records dropped" if should_warn
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|
|
122
|
+
end
|
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "json"
|
|
3
|
+
require "zlib"
|
|
4
|
+
require "lens/rails/backoff_loop"
|
|
5
|
+
|
|
6
|
+
module Lens
|
|
7
|
+
module Rails
|
|
8
|
+
# Rack middleware that tracks request counts and durations,
|
|
9
|
+
# flushing them to Lens every 30 s via POST /v1/metrics.
|
|
10
|
+
class MetricsExporter
|
|
11
|
+
RESERVOIR_SIZE = 1024
|
|
12
|
+
|
|
13
|
+
def initialize(app, url:, token:, service_name:,
|
|
14
|
+
open_timeout: 2,
|
|
15
|
+
read_timeout: 2)
|
|
16
|
+
@app = app
|
|
17
|
+
@token = token
|
|
18
|
+
@service_name = service_name
|
|
19
|
+
@uri = URI("#{url}/v1/metrics")
|
|
20
|
+
@open_timeout = open_timeout
|
|
21
|
+
@read_timeout = read_timeout
|
|
22
|
+
@mutex = Mutex.new
|
|
23
|
+
reset_window
|
|
24
|
+
restart_flush_thread
|
|
25
|
+
Lens::Rails.register_flushable(self)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def call(env)
|
|
29
|
+
Thread.current[:lens_request_id] = env["action_dispatch.request_id"]
|
|
30
|
+
wall_start = Time.now
|
|
31
|
+
start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
32
|
+
status, headers, body = @app.call(env)
|
|
33
|
+
duration = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000.0
|
|
34
|
+
queue_time = parse_queue_time(env, wall_start)
|
|
35
|
+
|
|
36
|
+
@mutex.synchronize do
|
|
37
|
+
@request_count += 1
|
|
38
|
+
@error_count += 1 if status >= 500
|
|
39
|
+
@duration_sum += duration
|
|
40
|
+
@duration_min = duration if @duration_min.nil? || duration < @duration_min
|
|
41
|
+
@duration_max = duration if @duration_max.nil? || duration > @duration_max
|
|
42
|
+
if @samples.size < RESERVOIR_SIZE
|
|
43
|
+
@samples << duration
|
|
44
|
+
else
|
|
45
|
+
idx = rand(@request_count)
|
|
46
|
+
@samples[idx] = duration if idx < RESERVOIR_SIZE
|
|
47
|
+
end
|
|
48
|
+
if queue_time
|
|
49
|
+
@queue_samples << queue_time
|
|
50
|
+
@queue_sum += queue_time
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
[status, headers, body]
|
|
55
|
+
ensure
|
|
56
|
+
Thread.current[:lens_request_id] = nil
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def flush
|
|
60
|
+
snapshot = @mutex.synchronize do
|
|
61
|
+
s = {count: @request_count, errors: @error_count,
|
|
62
|
+
sum: @duration_sum, min: @duration_min, max: @duration_max,
|
|
63
|
+
samples: @samples, queue_samples: @queue_samples, queue_sum: @queue_sum}
|
|
64
|
+
reset_window
|
|
65
|
+
s
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
return if snapshot[:count].zero?
|
|
69
|
+
|
|
70
|
+
now = Time.now.iso8601(3)
|
|
71
|
+
avg = snapshot[:sum] / snapshot[:count]
|
|
72
|
+
p50 = percentile(snapshot[:samples], 0.50)
|
|
73
|
+
p95 = percentile(snapshot[:samples], 0.95)
|
|
74
|
+
|
|
75
|
+
metrics = [
|
|
76
|
+
{name: "http.server.request_count", unit: "1", value: snapshot[:count].to_f, timestamp: now},
|
|
77
|
+
{name: "http.server.error_count", unit: "1", value: snapshot[:errors].to_f, timestamp: now},
|
|
78
|
+
{name: "http.server.duration", unit: "ms", value: avg, timestamp: now},
|
|
79
|
+
{name: "http.server.duration.p50", unit: "ms", value: p50, timestamp: now},
|
|
80
|
+
{name: "http.server.duration.p95", unit: "ms", value: p95, timestamp: now}
|
|
81
|
+
]
|
|
82
|
+
|
|
83
|
+
if snapshot[:queue_samples].any?
|
|
84
|
+
q_avg = snapshot[:queue_sum] / snapshot[:queue_samples].size
|
|
85
|
+
q_p50 = percentile(snapshot[:queue_samples], 0.50)
|
|
86
|
+
q_p95 = percentile(snapshot[:queue_samples], 0.95)
|
|
87
|
+
metrics += [
|
|
88
|
+
{name: "http.server.queue_time", unit: "ms", value: q_avg, timestamp: now},
|
|
89
|
+
{name: "http.server.queue_time.p50", unit: "ms", value: q_p50, timestamp: now},
|
|
90
|
+
{name: "http.server.queue_time.p95", unit: "ms", value: q_p95, timestamp: now}
|
|
91
|
+
]
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
body = JSON.generate(metrics: metrics, service: @service_name, version: Lens::Rails::VERSION)
|
|
95
|
+
|
|
96
|
+
req = Net::HTTP::Post.new(@uri)
|
|
97
|
+
req["Content-Type"] = "application/octet-stream"
|
|
98
|
+
req["Authorization"] = "Bearer #{@token}"
|
|
99
|
+
req.body = Zlib.gzip(body)
|
|
100
|
+
|
|
101
|
+
Thread.current[:lens_in_export] = true
|
|
102
|
+
begin
|
|
103
|
+
Net::HTTP.start(@uri.host, @uri.port,
|
|
104
|
+
use_ssl: @uri.scheme == "https",
|
|
105
|
+
open_timeout: @open_timeout,
|
|
106
|
+
read_timeout: @read_timeout) { |http| http.request(req) }
|
|
107
|
+
ensure
|
|
108
|
+
Thread.current[:lens_in_export] = false
|
|
109
|
+
end
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
def shutdown(timeout:)
|
|
113
|
+
Thread.new { flush }.join(timeout)
|
|
114
|
+
rescue
|
|
115
|
+
end
|
|
116
|
+
|
|
117
|
+
def restart_flush_thread
|
|
118
|
+
BackoffLoop.new(base: 30, max: 120, name: "lens-metrics-flush").start { flush }
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
private
|
|
122
|
+
|
|
123
|
+
def reset_window
|
|
124
|
+
@request_count = 0
|
|
125
|
+
@error_count = 0
|
|
126
|
+
@duration_sum = 0.0
|
|
127
|
+
@duration_min = nil
|
|
128
|
+
@duration_max = nil
|
|
129
|
+
@samples = []
|
|
130
|
+
@queue_samples = []
|
|
131
|
+
@queue_sum = 0.0
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
def parse_queue_time(env, wall_start)
|
|
135
|
+
raw = env["HTTP_X_REQUEST_START"]
|
|
136
|
+
return nil unless raw
|
|
137
|
+
val = raw.delete_prefix("t=").to_f
|
|
138
|
+
return nil if val.zero?
|
|
139
|
+
proxy_time = (val > 1e12) ? Time.at(val / 1000.0) : Time.at(val)
|
|
140
|
+
[(wall_start - proxy_time) * 1000.0, 0.0].max.round(2)
|
|
141
|
+
rescue
|
|
142
|
+
nil
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
def percentile(samples, p)
|
|
146
|
+
return 0.0 if samples.empty?
|
|
147
|
+
sorted = samples.sort
|
|
148
|
+
idx = (p * (sorted.size - 1)).round
|
|
149
|
+
sorted[idx]
|
|
150
|
+
end
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
end
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
require "lens/rails/log_exporter"
|
|
2
|
+
require "lens/rails/metrics_exporter"
|
|
3
|
+
require "lens/rails/requests_exporter"
|
|
4
|
+
|
|
5
|
+
module Lens
|
|
6
|
+
module Rails
|
|
7
|
+
class Railtie < ::Rails::Railtie
|
|
8
|
+
# Broadcast Rails.logger to Lens.
|
|
9
|
+
initializer "lens.configure_logging", after: :load_config_initializers do
|
|
10
|
+
cfg = Lens::Rails.configuration
|
|
11
|
+
next unless cfg.configured?
|
|
12
|
+
|
|
13
|
+
::Rails.logger.broadcast_to(
|
|
14
|
+
Lens::Rails::LogExporter.new(
|
|
15
|
+
url: cfg.url,
|
|
16
|
+
token: cfg.app_token,
|
|
17
|
+
service_name: cfg.service_name,
|
|
18
|
+
max_buffer: cfg.max_log_buffer,
|
|
19
|
+
min_severity: cfg.min_log_severity,
|
|
20
|
+
open_timeout: cfg.exporter_open_timeout,
|
|
21
|
+
read_timeout: cfg.exporter_read_timeout
|
|
22
|
+
)
|
|
23
|
+
)
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# Rack middleware for request metrics.
|
|
27
|
+
initializer "lens.configure_metrics", after: :load_config_initializers do
|
|
28
|
+
cfg = Lens::Rails.configuration
|
|
29
|
+
next unless cfg.configured?
|
|
30
|
+
|
|
31
|
+
::Rails.application.middleware.use(
|
|
32
|
+
Lens::Rails::MetricsExporter,
|
|
33
|
+
url: cfg.url,
|
|
34
|
+
token: cfg.app_token,
|
|
35
|
+
service_name: cfg.service_name,
|
|
36
|
+
open_timeout: cfg.exporter_open_timeout,
|
|
37
|
+
read_timeout: cfg.exporter_read_timeout
|
|
38
|
+
)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Rack middleware + AS notification subscriptions for request/job traces.
|
|
42
|
+
initializer "lens.configure_requests", after: :load_config_initializers do
|
|
43
|
+
cfg = Lens::Rails.configuration
|
|
44
|
+
next unless cfg.configured?
|
|
45
|
+
|
|
46
|
+
::Rails.application.middleware.use(Lens::Rails::RequestScopeMiddleware)
|
|
47
|
+
|
|
48
|
+
ActiveSupport.on_load(:active_job) do
|
|
49
|
+
include Lens::Rails::JobTracking
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
Lens::Rails::RequestsExporter.new(
|
|
53
|
+
url: cfg.url,
|
|
54
|
+
token: cfg.app_token,
|
|
55
|
+
service_name: cfg.service_name,
|
|
56
|
+
sql_threshold_ms: cfg.sql_threshold_ms,
|
|
57
|
+
sql_ignore: cfg.sql_ignore,
|
|
58
|
+
open_timeout: cfg.exporter_open_timeout,
|
|
59
|
+
read_timeout: cfg.exporter_read_timeout
|
|
60
|
+
)
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Flush pending data on clean shutdown, bounded by shutdown_timeout.
|
|
64
|
+
initializer "lens.at_exit", after: :load_config_initializers do
|
|
65
|
+
cfg = Lens::Rails.configuration
|
|
66
|
+
next unless cfg.configured?
|
|
67
|
+
|
|
68
|
+
at_exit { Lens::Rails.shutdown_flushables(timeout: cfg.shutdown_timeout) }
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# Restart background flush threads after a Puma/Unicorn fork.
|
|
72
|
+
initializer "lens.after_fork" do
|
|
73
|
+
cfg = Lens::Rails.configuration
|
|
74
|
+
next unless cfg.configured? && ::Rails.application.config.respond_to?(:after_fork)
|
|
75
|
+
|
|
76
|
+
::Rails.application.config.after_fork do
|
|
77
|
+
Lens::Rails.restart_flush_threads
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|
|
@@ -0,0 +1,253 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "json"
|
|
3
|
+
require "zlib"
|
|
4
|
+
require "lens/rails/backoff_loop"
|
|
5
|
+
|
|
6
|
+
module Lens
|
|
7
|
+
module Rails
|
|
8
|
+
# Tiny Rack middleware that opens a per-request operation accumulator so SQL
|
|
9
|
+
# and view notifications can attach to the right request on the current thread.
|
|
10
|
+
class RequestScopeMiddleware
|
|
11
|
+
def initialize(app)
|
|
12
|
+
@app = app
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def call(env)
|
|
16
|
+
wall_start = Time.now
|
|
17
|
+
Thread.current[:lens_request_ops] = []
|
|
18
|
+
Thread.current[:lens_request_start] = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
19
|
+
Thread.current[:lens_request_wall_start] = wall_start
|
|
20
|
+
Thread.current[:lens_queue_time_ms] = parse_queue_time(env, wall_start)
|
|
21
|
+
@app.call(env)
|
|
22
|
+
ensure
|
|
23
|
+
Thread.current[:lens_request_ops] = nil
|
|
24
|
+
Thread.current[:lens_request_start] = nil
|
|
25
|
+
Thread.current[:lens_request_wall_start] = nil
|
|
26
|
+
Thread.current[:lens_queue_time_ms] = nil
|
|
27
|
+
Thread.current[:lens_controller] = nil
|
|
28
|
+
Thread.current[:lens_action] = nil
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
private
|
|
32
|
+
|
|
33
|
+
# Parses X-Request-Start set by kamal-proxy and nginx.
|
|
34
|
+
# Formats: "t=1234567890.123456" (seconds) or "1234567890123" (milliseconds).
|
|
35
|
+
def parse_queue_time(env, wall_start)
|
|
36
|
+
raw = env["HTTP_X_REQUEST_START"]
|
|
37
|
+
return nil unless raw
|
|
38
|
+
val = raw.delete_prefix("t=").to_f
|
|
39
|
+
return nil if val.zero?
|
|
40
|
+
proxy_time = (val > 1e12) ? Time.at(val / 1000.0) : Time.at(val)
|
|
41
|
+
[(wall_start - proxy_time) * 1000.0, 0.0].max.round(2)
|
|
42
|
+
rescue
|
|
43
|
+
nil
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
# ActiveJob concern injected via on_load(:active_job) in the Railtie.
|
|
48
|
+
# Opens a per-job operation accumulator around every perform call.
|
|
49
|
+
module JobTracking
|
|
50
|
+
extend ActiveSupport::Concern
|
|
51
|
+
|
|
52
|
+
included do
|
|
53
|
+
around_perform do |_job, block|
|
|
54
|
+
Thread.current[:lens_job_ops] = []
|
|
55
|
+
Thread.current[:lens_job_start] = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
56
|
+
Thread.current[:lens_job_wall_start] = Time.now
|
|
57
|
+
block.call
|
|
58
|
+
ensure
|
|
59
|
+
Thread.current[:lens_job_ops] = nil
|
|
60
|
+
Thread.current[:lens_job_start] = nil
|
|
61
|
+
Thread.current[:lens_job_wall_start] = nil
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# Subscribes to ActionController, ActionView, ActiveRecord, and ActiveJob
|
|
67
|
+
# notifications and flushes collected records to POST /v1/requests.
|
|
68
|
+
class RequestsExporter
|
|
69
|
+
SKIP_SCHEMA = /\A(SCHEMA|ActiveRecord::SchemaMigration|ActiveRecord::InternalMetadata)\z/
|
|
70
|
+
SKIP_JOB_CLASS = /\ASolidQueue::/
|
|
71
|
+
|
|
72
|
+
def initialize(url:, token:, service_name:,
|
|
73
|
+
sql_threshold_ms: 0.0, sql_ignore: [],
|
|
74
|
+
open_timeout: 2, read_timeout: 2)
|
|
75
|
+
@uri = URI("#{url}/v1/requests")
|
|
76
|
+
@token = token
|
|
77
|
+
@service_name = service_name
|
|
78
|
+
@sql_threshold_ms = sql_threshold_ms
|
|
79
|
+
@sql_ignore = sql_ignore
|
|
80
|
+
@open_timeout = open_timeout
|
|
81
|
+
@read_timeout = read_timeout
|
|
82
|
+
@mutex = Mutex.new
|
|
83
|
+
@requests = []
|
|
84
|
+
@jobs = []
|
|
85
|
+
subscribe!
|
|
86
|
+
restart_flush_thread
|
|
87
|
+
Lens::Rails.register_flushable(self)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def flush
|
|
91
|
+
requests, jobs = @mutex.synchronize do
|
|
92
|
+
[@requests.tap { @requests = [] }, @jobs.tap { @jobs = [] }]
|
|
93
|
+
end
|
|
94
|
+
return if requests.empty? && jobs.empty?
|
|
95
|
+
|
|
96
|
+
body = JSON.generate(requests: requests, jobs: jobs,
|
|
97
|
+
service: @service_name, version: Lens::Rails::VERSION)
|
|
98
|
+
|
|
99
|
+
req = Net::HTTP::Post.new(@uri)
|
|
100
|
+
req["Content-Type"] = "application/octet-stream"
|
|
101
|
+
req["Authorization"] = "Bearer #{@token}"
|
|
102
|
+
req.body = Zlib.gzip(body)
|
|
103
|
+
|
|
104
|
+
Thread.current[:lens_in_export] = true
|
|
105
|
+
begin
|
|
106
|
+
Net::HTTP.start(@uri.host, @uri.port,
|
|
107
|
+
use_ssl: @uri.scheme == "https",
|
|
108
|
+
open_timeout: @open_timeout,
|
|
109
|
+
read_timeout: @read_timeout) { |http| http.request(req) }
|
|
110
|
+
ensure
|
|
111
|
+
Thread.current[:lens_in_export] = false
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def shutdown(timeout:)
|
|
116
|
+
Thread.new { flush }.join(timeout)
|
|
117
|
+
rescue
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
def restart_flush_thread
|
|
121
|
+
BackoffLoop.new(base: 30, max: 120, name: "lens-requests-flush").start { flush }
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
private
|
|
125
|
+
|
|
126
|
+
def subscribe!
|
|
127
|
+
# Tag current thread with controller/action for the log exporter to pick up.
|
|
128
|
+
ActiveSupport::Notifications.subscribe("start_processing.action_controller") do |event|
|
|
129
|
+
next if Thread.current[:lens_in_export]
|
|
130
|
+
Thread.current[:lens_controller] = event.payload[:controller]
|
|
131
|
+
Thread.current[:lens_action] = event.payload[:action]
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
# Accumulate SQL operations onto the current request or job.
|
|
135
|
+
ActiveSupport::Notifications.subscribe("sql.active_record") do |event|
|
|
136
|
+
next if Thread.current[:lens_in_export]
|
|
137
|
+
record_sql(event)
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
# Accumulate view renders onto the current request.
|
|
141
|
+
ActiveSupport::Notifications.subscribe("render_template.action_view") do |event|
|
|
142
|
+
next if Thread.current[:lens_in_export]
|
|
143
|
+
record_view(event)
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
# Finalize an HTTP request record.
|
|
147
|
+
ActiveSupport::Notifications.subscribe("process_action.action_controller") do |event|
|
|
148
|
+
next if Thread.current[:lens_in_export]
|
|
149
|
+
record_request(event)
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# Finalize a background job record.
|
|
153
|
+
ActiveSupport::Notifications.subscribe("perform.active_job") do |event|
|
|
154
|
+
next if Thread.current[:lens_in_export]
|
|
155
|
+
record_job(event)
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def record_sql(event)
|
|
160
|
+
ops = Thread.current[:lens_request_ops] || Thread.current[:lens_job_ops]
|
|
161
|
+
t0 = Thread.current[:lens_request_start] || Thread.current[:lens_job_start]
|
|
162
|
+
return unless ops && t0
|
|
163
|
+
|
|
164
|
+
name = event.payload[:name].to_s
|
|
165
|
+
return if SKIP_SCHEMA.match?(name)
|
|
166
|
+
|
|
167
|
+
duration = event.duration.round(2)
|
|
168
|
+
return if duration < @sql_threshold_ms
|
|
169
|
+
|
|
170
|
+
sql = event.payload[:sql].to_s
|
|
171
|
+
return if @sql_ignore.any? { |re| sql.match?(re) }
|
|
172
|
+
|
|
173
|
+
offset = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t0) * 1000 - duration).round(2).clamp(0.0, Float::INFINITY)
|
|
174
|
+
ops << {"type" => "sql", "name" => name, "duration_ms" => duration, "offset_ms" => offset, "sql" => sql}
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
def record_view(event)
|
|
178
|
+
ops = Thread.current[:lens_request_ops]
|
|
179
|
+
t0 = Thread.current[:lens_request_start]
|
|
180
|
+
return unless ops && t0
|
|
181
|
+
|
|
182
|
+
name = event.payload[:identifier].to_s.sub(%r{.*/app/views/}, "")
|
|
183
|
+
duration = event.duration.round(2)
|
|
184
|
+
offset = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - t0) * 1000 - duration).round(2).clamp(0.0, Float::INFINITY)
|
|
185
|
+
ops << {"type" => "view", "name" => name, "duration_ms" => duration, "offset_ms" => offset}
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
def record_request(event)
|
|
189
|
+
ops = Thread.current[:lens_request_ops] || []
|
|
190
|
+
Thread.current[:lens_request_ops] = nil
|
|
191
|
+
|
|
192
|
+
payload = event.payload
|
|
193
|
+
sql_ops = ops.select { |op| op["type"] == "sql" }
|
|
194
|
+
view_ops = ops.select { |op| op["type"] == "view" }
|
|
195
|
+
|
|
196
|
+
rec = {
|
|
197
|
+
"id" => Thread.current[:lens_request_id],
|
|
198
|
+
"controller" => payload[:controller],
|
|
199
|
+
"action" => payload[:action],
|
|
200
|
+
"method" => payload[:method],
|
|
201
|
+
"path" => payload[:path],
|
|
202
|
+
"status" => payload[:status],
|
|
203
|
+
"duration_ms" => event.duration.round(2),
|
|
204
|
+
"sql_count" => sql_ops.size,
|
|
205
|
+
"sql_duration_ms" => sql_ops.sum { |op| op["duration_ms"] }.round(2),
|
|
206
|
+
"view_duration_ms" => view_ops.sum { |op| op["duration_ms"] }.round(2),
|
|
207
|
+
"queue_time_ms" => Thread.current[:lens_queue_time_ms],
|
|
208
|
+
"started_at" => (Thread.current[:lens_request_wall_start] || Time.now).iso8601(3),
|
|
209
|
+
"error" => build_error(payload[:exception], payload[:exception_object]),
|
|
210
|
+
"operations" => ops
|
|
211
|
+
}.compact
|
|
212
|
+
|
|
213
|
+
@mutex.synchronize { @requests << rec }
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
def record_job(event)
|
|
217
|
+
ops = Thread.current[:lens_job_ops] || []
|
|
218
|
+
Thread.current[:lens_job_ops] = nil
|
|
219
|
+
|
|
220
|
+
payload = event.payload
|
|
221
|
+
job = payload[:job]
|
|
222
|
+
return if job.class.name&.match?(SKIP_JOB_CLASS)
|
|
223
|
+
|
|
224
|
+
wall_start = Thread.current[:lens_job_wall_start] || Time.now
|
|
225
|
+
sql_ops = ops.select { |op| op["type"] == "sql" }
|
|
226
|
+
enqueued = job.respond_to?(:enqueued_at) && job.enqueued_at
|
|
227
|
+
latency_ms = enqueued ? ((wall_start - enqueued) * 1000).round(2) : nil
|
|
228
|
+
|
|
229
|
+
rec = {
|
|
230
|
+
"id" => job.job_id,
|
|
231
|
+
"job_class" => job.class.name,
|
|
232
|
+
"queue" => job.queue_name,
|
|
233
|
+
"duration_ms" => event.duration.round(2),
|
|
234
|
+
"latency_ms" => latency_ms,
|
|
235
|
+
"sql_count" => sql_ops.size,
|
|
236
|
+
"sql_duration_ms" => sql_ops.sum { |op| op["duration_ms"] }.round(2),
|
|
237
|
+
"started_at" => wall_start.iso8601(3),
|
|
238
|
+
"error" => build_error(payload[:exception], payload[:exception_object]),
|
|
239
|
+
"operations" => ops
|
|
240
|
+
}.compact
|
|
241
|
+
|
|
242
|
+
@mutex.synchronize { @jobs << rec }
|
|
243
|
+
end
|
|
244
|
+
|
|
245
|
+
def build_error(exception, exception_object)
|
|
246
|
+
return nil unless exception
|
|
247
|
+
klass, message = exception
|
|
248
|
+
backtrace = exception_object&.backtrace&.first(20) || []
|
|
249
|
+
{"class" => klass, "message" => message, "backtrace" => backtrace}
|
|
250
|
+
end
|
|
251
|
+
end
|
|
252
|
+
end
|
|
253
|
+
end
|
data/lib/lens/rails.rb
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
require "lens/rails/version"
|
|
2
|
+
require "lens/rails/configuration"
|
|
3
|
+
|
|
4
|
+
module Lens
|
|
5
|
+
module Rails
|
|
6
|
+
class << self
|
|
7
|
+
def configuration
|
|
8
|
+
@configuration ||= Configuration.new
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def configure
|
|
12
|
+
yield configuration
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def register_flushable(exporter)
|
|
16
|
+
flushables << exporter
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def restart_flush_threads
|
|
20
|
+
flushables.each(&:restart_flush_thread)
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def shutdown_flushables(timeout:)
|
|
24
|
+
flushables.each { |f| f.shutdown(timeout: timeout) }
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def flushables
|
|
30
|
+
@flushables ||= []
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
require "lens/rails/railtie" if defined?(::Rails::Railtie)
|
data/lib/lens-rails.rb
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
require "lens/rails"
|
metadata
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: lens-rails
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Axel Gustav
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies: []
|
|
12
|
+
description: Drop-in Rails gem that collects logs, metrics, and request traces via
|
|
13
|
+
ActiveSupport::Notifications and ships them to a Lens instance. Uses gzip-compressed
|
|
14
|
+
JSON — no protobuf, no OpenTelemetry SDK required.
|
|
15
|
+
executables: []
|
|
16
|
+
extensions: []
|
|
17
|
+
extra_rdoc_files: []
|
|
18
|
+
files:
|
|
19
|
+
- README.md
|
|
20
|
+
- lib/lens-rails.rb
|
|
21
|
+
- lib/lens/rails.rb
|
|
22
|
+
- lib/lens/rails/backoff_loop.rb
|
|
23
|
+
- lib/lens/rails/configuration.rb
|
|
24
|
+
- lib/lens/rails/log_exporter.rb
|
|
25
|
+
- lib/lens/rails/metrics_exporter.rb
|
|
26
|
+
- lib/lens/rails/railtie.rb
|
|
27
|
+
- lib/lens/rails/requests_exporter.rb
|
|
28
|
+
- lib/lens/rails/version.rb
|
|
29
|
+
homepage: https://codefloe.com/GusTech/lens-rails
|
|
30
|
+
licenses:
|
|
31
|
+
- MIT
|
|
32
|
+
metadata: {}
|
|
33
|
+
rdoc_options: []
|
|
34
|
+
require_paths:
|
|
35
|
+
- lib
|
|
36
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
37
|
+
requirements:
|
|
38
|
+
- - ">="
|
|
39
|
+
- !ruby/object:Gem::Version
|
|
40
|
+
version: '3.1'
|
|
41
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
42
|
+
requirements:
|
|
43
|
+
- - ">="
|
|
44
|
+
- !ruby/object:Gem::Version
|
|
45
|
+
version: '0'
|
|
46
|
+
requirements: []
|
|
47
|
+
rubygems_version: 3.6.9
|
|
48
|
+
specification_version: 4
|
|
49
|
+
summary: Zero-config APM integration for Lens
|
|
50
|
+
test_files: []
|