packrat-rails 0.3.1 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +50 -7
- data/lib/packrat/config.rb +1 -1
- data/lib/packrat/otel/query_names.rb +44 -0
- data/lib/packrat/otel/scrubber.rb +18 -5
- data/lib/packrat/otel.rb +7 -6
- data/lib/packrat/version.rb +1 -1
- data/lib/packrat.rb +1 -0
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: eff67491572004ed16bdf75327dfdd9e7bcb449f8ab9e0ea62739b4c8bcb1cb8
|
|
4
|
+
data.tar.gz: 5a22cd57c3948731f30764e372b7b2abbc5dcbe730581f15127e1d8e43b676e3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b86a54aa7f48bfc4bef55707bbd2e3d8767bc25be922bf4f64073bc4945514b924f4ed833f75b63e3a1490b2540b4d2550051e94c4acdb2243e83fbe7a15aff8
|
|
7
|
+
data.tar.gz: 8cff66439b796c89f4760eba7dd1d77c2f515f0f8306ab6db0ffa6b747a0856ff537da800b79d3beb591e04cedaa8fcd1043a646145d9c0a21d1a1228daa1e66
|
data/README.md
CHANGED
|
@@ -37,13 +37,13 @@ On boot the gem writes one line to the Rails log. It names the service and envir
|
|
|
37
37
|
reports under:
|
|
38
38
|
|
|
39
39
|
```
|
|
40
|
-
Packrat: sending traces and logs for shop/production to https://ingest.packrat.sh (rack, action_pack, active_job,
|
|
40
|
+
Packrat: sending traces and logs for shop/production to https://ingest.packrat.sh (rack, action_pack, active_job, net_http, pg, logger); redacting email and ip
|
|
41
41
|
```
|
|
42
42
|
|
|
43
43
|
## Configuration
|
|
44
44
|
|
|
45
|
-
|
|
46
|
-
an environment variable.
|
|
45
|
+
`Packrat.configure` sets the ingest key, the ingest URL, the privacy level,
|
|
46
|
+
[your own redaction](#redact-your-own-data) and `debug_export`. Everything else is an environment variable.
|
|
47
47
|
|
|
48
48
|
| Variable | Default | Note |
|
|
49
49
|
| ------------------------------- | --------------------------- | ------------------------------------------------------------ |
|
|
@@ -53,13 +53,20 @@ an environment variable.
|
|
|
53
53
|
| `PACKRAT_OTEL_INSTRUMENTATIONS` | the list below | Comma-separated names to add; a leading `-` removes one |
|
|
54
54
|
| `PACKRAT_PRIVACY` | `config.privacy`, else `standard` | `standard` or `strict`. Wins over the initializer. Any other value is strict |
|
|
55
55
|
| `PACKRAT_REDACT` | `email,ip` | Patterns redacted from values; `-ip` stops masking IP addresses |
|
|
56
|
+
| `PACKRAT_DEBUG_EXPORT` | `config.debug_export`, else off | `1` prints everything sent to stdout. See [what is sent](#see-what-is-sent) |
|
|
56
57
|
|
|
57
|
-
The instrumentations are `rack`, `action_pack`, `action_view`, `active_job`, `
|
|
58
|
-
`
|
|
58
|
+
The instrumentations are `rack`, `action_pack`, `action_view`, `active_job`, `net_http`, `pg`,
|
|
59
|
+
`mysql2`, `sidekiq`, `redis`, `aws_sdk`, `concurrent_ruby`, `rake` and `logger`. One for a library that is not in the app installs nothing.
|
|
59
60
|
`PACKRAT_OTEL_INSTRUMENTATIONS=faraday,-action_view` adds Faraday (with
|
|
60
61
|
`opentelemetry-instrumentation-faraday` in the Gemfile) and removes Action View. `action_mailer` and
|
|
61
62
|
`active_storage` cannot be added: they must load before boot ends.
|
|
62
63
|
|
|
64
|
+
Each database query is one span. The `pg` or `mysql2` span is named after the query the way the
|
|
65
|
+
Rails log names it (`Order Load`, `Order Create`, `TRANSACTION`) and carries the SQL in
|
|
66
|
+
`db.statement`. A query Rails does not name, such as `connection.select_value`, keeps the driver's
|
|
67
|
+
name. `active_record` is not on the list because its span wraps the driver's and would record each
|
|
68
|
+
query twice. `PACKRAT_OTEL_INSTRUMENTATIONS=active_record` adds it back.
|
|
69
|
+
|
|
63
70
|
The service name is `OTEL_SERVICE_NAME`, else `service.name` in `OTEL_RESOURCE_ATTRIBUTES`, else the
|
|
64
71
|
app's module name (`Shop::Application` is `shop`). The environment is `Rails.env` unless
|
|
65
72
|
`OTEL_RESOURCE_ATTRIBUTES` names one. The version is `BUILD_COMMIT`, `HEROKU_SLUG_COMMIT` or
|
|
@@ -97,11 +104,47 @@ In standard, the gem redacts:
|
|
|
97
104
|
- **Credentials**: values named like a key, token, password, secret, signature or credential, and
|
|
98
105
|
`Bearer` or `Basic` values.
|
|
99
106
|
|
|
100
|
-
Patterns can't recognize other personal data, such as a name or
|
|
101
|
-
|
|
107
|
+
Patterns can't recognize other personal data, such as a name or an account number in a log line.
|
|
108
|
+
To remove that, [redact your own data](#redact-your-own-data). If your logs can hold text you can't
|
|
109
|
+
describe, such as a support ticket's body, use strict: pass `--strict` to the generator, set
|
|
102
110
|
`config.privacy = :strict`, or set `PACKRAT_PRIVACY=strict`. Strict keeps routes, durations, status
|
|
103
111
|
codes, errors by class and the shape of each query, which is what a deploy check needs.
|
|
104
112
|
|
|
113
|
+
### Redact your own data
|
|
114
|
+
|
|
115
|
+
Your own keys, patterns and block run after the gem's redaction, in standard and in strict:
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
Packrat.configure do |config|
|
|
119
|
+
config.redact_keys = %w[ssn account_number] # the whole value of these attributes and parameters
|
|
120
|
+
config.redact_patterns = [/acct \d{4}-\d{2}/] # each match, in every value and log line
|
|
121
|
+
config.redact { |value, name| value.gsub(Customer::NAME_PATTERN, "[name]") }
|
|
122
|
+
end
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
- **`redact_keys`** replaces the value with `[REDACTED]` in a span attribute, a query parameter, a
|
|
126
|
+
`Rails.error` context key or a key in a structured log line. An attribute matches on its whole name
|
|
127
|
+
or its last segment (`ssn` matches `enduser.ssn`). Unlike `filter_parameters`, a key matches only a
|
|
128
|
+
whole name: `ssn` doesn't match `ssn_last4`.
|
|
129
|
+
- **`redact_patterns`** replaces each match with `[REDACTED]`. A string is matched literally.
|
|
130
|
+
- **The `redact` block** receives each string value and its attribute name (`nil` for a log line) and
|
|
131
|
+
returns the string to send. If it raises or returns something other than a string, the gem sends
|
|
132
|
+
`[REDACTED]` and logs one warning.
|
|
133
|
+
|
|
134
|
+
### See what is sent
|
|
135
|
+
|
|
136
|
+
Set `PACKRAT_DEBUG_EXPORT=1`, or `config.debug_export = true`, and the gem prints each span and log
|
|
137
|
+
line to stdout as it is sent, after redaction:
|
|
138
|
+
|
|
139
|
+
```
|
|
140
|
+
Packrat export: {"span":"GET /orders/:id","kind":"server","attributes":{"url.query":"token=[REDACTED]&page=2",…},"events":[]}
|
|
141
|
+
Packrat export: {"log":"Refund issued to [email:4f2a91c0]","severity":"INFO","attributes":{}}
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Everything is still sent to Packrat. `PACKRAT_DEBUG_EXPORT=0` wins over the initializer.
|
|
145
|
+
|
|
146
|
+
How long Packrat keeps your data: https://packrat.sh/docs/ruby-on-rails#how-long-data-is-kept
|
|
147
|
+
|
|
105
148
|
## Beside other tools
|
|
106
149
|
|
|
107
150
|
- **Nothing else:** the gem configures OpenTelemetry.
|
data/lib/packrat/config.rb
CHANGED
|
@@ -6,7 +6,7 @@ module Packrat
|
|
|
6
6
|
# wins: the gem fills in only what the standard variables leave empty.
|
|
7
7
|
class Config
|
|
8
8
|
DEFAULT_URL = "https://ingest.packrat.sh".freeze
|
|
9
|
-
INSTRUMENTATIONS = %w[rack action_pack action_view active_job
|
|
9
|
+
INSTRUMENTATIONS = %w[rack action_pack action_view active_job net_http pg mysql2 sidekiq redis aws_sdk concurrent_ruby rake logger].freeze
|
|
10
10
|
COMMIT_VARIABLES = %w[BUILD_COMMIT HEROKU_SLUG_COMMIT SOURCE_VERSION].freeze
|
|
11
11
|
REDACTIONS = %w[email ip].freeze
|
|
12
12
|
PRIVACY = %w[standard strict].freeze
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
require "active_support/isolated_execution_state"
|
|
2
|
+
require "active_support/notifications"
|
|
3
|
+
|
|
4
|
+
module Packrat
|
|
5
|
+
module Otel
|
|
6
|
+
# One span per query, in place of the Active Record instrumentation's span around the driver's.
|
|
7
|
+
# Rails names each query in the sql.active_record event its log line prints ("Order Load",
|
|
8
|
+
# "Order Create"), and the driver's span starts inside that event, so the driver's span takes the
|
|
9
|
+
# name and keeps its SQL. A query Rails did not name keeps the driver's name.
|
|
10
|
+
class QueryNames < OpenTelemetry::SDK::Trace::SpanProcessor
|
|
11
|
+
EVENT = "sql.active_record".freeze
|
|
12
|
+
SCOPES = %w[OpenTelemetry::Instrumentation::PG OpenTelemetry::Instrumentation::Mysql2 OpenTelemetry::Instrumentation::Trilogy].freeze
|
|
13
|
+
KEY = :packrat_query_names
|
|
14
|
+
|
|
15
|
+
# The sql.active_record subscriber: the name of each query in flight, innermost last.
|
|
16
|
+
class Names
|
|
17
|
+
def start(_event, _id, payload)
|
|
18
|
+
name = payload[:name]
|
|
19
|
+
Names.stack << (name unless name.nil? || name.empty?)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def finish(_event, _id, _payload)
|
|
23
|
+
Names.stack.pop
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def self.stack
|
|
27
|
+
ActiveSupport::IsolatedExecutionState[KEY] ||= []
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def self.install(provider)
|
|
32
|
+
ActiveSupport::Notifications.subscribe(EVENT, Names.new)
|
|
33
|
+
provider.add_span_processor(new)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def on_start(span, _parent_context)
|
|
37
|
+
return unless SCOPES.include?(span.instrumentation_scope&.name)
|
|
38
|
+
|
|
39
|
+
name = Names.stack.last
|
|
40
|
+
span.name = name if name
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
end
|
|
@@ -229,28 +229,41 @@ module Packrat
|
|
|
229
229
|
class ScrubbingSpanExporter
|
|
230
230
|
PREFIX = "Packrat export: ".freeze
|
|
231
231
|
|
|
232
|
-
|
|
232
|
+
# With a block, a forked process builds its own exporter with it: the parent's holds a TLS connection
|
|
233
|
+
# that is open in both, and a write from the child corrupts the session ("bad record mac"). That
|
|
234
|
+
# includes the shutdown at exit, which closes the connection by writing to it.
|
|
235
|
+
def initialize(exporter, scrubber = Scrubber.new, io: nil, &rebuild)
|
|
233
236
|
@exporter = exporter
|
|
234
237
|
@scrubber = scrubber
|
|
235
238
|
@io = io
|
|
239
|
+
@rebuild = rebuild
|
|
240
|
+
@pid = Process.pid
|
|
236
241
|
end
|
|
237
242
|
|
|
238
243
|
def export(spans, timeout: nil)
|
|
239
244
|
spans = @scrubber.scrub_spans(spans)
|
|
240
245
|
write_lines(spans) { |span| span_line(span) }
|
|
241
|
-
|
|
246
|
+
exporter.export(spans, timeout:)
|
|
242
247
|
end
|
|
243
248
|
|
|
244
249
|
def force_flush(timeout: nil)
|
|
245
|
-
|
|
250
|
+
exporter.force_flush(timeout:)
|
|
246
251
|
end
|
|
247
252
|
|
|
248
253
|
def shutdown(timeout: nil)
|
|
249
|
-
|
|
254
|
+
exporter.shutdown(timeout:)
|
|
250
255
|
end
|
|
251
256
|
|
|
252
257
|
private
|
|
253
258
|
|
|
259
|
+
def exporter
|
|
260
|
+
if @rebuild && @pid != Process.pid
|
|
261
|
+
@pid = Process.pid
|
|
262
|
+
@exporter = @rebuild.call
|
|
263
|
+
end
|
|
264
|
+
@exporter
|
|
265
|
+
end
|
|
266
|
+
|
|
254
267
|
def write_lines(items)
|
|
255
268
|
return unless @io
|
|
256
269
|
|
|
@@ -272,7 +285,7 @@ module Packrat
|
|
|
272
285
|
def export(records, timeout: nil)
|
|
273
286
|
records = @scrubber.scrub_log_records(records)
|
|
274
287
|
write_lines(records) { |record| log_line(record) }
|
|
275
|
-
|
|
288
|
+
exporter.export(records, timeout:)
|
|
276
289
|
end
|
|
277
290
|
|
|
278
291
|
private
|
data/lib/packrat/otel.rb
CHANGED
|
@@ -21,6 +21,8 @@ module Packrat
|
|
|
21
21
|
BOOT_ONLY = %w[action_mailer active_storage].freeze
|
|
22
22
|
ACTION_VIEW_EVENTS = %w[render_template render_partial render_collection render_layout].map { |event| "#{event}.action_view" }.freeze
|
|
23
23
|
EXIT_TIMEOUT = 5
|
|
24
|
+
# The drivers whose spans take the name Rails gives each query (QueryNames).
|
|
25
|
+
QUERY_DRIVERS = %w[pg mysql2 trilogy].freeze
|
|
24
26
|
|
|
25
27
|
# dogstatsd-ruby defines Datadog::Statsd without the tracer, hence the method check.
|
|
26
28
|
def self.datadog?
|
|
@@ -84,6 +86,7 @@ module Packrat
|
|
|
84
86
|
@outcome = :configured
|
|
85
87
|
step("sampler") { use_sampler } unless @config.set?("OTEL_TRACES_SAMPLER")
|
|
86
88
|
@config.instrumentations.each { |name| step("instrumentation #{name}") { install(name) } }
|
|
89
|
+
step("query names") { QueryNames.install(OpenTelemetry.tracer_provider) } if (@installed & QUERY_DRIVERS).any?
|
|
87
90
|
step("Rails.error") { @errors.subscribe(ErrorSpans.new(scrubber)) }
|
|
88
91
|
step("at_exit") { at_exit { shutdown_providers } }
|
|
89
92
|
@line = "Packrat: sending #{signals(records)} for #{@config.service_name}/#{@config.environment} " \
|
|
@@ -140,15 +143,13 @@ module Packrat
|
|
|
140
143
|
end
|
|
141
144
|
|
|
142
145
|
def span_processor
|
|
143
|
-
OpenTelemetry::
|
|
144
|
-
|
|
145
|
-
)
|
|
146
|
+
build = -> { OpenTelemetry::Exporter::OTLP::Exporter.new(endpoint: @config.traces_endpoint, headers: @config.headers) }
|
|
147
|
+
OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(ScrubbingSpanExporter.new(build.call, scrubber, io: debug_io, &build))
|
|
146
148
|
end
|
|
147
149
|
|
|
148
150
|
def log_record_processor
|
|
149
|
-
OpenTelemetry::
|
|
150
|
-
|
|
151
|
-
)
|
|
151
|
+
build = -> { OpenTelemetry::Exporter::OTLP::Logs::LogsExporter.new(endpoint: @config.logs_endpoint, headers: @config.headers) }
|
|
152
|
+
OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(ScrubbingLogExporter.new(build.call, scrubber, io: debug_io, &build))
|
|
152
153
|
rescue NameError => error
|
|
153
154
|
raise unless error.name == :Utilities
|
|
154
155
|
|
data/lib/packrat/version.rb
CHANGED
data/lib/packrat.rb
CHANGED
|
@@ -6,6 +6,7 @@ require_relative "packrat/config"
|
|
|
6
6
|
require_relative "packrat/otel/scrubber"
|
|
7
7
|
require_relative "packrat/otel/error_spans"
|
|
8
8
|
require_relative "packrat/otel/root_kind_sampler"
|
|
9
|
+
require_relative "packrat/otel/query_names"
|
|
9
10
|
require_relative "packrat/otel"
|
|
10
11
|
require_relative "packrat/rack_span"
|
|
11
12
|
require_relative "packrat/railtie" if defined?(Rails::Railtie)
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: packrat-rails
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Packrat Labs
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-11-
|
|
11
|
+
date: 2026-11-18 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: railties
|
|
@@ -291,6 +291,7 @@ files:
|
|
|
291
291
|
- lib/packrat/config.rb
|
|
292
292
|
- lib/packrat/otel.rb
|
|
293
293
|
- lib/packrat/otel/error_spans.rb
|
|
294
|
+
- lib/packrat/otel/query_names.rb
|
|
294
295
|
- lib/packrat/otel/root_kind_sampler.rb
|
|
295
296
|
- lib/packrat/otel/scrubber.rb
|
|
296
297
|
- lib/packrat/rack_span.rb
|