zharp 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +33 -0
- data/LICENSE +21 -0
- data/README.md +170 -0
- data/lib/generators/zharp/install/install_generator.rb +39 -0
- data/lib/generators/zharp/install/templates/zharp.rb.tt +15 -0
- data/lib/zharp/bodies.rb +386 -0
- data/lib/zharp/capture.rb +361 -0
- data/lib/zharp/config.rb +129 -0
- data/lib/zharp/context.rb +133 -0
- data/lib/zharp/db_errors.rb +229 -0
- data/lib/zharp/errors.rb +158 -0
- data/lib/zharp/fork.rb +70 -0
- data/lib/zharp/headers.rb +63 -0
- data/lib/zharp/instrumentations.rb +17 -0
- data/lib/zharp/logs.rb +128 -0
- data/lib/zharp/rack.rb +148 -0
- data/lib/zharp/railtie.rb +47 -0
- data/lib/zharp/setup.rb +206 -0
- data/lib/zharp/sidekiq.rb +13 -0
- data/lib/zharp/source.rb +118 -0
- data/lib/zharp/stamp.rb +124 -0
- data/lib/zharp/version.rb +5 -0
- data/lib/zharp.rb +218 -0
- metadata +341 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 58fd6ce97a15cf4ba775b28da313a421cc5809e145ab7ba7f744034b9589e052
|
|
4
|
+
data.tar.gz: fd36fde29b5fe7681a8233493f037b117aa1d7e08af5beca0b6ed6531d669905
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: f1871fd7081b1815a909bf83699df5a9ba6576f687cdfb0a4c752f3d751d8fce303999a0d2ab1b2fab238f58c775cbf701d4ca8b0b4daeac4dc4b71d30795587
|
|
7
|
+
data.tar.gz: 860e2dce601c5de97655001888854f3bfb676d00198cbc4becc8bdbd0c58a4e34e1620d5daa57456e53973ee3944099f19c80535c249a8b3468e2e63e8b21374
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 (unreleased)
|
|
4
|
+
|
|
5
|
+
The first release: one `zharp` gem for Rails, Rack apps, Sidekiq and plain Ruby, built on the
|
|
6
|
+
official OpenTelemetry SDK (`opentelemetry-sdk` 1.13) and its instrumentations for Rack, Rails,
|
|
7
|
+
Sinatra, Grape, Net::HTTP, Faraday, HTTP.rb, HTTPX, pg, mysql2, trilogy, Redis, Sidekiq, Resque and
|
|
8
|
+
Delayed Job.
|
|
9
|
+
Ruby 3.3+, Rails 7.1+.
|
|
10
|
+
|
|
11
|
+
- Rails starts it by itself (a Railtie); `bin/rails generate zharp:install` writes
|
|
12
|
+
`config/initializers/zharp.rb` and can store the key in the credentials.
|
|
13
|
+
- Rack apps: `use Zharp::Rack`. Plain Ruby: `Zharp.init`. Sidekiq outside Rails: `require "zharp/sidekiq"` in the file Sidekiq loads.
|
|
14
|
+
- Settings from `ZHARP_KEY`, `ZHARP_SERVICE`, `ZHARP_ENVIRONMENT`, `ZHARP_RELEASE`,
|
|
15
|
+
`ZHARP_ENDPOINT`, `ZHARP_SAMPLE_RATE`, `ZHARP_LOGS`, `ZHARP_LOG_CORRELATION`,
|
|
16
|
+
`ZHARP_CAPTURE_HEADERS`, `ZHARP_CAPTURE_QUERY_ERRORS`, `ZHARP_CAPTURE_SOURCE_CONTEXT`,
|
|
17
|
+
`ZHARP_TRACING`, `ZHARP_DISABLED`; in Rails also `credentials.zharp.key`.
|
|
18
|
+
- Errors counted once (Action Pack, `Rails.error`, Rack, Sidekiq, Active Job, `at_exit`), with the
|
|
19
|
+
code around the app's frames as `zharp.source_context`.
|
|
20
|
+
- All request and response headers on the server span, secrets as `[REDACTED]`.
|
|
21
|
+
- Failed queries (pg, mysql2, trilogy, Active Record) reported once with system, database,
|
|
22
|
+
obfuscated statement and error code; Postgres's `DETAIL:` line dropped.
|
|
23
|
+
- Sidekiq and Active Job jobs in the trace of the request that queued them.
|
|
24
|
+
- ` trace_id=… span_id=…` at the end of each `Logger` and semantic_logger line in a request;
|
|
25
|
+
`ZHARP_LOGS=true` sends the lines from the SDK.
|
|
26
|
+
- `capture_bodies` (off by default): request and response bodies on Rack and Net::HTTP spans,
|
|
27
|
+
secrets filtered, files and binary bodies described, 32 KB cap. Secret query values are masked
|
|
28
|
+
in every span's URL.
|
|
29
|
+
- Database error messages carry no values: Postgres's `DETAIL:` line, MySQL's `Duplicate entry
|
|
30
|
+
'…'`, invalid-input values.
|
|
31
|
+
- Fork safety for Puma cluster mode and Unicorn: each worker's exporters get their own connection.
|
|
32
|
+
- `Zharp.capture_error`, `set_user`, `clear_user`, `set_tag`, `log`, `start_span`, `flush`,
|
|
33
|
+
`shutdown`.
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Zharp
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
# Zharp for Ruby
|
|
2
|
+
|
|
3
|
+
Application Monitoring for Ruby apps: traces, errors and logs, sent to [Zharp](https://zharp.io).
|
|
4
|
+
One gem for Rails, any Rack app (Sinatra, Hanami, Roda, Grape), Sidekiq and plain Ruby, built on
|
|
5
|
+
the official OpenTelemetry SDK.
|
|
6
|
+
|
|
7
|
+
- Requests named by route, continuing the caller's `traceparent`; database queries (pg, mysql2,
|
|
8
|
+
trilogy, Active Record), outgoing HTTP (Net::HTTP, Faraday, HTTP.rb, HTTPX), Redis, jobs
|
|
9
|
+
(Sidekiq, Active Job, Resque, Delayed Job), Action View and Action Mailer, each switched on when
|
|
10
|
+
its library is loaded. Anything else you instrument with OpenTelemetry is sent too.
|
|
11
|
+
- **Errors, each counted once**, with type, message, the stack and the code that ran (the lines
|
|
12
|
+
around your app's innermost five frames, read on your server; never gems or Ruby's library).
|
|
13
|
+
- **Request and response headers** on the request's span, secrets (authorization, cookies, API
|
|
14
|
+
keys, tokens, sessions) as `[REDACTED]`.
|
|
15
|
+
- **Failed queries** reported once, with the statement (obfuscated), the database and the error
|
|
16
|
+
code; Postgres's `DETAIL:` line (the row's values) is dropped.
|
|
17
|
+
- **Jobs in the request's trace**: Sidekiq and Active Job (Solid Queue, GoodJob, Delayed Job…) run
|
|
18
|
+
as children of the request that queued them.
|
|
19
|
+
- **Logs**: each line written through `Logger` (so `Rails.logger` and lograge) or semantic_logger
|
|
20
|
+
during a request ends with ` trace_id=… span_id=…` (two fields on a JSON line), so the Zharp
|
|
21
|
+
agent links it to its request. `ZHARP_LOGS=true` sends the lines from the SDK instead, where
|
|
22
|
+
there is no agent.
|
|
23
|
+
- **Forking servers**: Puma in cluster mode (`preload_app!`) and Unicorn need nothing.
|
|
24
|
+
|
|
25
|
+
Ruby 3.3+ (3.3, 3.4, 4.0), Rails 7.1+.
|
|
26
|
+
|
|
27
|
+
## Rails
|
|
28
|
+
|
|
29
|
+
```sh
|
|
30
|
+
bundle add zharp
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Set your key, from **Settings → Applications** in Zharp:
|
|
34
|
+
|
|
35
|
+
```sh
|
|
36
|
+
ZHARP_KEY=zpk_… bin/rails server
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
or keep it in the credentials (`bin/rails credentials:edit`):
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
zharp:
|
|
43
|
+
key: zpk_…
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
That's all: the gem's Railtie starts Zharp when Rails boots, in the web server, `bin/rails runner`,
|
|
47
|
+
Sidekiq and Solid Queue alike. To change a default, generate the initializer, which lists every
|
|
48
|
+
option commented out (with `--key` it also stores the key in the credentials):
|
|
49
|
+
|
|
50
|
+
```sh
|
|
51
|
+
bin/rails generate zharp:install --key=zpk_…
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Puma in cluster mode with `preload_app!` prints "Detected 2 Thread(s) started in app boot": those
|
|
55
|
+
are OpenTelemetry's export threads in the master process. Each worker starts its own after the fork
|
|
56
|
+
and sends over a connection of its own; nothing needs changing.
|
|
57
|
+
|
|
58
|
+
## Rack apps: Sinatra, Hanami, Roda, Grape, plain Rack
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
bundle add zharp
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
In `config.ru`:
|
|
65
|
+
|
|
66
|
+
```ruby
|
|
67
|
+
require "zharp"
|
|
68
|
+
use Zharp::Rack
|
|
69
|
+
|
|
70
|
+
run App
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
and `ZHARP_KEY` in the environment. Sinatra and Grape spans are named by their routes; others by
|
|
74
|
+
the path with ids replaced (`GET /orders/:id`). Call `Zharp.init(service: "shop")` before `use`
|
|
75
|
+
to set options in code.
|
|
76
|
+
|
|
77
|
+
## Plain Ruby: scripts, workers, CLIs
|
|
78
|
+
|
|
79
|
+
```ruby
|
|
80
|
+
require "zharp"
|
|
81
|
+
Zharp.init(service: "billing-worker")
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Call it after your gems are required, so their instrumentation is switched on. An exception that
|
|
85
|
+
ends the script is reported and sent before it exits.
|
|
86
|
+
|
|
87
|
+
### Sidekiq outside Rails
|
|
88
|
+
|
|
89
|
+
At the top of the file Sidekiq loads (`bundle exec sidekiq -r ./worker.rb`), with `ZHARP_KEY` in
|
|
90
|
+
the environment:
|
|
91
|
+
|
|
92
|
+
```ruby
|
|
93
|
+
require "zharp/sidekiq"
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Sidekiq keeps only one `-r`, so this goes in that file rather than on the command line. Jobs queued
|
|
97
|
+
from a traced request or span run in its trace. (In a Rails app Sidekiq needs nothing.)
|
|
98
|
+
|
|
99
|
+
## Settings
|
|
100
|
+
|
|
101
|
+
Every setting can be passed to `Zharp.init` or set in the environment.
|
|
102
|
+
|
|
103
|
+
| `Zharp.init` | Environment | Default |
|
|
104
|
+
|---|---|---|
|
|
105
|
+
| `key:` | `ZHARP_KEY` (Rails: then `credentials.zharp.key`) | — nothing is sent without one |
|
|
106
|
+
| `service:` | `ZHARP_SERVICE`, `OTEL_SERVICE_NAME` | the Rails app's name, else the project folder |
|
|
107
|
+
| `environment:` | `ZHARP_ENVIRONMENT` | `Rails.env` |
|
|
108
|
+
| `release:` | `ZHARP_RELEASE` | — |
|
|
109
|
+
| `endpoint:` | `ZHARP_ENDPOINT` | `https://api.zharp.io/api/v1/otlp` |
|
|
110
|
+
| `sample_rate:` | `ZHARP_SAMPLE_RATE` | `1.0` (errors are always sent) |
|
|
111
|
+
| `logs:` | `ZHARP_LOGS` | `false` |
|
|
112
|
+
| `log_correlation:` | `ZHARP_LOG_CORRELATION` | `true` |
|
|
113
|
+
| `capture_headers:` | `ZHARP_CAPTURE_HEADERS` | on; `{ redact: ["x-tenant"] }` filters more names |
|
|
114
|
+
| `capture_query_errors:` | `ZHARP_CAPTURE_QUERY_ERRORS` | `true` |
|
|
115
|
+
| `capture_source_context:` | `ZHARP_CAPTURE_SOURCE_CONTEXT` | `true` |
|
|
116
|
+
| `capture_bodies:` | `ZHARP_CAPTURE_BODIES` (`errors`, `all`, `off`) | off |
|
|
117
|
+
| `tracing:` | `ZHARP_TRACING` | `true`; `false` sends errors and logs only |
|
|
118
|
+
| `instrumentation:` | — | options per OpenTelemetry instrumentation, by class name |
|
|
119
|
+
| — | `ZHARP_DISABLED=true` | turns everything off |
|
|
120
|
+
|
|
121
|
+
### Request and response bodies
|
|
122
|
+
|
|
123
|
+
Off by default. `capture_bodies: "errors"` (or `true`) keeps the bodies of failed requests (status
|
|
124
|
+
400 and above, or no response), `"all"` of every request: incoming requests (any Rack app, Rails
|
|
125
|
+
included) and outgoing Net::HTTP calls (Faraday's default adapter too).
|
|
126
|
+
|
|
127
|
+
```ruby
|
|
128
|
+
Zharp.init(capture_bodies: { mode: "errors", max_size: 32_768, redact_keys: ["date_of_birth"], urls: ["api.example.com"] })
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Text is capped at 32 KB (`max_size`, at most 128 KB). Files in uploads, inline images and binary
|
|
132
|
+
bodies are described (name, type, size), never sent. Field names are kept; values under
|
|
133
|
+
secret-looking keys (`password`, `token`, `authorization`, `card`…) and `redact_keys` become
|
|
134
|
+
`[Filtered]` before anything is sent. `urls` limits it to some requests (a host, a URL prefix or a
|
|
135
|
+
Regexp). Secret-looking query values in URLs are masked on every span either way.
|
|
136
|
+
|
|
137
|
+
## API
|
|
138
|
+
|
|
139
|
+
```ruby
|
|
140
|
+
Zharp.capture_error(error, order_id: 42) # an exception you rescued, as a handled error
|
|
141
|
+
Zharp.set_user(id: user.id, email: user.email) # on the request's spans, logs and errors
|
|
142
|
+
Zharp.set_tag("tenant", "acme") # on every span and log from now on
|
|
143
|
+
Zharp.log.info("order paid", order_id: 42) # a log line sent from the SDK, whatever ZHARP_LOGS says
|
|
144
|
+
Zharp.start_span("render invoice") { render } # a span around some work
|
|
145
|
+
Zharp.flush # send what's queued now (end of a script or job)
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Rails' own `Rails.error.report` and `Rails.error.handle` reach Zharp too.
|
|
149
|
+
|
|
150
|
+
A browser or mobile app using a Zharp SDK sends its user and session with each request (W3C
|
|
151
|
+
`baggage`); they are put on the request's spans, logs and errors without any code.
|
|
152
|
+
|
|
153
|
+
## Development
|
|
154
|
+
|
|
155
|
+
No Ruby needed on the host: everything runs in Docker.
|
|
156
|
+
|
|
157
|
+
```sh
|
|
158
|
+
docker run --rm -v "$PWD:/src" -w /src ruby:3.4 sh -c "gem install bundler && bundle install && bundle exec rake"
|
|
159
|
+
e2e/unit.sh # unit tests: Ruby 3.3 / 3.4 / 4.0 × no Rails, Rails 7.1 / 7.2 / 8.1
|
|
160
|
+
e2e/generator.sh 3.4 8.1.0 # rails new, bundle add zharp, the generator, boot
|
|
161
|
+
# End-to-end against a Zharp backend: a throwaway app per case, its docker compose stack, what
|
|
162
|
+
# Zharp stored saved in e2e/out/<case>.json. Cases: rails81 rails71 sinatra rack hanami script semantic.
|
|
163
|
+
ZHARP_API=http://localhost:3000/api/v1 ZHARP_TOKEN=<user JWT> ZHARP_WORKSPACE=<id> node e2e/run.mjs rails81
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
`e2e/bundler-check.sh` times `bundle add zharp` in fresh Rails apps with Bundler 2.6, 2.7 and 4.
|
|
167
|
+
|
|
168
|
+
## License
|
|
169
|
+
|
|
170
|
+
MIT
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/generators"
|
|
4
|
+
|
|
5
|
+
module Zharp
|
|
6
|
+
module Generators
|
|
7
|
+
# bin/rails generate zharp:install [--key=zpk_…]
|
|
8
|
+
#
|
|
9
|
+
# Writes config/initializers/zharp.rb (every option there, commented out) and, given a key,
|
|
10
|
+
# stores it in the credentials as zharp.key. Without one it says how to set ZHARP_KEY.
|
|
11
|
+
class InstallGenerator < ::Rails::Generators::Base
|
|
12
|
+
source_root File.expand_path("templates", __dir__)
|
|
13
|
+
|
|
14
|
+
class_option :key, type: :string, desc: "Your Zharp ingest key (zpk_…), stored in the credentials as zharp.key"
|
|
15
|
+
|
|
16
|
+
def create_initializer
|
|
17
|
+
template "zharp.rb.tt", "config/initializers/zharp.rb"
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def store_key
|
|
21
|
+
key = options[:key].to_s.strip
|
|
22
|
+
if key.empty?
|
|
23
|
+
say "Set your key: ZHARP_KEY=zpk_… in the environment, or bin/rails credentials:edit and add\n zharp:\n key: zpk_…", :yellow
|
|
24
|
+
return
|
|
25
|
+
end
|
|
26
|
+
credentials = ::Rails.application.credentials
|
|
27
|
+
content = credentials.read.to_s
|
|
28
|
+
if content.match?(/^zharp:\s*$/)
|
|
29
|
+
say "credentials already have a zharp: section; left as it is", :yellow
|
|
30
|
+
return
|
|
31
|
+
end
|
|
32
|
+
credentials.write("#{content.chomp}\n\nzharp:\n key: #{key}\n".lstrip)
|
|
33
|
+
say_status :credentials, "zharp.key added (#{credentials.content_path.basename})"
|
|
34
|
+
rescue StandardError => e
|
|
35
|
+
say "Could not write the credentials (#{e.message}). Set ZHARP_KEY=#{key} in the environment instead.", :red
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Zharp Application Monitoring. Rails starts it without this file: what's here only changes the
|
|
2
|
+
# defaults. The key comes from ZHARP_KEY, then credentials (zharp.key).
|
|
3
|
+
Zharp.init(
|
|
4
|
+
<%= %( # service: "#{Rails.application.class.name.to_s.split("::").first.to_s.gsub(/([a-z\d])([A-Z])/, '\1-\2').downcase}",).ljust(37) %># ZHARP_SERVICE
|
|
5
|
+
# environment: Rails.env, # ZHARP_ENVIRONMENT
|
|
6
|
+
# release: ENV["GIT_SHA"], # ZHARP_RELEASE: the version you deploy, for "new in this release"
|
|
7
|
+
# sample_rate: 1.0, # ZHARP_SAMPLE_RATE: share of traces kept; errors are always sent
|
|
8
|
+
# logs: false, # ZHARP_LOGS: send log lines from here (off: the Zharp agent reads them)
|
|
9
|
+
# log_correlation: true, # ZHARP_LOG_CORRELATION: " trace_id=… span_id=…" at the end of each line
|
|
10
|
+
# capture_headers: { redact: ["x-tenant"] }, # more header names to filter; false turns headers off
|
|
11
|
+
# capture_query_errors: true, # a failed query is reported once, with its statement
|
|
12
|
+
# capture_source_context: true, # the code around each error's lines is sent with it
|
|
13
|
+
# capture_bodies: "errors", # ZHARP_CAPTURE_BODIES: bodies of failed requests ("all": every one)
|
|
14
|
+
# instrumentation: { "OpenTelemetry::Instrumentation::Sidekiq" => { span_naming: :job_class } },
|
|
15
|
+
)
|
data/lib/zharp/bodies.rb
ADDED
|
@@ -0,0 +1,386 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "uri"
|
|
5
|
+
require "bigdecimal"
|
|
6
|
+
|
|
7
|
+
module Zharp
|
|
8
|
+
# Request and response bodies on HTTP spans (capture_bodies), and secrets in URLs.
|
|
9
|
+
#
|
|
10
|
+
# A port of the rules every Zharp SDK shares (@zharp/internal/bodies, zharp-python's _bodies.py),
|
|
11
|
+
# so a payload looks the same whichever SDK sent it:
|
|
12
|
+
# - field names stay; values under keys that look secret (password, token, authorization, card…)
|
|
13
|
+
# become [Filtered], and an object under such a key keeps its keys with every value filtered;
|
|
14
|
+
# - an inline image or other long base64 value becomes [image/png · 24 KB], a file in a form
|
|
15
|
+
# upload [file photo.png · image/png · 240 KB], and a body that isn't text at all
|
|
16
|
+
# [binary · application/pdf · 1.2 MB] or [event stream];
|
|
17
|
+
# - text is capped (32 KB by default), with http.*.body.truncated set when it was cut.
|
|
18
|
+
module Bodies
|
|
19
|
+
DEFAULT_MAX_BODY = 32 * 1024
|
|
20
|
+
MAX_BODY = 128 * 1024
|
|
21
|
+
FILTERED = "[Filtered]"
|
|
22
|
+
|
|
23
|
+
REQUEST_BODY = "http.request.body"
|
|
24
|
+
REQUEST_TRUNCATED = "http.request.body.truncated"
|
|
25
|
+
REQUEST_TYPE = "http.request.header.content-type"
|
|
26
|
+
RESPONSE_BODY = "http.response.body"
|
|
27
|
+
RESPONSE_TRUNCATED = "http.response.body.truncated"
|
|
28
|
+
RESPONSE_TYPE = "http.response.header.content-type"
|
|
29
|
+
ATTRS = {
|
|
30
|
+
"request" => [REQUEST_BODY, REQUEST_TRUNCATED, REQUEST_TYPE],
|
|
31
|
+
"response" => [RESPONSE_BODY, RESPONSE_TRUNCATED, RESPONSE_TYPE]
|
|
32
|
+
}.freeze
|
|
33
|
+
|
|
34
|
+
# mode: "errors" (status 400 and above, or no response) or "all"; max_size: characters kept;
|
|
35
|
+
# redact_keys: more secret keys (a String matches ignoring case, - and _; a Regexp is searched
|
|
36
|
+
# in the key); urls: which requests (a bare host, a URL prefix, or a Regexp), nil for all.
|
|
37
|
+
Settings = Struct.new(:mode, :max_size, :redact_keys, :urls, keyword_init: true)
|
|
38
|
+
|
|
39
|
+
module_function
|
|
40
|
+
|
|
41
|
+
# capture_bodies: true, "errors", "all", false/"off", or a Hash with mode, max_size,
|
|
42
|
+
# redact_keys, urls. ZHARP_CAPTURE_BODIES (errors, all, off) when nothing is passed. Off by default.
|
|
43
|
+
def settings(option, env_value)
|
|
44
|
+
value = option
|
|
45
|
+
if value.nil? && env_value
|
|
46
|
+
text = env_value.to_s.strip.downcase
|
|
47
|
+
value = if %w[errors all].include?(text) then text
|
|
48
|
+
elsif Config::TRUE.match?(text) then true
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
return nil if value.nil? || value == false
|
|
52
|
+
|
|
53
|
+
value = { mode: value } if value.is_a?(String) || value.is_a?(Symbol)
|
|
54
|
+
value = {} if value == true
|
|
55
|
+
return nil unless value.is_a?(Hash)
|
|
56
|
+
|
|
57
|
+
value = value.transform_keys(&:to_sym)
|
|
58
|
+
mode = value[:mode].to_s.strip.downcase
|
|
59
|
+
return nil if ["", "off", "false", "no", "none", "0"].include?(mode) && value.key?(:mode)
|
|
60
|
+
|
|
61
|
+
size = value[:max_size]
|
|
62
|
+
size = size.is_a?(Numeric) && size.positive? ? [size.floor, MAX_BODY].min : DEFAULT_MAX_BODY
|
|
63
|
+
Settings.new(
|
|
64
|
+
mode: mode == "all" ? "all" : "errors",
|
|
65
|
+
max_size: size,
|
|
66
|
+
redact_keys: Array(value[:redact_keys]),
|
|
67
|
+
urls: value[:urls].nil? ? nil : Array(value[:urls])
|
|
68
|
+
)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# Raw text read before shortening and capping: inline images shrink to a few words.
|
|
72
|
+
def raw_limit(settings) = [64 * 1024, settings.max_size * 2].max
|
|
73
|
+
|
|
74
|
+
# Whether a request with this outcome has its bodies kept (no status: no response).
|
|
75
|
+
def wanted?(settings, status) = settings.mode == "all" || status.nil? || status.to_i.zero? || status.to_i >= 400
|
|
76
|
+
|
|
77
|
+
def allowed?(settings, url) = settings.urls.nil? || url_matches?(url.to_s, settings.urls)
|
|
78
|
+
|
|
79
|
+
# ─── content types ───────────────────────────────────────────────────────────
|
|
80
|
+
|
|
81
|
+
def mime(content_type) = content_type.to_s.split(";").first.to_s.strip.downcase
|
|
82
|
+
def event_stream?(content_type) = mime(content_type) == "text/event-stream"
|
|
83
|
+
def multipart?(content_type) = mime(content_type) == "multipart/form-data"
|
|
84
|
+
def form?(content_type) = mime(content_type) == "application/x-www-form-urlencoded"
|
|
85
|
+
|
|
86
|
+
TEXT_APPLICATION = %r{\Aapplication/(json|xml|x-www-form-urlencoded|graphql|javascript|x-ndjson|ld\+json)\z}
|
|
87
|
+
|
|
88
|
+
# Whether a body of this type is text worth showing. An unknown type gets the benefit of the doubt.
|
|
89
|
+
def textual?(content_type)
|
|
90
|
+
kind = mime(content_type)
|
|
91
|
+
return true if kind.empty?
|
|
92
|
+
return false if kind == "text/event-stream"
|
|
93
|
+
|
|
94
|
+
kind.start_with?("text/") || kind.end_with?("+json", "+xml") || TEXT_APPLICATION.match?(kind)
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
# 240 KB, 1.4 MB.
|
|
98
|
+
def format_size(size)
|
|
99
|
+
return "#{size} B" if size < 1024
|
|
100
|
+
return "#{(size / 1024.0 + 0.5).floor} KB" if size < 1024 * 1024
|
|
101
|
+
|
|
102
|
+
"#{format('%.1f', ((size / (1024.0 * 1024)) * 10 + 0.5).floor / 10.0)} MB"
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
# [file photo.png · image/png · 240 KB]
|
|
106
|
+
def file_label(name, content_type, size, partial = false)
|
|
107
|
+
parts = ["file #{name.nil? || name.to_s.empty? ? 'blob' : name}"]
|
|
108
|
+
parts << content_type if content_type && !content_type.to_s.empty?
|
|
109
|
+
parts << "#{format_size(size)}#{partial ? '+' : ''}" unless size.nil?
|
|
110
|
+
"[#{parts.join(' · ')}]"
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
# [binary · image/png · 24 KB], [event stream]
|
|
114
|
+
def describe_body(content_type, size = nil, partial = false)
|
|
115
|
+
return "[event stream]" if event_stream?(content_type)
|
|
116
|
+
|
|
117
|
+
parts = ["binary"]
|
|
118
|
+
kind = mime(content_type)
|
|
119
|
+
parts << kind unless kind.empty?
|
|
120
|
+
parts << "#{format_size(size)}#{partial ? '+' : ''}" if size&.positive?
|
|
121
|
+
"[#{parts.join(' · ')}]"
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
# ─── redaction ───────────────────────────────────────────────────────────────
|
|
125
|
+
|
|
126
|
+
# Built-in keys whose values never leave the process, compared without case, - or _.
|
|
127
|
+
CONTAINS = %w[
|
|
128
|
+
password passwd passphrase secret token apikey accesskey privatekey authorization cookie
|
|
129
|
+
creditcard cardnumber cvv cvc ssn iban sessionid signature credential
|
|
130
|
+
].freeze
|
|
131
|
+
EXACT = %w[auth pin pwd otp card pan jwt sid codeverifier].freeze
|
|
132
|
+
# Also secret in a query string, where they are API keys and signed-URL signatures.
|
|
133
|
+
URL_EXACT = %w[key sig code state session].freeze
|
|
134
|
+
|
|
135
|
+
def normalize(key) = key.to_s.downcase.gsub(/[^a-z0-9]/, "")
|
|
136
|
+
|
|
137
|
+
def secret_key?(key, extra = [])
|
|
138
|
+
k = normalize(key)
|
|
139
|
+
return false if k.empty?
|
|
140
|
+
return true if EXACT.include?(k) || CONTAINS.any? { |c| k.include?(c) }
|
|
141
|
+
|
|
142
|
+
extra.any? { |pattern| pattern.is_a?(Regexp) ? pattern.match?(key.to_s) : normalize(pattern) == k }
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
DATA_URI = %r{\Adata:([\w.+-]+/[\w.+-]+)?(?:;[\w=.+-]+)*;base64,([A-Za-z0-9+/=\s]*)\z}
|
|
146
|
+
BASE64 = %r{\A[A-Za-z0-9+/_-]{256,}={0,2}\z}
|
|
147
|
+
|
|
148
|
+
# A long base64 value as a short description; other strings as they are.
|
|
149
|
+
def shorten(value)
|
|
150
|
+
return value if value.length < 256
|
|
151
|
+
|
|
152
|
+
if (uri = DATA_URI.match(value))
|
|
153
|
+
return "[#{uri[1] || 'data'} · #{format_size(uri[2].length * 3 / 4)}]"
|
|
154
|
+
end
|
|
155
|
+
return "[base64 · #{format_size(value.length * 3 / 4)}]" if BASE64.match?(value) && value.match?(%r{[0-9+/]})
|
|
156
|
+
|
|
157
|
+
value
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def filter_all(value, depth)
|
|
161
|
+
return FILTERED if depth > 20
|
|
162
|
+
return value if value.nil? || value == ""
|
|
163
|
+
return value.map { |v| filter_all(v, depth + 1) } if value.is_a?(Array)
|
|
164
|
+
return value.transform_values { |v| filter_all(v, depth + 1) } if value.is_a?(Hash)
|
|
165
|
+
|
|
166
|
+
FILTERED
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
def redact_value(value, extra, depth = 0)
|
|
170
|
+
return shorten(value) if value.is_a?(String)
|
|
171
|
+
return value if depth > 20 || !(value.is_a?(Hash) || value.is_a?(Array))
|
|
172
|
+
return value.map { |v| redact_value(v, extra, depth + 1) } if value.is_a?(Array)
|
|
173
|
+
|
|
174
|
+
value.to_h { |k, v| [k, secret_key?(k, extra) ? filter_all(v, depth + 1) : redact_value(v, extra, depth + 1)] }
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
RUNS = %r{"((?:data:[\w.+-]+/[\w.+-]+(?:;[\w=.+-]+)*;base64,)?[A-Za-z0-9+/_=-]{256,})("?)}
|
|
178
|
+
PAIRS = /("([^"\\]{1,100})"\s*:\s*)("(?:[^"\\]|\\.)*"?|-?\d[\d.eE+-]*|true|false)/
|
|
179
|
+
|
|
180
|
+
# Text that didn't parse: secret "key": value pairs filtered, long base64 runs shortened.
|
|
181
|
+
def redact_loose(text, extra)
|
|
182
|
+
text = text.gsub(RUNS) do
|
|
183
|
+
m = Regexp.last_match
|
|
184
|
+
short = shorten(m[1])
|
|
185
|
+
if short == m[1] then m[0]
|
|
186
|
+
elsif m[2].empty? then %("#{short[0..-2]}+]")
|
|
187
|
+
else %("#{short}")
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
text.gsub(PAIRS) { (m = Regexp.last_match) && secret_key?(m[2], extra) ? %(#{m[1]}"#{FILTERED}") : m[0] }
|
|
191
|
+
end
|
|
192
|
+
|
|
193
|
+
def decode_component(text) = URI.decode_www_form_component(text)
|
|
194
|
+
|
|
195
|
+
def redact_form(text, extra)
|
|
196
|
+
text.split("&", -1).map do |pair|
|
|
197
|
+
eq = pair.index("=")
|
|
198
|
+
next pair unless eq
|
|
199
|
+
|
|
200
|
+
key = begin
|
|
201
|
+
decode_component(pair[0...eq])
|
|
202
|
+
rescue ArgumentError
|
|
203
|
+
pair[0...eq]
|
|
204
|
+
end
|
|
205
|
+
next "#{pair[0...eq]}=#{FILTERED}" if secret_key?(key, extra)
|
|
206
|
+
|
|
207
|
+
value = pair[(eq + 1)..]
|
|
208
|
+
value.length >= 256 ? "#{pair[0...eq]}=#{shorten_encoded(value)}" : pair
|
|
209
|
+
end.join("&")
|
|
210
|
+
end
|
|
211
|
+
|
|
212
|
+
def shorten_encoded(value)
|
|
213
|
+
decoded = decode_component(value)
|
|
214
|
+
short = shorten(decoded)
|
|
215
|
+
short == decoded ? value : short
|
|
216
|
+
rescue ArgumentError
|
|
217
|
+
value
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
# ─── JSON as JavaScript writes it ────────────────────────────────────────────
|
|
221
|
+
|
|
222
|
+
# JSON.stringify(value): no spaces, numbers as JavaScript writes them (1.0 is 1, 1e21 is 1e+21).
|
|
223
|
+
def stringify(value)
|
|
224
|
+
case value
|
|
225
|
+
when nil then "null"
|
|
226
|
+
when true then "true"
|
|
227
|
+
when false then "false"
|
|
228
|
+
when Integer then value.to_s
|
|
229
|
+
when Float then js_number(value)
|
|
230
|
+
when String then JSON.generate(value)
|
|
231
|
+
when Hash then "{#{value.map { |k, v| "#{JSON.generate(k.to_s)}:#{stringify(v)}" }.join(",")}}"
|
|
232
|
+
when Array then "[#{value.map { |v| stringify(v) }.join(",")}]"
|
|
233
|
+
else JSON.generate(value.to_s)
|
|
234
|
+
end
|
|
235
|
+
end
|
|
236
|
+
|
|
237
|
+
def js_number(value)
|
|
238
|
+
return "null" if value.nan? || value.infinite?
|
|
239
|
+
return "0" if value.zero?
|
|
240
|
+
return value.to_i.to_s if value == value.floor && value.abs < 1e21
|
|
241
|
+
|
|
242
|
+
text = value.to_s
|
|
243
|
+
return text unless text.include?("e")
|
|
244
|
+
|
|
245
|
+
mantissa, exponent = text.split("e")
|
|
246
|
+
exp = exponent.to_i
|
|
247
|
+
return BigDecimal(text).to_s("F").sub(/\.0\z/, "") if exp > -7 && exp < 21
|
|
248
|
+
|
|
249
|
+
"#{mantissa.sub(/\.0\z/, '')}e#{exp.positive? ? '+' : '-'}#{exp.abs}"
|
|
250
|
+
end
|
|
251
|
+
|
|
252
|
+
# ─── the attribute value ─────────────────────────────────────────────────────
|
|
253
|
+
|
|
254
|
+
# A body as it is sent: redacted, then cut to max_size. cut says the text was already cut
|
|
255
|
+
# short where it was read. nil for an empty body. Returns [value, truncated].
|
|
256
|
+
def format_body(text, content_type, settings, cut = false)
|
|
257
|
+
return nil if text.nil?
|
|
258
|
+
|
|
259
|
+
body = text.to_s.dup.force_encoding(Encoding::UTF_8).scrub("\uFFFD")
|
|
260
|
+
trimmed = body.strip
|
|
261
|
+
return nil if trimmed.empty?
|
|
262
|
+
|
|
263
|
+
if form?(content_type)
|
|
264
|
+
body = redact_form(body, settings.redact_keys)
|
|
265
|
+
else
|
|
266
|
+
parsed = nil
|
|
267
|
+
ok = false
|
|
268
|
+
if !cut && "{[".include?(trimmed[0])
|
|
269
|
+
begin
|
|
270
|
+
parsed = JSON.parse(trimmed)
|
|
271
|
+
ok = true
|
|
272
|
+
rescue JSON::ParserError, SystemStackError, EncodingError
|
|
273
|
+
ok = false
|
|
274
|
+
end
|
|
275
|
+
end
|
|
276
|
+
body = ok ? stringify(redact_value(parsed, settings.redact_keys)) : redact_loose(body, settings.redact_keys)
|
|
277
|
+
end
|
|
278
|
+
length = utf16_length(body)
|
|
279
|
+
truncated = cut || length > settings.max_size
|
|
280
|
+
[length > settings.max_size ? utf16_slice(body, settings.max_size) : body, truncated]
|
|
281
|
+
end
|
|
282
|
+
|
|
283
|
+
def utf16_length(text) = text.length + text.each_char.count { |c| c.ord > 0xFFFF }
|
|
284
|
+
|
|
285
|
+
def utf16_slice(text, limit)
|
|
286
|
+
count = 0
|
|
287
|
+
text.each_char.with_index do |c, i|
|
|
288
|
+
count += c.ord > 0xFFFF ? 2 : 1
|
|
289
|
+
return text[0, i] if count > limit
|
|
290
|
+
end
|
|
291
|
+
text
|
|
292
|
+
end
|
|
293
|
+
|
|
294
|
+
# The span attributes for one side of a request ("request" or "response").
|
|
295
|
+
def body_attributes(side, text, content_type, settings, cut = false)
|
|
296
|
+
return {} unless textual?(content_type) || multipart?(content_type)
|
|
297
|
+
|
|
298
|
+
formatted = format_body(text, content_type, settings, cut)
|
|
299
|
+
return {} unless formatted
|
|
300
|
+
|
|
301
|
+
body_key, truncated_key, type_key = ATTRS.fetch(side)
|
|
302
|
+
out = { body_key => formatted[0] }
|
|
303
|
+
out[truncated_key] = true if formatted[1]
|
|
304
|
+
out[type_key] = content_type if content_type && !content_type.to_s.empty?
|
|
305
|
+
out
|
|
306
|
+
end
|
|
307
|
+
|
|
308
|
+
# The attributes for a body that isn't text: what it was, not what it said.
|
|
309
|
+
def described_attributes(side, content_type, size = nil, partial = false)
|
|
310
|
+
body_key, _, type_key = ATTRS.fetch(side)
|
|
311
|
+
out = { body_key => describe_body(content_type, size, partial) }
|
|
312
|
+
out[type_key] = content_type if content_type && !content_type.to_s.empty?
|
|
313
|
+
out
|
|
314
|
+
end
|
|
315
|
+
|
|
316
|
+
# A body never read (a streamed response): text is [stream · application/json · 60 B].
|
|
317
|
+
def unread_attributes(side, content_type, size)
|
|
318
|
+
return described_attributes(side, content_type, size) unless textual?(content_type)
|
|
319
|
+
|
|
320
|
+
parts = ["stream"]
|
|
321
|
+
kind = mime(content_type)
|
|
322
|
+
parts << kind unless kind.empty?
|
|
323
|
+
parts << format_size(size) if size&.positive?
|
|
324
|
+
body_key, _, type_key = ATTRS.fetch(side)
|
|
325
|
+
out = { body_key => "[#{parts.join(' · ')}]" }
|
|
326
|
+
out[type_key] = content_type if content_type && !content_type.to_s.empty?
|
|
327
|
+
out
|
|
328
|
+
end
|
|
329
|
+
|
|
330
|
+
# ─── URLs ────────────────────────────────────────────────────────────────────
|
|
331
|
+
|
|
332
|
+
# A URL as it is recorded: secret-looking query values become [Filtered], user:password@ dropped.
|
|
333
|
+
def redact_url(url, extra = [])
|
|
334
|
+
return url if url.nil? || url.empty?
|
|
335
|
+
|
|
336
|
+
out = url.sub(%r{\A([a-z][a-z0-9+.-]*://)[^/?#@]*@}i, '\1')
|
|
337
|
+
q = out.index("?")
|
|
338
|
+
return out unless q
|
|
339
|
+
|
|
340
|
+
hash_at = out.index("#", q)
|
|
341
|
+
query = hash_at ? out[(q + 1)...hash_at] : out[(q + 1)..]
|
|
342
|
+
redacted = redact_query(query, extra)
|
|
343
|
+
redacted == query ? out : "#{out[0..q]}#{redacted}#{hash_at ? out[hash_at..] : ''}"
|
|
344
|
+
end
|
|
345
|
+
|
|
346
|
+
# A query string (without ?) with secret-looking values filtered.
|
|
347
|
+
def redact_query(query, extra = [])
|
|
348
|
+
return query if query.nil? || query.empty?
|
|
349
|
+
|
|
350
|
+
query.split("&", -1).map do |pair|
|
|
351
|
+
eq = pair.index("=")
|
|
352
|
+
next pair unless eq
|
|
353
|
+
|
|
354
|
+
key = begin
|
|
355
|
+
decode_component(pair[0...eq])
|
|
356
|
+
rescue ArgumentError
|
|
357
|
+
pair[0...eq]
|
|
358
|
+
end
|
|
359
|
+
secret = secret_key?(key, extra) || URL_EXACT.include?(normalize(key))
|
|
360
|
+
secret && eq < pair.length - 1 ? "#{pair[0...eq]}=#{FILTERED}" : pair
|
|
361
|
+
end.join("&")
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
DEFAULT_PORTS = { "http" => 80, "https" => 443, "ws" => 80, "wss" => 443, "ftp" => 21 }.freeze
|
|
365
|
+
|
|
366
|
+
# A bare host (with or without a port) matches that host; a string with a scheme matches as a
|
|
367
|
+
# prefix; a Regexp is searched in the URL.
|
|
368
|
+
def url_matches?(url, patterns)
|
|
369
|
+
host = hostname = ""
|
|
370
|
+
begin
|
|
371
|
+
uri = URI.parse(url)
|
|
372
|
+
hostname = uri.host.to_s.downcase
|
|
373
|
+
host = uri.port.nil? || DEFAULT_PORTS[uri.scheme.to_s.downcase] == uri.port ? hostname : "#{hostname}:#{uri.port}"
|
|
374
|
+
rescue URI::Error
|
|
375
|
+
nil
|
|
376
|
+
end
|
|
377
|
+
patterns.any? do |pattern|
|
|
378
|
+
if pattern.is_a?(Regexp) then pattern.match?(url)
|
|
379
|
+
elsif pattern.to_s.empty? then false
|
|
380
|
+
elsif pattern.include?("://") then url.start_with?(pattern)
|
|
381
|
+
else [host, hostname].include?(pattern)
|
|
382
|
+
end
|
|
383
|
+
end
|
|
384
|
+
end
|
|
385
|
+
end
|
|
386
|
+
end
|