rails-profiler 0.30.6 → 0.30.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +81 -0
- data/app/assets/builds/profiler.js +15 -2
- data/app/controllers/profiler/api/env_vars_controller.rb +24 -6
- data/lib/profiler/collectors/database_collector.rb +5 -1
- data/lib/profiler/collectors/env_collector.rb +1 -1
- data/lib/profiler/collectors/mailer_collector.rb +13 -1
- data/lib/profiler/collectors/request_collector.rb +1 -1
- data/lib/profiler/configuration.rb +22 -0
- data/lib/profiler/env_override_store.rb +131 -32
- data/lib/profiler/explain_runner.rb +4 -0
- data/lib/profiler/instrumentation/net_http_instrumentation.rb +6 -4
- data/lib/profiler/job_profiler.rb +2 -1
- data/lib/profiler/mcp/tools/list_env_vars.rb +4 -3
- data/lib/profiler/mcp/tools/reset_env_var.rb +3 -4
- data/lib/profiler/mcp/tools/set_env_var.rb +5 -0
- data/lib/profiler/models/profile.rb +9 -3
- data/lib/profiler/railtie.rb +7 -4
- data/lib/profiler/redaction.rb +421 -0
- data/lib/profiler/version.rb +1 -1
- data/lib/profiler.rb +1 -0
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7edf1c383b05c79e4013d76c96fa57077b376f3a312c27aa355927ca1bb7e362
|
|
4
|
+
data.tar.gz: b1b9b6bd25f7f739c14d2afe72309fac7455e8f2c62b5e7d81e206a09b8ba4fa
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ca0ca79568606603601b5b33aa3ee69fb54d6497e8c8e2087f45cb8d78092ad0264ed8a8794472955e6b5608f743d198de31fcdf63493e40b7f538fd4c4d8298
|
|
7
|
+
data.tar.gz: fd3f7cb5326209b731f11abbca7e7e0b3b226ce019ea62f32636a137d1d402c26f692dc11754e27d3144bae0814524ea7be282031d8adacfe3bda5a97d3d4cc9
|
data/CHANGELOG.md
CHANGED
|
@@ -19,6 +19,87 @@ every commit of every tag interval is accounted for one way or the other.
|
|
|
19
19
|
|
|
20
20
|
## [Unreleased]
|
|
21
21
|
|
|
22
|
+
## [0.30.8] - 2026-10-04
|
|
23
|
+
|
|
24
|
+
<!-- stamped -->
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
|
|
28
|
+
- **Env overrides:** Keep every override when several are set, deleted or reset at the same
|
|
29
|
+
time, from threads of the web process or from Sidekiq processes. Each writer read the file,
|
|
30
|
+
changed it and saved it over the others' changes, and a reader could see a half-written file as
|
|
31
|
+
no override at all. The file is now changed under a lock (`env_overrides.json.lock`, next to
|
|
32
|
+
it) and replaced in one step.
|
|
33
|
+
|
|
34
|
+
### Security
|
|
35
|
+
|
|
36
|
+
- **Env overrides:** Stop applying the environment variable overrides saved from the Env tab or
|
|
37
|
+
the MCP env tools in production, and while the profiler is disabled. They were written into
|
|
38
|
+
`ENV` at every boot whatever `enabled` said and whatever the environment, so an
|
|
39
|
+
`env_overrides.json` left under `tmp/` and shipped with a deployment changed the environment of
|
|
40
|
+
the production application, and an application with `config.enabled = false` still had them
|
|
41
|
+
applied without a word. Sidekiq jobs and console evaluations, which apply them again, follow the
|
|
42
|
+
same rules. When overrides are left out, the boot logs one warning to `Rails.logger` with the
|
|
43
|
+
file, the number of overrides and the reason, never a name or a value.
|
|
44
|
+
- **Env overrides:** Apply them at boot once the application's `config/initializers` have run,
|
|
45
|
+
where `enabled` is decided, instead of before. An initializer of the application that reads
|
|
46
|
+
`ENV` while it runs no longer sees them; requests, jobs, eager loading and `after_initialize`
|
|
47
|
+
still do.
|
|
48
|
+
- **Env overrides:** Resetting one override, or all of them, from the Env tab or the MCP env
|
|
49
|
+
tools no longer writes into `ENV` the "original" values saved in `env_overrides.json` in
|
|
50
|
+
production, nor while the profiler is disabled: those values come from the machine that wrote
|
|
51
|
+
the file, so a reset could put a development value into a production variable, or delete it.
|
|
52
|
+
There, a reset restores only the variables the running process changed itself, to the values
|
|
53
|
+
they had before.
|
|
54
|
+
- **Env overrides:** In production, or with the profiler disabled, typing back in the Env tab the
|
|
55
|
+
original value the overrides file shows for a variable now records the running value first, so
|
|
56
|
+
a later reset puts it back instead of keeping the file's value. A reset also restores a variable
|
|
57
|
+
this process changed even when its entry has gone from the file. The `reset_env_var` MCP tool
|
|
58
|
+
now says whether the original value was restored in the process or `ENV` was left unchanged,
|
|
59
|
+
and no longer prints a value, masked or not.
|
|
60
|
+
- **Upgrading:** to apply the env overrides while the profiler is disabled outside production,
|
|
61
|
+
as before, set `config.apply_env_overrides_when_disabled = true`. Nothing applies them in
|
|
62
|
+
production. If an initializer of yours needs them, call `Profiler.env_override_store.apply!` at
|
|
63
|
+
the end of `config/initializers/profiler.rb`; initializers loaded after it then see them, under
|
|
64
|
+
the same rules.
|
|
65
|
+
|
|
66
|
+
## [0.30.7] - 2026-10-04
|
|
67
|
+
|
|
68
|
+
<!-- stamped -->
|
|
69
|
+
|
|
70
|
+
### Security
|
|
71
|
+
|
|
72
|
+
- **Sensitive data:** Mask sensitive values with `[FILTERED]` before a profile is stored. Until
|
|
73
|
+
now `Authorization`, `Cookie` and `Set-Cookie` headers, raw request and response bodies
|
|
74
|
+
(passwords included), outbound request headers and bodies, and the whole of `ENV` were stored in
|
|
75
|
+
clear, and params lost only four exact root keys. The profiler now builds one filter from your
|
|
76
|
+
`Rails.application.config.filter_parameters` plus its own `config.filter_parameters`, with Rails
|
|
77
|
+
semantics (nested keys, any case, regexps, procs), and applies it to params and route params,
|
|
78
|
+
JSON, NDJSON and form bodies, incoming, response and outbound headers (the query string of
|
|
79
|
+
`Referer`, `Location` and `Content-Location` included), outbound URLs, SQL binds of filtered
|
|
80
|
+
columns, job arguments and mailer arguments. `multipart` bodies, and JSON bodies that cannot be
|
|
81
|
+
parsed and in which a filter matches, are masked entirely; HTML and other unstructured bodies are
|
|
82
|
+
kept. A params key that matches the filter is now kept with a masked value instead of being
|
|
83
|
+
removed.
|
|
84
|
+
- **Sensitive data:** `ENV` values are shown only for the variables in `config.env_allowlist`, in
|
|
85
|
+
the Env tab, the `env_vars` endpoint and the `list_env_vars`, `reset_env_var` and `get_profile`
|
|
86
|
+
MCP tools alike. `[FILTERED]` is refused as a value by the `env_vars` endpoint (422), the
|
|
87
|
+
`set_env_var` MCP tool and the Env tab import, so re-importing an export cannot overwrite a
|
|
88
|
+
secret. EXPLAIN is refused, with a message, for a query whose bind values were masked.
|
|
89
|
+
- **Sensitive data:** Procs of `filter_parameters` run once per string value, with the original
|
|
90
|
+
params for a proc of three arguments, as in Rails; other objects, Active Record models included,
|
|
91
|
+
are neither copied nor passed to them. When the filter fails, the value is masked rather than
|
|
92
|
+
the error reaching the application, and the error class is logged once, without the value. A
|
|
93
|
+
JSON body in whose text no filter matches is not parsed.
|
|
94
|
+
- **Upgrade note:** To get the previous behaviour back, set `config.redact_sensitive_data = false`
|
|
95
|
+
(no masking) and `config.env_allowlist = :all` (every `ENV` value); `config.filter_parameters = []`
|
|
96
|
+
keeps masking but relies on your application's list alone. Profiles stored before the upgrade
|
|
97
|
+
are not rewritten and keep their values in clear until they rotate out: clear them after
|
|
98
|
+
upgrading with `bin/rails runner 'Profiler.storage.clear'` (every storage backend), the
|
|
99
|
+
`clear_profiles` MCP tool or the dashboard; the memory store is emptied by a restart. Update
|
|
100
|
+
every node of a cluster: a node still on an older version returns its data in clear to the
|
|
101
|
+
master.
|
|
102
|
+
|
|
22
103
|
## [0.30.6] - 2026-10-03
|
|
23
104
|
|
|
24
105
|
<!-- stamped -->
|
|
@@ -4499,6 +4499,7 @@
|
|
|
4499
4499
|
}
|
|
4500
4500
|
|
|
4501
4501
|
// app/assets/typescript/profiler/components/dashboard/tabs/EnvTab.tsx
|
|
4502
|
+
var FILTERED_MASK = "[FILTERED]";
|
|
4502
4503
|
function detectType(value) {
|
|
4503
4504
|
if (value === "") return { label: "empty", color: "var(--profiler-text-muted)" };
|
|
4504
4505
|
if (/^(true|false|yes|no)$/i.test(value)) return { label: "bool", color: "#a78bfa" };
|
|
@@ -4653,8 +4654,13 @@
|
|
|
4653
4654
|
URL.revokeObjectURL(url);
|
|
4654
4655
|
showFlash("success", `Exported ${entries.length} variables`);
|
|
4655
4656
|
};
|
|
4656
|
-
const
|
|
4657
|
+
const importParsed = T2(() => parseEnvFile(importContent), [importContent]);
|
|
4658
|
+
const importPreview = T2(
|
|
4659
|
+
() => Object.fromEntries(Object.entries(importParsed).filter(([, v3]) => v3 !== FILTERED_MASK)),
|
|
4660
|
+
[importParsed]
|
|
4661
|
+
);
|
|
4657
4662
|
const importCount = Object.keys(importPreview).length;
|
|
4663
|
+
const importMaskedCount = Object.keys(importParsed).length - importCount;
|
|
4658
4664
|
const importEnv = async () => {
|
|
4659
4665
|
const entries = Object.entries(importPreview);
|
|
4660
4666
|
if (!entries.length) return;
|
|
@@ -4664,7 +4670,8 @@
|
|
|
4664
4670
|
setVariables((prev) => ({ ...prev, ...importPreview }));
|
|
4665
4671
|
setImportContent("");
|
|
4666
4672
|
setShowImport(false);
|
|
4667
|
-
|
|
4673
|
+
const skipped = importMaskedCount > 0 ? `, skipped ${importMaskedCount} masked` : "";
|
|
4674
|
+
showFlash("success", `Imported ${entries.length} variables${skipped}`);
|
|
4668
4675
|
} catch (e3) {
|
|
4669
4676
|
showFlash("error", e3.message ?? "Import failed");
|
|
4670
4677
|
} finally {
|
|
@@ -4933,6 +4940,12 @@
|
|
|
4933
4940
|
importCount !== 1 ? "s" : "",
|
|
4934
4941
|
" to import"
|
|
4935
4942
|
] }),
|
|
4943
|
+
importMaskedCount > 0 && /* @__PURE__ */ u3("span", { class: "profiler-text--xs profiler-text--muted", children: [
|
|
4944
|
+
importMaskedCount,
|
|
4945
|
+
" masked (",
|
|
4946
|
+
FILTERED_MASK,
|
|
4947
|
+
") skipped"
|
|
4948
|
+
] }),
|
|
4936
4949
|
/* @__PURE__ */ u3(
|
|
4937
4950
|
"button",
|
|
4938
4951
|
{
|
|
@@ -4,8 +4,8 @@ module Profiler
|
|
|
4
4
|
module Api
|
|
5
5
|
class EnvVarsController < ApplicationController
|
|
6
6
|
def show
|
|
7
|
-
variables =
|
|
8
|
-
overrides = Profiler.env_override_store.all_overrides
|
|
7
|
+
variables = Profiler::Redaction.env_snapshot
|
|
8
|
+
overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
|
|
9
9
|
render json: { variables: variables, total: variables.size, overrides: overrides }
|
|
10
10
|
end
|
|
11
11
|
|
|
@@ -19,19 +19,27 @@ module Profiler
|
|
|
19
19
|
|
|
20
20
|
value = params[:value]
|
|
21
21
|
|
|
22
|
+
if Profiler::Redaction.mask?(value)
|
|
23
|
+
render json: { error: "#{Profiler::Redaction::MASK} is the mask the profiler shows in place of a " \
|
|
24
|
+
"hidden value, not a value; #{key} was left unchanged" },
|
|
25
|
+
status: :unprocessable_entity
|
|
26
|
+
return
|
|
27
|
+
end
|
|
28
|
+
|
|
22
29
|
if value.nil? || value.to_s.empty?
|
|
23
30
|
Profiler.env_override_store.delete(key)
|
|
24
31
|
ENV.delete(key)
|
|
25
|
-
render json: { key: key, value: nil, deleted: true, override:
|
|
32
|
+
render json: { key: key, value: nil, deleted: true, override: redacted_override(key) }
|
|
26
33
|
else
|
|
27
34
|
current_original = Profiler.env_override_store.all_overrides.dig(key, "original")
|
|
28
|
-
|
|
35
|
+
# Where the overrides are blocked, set records this process's value, which reset would not.
|
|
36
|
+
if current_original == value.to_s && Profiler.env_override_store.blocked_reason.nil?
|
|
29
37
|
Profiler.env_override_store.reset(key)
|
|
30
38
|
else
|
|
31
39
|
Profiler.env_override_store.set(key, value.to_s)
|
|
32
40
|
end
|
|
33
41
|
ENV[key] = value.to_s
|
|
34
|
-
render json: { key: key, value:
|
|
42
|
+
render json: { key: key, value: redacted_value(key), override: redacted_override(key) }
|
|
35
43
|
end
|
|
36
44
|
end
|
|
37
45
|
|
|
@@ -44,13 +52,23 @@ module Profiler
|
|
|
44
52
|
end
|
|
45
53
|
|
|
46
54
|
Profiler.env_override_store.reset(key)
|
|
47
|
-
render json: { key: key, value:
|
|
55
|
+
render json: { key: key, value: redacted_value(key), reset: true }
|
|
48
56
|
end
|
|
49
57
|
|
|
50
58
|
def reset_all
|
|
51
59
|
Profiler.env_override_store.reset_all
|
|
52
60
|
render json: { reset: true }
|
|
53
61
|
end
|
|
62
|
+
|
|
63
|
+
private
|
|
64
|
+
|
|
65
|
+
def redacted_value(key)
|
|
66
|
+
ENV[key].nil? ? nil : Profiler::Redaction.env_value(key, ENV[key])
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def redacted_override(key)
|
|
70
|
+
Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)[key]
|
|
71
|
+
end
|
|
54
72
|
end
|
|
55
73
|
end
|
|
56
74
|
end
|
|
@@ -96,11 +96,15 @@ module Profiler
|
|
|
96
96
|
|
|
97
97
|
binds.map do |bind|
|
|
98
98
|
if bind.respond_to?(:value)
|
|
99
|
-
bind.
|
|
99
|
+
name = bind.name if bind.respond_to?(:name)
|
|
100
|
+
Profiler::Redaction.filter_named(name, bind.value)
|
|
100
101
|
else
|
|
101
102
|
bind
|
|
102
103
|
end
|
|
103
104
|
end
|
|
105
|
+
rescue StandardError
|
|
106
|
+
# Never let the profiler raise into the application's query.
|
|
107
|
+
binds.map { Profiler::Redaction::MASK }
|
|
104
108
|
end
|
|
105
109
|
|
|
106
110
|
def extract_backtrace
|
|
@@ -199,7 +199,19 @@ module Profiler
|
|
|
199
199
|
klass = Object.const_get(mailer_class)
|
|
200
200
|
params = klass.instance_method(action).parameters
|
|
201
201
|
params.each_with_index.each_with_object({}) do |((_, name), i), h|
|
|
202
|
-
h[name.to_s] =
|
|
202
|
+
h[name.to_s] = if Profiler::Redaction.sensitive_key?(name)
|
|
203
|
+
Profiler::Redaction::MASK
|
|
204
|
+
else
|
|
205
|
+
value = args[i]
|
|
206
|
+
# A hash or array goes through the filter by its keys, anything
|
|
207
|
+
# else by the parameter name: never both, so a proc runs once.
|
|
208
|
+
filtered = if value.is_a?(Hash) || value.is_a?(Array)
|
|
209
|
+
Profiler::Redaction.filter_value(value)
|
|
210
|
+
else
|
|
211
|
+
Profiler::Redaction.filter_named(name, value)
|
|
212
|
+
end
|
|
213
|
+
serialize_assign(filtered)
|
|
214
|
+
end
|
|
203
215
|
end
|
|
204
216
|
rescue StandardError
|
|
205
217
|
{}
|
|
@@ -91,7 +91,7 @@ module Profiler
|
|
|
91
91
|
{
|
|
92
92
|
route_name: route_name ? "#{route_name}_path" : nil,
|
|
93
93
|
route_pattern: matched_route&.path&.spec&.to_s&.sub(/\(\.:format\)$/, ""),
|
|
94
|
-
route_params: route_params,
|
|
94
|
+
route_params: Profiler::Redaction.filter_hash(route_params),
|
|
95
95
|
controller_action: controller_action
|
|
96
96
|
}
|
|
97
97
|
rescue StandardError
|
|
@@ -9,6 +9,22 @@ module Profiler
|
|
|
9
9
|
FORGERY_PROTECTION_HEADER = "X-Profiler-Request"
|
|
10
10
|
|
|
11
11
|
class Configuration
|
|
12
|
+
# Added to the application's own config.filter_parameters: the list of the
|
|
13
|
+
# Rails 7.1 application template, without :email.
|
|
14
|
+
DEFAULT_FILTER_PARAMETERS = %i[
|
|
15
|
+
passw secret token _key crypt salt certificate otp ssn cvv cvc
|
|
16
|
+
].freeze
|
|
17
|
+
|
|
18
|
+
# ENV variables whose values the profiler shows; every other one is listed
|
|
19
|
+
# with its value masked.
|
|
20
|
+
DEFAULT_ENV_ALLOWLIST = %w[
|
|
21
|
+
RAILS_ENV RACK_ENV NODE_ENV RAILS_LOG_LEVEL RAILS_LOG_TO_STDOUT RAILS_SERVE_STATIC_FILES
|
|
22
|
+
RAILS_MAX_THREADS RAILS_MIN_THREADS WEB_CONCURRENCY PORT PIDFILE
|
|
23
|
+
LANG LANGUAGE LC_ALL LC_CTYPE TZ HOME PWD PATH SHELL USER HOSTNAME TERM
|
|
24
|
+
RUBY_VERSION RUBYOPT RUBY_YJIT_ENABLE MALLOC_ARENA_MAX
|
|
25
|
+
BUNDLE_GEMFILE BUNDLE_PATH BUNDLE_WITHOUT GEM_HOME GEM_PATH BOOTSNAP_CACHE_DIR
|
|
26
|
+
].freeze
|
|
27
|
+
|
|
12
28
|
attr_accessor :enabled, :storage_options, :collectors,
|
|
13
29
|
:skip_paths, :slow_query_threshold, :max_queries_warning,
|
|
14
30
|
:track_memory, :memory_warning_threshold,
|
|
@@ -19,9 +35,11 @@ module Profiler
|
|
|
19
35
|
:track_http, :slow_http_threshold, :http_skip_hosts, :http_backtrace_depth,
|
|
20
36
|
:track_jobs,
|
|
21
37
|
:track_console,
|
|
38
|
+
:apply_env_overrides_when_disabled,
|
|
22
39
|
:track_tests,
|
|
23
40
|
:track_mailers, :capture_mail_body, :sanitize_mailer_recipients, :mailer_skip_actions,
|
|
24
41
|
:compress_bodies, :compress_body_threshold,
|
|
42
|
+
:redact_sensitive_data, :filter_parameters, :env_allowlist,
|
|
25
43
|
:name, :master_url, :self_url,
|
|
26
44
|
:cluster_heartbeat_interval, :cluster_offline_threshold
|
|
27
45
|
|
|
@@ -60,6 +78,7 @@ module Profiler
|
|
|
60
78
|
@http_backtrace_depth = 40
|
|
61
79
|
@track_jobs = true
|
|
62
80
|
@track_console = true
|
|
81
|
+
@apply_env_overrides_when_disabled = false
|
|
63
82
|
@track_tests = false
|
|
64
83
|
@track_mailers = true
|
|
65
84
|
@capture_mail_body = false
|
|
@@ -67,6 +86,9 @@ module Profiler
|
|
|
67
86
|
@mailer_skip_actions = []
|
|
68
87
|
@compress_bodies = true
|
|
69
88
|
@compress_body_threshold = 10 * 1024 # 10 KB
|
|
89
|
+
@redact_sensitive_data = true
|
|
90
|
+
@filter_parameters = DEFAULT_FILTER_PARAMETERS.dup
|
|
91
|
+
@env_allowlist = DEFAULT_ENV_ALLOWLIST.dup
|
|
70
92
|
@tmp_path = nil
|
|
71
93
|
@name = nil
|
|
72
94
|
@master_url = nil
|
|
@@ -8,63 +8,108 @@ module Profiler
|
|
|
8
8
|
DELETED_SENTINEL = "__profiler_deleted__"
|
|
9
9
|
RESTORE_SENTINEL = "__profiler_restore__"
|
|
10
10
|
|
|
11
|
+
# Why the persisted overrides are left out of ENV, as logged at boot.
|
|
12
|
+
BLOCKED_REASONS = {
|
|
13
|
+
production: "persisted overrides are never applied in production; " \
|
|
14
|
+
"delete the file if it was deployed by mistake",
|
|
15
|
+
disabled: "the profiler is disabled; set " \
|
|
16
|
+
"config.apply_env_overrides_when_disabled = true to apply them anyway"
|
|
17
|
+
}.freeze
|
|
18
|
+
|
|
11
19
|
# File format: { "KEY" => { "value" => "...", "original" => "..." } }
|
|
12
20
|
# Sentinels for "value":
|
|
13
21
|
# DELETED_SENTINEL → ENV.delete(key)
|
|
14
22
|
# RESTORE_SENTINEL → restore ENV[key] to "original" (cleaned up after apply!)
|
|
15
23
|
|
|
24
|
+
def initialize
|
|
25
|
+
# What ENV held in this process before set, delete or apply! changed a key. Where the
|
|
26
|
+
# overrides are blocked, a reset restores these and never the file's "original" values,
|
|
27
|
+
# which come from whatever machine wrote the file.
|
|
28
|
+
@process_originals = {}
|
|
29
|
+
@process_originals_lock = Mutex.new
|
|
30
|
+
end
|
|
31
|
+
|
|
16
32
|
def set(key, value)
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
33
|
+
remember_process_original(key)
|
|
34
|
+
update_overrides do |overrides|
|
|
35
|
+
overrides[key] = { "value" => value.to_s, "original" => original_for(overrides, key) }
|
|
36
|
+
end
|
|
21
37
|
rescue => e
|
|
22
38
|
warn "[Profiler] EnvOverrideStore: failed to set #{key}: #{e.message}"
|
|
23
39
|
end
|
|
24
40
|
|
|
25
41
|
def delete(key)
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
42
|
+
remember_process_original(key)
|
|
43
|
+
update_overrides do |overrides|
|
|
44
|
+
overrides[key] = { "value" => DELETED_SENTINEL, "original" => original_for(overrides, key) }
|
|
45
|
+
end
|
|
30
46
|
rescue => e
|
|
31
47
|
warn "[Profiler] EnvOverrideStore: failed to delete #{key}: #{e.message}"
|
|
32
48
|
end
|
|
33
49
|
|
|
50
|
+
# Returns whether ENV was written in this process.
|
|
34
51
|
def reset(key)
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
overrides[key] = { "value" => RESTORE_SENTINEL, "original" => original }
|
|
42
|
-
save_overrides(overrides)
|
|
52
|
+
entry = nil
|
|
53
|
+
update_overrides do |overrides|
|
|
54
|
+
entry = overrides[key]
|
|
55
|
+
# Keep a RESTORE entry so Sidekiq workers pick it up on next job
|
|
56
|
+
overrides[key] = { "value" => RESTORE_SENTINEL, "original" => entry["original"] } if entry
|
|
57
|
+
end
|
|
43
58
|
|
|
44
59
|
# Apply immediately to the current (web) process
|
|
45
|
-
|
|
60
|
+
restore_env(key, entry, blocked: blocked_reason)
|
|
46
61
|
rescue => e
|
|
47
62
|
warn "[Profiler] EnvOverrideStore: failed to reset #{key}: #{e.message}"
|
|
48
63
|
end
|
|
49
64
|
|
|
50
65
|
def reset_all
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
# Leave a RESTORE sentinel for Sidekiq workers to pick up
|
|
59
|
-
restore_overrides[key] = { "value" => RESTORE_SENTINEL, "original" => original }
|
|
66
|
+
originals = {}
|
|
67
|
+
update_overrides do |overrides|
|
|
68
|
+
overrides.each do |key, entry|
|
|
69
|
+
originals[key] = entry.is_a?(Hash) ? entry["original"] : nil
|
|
70
|
+
# Leave a RESTORE sentinel for Sidekiq workers to pick up
|
|
71
|
+
overrides[key] = { "value" => RESTORE_SENTINEL, "original" => originals[key] }
|
|
72
|
+
end
|
|
60
73
|
end
|
|
61
74
|
|
|
62
|
-
|
|
75
|
+
# Apply immediately to the current (web) process
|
|
76
|
+
blocked = blocked_reason
|
|
77
|
+
originals.each { |key, original| restore_env(key, { "original" => original }, blocked: blocked) }
|
|
63
78
|
rescue => e
|
|
64
79
|
warn "[Profiler] EnvOverrideStore: failed to reset all: #{e.message}"
|
|
65
80
|
end
|
|
66
81
|
|
|
82
|
+
# Why the persisted overrides must not reach ENV in this process, or nil when they may.
|
|
83
|
+
def blocked_reason
|
|
84
|
+
return :production if defined?(Rails) && Rails.respond_to?(:env) && Rails.env.production?
|
|
85
|
+
|
|
86
|
+
config = Profiler.configuration
|
|
87
|
+
return :disabled unless config.enabled || config.apply_env_overrides_when_disabled
|
|
88
|
+
|
|
89
|
+
nil
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
# Applies the overrides at boot, or says once in the log why they are left out. The message
|
|
93
|
+
# carries the file and the number of keys only, never a key or a value.
|
|
94
|
+
def apply_at_boot!(logger)
|
|
95
|
+
reason = blocked_reason
|
|
96
|
+
return apply! unless reason
|
|
97
|
+
return if @boot_warning_logged
|
|
98
|
+
|
|
99
|
+
count = all_overrides.size
|
|
100
|
+
return if count.zero?
|
|
101
|
+
|
|
102
|
+
@boot_warning_logged = true
|
|
103
|
+
noun, verb = count == 1 ? %w[override was] : %w[overrides were]
|
|
104
|
+
logger&.warn("[Profiler] #{count} persisted environment #{noun} in #{override_file_path} " \
|
|
105
|
+
"#{verb} not applied: #{BLOCKED_REASONS.fetch(reason)}.")
|
|
106
|
+
rescue => e
|
|
107
|
+
warn "[Profiler] EnvOverrideStore: failed to check overrides at boot: #{e.message}"
|
|
108
|
+
end
|
|
109
|
+
|
|
67
110
|
def apply!
|
|
111
|
+
return if blocked_reason
|
|
112
|
+
|
|
68
113
|
overrides = load_overrides
|
|
69
114
|
restore_keys = []
|
|
70
115
|
|
|
@@ -72,6 +117,8 @@ module Profiler
|
|
|
72
117
|
value = entry.is_a?(Hash) ? entry["value"] : entry
|
|
73
118
|
original = entry.is_a?(Hash) ? entry["original"] : nil
|
|
74
119
|
|
|
120
|
+
remember_process_original(key) unless value == RESTORE_SENTINEL
|
|
121
|
+
|
|
75
122
|
case value
|
|
76
123
|
when DELETED_SENTINEL
|
|
77
124
|
ENV.delete(key)
|
|
@@ -84,8 +131,13 @@ module Profiler
|
|
|
84
131
|
end
|
|
85
132
|
|
|
86
133
|
if restore_keys.any?
|
|
87
|
-
|
|
88
|
-
|
|
134
|
+
# Read again under the lock: another writer may have changed the file since.
|
|
135
|
+
update_overrides do |current|
|
|
136
|
+
restore_keys.each do |k|
|
|
137
|
+
entry = current[k]
|
|
138
|
+
current.delete(k) if entry.is_a?(Hash) && entry["value"] == RESTORE_SENTINEL
|
|
139
|
+
end
|
|
140
|
+
end
|
|
89
141
|
end
|
|
90
142
|
rescue => e
|
|
91
143
|
warn "[Profiler] EnvOverrideStore: failed to apply overrides: #{e.message}"
|
|
@@ -105,13 +157,34 @@ module Profiler
|
|
|
105
157
|
end
|
|
106
158
|
|
|
107
159
|
def clear
|
|
108
|
-
FileUtils.rm_f(override_file_path)
|
|
160
|
+
with_lock { FileUtils.rm_f(override_file_path) }
|
|
109
161
|
rescue => e
|
|
110
162
|
warn "[Profiler] EnvOverrideStore: failed to clear: #{e.message}"
|
|
111
163
|
end
|
|
112
164
|
|
|
113
165
|
private
|
|
114
166
|
|
|
167
|
+
def remember_process_original(key)
|
|
168
|
+
@process_originals_lock.synchronize do
|
|
169
|
+
@process_originals[key] = ENV[key] unless @process_originals.key?(key)
|
|
170
|
+
end
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
# Puts a reset key back to its original value and returns whether ENV was written. Where the
|
|
174
|
+
# overrides are blocked, or the file has no entry for the key, only a key this process
|
|
175
|
+
# changed itself is touched, back to the value it had here; ENV is left alone for any other.
|
|
176
|
+
def restore_env(key, file_entry, blocked:)
|
|
177
|
+
changed_here, process_original = @process_originals_lock.synchronize do
|
|
178
|
+
[@process_originals.key?(key), @process_originals.delete(key)]
|
|
179
|
+
end
|
|
180
|
+
from_process = blocked || file_entry.nil?
|
|
181
|
+
return false if from_process && !changed_here
|
|
182
|
+
|
|
183
|
+
original = from_process ? process_original : file_entry["original"]
|
|
184
|
+
original.nil? ? ENV.delete(key) : ENV[key] = original
|
|
185
|
+
true
|
|
186
|
+
end
|
|
187
|
+
|
|
115
188
|
def original_for(overrides, key)
|
|
116
189
|
overrides.key?(key) ? overrides[key]["original"] : ENV[key]
|
|
117
190
|
end
|
|
@@ -129,10 +202,36 @@ module Profiler
|
|
|
129
202
|
{}
|
|
130
203
|
end
|
|
131
204
|
|
|
132
|
-
|
|
205
|
+
# Reads, changes and writes the file under the lock, so that the web process, its threads and
|
|
206
|
+
# the Sidekiq processes never save over each other's changes. The file is removed when the
|
|
207
|
+
# block leaves it empty.
|
|
208
|
+
def update_overrides
|
|
209
|
+
with_lock do
|
|
210
|
+
overrides = load_overrides
|
|
211
|
+
yield overrides
|
|
212
|
+
overrides.empty? ? FileUtils.rm_f(override_file_path) : save_overrides(overrides)
|
|
213
|
+
end
|
|
214
|
+
end
|
|
215
|
+
|
|
216
|
+
# An exclusive flock on a file next to the overrides file. Not reentrant: never nest it.
|
|
217
|
+
def with_lock
|
|
133
218
|
path = override_file_path
|
|
134
219
|
FileUtils.mkdir_p(path.dirname)
|
|
135
|
-
File.
|
|
220
|
+
File.open("#{path}.lock", File::RDWR | File::CREAT, 0o644) do |lock|
|
|
221
|
+
lock.flock(File::LOCK_EX)
|
|
222
|
+
yield
|
|
223
|
+
end
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
# Written to a temporary file then renamed over the old one, so that a reader never sees a
|
|
227
|
+
# half-written file, which would parse as no override at all.
|
|
228
|
+
def save_overrides(overrides)
|
|
229
|
+
path = override_file_path
|
|
230
|
+
tmp = "#{path}.#{Process.pid}.#{Thread.current.object_id}.tmp"
|
|
231
|
+
File.write(tmp, JSON.generate(overrides))
|
|
232
|
+
File.rename(tmp, path)
|
|
233
|
+
ensure
|
|
234
|
+
FileUtils.rm_f(tmp) if tmp
|
|
136
235
|
end
|
|
137
236
|
end
|
|
138
237
|
end
|
|
@@ -28,6 +28,10 @@ module Profiler
|
|
|
28
28
|
query = queries[query_index]
|
|
29
29
|
sql = query["sql"].to_s
|
|
30
30
|
binds = Array(query["binds"])
|
|
31
|
+
if binds.include?(Redaction::MASK)
|
|
32
|
+
raise ArgumentError, "EXPLAIN refused for query #{query_index}: one of its bind values was " \
|
|
33
|
+
"filtered when captured (config.filter_parameters), so the query cannot be rebuilt"
|
|
34
|
+
end
|
|
31
35
|
|
|
32
36
|
conn = ActiveRecord::Base.connection
|
|
33
37
|
adapter = conn.adapter_name.downcase
|
|
@@ -5,6 +5,7 @@ require "base64"
|
|
|
5
5
|
require "zlib"
|
|
6
6
|
require "stringio"
|
|
7
7
|
require "securerandom"
|
|
8
|
+
require_relative "../redaction"
|
|
8
9
|
|
|
9
10
|
module Profiler
|
|
10
11
|
module Instrumentation
|
|
@@ -20,7 +21,7 @@ module Profiler
|
|
|
20
21
|
host = address.to_s
|
|
21
22
|
return super if NetHttpInstrumentation.skip_host?(host)
|
|
22
23
|
|
|
23
|
-
url = build_url(host, port, req.path, use_ssl?)
|
|
24
|
+
url = Redaction.filter_url(build_url(host, port, req.path, use_ssl?))
|
|
24
25
|
req_body = req.body.to_s
|
|
25
26
|
# Fallback: body may be passed as the 2nd argument and only applied
|
|
26
27
|
# to req inside super via req.set_body_internal(body)
|
|
@@ -43,7 +44,7 @@ module Profiler
|
|
|
43
44
|
req_body = ""
|
|
44
45
|
end
|
|
45
46
|
end
|
|
46
|
-
req_headers = req.to_hash.transform_values { |v| v.join(", ") }
|
|
47
|
+
req_headers = Redaction.filter_headers(req.to_hash.transform_values { |v| v.join(", ") })
|
|
47
48
|
req_content_type = req["content-type"].to_s
|
|
48
49
|
processed_req = req_body.empty? ? { body: nil, encoding: "text" } : NetHttpInstrumentation.process_body(req_body, req_content_type)
|
|
49
50
|
|
|
@@ -83,7 +84,7 @@ module Profiler
|
|
|
83
84
|
entry,
|
|
84
85
|
status: response.code.to_i,
|
|
85
86
|
duration: duration,
|
|
86
|
-
response_headers: response.to_hash.transform_values { |v| v.join(", ") },
|
|
87
|
+
response_headers: Redaction.filter_headers(response.to_hash.transform_values { |v| v.join(", ") }),
|
|
87
88
|
response_body: processed_resp[:body],
|
|
88
89
|
response_body_encoding: processed_resp[:encoding],
|
|
89
90
|
response_size: resp_body_raw.bytesize
|
|
@@ -151,7 +152,8 @@ module Profiler
|
|
|
151
152
|
if mime.match?(BINARY_CONTENT_TYPES)
|
|
152
153
|
{ body: Base64.strict_encode64(body.b), encoding: "base64" }
|
|
153
154
|
else
|
|
154
|
-
text =
|
|
155
|
+
text = Redaction.filter_body(body, content_type)
|
|
156
|
+
.encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")
|
|
155
157
|
{ body: text, encoding: "text" }
|
|
156
158
|
end
|
|
157
159
|
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative "models/profile"
|
|
4
4
|
require_relative "current_context"
|
|
5
|
+
require_relative "redaction"
|
|
5
6
|
require_relative "collectors/job_collector"
|
|
6
7
|
require_relative "collectors/database_collector"
|
|
7
8
|
require_relative "collectors/cache_collector"
|
|
@@ -115,7 +116,7 @@ module Profiler
|
|
|
115
116
|
def sanitize_arguments(args)
|
|
116
117
|
return [] unless args
|
|
117
118
|
|
|
118
|
-
args.map do |arg|
|
|
119
|
+
Redaction.filter_value(args).map do |arg|
|
|
119
120
|
case arg
|
|
120
121
|
when String then arg.length > 200 ? "#{arg[0, 200]}..." : arg
|
|
121
122
|
when Numeric, TrueClass, FalseClass, NilClass then arg
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative "../slave_support"
|
|
4
4
|
require_relative "../../env_override_store"
|
|
5
|
+
require_relative "../../redaction"
|
|
5
6
|
|
|
6
7
|
module Profiler
|
|
7
8
|
module MCP
|
|
@@ -19,8 +20,8 @@ module Profiler
|
|
|
19
20
|
filter = params["filter"]&.downcase
|
|
20
21
|
|
|
21
22
|
if include_all
|
|
22
|
-
vars =
|
|
23
|
-
overrides = Profiler.env_override_store.all_overrides
|
|
23
|
+
vars = Profiler::Redaction.env_snapshot
|
|
24
|
+
overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
|
|
24
25
|
|
|
25
26
|
vars = vars.select { |k, _| k.downcase.include?(filter) } if filter
|
|
26
27
|
|
|
@@ -39,7 +40,7 @@ module Profiler
|
|
|
39
40
|
text += "|-----|--------------|------------|\n"
|
|
40
41
|
text += rows.join("\n")
|
|
41
42
|
else
|
|
42
|
-
overrides = Profiler.env_override_store.all_overrides
|
|
43
|
+
overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
|
|
43
44
|
overrides = overrides.select { |k, _| k.downcase.include?(filter) } if filter
|
|
44
45
|
|
|
45
46
|
if overrides.empty?
|
|
@@ -23,11 +23,10 @@ module Profiler
|
|
|
23
23
|
return [{ type: "text", text: "No active override for #{key}." }]
|
|
24
24
|
end
|
|
25
25
|
|
|
26
|
-
|
|
27
|
-
Profiler.env_override_store.reset(key)
|
|
26
|
+
restored = Profiler.env_override_store.reset(key)
|
|
28
27
|
|
|
29
|
-
|
|
30
|
-
[{ type: "text", text: "Reset #{key}
|
|
28
|
+
done = restored ? "original value restored" : "ENV left unchanged"
|
|
29
|
+
[{ type: "text", text: "Reset #{key}: override removed, #{done} in this process." }]
|
|
31
30
|
end
|
|
32
31
|
end
|
|
33
32
|
end
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require_relative "../slave_support"
|
|
4
|
+
require_relative "../../redaction"
|
|
4
5
|
|
|
5
6
|
module Profiler
|
|
6
7
|
module MCP
|
|
@@ -11,6 +12,10 @@ module Profiler
|
|
|
11
12
|
value = params["value"].to_s
|
|
12
13
|
|
|
13
14
|
return [{ type: "text", text: "Error: key cannot be blank." }] if key.empty?
|
|
15
|
+
if Profiler::Redaction.mask?(value)
|
|
16
|
+
return [{ type: "text", text: "Error: #{Profiler::Redaction::MASK} is the mask the profiler shows " \
|
|
17
|
+
"in place of a hidden value, not a value; #{key} was left unchanged." }]
|
|
18
|
+
end
|
|
14
19
|
|
|
15
20
|
if (proxy = MCP::SlaveSupport.with_slave_proxy(params))
|
|
16
21
|
proxy.patch_json("/_profiler/api/env_vars", { key: key, value: value })
|
|
@@ -4,6 +4,7 @@ require "base64"
|
|
|
4
4
|
require "securerandom"
|
|
5
5
|
require "json"
|
|
6
6
|
require "zlib"
|
|
7
|
+
require_relative "../redaction"
|
|
7
8
|
|
|
8
9
|
module Profiler
|
|
9
10
|
module Models
|
|
@@ -46,7 +47,7 @@ module Profiler
|
|
|
46
47
|
@finished_at = Time.now
|
|
47
48
|
@duration = ((@finished_at - @started_at) * 1000).round(2) # milliseconds
|
|
48
49
|
@status = status
|
|
49
|
-
@response_headers = response_headers
|
|
50
|
+
@response_headers = Redaction.filter_headers(response_headers)
|
|
50
51
|
end
|
|
51
52
|
|
|
52
53
|
def add_collector_data(name, data)
|
|
@@ -161,7 +162,7 @@ module Profiler
|
|
|
161
162
|
if binary_content_type?(content_type)
|
|
162
163
|
{ body: Base64.strict_encode64(raw.b), encoding: "base64" }
|
|
163
164
|
else
|
|
164
|
-
text = raw.encode("UTF-8", invalid: :replace, undef: :replace)
|
|
165
|
+
text = Redaction.filter_body(raw, content_type).encode("UTF-8", invalid: :replace, undef: :replace)
|
|
165
166
|
if compress_body?(text)
|
|
166
167
|
{ body: Base64.strict_encode64(Zlib::Deflate.deflate(text)), encoding: "gzip+base64" }
|
|
167
168
|
else
|
|
@@ -189,10 +190,14 @@ module Profiler
|
|
|
189
190
|
ct.to_s.match?(%r{image/(?!svg)|application/(?:pdf|octet-stream|zip)|audio/|video/})
|
|
190
191
|
end
|
|
191
192
|
|
|
193
|
+
LEGACY_FILTERED_PARAMS = %w[password password_confirmation token secret].freeze
|
|
194
|
+
|
|
192
195
|
def sanitize_params(params)
|
|
193
196
|
return {} unless params
|
|
194
197
|
|
|
195
|
-
params.to_h.except(
|
|
198
|
+
return params.to_h.except(*LEGACY_FILTERED_PARAMS) unless Redaction.enabled?
|
|
199
|
+
|
|
200
|
+
Redaction.filter_hash(params.to_h)
|
|
196
201
|
end
|
|
197
202
|
|
|
198
203
|
ALLOWED_HEADERS = %w[
|
|
@@ -206,6 +211,7 @@ module Profiler
|
|
|
206
211
|
env.select { |k, _| k.start_with?("HTTP_") }
|
|
207
212
|
.transform_keys { |k| k.sub(/^HTTP_/, "").split("_").map(&:capitalize).join("-") }
|
|
208
213
|
.select { |k, _| ALLOWED_HEADERS.include?(k) }
|
|
214
|
+
.then { |headers| Redaction.filter_headers(headers) }
|
|
209
215
|
end
|
|
210
216
|
end
|
|
211
217
|
end
|
data/lib/profiler/railtie.rb
CHANGED
|
@@ -6,10 +6,6 @@ module Profiler
|
|
|
6
6
|
class Railtie < Rails::Railtie
|
|
7
7
|
config.profiler = ActiveSupport::OrderedOptions.new
|
|
8
8
|
|
|
9
|
-
initializer "profiler.apply_env_overrides" do
|
|
10
|
-
Profiler.env_override_store.apply!
|
|
11
|
-
end
|
|
12
|
-
|
|
13
9
|
initializer "profiler.set_configs" do |app|
|
|
14
10
|
# Set default configuration for Rails environment
|
|
15
11
|
Profiler.configure do |config|
|
|
@@ -103,6 +99,13 @@ module Profiler
|
|
|
103
99
|
end
|
|
104
100
|
end
|
|
105
101
|
|
|
102
|
+
# After the application's config/initializers, so that `enabled` is the application's own.
|
|
103
|
+
# Declared last: an initializer without `after:` runs after the one declared before it, so
|
|
104
|
+
# one declared below this one would move past the application's initializers too.
|
|
105
|
+
initializer "profiler.apply_env_overrides", after: :load_config_initializers do
|
|
106
|
+
Profiler.env_override_store.apply_at_boot!(Rails.logger)
|
|
107
|
+
end
|
|
108
|
+
|
|
106
109
|
console do
|
|
107
110
|
next unless Profiler.configuration.enabled && Profiler.configuration.track_console
|
|
108
111
|
|
|
@@ -0,0 +1,421 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "set"
|
|
5
|
+
require "rack/utils"
|
|
6
|
+
require "active_support/parameter_filter"
|
|
7
|
+
|
|
8
|
+
module Profiler
|
|
9
|
+
# The single filter applied to everything the profiler captures: request
|
|
10
|
+
# params, headers, bodies and URLs (incoming and outgoing), SQL binds, job
|
|
11
|
+
# arguments, mailer arguments and ENV.
|
|
12
|
+
#
|
|
13
|
+
# It is built on ActiveSupport::ParameterFilter from the host application's
|
|
14
|
+
# Rails.application.config.filter_parameters plus
|
|
15
|
+
# Profiler.configuration.filter_parameters, so it follows Rails semantics:
|
|
16
|
+
# a symbol or string matches any key containing it, case-insensitively, at
|
|
17
|
+
# any nesting depth; a regexp is used as is; a proc rewrites values.
|
|
18
|
+
#
|
|
19
|
+
# It never raises: it runs inside notification subscribers and the Net::HTTP
|
|
20
|
+
# patch, where an exception would reach the application. When the filter
|
|
21
|
+
# fails (a proc that does not expect a value, say), the value is masked and
|
|
22
|
+
# the failure is logged once per error class, without the value.
|
|
23
|
+
module Redaction
|
|
24
|
+
MASK = "[FILTERED]"
|
|
25
|
+
|
|
26
|
+
# Matched by the filter on top of filter_parameters.
|
|
27
|
+
ALWAYS_FILTERED_HEADERS = %w[authorization proxy-authorization cookie set-cookie].freeze
|
|
28
|
+
|
|
29
|
+
# Headers whose value is a URL: their query string goes through the filter.
|
|
30
|
+
URL_HEADERS = %w[referer location content-location].freeze
|
|
31
|
+
|
|
32
|
+
JSON_TYPES = %r{\A(?:application|text)/(?:[\w.+-]+\+)?json\z}i
|
|
33
|
+
NDJSON_TYPES = %r{\A[\w.+-]+/x-ndjson\z}i
|
|
34
|
+
FORM_TYPE = "application/x-www-form-urlencoded"
|
|
35
|
+
MULTIPART_TYPE = %r{\Amultipart/}i
|
|
36
|
+
|
|
37
|
+
# The characters outside ASCII whose case folding yields ASCII letters
|
|
38
|
+
# (Unicode CaseFolding.txt): with any of them, a case-insensitive regexp
|
|
39
|
+
# can match an ASCII filter where a plain ASCII search cannot.
|
|
40
|
+
FOLDS_TO_ASCII = /[ßİʼnſǰẖ-ẚẞKff-st]/
|
|
41
|
+
|
|
42
|
+
# What makes a regexp filter unfit for a search of the whole body text:
|
|
43
|
+
# anchors, word boundaries and lookarounds can match a key on its own and
|
|
44
|
+
# fail on the same key inside the text.
|
|
45
|
+
CONTEXT_SENSITIVE_REGEXP = /\\[AzZbBG]|[\^$]|\(\?<?[=!]/
|
|
46
|
+
|
|
47
|
+
# Bound on the memo of names already tested against the filter.
|
|
48
|
+
KEY_CACHE_LIMIT = 10_000
|
|
49
|
+
|
|
50
|
+
# The value a name alone is tested with: procs in filter_parameters
|
|
51
|
+
# rewrite it in place, so it has to be a string, and a throwaway one.
|
|
52
|
+
PROBE = "x"
|
|
53
|
+
|
|
54
|
+
class << self
|
|
55
|
+
def enabled?
|
|
56
|
+
Profiler.configuration.redact_sensitive_data != false
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# The ParameterFilter, rebuilt only when the list of filters changes.
|
|
60
|
+
def parameter_filter
|
|
61
|
+
compiled[:filter]
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
# Keys are masked by the filters that are not procs; the procs then
|
|
65
|
+
# rewrite each string left, once, under its own key. Running the procs
|
|
66
|
+
# through ParameterFilter instead would dup every other value first,
|
|
67
|
+
# Active Record models included.
|
|
68
|
+
def filter_hash(hash)
|
|
69
|
+
return hash unless enabled? && hash.is_a?(Hash)
|
|
70
|
+
|
|
71
|
+
state = compiled
|
|
72
|
+
masked = state[:plain].filter(hash)
|
|
73
|
+
state[:proc_filter] ? apply_procs_tree(masked, nil, hash, state[:procs]) : masked
|
|
74
|
+
rescue StandardError => e
|
|
75
|
+
report(e, "filtering a hash")
|
|
76
|
+
hash.transform_values { MASK }
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# Filters any value: hashes through the filter, arrays element by
|
|
80
|
+
# element, anything else unchanged (it carries no key to match).
|
|
81
|
+
def filter_value(value)
|
|
82
|
+
return value unless enabled?
|
|
83
|
+
|
|
84
|
+
case value
|
|
85
|
+
when Hash then filter_hash(value)
|
|
86
|
+
when Array then value.map { |v| filter_value(v) }
|
|
87
|
+
else value
|
|
88
|
+
end
|
|
89
|
+
rescue StandardError => e
|
|
90
|
+
report(e, "filtering a value")
|
|
91
|
+
MASK
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
# A value known by its name (SQL bind, mailer argument): MASK when the
|
|
95
|
+
# name matches the filter, else the value as the procs of
|
|
96
|
+
# filter_parameters rewrite it.
|
|
97
|
+
def filter_named(name, value)
|
|
98
|
+
return value unless enabled? && name
|
|
99
|
+
return MASK if sensitive_key?(name)
|
|
100
|
+
|
|
101
|
+
apply_procs([name.to_s], value)
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
# Whether the name alone matches the filter. Memoized per name: header,
|
|
105
|
+
# ENV, column and key names come from a small set. A filter that raises
|
|
106
|
+
# makes the name sensitive.
|
|
107
|
+
def sensitive_key?(name)
|
|
108
|
+
key = name.to_s
|
|
109
|
+
state = compiled
|
|
110
|
+
known = state[:keys]
|
|
111
|
+
return known[key] if known.key?(key)
|
|
112
|
+
|
|
113
|
+
result = state[:plain].filter(key => +PROBE)[key] == MASK
|
|
114
|
+
known[key] = result if known.size < KEY_CACHE_LIMIT
|
|
115
|
+
result
|
|
116
|
+
rescue StandardError => e
|
|
117
|
+
report(e, "testing a name")
|
|
118
|
+
true
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
# Header names are tested as written and with dashes as underscores, so
|
|
122
|
+
# that X-Api-Key matches :_key as x_api_key would.
|
|
123
|
+
def sensitive_header?(name)
|
|
124
|
+
key = name.to_s.downcase
|
|
125
|
+
ALWAYS_FILTERED_HEADERS.include?(key) || sensitive_key?(key) || sensitive_key?(key.tr("-", "_"))
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
def filter_headers(headers)
|
|
129
|
+
return headers unless enabled? && headers.respond_to?(:each_with_object)
|
|
130
|
+
|
|
131
|
+
headers.each_with_object({}) do |(name, value), out|
|
|
132
|
+
out[name] = filter_header(name, value)
|
|
133
|
+
end
|
|
134
|
+
end
|
|
135
|
+
|
|
136
|
+
# Filters a raw body according to its content type. JSON, NDJSON and
|
|
137
|
+
# form-urlencoded bodies have their filtered keys masked, multipart
|
|
138
|
+
# bodies are masked entirely, and every other type is kept as is: they
|
|
139
|
+
# carry no key the filter can read, and masking HTML would hide the very
|
|
140
|
+
# pages being profiled.
|
|
141
|
+
#
|
|
142
|
+
# Bodies usually arrive as binary strings (rack.input, Net::HTTP): they
|
|
143
|
+
# are read as UTF-8, or as bytes when they are not valid UTF-8.
|
|
144
|
+
def filter_body(raw, content_type)
|
|
145
|
+
return raw unless enabled? && raw.is_a?(String) && !raw.empty?
|
|
146
|
+
|
|
147
|
+
mime = content_type.to_s.split(";").first.to_s.strip.downcase
|
|
148
|
+
text = readable(raw)
|
|
149
|
+
if mime.match?(JSON_TYPES)
|
|
150
|
+
filter_json(text, mime)
|
|
151
|
+
elsif mime.match?(NDJSON_TYPES)
|
|
152
|
+
text.split("\n", -1).map { |line| line.strip.empty? ? line : filter_json(line, mime) }.join("\n")
|
|
153
|
+
elsif mime == FORM_TYPE
|
|
154
|
+
filter_query(text)
|
|
155
|
+
elsif mime.match?(MULTIPART_TYPE)
|
|
156
|
+
"[FILTERED: #{mime} body, #{raw.bytesize} bytes]"
|
|
157
|
+
else
|
|
158
|
+
raw
|
|
159
|
+
end
|
|
160
|
+
rescue StandardError => e
|
|
161
|
+
report(e, "filtering a #{mime} body")
|
|
162
|
+
"[FILTERED: #{mime} body that could not be filtered, #{raw.bytesize} bytes]"
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
# Masks the filtered values of a query string, pair by pair, leaving the
|
|
166
|
+
# rest of the string exactly as it was.
|
|
167
|
+
def filter_query(query)
|
|
168
|
+
return query unless enabled? && query.is_a?(String) && !query.empty?
|
|
169
|
+
|
|
170
|
+
readable(query).split("&", -1).map do |pair|
|
|
171
|
+
name, value = pair.split("=", 2)
|
|
172
|
+
next pair if value.nil? || value.empty?
|
|
173
|
+
|
|
174
|
+
parsed = Rack::Utils.parse_nested_query(pair)
|
|
175
|
+
parameter_filter.filter(parsed) == parsed ? pair : "#{name}=#{MASK}"
|
|
176
|
+
rescue StandardError
|
|
177
|
+
# A pair Rack or the filter cannot handle is masked rather than trusted.
|
|
178
|
+
"#{name}=#{MASK}"
|
|
179
|
+
end.join("&")
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
# A URL, or a list of them as Rack 3 gives a repeated header.
|
|
183
|
+
def filter_url(url)
|
|
184
|
+
return url unless enabled?
|
|
185
|
+
return url.map { |u| filter_url(u) } if url.is_a?(Array)
|
|
186
|
+
return url unless url.is_a?(String)
|
|
187
|
+
|
|
188
|
+
rest, fragment = readable(url).split("#", 2)
|
|
189
|
+
base, query = rest.split("?", 2)
|
|
190
|
+
return url if query.nil?
|
|
191
|
+
|
|
192
|
+
"#{base}?#{filter_query(query)}#{"##{fragment}" if fragment}"
|
|
193
|
+
rescue StandardError => e
|
|
194
|
+
report(e, "filtering a URL")
|
|
195
|
+
MASK
|
|
196
|
+
end
|
|
197
|
+
|
|
198
|
+
# ENV as the profiler shows it: names kept, values masked unless the
|
|
199
|
+
# name is in config.env_allowlist and does not match the filter.
|
|
200
|
+
def env_snapshot(env = ENV.to_h)
|
|
201
|
+
listed = env_allowlist_matcher
|
|
202
|
+
env.sort.to_h { |name, value| [name, env_visible?(name, listed) ? apply_procs([name], value) : MASK] }
|
|
203
|
+
end
|
|
204
|
+
|
|
205
|
+
def env_value(name, value)
|
|
206
|
+
env_visible?(name) ? apply_procs([name.to_s], value) : MASK
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
# ENV overrides with the same rule applied to their current and original
|
|
210
|
+
# values. The deleted and restore markers are kept: they are not values.
|
|
211
|
+
def env_overrides(overrides)
|
|
212
|
+
overrides.to_h do |name, entry|
|
|
213
|
+
next [name, entry] if env_visible?(name) || !entry.is_a?(Hash)
|
|
214
|
+
|
|
215
|
+
masked = entry.to_h do |field, value|
|
|
216
|
+
[field, keep_override_field?(field, value) ? value : MASK]
|
|
217
|
+
end
|
|
218
|
+
[name, masked]
|
|
219
|
+
end
|
|
220
|
+
end
|
|
221
|
+
|
|
222
|
+
def env_visible?(name, listed = env_allowlist_matcher)
|
|
223
|
+
listed.call(name.to_s) && !(enabled? && sensitive_key?(name))
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
# Whether +value+, stripped, is the mask: a value exported while masked,
|
|
227
|
+
# which must never be written back as if it were the real one.
|
|
228
|
+
def mask?(value)
|
|
229
|
+
value.to_s.strip == MASK
|
|
230
|
+
end
|
|
231
|
+
|
|
232
|
+
private
|
|
233
|
+
|
|
234
|
+
def compiled
|
|
235
|
+
filters = current_filters
|
|
236
|
+
state = @compiled
|
|
237
|
+
return state if state && state[:filters] == filters
|
|
238
|
+
|
|
239
|
+
@reported = Set.new
|
|
240
|
+
procs, plain = filters.partition { |f| f.respond_to?(:call) }
|
|
241
|
+
@compiled = {
|
|
242
|
+
filters: filters,
|
|
243
|
+
filter: ActiveSupport::ParameterFilter.new(filters, mask: MASK),
|
|
244
|
+
plain: ActiveSupport::ParameterFilter.new(plain, mask: MASK),
|
|
245
|
+
proc_filter: procs.empty? ? nil : ActiveSupport::ParameterFilter.new(procs, mask: MASK),
|
|
246
|
+
procs: procs,
|
|
247
|
+
keys: {}
|
|
248
|
+
}.merge(text_matchers(filters))
|
|
249
|
+
end
|
|
250
|
+
|
|
251
|
+
# What the JSON pre-test needs: the string filters, downcased, for a
|
|
252
|
+
# plain search, and as one case-insensitive alternation as
|
|
253
|
+
# ParameterFilter builds it, for text outside ASCII; the regexp filters
|
|
254
|
+
# that can be searched for in the whole text. Procs rewrite values
|
|
255
|
+
# whatever the key, dotted filters match a key path, and other regexps
|
|
256
|
+
# depend on where the key ends: any of them makes the filters opaque,
|
|
257
|
+
# and every body is then parsed.
|
|
258
|
+
def text_matchers(filters)
|
|
259
|
+
procs = filters.any? { |f| f.respond_to?(:call) }
|
|
260
|
+
regexps, strings = filters.reject { |f| f.respond_to?(:call) }.partition { |f| f.is_a?(Regexp) }
|
|
261
|
+
opaque = procs ||
|
|
262
|
+
filters.any? { |f| !f.respond_to?(:call) && f.to_s.include?(".") } ||
|
|
263
|
+
regexps.any? { |r| r.source.match?(CONTEXT_SENSITIVE_REGEXP) } ||
|
|
264
|
+
strings.any? { |s| !s.to_s.ascii_only? }
|
|
265
|
+
words = strings.map { |s| s.to_s.downcase }
|
|
266
|
+
{
|
|
267
|
+
opaque: opaque,
|
|
268
|
+
words: words,
|
|
269
|
+
words_regexp: words.empty? ? nil : Regexp.new(words.map { |w| Regexp.escape(w) }.join("|"), Regexp::IGNORECASE),
|
|
270
|
+
regexps: regexps
|
|
271
|
+
}
|
|
272
|
+
end
|
|
273
|
+
|
|
274
|
+
def filter_header(name, value)
|
|
275
|
+
return MASK if sensitive_header?(name)
|
|
276
|
+
|
|
277
|
+
value = filter_url(value) if URL_HEADERS.include?(name.to_s.downcase)
|
|
278
|
+
apply_procs([name.to_s, name.to_s.tr("-", "_")], value)
|
|
279
|
+
rescue StandardError => e
|
|
280
|
+
report(e, "filtering a header")
|
|
281
|
+
MASK
|
|
282
|
+
end
|
|
283
|
+
|
|
284
|
+
# The value as the procs of filter_parameters rewrite it under each of
|
|
285
|
+
# +names+. Nothing to do without procs: the names were already tested.
|
|
286
|
+
#
|
|
287
|
+
# Only strings: any other object would be duplicated for the procs,
|
|
288
|
+
# which runs the application's copy hooks; it is shown through its
|
|
289
|
+
# inspect, as before. The procs run once: under the first of +names+
|
|
290
|
+
# that changes the value, so that a proc which undoes itself (reverse!)
|
|
291
|
+
# is not applied twice.
|
|
292
|
+
def apply_procs(names, value)
|
|
293
|
+
return value unless enabled? && value.is_a?(String)
|
|
294
|
+
|
|
295
|
+
filter = compiled[:proc_filter]
|
|
296
|
+
return value unless filter
|
|
297
|
+
|
|
298
|
+
names.uniq.each do |name|
|
|
299
|
+
result = filter.filter(name => value)[name]
|
|
300
|
+
return result unless result == value
|
|
301
|
+
end
|
|
302
|
+
value
|
|
303
|
+
rescue StandardError => e
|
|
304
|
+
report(e, "applying a filter_parameters proc")
|
|
305
|
+
MASK
|
|
306
|
+
end
|
|
307
|
+
|
|
308
|
+
# Walks a tree the non-proc filters have already masked and lets the
|
|
309
|
+
# procs rewrite each string leaf, once, as ParameterFilter does: a copy
|
|
310
|
+
# of the key and of the value, and the original params for a proc of
|
|
311
|
+
# arity 3. A leaf masked by the filter is the MASK object itself and is
|
|
312
|
+
# left alone, also as ParameterFilter does; a value that only reads
|
|
313
|
+
# "[FILTERED]" is a different object and still goes to the procs.
|
|
314
|
+
def apply_procs_tree(value, key, original, procs)
|
|
315
|
+
case value
|
|
316
|
+
when Hash then value.each_with_object(value.class.new) { |(k, v), out| out[k] = apply_procs_tree(v, k, original, procs) }
|
|
317
|
+
when Array then value.map { |v| apply_procs_tree(v, key, original, procs) }
|
|
318
|
+
when String then key.nil? || value.equal?(MASK) ? value : call_procs(procs, key, value, original)
|
|
319
|
+
else value
|
|
320
|
+
end
|
|
321
|
+
end
|
|
322
|
+
|
|
323
|
+
def call_procs(procs, key, value, original)
|
|
324
|
+
key = key.dup if key.duplicable?
|
|
325
|
+
value = value.dup
|
|
326
|
+
procs.each { |b| b.arity == 2 ? b.call(key, value) : b.call(key, value, original) }
|
|
327
|
+
value
|
|
328
|
+
rescue StandardError => e
|
|
329
|
+
report(e, "applying a filter_parameters proc")
|
|
330
|
+
MASK
|
|
331
|
+
end
|
|
332
|
+
|
|
333
|
+
def keep_override_field?(field, value)
|
|
334
|
+
value.nil? ||
|
|
335
|
+
value == EnvOverrideStore::DELETED_SENTINEL ||
|
|
336
|
+
value == EnvOverrideStore::RESTORE_SENTINEL ||
|
|
337
|
+
!%w[value original].include?(field)
|
|
338
|
+
end
|
|
339
|
+
|
|
340
|
+
def env_allowlist_matcher
|
|
341
|
+
allowlist = Profiler.configuration.env_allowlist
|
|
342
|
+
return ->(_name) { true } if allowlist == :all
|
|
343
|
+
|
|
344
|
+
regexps, names = Array(allowlist).partition { |entry| entry.is_a?(Regexp) }
|
|
345
|
+
names = names.to_set(&:to_s)
|
|
346
|
+
->(name) { names.include?(name) || regexps.any? { |r| r.match?(name) } }
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
def current_filters
|
|
350
|
+
filters = Array(Profiler.configuration.filter_parameters)
|
|
351
|
+
app_filters = rails_filter_parameters
|
|
352
|
+
app_filters.empty? ? filters : app_filters + filters
|
|
353
|
+
end
|
|
354
|
+
|
|
355
|
+
def rails_filter_parameters
|
|
356
|
+
return [] unless defined?(::Rails) && ::Rails.respond_to?(:application) && ::Rails.application
|
|
357
|
+
|
|
358
|
+
Array(::Rails.application.config.filter_parameters)
|
|
359
|
+
rescue StandardError
|
|
360
|
+
[]
|
|
361
|
+
end
|
|
362
|
+
|
|
363
|
+
# The string as UTF-8 when it is valid UTF-8, else as bytes, so that
|
|
364
|
+
# string operations and regexps never meet an incompatible encoding.
|
|
365
|
+
def readable(string)
|
|
366
|
+
return string if string.encoding == Encoding::UTF_8 && string.valid_encoding?
|
|
367
|
+
|
|
368
|
+
utf8 = string.dup.force_encoding(Encoding::UTF_8)
|
|
369
|
+
utf8.valid_encoding? ? utf8 : string.b
|
|
370
|
+
end
|
|
371
|
+
|
|
372
|
+
def filter_json(text, mime)
|
|
373
|
+
return text unless json_needs_parse?(text)
|
|
374
|
+
|
|
375
|
+
parsed = JSON.parse(text)
|
|
376
|
+
filtered = filter_value(parsed)
|
|
377
|
+
filtered == parsed ? text : JSON.generate(filtered)
|
|
378
|
+
rescue JSON::ParserError, JSON::GeneratorError
|
|
379
|
+
"[FILTERED: unparseable #{mime} body, #{text.bytesize} bytes]"
|
|
380
|
+
end
|
|
381
|
+
|
|
382
|
+
# A cheap pre-test that spares the parse of a body in whose text no
|
|
383
|
+
# filter matches anywhere: no key of it can match either. Any match, in
|
|
384
|
+
# a key or in a value, sends the body to the parser, which also masks
|
|
385
|
+
# an unreadable body entirely. Text that is not valid UTF-8, holds an
|
|
386
|
+
# escape (a key can be written password) or, for a plain ASCII
|
|
387
|
+
# search, a character that case-folds onto ASCII letters is not judged
|
|
388
|
+
# here, nor is any body under opaque filters.
|
|
389
|
+
def json_needs_parse?(text)
|
|
390
|
+
state = compiled
|
|
391
|
+
return true if state[:opaque]
|
|
392
|
+
return true if text.encoding != Encoding::UTF_8 || text.include?("\\")
|
|
393
|
+
|
|
394
|
+
state[:regexps].any? { |r| r.match?(text) } || words_in_text?(state, text)
|
|
395
|
+
end
|
|
396
|
+
|
|
397
|
+
def words_in_text?(state, text)
|
|
398
|
+
words = state[:words]
|
|
399
|
+
return false if words.empty?
|
|
400
|
+
return state[:words_regexp].match?(text) unless text.ascii_only? || !text.match?(FOLDS_TO_ASCII)
|
|
401
|
+
|
|
402
|
+
lowered = text.downcase(:ascii)
|
|
403
|
+
words.any? { |word| lowered.include?(word) }
|
|
404
|
+
end
|
|
405
|
+
|
|
406
|
+
# Logs a failure of the filter once per error class, with no value and
|
|
407
|
+
# no message: either could carry what was being masked.
|
|
408
|
+
def report(error, during)
|
|
409
|
+
reported = (@reported ||= Set.new)
|
|
410
|
+
return if reported.include?(error.class)
|
|
411
|
+
|
|
412
|
+
reported << error.class
|
|
413
|
+
message = "[Profiler] Redaction: #{error.class} while #{during}; the value was masked instead."
|
|
414
|
+
logger = ::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger)
|
|
415
|
+
logger ? logger.warn(message) : Kernel.warn(message)
|
|
416
|
+
rescue StandardError
|
|
417
|
+
nil
|
|
418
|
+
end
|
|
419
|
+
end
|
|
420
|
+
end
|
|
421
|
+
end
|
data/lib/profiler/version.rb
CHANGED
data/lib/profiler.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: rails-profiler
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.30.
|
|
4
|
+
version: 0.30.8
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Sébastien Duplessy
|
|
@@ -209,6 +209,7 @@ files:
|
|
|
209
209
|
- lib/profiler/models/sql_query.rb
|
|
210
210
|
- lib/profiler/models/timeline_event.rb
|
|
211
211
|
- lib/profiler/railtie.rb
|
|
212
|
+
- lib/profiler/redaction.rb
|
|
212
213
|
- lib/profiler/sse/bus.rb
|
|
213
214
|
- lib/profiler/sse/event_bus.rb
|
|
214
215
|
- lib/profiler/sse/redis_event_bus.rb
|