rails-profiler 0.30.6 → 0.30.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e7a2ea61f4463670763c8c3c2d92f98a1f4ff2917fb9a2f5e0d245ea1860d2f1
4
- data.tar.gz: ba467072b67f027565a29c99f458cd959933de512cdfa054ddde052d3e65ddb1
3
+ metadata.gz: 7edf1c383b05c79e4013d76c96fa57077b376f3a312c27aa355927ca1bb7e362
4
+ data.tar.gz: b1b9b6bd25f7f739c14d2afe72309fac7455e8f2c62b5e7d81e206a09b8ba4fa
5
5
  SHA512:
6
- metadata.gz: b2a1049428bab86d06ba54bba7cd373bfc3ebe606137bd2e6bfda46d5b1501c10574ebcec632125efd9e5c6b6a0be2d11c763b15e670a06aaaac384c0e028c6e
7
- data.tar.gz: 561590eb7eadc20139ea79a0005efefbf9de88da0010db4ae20903360d3199342ba9648fe9d6d4bc1452c62e18c61f47311f19bb78f68a3307791f3089a11f06
6
+ metadata.gz: ca0ca79568606603601b5b33aa3ee69fb54d6497e8c8e2087f45cb8d78092ad0264ed8a8794472955e6b5608f743d198de31fcdf63493e40b7f538fd4c4d8298
7
+ data.tar.gz: fd3f7cb5326209b731f11abbca7e7e0b3b226ce019ea62f32636a137d1d402c26f692dc11754e27d3144bae0814524ea7be282031d8adacfe3bda5a97d3d4cc9
data/CHANGELOG.md CHANGED
@@ -19,6 +19,87 @@ every commit of every tag interval is accounted for one way or the other.
19
19
 
20
20
  ## [Unreleased]
21
21
 
22
+ ## [0.30.8] - 2026-10-04
23
+
24
+ <!-- stamped -->
25
+
26
+ ### Fixed
27
+
28
+ - **Env overrides:** Keep every override when several are set, deleted or reset at the same
29
+ time, from threads of the web process or from Sidekiq processes. Each writer read the file,
30
+ changed it and saved it over the others' changes, and a reader could see a half-written file as
31
+ no override at all. The file is now changed under a lock (`env_overrides.json.lock`, next to
32
+ it) and replaced in one step.
33
+
34
+ ### Security
35
+
36
+ - **Env overrides:** Stop applying the environment variable overrides saved from the Env tab or
37
+ the MCP env tools in production, and while the profiler is disabled. They were written into
38
+ `ENV` at every boot whatever `enabled` said and whatever the environment, so an
39
+ `env_overrides.json` left under `tmp/` and shipped with a deployment changed the environment of
40
+ the production application, and an application with `config.enabled = false` still had them
41
+ applied without a word. Sidekiq jobs and console evaluations, which apply them again, follow the
42
+ same rules. When overrides are left out, the boot logs one warning to `Rails.logger` with the
43
+ file, the number of overrides and the reason, never a name or a value.
44
+ - **Env overrides:** Apply them at boot once the application's `config/initializers` have run,
45
+ where `enabled` is decided, instead of before. An initializer of the application that reads
46
+ `ENV` while it runs no longer sees them; requests, jobs, eager loading and `after_initialize`
47
+ still do.
48
+ - **Env overrides:** Resetting one override, or all of them, from the Env tab or the MCP env
49
+ tools no longer writes into `ENV` the "original" values saved in `env_overrides.json` in
50
+ production, nor while the profiler is disabled: those values come from the machine that wrote
51
+ the file, so a reset could put a development value into a production variable, or delete it.
52
+ There, a reset restores only the variables the running process changed itself, to the values
53
+ they had before.
54
+ - **Env overrides:** In production, or with the profiler disabled, typing back in the Env tab the
55
+ original value the overrides file shows for a variable now records the running value first, so
56
+ a later reset puts it back instead of keeping the file's value. A reset also restores a variable
57
+ this process changed even when its entry has gone from the file. The `reset_env_var` MCP tool
58
+ now says whether the original value was restored in the process or `ENV` was left unchanged,
59
+ and no longer prints a value, masked or not.
60
+ - **Upgrading:** to apply the env overrides while the profiler is disabled outside production,
61
+ as before, set `config.apply_env_overrides_when_disabled = true`. Nothing applies them in
62
+ production. If an initializer of yours needs them, call `Profiler.env_override_store.apply!` at
63
+ the end of `config/initializers/profiler.rb`; initializers loaded after it then see them, under
64
+ the same rules.
65
+
66
+ ## [0.30.7] - 2026-10-04
67
+
68
+ <!-- stamped -->
69
+
70
+ ### Security
71
+
72
+ - **Sensitive data:** Mask sensitive values with `[FILTERED]` before a profile is stored. Until
73
+ now `Authorization`, `Cookie` and `Set-Cookie` headers, raw request and response bodies
74
+ (passwords included), outbound request headers and bodies, and the whole of `ENV` were stored in
75
+ clear, and params lost only four exact root keys. The profiler now builds one filter from your
76
+ `Rails.application.config.filter_parameters` plus its own `config.filter_parameters`, with Rails
77
+ semantics (nested keys, any case, regexps, procs), and applies it to params and route params,
78
+ JSON, NDJSON and form bodies, incoming, response and outbound headers (the query string of
79
+ `Referer`, `Location` and `Content-Location` included), outbound URLs, SQL binds of filtered
80
+ columns, job arguments and mailer arguments. `multipart` bodies, and JSON bodies that cannot be
81
+ parsed and in which a filter matches, are masked entirely; HTML and other unstructured bodies are
82
+ kept. A params key that matches the filter is now kept with a masked value instead of being
83
+ removed.
84
+ - **Sensitive data:** `ENV` values are shown only for the variables in `config.env_allowlist`, in
85
+ the Env tab, the `env_vars` endpoint and the `list_env_vars`, `reset_env_var` and `get_profile`
86
+ MCP tools alike. `[FILTERED]` is refused as a value by the `env_vars` endpoint (422), the
87
+ `set_env_var` MCP tool and the Env tab import, so re-importing an export cannot overwrite a
88
+ secret. EXPLAIN is refused, with a message, for a query whose bind values were masked.
89
+ - **Sensitive data:** Procs of `filter_parameters` run once per string value, with the original
90
+ params for a proc of three arguments, as in Rails; other objects, Active Record models included,
91
+ are neither copied nor passed to them. When the filter fails, the value is masked rather than
92
+ the error reaching the application, and the error class is logged once, without the value. A
93
+ JSON body in whose text no filter matches is not parsed.
94
+ - **Upgrade note:** To get the previous behaviour back, set `config.redact_sensitive_data = false`
95
+ (no masking) and `config.env_allowlist = :all` (every `ENV` value); `config.filter_parameters = []`
96
+ keeps masking but relies on your application's list alone. Profiles stored before the upgrade
97
+ are not rewritten and keep their values in clear until they rotate out: clear them after
98
+ upgrading with `bin/rails runner 'Profiler.storage.clear'` (every storage backend), the
99
+ `clear_profiles` MCP tool or the dashboard; the memory store is emptied by a restart. Update
100
+ every node of a cluster: a node still on an older version returns its data in clear to the
101
+ master.
102
+
22
103
  ## [0.30.6] - 2026-10-03
23
104
 
24
105
  <!-- stamped -->
@@ -4499,6 +4499,7 @@
4499
4499
  }
4500
4500
 
4501
4501
  // app/assets/typescript/profiler/components/dashboard/tabs/EnvTab.tsx
4502
+ var FILTERED_MASK = "[FILTERED]";
4502
4503
  function detectType(value) {
4503
4504
  if (value === "") return { label: "empty", color: "var(--profiler-text-muted)" };
4504
4505
  if (/^(true|false|yes|no)$/i.test(value)) return { label: "bool", color: "#a78bfa" };
@@ -4653,8 +4654,13 @@
4653
4654
  URL.revokeObjectURL(url);
4654
4655
  showFlash("success", `Exported ${entries.length} variables`);
4655
4656
  };
4656
- const importPreview = T2(() => parseEnvFile(importContent), [importContent]);
4657
+ const importParsed = T2(() => parseEnvFile(importContent), [importContent]);
4658
+ const importPreview = T2(
4659
+ () => Object.fromEntries(Object.entries(importParsed).filter(([, v3]) => v3 !== FILTERED_MASK)),
4660
+ [importParsed]
4661
+ );
4657
4662
  const importCount = Object.keys(importPreview).length;
4663
+ const importMaskedCount = Object.keys(importParsed).length - importCount;
4658
4664
  const importEnv = async () => {
4659
4665
  const entries = Object.entries(importPreview);
4660
4666
  if (!entries.length) return;
@@ -4664,7 +4670,8 @@
4664
4670
  setVariables((prev) => ({ ...prev, ...importPreview }));
4665
4671
  setImportContent("");
4666
4672
  setShowImport(false);
4667
- showFlash("success", `Imported ${entries.length} variables`);
4673
+ const skipped = importMaskedCount > 0 ? `, skipped ${importMaskedCount} masked` : "";
4674
+ showFlash("success", `Imported ${entries.length} variables${skipped}`);
4668
4675
  } catch (e3) {
4669
4676
  showFlash("error", e3.message ?? "Import failed");
4670
4677
  } finally {
@@ -4933,6 +4940,12 @@
4933
4940
  importCount !== 1 ? "s" : "",
4934
4941
  " to import"
4935
4942
  ] }),
4943
+ importMaskedCount > 0 && /* @__PURE__ */ u3("span", { class: "profiler-text--xs profiler-text--muted", children: [
4944
+ importMaskedCount,
4945
+ " masked (",
4946
+ FILTERED_MASK,
4947
+ ") skipped"
4948
+ ] }),
4936
4949
  /* @__PURE__ */ u3(
4937
4950
  "button",
4938
4951
  {
@@ -4,8 +4,8 @@ module Profiler
4
4
  module Api
5
5
  class EnvVarsController < ApplicationController
6
6
  def show
7
- variables = ENV.to_h.sort.to_h
8
- overrides = Profiler.env_override_store.all_overrides
7
+ variables = Profiler::Redaction.env_snapshot
8
+ overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
9
9
  render json: { variables: variables, total: variables.size, overrides: overrides }
10
10
  end
11
11
 
@@ -19,19 +19,27 @@ module Profiler
19
19
 
20
20
  value = params[:value]
21
21
 
22
+ if Profiler::Redaction.mask?(value)
23
+ render json: { error: "#{Profiler::Redaction::MASK} is the mask the profiler shows in place of a " \
24
+ "hidden value, not a value; #{key} was left unchanged" },
25
+ status: :unprocessable_entity
26
+ return
27
+ end
28
+
22
29
  if value.nil? || value.to_s.empty?
23
30
  Profiler.env_override_store.delete(key)
24
31
  ENV.delete(key)
25
- render json: { key: key, value: nil, deleted: true, override: Profiler.env_override_store.all_overrides[key] }
32
+ render json: { key: key, value: nil, deleted: true, override: redacted_override(key) }
26
33
  else
27
34
  current_original = Profiler.env_override_store.all_overrides.dig(key, "original")
28
- if current_original == value.to_s
35
+ # Where the overrides are blocked, set records this process's value, which reset would not.
36
+ if current_original == value.to_s && Profiler.env_override_store.blocked_reason.nil?
29
37
  Profiler.env_override_store.reset(key)
30
38
  else
31
39
  Profiler.env_override_store.set(key, value.to_s)
32
40
  end
33
41
  ENV[key] = value.to_s
34
- render json: { key: key, value: ENV[key], override: Profiler.env_override_store.all_overrides[key] }
42
+ render json: { key: key, value: redacted_value(key), override: redacted_override(key) }
35
43
  end
36
44
  end
37
45
 
@@ -44,13 +52,23 @@ module Profiler
44
52
  end
45
53
 
46
54
  Profiler.env_override_store.reset(key)
47
- render json: { key: key, value: ENV[key], reset: true }
55
+ render json: { key: key, value: redacted_value(key), reset: true }
48
56
  end
49
57
 
50
58
  def reset_all
51
59
  Profiler.env_override_store.reset_all
52
60
  render json: { reset: true }
53
61
  end
62
+
63
+ private
64
+
65
+ def redacted_value(key)
66
+ ENV[key].nil? ? nil : Profiler::Redaction.env_value(key, ENV[key])
67
+ end
68
+
69
+ def redacted_override(key)
70
+ Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)[key]
71
+ end
54
72
  end
55
73
  end
56
74
  end
@@ -96,11 +96,15 @@ module Profiler
96
96
 
97
97
  binds.map do |bind|
98
98
  if bind.respond_to?(:value)
99
- bind.value
99
+ name = bind.name if bind.respond_to?(:name)
100
+ Profiler::Redaction.filter_named(name, bind.value)
100
101
  else
101
102
  bind
102
103
  end
103
104
  end
105
+ rescue StandardError
106
+ # Never let the profiler raise into the application's query.
107
+ binds.map { Profiler::Redaction::MASK }
104
108
  end
105
109
 
106
110
  def extract_backtrace
@@ -25,7 +25,7 @@ module Profiler
25
25
  end
26
26
 
27
27
  def collect
28
- variables = ENV.to_h.sort.to_h
28
+ variables = Profiler::Redaction.env_snapshot
29
29
  store_data({ variables: variables, total: variables.size })
30
30
  end
31
31
 
@@ -199,7 +199,19 @@ module Profiler
199
199
  klass = Object.const_get(mailer_class)
200
200
  params = klass.instance_method(action).parameters
201
201
  params.each_with_index.each_with_object({}) do |((_, name), i), h|
202
- h[name.to_s] = serialize_assign(args[i])
202
+ h[name.to_s] = if Profiler::Redaction.sensitive_key?(name)
203
+ Profiler::Redaction::MASK
204
+ else
205
+ value = args[i]
206
+ # A hash or array goes through the filter by its keys, anything
207
+ # else by the parameter name: never both, so a proc runs once.
208
+ filtered = if value.is_a?(Hash) || value.is_a?(Array)
209
+ Profiler::Redaction.filter_value(value)
210
+ else
211
+ Profiler::Redaction.filter_named(name, value)
212
+ end
213
+ serialize_assign(filtered)
214
+ end
203
215
  end
204
216
  rescue StandardError
205
217
  {}
@@ -91,7 +91,7 @@ module Profiler
91
91
  {
92
92
  route_name: route_name ? "#{route_name}_path" : nil,
93
93
  route_pattern: matched_route&.path&.spec&.to_s&.sub(/\(\.:format\)$/, ""),
94
- route_params: route_params,
94
+ route_params: Profiler::Redaction.filter_hash(route_params),
95
95
  controller_action: controller_action
96
96
  }
97
97
  rescue StandardError
@@ -9,6 +9,22 @@ module Profiler
9
9
  FORGERY_PROTECTION_HEADER = "X-Profiler-Request"
10
10
 
11
11
  class Configuration
12
+ # Added to the application's own config.filter_parameters: the list of the
13
+ # Rails 7.1 application template, without :email.
14
+ DEFAULT_FILTER_PARAMETERS = %i[
15
+ passw secret token _key crypt salt certificate otp ssn cvv cvc
16
+ ].freeze
17
+
18
+ # ENV variables whose values the profiler shows; every other one is listed
19
+ # with its value masked.
20
+ DEFAULT_ENV_ALLOWLIST = %w[
21
+ RAILS_ENV RACK_ENV NODE_ENV RAILS_LOG_LEVEL RAILS_LOG_TO_STDOUT RAILS_SERVE_STATIC_FILES
22
+ RAILS_MAX_THREADS RAILS_MIN_THREADS WEB_CONCURRENCY PORT PIDFILE
23
+ LANG LANGUAGE LC_ALL LC_CTYPE TZ HOME PWD PATH SHELL USER HOSTNAME TERM
24
+ RUBY_VERSION RUBYOPT RUBY_YJIT_ENABLE MALLOC_ARENA_MAX
25
+ BUNDLE_GEMFILE BUNDLE_PATH BUNDLE_WITHOUT GEM_HOME GEM_PATH BOOTSNAP_CACHE_DIR
26
+ ].freeze
27
+
12
28
  attr_accessor :enabled, :storage_options, :collectors,
13
29
  :skip_paths, :slow_query_threshold, :max_queries_warning,
14
30
  :track_memory, :memory_warning_threshold,
@@ -19,9 +35,11 @@ module Profiler
19
35
  :track_http, :slow_http_threshold, :http_skip_hosts, :http_backtrace_depth,
20
36
  :track_jobs,
21
37
  :track_console,
38
+ :apply_env_overrides_when_disabled,
22
39
  :track_tests,
23
40
  :track_mailers, :capture_mail_body, :sanitize_mailer_recipients, :mailer_skip_actions,
24
41
  :compress_bodies, :compress_body_threshold,
42
+ :redact_sensitive_data, :filter_parameters, :env_allowlist,
25
43
  :name, :master_url, :self_url,
26
44
  :cluster_heartbeat_interval, :cluster_offline_threshold
27
45
 
@@ -60,6 +78,7 @@ module Profiler
60
78
  @http_backtrace_depth = 40
61
79
  @track_jobs = true
62
80
  @track_console = true
81
+ @apply_env_overrides_when_disabled = false
63
82
  @track_tests = false
64
83
  @track_mailers = true
65
84
  @capture_mail_body = false
@@ -67,6 +86,9 @@ module Profiler
67
86
  @mailer_skip_actions = []
68
87
  @compress_bodies = true
69
88
  @compress_body_threshold = 10 * 1024 # 10 KB
89
+ @redact_sensitive_data = true
90
+ @filter_parameters = DEFAULT_FILTER_PARAMETERS.dup
91
+ @env_allowlist = DEFAULT_ENV_ALLOWLIST.dup
70
92
  @tmp_path = nil
71
93
  @name = nil
72
94
  @master_url = nil
@@ -8,63 +8,108 @@ module Profiler
8
8
  DELETED_SENTINEL = "__profiler_deleted__"
9
9
  RESTORE_SENTINEL = "__profiler_restore__"
10
10
 
11
+ # Why the persisted overrides are left out of ENV, as logged at boot.
12
+ BLOCKED_REASONS = {
13
+ production: "persisted overrides are never applied in production; " \
14
+ "delete the file if it was deployed by mistake",
15
+ disabled: "the profiler is disabled; set " \
16
+ "config.apply_env_overrides_when_disabled = true to apply them anyway"
17
+ }.freeze
18
+
11
19
  # File format: { "KEY" => { "value" => "...", "original" => "..." } }
12
20
  # Sentinels for "value":
13
21
  # DELETED_SENTINEL → ENV.delete(key)
14
22
  # RESTORE_SENTINEL → restore ENV[key] to "original" (cleaned up after apply!)
15
23
 
24
+ def initialize
25
+ # What ENV held in this process before set, delete or apply! changed a key. Where the
26
+ # overrides are blocked, a reset restores these and never the file's "original" values,
27
+ # which come from whatever machine wrote the file.
28
+ @process_originals = {}
29
+ @process_originals_lock = Mutex.new
30
+ end
31
+
16
32
  def set(key, value)
17
- overrides = load_overrides
18
- original = original_for(overrides, key)
19
- overrides[key] = { "value" => value.to_s, "original" => original }
20
- save_overrides(overrides)
33
+ remember_process_original(key)
34
+ update_overrides do |overrides|
35
+ overrides[key] = { "value" => value.to_s, "original" => original_for(overrides, key) }
36
+ end
21
37
  rescue => e
22
38
  warn "[Profiler] EnvOverrideStore: failed to set #{key}: #{e.message}"
23
39
  end
24
40
 
25
41
  def delete(key)
26
- overrides = load_overrides
27
- original = original_for(overrides, key)
28
- overrides[key] = { "value" => DELETED_SENTINEL, "original" => original }
29
- save_overrides(overrides)
42
+ remember_process_original(key)
43
+ update_overrides do |overrides|
44
+ overrides[key] = { "value" => DELETED_SENTINEL, "original" => original_for(overrides, key) }
45
+ end
30
46
  rescue => e
31
47
  warn "[Profiler] EnvOverrideStore: failed to delete #{key}: #{e.message}"
32
48
  end
33
49
 
50
+ # Returns whether ENV was written in this process.
34
51
  def reset(key)
35
- overrides = load_overrides
36
- entry = overrides[key]
37
- return unless entry
38
-
39
- original = entry["original"]
40
- # Keep a RESTORE entry so Sidekiq workers pick it up on next job
41
- overrides[key] = { "value" => RESTORE_SENTINEL, "original" => original }
42
- save_overrides(overrides)
52
+ entry = nil
53
+ update_overrides do |overrides|
54
+ entry = overrides[key]
55
+ # Keep a RESTORE entry so Sidekiq workers pick it up on next job
56
+ overrides[key] = { "value" => RESTORE_SENTINEL, "original" => entry["original"] } if entry
57
+ end
43
58
 
44
59
  # Apply immediately to the current (web) process
45
- original.nil? ? ENV.delete(key) : ENV[key] = original
60
+ restore_env(key, entry, blocked: blocked_reason)
46
61
  rescue => e
47
62
  warn "[Profiler] EnvOverrideStore: failed to reset #{key}: #{e.message}"
48
63
  end
49
64
 
50
65
  def reset_all
51
- overrides = load_overrides
52
- restore_overrides = {}
53
-
54
- overrides.each do |key, entry|
55
- original = entry.is_a?(Hash) ? entry["original"] : nil
56
- # Apply immediately to the current (web) process
57
- original.nil? ? ENV.delete(key) : ENV[key] = original
58
- # Leave a RESTORE sentinel for Sidekiq workers to pick up
59
- restore_overrides[key] = { "value" => RESTORE_SENTINEL, "original" => original }
66
+ originals = {}
67
+ update_overrides do |overrides|
68
+ overrides.each do |key, entry|
69
+ originals[key] = entry.is_a?(Hash) ? entry["original"] : nil
70
+ # Leave a RESTORE sentinel for Sidekiq workers to pick up
71
+ overrides[key] = { "value" => RESTORE_SENTINEL, "original" => originals[key] }
72
+ end
60
73
  end
61
74
 
62
- restore_overrides.empty? ? clear : save_overrides(restore_overrides)
75
+ # Apply immediately to the current (web) process
76
+ blocked = blocked_reason
77
+ originals.each { |key, original| restore_env(key, { "original" => original }, blocked: blocked) }
63
78
  rescue => e
64
79
  warn "[Profiler] EnvOverrideStore: failed to reset all: #{e.message}"
65
80
  end
66
81
 
82
+ # Why the persisted overrides must not reach ENV in this process, or nil when they may.
83
+ def blocked_reason
84
+ return :production if defined?(Rails) && Rails.respond_to?(:env) && Rails.env.production?
85
+
86
+ config = Profiler.configuration
87
+ return :disabled unless config.enabled || config.apply_env_overrides_when_disabled
88
+
89
+ nil
90
+ end
91
+
92
+ # Applies the overrides at boot, or says once in the log why they are left out. The message
93
+ # carries the file and the number of keys only, never a key or a value.
94
+ def apply_at_boot!(logger)
95
+ reason = blocked_reason
96
+ return apply! unless reason
97
+ return if @boot_warning_logged
98
+
99
+ count = all_overrides.size
100
+ return if count.zero?
101
+
102
+ @boot_warning_logged = true
103
+ noun, verb = count == 1 ? %w[override was] : %w[overrides were]
104
+ logger&.warn("[Profiler] #{count} persisted environment #{noun} in #{override_file_path} " \
105
+ "#{verb} not applied: #{BLOCKED_REASONS.fetch(reason)}.")
106
+ rescue => e
107
+ warn "[Profiler] EnvOverrideStore: failed to check overrides at boot: #{e.message}"
108
+ end
109
+
67
110
  def apply!
111
+ return if blocked_reason
112
+
68
113
  overrides = load_overrides
69
114
  restore_keys = []
70
115
 
@@ -72,6 +117,8 @@ module Profiler
72
117
  value = entry.is_a?(Hash) ? entry["value"] : entry
73
118
  original = entry.is_a?(Hash) ? entry["original"] : nil
74
119
 
120
+ remember_process_original(key) unless value == RESTORE_SENTINEL
121
+
75
122
  case value
76
123
  when DELETED_SENTINEL
77
124
  ENV.delete(key)
@@ -84,8 +131,13 @@ module Profiler
84
131
  end
85
132
 
86
133
  if restore_keys.any?
87
- restore_keys.each { |k| overrides.delete(k) }
88
- save_overrides(overrides)
134
+ # Read again under the lock: another writer may have changed the file since.
135
+ update_overrides do |current|
136
+ restore_keys.each do |k|
137
+ entry = current[k]
138
+ current.delete(k) if entry.is_a?(Hash) && entry["value"] == RESTORE_SENTINEL
139
+ end
140
+ end
89
141
  end
90
142
  rescue => e
91
143
  warn "[Profiler] EnvOverrideStore: failed to apply overrides: #{e.message}"
@@ -105,13 +157,34 @@ module Profiler
105
157
  end
106
158
 
107
159
  def clear
108
- FileUtils.rm_f(override_file_path)
160
+ with_lock { FileUtils.rm_f(override_file_path) }
109
161
  rescue => e
110
162
  warn "[Profiler] EnvOverrideStore: failed to clear: #{e.message}"
111
163
  end
112
164
 
113
165
  private
114
166
 
167
+ def remember_process_original(key)
168
+ @process_originals_lock.synchronize do
169
+ @process_originals[key] = ENV[key] unless @process_originals.key?(key)
170
+ end
171
+ end
172
+
173
+ # Puts a reset key back to its original value and returns whether ENV was written. Where the
174
+ # overrides are blocked, or the file has no entry for the key, only a key this process
175
+ # changed itself is touched, back to the value it had here; ENV is left alone for any other.
176
+ def restore_env(key, file_entry, blocked:)
177
+ changed_here, process_original = @process_originals_lock.synchronize do
178
+ [@process_originals.key?(key), @process_originals.delete(key)]
179
+ end
180
+ from_process = blocked || file_entry.nil?
181
+ return false if from_process && !changed_here
182
+
183
+ original = from_process ? process_original : file_entry["original"]
184
+ original.nil? ? ENV.delete(key) : ENV[key] = original
185
+ true
186
+ end
187
+
115
188
  def original_for(overrides, key)
116
189
  overrides.key?(key) ? overrides[key]["original"] : ENV[key]
117
190
  end
@@ -129,10 +202,36 @@ module Profiler
129
202
  {}
130
203
  end
131
204
 
132
- def save_overrides(overrides)
205
+ # Reads, changes and writes the file under the lock, so that the web process, its threads and
206
+ # the Sidekiq processes never save over each other's changes. The file is removed when the
207
+ # block leaves it empty.
208
+ def update_overrides
209
+ with_lock do
210
+ overrides = load_overrides
211
+ yield overrides
212
+ overrides.empty? ? FileUtils.rm_f(override_file_path) : save_overrides(overrides)
213
+ end
214
+ end
215
+
216
+ # An exclusive flock on a file next to the overrides file. Not reentrant: never nest it.
217
+ def with_lock
133
218
  path = override_file_path
134
219
  FileUtils.mkdir_p(path.dirname)
135
- File.write(path, JSON.generate(overrides))
220
+ File.open("#{path}.lock", File::RDWR | File::CREAT, 0o644) do |lock|
221
+ lock.flock(File::LOCK_EX)
222
+ yield
223
+ end
224
+ end
225
+
226
+ # Written to a temporary file then renamed over the old one, so that a reader never sees a
227
+ # half-written file, which would parse as no override at all.
228
+ def save_overrides(overrides)
229
+ path = override_file_path
230
+ tmp = "#{path}.#{Process.pid}.#{Thread.current.object_id}.tmp"
231
+ File.write(tmp, JSON.generate(overrides))
232
+ File.rename(tmp, path)
233
+ ensure
234
+ FileUtils.rm_f(tmp) if tmp
136
235
  end
137
236
  end
138
237
  end
@@ -28,6 +28,10 @@ module Profiler
28
28
  query = queries[query_index]
29
29
  sql = query["sql"].to_s
30
30
  binds = Array(query["binds"])
31
+ if binds.include?(Redaction::MASK)
32
+ raise ArgumentError, "EXPLAIN refused for query #{query_index}: one of its bind values was " \
33
+ "filtered when captured (config.filter_parameters), so the query cannot be rebuilt"
34
+ end
31
35
 
32
36
  conn = ActiveRecord::Base.connection
33
37
  adapter = conn.adapter_name.downcase
@@ -5,6 +5,7 @@ require "base64"
5
5
  require "zlib"
6
6
  require "stringio"
7
7
  require "securerandom"
8
+ require_relative "../redaction"
8
9
 
9
10
  module Profiler
10
11
  module Instrumentation
@@ -20,7 +21,7 @@ module Profiler
20
21
  host = address.to_s
21
22
  return super if NetHttpInstrumentation.skip_host?(host)
22
23
 
23
- url = build_url(host, port, req.path, use_ssl?)
24
+ url = Redaction.filter_url(build_url(host, port, req.path, use_ssl?))
24
25
  req_body = req.body.to_s
25
26
  # Fallback: body may be passed as the 2nd argument and only applied
26
27
  # to req inside super via req.set_body_internal(body)
@@ -43,7 +44,7 @@ module Profiler
43
44
  req_body = ""
44
45
  end
45
46
  end
46
- req_headers = req.to_hash.transform_values { |v| v.join(", ") }
47
+ req_headers = Redaction.filter_headers(req.to_hash.transform_values { |v| v.join(", ") })
47
48
  req_content_type = req["content-type"].to_s
48
49
  processed_req = req_body.empty? ? { body: nil, encoding: "text" } : NetHttpInstrumentation.process_body(req_body, req_content_type)
49
50
 
@@ -83,7 +84,7 @@ module Profiler
83
84
  entry,
84
85
  status: response.code.to_i,
85
86
  duration: duration,
86
- response_headers: response.to_hash.transform_values { |v| v.join(", ") },
87
+ response_headers: Redaction.filter_headers(response.to_hash.transform_values { |v| v.join(", ") }),
87
88
  response_body: processed_resp[:body],
88
89
  response_body_encoding: processed_resp[:encoding],
89
90
  response_size: resp_body_raw.bytesize
@@ -151,7 +152,8 @@ module Profiler
151
152
  if mime.match?(BINARY_CONTENT_TYPES)
152
153
  { body: Base64.strict_encode64(body.b), encoding: "base64" }
153
154
  else
154
- text = body.encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")
155
+ text = Redaction.filter_body(body, content_type)
156
+ .encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")
155
157
  { body: text, encoding: "text" }
156
158
  end
157
159
  end
@@ -2,6 +2,7 @@
2
2
 
3
3
  require_relative "models/profile"
4
4
  require_relative "current_context"
5
+ require_relative "redaction"
5
6
  require_relative "collectors/job_collector"
6
7
  require_relative "collectors/database_collector"
7
8
  require_relative "collectors/cache_collector"
@@ -115,7 +116,7 @@ module Profiler
115
116
  def sanitize_arguments(args)
116
117
  return [] unless args
117
118
 
118
- args.map do |arg|
119
+ Redaction.filter_value(args).map do |arg|
119
120
  case arg
120
121
  when String then arg.length > 200 ? "#{arg[0, 200]}..." : arg
121
122
  when Numeric, TrueClass, FalseClass, NilClass then arg
@@ -2,6 +2,7 @@
2
2
 
3
3
  require_relative "../slave_support"
4
4
  require_relative "../../env_override_store"
5
+ require_relative "../../redaction"
5
6
 
6
7
  module Profiler
7
8
  module MCP
@@ -19,8 +20,8 @@ module Profiler
19
20
  filter = params["filter"]&.downcase
20
21
 
21
22
  if include_all
22
- vars = ENV.to_h.sort.to_h
23
- overrides = Profiler.env_override_store.all_overrides
23
+ vars = Profiler::Redaction.env_snapshot
24
+ overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
24
25
 
25
26
  vars = vars.select { |k, _| k.downcase.include?(filter) } if filter
26
27
 
@@ -39,7 +40,7 @@ module Profiler
39
40
  text += "|-----|--------------|------------|\n"
40
41
  text += rows.join("\n")
41
42
  else
42
- overrides = Profiler.env_override_store.all_overrides
43
+ overrides = Profiler::Redaction.env_overrides(Profiler.env_override_store.all_overrides)
43
44
  overrides = overrides.select { |k, _| k.downcase.include?(filter) } if filter
44
45
 
45
46
  if overrides.empty?
@@ -23,11 +23,10 @@ module Profiler
23
23
  return [{ type: "text", text: "No active override for #{key}." }]
24
24
  end
25
25
 
26
- original = overrides[key]["original"]
27
- Profiler.env_override_store.reset(key)
26
+ restored = Profiler.env_override_store.reset(key)
28
27
 
29
- restored = original || "(unset)"
30
- [{ type: "text", text: "Reset #{key} to original value: #{restored}" }]
28
+ done = restored ? "original value restored" : "ENV left unchanged"
29
+ [{ type: "text", text: "Reset #{key}: override removed, #{done} in this process." }]
31
30
  end
32
31
  end
33
32
  end
@@ -1,6 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require_relative "../slave_support"
4
+ require_relative "../../redaction"
4
5
 
5
6
  module Profiler
6
7
  module MCP
@@ -11,6 +12,10 @@ module Profiler
11
12
  value = params["value"].to_s
12
13
 
13
14
  return [{ type: "text", text: "Error: key cannot be blank." }] if key.empty?
15
+ if Profiler::Redaction.mask?(value)
16
+ return [{ type: "text", text: "Error: #{Profiler::Redaction::MASK} is the mask the profiler shows " \
17
+ "in place of a hidden value, not a value; #{key} was left unchanged." }]
18
+ end
14
19
 
15
20
  if (proxy = MCP::SlaveSupport.with_slave_proxy(params))
16
21
  proxy.patch_json("/_profiler/api/env_vars", { key: key, value: value })
@@ -4,6 +4,7 @@ require "base64"
4
4
  require "securerandom"
5
5
  require "json"
6
6
  require "zlib"
7
+ require_relative "../redaction"
7
8
 
8
9
  module Profiler
9
10
  module Models
@@ -46,7 +47,7 @@ module Profiler
46
47
  @finished_at = Time.now
47
48
  @duration = ((@finished_at - @started_at) * 1000).round(2) # milliseconds
48
49
  @status = status
49
- @response_headers = response_headers
50
+ @response_headers = Redaction.filter_headers(response_headers)
50
51
  end
51
52
 
52
53
  def add_collector_data(name, data)
@@ -161,7 +162,7 @@ module Profiler
161
162
  if binary_content_type?(content_type)
162
163
  { body: Base64.strict_encode64(raw.b), encoding: "base64" }
163
164
  else
164
- text = raw.encode("UTF-8", invalid: :replace, undef: :replace)
165
+ text = Redaction.filter_body(raw, content_type).encode("UTF-8", invalid: :replace, undef: :replace)
165
166
  if compress_body?(text)
166
167
  { body: Base64.strict_encode64(Zlib::Deflate.deflate(text)), encoding: "gzip+base64" }
167
168
  else
@@ -189,10 +190,14 @@ module Profiler
189
190
  ct.to_s.match?(%r{image/(?!svg)|application/(?:pdf|octet-stream|zip)|audio/|video/})
190
191
  end
191
192
 
193
+ LEGACY_FILTERED_PARAMS = %w[password password_confirmation token secret].freeze
194
+
192
195
  def sanitize_params(params)
193
196
  return {} unless params
194
197
 
195
- params.to_h.except("password", "password_confirmation", "token", "secret")
198
+ return params.to_h.except(*LEGACY_FILTERED_PARAMS) unless Redaction.enabled?
199
+
200
+ Redaction.filter_hash(params.to_h)
196
201
  end
197
202
 
198
203
  ALLOWED_HEADERS = %w[
@@ -206,6 +211,7 @@ module Profiler
206
211
  env.select { |k, _| k.start_with?("HTTP_") }
207
212
  .transform_keys { |k| k.sub(/^HTTP_/, "").split("_").map(&:capitalize).join("-") }
208
213
  .select { |k, _| ALLOWED_HEADERS.include?(k) }
214
+ .then { |headers| Redaction.filter_headers(headers) }
209
215
  end
210
216
  end
211
217
  end
@@ -6,10 +6,6 @@ module Profiler
6
6
  class Railtie < Rails::Railtie
7
7
  config.profiler = ActiveSupport::OrderedOptions.new
8
8
 
9
- initializer "profiler.apply_env_overrides" do
10
- Profiler.env_override_store.apply!
11
- end
12
-
13
9
  initializer "profiler.set_configs" do |app|
14
10
  # Set default configuration for Rails environment
15
11
  Profiler.configure do |config|
@@ -103,6 +99,13 @@ module Profiler
103
99
  end
104
100
  end
105
101
 
102
+ # After the application's config/initializers, so that `enabled` is the application's own.
103
+ # Declared last: an initializer without `after:` runs after the one declared before it, so
104
+ # one declared below this one would move past the application's initializers too.
105
+ initializer "profiler.apply_env_overrides", after: :load_config_initializers do
106
+ Profiler.env_override_store.apply_at_boot!(Rails.logger)
107
+ end
108
+
106
109
  console do
107
110
  next unless Profiler.configuration.enabled && Profiler.configuration.track_console
108
111
 
@@ -0,0 +1,421 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "json"
4
+ require "set"
5
+ require "rack/utils"
6
+ require "active_support/parameter_filter"
7
+
8
+ module Profiler
9
+ # The single filter applied to everything the profiler captures: request
10
+ # params, headers, bodies and URLs (incoming and outgoing), SQL binds, job
11
+ # arguments, mailer arguments and ENV.
12
+ #
13
+ # It is built on ActiveSupport::ParameterFilter from the host application's
14
+ # Rails.application.config.filter_parameters plus
15
+ # Profiler.configuration.filter_parameters, so it follows Rails semantics:
16
+ # a symbol or string matches any key containing it, case-insensitively, at
17
+ # any nesting depth; a regexp is used as is; a proc rewrites values.
18
+ #
19
+ # It never raises: it runs inside notification subscribers and the Net::HTTP
20
+ # patch, where an exception would reach the application. When the filter
21
+ # fails (a proc that does not expect a value, say), the value is masked and
22
+ # the failure is logged once per error class, without the value.
23
+ module Redaction
24
+ MASK = "[FILTERED]"
25
+
26
+ # Matched by the filter on top of filter_parameters.
27
+ ALWAYS_FILTERED_HEADERS = %w[authorization proxy-authorization cookie set-cookie].freeze
28
+
29
+ # Headers whose value is a URL: their query string goes through the filter.
30
+ URL_HEADERS = %w[referer location content-location].freeze
31
+
32
+ JSON_TYPES = %r{\A(?:application|text)/(?:[\w.+-]+\+)?json\z}i
33
+ NDJSON_TYPES = %r{\A[\w.+-]+/x-ndjson\z}i
34
+ FORM_TYPE = "application/x-www-form-urlencoded"
35
+ MULTIPART_TYPE = %r{\Amultipart/}i
36
+
37
+ # The characters outside ASCII whose case folding yields ASCII letters
38
+ # (Unicode CaseFolding.txt): with any of them, a case-insensitive regexp
39
+ # can match an ASCII filter where a plain ASCII search cannot.
40
+ FOLDS_TO_ASCII = /[ßİʼnſǰẖ-ẚẞKff-st]/
41
+
42
+ # What makes a regexp filter unfit for a search of the whole body text:
43
+ # anchors, word boundaries and lookarounds can match a key on its own and
44
+ # fail on the same key inside the text.
45
+ CONTEXT_SENSITIVE_REGEXP = /\\[AzZbBG]|[\^$]|\(\?<?[=!]/
46
+
47
+ # Bound on the memo of names already tested against the filter.
48
+ KEY_CACHE_LIMIT = 10_000
49
+
50
+ # The value a name alone is tested with: procs in filter_parameters
51
+ # rewrite it in place, so it has to be a string, and a throwaway one.
52
+ PROBE = "x"
53
+
54
+ class << self
55
+ def enabled?
56
+ Profiler.configuration.redact_sensitive_data != false
57
+ end
58
+
59
+ # The ParameterFilter, rebuilt only when the list of filters changes.
60
+ def parameter_filter
61
+ compiled[:filter]
62
+ end
63
+
64
+ # Keys are masked by the filters that are not procs; the procs then
65
+ # rewrite each string left, once, under its own key. Running the procs
66
+ # through ParameterFilter instead would dup every other value first,
67
+ # Active Record models included.
68
+ def filter_hash(hash)
69
+ return hash unless enabled? && hash.is_a?(Hash)
70
+
71
+ state = compiled
72
+ masked = state[:plain].filter(hash)
73
+ state[:proc_filter] ? apply_procs_tree(masked, nil, hash, state[:procs]) : masked
74
+ rescue StandardError => e
75
+ report(e, "filtering a hash")
76
+ hash.transform_values { MASK }
77
+ end
78
+
79
+ # Filters any value: hashes through the filter, arrays element by
80
+ # element, anything else unchanged (it carries no key to match).
81
+ def filter_value(value)
82
+ return value unless enabled?
83
+
84
+ case value
85
+ when Hash then filter_hash(value)
86
+ when Array then value.map { |v| filter_value(v) }
87
+ else value
88
+ end
89
+ rescue StandardError => e
90
+ report(e, "filtering a value")
91
+ MASK
92
+ end
93
+
94
+ # A value known by its name (SQL bind, mailer argument): MASK when the
95
+ # name matches the filter, else the value as the procs of
96
+ # filter_parameters rewrite it.
97
+ def filter_named(name, value)
98
+ return value unless enabled? && name
99
+ return MASK if sensitive_key?(name)
100
+
101
+ apply_procs([name.to_s], value)
102
+ end
103
+
104
+ # Whether the name alone matches the filter. Memoized per name: header,
105
+ # ENV, column and key names come from a small set. A filter that raises
106
+ # makes the name sensitive.
107
+ def sensitive_key?(name)
108
+ key = name.to_s
109
+ state = compiled
110
+ known = state[:keys]
111
+ return known[key] if known.key?(key)
112
+
113
+ result = state[:plain].filter(key => +PROBE)[key] == MASK
114
+ known[key] = result if known.size < KEY_CACHE_LIMIT
115
+ result
116
+ rescue StandardError => e
117
+ report(e, "testing a name")
118
+ true
119
+ end
120
+
121
+ # Header names are tested as written and with dashes as underscores, so
122
+ # that X-Api-Key matches :_key as x_api_key would.
123
+ def sensitive_header?(name)
124
+ key = name.to_s.downcase
125
+ ALWAYS_FILTERED_HEADERS.include?(key) || sensitive_key?(key) || sensitive_key?(key.tr("-", "_"))
126
+ end
127
+
128
+ def filter_headers(headers)
129
+ return headers unless enabled? && headers.respond_to?(:each_with_object)
130
+
131
+ headers.each_with_object({}) do |(name, value), out|
132
+ out[name] = filter_header(name, value)
133
+ end
134
+ end
135
+
136
+ # Filters a raw body according to its content type. JSON, NDJSON and
137
+ # form-urlencoded bodies have their filtered keys masked, multipart
138
+ # bodies are masked entirely, and every other type is kept as is: they
139
+ # carry no key the filter can read, and masking HTML would hide the very
140
+ # pages being profiled.
141
+ #
142
+ # Bodies usually arrive as binary strings (rack.input, Net::HTTP): they
143
+ # are read as UTF-8, or as bytes when they are not valid UTF-8.
144
+ def filter_body(raw, content_type)
145
+ return raw unless enabled? && raw.is_a?(String) && !raw.empty?
146
+
147
+ mime = content_type.to_s.split(";").first.to_s.strip.downcase
148
+ text = readable(raw)
149
+ if mime.match?(JSON_TYPES)
150
+ filter_json(text, mime)
151
+ elsif mime.match?(NDJSON_TYPES)
152
+ text.split("\n", -1).map { |line| line.strip.empty? ? line : filter_json(line, mime) }.join("\n")
153
+ elsif mime == FORM_TYPE
154
+ filter_query(text)
155
+ elsif mime.match?(MULTIPART_TYPE)
156
+ "[FILTERED: #{mime} body, #{raw.bytesize} bytes]"
157
+ else
158
+ raw
159
+ end
160
+ rescue StandardError => e
161
+ report(e, "filtering a #{mime} body")
162
+ "[FILTERED: #{mime} body that could not be filtered, #{raw.bytesize} bytes]"
163
+ end
164
+
165
+ # Masks the filtered values of a query string, pair by pair, leaving the
166
+ # rest of the string exactly as it was.
167
+ def filter_query(query)
168
+ return query unless enabled? && query.is_a?(String) && !query.empty?
169
+
170
+ readable(query).split("&", -1).map do |pair|
171
+ name, value = pair.split("=", 2)
172
+ next pair if value.nil? || value.empty?
173
+
174
+ parsed = Rack::Utils.parse_nested_query(pair)
175
+ parameter_filter.filter(parsed) == parsed ? pair : "#{name}=#{MASK}"
176
+ rescue StandardError
177
+ # A pair Rack or the filter cannot handle is masked rather than trusted.
178
+ "#{name}=#{MASK}"
179
+ end.join("&")
180
+ end
181
+
182
+ # A URL, or a list of them as Rack 3 gives a repeated header.
183
+ def filter_url(url)
184
+ return url unless enabled?
185
+ return url.map { |u| filter_url(u) } if url.is_a?(Array)
186
+ return url unless url.is_a?(String)
187
+
188
+ rest, fragment = readable(url).split("#", 2)
189
+ base, query = rest.split("?", 2)
190
+ return url if query.nil?
191
+
192
+ "#{base}?#{filter_query(query)}#{"##{fragment}" if fragment}"
193
+ rescue StandardError => e
194
+ report(e, "filtering a URL")
195
+ MASK
196
+ end
197
+
198
+ # ENV as the profiler shows it: names kept, values masked unless the
199
+ # name is in config.env_allowlist and does not match the filter.
200
+ def env_snapshot(env = ENV.to_h)
201
+ listed = env_allowlist_matcher
202
+ env.sort.to_h { |name, value| [name, env_visible?(name, listed) ? apply_procs([name], value) : MASK] }
203
+ end
204
+
205
+ def env_value(name, value)
206
+ env_visible?(name) ? apply_procs([name.to_s], value) : MASK
207
+ end
208
+
209
+ # ENV overrides with the same rule applied to their current and original
210
+ # values. The deleted and restore markers are kept: they are not values.
211
+ def env_overrides(overrides)
212
+ overrides.to_h do |name, entry|
213
+ next [name, entry] if env_visible?(name) || !entry.is_a?(Hash)
214
+
215
+ masked = entry.to_h do |field, value|
216
+ [field, keep_override_field?(field, value) ? value : MASK]
217
+ end
218
+ [name, masked]
219
+ end
220
+ end
221
+
222
+ def env_visible?(name, listed = env_allowlist_matcher)
223
+ listed.call(name.to_s) && !(enabled? && sensitive_key?(name))
224
+ end
225
+
226
+ # Whether +value+, stripped, is the mask: a value exported while masked,
227
+ # which must never be written back as if it were the real one.
228
+ def mask?(value)
229
+ value.to_s.strip == MASK
230
+ end
231
+
232
+ private
233
+
234
+ def compiled
235
+ filters = current_filters
236
+ state = @compiled
237
+ return state if state && state[:filters] == filters
238
+
239
+ @reported = Set.new
240
+ procs, plain = filters.partition { |f| f.respond_to?(:call) }
241
+ @compiled = {
242
+ filters: filters,
243
+ filter: ActiveSupport::ParameterFilter.new(filters, mask: MASK),
244
+ plain: ActiveSupport::ParameterFilter.new(plain, mask: MASK),
245
+ proc_filter: procs.empty? ? nil : ActiveSupport::ParameterFilter.new(procs, mask: MASK),
246
+ procs: procs,
247
+ keys: {}
248
+ }.merge(text_matchers(filters))
249
+ end
250
+
251
+ # What the JSON pre-test needs: the string filters, downcased, for a
252
+ # plain search, and as one case-insensitive alternation as
253
+ # ParameterFilter builds it, for text outside ASCII; the regexp filters
254
+ # that can be searched for in the whole text. Procs rewrite values
255
+ # whatever the key, dotted filters match a key path, and other regexps
256
+ # depend on where the key ends: any of them makes the filters opaque,
257
+ # and every body is then parsed.
258
+ def text_matchers(filters)
259
+ procs = filters.any? { |f| f.respond_to?(:call) }
260
+ regexps, strings = filters.reject { |f| f.respond_to?(:call) }.partition { |f| f.is_a?(Regexp) }
261
+ opaque = procs ||
262
+ filters.any? { |f| !f.respond_to?(:call) && f.to_s.include?(".") } ||
263
+ regexps.any? { |r| r.source.match?(CONTEXT_SENSITIVE_REGEXP) } ||
264
+ strings.any? { |s| !s.to_s.ascii_only? }
265
+ words = strings.map { |s| s.to_s.downcase }
266
+ {
267
+ opaque: opaque,
268
+ words: words,
269
+ words_regexp: words.empty? ? nil : Regexp.new(words.map { |w| Regexp.escape(w) }.join("|"), Regexp::IGNORECASE),
270
+ regexps: regexps
271
+ }
272
+ end
273
+
274
+ def filter_header(name, value)
275
+ return MASK if sensitive_header?(name)
276
+
277
+ value = filter_url(value) if URL_HEADERS.include?(name.to_s.downcase)
278
+ apply_procs([name.to_s, name.to_s.tr("-", "_")], value)
279
+ rescue StandardError => e
280
+ report(e, "filtering a header")
281
+ MASK
282
+ end
283
+
284
+ # The value as the procs of filter_parameters rewrite it under each of
285
+ # +names+. Nothing to do without procs: the names were already tested.
286
+ #
287
+ # Only strings: any other object would be duplicated for the procs,
288
+ # which runs the application's copy hooks; it is shown through its
289
+ # inspect, as before. The procs run once: under the first of +names+
290
+ # that changes the value, so that a proc which undoes itself (reverse!)
291
+ # is not applied twice.
292
+ def apply_procs(names, value)
293
+ return value unless enabled? && value.is_a?(String)
294
+
295
+ filter = compiled[:proc_filter]
296
+ return value unless filter
297
+
298
+ names.uniq.each do |name|
299
+ result = filter.filter(name => value)[name]
300
+ return result unless result == value
301
+ end
302
+ value
303
+ rescue StandardError => e
304
+ report(e, "applying a filter_parameters proc")
305
+ MASK
306
+ end
307
+
308
+ # Walks a tree the non-proc filters have already masked and lets the
309
+ # procs rewrite each string leaf, once, as ParameterFilter does: a copy
310
+ # of the key and of the value, and the original params for a proc of
311
+ # arity 3. A leaf masked by the filter is the MASK object itself and is
312
+ # left alone, also as ParameterFilter does; a value that only reads
313
+ # "[FILTERED]" is a different object and still goes to the procs.
314
+ def apply_procs_tree(value, key, original, procs)
315
+ case value
316
+ when Hash then value.each_with_object(value.class.new) { |(k, v), out| out[k] = apply_procs_tree(v, k, original, procs) }
317
+ when Array then value.map { |v| apply_procs_tree(v, key, original, procs) }
318
+ when String then key.nil? || value.equal?(MASK) ? value : call_procs(procs, key, value, original)
319
+ else value
320
+ end
321
+ end
322
+
323
+ def call_procs(procs, key, value, original)
324
+ key = key.dup if key.duplicable?
325
+ value = value.dup
326
+ procs.each { |b| b.arity == 2 ? b.call(key, value) : b.call(key, value, original) }
327
+ value
328
+ rescue StandardError => e
329
+ report(e, "applying a filter_parameters proc")
330
+ MASK
331
+ end
332
+
333
+ def keep_override_field?(field, value)
334
+ value.nil? ||
335
+ value == EnvOverrideStore::DELETED_SENTINEL ||
336
+ value == EnvOverrideStore::RESTORE_SENTINEL ||
337
+ !%w[value original].include?(field)
338
+ end
339
+
340
+ def env_allowlist_matcher
341
+ allowlist = Profiler.configuration.env_allowlist
342
+ return ->(_name) { true } if allowlist == :all
343
+
344
+ regexps, names = Array(allowlist).partition { |entry| entry.is_a?(Regexp) }
345
+ names = names.to_set(&:to_s)
346
+ ->(name) { names.include?(name) || regexps.any? { |r| r.match?(name) } }
347
+ end
348
+
349
+ def current_filters
350
+ filters = Array(Profiler.configuration.filter_parameters)
351
+ app_filters = rails_filter_parameters
352
+ app_filters.empty? ? filters : app_filters + filters
353
+ end
354
+
355
+ def rails_filter_parameters
356
+ return [] unless defined?(::Rails) && ::Rails.respond_to?(:application) && ::Rails.application
357
+
358
+ Array(::Rails.application.config.filter_parameters)
359
+ rescue StandardError
360
+ []
361
+ end
362
+
363
+ # The string as UTF-8 when it is valid UTF-8, else as bytes, so that
364
+ # string operations and regexps never meet an incompatible encoding.
365
+ def readable(string)
366
+ return string if string.encoding == Encoding::UTF_8 && string.valid_encoding?
367
+
368
+ utf8 = string.dup.force_encoding(Encoding::UTF_8)
369
+ utf8.valid_encoding? ? utf8 : string.b
370
+ end
371
+
372
+ def filter_json(text, mime)
373
+ return text unless json_needs_parse?(text)
374
+
375
+ parsed = JSON.parse(text)
376
+ filtered = filter_value(parsed)
377
+ filtered == parsed ? text : JSON.generate(filtered)
378
+ rescue JSON::ParserError, JSON::GeneratorError
379
+ "[FILTERED: unparseable #{mime} body, #{text.bytesize} bytes]"
380
+ end
381
+
382
+ # A cheap pre-test that spares the parse of a body in whose text no
383
+ # filter matches anywhere: no key of it can match either. Any match, in
384
+ # a key or in a value, sends the body to the parser, which also masks
385
+ # an unreadable body entirely. Text that is not valid UTF-8, holds an
386
+ # escape (a key can be written password) or, for a plain ASCII
387
+ # search, a character that case-folds onto ASCII letters is not judged
388
+ # here, nor is any body under opaque filters.
389
+ def json_needs_parse?(text)
390
+ state = compiled
391
+ return true if state[:opaque]
392
+ return true if text.encoding != Encoding::UTF_8 || text.include?("\\")
393
+
394
+ state[:regexps].any? { |r| r.match?(text) } || words_in_text?(state, text)
395
+ end
396
+
397
+ def words_in_text?(state, text)
398
+ words = state[:words]
399
+ return false if words.empty?
400
+ return state[:words_regexp].match?(text) unless text.ascii_only? || !text.match?(FOLDS_TO_ASCII)
401
+
402
+ lowered = text.downcase(:ascii)
403
+ words.any? { |word| lowered.include?(word) }
404
+ end
405
+
406
+ # Logs a failure of the filter once per error class, with no value and
407
+ # no message: either could carry what was being masked.
408
+ def report(error, during)
409
+ reported = (@reported ||= Set.new)
410
+ return if reported.include?(error.class)
411
+
412
+ reported << error.class
413
+ message = "[Profiler] Redaction: #{error.class} while #{during}; the value was masked instead."
414
+ logger = ::Rails.logger if defined?(::Rails) && ::Rails.respond_to?(:logger)
415
+ logger ? logger.warn(message) : Kernel.warn(message)
416
+ rescue StandardError
417
+ nil
418
+ end
419
+ end
420
+ end
421
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Profiler
4
- VERSION = "0.30.6"
4
+ VERSION = "0.30.8"
5
5
  end
data/lib/profiler.rb CHANGED
@@ -2,6 +2,7 @@
2
2
 
3
3
  require_relative "profiler/version"
4
4
  require_relative "profiler/configuration"
5
+ require_relative "profiler/redaction"
5
6
 
6
7
  module Profiler
7
8
  class Error < StandardError; end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rails-profiler
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.30.6
4
+ version: 0.30.8
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sébastien Duplessy
@@ -209,6 +209,7 @@ files:
209
209
  - lib/profiler/models/sql_query.rb
210
210
  - lib/profiler/models/timeline_event.rb
211
211
  - lib/profiler/railtie.rb
212
+ - lib/profiler/redaction.rb
212
213
  - lib/profiler/sse/bus.rb
213
214
  - lib/profiler/sse/event_bus.rb
214
215
  - lib/profiler/sse/redis_event_bus.rb