logstruct 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +42 -0
  3. data/README.md +1 -1
  4. data/lib/log_struct/bounded_input.rb +101 -0
  5. data/lib/log_struct/concerns/configuration.rb +10 -10
  6. data/lib/log_struct/concerns/error_handling.rb +21 -0
  7. data/lib/log_struct/config_struct/filters.rb +23 -9
  8. data/lib/log_struct/config_struct/limits.rb +1 -1
  9. data/lib/log_struct/enums/log_field.rb +0 -3
  10. data/lib/log_struct/formatter.rb +54 -58
  11. data/lib/log_struct/hash_utils.rb +11 -4
  12. data/lib/log_struct/integrations/ahoy.rb +3 -2
  13. data/lib/log_struct/integrations/carrierwave.rb +3 -6
  14. data/lib/log_struct/integrations/lograge.rb +4 -10
  15. data/lib/log_struct/integrations/shrine.rb +1 -1
  16. data/lib/log_struct/log/action_mailer/delivered.rb +10 -0
  17. data/lib/log_struct/log/action_mailer/delivery.rb +10 -0
  18. data/lib/log_struct/log/action_mailer/error.rb +10 -0
  19. data/lib/log_struct/log/carrierwave/delete.rb +0 -4
  20. data/lib/log_struct/log/carrierwave/download.rb +0 -6
  21. data/lib/log_struct/log/carrierwave/upload.rb +0 -6
  22. data/lib/log_struct/log/carrierwave.rb +2 -8
  23. data/lib/log_struct/log/error.rb +10 -0
  24. data/lib/log_struct/log/plain.rb +10 -0
  25. data/lib/log_struct/log/puma/error.rb +10 -0
  26. data/lib/log_struct/log/puma/shutdown.rb +10 -0
  27. data/lib/log_struct/log/puma/start.rb +10 -0
  28. data/lib/log_struct/log/request.rb +10 -0
  29. data/lib/log_struct/log/security/blocked_host.rb +10 -0
  30. data/lib/log_struct/log/security/csrf_violation.rb +10 -0
  31. data/lib/log_struct/log/security/ip_spoof.rb +10 -0
  32. data/lib/log_struct/log/sidekiq/error.rb +10 -0
  33. data/lib/log_struct/log/sidekiq/log.rb +10 -0
  34. data/lib/log_struct/log/sql.rb +10 -0
  35. data/lib/log_struct/monkey_patches/active_support/tagged_logging/formatter.rb +5 -3
  36. data/lib/log_struct/param_filters.rb +10 -28
  37. data/lib/log_struct/shared/merge_additional_data_fields.rb +21 -5
  38. data/lib/log_struct/string_scrubber.rb +7 -6
  39. data/lib/log_struct/version.rb +1 -1
  40. data/logstruct.gemspec +1 -1
  41. metadata +4 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 23182266ea1c9f53e0b824a8b806207b92255ed6a6d2755cf67c8f3d35680d19
4
- data.tar.gz: 4d684c903461a10aa875314f6dac3fca9978ac190ce0616c3f6d34076c3b8ba7
3
+ metadata.gz: 647c97ba4368b67d39333204ca35a5161b0905dda2c8ef72d01e15d58665d367
4
+ data.tar.gz: e86e688d516ef584c5f8994ecb0739599c09595a021ebc949f494e6d631f349d
5
5
  SHA512:
6
- metadata.gz: 4a3dc3b56efac2d68703b77210006852059188425e612f537be307e7740da79781600dc4147d955c52c0e65a39978d476fefd0a796b07b7fe5ee170d2ce5b07a
7
- data.tar.gz: bb328f655eda7dbcfb2ace155586966bfc47a4a60df88dcb7aa2e355b5213f0e76623d52e83906063a5aaff63039c8937c59bdd373c783faa7013a7c860acfac
6
+ metadata.gz: 6709f7868f43c348891761749400bb93c762e3df6f2d7093dab12f51c50a2ce33e26bfd94dab009093a1c4207f746735764c56ce20768394a9b8e81cce5d346d
7
+ data.tar.gz: 7ab05d71a64c08cd4ad747c7684d3b75553e2740c0539cefa2cab2f906f2c5023dd8696bb78129790472896f83008addde117f9e41229d563ad9003a76911fd9
data/CHANGELOG.md CHANGED
@@ -5,6 +5,48 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [Unreleased]
9
+
10
+ ## [0.3.0] - 2026-10-23
11
+
12
+ ### Changed
13
+
14
+ - **Breaking**: CarrierWave lines log `file_id` as an HMAC of the file's identifier (keyed like email hashes, see `hash_salt`), and no longer log `filename`, `store_path` or `ext`. CarrierWave names stored files after the uploaded file by default, and an uploaded file's name often names a person. `LogField::Filename`, `LogField::StorePath` and `LogField::Extension` are removed.
15
+ - **Breaking**: A filtered Hash's summary has its `_keys_count` but not its first 10 keys (`_keys`): a Hash's keys can be secrets themselves, like tokens or email addresses used as keys.
16
+ - **Breaking**: `additional_data` can't set any field of its line's own type, whether or not the line has it (an error line's `msg`, `backtrace` or `error_message`), or `req_id`. Like `src`, `evt`, `lvl` and `ts`, such a key is left out and handled as a LogStruct error (raised in `local_environments`). It used to replace the field. Other types' field names, like `name` on a plain line, can still be used.
17
+ - **Filtering**: Without a `hash_salt`, values hashed before the Rails app has its `secret_key_base` use a random key for each process, instead of a fixed key that was public in LogStruct's source. Set `hash_salt` to keep these hashes the same across processes.
18
+ - **Development**: `task ci` checks the locked Ruby and docs dependencies for known advisories, with `bundler-audit` and `pnpm audit`.
19
+
20
+ ### Fixed
21
+
22
+ - **Formatting**: Hash keys longer than `max_string_bytes` are left out with their values, and counted in `_oversized_keys`. A request's parameter names could make a line any size.
23
+ - **Formatting**: `ActionController::Parameters` are written like a Hash, within the size limits, instead of being converted whole first. Lograge's `params` and Ahoy's `properties` are converted at their first level only, and only as many keys as the line keeps; `_more_keys` and `_oversized_keys` count the rest.
24
+ - **Filtering**: A Rails filter block with three arguments gets a copy of the parameters that stops at 10,000 values, 20 levels of nesting or a value that contains itself. Past those, every key the block would check is filtered. The parameters used to be copied whole.
25
+ - **Errors**: An error raised while formatting, serializing or filtering a value is reported with its class and backtrace but not its message, which can quote the value. Errors in Rails filter blocks and LogStruct's filter matchers aren't reported with the key, which can be the secret. An object whose `as_json` raises is logged as `[SERIALIZATION_ERROR]` instead of its `to_s`, and its `inspect` isn't reported.
26
+ - **Formatting**: A Hash key that isn't a String or Symbol is written as its string, scrubbed, so an email address in its `to_s` is scrubbed.
27
+ - **Filtering**: A value longer than `max_string_bytes` is hashed by its first `max_string_bytes` bytes and its length.
28
+ - **Development**: Update Next.js to 16.4.1 for the docs site, which fixes three advisories in features the static site doesn't use, and the test-only Dalli to 3.2.13.
29
+
30
+ ## [0.2.2] - 2026-10-21
31
+
32
+ ### Changed
33
+
34
+ - **Development**: Update Prettier, cspell and pnpm's setup action. The development tools require Node.js 22.18 or later in the 22.x series.
35
+ - **Docs**: Upgrade the site to Next.js 16 and align its Node type declarations with the Node.js 22 runtime used in CI.
36
+
37
+ ### Fixed
38
+
39
+ - **Filtering**: Nested URL query parameters such as `password[]` and `token[value]` are filtered, including URL-encoded names and paths in Rails' `filter_parameters`.
40
+ - **Filtering**: Basic credentials are scrubbed even when their Base64 encoding is shorter than eight characters.
41
+ - **Filtering**: Symbols beyond the nesting limit are summarized instead of being written without scrubbing.
42
+ - **Filtering**: When a key or path matcher raises and error handling lets logging continue, its value is filtered. This also applies to Rails filter blocks.
43
+ - **Errors**: A failed filter or scrubber does not keep queuing or reporting errors when it fails again while formatting its own error line or report context.
44
+ - **Formatting**: Hash keys are normalized during filtering, so arrays, hashes and filtered values are no longer copied recursively before size limits are applied. Circular references are summarized at their first recurrence.
45
+ - **Docs**: Correct the supported security release series and contributor instructions for releases, generated log types and schema fields. Describe size limits as limits per value, not a guarantee on total formatting time. Ensure the site's integration tests run when a local Watchman installation is broken.
46
+ - **Development**: `task ci` verifies committed RBI files, matching GitHub CI, instead of downloading new annotations concurrently with typechecking.
47
+ - **Generation**: A failed formatter or Sorbet JSON export stops code generation instead of leaving incomplete artifacts with a successful exit status.
48
+ - **Development**: Update the locked `crass` dependency to 1.0.7, which fixes a denial-of-service issue in CSS number parsing.
49
+
8
50
  ## [0.2.1] - 2026-10-18
9
51
 
10
52
  ### Fixed
data/README.md CHANGED
@@ -28,7 +28,7 @@ any other log platform. Sensitive values are filtered and scrubbed before a line
28
28
  `Bearer` and `Basic` credentials and sensitive URL query params (`?token=`) are replaced too. IP and MAC address
29
29
  scrubbing are opt-in.
30
30
  - **Limits.** Huge values are cut down (100 keys per Hash, 10 items per Array, 20 lines per backtrace, 32 KB per
31
- String by default), so one large request body can't slow down your logging.
31
+ String by default), reducing log size and the work spent formatting large values.
32
32
  - **Error handling.** A mode for each kind of error decides whether it's raised, logged or reported, and whether
33
33
  that differs between development and production. Reports go to Sentry, Bugsnag, Rollbar, Honeybadger or your own
34
34
  reporter.
@@ -0,0 +1,101 @@
1
+ # typed: strict
2
+ # frozen_string_literal: true
3
+
4
+ require "action_controller"
5
+
6
+ module LogStruct
7
+ # Copies of logged Hashes and Arrays that stop before an input too large to copy in full
8
+ module BoundedInput
9
+ extend T::Sig
10
+
11
+ # A Rails filter block with three arguments gets every parameter, so a copy for it keeps what the formatter
12
+ # leaves out of the line. These limits only stop values no request sends: nesting deeper than the formatter
13
+ # writes, and more values than a form or JSON body has. A copy that reaches one is truncated, and the
14
+ # filters that read it fail closed.
15
+ MAX_DEPTH = 20
16
+ MAX_NODES = 10_000
17
+
18
+ # The keys that count what a Hash's copy left out. The formatter writes them as it writes its own counts:
19
+ # Rails' default filter_parameters include :_key, which would filter them.
20
+ COUNT_KEYS = T.let(Set[:_more_keys, :_oversized_keys].freeze, T::Set[Symbol])
21
+
22
+ class State < T::Struct
23
+ prop :remaining_nodes, Integer, default: MAX_NODES
24
+ prop :truncated, T::Boolean, default: false
25
+ end
26
+
27
+ sig do
28
+ params(value: T.untyped, state: State, depth: Integer, ancestors: T::Hash[T.untyped, T::Boolean]).returns(T.untyped)
29
+ end
30
+ def self.copy(value, state:, depth: 0, ancestors: {}.compare_by_identity)
31
+ if depth > MAX_DEPTH || ancestors.key?(value) || state.remaining_nodes <= 0
32
+ state.truncated = true
33
+ return nil
34
+ end
35
+ state.remaining_nodes -= 1
36
+
37
+ case value
38
+ when Hash, ::ActionController::Parameters
39
+ ancestors[value] = true
40
+ begin
41
+ result = {}
42
+ value.each_pair do |key, child|
43
+ break if state.truncated
44
+
45
+ result[key] = copy(child, state: state, depth: depth + 1, ancestors: ancestors)
46
+ end
47
+ result
48
+ ensure
49
+ ancestors.delete(value)
50
+ end
51
+ when Array
52
+ ancestors[value] = true
53
+ begin
54
+ result = []
55
+ value.each do |child|
56
+ break if state.truncated
57
+
58
+ result << copy(child, state: state, depth: depth + 1, ancestors: ancestors)
59
+ end
60
+ result
61
+ ensure
62
+ ancestors.delete(value)
63
+ end
64
+ else
65
+ value
66
+ end
67
+ end
68
+
69
+ # How many keys a Hash or ActionController::Parameters has, without converting its values
70
+ sig { params(hash: T.untyped).returns(Integer) }
71
+ def self.key_count(hash)
72
+ hash.is_a?(Hash) ? hash.size : hash.keys.size
73
+ end
74
+
75
+ # The first keys of a Hash or ActionController::Parameters as Symbols, for a log struct's Hash field, without
76
+ # converting the keys the formatter would leave out. The formatter writes max_hash_keys entries of a Hash,
77
+ # and the counts of the keys left out here are entries too: the last key's slot goes to _more_keys, and an
78
+ # omitted oversized key's slot to _oversized_keys, so the formatter writes both counts as they are.
79
+ sig { params(hash: T.untyped).returns(T::Hash[Symbol, T.untyped]) }
80
+ def self.symbolize_keys(hash)
81
+ limits = LogStruct.config.limits
82
+ total = key_count(hash)
83
+ examined = (total > limits.max_hash_keys) ? [limits.max_hash_keys - 1, 0].max : total
84
+ result = T.let({}, T::Hash[Symbol, T.untyped])
85
+ oversized_keys = 0
86
+ hash.each_pair.with_index do |(key, value), index|
87
+ break if index >= examined
88
+
89
+ if key.to_s.bytesize > limits.max_string_bytes
90
+ oversized_keys += 1
91
+ next
92
+ end
93
+
94
+ result[key.to_s.to_sym] = value
95
+ end
96
+ result[:_more_keys] = total - examined if examined < total
97
+ result[:_oversized_keys] = oversized_keys if oversized_keys.positive?
98
+ result
99
+ end
100
+ end
101
+ end
@@ -259,8 +259,8 @@ module LogStruct
259
259
  result = filter.call(key, value, root)
260
260
  !!result || (value.is_a?(String) && value != original_value)
261
261
  rescue => e
262
- handle_filter_error(e, filter, original_key || "")
263
- false
262
+ handle_filter_error(e, filter)
263
+ true
264
264
  end
265
265
  ConfigStruct::PathFilterMatcher.new(callable: callable, label: filter.inspect)
266
266
  end
@@ -290,12 +290,12 @@ module LogStruct
290
290
  begin
291
291
  !!filter.call(key)
292
292
  rescue => e
293
- handle_filter_error(e, filter, key)
294
- false
293
+ handle_filter_error(e, filter)
294
+ true
295
295
  end
296
296
  rescue => e
297
- handle_filter_error(e, filter, key)
298
- false
297
+ handle_filter_error(e, filter)
298
+ true
299
299
  end
300
300
  ConfigStruct::FilterMatcher.new(callable: callable, label: filter.inspect)
301
301
  end
@@ -305,11 +305,11 @@ module LogStruct
305
305
  filter.respond_to?(:arity) ? filter.arity : 2
306
306
  end
307
307
 
308
- sig { params(error: StandardError, filter: T.untyped, key: String).void }
309
- def handle_filter_error(error, filter, key)
308
+ # The key isn't reported: a key can be the secret (an email address or a token used as a key)
309
+ sig { params(error: StandardError, filter: T.untyped).void }
310
+ def handle_filter_error(error, filter)
310
311
  context = {
311
312
  filter: filter.class.name,
312
- key: key,
313
313
  filter_label: begin
314
314
  filter.inspect
315
315
  rescue
@@ -317,7 +317,7 @@ module LogStruct
317
317
  end
318
318
  }
319
319
 
320
- LogStruct.handle_exception(error, source: Source::Internal, context: context)
320
+ LogStruct.handle_value_error(error, "filter_parameters entry failed", context: context)
321
321
  end
322
322
  end
323
323
  end
@@ -78,10 +78,25 @@ module LogStruct
78
78
  # request's parameters or a mail's recipients
79
79
  sig { params(context: T.nilable(T::Hash[Symbol, T.untyped])).returns(T.nilable(T::Hash[Symbol, T.untyped])) }
80
80
  def filtered_context(context)
81
+ previous_internal_error = Thread.current[:logstruct_formatting_internal_error]
82
+ # The same filter may fail on the error's context. Keep that failure from reporting itself.
83
+ Thread.current[:logstruct_formatting_internal_error] = true
81
84
  return context if context.nil? || context.empty?
82
85
 
83
86
  filtered = Formatter.new.process_values(context)
84
87
  filtered.is_a?(Hash) ? filtered : {}
88
+ ensure
89
+ Thread.current[:logstruct_formatting_internal_error] = previous_internal_error
90
+ end
91
+
92
+ # Handle an error raised while LogStruct read a value it was logging, without the error's message: the
93
+ # message can quote that value, as a NoMethodError quotes its receiver's inspect before Ruby 3.4.
94
+ # failure says what failed, and the reported error keeps the original's class and backtrace.
95
+ sig { params(error: Exception, failure: String, context: T.nilable(T::Hash[Symbol, T.untyped])).void }
96
+ def handle_value_error(error, failure, context: nil)
97
+ reported = RuntimeError.new("#{error.class}: #{failure}")
98
+ reported.set_backtrace(error.backtrace)
99
+ handle_exception(reported, source: Source::Internal, context: context)
85
100
  end
86
101
 
87
102
  # Handle an error according to the configured error handling mode (log, report, raise, etc).
@@ -95,6 +110,12 @@ module LogStruct
95
110
  raise(error) if !LogStruct.is_production?
96
111
  end
97
112
 
113
+ # A broken filter or scrubber can also fail on its own error line. Enqueuing another error
114
+ # there would repeat forever on SemanticLogger's writer thread. Raise modes still raise.
115
+ if source == Source::Internal && Thread.current[:logstruct_formatting_internal_error] && mode != ErrorHandlingMode::Raise
116
+ return
117
+ end
118
+
98
119
  case mode
99
120
  when ErrorHandlingMode::Ignore
100
121
  # Do nothing
@@ -2,6 +2,7 @@
2
2
  # frozen_string_literal: true
3
3
 
4
4
  require "active_support/hash_with_indifferent_access"
5
+ require_relative "../bounded_input"
5
6
 
6
7
  module LogStruct
7
8
  # A key's words, lowercase and joined by underscores, so that keys match however they're written:
@@ -38,8 +39,8 @@ module LogStruct
38
39
  # A filter that needs more than the key: Rails' entries with a dot (`card.number`) match a key by its
39
40
  # path, and Rails gives filter blocks with three arguments the parameters around the key. The path
40
41
  # starts at the line's top-level field (`["params", "card", "number"]`); `root` is that field's value,
41
- # or the whole line for a top-level key, with indifferent access as Rails gives it. Building it copies
42
- # every value inside it, so a matcher that doesn't read it sets uses_root to false and gets nil.
42
+ # or the whole line for a top-level key, with indifferent access as Rails gives it. A matcher
43
+ # that doesn't read the root sets uses_root to false and gets nil.
43
44
  class PathFilterMatcher < T::Struct
44
45
  extend T::Sig
45
46
 
@@ -50,7 +51,10 @@ module LogStruct
50
51
  sig { params(path: T::Array[String], value: T.untyped, root: T.untyped).returns(T::Boolean) }
51
52
  def matches?(path, value, root)
52
53
  root = nil unless uses_root
53
- callable.call(path, value, root.is_a?(LazyRoot) ? root.value : root)
54
+ root = root.value if root.is_a?(LazyRoot)
55
+ return true if root.equal?(LazyRoot::UNAVAILABLE)
56
+
57
+ callable.call(path, value, root)
54
58
  end
55
59
  end
56
60
 
@@ -58,6 +62,8 @@ module LogStruct
58
62
  class LazyRoot
59
63
  extend T::Sig
60
64
 
65
+ UNAVAILABLE = T.let(Object.new.freeze, Object)
66
+
61
67
  sig { params(value: T.untyped).void }
62
68
  def initialize(value)
63
69
  @source = value
@@ -70,12 +76,20 @@ module LogStruct
70
76
  def value
71
77
  return @value if @built
72
78
 
73
- @built = true
74
- @value = begin
75
- @source.is_a?(Hash) ? ActiveSupport::HashWithIndifferentAccess.new(@source) : @source
76
- rescue SystemStackError, StandardError
77
- @source
79
+ state = BoundedInput::State.new
80
+ bounded = BoundedInput.copy(@source, state: state)
81
+ @value = if state.truncated
82
+ UNAVAILABLE
83
+ elsif bounded.is_a?(Hash)
84
+ ActiveSupport::HashWithIndifferentAccess.new(bounded)
85
+ else
86
+ bounded
78
87
  end
88
+ @built = true
89
+ @value
90
+ rescue SystemStackError, StandardError
91
+ @built = true
92
+ @value = UNAVAILABLE
79
93
  end
80
94
  end
81
95
 
@@ -86,7 +100,7 @@ module LogStruct
86
100
  # A filtered value is replaced by a summary of it, with no length (which would say how long a
87
101
  # secret is):
88
102
  #
89
- # { _filtered: { _class: "Hash", _keys_count: 3, _keys: [:key1, :key2, :key3] } } # first 10 keys
103
+ # { _filtered: { _class: "Hash", _keys_count: 3 } }
90
104
  # { _filtered: { _class: "Array", _count: 2 } }
91
105
  # { _filtered: { _class: "String" } } # plus its _hash for filter_keys_with_hashes
92
106
  #
@@ -21,7 +21,7 @@ module LogStruct
21
21
  # Default: 20
22
22
  prop :max_backtrace_lines, Integer, default: 20
23
23
 
24
- # Strings longer than this many bytes are cut, and say how long they were
24
+ # Strings longer than this many bytes are cut; oversized Hash keys and their values are omitted
25
25
  # Default: 32768 (32 KB)
26
26
  prop :max_string_bytes, Integer, default: 32_768
27
27
  end
@@ -120,7 +120,6 @@ module LogStruct
120
120
  Storage = new(:storage)
121
121
  Operation = new(:op)
122
122
  FileId = new(:file_id)
123
- Filename = new(:filename)
124
123
  MimeType = new(:mime_type)
125
124
  Size = new(:size)
126
125
  Metadata = new(:metadata)
@@ -134,8 +133,6 @@ module LogStruct
134
133
  Model = new(:model)
135
134
  MountPoint = new(:mount_point)
136
135
  Version = new(:version)
137
- StorePath = new(:store_path)
138
- Extension = new(:ext)
139
136
 
140
137
  # SQL-specific fields
141
138
  Sql = new(:sql)
@@ -10,6 +10,7 @@ require_relative "enums/source"
10
10
  require_relative "enums/event"
11
11
  require_relative "enums/log_field"
12
12
  require_relative "string_scrubber"
13
+ require_relative "bounded_input"
13
14
  require_relative "log"
14
15
  require_relative "param_filters"
15
16
  require_relative "multi_error_reporter"
@@ -80,13 +81,13 @@ module LogStruct
80
81
  return summarize_unprocessed_value(arg) if recursion_depth > MAX_RECURSION_DEPTH || ancestors.key?(arg)
81
82
 
82
83
  case arg
83
- when Hash, Array
84
+ when Hash, Array, ::ActionController::Parameters
84
85
  ancestors[arg] = true
85
86
  begin
86
- if arg.is_a?(Hash)
87
- process_hash(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
88
- else
87
+ if arg.is_a?(Array)
89
88
  process_array(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
89
+ else
90
+ process_hash(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
90
91
  end
91
92
  ensure
92
93
  ancestors.delete(arg)
@@ -107,7 +108,7 @@ module LogStruct
107
108
  scrub_string(String(object))
108
109
  end
109
110
  rescue => e
110
- LogStruct.handle_exception(e, source: Source::Internal)
111
+ LogStruct.handle_value_error(e, "GlobalID formatting failed")
111
112
  "[GLOBALID_ERROR]"
112
113
  end
113
114
  end
@@ -125,26 +126,24 @@ module LogStruct
125
126
  when Numeric, TrueClass, FalseClass, NilClass
126
127
  arg
127
128
  else
128
- # Structs, ActionController::Parameters and other objects: filter and scrub what
129
+ # Structs and other objects: filter and scrub what
129
130
  # to_json would write
130
131
  process_values(arg.as_json, recursion_depth: recursion_depth + 1, ancestors: ancestors, path: path, root: root)
131
132
  end
132
133
  rescue SystemStackError, StandardError => e
133
134
  # A SystemStackError comes from an object whose as_json refers back to itself
134
- e = RuntimeError.new("#{e.class}: #{e.message}") unless e.is_a?(StandardError)
135
- # Report error through LogStruct's framework
136
135
  context = {
137
136
  processor_method: "process_values",
138
137
  value_type: arg.class.name,
139
138
  recursion_depth: recursion_depth
140
139
  }
141
- LogStruct.handle_exception(e, source: Source::Internal, context: context)
140
+ LogStruct.handle_value_error(e, "value formatting failed", context: context)
142
141
  summarize_unprocessed_value(arg)
143
142
  end
144
143
 
145
144
  sig do
146
145
  params(
147
- hash: T::Hash[T.untyped, T.untyped],
146
+ hash: T.untyped,
148
147
  recursion_depth: Integer,
149
148
  ancestors: T::Hash[T.untyped, T::Boolean],
150
149
  path: T.nilable(T::Array[String]),
@@ -158,13 +157,30 @@ module LogStruct
158
157
  line_root = (path && recursion_depth == 0) ? ConfigStruct::LazyRoot.new(hash) : nil
159
158
 
160
159
  max_keys = LogStruct.config.limits.max_hash_keys
161
- hash.each_with_index do |(key, value), index|
160
+ max_key_bytes = LogStruct.config.limits.max_string_bytes
161
+ oversized_keys = 0
162
+ hash.each_pair.with_index do |(key, value), index|
162
163
  if index >= max_keys
163
- result[:_more_keys] = hash.size - max_keys
164
+ result[:_more_keys] = BoundedInput.key_count(hash) - max_keys
164
165
  break
165
166
  end
166
167
 
168
+ if BoundedInput::COUNT_KEYS.include?(key) && value.is_a?(Integer)
169
+ result[key] = value
170
+ next
171
+ end
172
+
173
+ if key.to_s.bytesize > max_key_bytes
174
+ oversized_keys += 1
175
+ next
176
+ end
177
+
178
+ # Checked again: each byte that isn't valid UTF-8 becomes a 3-byte U+FFFD
167
179
  key = valid_utf8_key(key)
180
+ if key.to_s.bytesize > max_key_bytes
181
+ oversized_keys += 1
182
+ next
183
+ end
168
184
  logstruct_field = recursion_depth == 0 && LOG_FIELD_KEYS.include?(key)
169
185
  key_path = path && [*path, key.to_s]
170
186
  # A top-level key's root is the line; the keys below it get the top-level field's value
@@ -173,7 +189,9 @@ module LogStruct
173
189
  # Check if this key should be filtered at any depth
174
190
  filtered = ParamFilters.should_filter_key?(key, value, logstruct_field: logstruct_field, path: key_path, root: key_root)
175
191
  # Keys are scrubbed too: an email address can be a key
176
- result[StringScrubber.scrub_name(key)] = if !filtered
192
+ scrubbed_key = StringScrubber.scrub_name(key)
193
+ scrubbed_key = scrub_limited_string(scrubbed_key) if scrubbed_key.is_a?(String) && scrubbed_key.bytesize > max_key_bytes
194
+ result[scrubbed_key] = if !filtered
177
195
  process_values(value, recursion_depth: recursion_depth + 1, ancestors: ancestors, path: key_path, root: children_root)
178
196
  elsif value == RAILS_FILTERED_MASK
179
197
  # Rails' parameter filter has already replaced the value
@@ -183,6 +201,8 @@ module LogStruct
183
201
  end
184
202
  end
185
203
 
204
+ result[:_oversized_keys] = oversized_keys if oversized_keys.positive?
205
+
186
206
  result
187
207
  end
188
208
 
@@ -194,18 +214,18 @@ module LogStruct
194
214
  StringScrubber.valid_utf8(string)
195
215
  end
196
216
 
197
- # A String or Symbol key as valid UTF-8 (symbolize_keys makes Symbols of binary String keys)
217
+ # Normalize only keys that will be logged, after applying the size limits.
198
218
  sig { params(key: T.untyped).returns(T.untyped) }
199
219
  def valid_utf8_key(key)
200
220
  case key
201
221
  when String
202
- valid_utf8(key)
222
+ valid_utf8(key).to_sym
203
223
  when Symbol
204
224
  name = key.name
205
225
  fixed = valid_utf8(name)
206
226
  fixed.equal?(name) ? key : fixed.to_sym
207
227
  else
208
- key
228
+ valid_utf8(key.to_s)
209
229
  end
210
230
  end
211
231
 
@@ -232,31 +252,6 @@ module LogStruct
232
252
  "#{kept}... (cut from #{string.bytesize} bytes)"
233
253
  end
234
254
 
235
- # Like deep_symbolize_keys, but stops at circular references and at the depth where
236
- # process_values stops, leaving those values for process_values to summarize
237
- sig { params(value: T.untyped, recursion_depth: Integer, ancestors: T::Hash[T.untyped, T::Boolean]).returns(T.untyped) }
238
- def symbolize_keys(value, recursion_depth: 0, ancestors: {}.compare_by_identity)
239
- return value if recursion_depth > MAX_RECURSION_DEPTH || ancestors.key?(value)
240
-
241
- case value
242
- when Hash
243
- ancestors[value] = true
244
- result = value.each_with_object({}) do |(key, nested_value), symbolized|
245
- symbol_key = key.respond_to?(:to_sym) ? key.to_sym : key
246
- symbolized[symbol_key] = symbolize_keys(nested_value, recursion_depth: recursion_depth + 1, ancestors: ancestors)
247
- end
248
- ancestors.delete(value)
249
- result
250
- when Array
251
- ancestors[value] = true
252
- result = value.map { |item| symbolize_keys(item, recursion_depth: recursion_depth + 1, ancestors: ancestors) }
253
- ancestors.delete(value)
254
- result
255
- else
256
- value
257
- end
258
- end
259
-
260
255
  # Describes a value without filtering, scrubbing or serializing it, so that values which
261
256
  # couldn't be processed are never logged raw. Hashes and arrays are not serialized because
262
257
  # they may contain circular references.
@@ -267,16 +262,16 @@ module LogStruct
267
262
  {_filtered: {_class: arg.class.name, _keys_count: arg.size}}
268
263
  when Array
269
264
  {_filtered: {_class: arg.class.name, _count: arg.size}}
270
- when String
265
+ when String, Symbol
271
266
  {_filtered: {_class: arg.class.name}}
272
- when Numeric, Symbol, TrueClass, FalseClass, NilClass
267
+ when Numeric, TrueClass, FalseClass, NilClass
273
268
  arg
274
269
  else
275
270
  {_filtered: {_class: arg.class.name}}
276
271
  end
277
272
  end
278
273
 
279
- sig { params(log_value: T.untyped, time: Time).returns(T::Hash[Symbol, T.untyped]) }
274
+ sig { params(log_value: T.untyped, time: Time).returns(T::Hash[T.untyped, T.untyped]) }
280
275
  def log_value_to_hash(log_value, time:)
281
276
  case log_value
282
277
  when Log::Interfaces::CommonFields
@@ -284,12 +279,10 @@ module LogStruct
284
279
  log_value.serialize
285
280
 
286
281
  when T::Struct
287
- # Default T::Struct.serialize methods returns a hash with string keys, so convert them to symbols
288
- symbolize_keys(log_value.serialize)
282
+ log_value.serialize
289
283
 
290
284
  when Hash
291
- # Use hash as is and convert string keys to symbols
292
- symbolize_keys(log_value)
285
+ log_value
293
286
 
294
287
  when Exception
295
288
  # logger.error(exception): its class, message and backtrace
@@ -322,15 +315,10 @@ module LogStruct
322
315
  log_value.to_s
323
316
  end
324
317
  rescue => e
325
- # Handle serialization errors
326
- context = {
327
- object_class: log_value.class.name,
328
- object_inspect: log_value.inspect.truncate(100)
329
- }
330
- LogStruct.handle_exception(e, source: Source::Internal, context: context)
331
-
332
- # Fall back to the string representation to ensure we continue processing
333
- log_value.to_s
318
+ LogStruct.handle_value_error(e, "value serialization failed", context: {object_class: log_value.class.name})
319
+
320
+ # An object's string representation may contain the same secret that broke serialization.
321
+ "[SERIALIZATION_ERROR]"
334
322
  end
335
323
  end
336
324
 
@@ -357,9 +345,15 @@ module LogStruct
357
345
  ).returns(String)
358
346
  end
359
347
  def call(severity, time, progname, log_value, request_id: nil, context: nil, tags: nil)
348
+ previous_internal_error = Thread.current[:logstruct_formatting_internal_error]
360
349
  level_enum = Level.from_severity(severity)
361
350
 
362
351
  data = log_value_to_hash(log_value, time: time)
352
+ source = data[:src] || data["src"]
353
+ event = data[:evt] || data["evt"]
354
+ if [Source::Internal, :logstruct, "logstruct"].include?(source) && [Event::Error, :error, "error"].include?(event)
355
+ Thread.current[:logstruct_formatting_internal_error] = true
356
+ end
363
357
 
364
358
  # Filter params, scrub sensitive values, format ActiveJob GlobalID arguments
365
359
  data = process_values(data)
@@ -376,6 +370,8 @@ module LogStruct
376
370
  rescue SystemStackError, StandardError => e
377
371
  unwritable_line_json(data, e)
378
372
  end
373
+ ensure
374
+ Thread.current[:logstruct_formatting_internal_error] = previous_internal_error
379
375
  end
380
376
 
381
377
  # A line that can't be written as JSON is replaced by one that says so, instead of being lost
@@ -420,7 +416,7 @@ module LogStruct
420
416
  if request_id.is_a?(String) && !request_id.empty? && !data.key?(:req_id)
421
417
  metadata[:req_id] = request_id
422
418
  end
423
- metadata[:ctx] = symbolize_keys(context) if context.present?
419
+ metadata[:ctx] = context if context.present?
424
420
  return if metadata.empty?
425
421
 
426
422
  processed = process_values(metadata)
@@ -2,13 +2,14 @@
2
2
  # frozen_string_literal: true
3
3
 
4
4
  require "openssl"
5
+ require "securerandom"
5
6
 
6
7
  module LogStruct
7
8
  # Utility module for hashing sensitive data
8
9
  module HashUtils
9
- # The key when there's neither a configured hash_salt nor a Rails app with a secret_key_base.
10
- # It's public, so these hashes can be matched to a list of candidate values.
11
- FALLBACK_KEY = "l0g5t0p"
10
+ # The key without a hash_salt, before the Rails app has its secret_key_base. A fixed key would be public,
11
+ # and anyone could match its hashes to a list of candidate emails; these hashes differ between processes.
12
+ FALLBACK_KEY = T.let(::SecureRandom.random_bytes(32).freeze, String)
12
13
 
13
14
  class << self
14
15
  extend T::Sig
@@ -17,7 +18,13 @@ module LogStruct
17
18
  sig { params(value: String).returns(String) }
18
19
  def hash_value(value)
19
20
  length = LogStruct.config.filters.hash_length
20
- OpenSSL::HMAC.hexdigest("SHA256", hash_key, value)[0...length] || "error"
21
+ max_bytes = LogStruct.config.limits.max_string_bytes
22
+ input = if value.bytesize > max_bytes
23
+ value.byteslice(0, max_bytes).to_s.b + "\0".b + value.bytesize.to_s.b
24
+ else
25
+ value
26
+ end
27
+ OpenSSL::HMAC.hexdigest("SHA256", hash_key, input)[0...length] || "error"
21
28
  end
22
29
 
23
30
  # The configured hash_salt, or a key derived from the app's secret_key_base