logstruct 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/README.md +1 -1
- data/lib/log_struct/bounded_input.rb +101 -0
- data/lib/log_struct/concerns/configuration.rb +10 -10
- data/lib/log_struct/concerns/error_handling.rb +21 -0
- data/lib/log_struct/config_struct/filters.rb +23 -9
- data/lib/log_struct/config_struct/limits.rb +1 -1
- data/lib/log_struct/enums/log_field.rb +0 -3
- data/lib/log_struct/formatter.rb +54 -58
- data/lib/log_struct/hash_utils.rb +11 -4
- data/lib/log_struct/integrations/ahoy.rb +3 -2
- data/lib/log_struct/integrations/carrierwave.rb +3 -6
- data/lib/log_struct/integrations/lograge.rb +4 -10
- data/lib/log_struct/integrations/shrine.rb +1 -1
- data/lib/log_struct/log/action_mailer/delivered.rb +10 -0
- data/lib/log_struct/log/action_mailer/delivery.rb +10 -0
- data/lib/log_struct/log/action_mailer/error.rb +10 -0
- data/lib/log_struct/log/carrierwave/delete.rb +0 -4
- data/lib/log_struct/log/carrierwave/download.rb +0 -6
- data/lib/log_struct/log/carrierwave/upload.rb +0 -6
- data/lib/log_struct/log/carrierwave.rb +2 -8
- data/lib/log_struct/log/error.rb +10 -0
- data/lib/log_struct/log/plain.rb +10 -0
- data/lib/log_struct/log/puma/error.rb +10 -0
- data/lib/log_struct/log/puma/shutdown.rb +10 -0
- data/lib/log_struct/log/puma/start.rb +10 -0
- data/lib/log_struct/log/request.rb +10 -0
- data/lib/log_struct/log/security/blocked_host.rb +10 -0
- data/lib/log_struct/log/security/csrf_violation.rb +10 -0
- data/lib/log_struct/log/security/ip_spoof.rb +10 -0
- data/lib/log_struct/log/sidekiq/error.rb +10 -0
- data/lib/log_struct/log/sidekiq/log.rb +10 -0
- data/lib/log_struct/log/sql.rb +10 -0
- data/lib/log_struct/monkey_patches/active_support/tagged_logging/formatter.rb +5 -3
- data/lib/log_struct/param_filters.rb +10 -28
- data/lib/log_struct/shared/merge_additional_data_fields.rb +21 -5
- data/lib/log_struct/string_scrubber.rb +7 -6
- data/lib/log_struct/version.rb +1 -1
- data/logstruct.gemspec +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 647c97ba4368b67d39333204ca35a5161b0905dda2c8ef72d01e15d58665d367
|
|
4
|
+
data.tar.gz: e86e688d516ef584c5f8994ecb0739599c09595a021ebc949f494e6d631f349d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6709f7868f43c348891761749400bb93c762e3df6f2d7093dab12f51c50a2ce33e26bfd94dab009093a1c4207f746735764c56ce20768394a9b8e81cce5d346d
|
|
7
|
+
data.tar.gz: 7ab05d71a64c08cd4ad747c7684d3b75553e2740c0539cefa2cab2f906f2c5023dd8696bb78129790472896f83008addde117f9e41229d563ad9003a76911fd9
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,48 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.3.0] - 2026-10-23
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- **Breaking**: CarrierWave lines log `file_id` as an HMAC of the file's identifier (keyed like email hashes, see `hash_salt`), and no longer log `filename`, `store_path` or `ext`. CarrierWave names stored files after the uploaded file by default, and an uploaded file's name often names a person. `LogField::Filename`, `LogField::StorePath` and `LogField::Extension` are removed.
|
|
15
|
+
- **Breaking**: A filtered Hash's summary has its `_keys_count` but not its first 10 keys (`_keys`): a Hash's keys can be secrets themselves, like tokens or email addresses used as keys.
|
|
16
|
+
- **Breaking**: `additional_data` can't set any field of its line's own type, whether or not the line has it (an error line's `msg`, `backtrace` or `error_message`), or `req_id`. Like `src`, `evt`, `lvl` and `ts`, such a key is left out and handled as a LogStruct error (raised in `local_environments`). It used to replace the field. Other types' field names, like `name` on a plain line, can still be used.
|
|
17
|
+
- **Filtering**: Without a `hash_salt`, values hashed before the Rails app has its `secret_key_base` use a random key for each process, instead of a fixed key that was public in LogStruct's source. Set `hash_salt` to keep these hashes the same across processes.
|
|
18
|
+
- **Development**: `task ci` checks the locked Ruby and docs dependencies for known advisories, with `bundler-audit` and `pnpm audit`.
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- **Formatting**: Hash keys longer than `max_string_bytes` are left out with their values, and counted in `_oversized_keys`. A request's parameter names could make a line any size.
|
|
23
|
+
- **Formatting**: `ActionController::Parameters` are written like a Hash, within the size limits, instead of being converted whole first. Lograge's `params` and Ahoy's `properties` are converted at their first level only, and only as many keys as the line keeps; `_more_keys` and `_oversized_keys` count the rest.
|
|
24
|
+
- **Filtering**: A Rails filter block with three arguments gets a copy of the parameters that stops at 10,000 values, 20 levels of nesting or a value that contains itself. Past those, every key the block would check is filtered. The parameters used to be copied whole.
|
|
25
|
+
- **Errors**: An error raised while formatting, serializing or filtering a value is reported with its class and backtrace but not its message, which can quote the value. Errors in Rails filter blocks and LogStruct's filter matchers aren't reported with the key, which can be the secret. An object whose `as_json` raises is logged as `[SERIALIZATION_ERROR]` instead of its `to_s`, and its `inspect` isn't reported.
|
|
26
|
+
- **Formatting**: A Hash key that isn't a String or Symbol is written as its string, scrubbed, so an email address in its `to_s` is scrubbed.
|
|
27
|
+
- **Filtering**: A value longer than `max_string_bytes` is hashed by its first `max_string_bytes` bytes and its length.
|
|
28
|
+
- **Development**: Update Next.js to 16.4.1 for the docs site, which fixes three advisories in features the static site doesn't use, and the test-only Dalli to 3.2.13.
|
|
29
|
+
|
|
30
|
+
## [0.2.2] - 2026-10-21
|
|
31
|
+
|
|
32
|
+
### Changed
|
|
33
|
+
|
|
34
|
+
- **Development**: Update Prettier, cspell and pnpm's setup action. The development tools require Node.js 22.18 or later in the 22.x series.
|
|
35
|
+
- **Docs**: Upgrade the site to Next.js 16 and align its Node type declarations with the Node.js 22 runtime used in CI.
|
|
36
|
+
|
|
37
|
+
### Fixed
|
|
38
|
+
|
|
39
|
+
- **Filtering**: Nested URL query parameters such as `password[]` and `token[value]` are filtered, including URL-encoded names and paths in Rails' `filter_parameters`.
|
|
40
|
+
- **Filtering**: Basic credentials are scrubbed even when their Base64 encoding is shorter than eight characters.
|
|
41
|
+
- **Filtering**: Symbols beyond the nesting limit are summarized instead of being written without scrubbing.
|
|
42
|
+
- **Filtering**: When a key or path matcher raises and error handling lets logging continue, its value is filtered. This also applies to Rails filter blocks.
|
|
43
|
+
- **Errors**: A failed filter or scrubber does not keep queuing or reporting errors when it fails again while formatting its own error line or report context.
|
|
44
|
+
- **Formatting**: Hash keys are normalized during filtering, so arrays, hashes and filtered values are no longer copied recursively before size limits are applied. Circular references are summarized at their first recurrence.
|
|
45
|
+
- **Docs**: Correct the supported security release series and contributor instructions for releases, generated log types and schema fields. Describe size limits as limits per value, not a guarantee on total formatting time. Ensure the site's integration tests run when a local Watchman installation is broken.
|
|
46
|
+
- **Development**: `task ci` verifies committed RBI files, matching GitHub CI, instead of downloading new annotations concurrently with typechecking.
|
|
47
|
+
- **Generation**: A failed formatter or Sorbet JSON export stops code generation instead of leaving incomplete artifacts with a successful exit status.
|
|
48
|
+
- **Development**: Update the locked `crass` dependency to 1.0.7, which fixes a denial-of-service issue in CSS number parsing.
|
|
49
|
+
|
|
8
50
|
## [0.2.1] - 2026-10-18
|
|
9
51
|
|
|
10
52
|
### Fixed
|
data/README.md
CHANGED
|
@@ -28,7 +28,7 @@ any other log platform. Sensitive values are filtered and scrubbed before a line
|
|
|
28
28
|
`Bearer` and `Basic` credentials and sensitive URL query params (`?token=`) are replaced too. IP and MAC address
|
|
29
29
|
scrubbing are opt-in.
|
|
30
30
|
- **Limits.** Huge values are cut down (100 keys per Hash, 10 items per Array, 20 lines per backtrace, 32 KB per
|
|
31
|
-
String by default),
|
|
31
|
+
String by default), reducing log size and the work spent formatting large values.
|
|
32
32
|
- **Error handling.** A mode for each kind of error decides whether it's raised, logged or reported, and whether
|
|
33
33
|
that differs between development and production. Reports go to Sentry, Bugsnag, Rollbar, Honeybadger or your own
|
|
34
34
|
reporter.
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# typed: strict
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require "action_controller"
|
|
5
|
+
|
|
6
|
+
module LogStruct
|
|
7
|
+
# Copies of logged Hashes and Arrays that stop before an input too large to copy in full
|
|
8
|
+
module BoundedInput
|
|
9
|
+
extend T::Sig
|
|
10
|
+
|
|
11
|
+
# A Rails filter block with three arguments gets every parameter, so a copy for it keeps what the formatter
|
|
12
|
+
# leaves out of the line. These limits only stop values no request sends: nesting deeper than the formatter
|
|
13
|
+
# writes, and more values than a form or JSON body has. A copy that reaches one is truncated, and the
|
|
14
|
+
# filters that read it fail closed.
|
|
15
|
+
MAX_DEPTH = 20
|
|
16
|
+
MAX_NODES = 10_000
|
|
17
|
+
|
|
18
|
+
# The keys that count what a Hash's copy left out. The formatter writes them as it writes its own counts:
|
|
19
|
+
# Rails' default filter_parameters include :_key, which would filter them.
|
|
20
|
+
COUNT_KEYS = T.let(Set[:_more_keys, :_oversized_keys].freeze, T::Set[Symbol])
|
|
21
|
+
|
|
22
|
+
class State < T::Struct
|
|
23
|
+
prop :remaining_nodes, Integer, default: MAX_NODES
|
|
24
|
+
prop :truncated, T::Boolean, default: false
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
sig do
|
|
28
|
+
params(value: T.untyped, state: State, depth: Integer, ancestors: T::Hash[T.untyped, T::Boolean]).returns(T.untyped)
|
|
29
|
+
end
|
|
30
|
+
def self.copy(value, state:, depth: 0, ancestors: {}.compare_by_identity)
|
|
31
|
+
if depth > MAX_DEPTH || ancestors.key?(value) || state.remaining_nodes <= 0
|
|
32
|
+
state.truncated = true
|
|
33
|
+
return nil
|
|
34
|
+
end
|
|
35
|
+
state.remaining_nodes -= 1
|
|
36
|
+
|
|
37
|
+
case value
|
|
38
|
+
when Hash, ::ActionController::Parameters
|
|
39
|
+
ancestors[value] = true
|
|
40
|
+
begin
|
|
41
|
+
result = {}
|
|
42
|
+
value.each_pair do |key, child|
|
|
43
|
+
break if state.truncated
|
|
44
|
+
|
|
45
|
+
result[key] = copy(child, state: state, depth: depth + 1, ancestors: ancestors)
|
|
46
|
+
end
|
|
47
|
+
result
|
|
48
|
+
ensure
|
|
49
|
+
ancestors.delete(value)
|
|
50
|
+
end
|
|
51
|
+
when Array
|
|
52
|
+
ancestors[value] = true
|
|
53
|
+
begin
|
|
54
|
+
result = []
|
|
55
|
+
value.each do |child|
|
|
56
|
+
break if state.truncated
|
|
57
|
+
|
|
58
|
+
result << copy(child, state: state, depth: depth + 1, ancestors: ancestors)
|
|
59
|
+
end
|
|
60
|
+
result
|
|
61
|
+
ensure
|
|
62
|
+
ancestors.delete(value)
|
|
63
|
+
end
|
|
64
|
+
else
|
|
65
|
+
value
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# How many keys a Hash or ActionController::Parameters has, without converting its values
|
|
70
|
+
sig { params(hash: T.untyped).returns(Integer) }
|
|
71
|
+
def self.key_count(hash)
|
|
72
|
+
hash.is_a?(Hash) ? hash.size : hash.keys.size
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
# The first keys of a Hash or ActionController::Parameters as Symbols, for a log struct's Hash field, without
|
|
76
|
+
# converting the keys the formatter would leave out. The formatter writes max_hash_keys entries of a Hash,
|
|
77
|
+
# and the counts of the keys left out here are entries too: the last key's slot goes to _more_keys, and an
|
|
78
|
+
# omitted oversized key's slot to _oversized_keys, so the formatter writes both counts as they are.
|
|
79
|
+
sig { params(hash: T.untyped).returns(T::Hash[Symbol, T.untyped]) }
|
|
80
|
+
def self.symbolize_keys(hash)
|
|
81
|
+
limits = LogStruct.config.limits
|
|
82
|
+
total = key_count(hash)
|
|
83
|
+
examined = (total > limits.max_hash_keys) ? [limits.max_hash_keys - 1, 0].max : total
|
|
84
|
+
result = T.let({}, T::Hash[Symbol, T.untyped])
|
|
85
|
+
oversized_keys = 0
|
|
86
|
+
hash.each_pair.with_index do |(key, value), index|
|
|
87
|
+
break if index >= examined
|
|
88
|
+
|
|
89
|
+
if key.to_s.bytesize > limits.max_string_bytes
|
|
90
|
+
oversized_keys += 1
|
|
91
|
+
next
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
result[key.to_s.to_sym] = value
|
|
95
|
+
end
|
|
96
|
+
result[:_more_keys] = total - examined if examined < total
|
|
97
|
+
result[:_oversized_keys] = oversized_keys if oversized_keys.positive?
|
|
98
|
+
result
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
end
|
|
@@ -259,8 +259,8 @@ module LogStruct
|
|
|
259
259
|
result = filter.call(key, value, root)
|
|
260
260
|
!!result || (value.is_a?(String) && value != original_value)
|
|
261
261
|
rescue => e
|
|
262
|
-
handle_filter_error(e, filter
|
|
263
|
-
|
|
262
|
+
handle_filter_error(e, filter)
|
|
263
|
+
true
|
|
264
264
|
end
|
|
265
265
|
ConfigStruct::PathFilterMatcher.new(callable: callable, label: filter.inspect)
|
|
266
266
|
end
|
|
@@ -290,12 +290,12 @@ module LogStruct
|
|
|
290
290
|
begin
|
|
291
291
|
!!filter.call(key)
|
|
292
292
|
rescue => e
|
|
293
|
-
handle_filter_error(e, filter
|
|
294
|
-
|
|
293
|
+
handle_filter_error(e, filter)
|
|
294
|
+
true
|
|
295
295
|
end
|
|
296
296
|
rescue => e
|
|
297
|
-
handle_filter_error(e, filter
|
|
298
|
-
|
|
297
|
+
handle_filter_error(e, filter)
|
|
298
|
+
true
|
|
299
299
|
end
|
|
300
300
|
ConfigStruct::FilterMatcher.new(callable: callable, label: filter.inspect)
|
|
301
301
|
end
|
|
@@ -305,11 +305,11 @@ module LogStruct
|
|
|
305
305
|
filter.respond_to?(:arity) ? filter.arity : 2
|
|
306
306
|
end
|
|
307
307
|
|
|
308
|
-
|
|
309
|
-
|
|
308
|
+
# The key isn't reported: a key can be the secret (an email address or a token used as a key)
|
|
309
|
+
sig { params(error: StandardError, filter: T.untyped).void }
|
|
310
|
+
def handle_filter_error(error, filter)
|
|
310
311
|
context = {
|
|
311
312
|
filter: filter.class.name,
|
|
312
|
-
key: key,
|
|
313
313
|
filter_label: begin
|
|
314
314
|
filter.inspect
|
|
315
315
|
rescue
|
|
@@ -317,7 +317,7 @@ module LogStruct
|
|
|
317
317
|
end
|
|
318
318
|
}
|
|
319
319
|
|
|
320
|
-
LogStruct.
|
|
320
|
+
LogStruct.handle_value_error(error, "filter_parameters entry failed", context: context)
|
|
321
321
|
end
|
|
322
322
|
end
|
|
323
323
|
end
|
|
@@ -78,10 +78,25 @@ module LogStruct
|
|
|
78
78
|
# request's parameters or a mail's recipients
|
|
79
79
|
sig { params(context: T.nilable(T::Hash[Symbol, T.untyped])).returns(T.nilable(T::Hash[Symbol, T.untyped])) }
|
|
80
80
|
def filtered_context(context)
|
|
81
|
+
previous_internal_error = Thread.current[:logstruct_formatting_internal_error]
|
|
82
|
+
# The same filter may fail on the error's context. Keep that failure from reporting itself.
|
|
83
|
+
Thread.current[:logstruct_formatting_internal_error] = true
|
|
81
84
|
return context if context.nil? || context.empty?
|
|
82
85
|
|
|
83
86
|
filtered = Formatter.new.process_values(context)
|
|
84
87
|
filtered.is_a?(Hash) ? filtered : {}
|
|
88
|
+
ensure
|
|
89
|
+
Thread.current[:logstruct_formatting_internal_error] = previous_internal_error
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
# Handle an error raised while LogStruct read a value it was logging, without the error's message: the
|
|
93
|
+
# message can quote that value, as a NoMethodError quotes its receiver's inspect before Ruby 3.4.
|
|
94
|
+
# failure says what failed, and the reported error keeps the original's class and backtrace.
|
|
95
|
+
sig { params(error: Exception, failure: String, context: T.nilable(T::Hash[Symbol, T.untyped])).void }
|
|
96
|
+
def handle_value_error(error, failure, context: nil)
|
|
97
|
+
reported = RuntimeError.new("#{error.class}: #{failure}")
|
|
98
|
+
reported.set_backtrace(error.backtrace)
|
|
99
|
+
handle_exception(reported, source: Source::Internal, context: context)
|
|
85
100
|
end
|
|
86
101
|
|
|
87
102
|
# Handle an error according to the configured error handling mode (log, report, raise, etc).
|
|
@@ -95,6 +110,12 @@ module LogStruct
|
|
|
95
110
|
raise(error) if !LogStruct.is_production?
|
|
96
111
|
end
|
|
97
112
|
|
|
113
|
+
# A broken filter or scrubber can also fail on its own error line. Enqueuing another error
|
|
114
|
+
# there would repeat forever on SemanticLogger's writer thread. Raise modes still raise.
|
|
115
|
+
if source == Source::Internal && Thread.current[:logstruct_formatting_internal_error] && mode != ErrorHandlingMode::Raise
|
|
116
|
+
return
|
|
117
|
+
end
|
|
118
|
+
|
|
98
119
|
case mode
|
|
99
120
|
when ErrorHandlingMode::Ignore
|
|
100
121
|
# Do nothing
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
# frozen_string_literal: true
|
|
3
3
|
|
|
4
4
|
require "active_support/hash_with_indifferent_access"
|
|
5
|
+
require_relative "../bounded_input"
|
|
5
6
|
|
|
6
7
|
module LogStruct
|
|
7
8
|
# A key's words, lowercase and joined by underscores, so that keys match however they're written:
|
|
@@ -38,8 +39,8 @@ module LogStruct
|
|
|
38
39
|
# A filter that needs more than the key: Rails' entries with a dot (`card.number`) match a key by its
|
|
39
40
|
# path, and Rails gives filter blocks with three arguments the parameters around the key. The path
|
|
40
41
|
# starts at the line's top-level field (`["params", "card", "number"]`); `root` is that field's value,
|
|
41
|
-
# or the whole line for a top-level key, with indifferent access as Rails gives it.
|
|
42
|
-
#
|
|
42
|
+
# or the whole line for a top-level key, with indifferent access as Rails gives it. A matcher
|
|
43
|
+
# that doesn't read the root sets uses_root to false and gets nil.
|
|
43
44
|
class PathFilterMatcher < T::Struct
|
|
44
45
|
extend T::Sig
|
|
45
46
|
|
|
@@ -50,7 +51,10 @@ module LogStruct
|
|
|
50
51
|
sig { params(path: T::Array[String], value: T.untyped, root: T.untyped).returns(T::Boolean) }
|
|
51
52
|
def matches?(path, value, root)
|
|
52
53
|
root = nil unless uses_root
|
|
53
|
-
|
|
54
|
+
root = root.value if root.is_a?(LazyRoot)
|
|
55
|
+
return true if root.equal?(LazyRoot::UNAVAILABLE)
|
|
56
|
+
|
|
57
|
+
callable.call(path, value, root)
|
|
54
58
|
end
|
|
55
59
|
end
|
|
56
60
|
|
|
@@ -58,6 +62,8 @@ module LogStruct
|
|
|
58
62
|
class LazyRoot
|
|
59
63
|
extend T::Sig
|
|
60
64
|
|
|
65
|
+
UNAVAILABLE = T.let(Object.new.freeze, Object)
|
|
66
|
+
|
|
61
67
|
sig { params(value: T.untyped).void }
|
|
62
68
|
def initialize(value)
|
|
63
69
|
@source = value
|
|
@@ -70,12 +76,20 @@ module LogStruct
|
|
|
70
76
|
def value
|
|
71
77
|
return @value if @built
|
|
72
78
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
79
|
+
state = BoundedInput::State.new
|
|
80
|
+
bounded = BoundedInput.copy(@source, state: state)
|
|
81
|
+
@value = if state.truncated
|
|
82
|
+
UNAVAILABLE
|
|
83
|
+
elsif bounded.is_a?(Hash)
|
|
84
|
+
ActiveSupport::HashWithIndifferentAccess.new(bounded)
|
|
85
|
+
else
|
|
86
|
+
bounded
|
|
78
87
|
end
|
|
88
|
+
@built = true
|
|
89
|
+
@value
|
|
90
|
+
rescue SystemStackError, StandardError
|
|
91
|
+
@built = true
|
|
92
|
+
@value = UNAVAILABLE
|
|
79
93
|
end
|
|
80
94
|
end
|
|
81
95
|
|
|
@@ -86,7 +100,7 @@ module LogStruct
|
|
|
86
100
|
# A filtered value is replaced by a summary of it, with no length (which would say how long a
|
|
87
101
|
# secret is):
|
|
88
102
|
#
|
|
89
|
-
# { _filtered: { _class: "Hash", _keys_count: 3
|
|
103
|
+
# { _filtered: { _class: "Hash", _keys_count: 3 } }
|
|
90
104
|
# { _filtered: { _class: "Array", _count: 2 } }
|
|
91
105
|
# { _filtered: { _class: "String" } } # plus its _hash for filter_keys_with_hashes
|
|
92
106
|
#
|
|
@@ -21,7 +21,7 @@ module LogStruct
|
|
|
21
21
|
# Default: 20
|
|
22
22
|
prop :max_backtrace_lines, Integer, default: 20
|
|
23
23
|
|
|
24
|
-
# Strings longer than this many bytes are cut
|
|
24
|
+
# Strings longer than this many bytes are cut; oversized Hash keys and their values are omitted
|
|
25
25
|
# Default: 32768 (32 KB)
|
|
26
26
|
prop :max_string_bytes, Integer, default: 32_768
|
|
27
27
|
end
|
|
@@ -120,7 +120,6 @@ module LogStruct
|
|
|
120
120
|
Storage = new(:storage)
|
|
121
121
|
Operation = new(:op)
|
|
122
122
|
FileId = new(:file_id)
|
|
123
|
-
Filename = new(:filename)
|
|
124
123
|
MimeType = new(:mime_type)
|
|
125
124
|
Size = new(:size)
|
|
126
125
|
Metadata = new(:metadata)
|
|
@@ -134,8 +133,6 @@ module LogStruct
|
|
|
134
133
|
Model = new(:model)
|
|
135
134
|
MountPoint = new(:mount_point)
|
|
136
135
|
Version = new(:version)
|
|
137
|
-
StorePath = new(:store_path)
|
|
138
|
-
Extension = new(:ext)
|
|
139
136
|
|
|
140
137
|
# SQL-specific fields
|
|
141
138
|
Sql = new(:sql)
|
data/lib/log_struct/formatter.rb
CHANGED
|
@@ -10,6 +10,7 @@ require_relative "enums/source"
|
|
|
10
10
|
require_relative "enums/event"
|
|
11
11
|
require_relative "enums/log_field"
|
|
12
12
|
require_relative "string_scrubber"
|
|
13
|
+
require_relative "bounded_input"
|
|
13
14
|
require_relative "log"
|
|
14
15
|
require_relative "param_filters"
|
|
15
16
|
require_relative "multi_error_reporter"
|
|
@@ -80,13 +81,13 @@ module LogStruct
|
|
|
80
81
|
return summarize_unprocessed_value(arg) if recursion_depth > MAX_RECURSION_DEPTH || ancestors.key?(arg)
|
|
81
82
|
|
|
82
83
|
case arg
|
|
83
|
-
when Hash, Array
|
|
84
|
+
when Hash, Array, ::ActionController::Parameters
|
|
84
85
|
ancestors[arg] = true
|
|
85
86
|
begin
|
|
86
|
-
if arg.is_a?(
|
|
87
|
-
process_hash(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
|
|
88
|
-
else
|
|
87
|
+
if arg.is_a?(Array)
|
|
89
88
|
process_array(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
|
|
89
|
+
else
|
|
90
|
+
process_hash(arg, recursion_depth: recursion_depth, ancestors: ancestors, path: path, root: root)
|
|
90
91
|
end
|
|
91
92
|
ensure
|
|
92
93
|
ancestors.delete(arg)
|
|
@@ -107,7 +108,7 @@ module LogStruct
|
|
|
107
108
|
scrub_string(String(object))
|
|
108
109
|
end
|
|
109
110
|
rescue => e
|
|
110
|
-
LogStruct.
|
|
111
|
+
LogStruct.handle_value_error(e, "GlobalID formatting failed")
|
|
111
112
|
"[GLOBALID_ERROR]"
|
|
112
113
|
end
|
|
113
114
|
end
|
|
@@ -125,26 +126,24 @@ module LogStruct
|
|
|
125
126
|
when Numeric, TrueClass, FalseClass, NilClass
|
|
126
127
|
arg
|
|
127
128
|
else
|
|
128
|
-
# Structs
|
|
129
|
+
# Structs and other objects: filter and scrub what
|
|
129
130
|
# to_json would write
|
|
130
131
|
process_values(arg.as_json, recursion_depth: recursion_depth + 1, ancestors: ancestors, path: path, root: root)
|
|
131
132
|
end
|
|
132
133
|
rescue SystemStackError, StandardError => e
|
|
133
134
|
# A SystemStackError comes from an object whose as_json refers back to itself
|
|
134
|
-
e = RuntimeError.new("#{e.class}: #{e.message}") unless e.is_a?(StandardError)
|
|
135
|
-
# Report error through LogStruct's framework
|
|
136
135
|
context = {
|
|
137
136
|
processor_method: "process_values",
|
|
138
137
|
value_type: arg.class.name,
|
|
139
138
|
recursion_depth: recursion_depth
|
|
140
139
|
}
|
|
141
|
-
LogStruct.
|
|
140
|
+
LogStruct.handle_value_error(e, "value formatting failed", context: context)
|
|
142
141
|
summarize_unprocessed_value(arg)
|
|
143
142
|
end
|
|
144
143
|
|
|
145
144
|
sig do
|
|
146
145
|
params(
|
|
147
|
-
hash: T
|
|
146
|
+
hash: T.untyped,
|
|
148
147
|
recursion_depth: Integer,
|
|
149
148
|
ancestors: T::Hash[T.untyped, T::Boolean],
|
|
150
149
|
path: T.nilable(T::Array[String]),
|
|
@@ -158,13 +157,30 @@ module LogStruct
|
|
|
158
157
|
line_root = (path && recursion_depth == 0) ? ConfigStruct::LazyRoot.new(hash) : nil
|
|
159
158
|
|
|
160
159
|
max_keys = LogStruct.config.limits.max_hash_keys
|
|
161
|
-
|
|
160
|
+
max_key_bytes = LogStruct.config.limits.max_string_bytes
|
|
161
|
+
oversized_keys = 0
|
|
162
|
+
hash.each_pair.with_index do |(key, value), index|
|
|
162
163
|
if index >= max_keys
|
|
163
|
-
result[:_more_keys] = hash
|
|
164
|
+
result[:_more_keys] = BoundedInput.key_count(hash) - max_keys
|
|
164
165
|
break
|
|
165
166
|
end
|
|
166
167
|
|
|
168
|
+
if BoundedInput::COUNT_KEYS.include?(key) && value.is_a?(Integer)
|
|
169
|
+
result[key] = value
|
|
170
|
+
next
|
|
171
|
+
end
|
|
172
|
+
|
|
173
|
+
if key.to_s.bytesize > max_key_bytes
|
|
174
|
+
oversized_keys += 1
|
|
175
|
+
next
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
# Checked again: each byte that isn't valid UTF-8 becomes a 3-byte U+FFFD
|
|
167
179
|
key = valid_utf8_key(key)
|
|
180
|
+
if key.to_s.bytesize > max_key_bytes
|
|
181
|
+
oversized_keys += 1
|
|
182
|
+
next
|
|
183
|
+
end
|
|
168
184
|
logstruct_field = recursion_depth == 0 && LOG_FIELD_KEYS.include?(key)
|
|
169
185
|
key_path = path && [*path, key.to_s]
|
|
170
186
|
# A top-level key's root is the line; the keys below it get the top-level field's value
|
|
@@ -173,7 +189,9 @@ module LogStruct
|
|
|
173
189
|
# Check if this key should be filtered at any depth
|
|
174
190
|
filtered = ParamFilters.should_filter_key?(key, value, logstruct_field: logstruct_field, path: key_path, root: key_root)
|
|
175
191
|
# Keys are scrubbed too: an email address can be a key
|
|
176
|
-
|
|
192
|
+
scrubbed_key = StringScrubber.scrub_name(key)
|
|
193
|
+
scrubbed_key = scrub_limited_string(scrubbed_key) if scrubbed_key.is_a?(String) && scrubbed_key.bytesize > max_key_bytes
|
|
194
|
+
result[scrubbed_key] = if !filtered
|
|
177
195
|
process_values(value, recursion_depth: recursion_depth + 1, ancestors: ancestors, path: key_path, root: children_root)
|
|
178
196
|
elsif value == RAILS_FILTERED_MASK
|
|
179
197
|
# Rails' parameter filter has already replaced the value
|
|
@@ -183,6 +201,8 @@ module LogStruct
|
|
|
183
201
|
end
|
|
184
202
|
end
|
|
185
203
|
|
|
204
|
+
result[:_oversized_keys] = oversized_keys if oversized_keys.positive?
|
|
205
|
+
|
|
186
206
|
result
|
|
187
207
|
end
|
|
188
208
|
|
|
@@ -194,18 +214,18 @@ module LogStruct
|
|
|
194
214
|
StringScrubber.valid_utf8(string)
|
|
195
215
|
end
|
|
196
216
|
|
|
197
|
-
#
|
|
217
|
+
# Normalize only keys that will be logged, after applying the size limits.
|
|
198
218
|
sig { params(key: T.untyped).returns(T.untyped) }
|
|
199
219
|
def valid_utf8_key(key)
|
|
200
220
|
case key
|
|
201
221
|
when String
|
|
202
|
-
valid_utf8(key)
|
|
222
|
+
valid_utf8(key).to_sym
|
|
203
223
|
when Symbol
|
|
204
224
|
name = key.name
|
|
205
225
|
fixed = valid_utf8(name)
|
|
206
226
|
fixed.equal?(name) ? key : fixed.to_sym
|
|
207
227
|
else
|
|
208
|
-
key
|
|
228
|
+
valid_utf8(key.to_s)
|
|
209
229
|
end
|
|
210
230
|
end
|
|
211
231
|
|
|
@@ -232,31 +252,6 @@ module LogStruct
|
|
|
232
252
|
"#{kept}... (cut from #{string.bytesize} bytes)"
|
|
233
253
|
end
|
|
234
254
|
|
|
235
|
-
# Like deep_symbolize_keys, but stops at circular references and at the depth where
|
|
236
|
-
# process_values stops, leaving those values for process_values to summarize
|
|
237
|
-
sig { params(value: T.untyped, recursion_depth: Integer, ancestors: T::Hash[T.untyped, T::Boolean]).returns(T.untyped) }
|
|
238
|
-
def symbolize_keys(value, recursion_depth: 0, ancestors: {}.compare_by_identity)
|
|
239
|
-
return value if recursion_depth > MAX_RECURSION_DEPTH || ancestors.key?(value)
|
|
240
|
-
|
|
241
|
-
case value
|
|
242
|
-
when Hash
|
|
243
|
-
ancestors[value] = true
|
|
244
|
-
result = value.each_with_object({}) do |(key, nested_value), symbolized|
|
|
245
|
-
symbol_key = key.respond_to?(:to_sym) ? key.to_sym : key
|
|
246
|
-
symbolized[symbol_key] = symbolize_keys(nested_value, recursion_depth: recursion_depth + 1, ancestors: ancestors)
|
|
247
|
-
end
|
|
248
|
-
ancestors.delete(value)
|
|
249
|
-
result
|
|
250
|
-
when Array
|
|
251
|
-
ancestors[value] = true
|
|
252
|
-
result = value.map { |item| symbolize_keys(item, recursion_depth: recursion_depth + 1, ancestors: ancestors) }
|
|
253
|
-
ancestors.delete(value)
|
|
254
|
-
result
|
|
255
|
-
else
|
|
256
|
-
value
|
|
257
|
-
end
|
|
258
|
-
end
|
|
259
|
-
|
|
260
255
|
# Describes a value without filtering, scrubbing or serializing it, so that values which
|
|
261
256
|
# couldn't be processed are never logged raw. Hashes and arrays are not serialized because
|
|
262
257
|
# they may contain circular references.
|
|
@@ -267,16 +262,16 @@ module LogStruct
|
|
|
267
262
|
{_filtered: {_class: arg.class.name, _keys_count: arg.size}}
|
|
268
263
|
when Array
|
|
269
264
|
{_filtered: {_class: arg.class.name, _count: arg.size}}
|
|
270
|
-
when String
|
|
265
|
+
when String, Symbol
|
|
271
266
|
{_filtered: {_class: arg.class.name}}
|
|
272
|
-
when Numeric,
|
|
267
|
+
when Numeric, TrueClass, FalseClass, NilClass
|
|
273
268
|
arg
|
|
274
269
|
else
|
|
275
270
|
{_filtered: {_class: arg.class.name}}
|
|
276
271
|
end
|
|
277
272
|
end
|
|
278
273
|
|
|
279
|
-
sig { params(log_value: T.untyped, time: Time).returns(T::Hash[
|
|
274
|
+
sig { params(log_value: T.untyped, time: Time).returns(T::Hash[T.untyped, T.untyped]) }
|
|
280
275
|
def log_value_to_hash(log_value, time:)
|
|
281
276
|
case log_value
|
|
282
277
|
when Log::Interfaces::CommonFields
|
|
@@ -284,12 +279,10 @@ module LogStruct
|
|
|
284
279
|
log_value.serialize
|
|
285
280
|
|
|
286
281
|
when T::Struct
|
|
287
|
-
|
|
288
|
-
symbolize_keys(log_value.serialize)
|
|
282
|
+
log_value.serialize
|
|
289
283
|
|
|
290
284
|
when Hash
|
|
291
|
-
|
|
292
|
-
symbolize_keys(log_value)
|
|
285
|
+
log_value
|
|
293
286
|
|
|
294
287
|
when Exception
|
|
295
288
|
# logger.error(exception): its class, message and backtrace
|
|
@@ -322,15 +315,10 @@ module LogStruct
|
|
|
322
315
|
log_value.to_s
|
|
323
316
|
end
|
|
324
317
|
rescue => e
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
}
|
|
330
|
-
LogStruct.handle_exception(e, source: Source::Internal, context: context)
|
|
331
|
-
|
|
332
|
-
# Fall back to the string representation to ensure we continue processing
|
|
333
|
-
log_value.to_s
|
|
318
|
+
LogStruct.handle_value_error(e, "value serialization failed", context: {object_class: log_value.class.name})
|
|
319
|
+
|
|
320
|
+
# An object's string representation may contain the same secret that broke serialization.
|
|
321
|
+
"[SERIALIZATION_ERROR]"
|
|
334
322
|
end
|
|
335
323
|
end
|
|
336
324
|
|
|
@@ -357,9 +345,15 @@ module LogStruct
|
|
|
357
345
|
).returns(String)
|
|
358
346
|
end
|
|
359
347
|
def call(severity, time, progname, log_value, request_id: nil, context: nil, tags: nil)
|
|
348
|
+
previous_internal_error = Thread.current[:logstruct_formatting_internal_error]
|
|
360
349
|
level_enum = Level.from_severity(severity)
|
|
361
350
|
|
|
362
351
|
data = log_value_to_hash(log_value, time: time)
|
|
352
|
+
source = data[:src] || data["src"]
|
|
353
|
+
event = data[:evt] || data["evt"]
|
|
354
|
+
if [Source::Internal, :logstruct, "logstruct"].include?(source) && [Event::Error, :error, "error"].include?(event)
|
|
355
|
+
Thread.current[:logstruct_formatting_internal_error] = true
|
|
356
|
+
end
|
|
363
357
|
|
|
364
358
|
# Filter params, scrub sensitive values, format ActiveJob GlobalID arguments
|
|
365
359
|
data = process_values(data)
|
|
@@ -376,6 +370,8 @@ module LogStruct
|
|
|
376
370
|
rescue SystemStackError, StandardError => e
|
|
377
371
|
unwritable_line_json(data, e)
|
|
378
372
|
end
|
|
373
|
+
ensure
|
|
374
|
+
Thread.current[:logstruct_formatting_internal_error] = previous_internal_error
|
|
379
375
|
end
|
|
380
376
|
|
|
381
377
|
# A line that can't be written as JSON is replaced by one that says so, instead of being lost
|
|
@@ -420,7 +416,7 @@ module LogStruct
|
|
|
420
416
|
if request_id.is_a?(String) && !request_id.empty? && !data.key?(:req_id)
|
|
421
417
|
metadata[:req_id] = request_id
|
|
422
418
|
end
|
|
423
|
-
metadata[:ctx] =
|
|
419
|
+
metadata[:ctx] = context if context.present?
|
|
424
420
|
return if metadata.empty?
|
|
425
421
|
|
|
426
422
|
processed = process_values(metadata)
|
|
@@ -2,13 +2,14 @@
|
|
|
2
2
|
# frozen_string_literal: true
|
|
3
3
|
|
|
4
4
|
require "openssl"
|
|
5
|
+
require "securerandom"
|
|
5
6
|
|
|
6
7
|
module LogStruct
|
|
7
8
|
# Utility module for hashing sensitive data
|
|
8
9
|
module HashUtils
|
|
9
|
-
# The key
|
|
10
|
-
#
|
|
11
|
-
FALLBACK_KEY =
|
|
10
|
+
# The key without a hash_salt, before the Rails app has its secret_key_base. A fixed key would be public,
|
|
11
|
+
# and anyone could match its hashes to a list of candidate emails; these hashes differ between processes.
|
|
12
|
+
FALLBACK_KEY = T.let(::SecureRandom.random_bytes(32).freeze, String)
|
|
12
13
|
|
|
13
14
|
class << self
|
|
14
15
|
extend T::Sig
|
|
@@ -17,7 +18,13 @@ module LogStruct
|
|
|
17
18
|
sig { params(value: String).returns(String) }
|
|
18
19
|
def hash_value(value)
|
|
19
20
|
length = LogStruct.config.filters.hash_length
|
|
20
|
-
|
|
21
|
+
max_bytes = LogStruct.config.limits.max_string_bytes
|
|
22
|
+
input = if value.bytesize > max_bytes
|
|
23
|
+
value.byteslice(0, max_bytes).to_s.b + "\0".b + value.bytesize.to_s.b
|
|
24
|
+
else
|
|
25
|
+
value
|
|
26
|
+
end
|
|
27
|
+
OpenSSL::HMAC.hexdigest("SHA256", hash_key, input)[0...length] || "error"
|
|
21
28
|
end
|
|
22
29
|
|
|
23
30
|
# The configured hash_salt, or a key derived from the app's secret_key_base
|