activesupport-json_logging 1.3.0 → 1.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +26 -2
- data/README.md +34 -20
- data/lib/json_logging/event_subscriber.rb +6 -2
- data/lib/json_logging/execution_state.rb +65 -0
- data/lib/json_logging/helpers.rb +17 -1
- data/lib/json_logging/json_logger.rb +1 -1
- data/lib/json_logging/json_logger_extension.rb +11 -18
- data/lib/json_logging/line_encoder.rb +27 -24
- data/lib/json_logging/message_parser.rb +4 -0
- data/lib/json_logging/payload_builder.rb +29 -15
- data/lib/json_logging/sanitizer.rb +52 -37
- data/lib/json_logging/sanitizer_hash_limit.rb +42 -0
- data/lib/json_logging/structured_hash_json_encoder.rb +7 -38
- data/lib/json_logging/structured_hash_sanitizer.rb +5 -12
- data/lib/json_logging/version.rb +1 -1
- data/lib/json_logging.rb +20 -5
- data/sig/json_logging.rbs +4 -0
- metadata +18 -30
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 34ff048fcdd516e4f0f8e2b5d6d9911e44a6a486ee166ee3287b5be217140385
|
|
4
|
+
data.tar.gz: af1f7dac1075da66a5e2fa4135d143cc58521f3cc848f9d40aeb54254f8cf907
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f3811c1fbf10caf6931e5eca77c645360def106697f651d7d6ba313415a5630f07517f0be570689bf25d0f8704c586614f23d71f75488211158b4d1783f47091
|
|
7
|
+
data.tar.gz: 5982806cd91b314b756e7d25dd9c2e6af92e58cfb41bef9413eec0621f18b76faa6c78d739329794a4bfdfaee0333f06e255b542233c9936fc686a30d634ca67
|
data/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,32 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
|
+
## 1.3.2 (2026-11-07)
|
|
4
|
+
|
|
5
|
+
- Prefer `JSON.fast_generate` on generic `to_json_line`, structured-hash encoder assembly, and `LineEncoder` fast-path lines
|
|
6
|
+
- Keep hot-path optimizations always on: timestamp cache per UTC second, sanitizer and context fast paths, tag-list cache, single `Severity.name_for` per line
|
|
7
|
+
- Experimental: add `JsonLogging.isolation_level = :fiber` so context, tags, and hot-path caches isolate per Fiber (`:thread` remains default)
|
|
8
|
+
- Emit valid JSON when a large structured hash body is empty after field overrides
|
|
9
|
+
- Resolve the home-directory prefix without ActiveSupport `blank?`
|
|
10
|
+
- Stop release when confirmation input closes before an answer
|
|
11
|
+
- Require a matching `CHANGELOG.md` version heading before publish
|
|
12
|
+
- Run quality, test, and coverage jobs on pull requests to `main`
|
|
13
|
+
|
|
14
|
+
## 1.3.1 (2026-09-04)
|
|
15
|
+
|
|
16
|
+
- Copy large structured hashes before Rails parameter filtering so caller fields stay intact
|
|
17
|
+
- Keep original key names on the large-hash encoder path when a Rails parameter filter is present
|
|
18
|
+
- Isolate nested hashes in `with_context` from later caller mutations
|
|
19
|
+
- Match fallback sensitive-key detection to Rails default filter substrings including email, otp, ssn, token, and credentials
|
|
20
|
+
- Rebuild the compiled Rails parameter filter after in-place `filter_parameters` changes
|
|
21
|
+
- Treat JSON object and array strings longer than 10,000 characters as truncated text
|
|
22
|
+
- Replace home-directory prefixes in log strings and backtraces with a tilde
|
|
23
|
+
- Skip EventSubscriber writes when the logger has INFO disabled; keep the Rails event timestamp as an integer
|
|
24
|
+
- Keep overflow sensitive keys as filtered after the 50-key hash cap without copying their values
|
|
25
|
+
- Require json 2.21.2 or newer
|
|
26
|
+
|
|
3
27
|
## 1.3.0 (2026-07-25)
|
|
4
28
|
|
|
5
|
-
- BREAKING: Require Ruby 3.2+ and Rails 7+ (drop Ruby 2.7
|
|
29
|
+
- BREAKING: Require Ruby 3.2+ and Rails 7+ (drop Ruby 2.7-3.1 and Rails 6 support)
|
|
6
30
|
- Add opt-in `JsonLogging::EventSubscriber` for Rails 8.1 `ActiveSupport::EventReporter` / `Rails.event` structured events
|
|
7
31
|
|
|
8
32
|
## 1.2.3 (2026-07-13)
|
|
@@ -36,7 +60,7 @@
|
|
|
36
60
|
## 1.2.0 (2025-11-07)
|
|
37
61
|
|
|
38
62
|
- Add support for service-specific tagged loggers: create loggers with permanent tags using `logger.tagged("service")` without a block
|
|
39
|
-
- Improve BroadcastLogger compatibility: service-specific loggers work
|
|
63
|
+
- Improve BroadcastLogger compatibility: service-specific loggers work with `ActiveSupport::BroadcastLogger`
|
|
40
64
|
- Fix LocalTagStorage implementation to match Rails' TaggedLogging behavior: use `tag_stack` attribute accessor pattern for proper tag isolation
|
|
41
65
|
- Add comprehensive examples in README for service-specific loggers and BroadcastLogger integration
|
|
42
66
|
|
data/README.md
CHANGED
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
[](https://badge.fury.io/rb/activesupport-json_logging) [](https://github.com/amkisko/activesupport-json_logging.rb/actions/workflows/test.yml) [](https://app.codecov.io/github/amkisko/activesupport-json_logging.rb) [](https://sonarcloud.io/project/overview?id=amkisko_activesupport-json_logging.rb)
|
|
4
4
|
|
|
5
5
|
Structured JSON logging for Rails and ActiveSupport with a safe, single-line formatter.
|
|
6
|
-
No dependencies beyond Rails and
|
|
7
|
-
Supports Ruby 3.2+ and Rails 7
|
|
6
|
+
No dependencies beyond Rails and ActiveSupport.
|
|
7
|
+
Supports Ruby 3.2+ and Rails 7-8.
|
|
8
8
|
|
|
9
9
|
## Installation
|
|
10
10
|
|
|
@@ -35,10 +35,10 @@ end
|
|
|
35
35
|
- `JsonLogging.logger(*args)` - Convenience method that creates an `ActiveSupport::Logger` and wraps it
|
|
36
36
|
- Safe JSON serialization that never raises from the formatter
|
|
37
37
|
- `JsonLogging.with_context` to attach contextual fields per-thread
|
|
38
|
-
-
|
|
38
|
+
- Message parsing for hashes, JSON strings, plain strings, and Exception objects
|
|
39
39
|
- Native `tagged` method support - use it just like Rails' tagged logger
|
|
40
40
|
- Service-specific tagged loggers - create loggers with permanent tags using `logger.tagged("service")` without a block
|
|
41
|
-
-
|
|
41
|
+
- Compatible with `ActiveSupport::BroadcastLogger` (Rails 7.1+)
|
|
42
42
|
- Opt-in `JsonLogging::EventSubscriber` for Rails 8.1 `ActiveSupport::EventReporter` / `Rails.event`
|
|
43
43
|
- Automatic Rails integration via Railtie (auto-requires the gem in Rails apps)
|
|
44
44
|
|
|
@@ -93,8 +93,21 @@ This gem does **not** automatically configure your Rails app. You set it up manu
|
|
|
93
93
|
|
|
94
94
|
**Note:**
|
|
95
95
|
- In Rails apps, the gem is automatically required via Railtie, so you typically don't need to manually `require "json_logging"` in initializers (though it's harmless if you do).
|
|
96
|
-
- In Rails 7.1+, Rails
|
|
97
|
-
- In Rails 7.1+, tag storage uses `ActiveSupport::IsolatedExecutionState`
|
|
96
|
+
- In Rails 7.1+, Rails wraps the logger in `ActiveSupport::BroadcastLogger` so one logger can write to several destinations (for example STDOUT and a file). `JsonLogging` stays compatible: the JSON logger is wrapped and method calls delegate as usual.
|
|
97
|
+
- In Rails 7.1+, default tag/context storage uses `ActiveSupport::IsolatedExecutionState` (thread isolation unless the app sets Rails fiber isolation).
|
|
98
|
+
|
|
99
|
+
### Experimental Fiber isolation
|
|
100
|
+
|
|
101
|
+
For Fiber-based servers or code that interleaves many Fibers on one Thread, enable JsonLogging's experimental fiber storage:
|
|
102
|
+
|
|
103
|
+
```ruby
|
|
104
|
+
# config/initializers/json_logging.rb
|
|
105
|
+
JsonLogging.isolation_level = :fiber
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
This isolates `with_context`, logger tags, and JsonLogging hot-path caches per Fiber. It does **not** flip Rails' own `config.active_support.isolation_level`; Falcon-style apps usually set both.
|
|
109
|
+
|
|
110
|
+
Default remains `:thread`.
|
|
98
111
|
|
|
99
112
|
### Service-specific loggers with tags
|
|
100
113
|
|
|
@@ -148,7 +161,7 @@ Or enable the Railtie opt-in (off by default):
|
|
|
148
161
|
config.json_logging.subscribe_event_reporter = true
|
|
149
162
|
```
|
|
150
163
|
|
|
151
|
-
The subscriber writes one JSON line per event and keeps the Rails event shape (`name`, `payload`, `tags`, `context`, `timestamp`, `source_location`). It writes through `Logger#<<` so the logger formatter does not wrap the event again. Payload and tag objects that implement `#serialize` are encoded via that method. Encode and write failures never raise from `#emit`.
|
|
164
|
+
The subscriber writes one JSON line per event and keeps the Rails event shape (`name`, `payload`, `tags`, `context`, `timestamp`, `source_location`). It writes through `Logger#<<` so the logger formatter does not wrap the event again. It skips that write when the logger reports that INFO is disabled (`logger.level` above INFO). An IO destination always writes. `timestamp` stays the Rails event integer (nanoseconds), not the ISO8601 string used by `JsonLoggerExtension`. Payload and tag objects that implement `#serialize` are encoded via that method. Encode and write failures never raise from `#emit`.
|
|
152
165
|
|
|
153
166
|
```ruby
|
|
154
167
|
Rails.event.set_context(request_id: "abc123")
|
|
@@ -159,7 +172,7 @@ end
|
|
|
159
172
|
|
|
160
173
|
### BroadcastLogger integration
|
|
161
174
|
|
|
162
|
-
`ActiveSupport::BroadcastLogger` (Rails 7.1+)
|
|
175
|
+
`ActiveSupport::BroadcastLogger` (Rails 7.1+) writes the same log stream to multiple destinations. `JsonLogging` works with `BroadcastLogger`:
|
|
163
176
|
|
|
164
177
|
```ruby
|
|
165
178
|
# Create JSON loggers for different destinations
|
|
@@ -203,7 +216,7 @@ end
|
|
|
203
216
|
- Tagged logging (`tagged`) works correctly through `BroadcastLogger`
|
|
204
217
|
- Service-specific tagged loggers work with `BroadcastLogger`
|
|
205
218
|
- Each destination receives properly formatted JSON logs
|
|
206
|
-
-
|
|
219
|
+
- Wrap your `JsonLogging` logger in `BroadcastLogger` when you need multiple destinations
|
|
207
220
|
|
|
208
221
|
### Basic setup
|
|
209
222
|
|
|
@@ -443,13 +456,13 @@ logger.tagged("BCX").tagged("Jason").info("Stuff")
|
|
|
443
456
|
**Wrapping Compatibility:** You can wrap loggers that have already been wrapped with `ActiveSupport::TaggedLogging`:
|
|
444
457
|
|
|
445
458
|
```ruby
|
|
446
|
-
# Wrap a TaggedLogging logger
|
|
459
|
+
# Wrap a TaggedLogging logger
|
|
447
460
|
tagged_logger = ActiveSupport::TaggedLogging.new(Logger.new(STDOUT))
|
|
448
461
|
json_logger = JsonLogging.new(tagged_logger)
|
|
449
462
|
json_logger.tagged("TEST") { json_logger.info("message") } # Tags appear at root level in JSON
|
|
450
463
|
```
|
|
451
464
|
|
|
452
|
-
**Note:** If you wrap a `JsonLogging` logger with `ActiveSupport::TaggedLogging`, the TaggedLogging's text-based tags will appear as part of the message string in the JSON output, not as structured tags at the root level.
|
|
465
|
+
**Note:** If you wrap a `JsonLogging` logger with `ActiveSupport::TaggedLogging`, the TaggedLogging's text-based tags will appear as part of the message string in the JSON output, not as structured tags at the root level. Wrap loggers with `JsonLogging` last.
|
|
453
466
|
|
|
454
467
|
### JsonLogging::JsonLogger
|
|
455
468
|
|
|
@@ -551,7 +564,7 @@ end
|
|
|
551
564
|
|
|
552
565
|
The transformer receives the current thread-local context hash and should return a hash. If the transformer raises an error, the base context will be returned to avoid breaking logging.
|
|
553
566
|
|
|
554
|
-
**Note:** The transformer is called every time a log entry is created, so keep it
|
|
567
|
+
**Note:** The transformer is called every time a log entry is created, so keep it cheap to avoid slowing each line.
|
|
555
568
|
|
|
556
569
|
### Inherited Rails Logger Features
|
|
557
570
|
|
|
@@ -607,24 +620,25 @@ logger.reopen # Reopen the logger (if supported by logdev)
|
|
|
607
620
|
## Features
|
|
608
621
|
|
|
609
622
|
- **Native tagged logging**: Use `logger.tagged("TAG")` just like Rails' tagged logger
|
|
610
|
-
- **
|
|
623
|
+
- **Message parsing**: Handles hashes, JSON strings, plain strings, and Exception objects (with class, message, and backtrace)
|
|
611
624
|
- **Thread-safe context**: `JsonLogging.with_context` works across threads
|
|
612
|
-
- **Rails 7.1+ thread
|
|
613
|
-
- **
|
|
625
|
+
- **Rails 7.1+ isolation**: Default `:thread` storage uses `ActiveSupport::IsolatedExecutionState` when available
|
|
626
|
+
- **Experimental Fiber isolation**: `JsonLogging.isolation_level = :fiber` isolates context, tags, and hot-path caches per Fiber (for Fiber servers / interleaved Fibers on one Thread). Does not change Rails `config.active_support.isolation_level`; set that separately for CurrentAttributes and the rest of Rails.
|
|
627
|
+
- **Safe on encode errors**: Formatter and logger methods fall back to safe entries instead of raising
|
|
614
628
|
- **Single-line JSON**: Each log entry is a single line, safe for log aggregation tools
|
|
615
629
|
|
|
616
630
|
## Security & privacy
|
|
617
631
|
|
|
618
632
|
- **Rails ParameterFilter integration**: Automatically uses `Rails.application.config.filter_parameters` to filter sensitive data (passwords, tokens, etc.). This includes encrypted attributes automatically. See [Rails parameter filtering guide](https://thoughtbot.com/blog/parameter-filtering).
|
|
619
|
-
- **Input sanitization**: Removes control characters, truncates long strings, and limits structure depth/size:
|
|
633
|
+
- **Input sanitization**: Removes control characters, truncates long strings, redacts home-directory prefixes to `~`, and limits structure depth/size:
|
|
620
634
|
- Maximum string length: 10,000 characters (truncated with `...[truncated]` suffix)
|
|
621
|
-
- Maximum context hash size: 50 keys (additional keys are truncated)
|
|
635
|
+
- Maximum context hash size: 50 keys (additional keys are truncated; overflow keys that match sensitive patterns or `filter_parameters` still appear as `[FILTERED]`)
|
|
622
636
|
- Maximum nesting depth: 10 levels (deeper structures return `{"error" => "max_depth_exceeded"}`)
|
|
623
637
|
- Maximum backtrace lines: 20 lines per exception
|
|
624
638
|
- **Single-line JSON**: Emits single-line JSON to avoid log injection via newlines
|
|
625
|
-
- **
|
|
639
|
+
- **Safe on encode errors**: Formatter and logger never raise; fallback to safe entries on serialization errors
|
|
626
640
|
- **Sensitive key detection**: Falls back to pattern matching when Rails ParameterFilter isn't available
|
|
627
|
-
- **
|
|
641
|
+
- **Caller-owned context**: You decide what goes into context via `JsonLogging.with_context`; avoid sensitive data
|
|
628
642
|
|
|
629
643
|
### Configuring sensitive parameter filtering
|
|
630
644
|
|
|
@@ -632,7 +646,7 @@ This gem automatically uses Rails' `config.filter_parameters` when available. Co
|
|
|
632
646
|
|
|
633
647
|
```ruby
|
|
634
648
|
Rails.application.config.filter_parameters += [
|
|
635
|
-
:passw, :secret, :token, :_key, :crypt, :salt, :certificate, :otp, :ssn
|
|
649
|
+
:passw, :email, :secret, :token, :_key, :crypt, :salt, :certificate, :otp, :ssn
|
|
636
650
|
]
|
|
637
651
|
```
|
|
638
652
|
|
|
@@ -121,9 +121,13 @@ module JsonLogging
|
|
|
121
121
|
def write_line(line)
|
|
122
122
|
if @io
|
|
123
123
|
@io.write(line)
|
|
124
|
-
|
|
125
|
-
resolve_logger << line
|
|
124
|
+
return
|
|
126
125
|
end
|
|
126
|
+
|
|
127
|
+
logger = resolve_logger
|
|
128
|
+
return if logger.respond_to?(:info?) && !logger.info?
|
|
129
|
+
|
|
130
|
+
logger << line
|
|
127
131
|
end
|
|
128
132
|
|
|
129
133
|
def resolve_logger
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module JsonLogging
|
|
4
|
+
# Storage for request-scoped logging state (context, tags, hot-path caches).
|
|
5
|
+
#
|
|
6
|
+
# Default +:thread+ follows ActiveSupport::IsolatedExecutionState when present
|
|
7
|
+
# (Rails 7.1+), otherwise Thread.current.
|
|
8
|
+
#
|
|
9
|
+
# Experimental +:fiber+ stores JsonLogging keys on the current Fiber so concurrent
|
|
10
|
+
# Fibers on one Thread do not share context or tags. Does not change Rails'
|
|
11
|
+
# ActiveSupport::IsolatedExecutionState.isolation_level; Falcon-style apps should
|
|
12
|
+
# still set config.active_support.isolation_level = :fiber for the rest of Rails.
|
|
13
|
+
module ExecutionState
|
|
14
|
+
PREPARED_TAGS_CACHE_KEY = :__json_logging_prepared_tags
|
|
15
|
+
|
|
16
|
+
module_function
|
|
17
|
+
|
|
18
|
+
def [](key)
|
|
19
|
+
backend[key]
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def []=(key, value)
|
|
23
|
+
backend[key] = value
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def delete(key)
|
|
27
|
+
backend.delete(key)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def key?(key)
|
|
31
|
+
backend.key?(key)
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def prepared_tags_cache
|
|
35
|
+
self[PREPARED_TAGS_CACHE_KEY] ||= {}
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def backend
|
|
39
|
+
if JsonLogging.isolation_level == :fiber
|
|
40
|
+
fiber_backend
|
|
41
|
+
else
|
|
42
|
+
thread_backend
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def fiber_backend
|
|
47
|
+
state = Fiber.current.json_logging_execution_state
|
|
48
|
+
unless state
|
|
49
|
+
state = {}
|
|
50
|
+
Fiber.current.json_logging_execution_state = state
|
|
51
|
+
end
|
|
52
|
+
state
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def thread_backend
|
|
56
|
+
if defined?(ActiveSupport::IsolatedExecutionState)
|
|
57
|
+
ActiveSupport::IsolatedExecutionState
|
|
58
|
+
else
|
|
59
|
+
Thread.current
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
Fiber.attr_accessor :json_logging_execution_state unless Fiber.method_defined?(:json_logging_execution_state)
|
data/lib/json_logging/helpers.rb
CHANGED
|
@@ -2,6 +2,9 @@ module JsonLogging
|
|
|
2
2
|
module Helpers
|
|
3
3
|
ISO8601_MICROSECOND_UTC_PATTERN = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{6}Z\z/
|
|
4
4
|
|
|
5
|
+
# Microsecond bundle exec / hot add path: avoid fresh iso8601(6) every line when still in the same UTC second.
|
|
6
|
+
TIMESTAMP_CACHE_KEY = :__json_logging_ts_cache
|
|
7
|
+
|
|
5
8
|
module_function
|
|
6
9
|
|
|
7
10
|
# Normalize timestamp to ISO8601 with microseconds
|
|
@@ -12,7 +15,20 @@ module JsonLogging
|
|
|
12
15
|
end
|
|
13
16
|
|
|
14
17
|
def current_timestamp
|
|
15
|
-
|
|
18
|
+
cached_log_timestamp
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def cached_log_timestamp
|
|
22
|
+
now = current_time
|
|
23
|
+
sec_key = now.to_i
|
|
24
|
+
cache = ExecutionState[TIMESTAMP_CACHE_KEY]
|
|
25
|
+
if cache.is_a?(Array) && cache[0] == sec_key && cache[1]
|
|
26
|
+
return cache[1]
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
formatted = format_timestamp(now)
|
|
30
|
+
ExecutionState[TIMESTAMP_CACHE_KEY] = [sec_key, formatted]
|
|
31
|
+
formatted
|
|
16
32
|
end
|
|
17
33
|
|
|
18
34
|
def format_timestamp(time)
|
|
@@ -12,7 +12,7 @@ module JsonLogging
|
|
|
12
12
|
shift_age = args[1] || 0
|
|
13
13
|
shift_size = args[2]
|
|
14
14
|
|
|
15
|
-
# Handle both positional and keyword arguments for Rails 7
|
|
15
|
+
# Handle both positional and keyword arguments for Rails 7-8 compatibility
|
|
16
16
|
if kwargs.empty? && shift_size
|
|
17
17
|
super(logdev, shift_age, shift_size)
|
|
18
18
|
elsif kwargs.empty?
|
|
@@ -22,10 +22,11 @@ module JsonLogging
|
|
|
22
22
|
|
|
23
23
|
msg = nil
|
|
24
24
|
msg = extract_log_message(message, progname, &block)
|
|
25
|
+
timestamp = Helpers.cached_log_timestamp
|
|
25
26
|
line = LineEncoder.build_line(
|
|
26
27
|
msg: msg,
|
|
27
28
|
severity: severity,
|
|
28
|
-
timestamp:
|
|
29
|
+
timestamp: timestamp,
|
|
29
30
|
tags: formatter.current_tags,
|
|
30
31
|
additional_context: JsonLogging.additional_context_for_payload,
|
|
31
32
|
additional_context_sanitized: true,
|
|
@@ -77,8 +78,9 @@ module JsonLogging
|
|
|
77
78
|
|
|
78
79
|
def write_add_fallback(severity, msg, error)
|
|
79
80
|
msg ||= "<uninitialized>"
|
|
81
|
+
ts = Helpers.cached_log_timestamp
|
|
80
82
|
fallback = {
|
|
81
|
-
timestamp:
|
|
83
|
+
timestamp: ts,
|
|
82
84
|
severity: Severity.name_for(severity),
|
|
83
85
|
message: Sanitizer.sanitize_string(Helpers.safe_string(msg)),
|
|
84
86
|
logger_error: {
|
|
@@ -94,27 +96,18 @@ module JsonLogging
|
|
|
94
96
|
end
|
|
95
97
|
|
|
96
98
|
def current_tags
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
else
|
|
102
|
-
Thread.current[tags_key] ||= []
|
|
103
|
-
end
|
|
99
|
+
tags = ExecutionState[tags_key]
|
|
100
|
+
return tags if tags
|
|
101
|
+
|
|
102
|
+
ExecutionState[tags_key] = []
|
|
104
103
|
end
|
|
105
104
|
|
|
106
105
|
def set_tags(new_tags)
|
|
107
|
-
|
|
108
|
-
# Falls back to Thread.current for Rails 7.0
|
|
109
|
-
if defined?(ActiveSupport::IsolatedExecutionState)
|
|
110
|
-
ActiveSupport::IsolatedExecutionState[tags_key] = new_tags
|
|
111
|
-
else
|
|
112
|
-
Thread.current[tags_key] = new_tags
|
|
113
|
-
end
|
|
106
|
+
ExecutionState[tags_key] = new_tags
|
|
114
107
|
end
|
|
115
108
|
|
|
116
109
|
def push_tags(tags)
|
|
117
|
-
flat = Sanitizer.prepare_tags(tags)
|
|
110
|
+
flat = Sanitizer.prepare_tags(tags, tag_source: :logger_extension_push)
|
|
118
111
|
return if flat.empty?
|
|
119
112
|
|
|
120
113
|
set_tags(current_tags + flat)
|
|
@@ -154,7 +147,7 @@ module JsonLogging
|
|
|
154
147
|
end
|
|
155
148
|
|
|
156
149
|
def push_tags(tags)
|
|
157
|
-
flat = Sanitizer.prepare_tags(tags)
|
|
150
|
+
flat = Sanitizer.prepare_tags(tags, tag_source: :local_tag_stack)
|
|
158
151
|
return [] if flat.empty?
|
|
159
152
|
|
|
160
153
|
@tags.concat(flat)
|
|
@@ -10,10 +10,12 @@ module JsonLogging
|
|
|
10
10
|
module_function
|
|
11
11
|
|
|
12
12
|
def build_line(msg:, severity:, timestamp:, tags:, additional_context:, additional_context_sanitized: false, sanitize_tags: true)
|
|
13
|
+
severity_string = Severity.name_for(severity)
|
|
14
|
+
|
|
13
15
|
if simple_string_line?(msg, tags, additional_context)
|
|
14
16
|
return simple_string_json_line(
|
|
15
17
|
message: msg,
|
|
16
|
-
severity:
|
|
18
|
+
severity: severity_string,
|
|
17
19
|
timestamp: timestamp
|
|
18
20
|
)
|
|
19
21
|
end
|
|
@@ -21,7 +23,7 @@ module JsonLogging
|
|
|
21
23
|
if tagged_simple_string_line?(msg, tags, additional_context)
|
|
22
24
|
return tagged_simple_string_json_line(
|
|
23
25
|
message: msg,
|
|
24
|
-
severity:
|
|
26
|
+
severity: severity_string,
|
|
25
27
|
timestamp: timestamp,
|
|
26
28
|
tags: tags,
|
|
27
29
|
sanitize_tags: sanitize_tags
|
|
@@ -31,7 +33,7 @@ module JsonLogging
|
|
|
31
33
|
if contextual_simple_string_line?(msg, tags, additional_context, additional_context_sanitized: additional_context_sanitized)
|
|
32
34
|
return contextual_simple_string_json_line(
|
|
33
35
|
message: msg,
|
|
34
|
-
severity:
|
|
36
|
+
severity: severity_string,
|
|
35
37
|
timestamp: timestamp,
|
|
36
38
|
additional_context: additional_context
|
|
37
39
|
)
|
|
@@ -40,7 +42,7 @@ module JsonLogging
|
|
|
40
42
|
if standalone_hash_line?(msg, tags, additional_context)
|
|
41
43
|
return standalone_hash_json_line(
|
|
42
44
|
message: msg,
|
|
43
|
-
severity:
|
|
45
|
+
severity: severity_string,
|
|
44
46
|
timestamp: timestamp
|
|
45
47
|
)
|
|
46
48
|
end
|
|
@@ -48,7 +50,7 @@ module JsonLogging
|
|
|
48
50
|
if tagged_hash_line?(msg, tags, additional_context)
|
|
49
51
|
return tagged_hash_json_line(
|
|
50
52
|
message: msg,
|
|
51
|
-
severity:
|
|
53
|
+
severity: severity_string,
|
|
52
54
|
timestamp: timestamp,
|
|
53
55
|
tags: tags,
|
|
54
56
|
sanitize_tags: sanitize_tags
|
|
@@ -58,13 +60,13 @@ module JsonLogging
|
|
|
58
60
|
if contextual_hash_line?(msg, tags, additional_context, additional_context_sanitized: additional_context_sanitized)
|
|
59
61
|
return contextual_hash_json_line(
|
|
60
62
|
message: msg,
|
|
61
|
-
severity:
|
|
63
|
+
severity: severity_string,
|
|
62
64
|
timestamp: timestamp,
|
|
63
65
|
additional_context: additional_context
|
|
64
66
|
)
|
|
65
67
|
end
|
|
66
68
|
|
|
67
|
-
sev =
|
|
69
|
+
sev = severity_string
|
|
68
70
|
payload = PayloadBuilder.build_base_payload(msg, severity: sev, timestamp: timestamp)
|
|
69
71
|
unless tags.empty? && PayloadBuilder.empty_additional_context?(additional_context)
|
|
70
72
|
payload = PayloadBuilder.merge_context(
|
|
@@ -81,7 +83,7 @@ module JsonLogging
|
|
|
81
83
|
def to_json_line(payload_hash)
|
|
82
84
|
compacted = payload_hash.compact
|
|
83
85
|
json_payload = string_keyed_structure?(compacted) ? compacted : compacted.deep_stringify_keys
|
|
84
|
-
"#{json_payload
|
|
86
|
+
"#{JSON.fast_generate(json_payload)}\n"
|
|
85
87
|
end
|
|
86
88
|
|
|
87
89
|
def string_keyed_structure?(object)
|
|
@@ -104,7 +106,7 @@ module JsonLogging
|
|
|
104
106
|
|
|
105
107
|
def simple_string_json_line(message:, severity:, timestamp:)
|
|
106
108
|
sanitized_message = Sanitizer.sanitize_string(message)
|
|
107
|
-
"#{JSON.
|
|
109
|
+
"#{JSON.fast_generate("message" => sanitized_message, "severity" => severity, "timestamp" => timestamp)}\n"
|
|
108
110
|
end
|
|
109
111
|
|
|
110
112
|
def tagged_simple_string_line?(message, tags, additional_context)
|
|
@@ -116,8 +118,13 @@ module JsonLogging
|
|
|
116
118
|
|
|
117
119
|
def tagged_simple_string_json_line(message:, severity:, timestamp:, tags:, sanitize_tags:)
|
|
118
120
|
sanitized_message = Sanitizer.sanitize_string(message)
|
|
119
|
-
prepared_tags = sanitize_tags
|
|
120
|
-
|
|
121
|
+
prepared_tags = if sanitize_tags
|
|
122
|
+
t = Sanitizer.prepare_tags(tags, tag_source: :line_encoder_tagged_simple)
|
|
123
|
+
t.uniq
|
|
124
|
+
else
|
|
125
|
+
tags
|
|
126
|
+
end
|
|
127
|
+
"#{JSON.fast_generate("message" => sanitized_message, "severity" => severity, "timestamp" => timestamp, "tags" => prepared_tags)}\n"
|
|
121
128
|
end
|
|
122
129
|
|
|
123
130
|
def contextual_simple_string_line?(message, tags, additional_context, additional_context_sanitized:)
|
|
@@ -137,7 +144,7 @@ module JsonLogging
|
|
|
137
144
|
}
|
|
138
145
|
context = PayloadBuilder.context_payload_portion(additional_context)
|
|
139
146
|
payload["context"] = context unless context.empty?
|
|
140
|
-
"#{JSON.
|
|
147
|
+
"#{JSON.fast_generate(payload)}\n"
|
|
141
148
|
end
|
|
142
149
|
|
|
143
150
|
def standalone_hash_line?(message, tags, additional_context)
|
|
@@ -158,7 +165,7 @@ module JsonLogging
|
|
|
158
165
|
payload = Sanitizer.sanitize_hash(hash)
|
|
159
166
|
payload["severity"] = severity
|
|
160
167
|
payload["timestamp"] = timestamp
|
|
161
|
-
"#{JSON.
|
|
168
|
+
"#{JSON.fast_generate(payload)}\n"
|
|
162
169
|
end
|
|
163
170
|
|
|
164
171
|
def tagged_hash_line?(message, tags, additional_context)
|
|
@@ -169,7 +176,11 @@ module JsonLogging
|
|
|
169
176
|
|
|
170
177
|
def tagged_hash_json_line(message:, severity:, timestamp:, tags:, sanitize_tags:)
|
|
171
178
|
hash = message.is_a?(Hash) ? message : message.to_hash
|
|
172
|
-
prepared_tags = sanitize_tags
|
|
179
|
+
prepared_tags = if sanitize_tags
|
|
180
|
+
Sanitizer.prepare_tags(tags, tag_source: :line_encoder_tagged_hash)
|
|
181
|
+
else
|
|
182
|
+
tags
|
|
183
|
+
end
|
|
173
184
|
merged_tags = (Array(hash[:tags] || hash["tags"]) + prepared_tags).uniq
|
|
174
185
|
field_overrides = {"tags" => merged_tags}
|
|
175
186
|
|
|
@@ -185,7 +196,7 @@ module JsonLogging
|
|
|
185
196
|
payload["severity"] = severity
|
|
186
197
|
payload["timestamp"] = timestamp
|
|
187
198
|
payload["tags"] = merged_tags
|
|
188
|
-
"#{JSON.
|
|
199
|
+
"#{JSON.fast_generate(payload)}\n"
|
|
189
200
|
end
|
|
190
201
|
|
|
191
202
|
def contextual_hash_line?(message, tags, additional_context, additional_context_sanitized:)
|
|
@@ -220,15 +231,7 @@ module JsonLogging
|
|
|
220
231
|
existing_context = payload["context"].is_a?(Hash) ? payload["context"] : {}
|
|
221
232
|
payload["context"] = existing_context.merge(merged_context)
|
|
222
233
|
end
|
|
223
|
-
"#{JSON.
|
|
224
|
-
end
|
|
225
|
-
|
|
226
|
-
def build_sanitized_hash_payload(message:, severity:, timestamp:)
|
|
227
|
-
hash = message.is_a?(Hash) ? message : message.to_hash
|
|
228
|
-
payload = Sanitizer.sanitize_hash(hash)
|
|
229
|
-
payload["severity"] = severity
|
|
230
|
-
payload["timestamp"] = timestamp
|
|
231
|
-
payload
|
|
234
|
+
"#{JSON.fast_generate(payload)}\n"
|
|
232
235
|
end
|
|
233
236
|
|
|
234
237
|
def deep_stringify_structure(obj)
|
|
@@ -25,6 +25,10 @@ module JsonLogging
|
|
|
25
25
|
end
|
|
26
26
|
|
|
27
27
|
def json_string?(str)
|
|
28
|
+
# Short strings cannot be JSON object/array literals with both delimiters.
|
|
29
|
+
return false if str.length < 2
|
|
30
|
+
return false if str.length > Sanitizer::MAX_STRING_LENGTH
|
|
31
|
+
|
|
28
32
|
(str.start_with?("{") && str.end_with?("}")) ||
|
|
29
33
|
(str.start_with?("[") && str.end_with?("]"))
|
|
30
34
|
end
|
|
@@ -25,30 +25,42 @@ module JsonLogging
|
|
|
25
25
|
return merge_tags_only(payload, tags: tags, sanitize_tags: sanitize_tags) if tags_only_merge?(payload, additional_context: additional_context, tags: tags)
|
|
26
26
|
|
|
27
27
|
existing_context = payload["context"].is_a?(Hash) ? payload["context"] : {}
|
|
28
|
+
user_context = filtered_user_context(additional_context, sanitized: additional_context_sanitized)
|
|
29
|
+
deduped = user_context.reject { |key, _| payload.key?(key) || payload.key?(key.to_s) }
|
|
30
|
+
merged_context = existing_context.merge(deduped)
|
|
28
31
|
|
|
29
|
-
|
|
32
|
+
assign_tags!(payload, tags: tags, sanitize_tags: sanitize_tags) unless tags.empty?
|
|
33
|
+
assign_context!(payload, merged_context) unless merged_context.empty?
|
|
34
|
+
payload
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def filtered_user_context(additional_context, sanitized:)
|
|
38
|
+
context = resolve_additional_context(additional_context, sanitized: sanitized)
|
|
39
|
+
context.except(*SYSTEM_CONTROLLED_KEYS)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def resolve_additional_context(additional_context, sanitized:)
|
|
43
|
+
if sanitized
|
|
30
44
|
additional_context.is_a?(Hash) ? additional_context : {}
|
|
31
45
|
elsif additional_context.is_a?(Hash) && !additional_context.empty?
|
|
32
46
|
Sanitizer.sanitize_hash(additional_context)
|
|
33
47
|
else
|
|
34
48
|
additional_context || {}
|
|
35
49
|
end
|
|
50
|
+
end
|
|
36
51
|
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
unless tags.empty?
|
|
43
|
-
existing_tags = Array(payload["tags"])
|
|
44
|
-
prepared_tags = sanitize_tags ? tags.map { |tag| Sanitizer.sanitize_string(tag.to_s) } : tags
|
|
45
|
-
payload["tags"] = (existing_tags + prepared_tags).uniq
|
|
46
|
-
end
|
|
52
|
+
def assign_tags!(payload, tags:, sanitize_tags:)
|
|
53
|
+
existing_tags = Array(payload["tags"])
|
|
54
|
+
prepared_tags = sanitize_tags ? tags.map { |tag| Sanitizer.sanitize_string(tag.to_s) } : tags
|
|
55
|
+
payload["tags"] = (existing_tags + prepared_tags).uniq
|
|
56
|
+
end
|
|
47
57
|
|
|
48
|
-
|
|
49
|
-
|
|
58
|
+
def assign_context!(payload, merged_context)
|
|
59
|
+
payload["context"] = if merged_context.keys.all? { |key| key.is_a?(String) }
|
|
60
|
+
merged_context
|
|
61
|
+
else
|
|
62
|
+
merged_context.transform_keys(&:to_s)
|
|
50
63
|
end
|
|
51
|
-
payload
|
|
52
64
|
end
|
|
53
65
|
|
|
54
66
|
def empty_additional_context?(additional_context)
|
|
@@ -73,7 +85,8 @@ module JsonLogging
|
|
|
73
85
|
end
|
|
74
86
|
|
|
75
87
|
def merge_tags_only(payload, tags:, sanitize_tags:)
|
|
76
|
-
|
|
88
|
+
raw = payload["tags"]
|
|
89
|
+
existing_tags = raw ? Array(raw) : []
|
|
77
90
|
prepared_tags = sanitize_tags ? tags.map { |tag| Sanitizer.sanitize_string(tag.to_s) } : tags
|
|
78
91
|
payload["tags"] = (existing_tags + prepared_tags).uniq
|
|
79
92
|
payload
|
|
@@ -81,6 +94,7 @@ module JsonLogging
|
|
|
81
94
|
|
|
82
95
|
def context_payload_portion(additional_context)
|
|
83
96
|
return {} unless additional_context.is_a?(Hash)
|
|
97
|
+
return additional_context if additional_context.empty?
|
|
84
98
|
|
|
85
99
|
additional_context.reject do |key, _|
|
|
86
100
|
SYSTEM_CONTROLLED_KEYS.include?(key) || SYSTEM_CONTROLLED_KEYS.include?(key.to_s)
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
require_relative "structured_hash_sanitizer"
|
|
2
|
+
require_relative "sanitizer_hash_limit"
|
|
2
3
|
|
|
3
4
|
module JsonLogging
|
|
5
|
+
# rubocop:disable Metrics/ModuleLength -- sanitizer owns string/hash/tag policy in one place
|
|
4
6
|
module Sanitizer
|
|
5
7
|
# Control characters that should be escaped or removed from log messages
|
|
6
8
|
CONTROL_CHARS = /[\x00-\x1F\x7F]/
|
|
@@ -18,7 +20,7 @@ module JsonLogging
|
|
|
18
20
|
MAX_BACKTRACE_LINES = 20
|
|
19
21
|
|
|
20
22
|
# Common sensitive key patterns (case insensitive) - fallback when Rails ParameterFilter not available
|
|
21
|
-
SENSITIVE_KEY_PATTERNS =
|
|
23
|
+
SENSITIVE_KEY_PATTERNS = /(passw|pwd|email|secret|token|_key|crypt|salt|certificate|otp|ssn|cvv|cvc|credential)/i
|
|
22
24
|
|
|
23
25
|
@parameter_filter = nil
|
|
24
26
|
@parameter_filter_config = nil
|
|
@@ -38,7 +40,7 @@ module JsonLogging
|
|
|
38
40
|
return @parameter_filter
|
|
39
41
|
end
|
|
40
42
|
|
|
41
|
-
@parameter_filter_config = filter_params
|
|
43
|
+
@parameter_filter_config = filter_params.dup.freeze
|
|
42
44
|
@parameter_filter_requires_full_tree_walk = parameter_filter_requires_full_tree_walk?(filter_params)
|
|
43
45
|
@parameter_filter = ActiveSupport::ParameterFilter.new(filter_params)
|
|
44
46
|
rescue
|
|
@@ -69,19 +71,34 @@ module JsonLogging
|
|
|
69
71
|
@parameter_filter_requires_full_tree_walk = false
|
|
70
72
|
end
|
|
71
73
|
|
|
72
|
-
def prepare_tags(tags)
|
|
73
|
-
tags.flatten.compact.map(&:to_s).reject(&:empty?)
|
|
74
|
+
def prepare_tags(tags, tag_source: nil)
|
|
75
|
+
flat = tags.flatten.compact.map(&:to_s).reject(&:empty?)
|
|
76
|
+
return [] if flat.empty?
|
|
77
|
+
|
|
78
|
+
if tag_source && flat.none? { |t| t.match?(CONTROL_CHARS) } && flat.all? { |t| t.length <= MAX_STRING_LENGTH }
|
|
79
|
+
cache = ExecutionState.prepared_tags_cache
|
|
80
|
+
key = [tag_source, flat]
|
|
81
|
+
return cache[key] if cache.key?(key)
|
|
82
|
+
|
|
83
|
+
prepared = flat.map { |tag| sanitize_string(tag) }
|
|
84
|
+
cache[key] = prepared.freeze
|
|
85
|
+
return prepared
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
flat.map { |tag| sanitize_string(tag) }
|
|
74
89
|
end
|
|
75
90
|
|
|
76
91
|
# Sanitize a string by removing/escaping control characters and truncating
|
|
77
92
|
def sanitize_string(str)
|
|
78
93
|
return str unless str.is_a?(String)
|
|
79
|
-
return str if str.length <= MAX_STRING_LENGTH && !str.match?(CONTROL_CHARS)
|
|
80
94
|
|
|
81
|
-
|
|
95
|
+
home_prefix = home_directory_prefix
|
|
96
|
+
has_home_directory = home_prefix && str.include?(home_prefix)
|
|
97
|
+
return str if str.length <= MAX_STRING_LENGTH && !str.match?(CONTROL_CHARS) && !has_home_directory
|
|
98
|
+
|
|
82
99
|
sanitized = str.gsub(CONTROL_CHARS, "")
|
|
100
|
+
sanitized = redact_home_directory(sanitized, home_prefix) if has_home_directory
|
|
83
101
|
|
|
84
|
-
# Truncate if too long
|
|
85
102
|
if sanitized.length > MAX_STRING_LENGTH
|
|
86
103
|
sanitized = sanitized[0, MAX_STRING_LENGTH] + "...[truncated]"
|
|
87
104
|
end
|
|
@@ -91,13 +108,10 @@ module JsonLogging
|
|
|
91
108
|
"<sanitization_error>"
|
|
92
109
|
end
|
|
93
110
|
|
|
94
|
-
# Sanitize a hash, removing sensitive keys and limiting size/depth
|
|
95
|
-
# Uses Rails ParameterFilter when available, falls back to pattern matching
|
|
96
111
|
def sanitize_hash(hash, depth: 0)
|
|
97
112
|
return hash unless hash.is_a?(Hash)
|
|
98
|
-
|
|
99
|
-
# Prevent excessive nesting
|
|
100
113
|
return {"error" => "max_depth_exceeded"} if depth > MAX_DEPTH
|
|
114
|
+
return hash if hash.empty?
|
|
101
115
|
|
|
102
116
|
limited_hash = limited_hash_for_sanitization(hash)
|
|
103
117
|
fast_path_result = fast_path_sanitized_hash(limited_hash, depth: depth)
|
|
@@ -109,13 +123,11 @@ module JsonLogging
|
|
|
109
123
|
end
|
|
110
124
|
|
|
111
125
|
def limited_hash_for_sanitization(hash)
|
|
112
|
-
if hash.size
|
|
113
|
-
|
|
114
|
-
truncated["_truncated"] = true
|
|
115
|
-
truncated
|
|
116
|
-
else
|
|
117
|
-
hash
|
|
126
|
+
if !rails_parameter_filter && hash.is_a?(Hash) && hash.size <= MAX_CONTEXT_SIZE
|
|
127
|
+
return hash
|
|
118
128
|
end
|
|
129
|
+
|
|
130
|
+
SanitizerHashLimit.apply(hash, filter_config: rails_parameter_filter && @parameter_filter_config)
|
|
119
131
|
end
|
|
120
132
|
|
|
121
133
|
def fast_path_sanitized_hash(hash, depth: 0)
|
|
@@ -126,38 +138,26 @@ module JsonLogging
|
|
|
126
138
|
end
|
|
127
139
|
|
|
128
140
|
def sanitize_hash_with_filtering(limited_hash, depth: 0)
|
|
129
|
-
# Use Rails ParameterFilter if available (handles encrypted attributes automatically)
|
|
130
141
|
filter = rails_parameter_filter
|
|
131
142
|
if filter
|
|
132
|
-
# ParameterFilter
|
|
133
|
-
# This includes encrypted attributes automatically
|
|
134
|
-
# Create a deep copy since filter modifies in place
|
|
143
|
+
# ParameterFilter mutates in place; copy first
|
|
135
144
|
filtered = limited_hash.respond_to?(:deep_dup) ? limited_hash.deep_dup : limited_hash.dup
|
|
136
145
|
filtered = filter.filter(filtered)
|
|
137
|
-
|
|
138
|
-
# Then sanitize values (strings, control chars, etc.) preserving filtered structure
|
|
139
146
|
filtered.each_with_object({}) do |(key, value), result|
|
|
140
147
|
result[key.to_s] = sanitize_value(value, depth: depth + 1)
|
|
141
148
|
end
|
|
142
|
-
|
|
143
149
|
else
|
|
144
|
-
# Fallback: use pattern matching for sensitive keys
|
|
145
150
|
limited_hash.each_with_object({}) do |(key, value), result|
|
|
146
151
|
key_string = key.to_s
|
|
147
|
-
|
|
148
|
-
# Skip sensitive keys
|
|
149
152
|
if SENSITIVE_KEY_PATTERNS.match?(key_string)
|
|
150
153
|
result[sensitive_filtered_key_name(key_string)] = "[FILTERED]"
|
|
151
154
|
next
|
|
152
155
|
end
|
|
153
|
-
|
|
154
156
|
result[key_string] = sanitize_value(value, depth: depth + 1)
|
|
155
157
|
end
|
|
156
158
|
end
|
|
157
159
|
end
|
|
158
160
|
|
|
159
|
-
# Sanitize a value (handles strings, hashes, arrays, etc.)
|
|
160
|
-
# Preserves numeric, boolean, and nil types
|
|
161
161
|
def sanitize_value(value, depth: 0)
|
|
162
162
|
case value
|
|
163
163
|
when String
|
|
@@ -165,24 +165,22 @@ module JsonLogging
|
|
|
165
165
|
when Hash
|
|
166
166
|
sanitize_hash(value, depth: depth)
|
|
167
167
|
when Array
|
|
168
|
-
|
|
168
|
+
return [] if value.empty?
|
|
169
|
+
|
|
169
170
|
sanitized = value.first(MAX_CONTEXT_SIZE).map { |v| sanitize_value(v, depth: depth + 1) }
|
|
170
171
|
sanitized << "[truncated]" if value.size > MAX_CONTEXT_SIZE
|
|
171
172
|
sanitized
|
|
172
173
|
when Exception
|
|
173
174
|
sanitize_exception(value)
|
|
174
175
|
when Numeric, TrueClass, FalseClass, NilClass
|
|
175
|
-
# Preserve numeric, boolean, and nil types
|
|
176
176
|
value
|
|
177
177
|
else
|
|
178
|
-
# For other types, convert to string and sanitize
|
|
179
178
|
sanitize_string(value.to_s)
|
|
180
179
|
end
|
|
181
180
|
rescue
|
|
182
181
|
"<unprintable>"
|
|
183
182
|
end
|
|
184
183
|
|
|
185
|
-
# Sanitize exception, including backtrace
|
|
186
184
|
def sanitize_exception(ex)
|
|
187
185
|
{
|
|
188
186
|
"error" => {
|
|
@@ -195,11 +193,9 @@ module JsonLogging
|
|
|
195
193
|
{"error" => {"class" => "Exception", "message" => "<sanitization_failed>"}}
|
|
196
194
|
end
|
|
197
195
|
|
|
198
|
-
# Sanitize backtrace - truncate and remove sensitive paths
|
|
199
196
|
def sanitize_backtrace(backtrace)
|
|
200
197
|
return [] unless backtrace.is_a?(Array)
|
|
201
198
|
|
|
202
|
-
# Take first MAX_BACKTRACE_LINES, sanitize each
|
|
203
199
|
backtrace.first(MAX_BACKTRACE_LINES).map do |line|
|
|
204
200
|
sanitize_string(line.to_s)
|
|
205
201
|
end
|
|
@@ -207,7 +203,6 @@ module JsonLogging
|
|
|
207
203
|
[]
|
|
208
204
|
end
|
|
209
205
|
|
|
210
|
-
# Check if a key looks sensitive
|
|
211
206
|
def sensitive_key?(key)
|
|
212
207
|
SENSITIVE_KEY_PATTERNS.match?(key.to_s)
|
|
213
208
|
end
|
|
@@ -255,5 +250,25 @@ module JsonLogging
|
|
|
255
250
|
def sensitive_filtered_key_name(key_string)
|
|
256
251
|
key_string.gsub(/(?<!^)(?=[A-Z])/, "_").downcase + "_filtered"
|
|
257
252
|
end
|
|
253
|
+
|
|
254
|
+
def home_directory_prefix
|
|
255
|
+
prefix = ENV["HOME"]
|
|
256
|
+
# rubocop:disable Rails/Blank -- string path must not require ActiveSupport blank?
|
|
257
|
+
prefix = Dir.home if prefix.nil? || prefix.empty?
|
|
258
|
+
# rubocop:enable Rails/Blank
|
|
259
|
+
prefix = prefix.to_s.chomp("/").chomp("\\")
|
|
260
|
+
return if prefix.empty?
|
|
261
|
+
|
|
262
|
+
prefix
|
|
263
|
+
rescue ArgumentError, RuntimeError
|
|
264
|
+
nil
|
|
265
|
+
end
|
|
266
|
+
|
|
267
|
+
def redact_home_directory(str, prefix)
|
|
268
|
+
return str unless prefix
|
|
269
|
+
|
|
270
|
+
str.gsub(/#{Regexp.escape(prefix)}(?=\/|\\|\z)/, "~")
|
|
271
|
+
end
|
|
258
272
|
end
|
|
273
|
+
# rubocop:enable Metrics/ModuleLength
|
|
259
274
|
end
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
module JsonLogging
|
|
2
|
+
module SanitizerHashLimit
|
|
3
|
+
FILTERED = "[FILTERED]"
|
|
4
|
+
|
|
5
|
+
module_function
|
|
6
|
+
|
|
7
|
+
def apply(hash, filter_config:)
|
|
8
|
+
return hash if hash.size <= Sanitizer::MAX_CONTEXT_SIZE
|
|
9
|
+
|
|
10
|
+
truncated = {}
|
|
11
|
+
kept = 0
|
|
12
|
+
extra_filtered = 0
|
|
13
|
+
hash.each do |key, value|
|
|
14
|
+
if kept < Sanitizer::MAX_CONTEXT_SIZE
|
|
15
|
+
truncated[key] = value
|
|
16
|
+
kept += 1
|
|
17
|
+
elsif extra_filtered < Sanitizer::MAX_CONTEXT_SIZE && key_requires_filtering?(key, filter_config)
|
|
18
|
+
truncated[key] = FILTERED
|
|
19
|
+
extra_filtered += 1
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
truncated["_truncated"] = true
|
|
23
|
+
truncated
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def key_requires_filtering?(key, filter_config)
|
|
27
|
+
return true if Sanitizer.sensitive_key?(key)
|
|
28
|
+
|
|
29
|
+
Array(filter_config).any? do |filter|
|
|
30
|
+
next false if filter.is_a?(Proc)
|
|
31
|
+
|
|
32
|
+
key_string = key.to_s
|
|
33
|
+
if filter.is_a?(Regexp)
|
|
34
|
+
filter.match?(key_string)
|
|
35
|
+
else
|
|
36
|
+
token = filter.to_s
|
|
37
|
+
!token.empty? && !token.include?(".") && key_string.match?(/#{Regexp.escape(token)}/i)
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
@@ -6,35 +6,6 @@ module JsonLogging
|
|
|
6
6
|
|
|
7
7
|
module_function
|
|
8
8
|
|
|
9
|
-
def eligible?(hash)
|
|
10
|
-
return false if Sanitizer.primitive_log_hash?(hash)
|
|
11
|
-
return false unless Sanitizer::StructuredHash.structured_log_hash?(hash)
|
|
12
|
-
return false unless large_structured_hash?(hash)
|
|
13
|
-
|
|
14
|
-
filter = Sanitizer.rails_parameter_filter
|
|
15
|
-
return true unless filter
|
|
16
|
-
|
|
17
|
-
!Sanitizer.rails_parameter_filter_requires_full_tree_walk?
|
|
18
|
-
end
|
|
19
|
-
|
|
20
|
-
def large_structured_hash?(hash, leaf_threshold: LEAF_THRESHOLD)
|
|
21
|
-
leaf_count = 0
|
|
22
|
-
walker = lambda do |value|
|
|
23
|
-
case value
|
|
24
|
-
when Hash
|
|
25
|
-
value.each_value { |entry| walker.call(entry) }
|
|
26
|
-
when Array
|
|
27
|
-
value.each { |entry| walker.call(entry) }
|
|
28
|
-
else
|
|
29
|
-
leaf_count += 1
|
|
30
|
-
throw(:enough, true) if leaf_count > leaf_threshold
|
|
31
|
-
end
|
|
32
|
-
end
|
|
33
|
-
|
|
34
|
-
catch(:enough) { walker.call(hash) }
|
|
35
|
-
leaf_count > leaf_threshold
|
|
36
|
-
end
|
|
37
|
-
|
|
38
9
|
def try_encode_line(hash, severity:, timestamp:, field_overrides: {})
|
|
39
10
|
tree = prepared_tree(hash, field_overrides: field_overrides)
|
|
40
11
|
return nil unless tree
|
|
@@ -88,19 +59,17 @@ module JsonLogging
|
|
|
88
59
|
return jsonable.tree unless filter
|
|
89
60
|
return jsonable.tree if Sanitizer.rails_parameter_filter_requires_full_tree_walk?
|
|
90
61
|
|
|
91
|
-
|
|
92
|
-
jsonable.tree
|
|
93
|
-
else
|
|
94
|
-
Sanitizer::StructuredHash.stringify_keys_copy(jsonable.tree)
|
|
95
|
-
end
|
|
96
|
-
|
|
97
|
-
filter.filter(tree)
|
|
62
|
+
filter.filter(Sanitizer::StructuredHash.stringify(jsonable.tree))
|
|
98
63
|
end
|
|
99
64
|
|
|
100
65
|
def append_json_line(tree, severity:, timestamp:, field_overrides:)
|
|
101
|
-
body = JSON.generate(tree)
|
|
102
66
|
extras = field_overrides.merge("severity" => severity, "timestamp" => timestamp)
|
|
103
|
-
|
|
67
|
+
# rubocop:disable Rails/Blank -- encoder path must not require ActiveSupport blank?
|
|
68
|
+
return "#{JSON.fast_generate(extras)}\n" if tree.nil? || tree.empty?
|
|
69
|
+
# rubocop:enable Rails/Blank
|
|
70
|
+
|
|
71
|
+
body = JSON.fast_generate(tree)
|
|
72
|
+
extra_json = extras.map { |key, value| "#{JSON.fast_generate(key.to_s)}:#{JSON.fast_generate(value)}" }.join(",")
|
|
104
73
|
"#{body.chop},#{extra_json}}\n"
|
|
105
74
|
end
|
|
106
75
|
|
|
@@ -38,17 +38,15 @@ module JsonLogging
|
|
|
38
38
|
if filter
|
|
39
39
|
filter.filter(stringify(hash))
|
|
40
40
|
else
|
|
41
|
+
return Sanitizer.sanitize_primitive_hash(hash) if Sanitizer.primitive_log_hash?(hash)
|
|
42
|
+
|
|
41
43
|
sanitize_without_filter(hash, depth: depth)
|
|
42
44
|
end
|
|
43
45
|
end
|
|
44
46
|
|
|
45
47
|
def sanitize_without_filter(hash, depth: 0)
|
|
46
48
|
jsonable = jsonable_tree(hash, depth: depth)
|
|
47
|
-
|
|
48
|
-
stringify(jsonable.tree)
|
|
49
|
-
else
|
|
50
|
-
jsonable.tree.dup
|
|
51
|
-
end
|
|
49
|
+
stringify(jsonable.tree)
|
|
52
50
|
end
|
|
53
51
|
|
|
54
52
|
def jsonable_tree(hash, depth: 0, seen: nil, parent_key: nil, omit_keys: nil, count_leaves: false, leaf_counter: nil)
|
|
@@ -133,7 +131,8 @@ module JsonLogging
|
|
|
133
131
|
return
|
|
134
132
|
end
|
|
135
133
|
|
|
136
|
-
|
|
134
|
+
# ParameterFilter keeps original key names; fallback rename would hide them from it.
|
|
135
|
+
if Sanitizer.rails_parameter_filter.nil? && Sanitizer::SENSITIVE_KEY_PATTERNS.match?(key_string)
|
|
137
136
|
return [Sanitizer.sensitive_filtered_key_name(key_string), "[FILTERED]", true]
|
|
138
137
|
end
|
|
139
138
|
|
|
@@ -195,12 +194,6 @@ module JsonLogging
|
|
|
195
194
|
value
|
|
196
195
|
end
|
|
197
196
|
end
|
|
198
|
-
|
|
199
|
-
def stringify_keys_copy(hash)
|
|
200
|
-
return hash if hash.keys.all? { |key| key.is_a?(String) }
|
|
201
|
-
|
|
202
|
-
hash.transform_keys(&:to_s)
|
|
203
|
-
end
|
|
204
197
|
end
|
|
205
198
|
end
|
|
206
199
|
end
|
data/lib/json_logging/version.rb
CHANGED
data/lib/json_logging.rb
CHANGED
|
@@ -3,6 +3,7 @@ require "json"
|
|
|
3
3
|
require "time"
|
|
4
4
|
|
|
5
5
|
require_relative "json_logging/version"
|
|
6
|
+
require_relative "json_logging/execution_state"
|
|
6
7
|
require_relative "json_logging/helpers"
|
|
7
8
|
require_relative "json_logging/sanitizer"
|
|
8
9
|
require_relative "json_logging/message_parser"
|
|
@@ -17,15 +18,29 @@ require_relative "json_logging/event_subscriber"
|
|
|
17
18
|
module JsonLogging
|
|
18
19
|
THREAD_CONTEXT_KEY = :__json_logging_context
|
|
19
20
|
SANITIZED_CONTEXT_CACHE_KEY = :__json_logging_sanitized_context
|
|
21
|
+
ISOLATION_LEVELS = [:thread, :fiber].freeze
|
|
22
|
+
|
|
20
23
|
@additional_context_warned = Set.new
|
|
21
24
|
@additional_context_transformer = nil
|
|
25
|
+
@isolation_level = :thread
|
|
22
26
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
27
|
+
# Experimental: +:fiber+ isolates JsonLogging context, tags, and caches per Fiber.
|
|
28
|
+
# Default +:thread+ keeps Rails IsolatedExecutionState / Thread.current behavior.
|
|
29
|
+
def self.isolation_level
|
|
30
|
+
@isolation_level
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def self.isolation_level=(level)
|
|
34
|
+
normalized = level.to_sym
|
|
35
|
+
unless ISOLATION_LEVELS.include?(normalized)
|
|
36
|
+
raise ArgumentError, "isolation_level must be :thread or :fiber, got: #{level.inspect}"
|
|
28
37
|
end
|
|
38
|
+
|
|
39
|
+
@isolation_level = normalized
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def self.context_storage
|
|
43
|
+
ExecutionState
|
|
29
44
|
end
|
|
30
45
|
private_class_method :context_storage
|
|
31
46
|
|
data/sig/json_logging.rbs
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: activesupport-json_logging
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.3.
|
|
4
|
+
version: 1.3.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrei Makarov
|
|
@@ -50,35 +50,21 @@ dependencies:
|
|
|
50
50
|
- !ruby/object:Gem::Version
|
|
51
51
|
version: '9.0'
|
|
52
52
|
- !ruby/object:Gem::Dependency
|
|
53
|
-
name:
|
|
54
|
-
requirement: !ruby/object:Gem::Requirement
|
|
55
|
-
requirements:
|
|
56
|
-
- - "~>"
|
|
57
|
-
- !ruby/object:Gem::Version
|
|
58
|
-
version: '3'
|
|
59
|
-
type: :development
|
|
60
|
-
prerelease: false
|
|
61
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
62
|
-
requirements:
|
|
63
|
-
- - "~>"
|
|
64
|
-
- !ruby/object:Gem::Version
|
|
65
|
-
version: '3'
|
|
66
|
-
- !ruby/object:Gem::Dependency
|
|
67
|
-
name: polyrun
|
|
53
|
+
name: json
|
|
68
54
|
requirement: !ruby/object:Gem::Requirement
|
|
69
55
|
requirements:
|
|
70
56
|
- - ">="
|
|
71
57
|
- !ruby/object:Gem::Version
|
|
72
|
-
version: 2.2
|
|
73
|
-
type: :
|
|
58
|
+
version: 2.21.2
|
|
59
|
+
type: :runtime
|
|
74
60
|
prerelease: false
|
|
75
61
|
version_requirements: !ruby/object:Gem::Requirement
|
|
76
62
|
requirements:
|
|
77
63
|
- - ">="
|
|
78
64
|
- !ruby/object:Gem::Version
|
|
79
|
-
version: 2.2
|
|
65
|
+
version: 2.21.2
|
|
80
66
|
- !ruby/object:Gem::Dependency
|
|
81
|
-
name:
|
|
67
|
+
name: rspec
|
|
82
68
|
requirement: !ruby/object:Gem::Requirement
|
|
83
69
|
requirements:
|
|
84
70
|
- - "~>"
|
|
@@ -92,33 +78,33 @@ dependencies:
|
|
|
92
78
|
- !ruby/object:Gem::Version
|
|
93
79
|
version: '3'
|
|
94
80
|
- !ruby/object:Gem::Dependency
|
|
95
|
-
name:
|
|
81
|
+
name: polyrun
|
|
96
82
|
requirement: !ruby/object:Gem::Requirement
|
|
97
83
|
requirements:
|
|
98
|
-
- - "
|
|
84
|
+
- - ">="
|
|
99
85
|
- !ruby/object:Gem::Version
|
|
100
|
-
version:
|
|
86
|
+
version: 2.2.9
|
|
101
87
|
type: :development
|
|
102
88
|
prerelease: false
|
|
103
89
|
version_requirements: !ruby/object:Gem::Requirement
|
|
104
90
|
requirements:
|
|
105
|
-
- - "
|
|
91
|
+
- - ">="
|
|
106
92
|
- !ruby/object:Gem::Version
|
|
107
|
-
version:
|
|
93
|
+
version: 2.2.9
|
|
108
94
|
- !ruby/object:Gem::Dependency
|
|
109
|
-
name:
|
|
95
|
+
name: rake
|
|
110
96
|
requirement: !ruby/object:Gem::Requirement
|
|
111
97
|
requirements:
|
|
112
98
|
- - "~>"
|
|
113
99
|
- !ruby/object:Gem::Version
|
|
114
|
-
version: '
|
|
100
|
+
version: '13'
|
|
115
101
|
type: :development
|
|
116
102
|
prerelease: false
|
|
117
103
|
version_requirements: !ruby/object:Gem::Requirement
|
|
118
104
|
requirements:
|
|
119
105
|
- - "~>"
|
|
120
106
|
- !ruby/object:Gem::Version
|
|
121
|
-
version: '
|
|
107
|
+
version: '13'
|
|
122
108
|
- !ruby/object:Gem::Dependency
|
|
123
109
|
name: standard
|
|
124
110
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -315,8 +301,8 @@ dependencies:
|
|
|
315
301
|
- - "~>"
|
|
316
302
|
- !ruby/object:Gem::Version
|
|
317
303
|
version: '0.2'
|
|
318
|
-
description:
|
|
319
|
-
|
|
304
|
+
description: JSON logger and formatter for Rails/ActiveSupport. No extra deps beyond
|
|
305
|
+
Rails/ActiveSupport. Compatible with Rails 7-8.
|
|
320
306
|
email:
|
|
321
307
|
- contact@kiskolabs.com
|
|
322
308
|
executables: []
|
|
@@ -330,6 +316,7 @@ files:
|
|
|
330
316
|
- lib/activesupport/json_logging/railtie.rb
|
|
331
317
|
- lib/json_logging.rb
|
|
332
318
|
- lib/json_logging/event_subscriber.rb
|
|
319
|
+
- lib/json_logging/execution_state.rb
|
|
333
320
|
- lib/json_logging/formatter.rb
|
|
334
321
|
- lib/json_logging/formatter_with_tags.rb
|
|
335
322
|
- lib/json_logging/helpers.rb
|
|
@@ -339,6 +326,7 @@ files:
|
|
|
339
326
|
- lib/json_logging/message_parser.rb
|
|
340
327
|
- lib/json_logging/payload_builder.rb
|
|
341
328
|
- lib/json_logging/sanitizer.rb
|
|
329
|
+
- lib/json_logging/sanitizer_hash_limit.rb
|
|
342
330
|
- lib/json_logging/severity.rb
|
|
343
331
|
- lib/json_logging/structured_hash_json_encoder.rb
|
|
344
332
|
- lib/json_logging/structured_hash_sanitizer.rb
|