julewire-redaction 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +6 -0
- data/LICENSE.txt +21 -0
- data/README.md +42 -0
- data/docs/advanced-configuration.md +62 -0
- data/docs/configuration.md +40 -0
- data/docs/failure-semantics.md +15 -0
- data/docs/filter-rules.md +91 -0
- data/docs/string-values.md +37 -0
- data/julewire-redaction.gemspec +37 -0
- data/lib/julewire/redaction/configuration.rb +22 -0
- data/lib/julewire/redaction/matcher.rb +67 -0
- data/lib/julewire/redaction/processor.rb +110 -0
- data/lib/julewire/redaction/string_redactor.rb +83 -0
- data/lib/julewire/redaction/version.rb +7 -0
- data/lib/julewire/redaction.rb +66 -0
- data/lib/julewire-redaction.rb +3 -0
- metadata +89 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: b67efb43089950743fb1e95f25c3086c2a6b7df60be5928df509f21807313d57
|
|
4
|
+
data.tar.gz: 73e0a830b9d4a325df1095f7a043caf3f7e80622caf0b6c23ad8f02d56d2a415
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 832b8609e996a5bd201cf1c3418a0c02677b38bdc6623b8ee0dc85bacf56a37005a9929537c1e385ac667aabb606842b4cd4bee0bc663d1d32201eb2ffe68d0b
|
|
7
|
+
data.tar.gz: a8aee3f3baab8ba0e407c0124f30ddb1646d00a57631ec77712454fd2c74a9edf4542bd1fb14aa0b3f61f4c495bfe966b994a23852da513bd96e131ddd7142c3
|
data/CHANGELOG.md
ADDED
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Alexander Grebennik
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Julewire Redaction
|
|
2
|
+
|
|
3
|
+
`julewire-redaction` is a structured redaction processor for Julewire records.
|
|
4
|
+
|
|
5
|
+
It redacts data inside the core processor pipeline before records are frozen,
|
|
6
|
+
formatted, encoded, and written.
|
|
7
|
+
|
|
8
|
+
## Install
|
|
9
|
+
|
|
10
|
+
```ruby
|
|
11
|
+
gem "julewire-redaction"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Quickstart
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
Julewire::Redaction.configure do |config|
|
|
18
|
+
config.filters = Julewire::Redaction::DEFAULT_FILTERS + %i[session_id]
|
|
19
|
+
config.mask = "[FILTERED]"
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
Julewire.configure do |config|
|
|
23
|
+
config.processors.prepend :redaction, on_error: :fail_closed
|
|
24
|
+
end
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Configure redaction before `Julewire.configure`, or pass explicit options to
|
|
28
|
+
the processor registration.
|
|
29
|
+
|
|
30
|
+
`DEFAULT_FILTERS` combines `SECRET_FILTERS` and `PII_FILTERS`; use the narrower
|
|
31
|
+
profile when fields such as email are acceptable in your logs.
|
|
32
|
+
|
|
33
|
+
Prepend the processor when redaction should run before enrichment and output
|
|
34
|
+
formatting.
|
|
35
|
+
|
|
36
|
+
## Docs
|
|
37
|
+
|
|
38
|
+
- [Configuration](docs/configuration.md)
|
|
39
|
+
- [Advanced Configuration](docs/advanced-configuration.md)
|
|
40
|
+
- [Filter Rules](docs/filter-rules.md)
|
|
41
|
+
- [String Values](docs/string-values.md)
|
|
42
|
+
- [Failure Semantics](docs/failure-semantics.md)
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Advanced Configuration
|
|
2
|
+
|
|
3
|
+
## Traversal Limits
|
|
4
|
+
|
|
5
|
+
The processor bounds its walk before the final core serializer runs.
|
|
6
|
+
When a redaction walk hits these limits, it may truncate values and add
|
|
7
|
+
`_julewire_truncation` metadata before later processors or destinations see the
|
|
8
|
+
affected field or section.
|
|
9
|
+
|
|
10
|
+
| Setting | Default |
|
|
11
|
+
| --- | --- |
|
|
12
|
+
| `max_depth` | `Julewire::Serializer::DEFAULT_MAX_DEPTH` |
|
|
13
|
+
| `max_hash_keys` | `Julewire::Serializer::DEFAULT_MAX_HASH_KEYS` |
|
|
14
|
+
| `max_array_items` | `Julewire::Serializer::DEFAULT_MAX_ARRAY_ITEMS` |
|
|
15
|
+
| `max_string_bytes` | `Julewire::Serializer::DEFAULT_MAX_STRING_BYTES` |
|
|
16
|
+
|
|
17
|
+
Override these only when the redaction walk must be tighter or looser than
|
|
18
|
+
core's default serializer bounds:
|
|
19
|
+
|
|
20
|
+
```ruby
|
|
21
|
+
Julewire::Redaction.configure do |config|
|
|
22
|
+
config.max_depth = 8
|
|
23
|
+
config.max_hash_keys = 1_000
|
|
24
|
+
config.max_array_items = 1_000
|
|
25
|
+
config.max_string_bytes = 16_384
|
|
26
|
+
end
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
`max_depth` must be positive. The other limits must be non-negative integers.
|
|
30
|
+
|
|
31
|
+
## Per-Processor Options
|
|
32
|
+
|
|
33
|
+
Global configuration is only the default. A processor registration can override
|
|
34
|
+
the same options:
|
|
35
|
+
|
|
36
|
+
```ruby
|
|
37
|
+
Julewire.configure do |config|
|
|
38
|
+
config.processors.prepend(
|
|
39
|
+
:redaction,
|
|
40
|
+
%i[password api_key],
|
|
41
|
+
mask: "[SECRET]",
|
|
42
|
+
string_values: false,
|
|
43
|
+
on_error: :fail_closed
|
|
44
|
+
)
|
|
45
|
+
end
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Registration options:
|
|
49
|
+
|
|
50
|
+
```ruby
|
|
51
|
+
config.processors.use(
|
|
52
|
+
:redaction,
|
|
53
|
+
filters,
|
|
54
|
+
mask: "[FILTERED]",
|
|
55
|
+
max_array_items: 1_000,
|
|
56
|
+
max_depth: 8,
|
|
57
|
+
max_hash_keys: 1_000,
|
|
58
|
+
max_string_bytes: 16_384,
|
|
59
|
+
string_values: false,
|
|
60
|
+
authorization_header: true
|
|
61
|
+
)
|
|
62
|
+
```
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Configuration
|
|
2
|
+
|
|
3
|
+
`Julewire::Redaction.configure` changes the defaults used by the registered
|
|
4
|
+
`:redaction` processor. Configure redaction before building the core pipeline,
|
|
5
|
+
or pass explicit options to `config.processors.use`.
|
|
6
|
+
|
|
7
|
+
```ruby
|
|
8
|
+
Julewire::Redaction.configure do |config|
|
|
9
|
+
config.filters = Julewire::Redaction::DEFAULT_FILTERS + %i[session_id]
|
|
10
|
+
config.mask = "[FILTERED]"
|
|
11
|
+
config.string_values = false
|
|
12
|
+
end
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Defaults
|
|
16
|
+
|
|
17
|
+
| Setting | Default | Meaning |
|
|
18
|
+
| --- | --- | --- |
|
|
19
|
+
| `filters` | `Julewire::Redaction::DEFAULT_FILTERS` | Structured keys to redact. |
|
|
20
|
+
| `mask` | `"[FILTERED]"` | Replacement value. |
|
|
21
|
+
| `string_values` | `false` | Also scrub embedded secrets in string values. |
|
|
22
|
+
| `authorization_header` | `true` | Always redact `Authorization:` header lines when string scrubbing is enabled. |
|
|
23
|
+
|
|
24
|
+
`DEFAULT_FILTERS` combines the secret and PII profiles:
|
|
25
|
+
|
|
26
|
+
- auth keys: `access_token`, `refresh_token`, `id_token`, `client_secret`,
|
|
27
|
+
`assertion`, `code_verifier`, `token`, `authorization`, `cookie`,
|
|
28
|
+
`set_cookie`, `x_api_key`, `set-cookie`, `x-api-key`
|
|
29
|
+
- common secret keys: `api_key`, `password`, `passwd`, `private_key`, `secret`
|
|
30
|
+
- extra secret keys: `crypt`, `salt`, `certificate`, `otp`, `cvv`, `cvc`
|
|
31
|
+
- PII keys: `email`, `ssn`
|
|
32
|
+
|
|
33
|
+
Use `Julewire::Redaction::SECRET_FILTERS` without
|
|
34
|
+
`Julewire::Redaction::PII_FILTERS` when PII fields are allowed in your logs.
|
|
35
|
+
|
|
36
|
+
String and symbol filters are case-insensitive exact matches. Use regexes for
|
|
37
|
+
partial matching.
|
|
38
|
+
|
|
39
|
+
For traversal limits and per-processor overrides, see
|
|
40
|
+
[Advanced Configuration](advanced-configuration.md).
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Failure Semantics
|
|
2
|
+
|
|
3
|
+
Redaction is fail-closed at the record boundary.
|
|
4
|
+
|
|
5
|
+
Register redaction with core's default `on_error: :fail_closed` policy. If a
|
|
6
|
+
custom filter proc raises, the processor raises. Core treats that as a processor
|
|
7
|
+
failure and does not emit the original unredacted record. Core records the
|
|
8
|
+
failure in health and emits its contained failure record when possible.
|
|
9
|
+
|
|
10
|
+
The failure is record-level, not field-level. A failing custom filter loses the
|
|
11
|
+
record rather than keeping the record with only that one field unredacted.
|
|
12
|
+
|
|
13
|
+
The processor receives a `Julewire::RecordDraft` and applies its
|
|
14
|
+
whole-record transform through `transform_record!`, so cache invalidation and
|
|
15
|
+
lineage preservation stay inside core.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Filter Rules
|
|
2
|
+
|
|
3
|
+
The redaction processor walks the normalized Julewire record and replaces
|
|
4
|
+
matching values.
|
|
5
|
+
|
|
6
|
+
Matching applies wherever the key appears, including `execution`, `context`,
|
|
7
|
+
`carry`, `neutral`, `attributes`, `labels`, `payload`, `metrics`, and `error`.
|
|
8
|
+
Top-level record container fields keep their required Julewire shape even if a
|
|
9
|
+
filter matches the section name.
|
|
10
|
+
|
|
11
|
+
## Exact Keys
|
|
12
|
+
|
|
13
|
+
String and symbol filters are exact, case-insensitive key matches:
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
Julewire::Redaction.configure do |config|
|
|
17
|
+
config.filters = %i[password api_key]
|
|
18
|
+
end
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
This redacts `password` and `api_key`, but not `old_password_hash` unless that
|
|
22
|
+
key is listed or matched by a regex.
|
|
23
|
+
|
|
24
|
+
## Nested Keys
|
|
25
|
+
|
|
26
|
+
Dot notation scopes a filter to a nested structured path:
|
|
27
|
+
|
|
28
|
+
```ruby
|
|
29
|
+
Julewire::Redaction.configure do |config|
|
|
30
|
+
config.filters = ["credit_card.code"]
|
|
31
|
+
end
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
This redacts:
|
|
35
|
+
|
|
36
|
+
```ruby
|
|
37
|
+
{ credit_card: { code: "123" } }
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
It does not redact:
|
|
41
|
+
|
|
42
|
+
```ruby
|
|
43
|
+
{ file: { code: "123" } }
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Regex Rules
|
|
47
|
+
|
|
48
|
+
Use regexes for deliberate partial matching:
|
|
49
|
+
|
|
50
|
+
```ruby
|
|
51
|
+
Julewire::Redaction.configure do |config|
|
|
52
|
+
config.filters = Julewire::Redaction::DEFAULT_FILTERS + [/password|secret/i]
|
|
53
|
+
end
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Regexes match leaf keys. Use `Julewire::Redaction.path(...)` for path-aware
|
|
57
|
+
regexes that match a dotted structured path:
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
Julewire::Redaction.configure do |config|
|
|
61
|
+
config.filters = [Julewire::Redaction.path(/user.email/i)]
|
|
62
|
+
end
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Use `Julewire::Redaction::SECRET_FILTERS` when the default PII profile is too
|
|
66
|
+
broad for your application.
|
|
67
|
+
|
|
68
|
+
## Proc Rules
|
|
69
|
+
|
|
70
|
+
Proc filters follow Rails' shape:
|
|
71
|
+
|
|
72
|
+
- 2-arity procs receive `key, value`
|
|
73
|
+
- 3-arity procs receive `key, value, original_record`
|
|
74
|
+
|
|
75
|
+
`original_record` is the root normalized record hash.
|
|
76
|
+
|
|
77
|
+
```ruby
|
|
78
|
+
Julewire::Redaction.configure do |config|
|
|
79
|
+
config.filters = [
|
|
80
|
+
lambda do |key, value|
|
|
81
|
+
value.replace("[FILTERED]") if key == "customer_note" && value.include?("ssn=")
|
|
82
|
+
end
|
|
83
|
+
]
|
|
84
|
+
end
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Proc rules mutate a duplicate of string scalar values. If a proc raises, the
|
|
88
|
+
record is not emitted.
|
|
89
|
+
|
|
90
|
+
String filters with dots are path-aware. Raw regexes are key matchers unless
|
|
91
|
+
wrapped with `Julewire::Redaction.path(...)`.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# String Values
|
|
2
|
+
|
|
3
|
+
Structured key/value redaction is the primary path.
|
|
4
|
+
|
|
5
|
+
`string_values` defaults to `false`. When enabled, the processor also tries to
|
|
6
|
+
scrub embedded secrets inside string values.
|
|
7
|
+
|
|
8
|
+
```ruby
|
|
9
|
+
Julewire::Redaction.configure do |config|
|
|
10
|
+
config.string_values = true
|
|
11
|
+
end
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The string scrubber handles common opaque-string shapes:
|
|
15
|
+
|
|
16
|
+
- header lines such as `Authorization:`, `Cookie:`, `Set-Cookie:`, and
|
|
17
|
+
`X-Api-Key:`
|
|
18
|
+
- JSON-like string pairs such as `"access_token": "abc"`
|
|
19
|
+
- query/form pairs such as `access_token=abc&scope=read`
|
|
20
|
+
|
|
21
|
+
This is defense-in-depth. It is not a JSON parser and it does not replace
|
|
22
|
+
structured key/value redaction.
|
|
23
|
+
|
|
24
|
+
String scrubbing scans the full string value before core output truncation. It
|
|
25
|
+
is therefore proportional to captured body size. Avoid pairing
|
|
26
|
+
`string_values: true` with unbounded request or response body capture unless the
|
|
27
|
+
volume and body sizes are known to be small.
|
|
28
|
+
|
|
29
|
+
Regexp filters disable the cheap string-key pre-scan, so every colon/equal style
|
|
30
|
+
string is checked against the pair scrubbers when `string_values` is enabled.
|
|
31
|
+
|
|
32
|
+
The matcher intentionally covers only simple string patterns. Escaped quotes,
|
|
33
|
+
unquoted or numeric JSON values, multiline serialized data, and custom formats
|
|
34
|
+
can remain unchanged unless the surrounding structured key is also redacted.
|
|
35
|
+
|
|
36
|
+
When `authorization_header` is true, `Authorization:` header lines are redacted
|
|
37
|
+
even if `authorization` is not present in the filter list.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "lib/julewire/redaction/version"
|
|
4
|
+
|
|
5
|
+
Gem::Specification.new do |spec|
|
|
6
|
+
spec.name = "julewire-redaction"
|
|
7
|
+
spec.version = Julewire::Redaction::VERSION
|
|
8
|
+
spec.authors = ["Alexander Grebennik"]
|
|
9
|
+
spec.email = ["slbug@users.noreply.github.com", "sl.bug.sl@gmail.com"]
|
|
10
|
+
|
|
11
|
+
spec.summary = "Structured redaction processor for Julewire."
|
|
12
|
+
spec.description = "Structured key/value and string redaction processor for Julewire records."
|
|
13
|
+
spec.homepage = "https://github.com/slbug/julewire"
|
|
14
|
+
spec.license = "MIT"
|
|
15
|
+
spec.required_ruby_version = ">= 3.4"
|
|
16
|
+
spec.metadata["homepage_uri"] = spec.homepage
|
|
17
|
+
spec.metadata["source_code_uri"] = "https://github.com/slbug/julewire/tree/main/gems/redaction"
|
|
18
|
+
spec.metadata["changelog_uri"] = "https://github.com/slbug/julewire/blob/main/gems/redaction/CHANGELOG.md"
|
|
19
|
+
|
|
20
|
+
spec.metadata["rubygems_mfa_required"] = "true"
|
|
21
|
+
|
|
22
|
+
spec.files = Dir.chdir(__dir__) do
|
|
23
|
+
Dir[
|
|
24
|
+
"CHANGELOG.md",
|
|
25
|
+
"LICENSE.txt",
|
|
26
|
+
"README.md",
|
|
27
|
+
"docs/**/*.md",
|
|
28
|
+
"julewire-redaction.gemspec",
|
|
29
|
+
"lib/**/*.rb"
|
|
30
|
+
]
|
|
31
|
+
end
|
|
32
|
+
spec.executables = []
|
|
33
|
+
spec.require_paths = ["lib"]
|
|
34
|
+
|
|
35
|
+
spec.add_dependency "julewire-core", ">= 1.0"
|
|
36
|
+
spec.add_dependency "zeitwerk", ">= 2.8.1"
|
|
37
|
+
end
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Julewire
|
|
4
|
+
module Redaction
|
|
5
|
+
class Configuration
|
|
6
|
+
include Julewire::Core::Integration::Settings
|
|
7
|
+
|
|
8
|
+
setting :authorization_header, default: true
|
|
9
|
+
setting :filters, default: DEFAULT_FILTERS
|
|
10
|
+
setting :mask, default: DEFAULT_MASK
|
|
11
|
+
setting :max_array_items, default: Julewire::Serializer::DEFAULT_MAX_ARRAY_ITEMS,
|
|
12
|
+
validate: integer_limit
|
|
13
|
+
setting :max_depth, default: Julewire::Serializer::DEFAULT_MAX_DEPTH,
|
|
14
|
+
validate: integer_limit(positive: true)
|
|
15
|
+
setting :max_hash_keys, default: Julewire::Serializer::DEFAULT_MAX_HASH_KEYS,
|
|
16
|
+
validate: integer_limit
|
|
17
|
+
setting :max_string_bytes, default: Julewire::Serializer::DEFAULT_MAX_STRING_BYTES,
|
|
18
|
+
validate: integer_limit
|
|
19
|
+
setting :string_values, default: false
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Julewire
|
|
4
|
+
module Redaction
|
|
5
|
+
class Matcher
|
|
6
|
+
attr_reader :blocks
|
|
7
|
+
|
|
8
|
+
def initialize(filters)
|
|
9
|
+
@blocks, patterns = Array(filters).partition { it.is_a?(Proc) }
|
|
10
|
+
normal_filters, deep_filters = patterns.partition { !deep_filter?(it) }
|
|
11
|
+
@pattern = compile(normal_filters, deep: false)
|
|
12
|
+
@deep_pattern = compile(deep_filters, deep: true)
|
|
13
|
+
@string_scan_pattern = string_scan_pattern(normal_filters)
|
|
14
|
+
@blocks.freeze
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def match?(key, path: nil)
|
|
18
|
+
key_string = key.to_s
|
|
19
|
+
!!(pattern_match?(@pattern, key_string) || pattern_match?(@deep_pattern, path))
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def empty? = @pattern.nil? && @deep_pattern.nil?
|
|
23
|
+
|
|
24
|
+
def path_dependent? = !@deep_pattern.nil?
|
|
25
|
+
|
|
26
|
+
def string_key_possible?(value)
|
|
27
|
+
return true unless @string_scan_pattern
|
|
28
|
+
|
|
29
|
+
@string_scan_pattern.match?(value)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
private
|
|
33
|
+
|
|
34
|
+
def compile(filters, deep:)
|
|
35
|
+
patterns = filters.map do |filter|
|
|
36
|
+
filter = filter.filter if filter.instance_of?(PathFilter)
|
|
37
|
+
case filter
|
|
38
|
+
when Regexp
|
|
39
|
+
filter
|
|
40
|
+
else
|
|
41
|
+
escaped = Regexp.escape(filter.to_s)
|
|
42
|
+
pattern = deep ? "(?:\\A|\\.)#{escaped}\\z" : "\\A#{escaped}\\z"
|
|
43
|
+
Regexp.new(pattern, Regexp::IGNORECASE)
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
Regexp.union(patterns) unless patterns.empty?
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def string_scan_pattern(filters)
|
|
50
|
+
return if filters.any?(Regexp)
|
|
51
|
+
|
|
52
|
+
Regexp.new(filters.map { Regexp.escape(it.to_s) }.join("|"), Regexp::IGNORECASE)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def pattern_match?(pattern, value)
|
|
56
|
+
pattern&.match?(value)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def deep_filter?(filter)
|
|
60
|
+
return true if filter.instance_of?(PathFilter)
|
|
61
|
+
return false if filter.instance_of?(Regexp)
|
|
62
|
+
|
|
63
|
+
filter.to_s.include?(".")
|
|
64
|
+
end
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
end
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Julewire
|
|
4
|
+
module Redaction
|
|
5
|
+
class Processor
|
|
6
|
+
PRESERVED_TOP_LEVEL_KEYS = (
|
|
7
|
+
Core::Fields::Bags.record_scalar_keys - %i[message]
|
|
8
|
+
).freeze
|
|
9
|
+
private_constant :PRESERVED_TOP_LEVEL_KEYS
|
|
10
|
+
|
|
11
|
+
def initialize(
|
|
12
|
+
filters = Redaction.config.filters,
|
|
13
|
+
mask: Redaction.config.mask,
|
|
14
|
+
max_array_items: Redaction.config.max_array_items,
|
|
15
|
+
max_depth: Redaction.config.max_depth,
|
|
16
|
+
max_hash_keys: Redaction.config.max_hash_keys,
|
|
17
|
+
max_string_bytes: Redaction.config.max_string_bytes,
|
|
18
|
+
string_values: Redaction.config.string_values,
|
|
19
|
+
authorization_header: Redaction.config.authorization_header
|
|
20
|
+
)
|
|
21
|
+
@matcher = Matcher.new(filters)
|
|
22
|
+
@blocks = @matcher.blocks
|
|
23
|
+
@mask = mask.to_s
|
|
24
|
+
@max_array_items = Core::Validation.validate_integer_limit!(max_array_items, name: :max_array_items)
|
|
25
|
+
@max_depth = Core::Validation.validate_integer_limit!(max_depth, name: :max_depth, positive: true)
|
|
26
|
+
@max_hash_keys = Core::Validation.validate_integer_limit!(max_hash_keys, name: :max_hash_keys)
|
|
27
|
+
@max_string_bytes = Core::Validation.validate_integer_limit!(max_string_bytes, name: :max_string_bytes)
|
|
28
|
+
@string_redactor = if string_values
|
|
29
|
+
StringRedactor.new(
|
|
30
|
+
matcher: @matcher,
|
|
31
|
+
mask: @mask,
|
|
32
|
+
authorization_header: authorization_header
|
|
33
|
+
)
|
|
34
|
+
end
|
|
35
|
+
@redact_keys = !@matcher.empty?
|
|
36
|
+
@redact_scalars = @string_redactor || !@blocks.empty?
|
|
37
|
+
@enabled = @redact_keys || @redact_scalars
|
|
38
|
+
@record_transform = Core::Processing::RecordFieldTransform.new(
|
|
39
|
+
max_array_items: @max_array_items,
|
|
40
|
+
max_depth: @max_depth,
|
|
41
|
+
max_hash_keys: @max_hash_keys,
|
|
42
|
+
max_string_bytes: @max_string_bytes,
|
|
43
|
+
preserve_top_level_keys: PRESERVED_TOP_LEVEL_KEYS,
|
|
44
|
+
track_paths: @matcher.path_dependent?
|
|
45
|
+
)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def call(draft)
|
|
49
|
+
validate_draft!(draft)
|
|
50
|
+
return draft unless @enabled
|
|
51
|
+
|
|
52
|
+
draft.transform_record! { redact_record(it) }
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
private
|
|
56
|
+
|
|
57
|
+
def validate_draft!(draft)
|
|
58
|
+
return if draft.instance_of?(Julewire::RecordDraft)
|
|
59
|
+
|
|
60
|
+
raise TypeError, "expected Julewire::RecordDraft"
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def redact_record(record)
|
|
64
|
+
@record_transform.call(record) do |item, key:, path:, prefixed_path:, original:, **|
|
|
65
|
+
redact_item(
|
|
66
|
+
item,
|
|
67
|
+
key: key,
|
|
68
|
+
path: path,
|
|
69
|
+
prefixed_path: prefixed_path,
|
|
70
|
+
original: original
|
|
71
|
+
)
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def redact_item(item, key:, path:, original:, prefixed_path:, **)
|
|
76
|
+
return @mask if redacted_key?(key, path: path, prefixed_path: prefixed_path)
|
|
77
|
+
if @redact_scalars && !item.is_a?(Hash) && !item.is_a?(Array)
|
|
78
|
+
return redact_scalar(item, key: key, original: original)
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
Core::Serialization::BoundedTransform::CONTINUE
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def redacted_key?(key, path:, prefixed_path:)
|
|
85
|
+
return false unless @redact_keys && key
|
|
86
|
+
return true if @matcher.match?(key, path: path)
|
|
87
|
+
|
|
88
|
+
prefixed_path && @matcher.match?(key, path: prefixed_path)
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def redact_scalar(value, key:, original:)
|
|
92
|
+
value = apply_block_filters(key, value, original) if key && !@blocks.empty?
|
|
93
|
+
value.is_a?(String) && @string_redactor ? @string_redactor.call(value) : value
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def apply_block_filters(key, value, original)
|
|
97
|
+
key_copy = key.to_s.dup
|
|
98
|
+
value_copy = duplicate_filter_value(value)
|
|
99
|
+
@blocks.each do |block|
|
|
100
|
+
block.arity == 2 ? block.call(key_copy, value_copy) : block.call(key_copy, value_copy, original)
|
|
101
|
+
end
|
|
102
|
+
value_copy
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def duplicate_filter_value(value)
|
|
106
|
+
value.is_a?(String) ? value.dup : value
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
end
|
|
110
|
+
end
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Julewire
|
|
4
|
+
module Redaction
|
|
5
|
+
class StringRedactor
|
|
6
|
+
def initialize(matcher:, mask:, authorization_header: true)
|
|
7
|
+
@matcher = matcher
|
|
8
|
+
@mask = mask.to_s
|
|
9
|
+
@authorization_header = authorization_header
|
|
10
|
+
@redact_headers = @authorization_header || !@matcher.empty?
|
|
11
|
+
@redact_pairs = !@matcher.empty?
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def call(value)
|
|
15
|
+
return value unless value.is_a?(String)
|
|
16
|
+
|
|
17
|
+
has_colon = value.include?(":")
|
|
18
|
+
has_equals = value.include?("=")
|
|
19
|
+
return value unless redaction_possible?(has_colon, has_equals)
|
|
20
|
+
|
|
21
|
+
redacted = value.dup
|
|
22
|
+
redact_header_lines!(redacted) if header_redaction_possible?(has_colon)
|
|
23
|
+
return redacted unless @redact_pairs
|
|
24
|
+
return redacted unless @matcher.string_key_possible?(value)
|
|
25
|
+
|
|
26
|
+
redact_json_pairs!(redacted) if json_pair_possible?(value, has_colon)
|
|
27
|
+
redact_form_pairs!(redacted) if form_pair_possible?(has_equals)
|
|
28
|
+
redacted
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
private
|
|
32
|
+
|
|
33
|
+
def redaction_possible?(has_colon, has_equals)
|
|
34
|
+
header_redaction_possible?(has_colon) || (@redact_pairs && (has_colon || has_equals))
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def header_redaction_possible?(has_colon)
|
|
38
|
+
@redact_headers && has_colon
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def json_pair_possible?(value, has_colon)
|
|
42
|
+
@redact_pairs && has_colon && (value.include?('"') || value.include?("'"))
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def form_pair_possible?(has_equals)
|
|
46
|
+
@redact_pairs && has_equals
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def redact_header_lines!(value)
|
|
50
|
+
value.gsub!(/(^|[\r\n])([A-Za-z0-9-]+:\s*)(?:"[^"\r\n]*"|[^\r\n]*)/i) do |line|
|
|
51
|
+
name = Regexp.last_match(2).split(":", 2).fetch(0)
|
|
52
|
+
next line unless redact_header?(name)
|
|
53
|
+
|
|
54
|
+
"#{Regexp.last_match(1)}#{Regexp.last_match(2)}#{@mask}"
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
def redact_header?(name)
|
|
59
|
+
return true if @authorization_header && name.casecmp?("authorization")
|
|
60
|
+
|
|
61
|
+
@matcher.match?(name) || @matcher.match?(name.tr("-", "_"))
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
def redact_json_pairs!(value)
|
|
65
|
+
# Defense-in-depth for short log strings; large bodies should stay bounded.
|
|
66
|
+
value.gsub!(/(["'])([^"']+)\1(\s*:\s*)(["'])(.*?)\4/i) do |pair|
|
|
67
|
+
next pair unless @matcher.match?(Regexp.last_match(2))
|
|
68
|
+
|
|
69
|
+
"#{Regexp.last_match(1)}#{Regexp.last_match(2)}#{Regexp.last_match(1)}" \
|
|
70
|
+
"#{Regexp.last_match(3)}#{Regexp.last_match(4)}#{@mask}#{Regexp.last_match(4)}"
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def redact_form_pairs!(value)
|
|
75
|
+
value.gsub!(/(^|[?&\s])([^?&\s"=]+)(=)([^&\s"]+)/i) do |pair|
|
|
76
|
+
next pair unless @matcher.match?(Regexp.last_match(2))
|
|
77
|
+
|
|
78
|
+
"#{Regexp.last_match(1)}#{Regexp.last_match(2)}#{Regexp.last_match(3)}#{@mask}"
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
end
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "zeitwerk"
|
|
4
|
+
require "julewire/core"
|
|
5
|
+
|
|
6
|
+
module Julewire
|
|
7
|
+
module Redaction
|
|
8
|
+
# Header names may arrive as normalized symbols or literal HTTP spellings.
|
|
9
|
+
AUTH_FILTERS = %i[
|
|
10
|
+
access_token
|
|
11
|
+
refresh_token
|
|
12
|
+
id_token
|
|
13
|
+
client_secret
|
|
14
|
+
assertion
|
|
15
|
+
code_verifier
|
|
16
|
+
token
|
|
17
|
+
authorization
|
|
18
|
+
cookie
|
|
19
|
+
set_cookie
|
|
20
|
+
x_api_key
|
|
21
|
+
set-cookie
|
|
22
|
+
x-api-key
|
|
23
|
+
].freeze
|
|
24
|
+
|
|
25
|
+
COMMON_FILTERS = %i[
|
|
26
|
+
api_key
|
|
27
|
+
password
|
|
28
|
+
passwd
|
|
29
|
+
private_key
|
|
30
|
+
secret
|
|
31
|
+
].freeze
|
|
32
|
+
|
|
33
|
+
SECRET_FILTERS = (AUTH_FILTERS + COMMON_FILTERS + %i[
|
|
34
|
+
crypt
|
|
35
|
+
salt
|
|
36
|
+
certificate
|
|
37
|
+
otp
|
|
38
|
+
cvv
|
|
39
|
+
cvc
|
|
40
|
+
]).uniq.freeze
|
|
41
|
+
|
|
42
|
+
PII_FILTERS = %i[
|
|
43
|
+
email
|
|
44
|
+
ssn
|
|
45
|
+
].freeze
|
|
46
|
+
|
|
47
|
+
DEFAULT_FILTERS = (SECRET_FILTERS + PII_FILTERS).uniq.freeze
|
|
48
|
+
DEFAULT_MASK = "[FILTERED]"
|
|
49
|
+
|
|
50
|
+
PathFilter = Data.define(:filter)
|
|
51
|
+
|
|
52
|
+
class << self
|
|
53
|
+
def path(filter)
|
|
54
|
+
PathFilter.new(filter)
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
extend Core::Integration::Configurable
|
|
59
|
+
|
|
60
|
+
configurable_with { Configuration }
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
loader = Zeitwerk::Loader.for_gem_extension(self)
|
|
64
|
+
loader.setup
|
|
65
|
+
Core::Processing.register(:redaction) { |*args, **options| Redaction::Processor.new(*args, **options) }
|
|
66
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: julewire-redaction
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 1.0.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Alexander Grebennik
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: julewire-core
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '1.0'
|
|
19
|
+
type: :runtime
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - ">="
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '1.0'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: zeitwerk
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - ">="
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: 2.8.1
|
|
33
|
+
type: :runtime
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - ">="
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: 2.8.1
|
|
40
|
+
description: Structured key/value and string redaction processor for Julewire records.
|
|
41
|
+
email:
|
|
42
|
+
- slbug@users.noreply.github.com
|
|
43
|
+
- sl.bug.sl@gmail.com
|
|
44
|
+
executables: []
|
|
45
|
+
extensions: []
|
|
46
|
+
extra_rdoc_files: []
|
|
47
|
+
files:
|
|
48
|
+
- CHANGELOG.md
|
|
49
|
+
- LICENSE.txt
|
|
50
|
+
- README.md
|
|
51
|
+
- docs/advanced-configuration.md
|
|
52
|
+
- docs/configuration.md
|
|
53
|
+
- docs/failure-semantics.md
|
|
54
|
+
- docs/filter-rules.md
|
|
55
|
+
- docs/string-values.md
|
|
56
|
+
- julewire-redaction.gemspec
|
|
57
|
+
- lib/julewire-redaction.rb
|
|
58
|
+
- lib/julewire/redaction.rb
|
|
59
|
+
- lib/julewire/redaction/configuration.rb
|
|
60
|
+
- lib/julewire/redaction/matcher.rb
|
|
61
|
+
- lib/julewire/redaction/processor.rb
|
|
62
|
+
- lib/julewire/redaction/string_redactor.rb
|
|
63
|
+
- lib/julewire/redaction/version.rb
|
|
64
|
+
homepage: https://github.com/slbug/julewire
|
|
65
|
+
licenses:
|
|
66
|
+
- MIT
|
|
67
|
+
metadata:
|
|
68
|
+
homepage_uri: https://github.com/slbug/julewire
|
|
69
|
+
source_code_uri: https://github.com/slbug/julewire/tree/main/gems/redaction
|
|
70
|
+
changelog_uri: https://github.com/slbug/julewire/blob/main/gems/redaction/CHANGELOG.md
|
|
71
|
+
rubygems_mfa_required: 'true'
|
|
72
|
+
rdoc_options: []
|
|
73
|
+
require_paths:
|
|
74
|
+
- lib
|
|
75
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
76
|
+
requirements:
|
|
77
|
+
- - ">="
|
|
78
|
+
- !ruby/object:Gem::Version
|
|
79
|
+
version: '3.4'
|
|
80
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
81
|
+
requirements:
|
|
82
|
+
- - ">="
|
|
83
|
+
- !ruby/object:Gem::Version
|
|
84
|
+
version: '0'
|
|
85
|
+
requirements: []
|
|
86
|
+
rubygems_version: 4.0.14
|
|
87
|
+
specification_version: 4
|
|
88
|
+
summary: Structured redaction processor for Julewire.
|
|
89
|
+
test_files: []
|