data_redactor 0.17.0 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +122 -1
- data/README.md +135 -46
- data/ext/data_redactor/matcher.c +53 -26
- data/lib/data_redactor/integrations/logger.rb +11 -1
- data/lib/data_redactor/integrations/ruby_llm.rb +152 -88
- data/lib/data_redactor/railtie.rb +91 -0
- data/lib/data_redactor/version.rb +1 -1
- data/lib/data_redactor.rb +108 -7
- metadata +17 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6c5247f79af00e150cb2fb5c688533dd4f91c6797e0958d894032efb766a1230
|
|
4
|
+
data.tar.gz: bee5d8338d43fd4f373e6fc06c7ca2402c2bbfa64a3109a31d8248a8a38a9c07
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 85ed112087268b8046b7d3131c1bcbabc4e7a3f69ccccc2f96e399f8b6fd4fd13f253cf0d48699606b39509a7d5df801eb2e8104fc590c15a7ec1bee814edc29
|
|
7
|
+
data.tar.gz: fcd438b03fd793200ff4f42e4f7a5ecc7437c4609c53c92ecf1869276ee2232e86cbc8e0b9075e32d526aba4bad96b64dfaaf3d977ae3740a26907829f67a671
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,126 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.18.0] - 2026-09-10
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **RubyLLM integration rebuilt on the public request hook.**
|
|
14
|
+
`Integrations::RubyLLM.chat(...)` is a drop-in for `RubyLLM.chat` that returns a
|
|
15
|
+
chat whose every request is redacted; `attach!(target)` does the same for a chat,
|
|
16
|
+
an `Agent`, or an `acts_as_chat` record you were handed (it finds the chat inside
|
|
17
|
+
whichever you pass); `hook(...)` returns the callback itself. All three sit on
|
|
18
|
+
`Chat#before_request` (ruby_llm 2.0+), which receives the fully rendered payload
|
|
19
|
+
and lets a callback edit it in place — **no monkeypatching**. This is the only way
|
|
20
|
+
to scrub tool results, which an agent inlines into the next request and the user
|
|
21
|
+
never typed. Redaction is per chat and applies to the rendered payload only, so
|
|
22
|
+
the stored conversation is untouched. RubyLLM runs the hooks on conversation
|
|
23
|
+
**compaction** requests too (`Protocol#compact`), so the history a summarisation
|
|
24
|
+
call sends is redacted as well.
|
|
25
|
+
- **`skip_keys:` on `redact_deep`, `redact_deep!` and `redact_json`.** Hash keys
|
|
26
|
+
whose values are left verbatim, matched by name at any depth (Symbol and String
|
|
27
|
+
keys are equivalent) and skipping the key's whole subtree. For structural fields
|
|
28
|
+
a receiving API validates. It is a denylist, not a filter — anything unlisted is
|
|
29
|
+
still redacted, so an unfamiliar field cannot slip through. The RubyLLM hook
|
|
30
|
+
defaults to `skip_keys: [:model]`: dated model ids such as
|
|
31
|
+
`claude-haiku-4-5-20251001` end in eight digits, which the national-ID patterns
|
|
32
|
+
match, and a provider rejects a redacted model id.
|
|
33
|
+
- **`DataRedactor.redact_deep!` — in-place deep redaction.** The sibling of
|
|
34
|
+
`redact_deep` for callers that must scrub the structure they were handed rather
|
|
35
|
+
than return a copy: hooks and middleware whose return value is discarded (a
|
|
36
|
+
`ruby_llm` `before_request` hook is the motivating case). Mutates and returns the
|
|
37
|
+
same Hash/Array; only container contents change — hash keys are untouched and
|
|
38
|
+
String leaves are replaced rather than mutated, so frozen leaves are safe. Raises
|
|
39
|
+
`ArgumentError` on anything that is not a Hash or Array, pointing at `redact` /
|
|
40
|
+
`redact_deep`. The copy-returning methods are unchanged.
|
|
41
|
+
- **Rails Railtie — zero-config onboarding.** `require "data_redactor/railtie"`
|
|
42
|
+
(e.g. `gem "data_redactor", require: "data_redactor/railtie"`) now wires both
|
|
43
|
+
Rails surfaces automatically: the redacting Logger formatter and a
|
|
44
|
+
`config.filter_parameters` entry. Everything is tunable from an initializer via
|
|
45
|
+
`config.data_redactor.{logger,filter_parameters,only,except,placeholder}`, and
|
|
46
|
+
either surface can be switched off. The logger initializer runs *after*
|
|
47
|
+
`initialize_logger` and **wraps** the app's existing formatter rather than
|
|
48
|
+
replacing it, so lograge/JSON formatters keep working; it descends into
|
|
49
|
+
`ActiveSupport::BroadcastLogger` and wraps each sink, since a formatter set on
|
|
50
|
+
the broadcast itself never runs. Already-wrapped formatters are left alone.
|
|
51
|
+
Rails is a development dependency only — the gem keeps zero runtime deps, and
|
|
52
|
+
the file is loaded only when the app requires it.
|
|
53
|
+
- **CI: the Ruby 2.7 floor is now tested, not just claimed.** New `ruby-floor`
|
|
54
|
+
job compiles the C extension and runs the full suite on Ruby 2.7 and 3.0 — the
|
|
55
|
+
bottom of the `required_ruby_version >= 2.7` range, which the 3.1–3.4 `test`
|
|
56
|
+
matrix never covered. It pins `ubuntu-24.04` (ruby-builder has no 2.7/3.0
|
|
57
|
+
binaries for 26.04) and resolves dependencies without the committed lockfile,
|
|
58
|
+
which pins a Rails that requires Ruby >= 3.1. No source changes were needed:
|
|
59
|
+
both Rubies were already green.
|
|
60
|
+
- **CI: Ruby 4.0 is tested, and `ruby-head` is watched.** 4.0 joins the `test`
|
|
61
|
+
matrix as a supported release (green as-is, no source changes) — resolving its
|
|
62
|
+
own dependency set, since the committed lockfile's nokogiri caps at
|
|
63
|
+
`< 3.5.dev` and `bundler-cache` installs frozen — and a new
|
|
64
|
+
allow-failure `ruby-next` job compiles the extension and runs the specs
|
|
65
|
+
against `ruby-head` so C-API or stdlib breakage surfaces months before a
|
|
66
|
+
release week rather than during one. `ruby-next` excludes Rails: nokogiri has
|
|
67
|
+
no precompiled gem for head, and letting it fail there would mask the signal
|
|
68
|
+
the job exists for. Precompiled binaries still cover 3.1–3.4 only, so Ruby 4.0
|
|
69
|
+
installs the source gem for now.
|
|
70
|
+
- **Project wiki.** Set up the GitHub wiki as the home for deep material so the
|
|
71
|
+
README stays a focused entry point: pattern catalogue (grouped by tag/country),
|
|
72
|
+
C engine internals (NFA → bytecode → lazy DFA, the v19 story), integration
|
|
73
|
+
guides, a dedicated RubyLLM page (per-call + transparent `install!`), custom /
|
|
74
|
+
name-pattern cookbook, benchmark methodology, and FAQ. README now links to it
|
|
75
|
+
and promotes RubyLLM higher in Usage and in the use-case list.
|
|
76
|
+
|
|
77
|
+
### Removed
|
|
78
|
+
- **`Integrations::RubyLLM.install!` (transparent mode).** It prepended
|
|
79
|
+
`RubyLLM::Protocol#render`, an internal that only ever existed in ruby_llm 2.0
|
|
80
|
+
pre-release, and its whole justification was that RubyLLM exposed no public seam
|
|
81
|
+
to rewrite an outbound request. 2.0 exposes one (`Chat#before_request`), so the
|
|
82
|
+
monkeypatch is gone. `install!` still exists for one release and raises a message
|
|
83
|
+
naming `chat` / `attach!`; it will be deleted in the next minor. The upstream
|
|
84
|
+
request we were tracking for this, [crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765),
|
|
85
|
+
was closed on 2026-08-12 as superseded by 2.0's instrumentation surface — which is
|
|
86
|
+
observe-only and does not solve payload rewriting, but `before_request` does.
|
|
87
|
+
Removing a shipped public method would normally force a major bump; this one
|
|
88
|
+
lands in a minor because `RubyLLM::Protocol` exists in no *stable* `ruby_llm`
|
|
89
|
+
release. The only versions carrying it are the 2.0 release candidates published
|
|
90
|
+
days before this change, so `install!` worked for nobody on 1.x and for almost
|
|
91
|
+
nobody on 2.0 — and `attach!` covers the same ground on the same versions.
|
|
92
|
+
|
|
93
|
+
### Changed
|
|
94
|
+
- **Engine re-entrancy: selective-merge cursors are now per-call, not per-thread.**
|
|
95
|
+
The digit-run and IBAN-union passes kept their non-overlap cursors in the
|
|
96
|
+
per-thread scan cache; they now live in a stack-allocated context, one set per
|
|
97
|
+
`mm_scan` call. Output is byte-for-byte unchanged — this is an internal
|
|
98
|
+
threading change with no API or behaviour difference. It makes the C engine
|
|
99
|
+
genuinely re-entrant (a prerequisite for Ractors and for widening the
|
|
100
|
+
GVL-free region) rather than relying on thread-local storage to keep concurrent
|
|
101
|
+
scans apart, and lifts the per-thread cache lookup out of the per-pattern inner
|
|
102
|
+
loop. New spec asserts merge-cursor output stays call-private under parallel
|
|
103
|
+
GVL-released digit/IBAN load.
|
|
104
|
+
|
|
105
|
+
### Fixed
|
|
106
|
+
- **`Integrations::RubyLLM.install!` now states the version it actually needs.**
|
|
107
|
+
It declared `~> 1.16`, but `RubyLLM::Protocol` — the chokepoint it prepends —
|
|
108
|
+
exists only in the 2.0 line; the 1.x series assembles the request
|
|
109
|
+
inside `Provider#complete`. On a real `ruby_llm` 1.16.0 the version check
|
|
110
|
+
passed and the next line died with `NameError: uninitialized constant
|
|
111
|
+
RubyLLM::Protocol`. The pin is now `>= 2.0.0.pre`, so 1.x users get a clear
|
|
112
|
+
message pointing at per-call `DataRedactor.redact`, and a missing `Protocol`
|
|
113
|
+
constant raises the same fail-fast error as a missing `Protocol#render`.
|
|
114
|
+
Redaction was never affected — the integration failed loudly at `install!`
|
|
115
|
+
rather than leaking — but it worked on no released `ruby_llm` version.
|
|
116
|
+
- **Logger integration no longer raises `LoadError` on Ruby 4.0.** Ruby 4.0
|
|
117
|
+
demoted `logger` from a default gem to a bundled one, so `require "logger"`
|
|
118
|
+
only resolves when something declares it — and this gem declares no runtime
|
|
119
|
+
dependencies. `integrations/logger.rb` now soft-requires it: anyone assigning
|
|
120
|
+
the redacting formatter already holds a `::Logger`, so their own require
|
|
121
|
+
defines the constant. Rails apps were never affected (activesupport declares
|
|
122
|
+
`logger`); plain Ruby 4.0 apps hit it the moment they loaded the integration.
|
|
123
|
+
The gem stays dependency-free. Found by the new `ruby-next` job on its first
|
|
124
|
+
run. **If you use the Logger integration on Ruby 4.0 without Rails, add
|
|
125
|
+
`gem "logger"` to your Gemfile** — Ruby 4.0 requires that of every caller,
|
|
126
|
+
not just this gem.
|
|
127
|
+
- Gemspec description said "85 sensitive patterns" while the engine ships 89.
|
|
128
|
+
The description no longer hardcodes a count, so it can't drift again.
|
|
129
|
+
|
|
10
130
|
## [0.17.0] - 2026-06-21
|
|
11
131
|
|
|
12
132
|
### Added
|
|
@@ -354,7 +474,8 @@ features as 0.7.1 plus the pipeline fix.
|
|
|
354
474
|
- `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
|
|
355
475
|
- RSpec suite with one example per pattern.
|
|
356
476
|
|
|
357
|
-
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.
|
|
477
|
+
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.18.0...HEAD
|
|
478
|
+
[0.18.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.17.0...v0.18.0
|
|
358
479
|
[0.17.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.16.0...v0.17.0
|
|
359
480
|
[0.16.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...v0.16.0
|
|
360
481
|
[0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
|
data/README.md
CHANGED
|
@@ -6,6 +6,8 @@
|
|
|
6
6
|
|
|
7
7
|
A Ruby gem with a C extension for high-performance regex-based redaction of sensitive data from strings.
|
|
8
8
|
|
|
9
|
+
📚 **Deeper docs live in the [wiki](https://github.com/danielefrisanco/data_redactor/wiki):** the full [pattern catalogue](https://github.com/danielefrisanco/data_redactor/wiki/Pattern-Catalogue), [C engine internals](https://github.com/danielefrisanco/data_redactor/wiki/C-Engine-Internals), [integration guides](https://github.com/danielefrisanco/data_redactor/wiki/Integration-Guides) (incl. [RubyLLM](https://github.com/danielefrisanco/data_redactor/wiki/RubyLLM-Integration)), [benchmark methodology](https://github.com/danielefrisanco/data_redactor/wiki/Performance-and-Benchmarks), and [FAQ](https://github.com/danielefrisanco/data_redactor/wiki/FAQ). The [API reference](https://danielefrisanco.github.io/data_redactor/) is on GitHub Pages.
|
|
10
|
+
|
|
9
11
|
> 📄 The engineering behind the v19 matching engine is written up as an experience
|
|
10
12
|
> report, *"The Fastest Engine Is Not the Shippable Engine: Replacing a Regex Engine
|
|
11
13
|
> for Data Redaction Under Production Constraints,"* currently under review at
|
|
@@ -31,12 +33,15 @@ Rack. You can also register your own patterns — at boot or at runtime from any
|
|
|
31
33
|
|
|
32
34
|
- **Log scrubbing** — drop the `Logger` formatter in so no secret or PII ever
|
|
33
35
|
reaches disk or your log aggregator.
|
|
34
|
-
- **Rails
|
|
35
|
-
|
|
36
|
+
- **Rails, zero-config** — one Gemfile line wires both the log formatter and
|
|
37
|
+
`filter_parameters`; unlike `filter_parameters` alone, it also catches secrets
|
|
38
|
+
in free text you can't enumerate by key.
|
|
36
39
|
- **HTTP request/response sanitising** — Rack middleware scrubs response bodies
|
|
37
40
|
and sensitive headers in flight.
|
|
38
|
-
- **
|
|
39
|
-
|
|
41
|
+
- **Scrubbing prompts before an LLM** — with [RubyLLM](#rubyllm--redact-before-it-reaches-the-model), redact every
|
|
42
|
+
prompt, system instruction, and tool result before it reaches the model —
|
|
43
|
+
per-call, or on every request through RubyLLM's own request hook. Also `redact_deep` /
|
|
44
|
+
`redact_json` over any params hash or JSON body before it leaves the process.
|
|
40
45
|
- **Compliance & auditing** — `scan` reports every match with byte offsets, tag,
|
|
41
46
|
and pattern name without changing the text, for false-positive tuning.
|
|
42
47
|
- **Internal identifiers** — register company-specific patterns (`add_pattern`)
|
|
@@ -58,6 +63,70 @@ custom patterns, deep/JSON traversal, and the Logger / Rack / Rails / LLM
|
|
|
58
63
|
integrations. Run any of them with `bundle exec ruby examples/<name>.rb` (see
|
|
59
64
|
[examples/README.md](examples/README.md)).
|
|
60
65
|
|
|
66
|
+
### RubyLLM — redact before it reaches the model
|
|
67
|
+
|
|
68
|
+
[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — Anthropic, OpenAI, Gemini, Bedrock, and more — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
require "ruby_llm"
|
|
72
|
+
require "data_redactor"
|
|
73
|
+
|
|
74
|
+
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
75
|
+
chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
|
|
76
|
+
|
|
77
|
+
user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
|
|
78
|
+
chat.ask(DataRedactor.redact(user_input))
|
|
79
|
+
# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into, and it's the recommended approach.
|
|
83
|
+
|
|
84
|
+
#### Every request, no per-call wrapping
|
|
85
|
+
|
|
86
|
+
To redact **every** outbound request — including the system prompt, the whole history, tool definitions, and any file contents or shell-command output an agent feeds back as a tool result — build the chat through the integration:
|
|
87
|
+
|
|
88
|
+
> **Requires `ruby_llm` 2.0 or newer** (verified against `2.0.0.rc2`). 1.x has no request hook at all; on 1.x use the per-call `DataRedactor.redact` form above.
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
require "ruby_llm"
|
|
92
|
+
require "data_redactor/integrations/ruby_llm"
|
|
93
|
+
|
|
94
|
+
chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
95
|
+
chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
`chat` is a drop-in for `RubyLLM.chat` — every argument is forwarded and you get a real chat back, so the fluent API keeps working. It registers RubyLLM's public [`before_request`](https://rubyllm.com) hook, which receives the fully rendered payload just before it's posted and lets a callback edit it in place. **Nothing is monkeypatched.**
|
|
99
|
+
|
|
100
|
+
Already holding a chat, an [`Agent`](https://rubyllm.com), or an `acts_as_chat` record? Attach to it — `attach!` finds the chat inside whichever you pass and returns your object:
|
|
101
|
+
|
|
102
|
+
```ruby
|
|
103
|
+
DataRedactor::Integrations::RubyLLM.attach!(agent, only: [:financial])
|
|
104
|
+
|
|
105
|
+
# or register the callback yourself
|
|
106
|
+
chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
For an **agent**, the tidiest spelling is to hand it a chat that's already redacted — an agent keeps the chat you pass and layers its own configuration on top:
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
113
|
+
agent = SupportAgent.new(chat: chat)
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Every request in the agent's tool loop is then redacted — and an agent issues one per turn, since `Chat#complete` steps until the loop settles.
|
|
117
|
+
|
|
118
|
+
This is the only way to scrub **tool results** — the file an agent read or the command it ran gets inlined into the *next* request, and the user never typed it, so per-call redaction can't reach it.
|
|
119
|
+
|
|
120
|
+
Five things to know:
|
|
121
|
+
|
|
122
|
+
- **It's per chat.** RubyLLM has no global callback registry, so a chat you neither built with `.chat` nor passed to `attach!` is not redacted.
|
|
123
|
+
- **Chat only.** Embeddings, moderation, image generation and transcription don't run request hooks.
|
|
124
|
+
- **The `model` key is skipped by default** (`skip_keys:`). Dated model ids like `claude-haiku-4-5-20251001` end in eight digits, which the national-ID patterns match, and a provider rejects a redacted model id. Pass `skip_keys: [:model, :metadata]` to protect more; pass `skip_keys: []` to redact everything.
|
|
125
|
+
- **A digits-only tool-call id would be redacted.** `_` counts as a boundary, so an id like `toolu_012345678` matches a national-ID pattern and breaks the `tool_use_id` correlation the provider validates. Anthropic and OpenAI mint mixed alphanumeric ids (`toolu_01A09q…`, `call_abc123…`), which are untouched; if a provider ever changes that, add `:id` to `skip_keys:`.
|
|
126
|
+
- **Base64 attachments** (PDFs, images, audio sent inline) and **URL-referenced files** are not redacted — the sensitive bytes are encoded or remote, so patterns cannot see them.
|
|
127
|
+
|
|
128
|
+
Redaction applies to the rendered payload and nothing is persisted, so your stored conversation keeps its original text — this scrubs the wire, per request. (We asked for a connection-middleware hook in [crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765); it was declined in favour of the instrumentation surface, but `before_request` gives us what we needed.)
|
|
129
|
+
|
|
61
130
|
### Filtering by tag or pattern name
|
|
62
131
|
|
|
63
132
|
`only:` and `except:` both accept a single value or an Array, mixing **Symbols** (tag names) and **Strings** (specific pattern names).
|
|
@@ -171,6 +240,28 @@ DataRedactor.redact_deep(params, only: :credentials)
|
|
|
171
240
|
DataRedactor.redact_deep(payload, except: :network, placeholder: :tagged)
|
|
172
241
|
```
|
|
173
242
|
|
|
243
|
+
Need the original scrubbed rather than a copy? `redact_deep!` is the in-place sibling — it mutates the Hash or Array you hand it and returns that same object. Use it when the caller ignores your return value, such as a hook or a middleware that must edit the structure it was given:
|
|
244
|
+
|
|
245
|
+
```ruby
|
|
246
|
+
params = { "user" => { "email" => "alice@example.com" } }
|
|
247
|
+
DataRedactor.redact_deep!(params)
|
|
248
|
+
params # => { "user" => { "email" => "[REDACTED]" } }
|
|
249
|
+
|
|
250
|
+
# Same filters; raises ArgumentError on anything that isn't a Hash or Array
|
|
251
|
+
DataRedactor.redact_deep!(payload, only: :credentials)
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Only the containers change: hash keys are untouched and string leaves are *replaced*, never mutated, so frozen strings are safe.
|
|
255
|
+
|
|
256
|
+
Some fields must survive verbatim — an identifier the receiving API validates, for instance. `skip_keys:` leaves them alone, matched by name at any depth (Symbol and String keys are equivalent), skipping the key's whole subtree:
|
|
257
|
+
|
|
258
|
+
```ruby
|
|
259
|
+
DataRedactor.redact_deep(request, skip_keys: [:model])
|
|
260
|
+
# "claude-haiku-4-5-20251001" survives; everything else is still redacted
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
It's a denylist, not a filter: anything you don't list is still redacted, so a field you've never seen can't slip through. Use `only:`/`except:` to choose *what* counts as sensitive.
|
|
264
|
+
|
|
174
265
|
```ruby
|
|
175
266
|
# JSON string — parse → redact_deep → re-serialise
|
|
176
267
|
safe_json = DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
|
|
@@ -267,6 +358,39 @@ DataRedactor.name_pattern("Mario", "Rossi", middle: "Luigi")
|
|
|
267
358
|
|
|
268
359
|
Optional adapters for Logger, Rails, and Rack. None are loaded automatically — `require` only what you use, and the gem adds zero runtime dependencies in the gemspec.
|
|
269
360
|
|
|
361
|
+
### Rails (zero-config)
|
|
362
|
+
|
|
363
|
+
One Gemfile line wires both Rails surfaces — the log formatter and `filter_parameters`:
|
|
364
|
+
|
|
365
|
+
```ruby
|
|
366
|
+
# Gemfile
|
|
367
|
+
gem "data_redactor", require: "data_redactor/railtie"
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
That's the whole setup. `filter_parameters` on its own only redacts the keys you name; data_redactor also catches the card number sitting inside an exception message or a free-text field you can't enumerate in advance.
|
|
371
|
+
|
|
372
|
+
Tune it from an initializer:
|
|
373
|
+
|
|
374
|
+
```ruby
|
|
375
|
+
# config/initializers/data_redactor.rb
|
|
376
|
+
Rails.application.configure do
|
|
377
|
+
config.data_redactor.only = [:credentials, :financial]
|
|
378
|
+
config.data_redactor.placeholder = :tagged
|
|
379
|
+
config.data_redactor.logger = false # leave the logger alone
|
|
380
|
+
end
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
| Option | Default | Effect |
|
|
384
|
+
| --- | --- | --- |
|
|
385
|
+
| `logger` | `true` | Wrap the Rails logger's formatter |
|
|
386
|
+
| `filter_parameters` | `true` | Append a redacting `filter_parameters` entry |
|
|
387
|
+
| `only` / `except` | `nil` | Forwarded to `DataRedactor.redact` |
|
|
388
|
+
| `placeholder` | `"[REDACTED]"` | Forwarded to `DataRedactor.redact` |
|
|
389
|
+
|
|
390
|
+
The logger initializer runs after Rails' own, so it **wraps** whatever formatter your app ended up with (lograge, a JSON formatter) rather than replacing it, and it descends into `ActiveSupport::BroadcastLogger` to wrap each sink. Rails stays a development dependency — the Railtie file is only loaded when you require it.
|
|
391
|
+
|
|
392
|
+
Prefer wiring the two surfaces by hand? Both are documented below.
|
|
393
|
+
|
|
270
394
|
### Logger formatter
|
|
271
395
|
|
|
272
396
|
Drop-in `Logger::Formatter` replacement that scrubs every emitted line:
|
|
@@ -282,6 +406,8 @@ logger.info("Auth failed for alice@example.com")
|
|
|
282
406
|
|
|
283
407
|
Wraps an inner formatter (defaults to `Logger::Formatter`), so it composes with structured loggers. Forwards `only:`, `except:`, `placeholder:` to `DataRedactor.redact`. Exception messages and arbitrary objects are scrubbed too — the wrapped object is passed unchanged to the inner formatter so the exception cause chain is preserved; only the rendered string is redacted.
|
|
284
408
|
|
|
409
|
+
> **Ruby 4.0:** `logger` became a bundled gem rather than a default one, so `require "logger"` resolves only when your Gemfile declares it. Add `gem "logger"` if you use this integration outside Rails — Ruby 4.0 asks that of every caller, not just this gem. Rails apps already have it via `activesupport`.
|
|
410
|
+
|
|
285
411
|
### Rails `filter_parameters` adapter
|
|
286
412
|
|
|
287
413
|
```ruby
|
|
@@ -340,46 +466,7 @@ client.chat(parameters: { model: "gpt-4o", messages: safe_messages })
|
|
|
340
466
|
safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
|
|
341
467
|
```
|
|
342
468
|
|
|
343
|
-
`content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
|
|
344
|
-
|
|
345
|
-
### RubyLLM
|
|
346
|
-
|
|
347
|
-
[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
|
|
348
|
-
|
|
349
|
-
```ruby
|
|
350
|
-
require "ruby_llm"
|
|
351
|
-
require "data_redactor"
|
|
352
|
-
|
|
353
|
-
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
354
|
-
chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
|
|
355
|
-
|
|
356
|
-
user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
|
|
357
|
-
chat.ask(DataRedactor.redact(user_input))
|
|
358
|
-
# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
|
|
359
|
-
```
|
|
360
|
-
|
|
361
|
-
Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into, and it's the recommended approach.
|
|
362
|
-
|
|
363
|
-
#### Transparent mode (every request, no per-call wrapping)
|
|
364
|
-
|
|
365
|
-
If you'd rather redact **every** outbound request automatically — including the system prompt, tool definitions, and any file contents or shell-command output an agent feeds back as a tool result — opt into the monkeypatch:
|
|
366
|
-
|
|
367
|
-
```ruby
|
|
368
|
-
require "ruby_llm"
|
|
369
|
-
require "data_redactor/integrations/ruby_llm"
|
|
370
|
-
|
|
371
|
-
DataRedactor::Integrations::RubyLLM.install! # once, at boot
|
|
372
|
-
|
|
373
|
-
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
374
|
-
chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
375
|
-
```
|
|
376
|
-
|
|
377
|
-
`install!` prepends a patch onto `RubyLLM::Protocol#render` — the one point where every provider (Anthropic, OpenAI, Gemini, Bedrock, Responses) has assembled its final request — and deep-redacts the payload before it's posted. It forwards `only:`/`except:`/`placeholder:`, is idempotent, and **fails fast** at `install!` if an unsupported `ruby_llm` version is loaded or the internal API has moved (so it never silently leaks).
|
|
378
|
-
|
|
379
|
-
Two caveats, by design:
|
|
380
|
-
|
|
381
|
-
- **It's a monkeypatch on RubyLLM internals**, pinned to a supported version range. Prefer per-call `DataRedactor.redact` (above) unless you specifically need transparency. RubyLLM does not yet expose a public request hook ([crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765) tracks the connection-middleware hook that would let us drop the patch).
|
|
382
|
-
- **Base64 attachments** (PDFs, images, audio sent inline) and **URL-referenced files** are not redacted — the sensitive bytes are encoded or remote, so patterns cannot see them.
|
|
469
|
+
`content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works. For [RubyLLM](#rubyllm--redact-before-it-reaches-the-model) — a unified client across every provider — see the dedicated section above, including whole-request redaction through its `before_request` hook.
|
|
383
470
|
|
|
384
471
|
## Detected patterns (89 total)
|
|
385
472
|
|
|
@@ -480,6 +567,7 @@ redactor/
|
|
|
480
567
|
│ └── data_redactor/
|
|
481
568
|
│ ├── version.rb
|
|
482
569
|
│ ├── name_pattern.rb # name_pattern helper — generates a name regex for add_pattern
|
|
570
|
+
│ ├── railtie.rb # opt-in Rails Railtie — wires logger + filter_parameters
|
|
483
571
|
│ └── integrations/ # soft-required Logger / Rails / Rack adapters
|
|
484
572
|
├── ext/
|
|
485
573
|
│ └── data_redactor/
|
|
@@ -502,6 +590,7 @@ redactor/
|
|
|
502
590
|
│ ├── logger.rb # Logger::Formatter integration
|
|
503
591
|
│ ├── rack_middleware.rb # Rack middleware (body + headers)
|
|
504
592
|
│ ├── rails_filter.rb # filter_parameters adapter
|
|
593
|
+
│ ├── rails_logger.rb # Rack middleware + redacting Logger, end-to-end
|
|
505
594
|
│ └── llm_payload.rb # Claude / OpenAI message + response redaction
|
|
506
595
|
├── benchmark/ # Repo-only perf scripts (not packaged in the gem)
|
|
507
596
|
│ ├── README.md # How to run, what each script measures
|
|
@@ -517,7 +606,7 @@ redactor/
|
|
|
517
606
|
|
|
518
607
|
## Requirements
|
|
519
608
|
|
|
520
|
-
- Ruby >= 2.7
|
|
609
|
+
- Ruby >= 2.7 — CI runs the full suite on 2.7 and 3.0 as well as 3.1–3.4 and 4.0
|
|
521
610
|
- A C compiler (`gcc` or `clang`) — only required when installing the source gem
|
|
522
611
|
- POSIX `regex.h` — only required when installing the source gem (standard on Linux and macOS)
|
|
523
612
|
|
|
@@ -541,7 +630,7 @@ That's it — there is nothing extra to configure for precompiled binaries. Bund
|
|
|
541
630
|
|
|
542
631
|
### Supported precompiled targets
|
|
543
632
|
|
|
544
|
-
Each precompiled gem ships compiled binaries for Ruby 3.1, 3.2, 3.3, and 3.4.
|
|
633
|
+
Each precompiled gem ships compiled binaries for Ruby 3.1, 3.2, 3.3, and 3.4. Ruby 2.7 and 3.0 are supported and tested, but fall back to the source gem — bundler compiles the extension on install, so those need a C compiler.
|
|
545
634
|
|
|
546
635
|
| Platform | Targets |
|
|
547
636
|
|---|---|
|
data/ext/data_redactor/matcher.c
CHANGED
|
@@ -389,10 +389,11 @@ typedef struct {
|
|
|
389
389
|
|
|
390
390
|
/* engine_t holds ONLY immutable, compiled state — built once at mm_init()/mm_add()
|
|
391
391
|
* and never written during a scan, so it is safe to share read-only across
|
|
392
|
-
* threads.
|
|
393
|
-
*
|
|
394
|
-
*
|
|
395
|
-
* scan
|
|
392
|
+
* threads. Mutable state is split two ways: the reusable NFA scratch + lazy DFA
|
|
393
|
+
* cache live in scan_state_t, which is per-thread (t_block below); the genuinely
|
|
394
|
+
* per-call merge cursors live in scan_ctx_t, stack-allocated by each mm_scan.
|
|
395
|
+
* This split is what lets redact/scan release the GVL: with no shared writes
|
|
396
|
+
* during a scan, concurrent scans on distinct threads cannot race. */
|
|
396
397
|
typedef struct {
|
|
397
398
|
prog_t prog;
|
|
398
399
|
size_t min_len;
|
|
@@ -414,9 +415,13 @@ typedef struct {
|
|
|
414
415
|
int iban_member;
|
|
415
416
|
} engine_t;
|
|
416
417
|
|
|
417
|
-
/* Per-engine
|
|
418
|
-
* The DFA cache warms lazily across this thread's scans; the
|
|
419
|
-
*
|
|
418
|
+
/* Per-engine REUSABLE scan cache. One per engine, owned per-thread (t_block).
|
|
419
|
+
* The DFA cache warms lazily across this thread's scans; the NFA-VM scratch
|
|
420
|
+
* (seen/clist/nlist/estack/gen) is reset within each scan but its buffers are
|
|
421
|
+
* reused. This is pure cache — sharing it across scans changes performance, not
|
|
422
|
+
* results. seen_cap==0 / dfa.n_states==0 means "not yet warmed" for this thread.
|
|
423
|
+
* The genuinely per-CALL state (the selective-merge cursors) lives in scan_ctx_t
|
|
424
|
+
* instead, so two concurrent scans (GVL released) never share a writable cursor. */
|
|
420
425
|
typedef struct {
|
|
421
426
|
dfa_t dfa;
|
|
422
427
|
int *seen;
|
|
@@ -424,10 +429,21 @@ typedef struct {
|
|
|
424
429
|
tlist_t clist, nlist;
|
|
425
430
|
int *estack;
|
|
426
431
|
int gen;
|
|
427
|
-
int digit_last_end; /* selective-merge non-overlap cursors */
|
|
428
|
-
size_t iban_last_end;
|
|
429
432
|
} scan_state_t;
|
|
430
433
|
|
|
434
|
+
/* Per-CALL mutable scan context. One scan_ctx_t per mm_scan() call, owned by the
|
|
435
|
+
* caller (stack-allocated in mm_scan). Holds the only state that must NOT be
|
|
436
|
+
* shared between concurrent scans: the selective-merge non-overlap cursors, one
|
|
437
|
+
* per engine. Keeping these out of the per-thread cache is what makes the engine
|
|
438
|
+
* re-entrant — a prerequisite for Ractors and for widening the GVL-free region.
|
|
439
|
+
* `cache` points at this thread's reusable scan_state_t array (the DFA cache);
|
|
440
|
+
* the ctx borrows it, it does not own it. */
|
|
441
|
+
typedef struct {
|
|
442
|
+
scan_state_t *cache; /* borrowed: this thread's per-engine cache */
|
|
443
|
+
int *digit_last_end; /* selective-merge cursors, one per engine */
|
|
444
|
+
size_t *iban_last_end;
|
|
445
|
+
} scan_ctx_t;
|
|
446
|
+
|
|
431
447
|
static engine_t *g_eng = NULL;
|
|
432
448
|
static int g_eng_n = 0; /* engines built (NUM_PATTERNS + custom_n) */
|
|
433
449
|
static int g_eng_cap= 0;
|
|
@@ -1048,11 +1064,11 @@ static size_t scan_one(int p, scan_state_t *state, const char *input, size_t len
|
|
|
1048
1064
|
* 10. Selective merges (digit run pass + IBAN union pass)
|
|
1049
1065
|
* ======================================================================== */
|
|
1050
1066
|
|
|
1051
|
-
static size_t scan_digit_group(
|
|
1067
|
+
static size_t scan_digit_group(scan_ctx_t *ctx, const char *input, size_t len,
|
|
1052
1068
|
const int *enable_bits, size_t n_bits,
|
|
1053
1069
|
mm_match_t *out, size_t max, size_t count) {
|
|
1054
1070
|
for (int p = 0; p < g_eng_n; p++)
|
|
1055
|
-
if (g_eng[p].digit_member)
|
|
1071
|
+
if (g_eng[p].digit_member) ctx->digit_last_end[p] = 0;
|
|
1056
1072
|
|
|
1057
1073
|
size_t i = 0;
|
|
1058
1074
|
while (i < len) {
|
|
@@ -1076,7 +1092,7 @@ static size_t scan_digit_group(scan_state_t *state, const char *input, size_t le
|
|
|
1076
1092
|
|
|
1077
1093
|
size_t start;
|
|
1078
1094
|
if (rs > 0 && !isalnum((unsigned char)input[rs-1]) &&
|
|
1079
|
-
rs - 1 >= (size_t)
|
|
1095
|
+
rs - 1 >= (size_t)ctx->digit_last_end[p]) {
|
|
1080
1096
|
start = rs - 1;
|
|
1081
1097
|
} else if (rs == 0 || input[rs-1] == '\n') {
|
|
1082
1098
|
start = rs;
|
|
@@ -1091,22 +1107,23 @@ static size_t scan_digit_group(scan_state_t *state, const char *input, size_t le
|
|
|
1091
1107
|
* separator are resolved exactly as gsub would. */
|
|
1092
1108
|
(void)start;
|
|
1093
1109
|
out[count++] = (mm_match_t){p, rs, re - rs};
|
|
1094
|
-
|
|
1110
|
+
ctx->digit_last_end[p] = (int)end;
|
|
1095
1111
|
}
|
|
1096
1112
|
if (count >= max) break;
|
|
1097
1113
|
}
|
|
1098
1114
|
return count;
|
|
1099
1115
|
}
|
|
1100
1116
|
|
|
1101
|
-
static size_t scan_iban_group(
|
|
1117
|
+
static size_t scan_iban_group(scan_ctx_t *ctx, const char *input, size_t len,
|
|
1102
1118
|
const int *enable_bits, size_t n_bits,
|
|
1103
1119
|
mm_match_t *out, size_t max, size_t count) {
|
|
1120
|
+
scan_state_t *cache = ctx->cache;
|
|
1104
1121
|
for (int p = 0; p < g_eng_n; p++)
|
|
1105
1122
|
if (g_eng[p].iban_member) {
|
|
1106
|
-
|
|
1123
|
+
ctx->iban_last_end[p] = 0;
|
|
1107
1124
|
engine_t *eng = &g_eng[p];
|
|
1108
|
-
if (eng->use_dfa &&
|
|
1109
|
-
ensure_scratch(eng, &
|
|
1125
|
+
if (eng->use_dfa && cache[p].dfa.n_states == 0) {
|
|
1126
|
+
ensure_scratch(eng, &cache[p]); dfa_build_start(eng, &cache[p]);
|
|
1110
1127
|
}
|
|
1111
1128
|
}
|
|
1112
1129
|
|
|
@@ -1117,10 +1134,10 @@ static size_t scan_iban_group(scan_state_t *state, const char *input, size_t len
|
|
|
1117
1134
|
int p = g_iban_pair[c0][(unsigned char)input[i + 1]];
|
|
1118
1135
|
if (p < 0) { i++; continue; }
|
|
1119
1136
|
if ((size_t)p < n_bits && !enable_bits[p]) { i++; continue; }
|
|
1120
|
-
if (i <
|
|
1137
|
+
if (i < ctx->iban_last_end[p]) { i++; continue; }
|
|
1121
1138
|
|
|
1122
1139
|
engine_t *eng = &g_eng[p];
|
|
1123
|
-
scan_state_t *sst = &
|
|
1140
|
+
scan_state_t *sst = &cache[p];
|
|
1124
1141
|
dfa_t *d = &sst->dfa;
|
|
1125
1142
|
size_t match_end = (size_t)-1, sp = i;
|
|
1126
1143
|
int st = 0;
|
|
@@ -1137,7 +1154,7 @@ static size_t scan_iban_group(scan_state_t *state, const char *input, size_t len
|
|
|
1137
1154
|
if (match_end != (size_t)-1) {
|
|
1138
1155
|
size_t span = match_end - i;
|
|
1139
1156
|
out[count++] = (mm_match_t){p, i, span};
|
|
1140
|
-
|
|
1157
|
+
ctx->iban_last_end[p] = match_end;
|
|
1141
1158
|
i = (span == 0) ? i + 1 : match_end;
|
|
1142
1159
|
} else {
|
|
1143
1160
|
i++;
|
|
@@ -1244,19 +1261,29 @@ size_t mm_scan(const char *input, size_t len,
|
|
|
1244
1261
|
const int *enable_bits, size_t n_bits,
|
|
1245
1262
|
mm_match_t *out, size_t max) {
|
|
1246
1263
|
if (!g_initialized) mm_init();
|
|
1247
|
-
|
|
1264
|
+
/* Per-thread reusable cache (DFA + NFA scratch); fetched once, not per
|
|
1265
|
+
* pattern, so the generation-guard indirection stays out of the inner loop. */
|
|
1266
|
+
scan_state_t *cache = thread_state();
|
|
1248
1267
|
size_t count = 0;
|
|
1249
1268
|
|
|
1250
1269
|
for (int p = 0; p < g_eng_n && count < max; p++) {
|
|
1251
1270
|
if (g_eng[p].digit_member) continue;
|
|
1252
1271
|
if (g_eng[p].iban_member) continue;
|
|
1253
1272
|
if (!enabled(enable_bits, n_bits, p)) continue;
|
|
1254
|
-
count = scan_one(p,
|
|
1273
|
+
count = scan_one(p, cache, input, len, out, max, count);
|
|
1274
|
+
}
|
|
1275
|
+
/* Per-call cursors: stack-allocated, one set per mm_scan, so concurrent
|
|
1276
|
+
* scans (GVL released for large inputs) never share a writable cursor. Sized
|
|
1277
|
+
* to g_eng_n (bounded by NUM_PATTERNS + customs); a VLA keeps it heap-free. */
|
|
1278
|
+
if ((g_have_iban_group || g_have_digit_group) && count < max) {
|
|
1279
|
+
int digit_cursors[g_eng_n];
|
|
1280
|
+
size_t iban_cursors[g_eng_n];
|
|
1281
|
+
scan_ctx_t ctx = { cache, digit_cursors, iban_cursors };
|
|
1282
|
+
if (g_have_iban_group && count < max)
|
|
1283
|
+
count = scan_iban_group(&ctx, input, len, enable_bits, n_bits, out, max, count);
|
|
1284
|
+
if (g_have_digit_group && count < max)
|
|
1285
|
+
count = scan_digit_group(&ctx, input, len, enable_bits, n_bits, out, max, count);
|
|
1255
1286
|
}
|
|
1256
|
-
if (g_have_iban_group && count < max)
|
|
1257
|
-
count = scan_iban_group(state, input, len, enable_bits, n_bits, out, max, count);
|
|
1258
|
-
if (g_have_digit_group && count < max)
|
|
1259
|
-
count = scan_digit_group(state, input, len, enable_bits, n_bits, out, max, count);
|
|
1260
1287
|
return count;
|
|
1261
1288
|
}
|
|
1262
1289
|
|
|
@@ -1,4 +1,14 @@
|
|
|
1
|
-
|
|
1
|
+
# Ruby 4.0 demoted logger from a default gem to a bundled one: it still ships
|
|
2
|
+
# with Ruby, but under Bundler it only resolves when something declares it, and
|
|
3
|
+
# this gem declares no runtime dependencies. Swallowing the LoadError costs
|
|
4
|
+
# nothing — anyone assigning this formatter holds a ::Logger instance already,
|
|
5
|
+
# so their own require has defined the constant this file needs.
|
|
6
|
+
begin
|
|
7
|
+
require "logger"
|
|
8
|
+
rescue LoadError
|
|
9
|
+
nil
|
|
10
|
+
end
|
|
11
|
+
|
|
2
12
|
require "data_redactor"
|
|
3
13
|
|
|
4
14
|
module DataRedactor
|
|
@@ -2,118 +2,182 @@ require "data_redactor"
|
|
|
2
2
|
|
|
3
3
|
module DataRedactor
|
|
4
4
|
module Integrations
|
|
5
|
-
#
|
|
5
|
+
# Outbound redaction for the `ruby_llm` gem (crmne/ruby_llm), built on
|
|
6
|
+
# RubyLLM's public request hook — nothing here monkeypatches RubyLLM.
|
|
6
7
|
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
#
|
|
11
|
-
#
|
|
12
|
-
# knowing any provider-specific shape.
|
|
8
|
+
# `chat.before_request { |payload| ... }` (ruby_llm 2.0+) hands a callback the
|
|
9
|
+
# fully rendered request payload — after all RubyLLM formatting and
|
|
10
|
+
# `with_provider_options` merging, immediately before it is posted — and
|
|
11
|
+
# discards the callback's return value, so hooks edit the payload in place.
|
|
12
|
+
# {hook} builds such a callback; {chat} and {attach!} register it for you.
|
|
13
13
|
#
|
|
14
|
-
# Because the payload is walked with {DataRedactor.redact_deep}, this scrubs
|
|
15
|
-
# **every String leaf**
|
|
16
|
-
# tool definitions, and —
|
|
17
|
-
#
|
|
18
|
-
#
|
|
14
|
+
# Because the payload is walked with {DataRedactor.redact_deep!}, this scrubs
|
|
15
|
+
# **every String leaf** of the request: the user prompt, the system prompt,
|
|
16
|
+
# the conversation history, tool definitions, and — the case per-call
|
|
17
|
+
# redaction cannot reach — **tool results**. An agent that reads a file or
|
|
18
|
+
# runs a command feeds that output back as a tool message, and it is inlined
|
|
19
|
+
# as a String in the next request's payload; the user never typed it, so only
|
|
20
|
+
# a request hook sees it before it leaves.
|
|
19
21
|
#
|
|
20
|
-
#
|
|
21
|
-
#
|
|
22
|
-
# is loaded and `RubyLLM::Protocol#render` still exists, so an upstream
|
|
23
|
-
# refactor fails loudly at install time rather than silently leaking data.
|
|
24
|
-
# Prefer this only when you need redaction to be *transparent*; otherwise
|
|
25
|
-
# redact per call with {DataRedactor.redact} before `chat.ask`.
|
|
22
|
+
# Redaction is applied to the rendered payload and nothing is persisted, so
|
|
23
|
+
# the conversation you keep is untouched: this scrubs the wire, per request.
|
|
26
24
|
#
|
|
27
|
-
# ##
|
|
28
|
-
# - **
|
|
29
|
-
#
|
|
30
|
-
# - **
|
|
31
|
-
#
|
|
25
|
+
# ## Limits, stated plainly
|
|
26
|
+
# - **Per chat.** RubyLLM has no global callback registry, so a chat you did
|
|
27
|
+
# not build with {chat} or pass to {attach!} is *not* redacted.
|
|
28
|
+
# - **Chat only.** Embeddings, moderation, image generation and transcription
|
|
29
|
+
# do not run request hooks.
|
|
30
|
+
# - **Base64 attachments** (PDFs, images, audio inlined as base64) and
|
|
31
|
+
# **URL-referenced files** are not redacted — the bytes are encoded or
|
|
32
|
+
# remote, so patterns cannot see into them.
|
|
33
|
+
# - **A digits-only tool-call id would be redacted**, breaking the
|
|
34
|
+
# `tool_use_id` correlation a provider validates: `_` counts as a boundary,
|
|
35
|
+
# so `toolu_012345678` matches a national-ID pattern. Real Anthropic and
|
|
36
|
+
# OpenAI ids are mixed alphanumeric and unaffected; add `:id` to
|
|
37
|
+
# `skip_keys:` if that ever changes.
|
|
38
|
+
# - Redacting tool results can break an agent that needs a value a tool
|
|
39
|
+
# returned. Scope with `only:`/`except:` when that matters.
|
|
32
40
|
#
|
|
33
|
-
# @example
|
|
41
|
+
# @example Build a redacted chat (nothing to remember later)
|
|
42
|
+
# require "ruby_llm"
|
|
34
43
|
# require "data_redactor/integrations/ruby_llm"
|
|
35
|
-
# DataRedactor::Integrations::RubyLLM.install!
|
|
36
44
|
#
|
|
37
|
-
# chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
45
|
+
# chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
38
46
|
# chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
39
47
|
#
|
|
40
|
-
# @example
|
|
41
|
-
# DataRedactor::Integrations::RubyLLM.
|
|
48
|
+
# @example Redact a chat, agent, or acts_as_chat record you were handed
|
|
49
|
+
# DataRedactor::Integrations::RubyLLM.attach!(agent, only: [:financial])
|
|
50
|
+
#
|
|
51
|
+
# @example Redact an agent by handing it a chat that is already redacted
|
|
52
|
+
# chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
53
|
+
# agent = SupportAgent.new(chat: chat) # the agent keeps that chat
|
|
54
|
+
#
|
|
55
|
+
# # Every request of the agent's tool loop is redacted, tool results
|
|
56
|
+
# # included. Reaches nothing internal, so it is the tidiest spelling
|
|
57
|
+
# # for agents.
|
|
58
|
+
#
|
|
59
|
+
# @example Register the callback yourself
|
|
60
|
+
# chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
42
61
|
module RubyLLM
|
|
43
62
|
module_function
|
|
44
63
|
|
|
45
|
-
# ruby_llm
|
|
46
|
-
#
|
|
47
|
-
SUPPORTED_VERSION = "
|
|
64
|
+
# The `ruby_llm` line that exposes `Chat#before_request`. 1.x has no
|
|
65
|
+
# request hook at all — redact per call with {DataRedactor.redact} there.
|
|
66
|
+
SUPPORTED_VERSION = ">= 2.0.0.pre"
|
|
48
67
|
|
|
49
|
-
#
|
|
50
|
-
# second call with the patch already installed is a no-op (the filter
|
|
51
|
-
# options from the first successful install are kept).
|
|
68
|
+
# Payload keys left untouched by default.
|
|
52
69
|
#
|
|
53
|
-
#
|
|
54
|
-
#
|
|
55
|
-
#
|
|
56
|
-
#
|
|
57
|
-
#
|
|
58
|
-
|
|
59
|
-
# @return [void]
|
|
60
|
-
# @raise [RuntimeError] if `ruby_llm` is not loaded, the loaded version is
|
|
61
|
-
# outside {SUPPORTED_VERSION}, or `RubyLLM::Protocol#render` is missing
|
|
62
|
-
# (i.e. an upstream refactor moved the chokepoint).
|
|
63
|
-
def install!(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
64
|
-
ensure_compatible!
|
|
70
|
+
# `model` is structural, not content: dated model ids carry an eight-digit
|
|
71
|
+
# suffix (`claude-haiku-4-5-20251001`), which the national-ID patterns
|
|
72
|
+
# match, and a redacted model id is rejected by the provider. Nothing else
|
|
73
|
+
# is skipped by default — an unlisted key is always redacted, so a field we
|
|
74
|
+
# have never seen cannot leak silently.
|
|
75
|
+
DEFAULT_SKIP_KEYS = [:model].freeze
|
|
65
76
|
|
|
66
|
-
|
|
67
|
-
|
|
77
|
+
# Build a chat with redaction already attached.
|
|
78
|
+
#
|
|
79
|
+
# A drop-in for `RubyLLM.chat`: every argument is forwarded to it
|
|
80
|
+
# untouched, and the returned chat is the real thing, so the fluent API
|
|
81
|
+
# keeps working (`.with_temperature(0.2).ask(...)`).
|
|
82
|
+
#
|
|
83
|
+
# @param only [Symbol, String, Array, nil] forwarded to {hook}.
|
|
84
|
+
# @param except [Symbol, String, Array, nil] forwarded to {hook}.
|
|
85
|
+
# @param placeholder [String, Symbol] forwarded to {hook}.
|
|
86
|
+
# @param skip_keys [Symbol, String, Array] forwarded to {hook}.
|
|
87
|
+
# @param kwargs [Hash] forwarded to `RubyLLM.chat` (`model:`, `provider:`,
|
|
88
|
+
# `protocol:`, `assume_model_exists:`, `context:`).
|
|
89
|
+
# @return [RubyLLM::Chat] a chat whose every request is redacted.
|
|
90
|
+
#
|
|
91
|
+
# @example
|
|
92
|
+
# DataRedactor::Integrations::RubyLLM.chat(model: "gpt-5.4", only: [:financial])
|
|
93
|
+
def chat(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
94
|
+
skip_keys: DEFAULT_SKIP_KEYS, **kwargs, &block)
|
|
95
|
+
attach!(::RubyLLM.chat(**kwargs, &block),
|
|
96
|
+
only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
97
|
+
end
|
|
68
98
|
|
|
69
|
-
|
|
99
|
+
# Register {hook} on a chat you did not build.
|
|
100
|
+
#
|
|
101
|
+
# Accepts whatever holds the chat, so callers need not know which object
|
|
102
|
+
# carries the hook: a `RubyLLM::Chat`, a `RubyLLM::Agent`, or an
|
|
103
|
+
# `acts_as_chat` record (its `#to_llm` chat is memoized, so one call covers
|
|
104
|
+
# the record). The agent's wrapped `#chat` is always used rather than its
|
|
105
|
+
# delegated `before_request`, because in Rails mode that delegator forwards
|
|
106
|
+
# to a record which has no such method.
|
|
107
|
+
#
|
|
108
|
+
# @param target [Object] a chat, agent, or `acts_as_chat` record.
|
|
109
|
+
# @param only [Symbol, String, Array, nil] forwarded to {hook}.
|
|
110
|
+
# @param except [Symbol, String, Array, nil] forwarded to {hook}.
|
|
111
|
+
# @param placeholder [String, Symbol] forwarded to {hook}.
|
|
112
|
+
# @param skip_keys [Symbol, String, Array] forwarded to {hook}.
|
|
113
|
+
# @return [Object] +target+, so calls chain.
|
|
114
|
+
# @raise [ArgumentError] if no chat with a `before_request` hook can be
|
|
115
|
+
# reached from +target+ — a `ruby_llm` older than {SUPPORTED_VERSION}.
|
|
116
|
+
#
|
|
117
|
+
# @example
|
|
118
|
+
# DataRedactor::Integrations::RubyLLM.attach!(chat).ask("...")
|
|
119
|
+
def attach!(target, only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
120
|
+
skip_keys: DEFAULT_SKIP_KEYS)
|
|
121
|
+
resolve(target).before_request(
|
|
122
|
+
&hook(only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
123
|
+
)
|
|
124
|
+
target
|
|
70
125
|
end
|
|
71
126
|
|
|
72
|
-
#
|
|
73
|
-
#
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
127
|
+
# Build the `before_request` callback.
|
|
128
|
+
#
|
|
129
|
+
# It redacts the payload **in place**, which is what RubyLLM's contract
|
|
130
|
+
# requires: hooks mutate what they are given and their return value is
|
|
131
|
+
# discarded.
|
|
132
|
+
#
|
|
133
|
+
# @param only [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep!}.
|
|
134
|
+
# @param except [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep!}.
|
|
135
|
+
# @param placeholder [String, Symbol] forwarded to {DataRedactor.redact_deep!}.
|
|
136
|
+
# @param skip_keys [Symbol, String, Array] payload keys to leave verbatim.
|
|
137
|
+
# Defaults to {DEFAULT_SKIP_KEYS}; pass more to protect provider fields
|
|
138
|
+
# your app relies on (`skip_keys: [:model, :metadata]`).
|
|
139
|
+
# @return [Proc] a one-argument callback for `chat.before_request`.
|
|
140
|
+
#
|
|
141
|
+
# @example
|
|
142
|
+
# chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
143
|
+
def hook(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
144
|
+
skip_keys: DEFAULT_SKIP_KEYS)
|
|
145
|
+
lambda do |payload|
|
|
146
|
+
DataRedactor.redact_deep!(payload, only: only, except: except,
|
|
147
|
+
placeholder: placeholder, skip_keys: skip_keys)
|
|
148
|
+
end
|
|
77
149
|
end
|
|
78
150
|
|
|
79
|
-
#
|
|
80
|
-
#
|
|
81
|
-
|
|
82
|
-
|
|
151
|
+
# @deprecated Transparent app-wide mode is gone. It prepended
|
|
152
|
+
# `RubyLLM::Protocol#render`; ruby_llm 2.0 exposes a public request hook,
|
|
153
|
+
# so redaction no longer patches RubyLLM. Use {chat} or {attach!}. This
|
|
154
|
+
# method exists only to say so and will be deleted in the next minor.
|
|
155
|
+
# @raise [RuntimeError] always.
|
|
156
|
+
def install!(*_args, **_kwargs)
|
|
157
|
+
raise "DataRedactor::Integrations::RubyLLM.install! has been removed: ruby_llm 2.0 exposes a " \
|
|
158
|
+
"public request hook, so redaction no longer patches RubyLLM. Build chats with " \
|
|
159
|
+
"DataRedactor::Integrations::RubyLLM.chat(...), or attach to an existing chat, agent, " \
|
|
160
|
+
"or record with DataRedactor::Integrations::RubyLLM.attach!(target)."
|
|
83
161
|
end
|
|
84
162
|
|
|
85
163
|
# @!visibility private
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
end
|
|
100
|
-
end
|
|
164
|
+
# Walks to the `RubyLLM::Chat` that actually holds the hook: an agent's
|
|
165
|
+
# wrapped chat, then a record's `to_llm` chat (an agent in Rails mode wraps
|
|
166
|
+
# the record, hence two hops), then the chat itself.
|
|
167
|
+
#
|
|
168
|
+
# The hops come first on purpose. `Agent` delegates `before_request` to
|
|
169
|
+
# whatever it wraps, so a Rails-mode agent answers `respond_to?` with true
|
|
170
|
+
# while the call forwards to a record that has no such method. Resolving to
|
|
171
|
+
# the real chat sidesteps the delegator instead of trusting it.
|
|
172
|
+
def resolve(target)
|
|
173
|
+
candidate = target
|
|
174
|
+
candidate = candidate.chat if candidate.respond_to?(:chat)
|
|
175
|
+
candidate = candidate.to_llm if candidate.respond_to?(:to_llm)
|
|
176
|
+
return candidate if candidate.respond_to?(:before_request)
|
|
101
177
|
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
# else in the send path.
|
|
106
|
-
module PayloadPatch
|
|
107
|
-
def render(*args, **kwargs)
|
|
108
|
-
payload = super
|
|
109
|
-
opts = DataRedactor::Integrations::RubyLLM.options
|
|
110
|
-
DataRedactor.redact_deep(
|
|
111
|
-
payload,
|
|
112
|
-
only: opts[:only],
|
|
113
|
-
except: opts[:except],
|
|
114
|
-
placeholder: opts[:placeholder]
|
|
115
|
-
)
|
|
116
|
-
end
|
|
178
|
+
raise ArgumentError, "data_redactor ruby_llm integration: no #before_request hook reachable from " \
|
|
179
|
+
"#{target.class}. It needs ruby_llm #{SUPPORTED_VERSION}; on 1.x, redact per " \
|
|
180
|
+
"call with DataRedactor.redact before chat.ask."
|
|
117
181
|
end
|
|
118
182
|
end
|
|
119
183
|
end
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/railtie"
|
|
4
|
+
require "data_redactor"
|
|
5
|
+
require "data_redactor/integrations/logger"
|
|
6
|
+
require "data_redactor/integrations/rails"
|
|
7
|
+
|
|
8
|
+
module DataRedactor
|
|
9
|
+
# Rails wiring for data_redactor. Requiring this file installs a Railtie that
|
|
10
|
+
# wraps the Rails logger's formatter and appends a `filter_parameters` entry,
|
|
11
|
+
# so an app gets redaction without writing either integration by hand.
|
|
12
|
+
#
|
|
13
|
+
# Rails is never a runtime dependency of the gem: this file is only loaded
|
|
14
|
+
# when the application requires it, following the same opt-in pattern as
|
|
15
|
+
# every other integration.
|
|
16
|
+
#
|
|
17
|
+
# @example Enable everything with the defaults
|
|
18
|
+
# # Gemfile
|
|
19
|
+
# gem "data_redactor", require: "data_redactor/railtie"
|
|
20
|
+
#
|
|
21
|
+
# @example Tune it from an initializer
|
|
22
|
+
# # config/initializers/data_redactor.rb
|
|
23
|
+
# Rails.application.configure do
|
|
24
|
+
# config.data_redactor.only = [:credentials, :financial]
|
|
25
|
+
# config.data_redactor.placeholder = :tagged
|
|
26
|
+
# config.data_redactor.logger = false # leave the logger alone
|
|
27
|
+
# end
|
|
28
|
+
class Railtie < ::Rails::Railtie
|
|
29
|
+
config.data_redactor = ActiveSupport::OrderedOptions.new
|
|
30
|
+
config.data_redactor.logger = true
|
|
31
|
+
config.data_redactor.filter_parameters = true
|
|
32
|
+
config.data_redactor.placeholder = DataRedactor::PLACEHOLDER_DEFAULT
|
|
33
|
+
|
|
34
|
+
# `only`/`except` are read with `[]` rather than the OrderedOptions reader:
|
|
35
|
+
# `only` and `except` are Enumerable methods, so the reader would return a
|
|
36
|
+
# Method-backed result instead of nil when the app never set them.
|
|
37
|
+
#
|
|
38
|
+
# @api private
|
|
39
|
+
# @param config [ActiveSupport::OrderedOptions] the `config.data_redactor` options
|
|
40
|
+
# @return [Hash] kwargs for {DataRedactor.redact}
|
|
41
|
+
def self.redaction_options(config)
|
|
42
|
+
{
|
|
43
|
+
only: config[:only],
|
|
44
|
+
except: config[:except],
|
|
45
|
+
placeholder: config.placeholder
|
|
46
|
+
}
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
initializer "data_redactor.filter_parameters" do |app|
|
|
50
|
+
options = app.config.data_redactor
|
|
51
|
+
next unless options.filter_parameters
|
|
52
|
+
|
|
53
|
+
app.config.filter_parameters += [
|
|
54
|
+
Integrations::Rails.filter(**Railtie.redaction_options(options))
|
|
55
|
+
]
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Runs after `initialize_logger` so we wrap whatever formatter the app
|
|
59
|
+
# ended up with (Rails' own, lograge, a JSON formatter) rather than
|
|
60
|
+
# replacing it.
|
|
61
|
+
initializer "data_redactor.logger", after: :initialize_logger do |app|
|
|
62
|
+
options = app.config.data_redactor
|
|
63
|
+
next unless options.logger
|
|
64
|
+
|
|
65
|
+
Railtie.wrap_logger(::Rails.logger, Railtie.redaction_options(options))
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# A BroadcastLogger writes to each of its sinks directly, so a formatter set
|
|
69
|
+
# on the broadcast itself never runs — each sink has to be wrapped instead.
|
|
70
|
+
#
|
|
71
|
+
# @api private
|
|
72
|
+
# @param logger [#formatter, ActiveSupport::BroadcastLogger, nil] logger to wrap
|
|
73
|
+
# @param options [Hash] kwargs forwarded to {Integrations::Logger}
|
|
74
|
+
# @return [void]
|
|
75
|
+
def self.wrap_logger(logger, options)
|
|
76
|
+
return if logger.nil?
|
|
77
|
+
|
|
78
|
+
if logger.respond_to?(:broadcasts)
|
|
79
|
+
logger.broadcasts.each { |sink| wrap_logger(sink, options) }
|
|
80
|
+
return
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
return unless logger.respond_to?(:formatter) && logger.respond_to?(:formatter=)
|
|
84
|
+
|
|
85
|
+
inner = logger.formatter || ::Logger::Formatter.new
|
|
86
|
+
return if inner.is_a?(Integrations::Logger)
|
|
87
|
+
|
|
88
|
+
logger.formatter = Integrations::Logger.new(inner: inner, **options)
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
end
|
data/lib/data_redactor.rb
CHANGED
|
@@ -200,6 +200,11 @@ module DataRedactor
|
|
|
200
200
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
201
201
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
202
202
|
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
203
|
+
# @param skip_keys [Symbol, String, Array, nil] hash keys whose values are left
|
|
204
|
+
# alone, matched at any depth and by name, so Symbol and String keys are
|
|
205
|
+
# equivalent. A skipped key's whole subtree is untouched. Use it for
|
|
206
|
+
# structural fields that must survive verbatim — an identifier a downstream
|
|
207
|
+
# API validates, say — not as a filter (that is what +only+/+except+ are).
|
|
203
208
|
# @return [Hash, Array, String, Object] a new structure of the same shape
|
|
204
209
|
# with all String leaves redacted.
|
|
205
210
|
# @raise [ArgumentError] if the structure contains a circular reference.
|
|
@@ -209,8 +214,51 @@ module DataRedactor
|
|
|
209
214
|
#
|
|
210
215
|
# @example Mixed filter
|
|
211
216
|
# DataRedactor.redact_deep(payload, only: :credentials, placeholder: :tagged)
|
|
212
|
-
|
|
213
|
-
|
|
217
|
+
#
|
|
218
|
+
# @example Keep a structural field intact
|
|
219
|
+
# DataRedactor.redact_deep(request, skip_keys: [:model])
|
|
220
|
+
def redact_deep(data, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
221
|
+
_walk(data, only: only, except: except, placeholder: placeholder,
|
|
222
|
+
skip: _skip_set(skip_keys), seen: Set.new)
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
# Redact every String value in a nested Hash/Array structure **in place**.
|
|
226
|
+
#
|
|
227
|
+
# The in-place sibling of {redact_deep}: same traversal and the same
|
|
228
|
+
# filtering, but the containers you pass in are mutated and returned instead
|
|
229
|
+
# of copied. Written for callers that hand a structure to something which
|
|
230
|
+
# ignores return values — a `ruby_llm` +before_request+ hook, a middleware
|
|
231
|
+
# that must edit the params Hash it was given — where {redact_deep} would
|
|
232
|
+
# force a full copy only to overwrite the original with it.
|
|
233
|
+
#
|
|
234
|
+
# Only container contents change: Hash values and Array elements are replaced
|
|
235
|
+
# with redacted Strings. Hash keys are never modified, and the String objects
|
|
236
|
+
# themselves are not mutated (frozen leaves are fine — the container's
|
|
237
|
+
# reference is repointed at a new String).
|
|
238
|
+
#
|
|
239
|
+
# @param data [Hash, Array] the structure to redact in place.
|
|
240
|
+
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
241
|
+
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
242
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
243
|
+
# @param skip_keys [Symbol, String, Array, nil] hash keys to leave alone, as
|
|
244
|
+
# in {redact_deep}.
|
|
245
|
+
# @return [Hash, Array] +data+ itself, redacted.
|
|
246
|
+
# @raise [ArgumentError] if +data+ is not a Hash or an Array, or if the
|
|
247
|
+
# structure contains a circular reference.
|
|
248
|
+
#
|
|
249
|
+
# @example A hook that must mutate what it is given
|
|
250
|
+
# chat.before_request { |payload| DataRedactor.redact_deep!(payload) }
|
|
251
|
+
#
|
|
252
|
+
# @example Scrubbing a params Hash in place
|
|
253
|
+
# DataRedactor.redact_deep!(params, only: :credentials)
|
|
254
|
+
def redact_deep!(data, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
255
|
+
unless data.is_a?(Hash) || data.is_a?(Array)
|
|
256
|
+
raise ArgumentError, "redact_deep!: expected a Hash or Array to mutate, got #{data.class}. " \
|
|
257
|
+
"Use redact or redact_deep for other types."
|
|
258
|
+
end
|
|
259
|
+
|
|
260
|
+
_walk!(data, only: only, except: except, placeholder: placeholder,
|
|
261
|
+
skip: _skip_set(skip_keys), seen: Set.new)
|
|
214
262
|
end
|
|
215
263
|
|
|
216
264
|
# Parse +json_string+, redact every String value in the resulting structure,
|
|
@@ -223,15 +271,16 @@ module DataRedactor
|
|
|
223
271
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
224
272
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
225
273
|
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
274
|
+
# @param skip_keys [Symbol, String, Array, nil] forwarded to {redact_deep}.
|
|
226
275
|
# @return [String] a JSON string with all String values redacted.
|
|
227
276
|
# @raise [JSON::ParserError] if +json_string+ is not valid JSON.
|
|
228
277
|
#
|
|
229
278
|
# @example
|
|
230
279
|
# DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
|
|
231
280
|
# # => '{"email":"[REDACTED]","count":3}'
|
|
232
|
-
def redact_json(json_string, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT)
|
|
281
|
+
def redact_json(json_string, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
233
282
|
parsed = JSON.parse(json_string)
|
|
234
|
-
redacted = redact_deep(parsed, only: only, except: except, placeholder: placeholder)
|
|
283
|
+
redacted = redact_deep(parsed, only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
235
284
|
JSON.generate(redacted)
|
|
236
285
|
end
|
|
237
286
|
|
|
@@ -395,20 +444,56 @@ module DataRedactor
|
|
|
395
444
|
# Depth-first recursive walker for {redact_deep}.
|
|
396
445
|
# +seen+ is a Set of object_ids already on the current traversal stack,
|
|
397
446
|
# used to detect circular references.
|
|
398
|
-
|
|
447
|
+
EMPTY_SKIP = Set.new.freeze
|
|
448
|
+
private_constant :EMPTY_SKIP
|
|
449
|
+
|
|
450
|
+
def _skip_set(keys)
|
|
451
|
+
return EMPTY_SKIP if keys.nil?
|
|
452
|
+
|
|
453
|
+
Set.new(Array(keys).map(&:to_s))
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
def _walk!(node, only:, except:, placeholder:, skip:, seen:)
|
|
457
|
+
case node
|
|
458
|
+
when String
|
|
459
|
+
redact(node, only: only, except: except, placeholder: placeholder)
|
|
460
|
+
when Hash
|
|
461
|
+
raise ArgumentError, "redact_deep!: circular reference detected" if seen.include?(node.object_id)
|
|
462
|
+
seen.add(node.object_id)
|
|
463
|
+
node.each_pair do |k, v|
|
|
464
|
+
next if skip.include?(k.to_s)
|
|
465
|
+
|
|
466
|
+
node[k] = _walk!(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
467
|
+
end
|
|
468
|
+
seen.delete(node.object_id)
|
|
469
|
+
node
|
|
470
|
+
when Array
|
|
471
|
+
raise ArgumentError, "redact_deep!: circular reference detected" if seen.include?(node.object_id)
|
|
472
|
+
seen.add(node.object_id)
|
|
473
|
+
node.each_index do |i|
|
|
474
|
+
node[i] = _walk!(node[i], only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
475
|
+
end
|
|
476
|
+
seen.delete(node.object_id)
|
|
477
|
+
node
|
|
478
|
+
else
|
|
479
|
+
node
|
|
480
|
+
end
|
|
481
|
+
end
|
|
482
|
+
|
|
483
|
+
def _walk(node, only:, except:, placeholder:, skip:, seen:)
|
|
399
484
|
case node
|
|
400
485
|
when String
|
|
401
486
|
redact(node, only: only, except: except, placeholder: placeholder)
|
|
402
487
|
when Hash
|
|
403
488
|
raise ArgumentError, "redact_deep: circular reference detected" if seen.include?(node.object_id)
|
|
404
489
|
seen.add(node.object_id)
|
|
405
|
-
result = node
|
|
490
|
+
result = _walk_hash(node, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
406
491
|
seen.delete(node.object_id)
|
|
407
492
|
result
|
|
408
493
|
when Array
|
|
409
494
|
raise ArgumentError, "redact_deep: circular reference detected" if seen.include?(node.object_id)
|
|
410
495
|
seen.add(node.object_id)
|
|
411
|
-
result = node.map { |v| _walk(v, only: only, except: except, placeholder: placeholder, seen: seen) }
|
|
496
|
+
result = node.map { |v| _walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen) }
|
|
412
497
|
seen.delete(node.object_id)
|
|
413
498
|
result
|
|
414
499
|
else
|
|
@@ -416,6 +501,22 @@ module DataRedactor
|
|
|
416
501
|
end
|
|
417
502
|
end
|
|
418
503
|
|
|
504
|
+
# transform_values is the fast path and cannot see keys, so it stays in use
|
|
505
|
+
# whenever nothing is being skipped — which is every call that predates skip_keys.
|
|
506
|
+
def _walk_hash(node, only:, except:, placeholder:, skip:, seen:)
|
|
507
|
+
if skip.empty?
|
|
508
|
+
node.transform_values { |v| _walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen) }
|
|
509
|
+
else
|
|
510
|
+
node.each_with_object({}) do |(k, v), acc|
|
|
511
|
+
acc[k] = if skip.include?(k.to_s)
|
|
512
|
+
v
|
|
513
|
+
else
|
|
514
|
+
_walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
515
|
+
end
|
|
516
|
+
end
|
|
517
|
+
end
|
|
518
|
+
end
|
|
519
|
+
|
|
419
520
|
# @api private
|
|
420
521
|
def pattern_enabled?(name, tag_bit, only_present, only_bits, only_names,
|
|
421
522
|
except_bits, except_names)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_redactor
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.18.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Daniele Frisanco
|
|
@@ -79,6 +79,20 @@ dependencies:
|
|
|
79
79
|
- - ">="
|
|
80
80
|
- !ruby/object:Gem::Version
|
|
81
81
|
version: '2.0'
|
|
82
|
+
- !ruby/object:Gem::Dependency
|
|
83
|
+
name: logger
|
|
84
|
+
requirement: !ruby/object:Gem::Requirement
|
|
85
|
+
requirements:
|
|
86
|
+
- - ">="
|
|
87
|
+
- !ruby/object:Gem::Version
|
|
88
|
+
version: '1.5'
|
|
89
|
+
type: :development
|
|
90
|
+
prerelease: false
|
|
91
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
92
|
+
requirements:
|
|
93
|
+
- - ">="
|
|
94
|
+
- !ruby/object:Gem::Version
|
|
95
|
+
version: '1.5'
|
|
82
96
|
- !ruby/object:Gem::Dependency
|
|
83
97
|
name: benchmark-ips
|
|
84
98
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -108,7 +122,7 @@ dependencies:
|
|
|
108
122
|
- !ruby/object:Gem::Version
|
|
109
123
|
version: '0.2'
|
|
110
124
|
description: A Ruby gem with a C extension for high-performance scanning and redaction
|
|
111
|
-
of
|
|
125
|
+
of sensitive data — API keys, tokens, credentials, IBANs, national IDs, emails,
|
|
112
126
|
phone numbers, and PII from 15+ countries. Optional Logger formatter, Rails filter_parameters
|
|
113
127
|
adapter, and Rack middleware. Designed to sanitize text before sending to LLMs,
|
|
114
128
|
logging systems, or any public/third-party API.
|
|
@@ -146,6 +160,7 @@ files:
|
|
|
146
160
|
- lib/data_redactor/integrations/rails.rb
|
|
147
161
|
- lib/data_redactor/integrations/ruby_llm.rb
|
|
148
162
|
- lib/data_redactor/name_pattern.rb
|
|
163
|
+
- lib/data_redactor/railtie.rb
|
|
149
164
|
- lib/data_redactor/refinements.rb
|
|
150
165
|
- lib/data_redactor/version.rb
|
|
151
166
|
homepage: https://github.com/danielefrisanco/data_redactor
|