data_redactor 0.17.0-x86_64-linux → 0.18.0-x86_64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +122 -1
- data/README.md +135 -46
- data/lib/data_redactor/3.0/data_redactor.so +0 -0
- data/lib/data_redactor/3.1/data_redactor.so +0 -0
- data/lib/data_redactor/3.2/data_redactor.so +0 -0
- data/lib/data_redactor/3.3/data_redactor.so +0 -0
- data/lib/data_redactor/3.4/data_redactor.so +0 -0
- data/lib/data_redactor/4.0/data_redactor.so +0 -0
- data/lib/data_redactor/integrations/logger.rb +11 -1
- data/lib/data_redactor/integrations/ruby_llm.rb +152 -88
- data/lib/data_redactor/railtie.rb +91 -0
- data/lib/data_redactor/version.rb +1 -1
- data/lib/data_redactor.rb +108 -7
- metadata +17 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 737bd3a71303be2d58c711ca85a090e510dcf558a1c7f2500db634e9b0418205
|
|
4
|
+
data.tar.gz: f38bb011758fae6bfad6280c0dd16248729adb86cd924a65d146f8ebc9aa7e3f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c16cd72b559998ceebad0068829efd611ffa41f7de09a825a8ebd214d4377fa975b22335a5c3e14b0b30723bd6d9df49fd84d5ba24f9ab2364427ebc871ae7ba
|
|
7
|
+
data.tar.gz: 4e9982034e40f5d0436075a3568d47dae621aac5d50fafa1a1fab566eb189c5188f06f6238a74cdc1489fd1eac710ee9e8e6b110a672d2ee730afaac759a56e5
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,126 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.18.0] - 2026-09-10
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **RubyLLM integration rebuilt on the public request hook.**
|
|
14
|
+
`Integrations::RubyLLM.chat(...)` is a drop-in for `RubyLLM.chat` that returns a
|
|
15
|
+
chat whose every request is redacted; `attach!(target)` does the same for a chat,
|
|
16
|
+
an `Agent`, or an `acts_as_chat` record you were handed (it finds the chat inside
|
|
17
|
+
whichever you pass); `hook(...)` returns the callback itself. All three sit on
|
|
18
|
+
`Chat#before_request` (ruby_llm 2.0+), which receives the fully rendered payload
|
|
19
|
+
and lets a callback edit it in place — **no monkeypatching**. This is the only way
|
|
20
|
+
to scrub tool results, which an agent inlines into the next request and the user
|
|
21
|
+
never typed. Redaction is per chat and applies to the rendered payload only, so
|
|
22
|
+
the stored conversation is untouched. RubyLLM runs the hooks on conversation
|
|
23
|
+
**compaction** requests too (`Protocol#compact`), so the history a summarisation
|
|
24
|
+
call sends is redacted as well.
|
|
25
|
+
- **`skip_keys:` on `redact_deep`, `redact_deep!` and `redact_json`.** Hash keys
|
|
26
|
+
whose values are left verbatim, matched by name at any depth (Symbol and String
|
|
27
|
+
keys are equivalent) and skipping the key's whole subtree. For structural fields
|
|
28
|
+
a receiving API validates. It is a denylist, not a filter — anything unlisted is
|
|
29
|
+
still redacted, so an unfamiliar field cannot slip through. The RubyLLM hook
|
|
30
|
+
defaults to `skip_keys: [:model]`: dated model ids such as
|
|
31
|
+
`claude-haiku-4-5-20251001` end in eight digits, which the national-ID patterns
|
|
32
|
+
match, and a provider rejects a redacted model id.
|
|
33
|
+
- **`DataRedactor.redact_deep!` — in-place deep redaction.** The sibling of
|
|
34
|
+
`redact_deep` for callers that must scrub the structure they were handed rather
|
|
35
|
+
than return a copy: hooks and middleware whose return value is discarded (a
|
|
36
|
+
`ruby_llm` `before_request` hook is the motivating case). Mutates and returns the
|
|
37
|
+
same Hash/Array; only container contents change — hash keys are untouched and
|
|
38
|
+
String leaves are replaced rather than mutated, so frozen leaves are safe. Raises
|
|
39
|
+
`ArgumentError` on anything that is not a Hash or Array, pointing at `redact` /
|
|
40
|
+
`redact_deep`. The copy-returning methods are unchanged.
|
|
41
|
+
- **Rails Railtie — zero-config onboarding.** `require "data_redactor/railtie"`
|
|
42
|
+
(e.g. `gem "data_redactor", require: "data_redactor/railtie"`) now wires both
|
|
43
|
+
Rails surfaces automatically: the redacting Logger formatter and a
|
|
44
|
+
`config.filter_parameters` entry. Everything is tunable from an initializer via
|
|
45
|
+
`config.data_redactor.{logger,filter_parameters,only,except,placeholder}`, and
|
|
46
|
+
either surface can be switched off. The logger initializer runs *after*
|
|
47
|
+
`initialize_logger` and **wraps** the app's existing formatter rather than
|
|
48
|
+
replacing it, so lograge/JSON formatters keep working; it descends into
|
|
49
|
+
`ActiveSupport::BroadcastLogger` and wraps each sink, since a formatter set on
|
|
50
|
+
the broadcast itself never runs. Already-wrapped formatters are left alone.
|
|
51
|
+
Rails is a development dependency only — the gem keeps zero runtime deps, and
|
|
52
|
+
the file is loaded only when the app requires it.
|
|
53
|
+
- **CI: the Ruby 2.7 floor is now tested, not just claimed.** New `ruby-floor`
|
|
54
|
+
job compiles the C extension and runs the full suite on Ruby 2.7 and 3.0 — the
|
|
55
|
+
bottom of the `required_ruby_version >= 2.7` range, which the 3.1–3.4 `test`
|
|
56
|
+
matrix never covered. It pins `ubuntu-24.04` (ruby-builder has no 2.7/3.0
|
|
57
|
+
binaries for 26.04) and resolves dependencies without the committed lockfile,
|
|
58
|
+
which pins a Rails that requires Ruby >= 3.1. No source changes were needed:
|
|
59
|
+
both Rubies were already green.
|
|
60
|
+
- **CI: Ruby 4.0 is tested, and `ruby-head` is watched.** 4.0 joins the `test`
|
|
61
|
+
matrix as a supported release (green as-is, no source changes) — resolving its
|
|
62
|
+
own dependency set, since the committed lockfile's nokogiri caps at
|
|
63
|
+
`< 3.5.dev` and `bundler-cache` installs frozen — and a new
|
|
64
|
+
allow-failure `ruby-next` job compiles the extension and runs the specs
|
|
65
|
+
against `ruby-head` so C-API or stdlib breakage surfaces months before a
|
|
66
|
+
release week rather than during one. `ruby-next` excludes Rails: nokogiri has
|
|
67
|
+
no precompiled gem for head, and letting it fail there would mask the signal
|
|
68
|
+
the job exists for. Precompiled binaries still cover 3.1–3.4 only, so Ruby 4.0
|
|
69
|
+
installs the source gem for now.
|
|
70
|
+
- **Project wiki.** Set up the GitHub wiki as the home for deep material so the
|
|
71
|
+
README stays a focused entry point: pattern catalogue (grouped by tag/country),
|
|
72
|
+
C engine internals (NFA → bytecode → lazy DFA, the v19 story), integration
|
|
73
|
+
guides, a dedicated RubyLLM page (per-call + transparent `install!`), custom /
|
|
74
|
+
name-pattern cookbook, benchmark methodology, and FAQ. README now links to it
|
|
75
|
+
and promotes RubyLLM higher in Usage and in the use-case list.
|
|
76
|
+
|
|
77
|
+
### Removed
|
|
78
|
+
- **`Integrations::RubyLLM.install!` (transparent mode).** It prepended
|
|
79
|
+
`RubyLLM::Protocol#render`, an internal that only ever existed in ruby_llm 2.0
|
|
80
|
+
pre-release, and its whole justification was that RubyLLM exposed no public seam
|
|
81
|
+
to rewrite an outbound request. 2.0 exposes one (`Chat#before_request`), so the
|
|
82
|
+
monkeypatch is gone. `install!` still exists for one release and raises a message
|
|
83
|
+
naming `chat` / `attach!`; it will be deleted in the next minor. The upstream
|
|
84
|
+
request we were tracking for this, [crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765),
|
|
85
|
+
was closed on 2026-08-12 as superseded by 2.0's instrumentation surface — which is
|
|
86
|
+
observe-only and does not solve payload rewriting, but `before_request` does.
|
|
87
|
+
Removing a shipped public method would normally force a major bump; this one
|
|
88
|
+
lands in a minor because `RubyLLM::Protocol` exists in no *stable* `ruby_llm`
|
|
89
|
+
release. The only versions carrying it are the 2.0 release candidates published
|
|
90
|
+
days before this change, so `install!` worked for nobody on 1.x and for almost
|
|
91
|
+
nobody on 2.0 — and `attach!` covers the same ground on the same versions.
|
|
92
|
+
|
|
93
|
+
### Changed
|
|
94
|
+
- **Engine re-entrancy: selective-merge cursors are now per-call, not per-thread.**
|
|
95
|
+
The digit-run and IBAN-union passes kept their non-overlap cursors in the
|
|
96
|
+
per-thread scan cache; they now live in a stack-allocated context, one set per
|
|
97
|
+
`mm_scan` call. Output is byte-for-byte unchanged — this is an internal
|
|
98
|
+
threading change with no API or behaviour difference. It makes the C engine
|
|
99
|
+
genuinely re-entrant (a prerequisite for Ractors and for widening the
|
|
100
|
+
GVL-free region) rather than relying on thread-local storage to keep concurrent
|
|
101
|
+
scans apart, and lifts the per-thread cache lookup out of the per-pattern inner
|
|
102
|
+
loop. New spec asserts merge-cursor output stays call-private under parallel
|
|
103
|
+
GVL-released digit/IBAN load.
|
|
104
|
+
|
|
105
|
+
### Fixed
|
|
106
|
+
- **`Integrations::RubyLLM.install!` now states the version it actually needs.**
|
|
107
|
+
It declared `~> 1.16`, but `RubyLLM::Protocol` — the chokepoint it prepends —
|
|
108
|
+
exists only in the 2.0 line; the 1.x series assembles the request
|
|
109
|
+
inside `Provider#complete`. On a real `ruby_llm` 1.16.0 the version check
|
|
110
|
+
passed and the next line died with `NameError: uninitialized constant
|
|
111
|
+
RubyLLM::Protocol`. The pin is now `>= 2.0.0.pre`, so 1.x users get a clear
|
|
112
|
+
message pointing at per-call `DataRedactor.redact`, and a missing `Protocol`
|
|
113
|
+
constant raises the same fail-fast error as a missing `Protocol#render`.
|
|
114
|
+
Redaction was never affected — the integration failed loudly at `install!`
|
|
115
|
+
rather than leaking — but it worked on no released `ruby_llm` version.
|
|
116
|
+
- **Logger integration no longer raises `LoadError` on Ruby 4.0.** Ruby 4.0
|
|
117
|
+
demoted `logger` from a default gem to a bundled one, so `require "logger"`
|
|
118
|
+
only resolves when something declares it — and this gem declares no runtime
|
|
119
|
+
dependencies. `integrations/logger.rb` now soft-requires it: anyone assigning
|
|
120
|
+
the redacting formatter already holds a `::Logger`, so their own require
|
|
121
|
+
defines the constant. Rails apps were never affected (activesupport declares
|
|
122
|
+
`logger`); plain Ruby 4.0 apps hit it the moment they loaded the integration.
|
|
123
|
+
The gem stays dependency-free. Found by the new `ruby-next` job on its first
|
|
124
|
+
run. **If you use the Logger integration on Ruby 4.0 without Rails, add
|
|
125
|
+
`gem "logger"` to your Gemfile** — Ruby 4.0 requires that of every caller,
|
|
126
|
+
not just this gem.
|
|
127
|
+
- Gemspec description said "85 sensitive patterns" while the engine ships 89.
|
|
128
|
+
The description no longer hardcodes a count, so it can't drift again.
|
|
129
|
+
|
|
10
130
|
## [0.17.0] - 2026-06-21
|
|
11
131
|
|
|
12
132
|
### Added
|
|
@@ -354,7 +474,8 @@ features as 0.7.1 plus the pipeline fix.
|
|
|
354
474
|
- `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
|
|
355
475
|
- RSpec suite with one example per pattern.
|
|
356
476
|
|
|
357
|
-
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.
|
|
477
|
+
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.18.0...HEAD
|
|
478
|
+
[0.18.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.17.0...v0.18.0
|
|
358
479
|
[0.17.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.16.0...v0.17.0
|
|
359
480
|
[0.16.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...v0.16.0
|
|
360
481
|
[0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
|
data/README.md
CHANGED
|
@@ -6,6 +6,8 @@
|
|
|
6
6
|
|
|
7
7
|
A Ruby gem with a C extension for high-performance regex-based redaction of sensitive data from strings.
|
|
8
8
|
|
|
9
|
+
📚 **Deeper docs live in the [wiki](https://github.com/danielefrisanco/data_redactor/wiki):** the full [pattern catalogue](https://github.com/danielefrisanco/data_redactor/wiki/Pattern-Catalogue), [C engine internals](https://github.com/danielefrisanco/data_redactor/wiki/C-Engine-Internals), [integration guides](https://github.com/danielefrisanco/data_redactor/wiki/Integration-Guides) (incl. [RubyLLM](https://github.com/danielefrisanco/data_redactor/wiki/RubyLLM-Integration)), [benchmark methodology](https://github.com/danielefrisanco/data_redactor/wiki/Performance-and-Benchmarks), and [FAQ](https://github.com/danielefrisanco/data_redactor/wiki/FAQ). The [API reference](https://danielefrisanco.github.io/data_redactor/) is on GitHub Pages.
|
|
10
|
+
|
|
9
11
|
> 📄 The engineering behind the v19 matching engine is written up as an experience
|
|
10
12
|
> report, *"The Fastest Engine Is Not the Shippable Engine: Replacing a Regex Engine
|
|
11
13
|
> for Data Redaction Under Production Constraints,"* currently under review at
|
|
@@ -31,12 +33,15 @@ Rack. You can also register your own patterns — at boot or at runtime from any
|
|
|
31
33
|
|
|
32
34
|
- **Log scrubbing** — drop the `Logger` formatter in so no secret or PII ever
|
|
33
35
|
reaches disk or your log aggregator.
|
|
34
|
-
- **Rails
|
|
35
|
-
|
|
36
|
+
- **Rails, zero-config** — one Gemfile line wires both the log formatter and
|
|
37
|
+
`filter_parameters`; unlike `filter_parameters` alone, it also catches secrets
|
|
38
|
+
in free text you can't enumerate by key.
|
|
36
39
|
- **HTTP request/response sanitising** — Rack middleware scrubs response bodies
|
|
37
40
|
and sensitive headers in flight.
|
|
38
|
-
- **
|
|
39
|
-
|
|
41
|
+
- **Scrubbing prompts before an LLM** — with [RubyLLM](#rubyllm--redact-before-it-reaches-the-model), redact every
|
|
42
|
+
prompt, system instruction, and tool result before it reaches the model —
|
|
43
|
+
per-call, or on every request through RubyLLM's own request hook. Also `redact_deep` /
|
|
44
|
+
`redact_json` over any params hash or JSON body before it leaves the process.
|
|
40
45
|
- **Compliance & auditing** — `scan` reports every match with byte offsets, tag,
|
|
41
46
|
and pattern name without changing the text, for false-positive tuning.
|
|
42
47
|
- **Internal identifiers** — register company-specific patterns (`add_pattern`)
|
|
@@ -58,6 +63,70 @@ custom patterns, deep/JSON traversal, and the Logger / Rack / Rails / LLM
|
|
|
58
63
|
integrations. Run any of them with `bundle exec ruby examples/<name>.rb` (see
|
|
59
64
|
[examples/README.md](examples/README.md)).
|
|
60
65
|
|
|
66
|
+
### RubyLLM — redact before it reaches the model
|
|
67
|
+
|
|
68
|
+
[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — Anthropic, OpenAI, Gemini, Bedrock, and more — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
require "ruby_llm"
|
|
72
|
+
require "data_redactor"
|
|
73
|
+
|
|
74
|
+
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
75
|
+
chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
|
|
76
|
+
|
|
77
|
+
user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
|
|
78
|
+
chat.ask(DataRedactor.redact(user_input))
|
|
79
|
+
# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into, and it's the recommended approach.
|
|
83
|
+
|
|
84
|
+
#### Every request, no per-call wrapping
|
|
85
|
+
|
|
86
|
+
To redact **every** outbound request — including the system prompt, the whole history, tool definitions, and any file contents or shell-command output an agent feeds back as a tool result — build the chat through the integration:
|
|
87
|
+
|
|
88
|
+
> **Requires `ruby_llm` 2.0 or newer** (verified against `2.0.0.rc2`). 1.x has no request hook at all; on 1.x use the per-call `DataRedactor.redact` form above.
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
require "ruby_llm"
|
|
92
|
+
require "data_redactor/integrations/ruby_llm"
|
|
93
|
+
|
|
94
|
+
chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
95
|
+
chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
`chat` is a drop-in for `RubyLLM.chat` — every argument is forwarded and you get a real chat back, so the fluent API keeps working. It registers RubyLLM's public [`before_request`](https://rubyllm.com) hook, which receives the fully rendered payload just before it's posted and lets a callback edit it in place. **Nothing is monkeypatched.**
|
|
99
|
+
|
|
100
|
+
Already holding a chat, an [`Agent`](https://rubyllm.com), or an `acts_as_chat` record? Attach to it — `attach!` finds the chat inside whichever you pass and returns your object:
|
|
101
|
+
|
|
102
|
+
```ruby
|
|
103
|
+
DataRedactor::Integrations::RubyLLM.attach!(agent, only: [:financial])
|
|
104
|
+
|
|
105
|
+
# or register the callback yourself
|
|
106
|
+
chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
For an **agent**, the tidiest spelling is to hand it a chat that's already redacted — an agent keeps the chat you pass and layers its own configuration on top:
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
113
|
+
agent = SupportAgent.new(chat: chat)
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Every request in the agent's tool loop is then redacted — and an agent issues one per turn, since `Chat#complete` steps until the loop settles.
|
|
117
|
+
|
|
118
|
+
This is the only way to scrub **tool results** — the file an agent read or the command it ran gets inlined into the *next* request, and the user never typed it, so per-call redaction can't reach it.
|
|
119
|
+
|
|
120
|
+
Five things to know:
|
|
121
|
+
|
|
122
|
+
- **It's per chat.** RubyLLM has no global callback registry, so a chat you neither built with `.chat` nor passed to `attach!` is not redacted.
|
|
123
|
+
- **Chat only.** Embeddings, moderation, image generation and transcription don't run request hooks.
|
|
124
|
+
- **The `model` key is skipped by default** (`skip_keys:`). Dated model ids like `claude-haiku-4-5-20251001` end in eight digits, which the national-ID patterns match, and a provider rejects a redacted model id. Pass `skip_keys: [:model, :metadata]` to protect more; pass `skip_keys: []` to redact everything.
|
|
125
|
+
- **A digits-only tool-call id would be redacted.** `_` counts as a boundary, so an id like `toolu_012345678` matches a national-ID pattern and breaks the `tool_use_id` correlation the provider validates. Anthropic and OpenAI mint mixed alphanumeric ids (`toolu_01A09q…`, `call_abc123…`), which are untouched; if a provider ever changes that, add `:id` to `skip_keys:`.
|
|
126
|
+
- **Base64 attachments** (PDFs, images, audio sent inline) and **URL-referenced files** are not redacted — the sensitive bytes are encoded or remote, so patterns cannot see them.
|
|
127
|
+
|
|
128
|
+
Redaction applies to the rendered payload and nothing is persisted, so your stored conversation keeps its original text — this scrubs the wire, per request. (We asked for a connection-middleware hook in [crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765); it was declined in favour of the instrumentation surface, but `before_request` gives us what we needed.)
|
|
129
|
+
|
|
61
130
|
### Filtering by tag or pattern name
|
|
62
131
|
|
|
63
132
|
`only:` and `except:` both accept a single value or an Array, mixing **Symbols** (tag names) and **Strings** (specific pattern names).
|
|
@@ -171,6 +240,28 @@ DataRedactor.redact_deep(params, only: :credentials)
|
|
|
171
240
|
DataRedactor.redact_deep(payload, except: :network, placeholder: :tagged)
|
|
172
241
|
```
|
|
173
242
|
|
|
243
|
+
Need the original scrubbed rather than a copy? `redact_deep!` is the in-place sibling — it mutates the Hash or Array you hand it and returns that same object. Use it when the caller ignores your return value, such as a hook or a middleware that must edit the structure it was given:
|
|
244
|
+
|
|
245
|
+
```ruby
|
|
246
|
+
params = { "user" => { "email" => "alice@example.com" } }
|
|
247
|
+
DataRedactor.redact_deep!(params)
|
|
248
|
+
params # => { "user" => { "email" => "[REDACTED]" } }
|
|
249
|
+
|
|
250
|
+
# Same filters; raises ArgumentError on anything that isn't a Hash or Array
|
|
251
|
+
DataRedactor.redact_deep!(payload, only: :credentials)
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Only the containers change: hash keys are untouched and string leaves are *replaced*, never mutated, so frozen strings are safe.
|
|
255
|
+
|
|
256
|
+
Some fields must survive verbatim — an identifier the receiving API validates, for instance. `skip_keys:` leaves them alone, matched by name at any depth (Symbol and String keys are equivalent), skipping the key's whole subtree:
|
|
257
|
+
|
|
258
|
+
```ruby
|
|
259
|
+
DataRedactor.redact_deep(request, skip_keys: [:model])
|
|
260
|
+
# "claude-haiku-4-5-20251001" survives; everything else is still redacted
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
It's a denylist, not a filter: anything you don't list is still redacted, so a field you've never seen can't slip through. Use `only:`/`except:` to choose *what* counts as sensitive.
|
|
264
|
+
|
|
174
265
|
```ruby
|
|
175
266
|
# JSON string — parse → redact_deep → re-serialise
|
|
176
267
|
safe_json = DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
|
|
@@ -267,6 +358,39 @@ DataRedactor.name_pattern("Mario", "Rossi", middle: "Luigi")
|
|
|
267
358
|
|
|
268
359
|
Optional adapters for Logger, Rails, and Rack. None are loaded automatically — `require` only what you use, and the gem adds zero runtime dependencies in the gemspec.
|
|
269
360
|
|
|
361
|
+
### Rails (zero-config)
|
|
362
|
+
|
|
363
|
+
One Gemfile line wires both Rails surfaces — the log formatter and `filter_parameters`:
|
|
364
|
+
|
|
365
|
+
```ruby
|
|
366
|
+
# Gemfile
|
|
367
|
+
gem "data_redactor", require: "data_redactor/railtie"
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
That's the whole setup. `filter_parameters` on its own only redacts the keys you name; data_redactor also catches the card number sitting inside an exception message or a free-text field you can't enumerate in advance.
|
|
371
|
+
|
|
372
|
+
Tune it from an initializer:
|
|
373
|
+
|
|
374
|
+
```ruby
|
|
375
|
+
# config/initializers/data_redactor.rb
|
|
376
|
+
Rails.application.configure do
|
|
377
|
+
config.data_redactor.only = [:credentials, :financial]
|
|
378
|
+
config.data_redactor.placeholder = :tagged
|
|
379
|
+
config.data_redactor.logger = false # leave the logger alone
|
|
380
|
+
end
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
| Option | Default | Effect |
|
|
384
|
+
| --- | --- | --- |
|
|
385
|
+
| `logger` | `true` | Wrap the Rails logger's formatter |
|
|
386
|
+
| `filter_parameters` | `true` | Append a redacting `filter_parameters` entry |
|
|
387
|
+
| `only` / `except` | `nil` | Forwarded to `DataRedactor.redact` |
|
|
388
|
+
| `placeholder` | `"[REDACTED]"` | Forwarded to `DataRedactor.redact` |
|
|
389
|
+
|
|
390
|
+
The logger initializer runs after Rails' own, so it **wraps** whatever formatter your app ended up with (lograge, a JSON formatter) rather than replacing it, and it descends into `ActiveSupport::BroadcastLogger` to wrap each sink. Rails stays a development dependency — the Railtie file is only loaded when you require it.
|
|
391
|
+
|
|
392
|
+
Prefer wiring the two surfaces by hand? Both are documented below.
|
|
393
|
+
|
|
270
394
|
### Logger formatter
|
|
271
395
|
|
|
272
396
|
Drop-in `Logger::Formatter` replacement that scrubs every emitted line:
|
|
@@ -282,6 +406,8 @@ logger.info("Auth failed for alice@example.com")
|
|
|
282
406
|
|
|
283
407
|
Wraps an inner formatter (defaults to `Logger::Formatter`), so it composes with structured loggers. Forwards `only:`, `except:`, `placeholder:` to `DataRedactor.redact`. Exception messages and arbitrary objects are scrubbed too — the wrapped object is passed unchanged to the inner formatter so the exception cause chain is preserved; only the rendered string is redacted.
|
|
284
408
|
|
|
409
|
+
> **Ruby 4.0:** `logger` became a bundled gem rather than a default one, so `require "logger"` resolves only when your Gemfile declares it. Add `gem "logger"` if you use this integration outside Rails — Ruby 4.0 asks that of every caller, not just this gem. Rails apps already have it via `activesupport`.
|
|
410
|
+
|
|
285
411
|
### Rails `filter_parameters` adapter
|
|
286
412
|
|
|
287
413
|
```ruby
|
|
@@ -340,46 +466,7 @@ client.chat(parameters: { model: "gpt-4o", messages: safe_messages })
|
|
|
340
466
|
safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
|
|
341
467
|
```
|
|
342
468
|
|
|
343
|
-
`content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
|
|
344
|
-
|
|
345
|
-
### RubyLLM
|
|
346
|
-
|
|
347
|
-
[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
|
|
348
|
-
|
|
349
|
-
```ruby
|
|
350
|
-
require "ruby_llm"
|
|
351
|
-
require "data_redactor"
|
|
352
|
-
|
|
353
|
-
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
354
|
-
chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
|
|
355
|
-
|
|
356
|
-
user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
|
|
357
|
-
chat.ask(DataRedactor.redact(user_input))
|
|
358
|
-
# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
|
|
359
|
-
```
|
|
360
|
-
|
|
361
|
-
Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into, and it's the recommended approach.
|
|
362
|
-
|
|
363
|
-
#### Transparent mode (every request, no per-call wrapping)
|
|
364
|
-
|
|
365
|
-
If you'd rather redact **every** outbound request automatically — including the system prompt, tool definitions, and any file contents or shell-command output an agent feeds back as a tool result — opt into the monkeypatch:
|
|
366
|
-
|
|
367
|
-
```ruby
|
|
368
|
-
require "ruby_llm"
|
|
369
|
-
require "data_redactor/integrations/ruby_llm"
|
|
370
|
-
|
|
371
|
-
DataRedactor::Integrations::RubyLLM.install! # once, at boot
|
|
372
|
-
|
|
373
|
-
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
374
|
-
chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
375
|
-
```
|
|
376
|
-
|
|
377
|
-
`install!` prepends a patch onto `RubyLLM::Protocol#render` — the one point where every provider (Anthropic, OpenAI, Gemini, Bedrock, Responses) has assembled its final request — and deep-redacts the payload before it's posted. It forwards `only:`/`except:`/`placeholder:`, is idempotent, and **fails fast** at `install!` if an unsupported `ruby_llm` version is loaded or the internal API has moved (so it never silently leaks).
|
|
378
|
-
|
|
379
|
-
Two caveats, by design:
|
|
380
|
-
|
|
381
|
-
- **It's a monkeypatch on RubyLLM internals**, pinned to a supported version range. Prefer per-call `DataRedactor.redact` (above) unless you specifically need transparency. RubyLLM does not yet expose a public request hook ([crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765) tracks the connection-middleware hook that would let us drop the patch).
|
|
382
|
-
- **Base64 attachments** (PDFs, images, audio sent inline) and **URL-referenced files** are not redacted — the sensitive bytes are encoded or remote, so patterns cannot see them.
|
|
469
|
+
`content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works. For [RubyLLM](#rubyllm--redact-before-it-reaches-the-model) — a unified client across every provider — see the dedicated section above, including whole-request redaction through its `before_request` hook.
|
|
383
470
|
|
|
384
471
|
## Detected patterns (89 total)
|
|
385
472
|
|
|
@@ -480,6 +567,7 @@ redactor/
|
|
|
480
567
|
│ └── data_redactor/
|
|
481
568
|
│ ├── version.rb
|
|
482
569
|
│ ├── name_pattern.rb # name_pattern helper — generates a name regex for add_pattern
|
|
570
|
+
│ ├── railtie.rb # opt-in Rails Railtie — wires logger + filter_parameters
|
|
483
571
|
│ └── integrations/ # soft-required Logger / Rails / Rack adapters
|
|
484
572
|
├── ext/
|
|
485
573
|
│ └── data_redactor/
|
|
@@ -502,6 +590,7 @@ redactor/
|
|
|
502
590
|
│ ├── logger.rb # Logger::Formatter integration
|
|
503
591
|
│ ├── rack_middleware.rb # Rack middleware (body + headers)
|
|
504
592
|
│ ├── rails_filter.rb # filter_parameters adapter
|
|
593
|
+
│ ├── rails_logger.rb # Rack middleware + redacting Logger, end-to-end
|
|
505
594
|
│ └── llm_payload.rb # Claude / OpenAI message + response redaction
|
|
506
595
|
├── benchmark/ # Repo-only perf scripts (not packaged in the gem)
|
|
507
596
|
│ ├── README.md # How to run, what each script measures
|
|
@@ -517,7 +606,7 @@ redactor/
|
|
|
517
606
|
|
|
518
607
|
## Requirements
|
|
519
608
|
|
|
520
|
-
- Ruby >= 2.7
|
|
609
|
+
- Ruby >= 2.7 — CI runs the full suite on 2.7 and 3.0 as well as 3.1–3.4 and 4.0
|
|
521
610
|
- A C compiler (`gcc` or `clang`) — only required when installing the source gem
|
|
522
611
|
- POSIX `regex.h` — only required when installing the source gem (standard on Linux and macOS)
|
|
523
612
|
|
|
@@ -541,7 +630,7 @@ That's it — there is nothing extra to configure for precompiled binaries. Bund
|
|
|
541
630
|
|
|
542
631
|
### Supported precompiled targets
|
|
543
632
|
|
|
544
|
-
Each precompiled gem ships compiled binaries for Ruby 3.1, 3.2, 3.3, and 3.4.
|
|
633
|
+
Each precompiled gem ships compiled binaries for Ruby 3.1, 3.2, 3.3, and 3.4. Ruby 2.7 and 3.0 are supported and tested, but fall back to the source gem — bundler compiles the extension on install, so those need a C compiler.
|
|
545
634
|
|
|
546
635
|
| Platform | Targets |
|
|
547
636
|
|---|---|
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -1,4 +1,14 @@
|
|
|
1
|
-
|
|
1
|
+
# Ruby 4.0 demoted logger from a default gem to a bundled one: it still ships
|
|
2
|
+
# with Ruby, but under Bundler it only resolves when something declares it, and
|
|
3
|
+
# this gem declares no runtime dependencies. Swallowing the LoadError costs
|
|
4
|
+
# nothing — anyone assigning this formatter holds a ::Logger instance already,
|
|
5
|
+
# so their own require has defined the constant this file needs.
|
|
6
|
+
begin
|
|
7
|
+
require "logger"
|
|
8
|
+
rescue LoadError
|
|
9
|
+
nil
|
|
10
|
+
end
|
|
11
|
+
|
|
2
12
|
require "data_redactor"
|
|
3
13
|
|
|
4
14
|
module DataRedactor
|
|
@@ -2,118 +2,182 @@ require "data_redactor"
|
|
|
2
2
|
|
|
3
3
|
module DataRedactor
|
|
4
4
|
module Integrations
|
|
5
|
-
#
|
|
5
|
+
# Outbound redaction for the `ruby_llm` gem (crmne/ruby_llm), built on
|
|
6
|
+
# RubyLLM's public request hook — nothing here monkeypatches RubyLLM.
|
|
6
7
|
#
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
#
|
|
10
|
-
#
|
|
11
|
-
#
|
|
12
|
-
# knowing any provider-specific shape.
|
|
8
|
+
# `chat.before_request { |payload| ... }` (ruby_llm 2.0+) hands a callback the
|
|
9
|
+
# fully rendered request payload — after all RubyLLM formatting and
|
|
10
|
+
# `with_provider_options` merging, immediately before it is posted — and
|
|
11
|
+
# discards the callback's return value, so hooks edit the payload in place.
|
|
12
|
+
# {hook} builds such a callback; {chat} and {attach!} register it for you.
|
|
13
13
|
#
|
|
14
|
-
# Because the payload is walked with {DataRedactor.redact_deep}, this scrubs
|
|
15
|
-
# **every String leaf**
|
|
16
|
-
# tool definitions, and —
|
|
17
|
-
#
|
|
18
|
-
#
|
|
14
|
+
# Because the payload is walked with {DataRedactor.redact_deep!}, this scrubs
|
|
15
|
+
# **every String leaf** of the request: the user prompt, the system prompt,
|
|
16
|
+
# the conversation history, tool definitions, and — the case per-call
|
|
17
|
+
# redaction cannot reach — **tool results**. An agent that reads a file or
|
|
18
|
+
# runs a command feeds that output back as a tool message, and it is inlined
|
|
19
|
+
# as a String in the next request's payload; the user never typed it, so only
|
|
20
|
+
# a request hook sees it before it leaves.
|
|
19
21
|
#
|
|
20
|
-
#
|
|
21
|
-
#
|
|
22
|
-
# is loaded and `RubyLLM::Protocol#render` still exists, so an upstream
|
|
23
|
-
# refactor fails loudly at install time rather than silently leaking data.
|
|
24
|
-
# Prefer this only when you need redaction to be *transparent*; otherwise
|
|
25
|
-
# redact per call with {DataRedactor.redact} before `chat.ask`.
|
|
22
|
+
# Redaction is applied to the rendered payload and nothing is persisted, so
|
|
23
|
+
# the conversation you keep is untouched: this scrubs the wire, per request.
|
|
26
24
|
#
|
|
27
|
-
# ##
|
|
28
|
-
# - **
|
|
29
|
-
#
|
|
30
|
-
# - **
|
|
31
|
-
#
|
|
25
|
+
# ## Limits, stated plainly
|
|
26
|
+
# - **Per chat.** RubyLLM has no global callback registry, so a chat you did
|
|
27
|
+
# not build with {chat} or pass to {attach!} is *not* redacted.
|
|
28
|
+
# - **Chat only.** Embeddings, moderation, image generation and transcription
|
|
29
|
+
# do not run request hooks.
|
|
30
|
+
# - **Base64 attachments** (PDFs, images, audio inlined as base64) and
|
|
31
|
+
# **URL-referenced files** are not redacted — the bytes are encoded or
|
|
32
|
+
# remote, so patterns cannot see into them.
|
|
33
|
+
# - **A digits-only tool-call id would be redacted**, breaking the
|
|
34
|
+
# `tool_use_id` correlation a provider validates: `_` counts as a boundary,
|
|
35
|
+
# so `toolu_012345678` matches a national-ID pattern. Real Anthropic and
|
|
36
|
+
# OpenAI ids are mixed alphanumeric and unaffected; add `:id` to
|
|
37
|
+
# `skip_keys:` if that ever changes.
|
|
38
|
+
# - Redacting tool results can break an agent that needs a value a tool
|
|
39
|
+
# returned. Scope with `only:`/`except:` when that matters.
|
|
32
40
|
#
|
|
33
|
-
# @example
|
|
41
|
+
# @example Build a redacted chat (nothing to remember later)
|
|
42
|
+
# require "ruby_llm"
|
|
34
43
|
# require "data_redactor/integrations/ruby_llm"
|
|
35
|
-
# DataRedactor::Integrations::RubyLLM.install!
|
|
36
44
|
#
|
|
37
|
-
# chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
45
|
+
# chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
38
46
|
# chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
39
47
|
#
|
|
40
|
-
# @example
|
|
41
|
-
# DataRedactor::Integrations::RubyLLM.
|
|
48
|
+
# @example Redact a chat, agent, or acts_as_chat record you were handed
|
|
49
|
+
# DataRedactor::Integrations::RubyLLM.attach!(agent, only: [:financial])
|
|
50
|
+
#
|
|
51
|
+
# @example Redact an agent by handing it a chat that is already redacted
|
|
52
|
+
# chat = DataRedactor::Integrations::RubyLLM.chat(model: "claude-opus-4-8")
|
|
53
|
+
# agent = SupportAgent.new(chat: chat) # the agent keeps that chat
|
|
54
|
+
#
|
|
55
|
+
# # Every request of the agent's tool loop is redacted, tool results
|
|
56
|
+
# # included. Reaches nothing internal, so it is the tidiest spelling
|
|
57
|
+
# # for agents.
|
|
58
|
+
#
|
|
59
|
+
# @example Register the callback yourself
|
|
60
|
+
# chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
42
61
|
module RubyLLM
|
|
43
62
|
module_function
|
|
44
63
|
|
|
45
|
-
# ruby_llm
|
|
46
|
-
#
|
|
47
|
-
SUPPORTED_VERSION = "
|
|
64
|
+
# The `ruby_llm` line that exposes `Chat#before_request`. 1.x has no
|
|
65
|
+
# request hook at all — redact per call with {DataRedactor.redact} there.
|
|
66
|
+
SUPPORTED_VERSION = ">= 2.0.0.pre"
|
|
48
67
|
|
|
49
|
-
#
|
|
50
|
-
# second call with the patch already installed is a no-op (the filter
|
|
51
|
-
# options from the first successful install are kept).
|
|
68
|
+
# Payload keys left untouched by default.
|
|
52
69
|
#
|
|
53
|
-
#
|
|
54
|
-
#
|
|
55
|
-
#
|
|
56
|
-
#
|
|
57
|
-
#
|
|
58
|
-
|
|
59
|
-
# @return [void]
|
|
60
|
-
# @raise [RuntimeError] if `ruby_llm` is not loaded, the loaded version is
|
|
61
|
-
# outside {SUPPORTED_VERSION}, or `RubyLLM::Protocol#render` is missing
|
|
62
|
-
# (i.e. an upstream refactor moved the chokepoint).
|
|
63
|
-
def install!(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
64
|
-
ensure_compatible!
|
|
70
|
+
# `model` is structural, not content: dated model ids carry an eight-digit
|
|
71
|
+
# suffix (`claude-haiku-4-5-20251001`), which the national-ID patterns
|
|
72
|
+
# match, and a redacted model id is rejected by the provider. Nothing else
|
|
73
|
+
# is skipped by default — an unlisted key is always redacted, so a field we
|
|
74
|
+
# have never seen cannot leak silently.
|
|
75
|
+
DEFAULT_SKIP_KEYS = [:model].freeze
|
|
65
76
|
|
|
66
|
-
|
|
67
|
-
|
|
77
|
+
# Build a chat with redaction already attached.
|
|
78
|
+
#
|
|
79
|
+
# A drop-in for `RubyLLM.chat`: every argument is forwarded to it
|
|
80
|
+
# untouched, and the returned chat is the real thing, so the fluent API
|
|
81
|
+
# keeps working (`.with_temperature(0.2).ask(...)`).
|
|
82
|
+
#
|
|
83
|
+
# @param only [Symbol, String, Array, nil] forwarded to {hook}.
|
|
84
|
+
# @param except [Symbol, String, Array, nil] forwarded to {hook}.
|
|
85
|
+
# @param placeholder [String, Symbol] forwarded to {hook}.
|
|
86
|
+
# @param skip_keys [Symbol, String, Array] forwarded to {hook}.
|
|
87
|
+
# @param kwargs [Hash] forwarded to `RubyLLM.chat` (`model:`, `provider:`,
|
|
88
|
+
# `protocol:`, `assume_model_exists:`, `context:`).
|
|
89
|
+
# @return [RubyLLM::Chat] a chat whose every request is redacted.
|
|
90
|
+
#
|
|
91
|
+
# @example
|
|
92
|
+
# DataRedactor::Integrations::RubyLLM.chat(model: "gpt-5.4", only: [:financial])
|
|
93
|
+
def chat(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
94
|
+
skip_keys: DEFAULT_SKIP_KEYS, **kwargs, &block)
|
|
95
|
+
attach!(::RubyLLM.chat(**kwargs, &block),
|
|
96
|
+
only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
97
|
+
end
|
|
68
98
|
|
|
69
|
-
|
|
99
|
+
# Register {hook} on a chat you did not build.
|
|
100
|
+
#
|
|
101
|
+
# Accepts whatever holds the chat, so callers need not know which object
|
|
102
|
+
# carries the hook: a `RubyLLM::Chat`, a `RubyLLM::Agent`, or an
|
|
103
|
+
# `acts_as_chat` record (its `#to_llm` chat is memoized, so one call covers
|
|
104
|
+
# the record). The agent's wrapped `#chat` is always used rather than its
|
|
105
|
+
# delegated `before_request`, because in Rails mode that delegator forwards
|
|
106
|
+
# to a record which has no such method.
|
|
107
|
+
#
|
|
108
|
+
# @param target [Object] a chat, agent, or `acts_as_chat` record.
|
|
109
|
+
# @param only [Symbol, String, Array, nil] forwarded to {hook}.
|
|
110
|
+
# @param except [Symbol, String, Array, nil] forwarded to {hook}.
|
|
111
|
+
# @param placeholder [String, Symbol] forwarded to {hook}.
|
|
112
|
+
# @param skip_keys [Symbol, String, Array] forwarded to {hook}.
|
|
113
|
+
# @return [Object] +target+, so calls chain.
|
|
114
|
+
# @raise [ArgumentError] if no chat with a `before_request` hook can be
|
|
115
|
+
# reached from +target+ — a `ruby_llm` older than {SUPPORTED_VERSION}.
|
|
116
|
+
#
|
|
117
|
+
# @example
|
|
118
|
+
# DataRedactor::Integrations::RubyLLM.attach!(chat).ask("...")
|
|
119
|
+
def attach!(target, only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
120
|
+
skip_keys: DEFAULT_SKIP_KEYS)
|
|
121
|
+
resolve(target).before_request(
|
|
122
|
+
&hook(only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
123
|
+
)
|
|
124
|
+
target
|
|
70
125
|
end
|
|
71
126
|
|
|
72
|
-
#
|
|
73
|
-
#
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
127
|
+
# Build the `before_request` callback.
|
|
128
|
+
#
|
|
129
|
+
# It redacts the payload **in place**, which is what RubyLLM's contract
|
|
130
|
+
# requires: hooks mutate what they are given and their return value is
|
|
131
|
+
# discarded.
|
|
132
|
+
#
|
|
133
|
+
# @param only [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep!}.
|
|
134
|
+
# @param except [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep!}.
|
|
135
|
+
# @param placeholder [String, Symbol] forwarded to {DataRedactor.redact_deep!}.
|
|
136
|
+
# @param skip_keys [Symbol, String, Array] payload keys to leave verbatim.
|
|
137
|
+
# Defaults to {DEFAULT_SKIP_KEYS}; pass more to protect provider fields
|
|
138
|
+
# your app relies on (`skip_keys: [:model, :metadata]`).
|
|
139
|
+
# @return [Proc] a one-argument callback for `chat.before_request`.
|
|
140
|
+
#
|
|
141
|
+
# @example
|
|
142
|
+
# chat.before_request(&DataRedactor::Integrations::RubyLLM.hook)
|
|
143
|
+
def hook(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT,
|
|
144
|
+
skip_keys: DEFAULT_SKIP_KEYS)
|
|
145
|
+
lambda do |payload|
|
|
146
|
+
DataRedactor.redact_deep!(payload, only: only, except: except,
|
|
147
|
+
placeholder: placeholder, skip_keys: skip_keys)
|
|
148
|
+
end
|
|
77
149
|
end
|
|
78
150
|
|
|
79
|
-
#
|
|
80
|
-
#
|
|
81
|
-
|
|
82
|
-
|
|
151
|
+
# @deprecated Transparent app-wide mode is gone. It prepended
|
|
152
|
+
# `RubyLLM::Protocol#render`; ruby_llm 2.0 exposes a public request hook,
|
|
153
|
+
# so redaction no longer patches RubyLLM. Use {chat} or {attach!}. This
|
|
154
|
+
# method exists only to say so and will be deleted in the next minor.
|
|
155
|
+
# @raise [RuntimeError] always.
|
|
156
|
+
def install!(*_args, **_kwargs)
|
|
157
|
+
raise "DataRedactor::Integrations::RubyLLM.install! has been removed: ruby_llm 2.0 exposes a " \
|
|
158
|
+
"public request hook, so redaction no longer patches RubyLLM. Build chats with " \
|
|
159
|
+
"DataRedactor::Integrations::RubyLLM.chat(...), or attach to an existing chat, agent, " \
|
|
160
|
+
"or record with DataRedactor::Integrations::RubyLLM.attach!(target)."
|
|
83
161
|
end
|
|
84
162
|
|
|
85
163
|
# @!visibility private
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
end
|
|
100
|
-
end
|
|
164
|
+
# Walks to the `RubyLLM::Chat` that actually holds the hook: an agent's
|
|
165
|
+
# wrapped chat, then a record's `to_llm` chat (an agent in Rails mode wraps
|
|
166
|
+
# the record, hence two hops), then the chat itself.
|
|
167
|
+
#
|
|
168
|
+
# The hops come first on purpose. `Agent` delegates `before_request` to
|
|
169
|
+
# whatever it wraps, so a Rails-mode agent answers `respond_to?` with true
|
|
170
|
+
# while the call forwards to a record that has no such method. Resolving to
|
|
171
|
+
# the real chat sidesteps the delegator instead of trusting it.
|
|
172
|
+
def resolve(target)
|
|
173
|
+
candidate = target
|
|
174
|
+
candidate = candidate.chat if candidate.respond_to?(:chat)
|
|
175
|
+
candidate = candidate.to_llm if candidate.respond_to?(:to_llm)
|
|
176
|
+
return candidate if candidate.respond_to?(:before_request)
|
|
101
177
|
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
# else in the send path.
|
|
106
|
-
module PayloadPatch
|
|
107
|
-
def render(*args, **kwargs)
|
|
108
|
-
payload = super
|
|
109
|
-
opts = DataRedactor::Integrations::RubyLLM.options
|
|
110
|
-
DataRedactor.redact_deep(
|
|
111
|
-
payload,
|
|
112
|
-
only: opts[:only],
|
|
113
|
-
except: opts[:except],
|
|
114
|
-
placeholder: opts[:placeholder]
|
|
115
|
-
)
|
|
116
|
-
end
|
|
178
|
+
raise ArgumentError, "data_redactor ruby_llm integration: no #before_request hook reachable from " \
|
|
179
|
+
"#{target.class}. It needs ruby_llm #{SUPPORTED_VERSION}; on 1.x, redact per " \
|
|
180
|
+
"call with DataRedactor.redact before chat.ask."
|
|
117
181
|
end
|
|
118
182
|
end
|
|
119
183
|
end
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/railtie"
|
|
4
|
+
require "data_redactor"
|
|
5
|
+
require "data_redactor/integrations/logger"
|
|
6
|
+
require "data_redactor/integrations/rails"
|
|
7
|
+
|
|
8
|
+
module DataRedactor
|
|
9
|
+
# Rails wiring for data_redactor. Requiring this file installs a Railtie that
|
|
10
|
+
# wraps the Rails logger's formatter and appends a `filter_parameters` entry,
|
|
11
|
+
# so an app gets redaction without writing either integration by hand.
|
|
12
|
+
#
|
|
13
|
+
# Rails is never a runtime dependency of the gem: this file is only loaded
|
|
14
|
+
# when the application requires it, following the same opt-in pattern as
|
|
15
|
+
# every other integration.
|
|
16
|
+
#
|
|
17
|
+
# @example Enable everything with the defaults
|
|
18
|
+
# # Gemfile
|
|
19
|
+
# gem "data_redactor", require: "data_redactor/railtie"
|
|
20
|
+
#
|
|
21
|
+
# @example Tune it from an initializer
|
|
22
|
+
# # config/initializers/data_redactor.rb
|
|
23
|
+
# Rails.application.configure do
|
|
24
|
+
# config.data_redactor.only = [:credentials, :financial]
|
|
25
|
+
# config.data_redactor.placeholder = :tagged
|
|
26
|
+
# config.data_redactor.logger = false # leave the logger alone
|
|
27
|
+
# end
|
|
28
|
+
class Railtie < ::Rails::Railtie
|
|
29
|
+
config.data_redactor = ActiveSupport::OrderedOptions.new
|
|
30
|
+
config.data_redactor.logger = true
|
|
31
|
+
config.data_redactor.filter_parameters = true
|
|
32
|
+
config.data_redactor.placeholder = DataRedactor::PLACEHOLDER_DEFAULT
|
|
33
|
+
|
|
34
|
+
# `only`/`except` are read with `[]` rather than the OrderedOptions reader:
|
|
35
|
+
# `only` and `except` are Enumerable methods, so the reader would return a
|
|
36
|
+
# Method-backed result instead of nil when the app never set them.
|
|
37
|
+
#
|
|
38
|
+
# @api private
|
|
39
|
+
# @param config [ActiveSupport::OrderedOptions] the `config.data_redactor` options
|
|
40
|
+
# @return [Hash] kwargs for {DataRedactor.redact}
|
|
41
|
+
def self.redaction_options(config)
|
|
42
|
+
{
|
|
43
|
+
only: config[:only],
|
|
44
|
+
except: config[:except],
|
|
45
|
+
placeholder: config.placeholder
|
|
46
|
+
}
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
initializer "data_redactor.filter_parameters" do |app|
|
|
50
|
+
options = app.config.data_redactor
|
|
51
|
+
next unless options.filter_parameters
|
|
52
|
+
|
|
53
|
+
app.config.filter_parameters += [
|
|
54
|
+
Integrations::Rails.filter(**Railtie.redaction_options(options))
|
|
55
|
+
]
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Runs after `initialize_logger` so we wrap whatever formatter the app
|
|
59
|
+
# ended up with (Rails' own, lograge, a JSON formatter) rather than
|
|
60
|
+
# replacing it.
|
|
61
|
+
initializer "data_redactor.logger", after: :initialize_logger do |app|
|
|
62
|
+
options = app.config.data_redactor
|
|
63
|
+
next unless options.logger
|
|
64
|
+
|
|
65
|
+
Railtie.wrap_logger(::Rails.logger, Railtie.redaction_options(options))
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# A BroadcastLogger writes to each of its sinks directly, so a formatter set
|
|
69
|
+
# on the broadcast itself never runs — each sink has to be wrapped instead.
|
|
70
|
+
#
|
|
71
|
+
# @api private
|
|
72
|
+
# @param logger [#formatter, ActiveSupport::BroadcastLogger, nil] logger to wrap
|
|
73
|
+
# @param options [Hash] kwargs forwarded to {Integrations::Logger}
|
|
74
|
+
# @return [void]
|
|
75
|
+
def self.wrap_logger(logger, options)
|
|
76
|
+
return if logger.nil?
|
|
77
|
+
|
|
78
|
+
if logger.respond_to?(:broadcasts)
|
|
79
|
+
logger.broadcasts.each { |sink| wrap_logger(sink, options) }
|
|
80
|
+
return
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
return unless logger.respond_to?(:formatter) && logger.respond_to?(:formatter=)
|
|
84
|
+
|
|
85
|
+
inner = logger.formatter || ::Logger::Formatter.new
|
|
86
|
+
return if inner.is_a?(Integrations::Logger)
|
|
87
|
+
|
|
88
|
+
logger.formatter = Integrations::Logger.new(inner: inner, **options)
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
end
|
data/lib/data_redactor.rb
CHANGED
|
@@ -200,6 +200,11 @@ module DataRedactor
|
|
|
200
200
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
201
201
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
202
202
|
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
203
|
+
# @param skip_keys [Symbol, String, Array, nil] hash keys whose values are left
|
|
204
|
+
# alone, matched at any depth and by name, so Symbol and String keys are
|
|
205
|
+
# equivalent. A skipped key's whole subtree is untouched. Use it for
|
|
206
|
+
# structural fields that must survive verbatim — an identifier a downstream
|
|
207
|
+
# API validates, say — not as a filter (that is what +only+/+except+ are).
|
|
203
208
|
# @return [Hash, Array, String, Object] a new structure of the same shape
|
|
204
209
|
# with all String leaves redacted.
|
|
205
210
|
# @raise [ArgumentError] if the structure contains a circular reference.
|
|
@@ -209,8 +214,51 @@ module DataRedactor
|
|
|
209
214
|
#
|
|
210
215
|
# @example Mixed filter
|
|
211
216
|
# DataRedactor.redact_deep(payload, only: :credentials, placeholder: :tagged)
|
|
212
|
-
|
|
213
|
-
|
|
217
|
+
#
|
|
218
|
+
# @example Keep a structural field intact
|
|
219
|
+
# DataRedactor.redact_deep(request, skip_keys: [:model])
|
|
220
|
+
def redact_deep(data, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
221
|
+
_walk(data, only: only, except: except, placeholder: placeholder,
|
|
222
|
+
skip: _skip_set(skip_keys), seen: Set.new)
|
|
223
|
+
end
|
|
224
|
+
|
|
225
|
+
# Redact every String value in a nested Hash/Array structure **in place**.
|
|
226
|
+
#
|
|
227
|
+
# The in-place sibling of {redact_deep}: same traversal and the same
|
|
228
|
+
# filtering, but the containers you pass in are mutated and returned instead
|
|
229
|
+
# of copied. Written for callers that hand a structure to something which
|
|
230
|
+
# ignores return values — a `ruby_llm` +before_request+ hook, a middleware
|
|
231
|
+
# that must edit the params Hash it was given — where {redact_deep} would
|
|
232
|
+
# force a full copy only to overwrite the original with it.
|
|
233
|
+
#
|
|
234
|
+
# Only container contents change: Hash values and Array elements are replaced
|
|
235
|
+
# with redacted Strings. Hash keys are never modified, and the String objects
|
|
236
|
+
# themselves are not mutated (frozen leaves are fine — the container's
|
|
237
|
+
# reference is repointed at a new String).
|
|
238
|
+
#
|
|
239
|
+
# @param data [Hash, Array] the structure to redact in place.
|
|
240
|
+
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
241
|
+
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
242
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
243
|
+
# @param skip_keys [Symbol, String, Array, nil] hash keys to leave alone, as
|
|
244
|
+
# in {redact_deep}.
|
|
245
|
+
# @return [Hash, Array] +data+ itself, redacted.
|
|
246
|
+
# @raise [ArgumentError] if +data+ is not a Hash or an Array, or if the
|
|
247
|
+
# structure contains a circular reference.
|
|
248
|
+
#
|
|
249
|
+
# @example A hook that must mutate what it is given
|
|
250
|
+
# chat.before_request { |payload| DataRedactor.redact_deep!(payload) }
|
|
251
|
+
#
|
|
252
|
+
# @example Scrubbing a params Hash in place
|
|
253
|
+
# DataRedactor.redact_deep!(params, only: :credentials)
|
|
254
|
+
def redact_deep!(data, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
255
|
+
unless data.is_a?(Hash) || data.is_a?(Array)
|
|
256
|
+
raise ArgumentError, "redact_deep!: expected a Hash or Array to mutate, got #{data.class}. " \
|
|
257
|
+
"Use redact or redact_deep for other types."
|
|
258
|
+
end
|
|
259
|
+
|
|
260
|
+
_walk!(data, only: only, except: except, placeholder: placeholder,
|
|
261
|
+
skip: _skip_set(skip_keys), seen: Set.new)
|
|
214
262
|
end
|
|
215
263
|
|
|
216
264
|
# Parse +json_string+, redact every String value in the resulting structure,
|
|
@@ -223,15 +271,16 @@ module DataRedactor
|
|
|
223
271
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
224
272
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
225
273
|
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
274
|
+
# @param skip_keys [Symbol, String, Array, nil] forwarded to {redact_deep}.
|
|
226
275
|
# @return [String] a JSON string with all String values redacted.
|
|
227
276
|
# @raise [JSON::ParserError] if +json_string+ is not valid JSON.
|
|
228
277
|
#
|
|
229
278
|
# @example
|
|
230
279
|
# DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
|
|
231
280
|
# # => '{"email":"[REDACTED]","count":3}'
|
|
232
|
-
def redact_json(json_string, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT)
|
|
281
|
+
def redact_json(json_string, only: nil, except: nil, placeholder: PLACEHOLDER_DEFAULT, skip_keys: nil)
|
|
233
282
|
parsed = JSON.parse(json_string)
|
|
234
|
-
redacted = redact_deep(parsed, only: only, except: except, placeholder: placeholder)
|
|
283
|
+
redacted = redact_deep(parsed, only: only, except: except, placeholder: placeholder, skip_keys: skip_keys)
|
|
235
284
|
JSON.generate(redacted)
|
|
236
285
|
end
|
|
237
286
|
|
|
@@ -395,20 +444,56 @@ module DataRedactor
|
|
|
395
444
|
# Depth-first recursive walker for {redact_deep}.
|
|
396
445
|
# +seen+ is a Set of object_ids already on the current traversal stack,
|
|
397
446
|
# used to detect circular references.
|
|
398
|
-
|
|
447
|
+
EMPTY_SKIP = Set.new.freeze
|
|
448
|
+
private_constant :EMPTY_SKIP
|
|
449
|
+
|
|
450
|
+
def _skip_set(keys)
|
|
451
|
+
return EMPTY_SKIP if keys.nil?
|
|
452
|
+
|
|
453
|
+
Set.new(Array(keys).map(&:to_s))
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
def _walk!(node, only:, except:, placeholder:, skip:, seen:)
|
|
457
|
+
case node
|
|
458
|
+
when String
|
|
459
|
+
redact(node, only: only, except: except, placeholder: placeholder)
|
|
460
|
+
when Hash
|
|
461
|
+
raise ArgumentError, "redact_deep!: circular reference detected" if seen.include?(node.object_id)
|
|
462
|
+
seen.add(node.object_id)
|
|
463
|
+
node.each_pair do |k, v|
|
|
464
|
+
next if skip.include?(k.to_s)
|
|
465
|
+
|
|
466
|
+
node[k] = _walk!(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
467
|
+
end
|
|
468
|
+
seen.delete(node.object_id)
|
|
469
|
+
node
|
|
470
|
+
when Array
|
|
471
|
+
raise ArgumentError, "redact_deep!: circular reference detected" if seen.include?(node.object_id)
|
|
472
|
+
seen.add(node.object_id)
|
|
473
|
+
node.each_index do |i|
|
|
474
|
+
node[i] = _walk!(node[i], only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
475
|
+
end
|
|
476
|
+
seen.delete(node.object_id)
|
|
477
|
+
node
|
|
478
|
+
else
|
|
479
|
+
node
|
|
480
|
+
end
|
|
481
|
+
end
|
|
482
|
+
|
|
483
|
+
def _walk(node, only:, except:, placeholder:, skip:, seen:)
|
|
399
484
|
case node
|
|
400
485
|
when String
|
|
401
486
|
redact(node, only: only, except: except, placeholder: placeholder)
|
|
402
487
|
when Hash
|
|
403
488
|
raise ArgumentError, "redact_deep: circular reference detected" if seen.include?(node.object_id)
|
|
404
489
|
seen.add(node.object_id)
|
|
405
|
-
result = node
|
|
490
|
+
result = _walk_hash(node, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
406
491
|
seen.delete(node.object_id)
|
|
407
492
|
result
|
|
408
493
|
when Array
|
|
409
494
|
raise ArgumentError, "redact_deep: circular reference detected" if seen.include?(node.object_id)
|
|
410
495
|
seen.add(node.object_id)
|
|
411
|
-
result = node.map { |v| _walk(v, only: only, except: except, placeholder: placeholder, seen: seen) }
|
|
496
|
+
result = node.map { |v| _walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen) }
|
|
412
497
|
seen.delete(node.object_id)
|
|
413
498
|
result
|
|
414
499
|
else
|
|
@@ -416,6 +501,22 @@ module DataRedactor
|
|
|
416
501
|
end
|
|
417
502
|
end
|
|
418
503
|
|
|
504
|
+
# transform_values is the fast path and cannot see keys, so it stays in use
|
|
505
|
+
# whenever nothing is being skipped — which is every call that predates skip_keys.
|
|
506
|
+
def _walk_hash(node, only:, except:, placeholder:, skip:, seen:)
|
|
507
|
+
if skip.empty?
|
|
508
|
+
node.transform_values { |v| _walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen) }
|
|
509
|
+
else
|
|
510
|
+
node.each_with_object({}) do |(k, v), acc|
|
|
511
|
+
acc[k] = if skip.include?(k.to_s)
|
|
512
|
+
v
|
|
513
|
+
else
|
|
514
|
+
_walk(v, only: only, except: except, placeholder: placeholder, skip: skip, seen: seen)
|
|
515
|
+
end
|
|
516
|
+
end
|
|
517
|
+
end
|
|
518
|
+
end
|
|
519
|
+
|
|
419
520
|
# @api private
|
|
420
521
|
def pattern_enabled?(name, tag_bit, only_present, only_bits, only_names,
|
|
421
522
|
except_bits, except_names)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_redactor
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.18.0
|
|
5
5
|
platform: x86_64-linux
|
|
6
6
|
authors:
|
|
7
7
|
- Daniele Frisanco
|
|
@@ -65,6 +65,20 @@ dependencies:
|
|
|
65
65
|
- - ">="
|
|
66
66
|
- !ruby/object:Gem::Version
|
|
67
67
|
version: '2.0'
|
|
68
|
+
- !ruby/object:Gem::Dependency
|
|
69
|
+
name: logger
|
|
70
|
+
requirement: !ruby/object:Gem::Requirement
|
|
71
|
+
requirements:
|
|
72
|
+
- - ">="
|
|
73
|
+
- !ruby/object:Gem::Version
|
|
74
|
+
version: '1.5'
|
|
75
|
+
type: :development
|
|
76
|
+
prerelease: false
|
|
77
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
78
|
+
requirements:
|
|
79
|
+
- - ">="
|
|
80
|
+
- !ruby/object:Gem::Version
|
|
81
|
+
version: '1.5'
|
|
68
82
|
- !ruby/object:Gem::Dependency
|
|
69
83
|
name: benchmark-ips
|
|
70
84
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -94,7 +108,7 @@ dependencies:
|
|
|
94
108
|
- !ruby/object:Gem::Version
|
|
95
109
|
version: '0.2'
|
|
96
110
|
description: A Ruby gem with a C extension for high-performance scanning and redaction
|
|
97
|
-
of
|
|
111
|
+
of sensitive data — API keys, tokens, credentials, IBANs, national IDs, emails,
|
|
98
112
|
phone numbers, and PII from 15+ countries. Optional Logger formatter, Rails filter_parameters
|
|
99
113
|
adapter, and Rack middleware. Designed to sanitize text before sending to LLMs,
|
|
100
114
|
logging systems, or any public/third-party API.
|
|
@@ -122,6 +136,7 @@ files:
|
|
|
122
136
|
- lib/data_redactor/integrations/rails.rb
|
|
123
137
|
- lib/data_redactor/integrations/ruby_llm.rb
|
|
124
138
|
- lib/data_redactor/name_pattern.rb
|
|
139
|
+
- lib/data_redactor/railtie.rb
|
|
125
140
|
- lib/data_redactor/refinements.rb
|
|
126
141
|
- lib/data_redactor/version.rb
|
|
127
142
|
homepage: https://github.com/danielefrisanco/data_redactor
|