data_redactor 0.15.0-aarch64-linux → 0.17.0-aarch64-linux
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +45 -1
- data/README.md +73 -1
- data/lib/data_redactor/3.0/data_redactor.so +0 -0
- data/lib/data_redactor/3.1/data_redactor.so +0 -0
- data/lib/data_redactor/3.2/data_redactor.so +0 -0
- data/lib/data_redactor/3.3/data_redactor.so +0 -0
- data/lib/data_redactor/3.4/data_redactor.so +0 -0
- data/lib/data_redactor/4.0/data_redactor.so +0 -0
- data/lib/data_redactor/integrations/ruby_llm.rb +120 -0
- data/lib/data_redactor/refinements.rb +69 -0
- data/lib/data_redactor/version.rb +1 -1
- data/lib/data_redactor.rb +22 -14
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 206353fa82d49af6e4b9d528eb27d95047d6121e575bc824640aa3595f785f2c
|
|
4
|
+
data.tar.gz: 1b1207d478038a66887af84363242913344345fb317277ac41554131c3989810
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 9e09a76ab4c81bc1b148b9b4f06d93be6df2a260608e195d2a59d424aeec55bc99f004cdfb2a8f22f7136e7200ba4350847b7ffbf260fa5461d141de4e6686cb
|
|
7
|
+
data.tar.gz: e75715b813870bdcf5fba2aa7ad7a4b8e87e0b15294473ba481c4b260a4e3ca037d32a5b669800448e821463f2fe9c95b9a45b5b829932dd3fcbf8adf463eccc
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.17.0] - 2026-06-21
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **Transparent `ruby_llm` integration (opt-in monkeypatch).**
|
|
14
|
+
`require "data_redactor/integrations/ruby_llm"` then
|
|
15
|
+
`DataRedactor::Integrations::RubyLLM.install!` prepends a small patch onto
|
|
16
|
+
`RubyLLM::Protocol#render`, so **every** outbound request is deep-redacted
|
|
17
|
+
before it is posted — no per-call `.redact`. One hook covers all providers
|
|
18
|
+
(Anthropic, OpenAI, Gemini, Bedrock, Responses) and scrubs the user prompt,
|
|
19
|
+
system prompt, tool definitions, and any file/command-output that an agent fed
|
|
20
|
+
back as a tool result (all inlined as strings in the payload). Forwards
|
|
21
|
+
`only:`/`except:`/`placeholder:`; idempotent; fails fast at `install!` if an
|
|
22
|
+
unsupported `ruby_llm` version is loaded or `Protocol#render` is missing.
|
|
23
|
+
**Limitation:** base64 attachments (PDFs/images/audio) and URL-referenced
|
|
24
|
+
files are not redacted — the secret bytes are encoded or remote, so patterns
|
|
25
|
+
cannot see them. This is a monkeypatch on internal API and is version-pinned;
|
|
26
|
+
the clean alternative remains per-call `DataRedactor.redact` before `chat.ask`.
|
|
27
|
+
|
|
28
|
+
## [0.16.0] - 2026-06-21
|
|
29
|
+
|
|
30
|
+
### Added
|
|
31
|
+
- **Opt-in `#redact` refinements.** `require "data_redactor/refinements"` then
|
|
32
|
+
`using DataRedactor::Refinements` adds `#redact` to `String` (→
|
|
33
|
+
`DataRedactor.redact`) and to `Hash`/`Array` (→ `DataRedactor.redact_deep`), e.g.
|
|
34
|
+
`"email a@b.com".redact` and `chat.ask(user_input.redact)`. Refinements are
|
|
35
|
+
lexically scoped, so they never pollute the core classes globally — apps that
|
|
36
|
+
don't opt in are unaffected and there is no collision risk. Forwards
|
|
37
|
+
`only:`/`except:`/`placeholder:`; never mutates the receiver.
|
|
38
|
+
`DataRedactor.redact` remains the primary API.
|
|
39
|
+
- **Length-aware placeholder modes.** `placeholder: :length` replaces each match
|
|
40
|
+
with `[REDACTED:N]` and `placeholder: :tagged_length` with `[REDACTED:TAGNAME:N]`,
|
|
41
|
+
where `N` is the **byte length** of the redacted value. Readers can gauge what
|
|
42
|
+
was there without seeing it. Both compose with `only:`/`except:` and are
|
|
43
|
+
forwarded by `redact_deep`, `redact_json`, and the integrations. Additive —
|
|
44
|
+
two new values for the existing `placeholder:` keyword; no behaviour changes.
|
|
45
|
+
- **CI: ASan/UBSan memory-safety gate.** New job builds the matcher engine
|
|
46
|
+
standalone under `-fsanitize=address,undefined` and drives it over an
|
|
47
|
+
adversarial corpus + seeded fuzz loop (catches the `OP_EOL`-class OOB read).
|
|
48
|
+
- **CI: throughput-trend history + PR comment.** New job records the C/pure-Ruby
|
|
49
|
+
ratio over time (history in `actions/cache`, no gh-pages), posts a sticky PR
|
|
50
|
+
comment comparing each run to the previous point, and fails on a >10% drop.
|
|
51
|
+
|
|
10
52
|
## [0.15.0] - 2026-06-17
|
|
11
53
|
|
|
12
54
|
### Changed
|
|
@@ -312,7 +354,9 @@ features as 0.7.1 plus the pipeline fix.
|
|
|
312
354
|
- `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
|
|
313
355
|
- RSpec suite with one example per pattern.
|
|
314
356
|
|
|
315
|
-
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.
|
|
357
|
+
[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.17.0...HEAD
|
|
358
|
+
[0.17.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.16.0...v0.17.0
|
|
359
|
+
[0.16.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...v0.16.0
|
|
316
360
|
[0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
|
|
317
361
|
[0.14.1]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...v0.14.1
|
|
318
362
|
[0.14.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.13.0...v0.14.0
|
data/README.md
CHANGED
|
@@ -6,6 +6,12 @@
|
|
|
6
6
|
|
|
7
7
|
A Ruby gem with a C extension for high-performance regex-based redaction of sensitive data from strings.
|
|
8
8
|
|
|
9
|
+
> 📄 The engineering behind the v19 matching engine is written up as an experience
|
|
10
|
+
> report, *"The Fastest Engine Is Not the Shippable Engine: Replacing a Regex Engine
|
|
11
|
+
> for Data Redaction Under Production Constraints,"* currently under review at
|
|
12
|
+
> *Software: Practice and Experience* (Manuscript ID 7985366). Source and the
|
|
13
|
+
> reproducibility bundle are in [`paper/`](paper/).
|
|
14
|
+
|
|
9
15
|
## What it does
|
|
10
16
|
|
|
11
17
|
DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
|
|
@@ -103,9 +109,18 @@ DataRedactor.redact(text, placeholder: :hash)
|
|
|
103
109
|
# "user@example.com" → "[CONTACT_3d7a]"
|
|
104
110
|
# "user@example.com" → "[CONTACT_3d7a]" (same every time)
|
|
105
111
|
# "other@example.com" → "[CONTACT_91fc]" (different value, different hash)
|
|
112
|
+
|
|
113
|
+
# Length — embeds the byte length of the redacted value, so readers can
|
|
114
|
+
# gauge what was there without seeing it.
|
|
115
|
+
DataRedactor.redact(text, placeholder: :length)
|
|
116
|
+
# "user@example.com" → "[REDACTED:16]"
|
|
117
|
+
|
|
118
|
+
# Tagged length — tag name plus byte length.
|
|
119
|
+
DataRedactor.redact(text, placeholder: :tagged_length)
|
|
120
|
+
# "user@example.com" → "[REDACTED:CONTACT:16]"
|
|
106
121
|
```
|
|
107
122
|
|
|
108
|
-
All
|
|
123
|
+
All modes compose with `only:` and `except:`:
|
|
109
124
|
|
|
110
125
|
```ruby
|
|
111
126
|
DataRedactor.redact(text, only: :contact, placeholder: :tagged)
|
|
@@ -165,6 +180,24 @@ safe_json = DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
|
|
|
165
180
|
DataRedactor.redact_json("not json") # => JSON::ParserError
|
|
166
181
|
```
|
|
167
182
|
|
|
183
|
+
### `#redact` refinements (opt-in)
|
|
184
|
+
|
|
185
|
+
Prefer `"text".redact` over `DataRedactor.redact("text")`? Opt into the refinement. It adds `#redact` to `String` (via `redact`) and to `Hash`/`Array` (via `redact_deep`) **only in the files that `using` it** — refinements are lexically scoped, so the core classes are never monkey-patched globally and there is no collision risk for apps that don't opt in. `DataRedactor.redact` remains the primary API.
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
require "data_redactor/refinements"
|
|
189
|
+
using DataRedactor::Refinements
|
|
190
|
+
|
|
191
|
+
"email alice@example.com".redact # => "email [REDACTED]"
|
|
192
|
+
{ token: "AKIAIOSFODNN7EXAMPLE" }.redact # => { token: "[REDACTED]" }
|
|
193
|
+
["a@b.com", 3].redact # => ["[REDACTED]", 3]
|
|
194
|
+
|
|
195
|
+
# Handy right before sending text to an LLM:
|
|
196
|
+
chat.ask(user_input.redact)
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
`#redact` forwards `only:`/`except:`/`placeholder:` and never mutates the receiver. Without `using DataRedactor::Refinements` in the current file, `#redact` is not defined.
|
|
200
|
+
|
|
168
201
|
### Custom patterns
|
|
169
202
|
|
|
170
203
|
Teams often have internal IDs that the gem can't ship. Register them at boot — or at runtime from any thread (registration is thread-safe, see [Thread safety](#thread-safety)):
|
|
@@ -309,6 +342,45 @@ safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
|
|
|
309
342
|
|
|
310
343
|
`content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
|
|
311
344
|
|
|
345
|
+
### RubyLLM
|
|
346
|
+
|
|
347
|
+
[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
|
|
348
|
+
|
|
349
|
+
```ruby
|
|
350
|
+
require "ruby_llm"
|
|
351
|
+
require "data_redactor"
|
|
352
|
+
|
|
353
|
+
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
354
|
+
chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
|
|
355
|
+
|
|
356
|
+
user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
|
|
357
|
+
chat.ask(DataRedactor.redact(user_input))
|
|
358
|
+
# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into, and it's the recommended approach.
|
|
362
|
+
|
|
363
|
+
#### Transparent mode (every request, no per-call wrapping)
|
|
364
|
+
|
|
365
|
+
If you'd rather redact **every** outbound request automatically — including the system prompt, tool definitions, and any file contents or shell-command output an agent feeds back as a tool result — opt into the monkeypatch:
|
|
366
|
+
|
|
367
|
+
```ruby
|
|
368
|
+
require "ruby_llm"
|
|
369
|
+
require "data_redactor/integrations/ruby_llm"
|
|
370
|
+
|
|
371
|
+
DataRedactor::Integrations::RubyLLM.install! # once, at boot
|
|
372
|
+
|
|
373
|
+
chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
374
|
+
chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
`install!` prepends a patch onto `RubyLLM::Protocol#render` — the one point where every provider (Anthropic, OpenAI, Gemini, Bedrock, Responses) has assembled its final request — and deep-redacts the payload before it's posted. It forwards `only:`/`except:`/`placeholder:`, is idempotent, and **fails fast** at `install!` if an unsupported `ruby_llm` version is loaded or the internal API has moved (so it never silently leaks).
|
|
378
|
+
|
|
379
|
+
Two caveats, by design:
|
|
380
|
+
|
|
381
|
+
- **It's a monkeypatch on RubyLLM internals**, pinned to a supported version range. Prefer per-call `DataRedactor.redact` (above) unless you specifically need transparency. RubyLLM does not yet expose a public request hook ([crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765) tracks the connection-middleware hook that would let us drop the patch).
|
|
382
|
+
- **Base64 attachments** (PDFs, images, audio sent inline) and **URL-referenced files** are not redacted — the sensitive bytes are encoded or remote, so patterns cannot see them.
|
|
383
|
+
|
|
312
384
|
## Detected patterns (89 total)
|
|
313
385
|
|
|
314
386
|
The table below is a representative sample. Use `DataRedactor.pattern_names` for the canonical, machine-readable list — it stays in sync with the C extension automatically.
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
require "data_redactor"
|
|
2
|
+
|
|
3
|
+
module DataRedactor
|
|
4
|
+
module Integrations
|
|
5
|
+
# Transparent outbound redaction for the `ruby_llm` gem (crmne/ruby_llm).
|
|
6
|
+
#
|
|
7
|
+
# Calling {install!} prepends a small module onto `RubyLLM::Protocol` that
|
|
8
|
+
# deep-redacts the **rendered request payload** before it is posted to any
|
|
9
|
+
# provider. `Protocol#render` is the single point where every provider
|
|
10
|
+
# (Anthropic, OpenAI/chat_completions, Gemini, Bedrock/Converse, Responses)
|
|
11
|
+
# has assembled its final request Hash, so one hook covers them all without
|
|
12
|
+
# knowing any provider-specific shape.
|
|
13
|
+
#
|
|
14
|
+
# Because the payload is walked with {DataRedactor.redact_deep}, this scrubs
|
|
15
|
+
# **every String leaf** in the request: the user prompt, the system prompt,
|
|
16
|
+
# tool definitions, and — crucially — any file contents or shell-command
|
|
17
|
+
# output that an agent fed back in as a tool result, since those are already
|
|
18
|
+
# inlined as strings in `messages` by the time `render` runs.
|
|
19
|
+
#
|
|
20
|
+
# This is a monkeypatch (a `prepend` onto a private internal class). It is
|
|
21
|
+
# opt-in and pinned: {install!} raises unless a supported `ruby_llm` version
|
|
22
|
+
# is loaded and `RubyLLM::Protocol#render` still exists, so an upstream
|
|
23
|
+
# refactor fails loudly at install time rather than silently leaking data.
|
|
24
|
+
# Prefer this only when you need redaction to be *transparent*; otherwise
|
|
25
|
+
# redact per call with {DataRedactor.redact} before `chat.ask`.
|
|
26
|
+
#
|
|
27
|
+
# ## What is NOT redacted
|
|
28
|
+
# - **Base64 attachments** (PDFs, images, audio sent inline as base64) — the
|
|
29
|
+
# sensitive bytes are encoded, so patterns cannot see into them.
|
|
30
|
+
# - **URL-referenced files/images** — the content lives on a remote server
|
|
31
|
+
# and never enters the payload.
|
|
32
|
+
#
|
|
33
|
+
# @example Make every ruby_llm request redacted, app-wide
|
|
34
|
+
# require "data_redactor/integrations/ruby_llm"
|
|
35
|
+
# DataRedactor::Integrations::RubyLLM.install!
|
|
36
|
+
#
|
|
37
|
+
# chat = RubyLLM.chat(model: "claude-opus-4-8")
|
|
38
|
+
# chat.ask("my card is 4111111111111111") # sent as "my card is [REDACTED]"
|
|
39
|
+
#
|
|
40
|
+
# @example Scope the redaction with the usual filters
|
|
41
|
+
# DataRedactor::Integrations::RubyLLM.install!(only: [:financial, :contact])
|
|
42
|
+
module RubyLLM
|
|
43
|
+
module_function
|
|
44
|
+
|
|
45
|
+
# ruby_llm versions whose `Protocol#render` chokepoint this integration
|
|
46
|
+
# has been verified against. Bump (and re-verify) on each ruby_llm release.
|
|
47
|
+
SUPPORTED_VERSION = "~> 1.16"
|
|
48
|
+
|
|
49
|
+
# Prepend the redaction patch onto `RubyLLM::Protocol`. Idempotent: a
|
|
50
|
+
# second call with the patch already installed is a no-op (the filter
|
|
51
|
+
# options from the first successful install are kept).
|
|
52
|
+
#
|
|
53
|
+
# The `only:`/`except:`/`placeholder:` filters are captured here and
|
|
54
|
+
# applied to every subsequent request.
|
|
55
|
+
#
|
|
56
|
+
# @param only [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep}.
|
|
57
|
+
# @param except [Symbol, String, Array, nil] forwarded to {DataRedactor.redact_deep}.
|
|
58
|
+
# @param placeholder [String, Symbol] forwarded to {DataRedactor.redact_deep}.
|
|
59
|
+
# @return [void]
|
|
60
|
+
# @raise [RuntimeError] if `ruby_llm` is not loaded, the loaded version is
|
|
61
|
+
# outside {SUPPORTED_VERSION}, or `RubyLLM::Protocol#render` is missing
|
|
62
|
+
# (i.e. an upstream refactor moved the chokepoint).
|
|
63
|
+
def install!(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
64
|
+
ensure_compatible!
|
|
65
|
+
|
|
66
|
+
@options = { only: only, except: except, placeholder: placeholder }
|
|
67
|
+
return if installed?
|
|
68
|
+
|
|
69
|
+
::RubyLLM::Protocol.prepend(PayloadPatch)
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
# @return [Boolean] whether the redaction patch is currently on
|
|
73
|
+
# `RubyLLM::Protocol`.
|
|
74
|
+
def installed?
|
|
75
|
+
defined?(::RubyLLM::Protocol) &&
|
|
76
|
+
::RubyLLM::Protocol.ancestors.include?(PayloadPatch)
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
# @!visibility private
|
|
80
|
+
# @return [Hash] the filter options captured at {install!}.
|
|
81
|
+
def options
|
|
82
|
+
@options ||= { only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT }
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
# @!visibility private
|
|
86
|
+
def ensure_compatible!
|
|
87
|
+
unless defined?(::RubyLLM::VERSION)
|
|
88
|
+
raise "data_redactor ruby_llm integration: require \"ruby_llm\" before calling install!"
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
unless Gem::Requirement.new(SUPPORTED_VERSION).satisfied_by?(Gem::Version.new(::RubyLLM::VERSION))
|
|
92
|
+
raise "data_redactor ruby_llm integration supports ruby_llm #{SUPPORTED_VERSION}, " \
|
|
93
|
+
"got #{::RubyLLM::VERSION}. Check for a newer data_redactor or pin ruby_llm."
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
unless ::RubyLLM::Protocol.method_defined?(:render) || ::RubyLLM::Protocol.private_method_defined?(:render)
|
|
97
|
+
raise "data_redactor ruby_llm integration: RubyLLM::Protocol#render not found — " \
|
|
98
|
+
"the upstream request-rendering API changed. This integration needs an update."
|
|
99
|
+
end
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
# Prepended onto `RubyLLM::Protocol`. `Protocol#complete` calls
|
|
103
|
+
# `payload = render(...)` and then posts that payload, so redacting the
|
|
104
|
+
# return value of `render` redacts the request without touching anything
|
|
105
|
+
# else in the send path.
|
|
106
|
+
module PayloadPatch
|
|
107
|
+
def render(*args, **kwargs)
|
|
108
|
+
payload = super
|
|
109
|
+
opts = DataRedactor::Integrations::RubyLLM.options
|
|
110
|
+
DataRedactor.redact_deep(
|
|
111
|
+
payload,
|
|
112
|
+
only: opts[:only],
|
|
113
|
+
except: opts[:except],
|
|
114
|
+
placeholder: opts[:placeholder]
|
|
115
|
+
)
|
|
116
|
+
end
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
end
|
|
120
|
+
end
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "data_redactor"
|
|
4
|
+
|
|
5
|
+
module DataRedactor
|
|
6
|
+
# Opt-in refinements that add a `#redact` method to `String`, `Hash`, and
|
|
7
|
+
# `Array` as sugar over {DataRedactor.redact} / {DataRedactor.redact_deep}.
|
|
8
|
+
#
|
|
9
|
+
# Refinements are lexically scoped: `#redact` exists only in files that
|
|
10
|
+
# `using DataRedactor::Refinements`, so loading this file never pollutes the
|
|
11
|
+
# core classes globally. Apps that don't opt in are unaffected, and there is
|
|
12
|
+
# no collision risk with other libraries' `String#redact`.
|
|
13
|
+
#
|
|
14
|
+
# `DataRedactor.redact` remains the primary API; this is convenience only.
|
|
15
|
+
#
|
|
16
|
+
# @example
|
|
17
|
+
# require "data_redactor/refinements"
|
|
18
|
+
# using DataRedactor::Refinements
|
|
19
|
+
#
|
|
20
|
+
# "email alice@example.com".redact #=> "email [REDACTED]"
|
|
21
|
+
# { token: "AKIAIOSFODNN7EXAMPLE" }.redact #=> { token: "[REDACTED]" }
|
|
22
|
+
# chat.ask(user_input.redact) # scrub before sending to an LLM
|
|
23
|
+
module Refinements
|
|
24
|
+
refine String do
|
|
25
|
+
# Redact this String via {DataRedactor.redact}. Returns a new String; the
|
|
26
|
+
# receiver is not mutated.
|
|
27
|
+
#
|
|
28
|
+
# @param only forwarded to {DataRedactor.redact}
|
|
29
|
+
# @param except forwarded to {DataRedactor.redact}
|
|
30
|
+
# @param placeholder forwarded to {DataRedactor.redact}
|
|
31
|
+
# @return [String] the redacted copy.
|
|
32
|
+
# @example
|
|
33
|
+
# "ssn 123-45-6789".redact #=> "ssn [REDACTED]"
|
|
34
|
+
def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
35
|
+
DataRedactor.redact(self, only: only, except: except, placeholder: placeholder)
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
refine Hash do
|
|
40
|
+
# Deep-redact this Hash's String values via {DataRedactor.redact_deep}.
|
|
41
|
+
# Returns a deep copy; the receiver is not mutated and keys are untouched.
|
|
42
|
+
#
|
|
43
|
+
# @param only forwarded to {DataRedactor.redact_deep}
|
|
44
|
+
# @param except forwarded to {DataRedactor.redact_deep}
|
|
45
|
+
# @param placeholder forwarded to {DataRedactor.redact_deep}
|
|
46
|
+
# @return [Hash] a deep copy with String leaves redacted.
|
|
47
|
+
# @example
|
|
48
|
+
# { email: "a@b.com" }.redact #=> { email: "[REDACTED]" }
|
|
49
|
+
def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
50
|
+
DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
refine Array do
|
|
55
|
+
# Deep-redact this Array's String elements via {DataRedactor.redact_deep}.
|
|
56
|
+
# Returns a deep copy; the receiver is not mutated.
|
|
57
|
+
#
|
|
58
|
+
# @param only forwarded to {DataRedactor.redact_deep}
|
|
59
|
+
# @param except forwarded to {DataRedactor.redact_deep}
|
|
60
|
+
# @param placeholder forwarded to {DataRedactor.redact_deep}
|
|
61
|
+
# @return [Array] a deep copy with String leaves redacted.
|
|
62
|
+
# @example
|
|
63
|
+
# ["a@b.com", 3].redact #=> ["[REDACTED]", 3]
|
|
64
|
+
def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
|
|
65
|
+
DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
data/lib/data_redactor.rb
CHANGED
|
@@ -23,8 +23,10 @@ require_relative "data_redactor/name_pattern"
|
|
|
23
23
|
#
|
|
24
24
|
# @example Custom placeholder
|
|
25
25
|
# DataRedactor.redact(text, placeholder: "***")
|
|
26
|
-
# DataRedactor.redact(text, placeholder: :tagged)
|
|
27
|
-
# DataRedactor.redact(text, placeholder: :hash)
|
|
26
|
+
# DataRedactor.redact(text, placeholder: :tagged) # => "[REDACTED:CONTACT]"
|
|
27
|
+
# DataRedactor.redact(text, placeholder: :hash) # => "[CONTACT_a3f9]"
|
|
28
|
+
# DataRedactor.redact(text, placeholder: :length) # => "[REDACTED:16]"
|
|
29
|
+
# DataRedactor.redact(text, placeholder: :tagged_length) # => "[REDACTED:CONTACT:16]"
|
|
28
30
|
#
|
|
29
31
|
# @example Audit / dry-run
|
|
30
32
|
# DataRedactor.scan(text)
|
|
@@ -125,12 +127,15 @@ module DataRedactor
|
|
|
125
127
|
# and/or pattern name(s).
|
|
126
128
|
# @param except [Symbol, String, Array, nil] exclude the given tag(s)
|
|
127
129
|
# and/or pattern name(s). May be combined with +only:+.
|
|
128
|
-
# @param placeholder [String, :tagged, :hash
|
|
129
|
-
# A String is used verbatim. +:tagged+ produces
|
|
130
|
-
# +:hash+ produces a deterministic +[TAGNAME_xxxx]+
|
|
131
|
-
# so the same input value always maps to the same token.
|
|
130
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length]
|
|
131
|
+
# replacement strategy. A String is used verbatim. +:tagged+ produces
|
|
132
|
+
# +[REDACTED:TAGNAME]+. +:hash+ produces a deterministic +[TAGNAME_xxxx]+
|
|
133
|
+
# token (4-hex djb2) so the same input value always maps to the same token.
|
|
134
|
+
# +:length+ produces +[REDACTED:N]+ and +:tagged_length+ produces
|
|
135
|
+
# +[REDACTED:TAGNAME:N]+, where +N+ is the byte length of the redacted value.
|
|
132
136
|
# @return [String] a new string with every match replaced.
|
|
133
|
-
# @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash
|
|
137
|
+
# @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash/
|
|
138
|
+
# :length/:tagged_length.
|
|
134
139
|
# @raise [UnknownTagError] if any Symbol in +only:+/+except:+ is not in {TAGS}.
|
|
135
140
|
# @raise [UnknownPatternError] if any String in +only:+/+except:+ is not in {pattern_names}.
|
|
136
141
|
#
|
|
@@ -194,7 +199,7 @@ module DataRedactor
|
|
|
194
199
|
# Any type is accepted; non-String scalars are returned as-is.
|
|
195
200
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
196
201
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
197
|
-
# @param placeholder [String, :tagged, :hash] forwarded to {redact}.
|
|
202
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
198
203
|
# @return [Hash, Array, String, Object] a new structure of the same shape
|
|
199
204
|
# with all String leaves redacted.
|
|
200
205
|
# @raise [ArgumentError] if the structure contains a circular reference.
|
|
@@ -217,7 +222,7 @@ module DataRedactor
|
|
|
217
222
|
# @param json_string [String] valid JSON input.
|
|
218
223
|
# @param only [Symbol, String, Array, nil] forwarded to {redact}.
|
|
219
224
|
# @param except [Symbol, String, Array, nil] forwarded to {redact}.
|
|
220
|
-
# @param placeholder [String, :tagged, :hash] forwarded to {redact}.
|
|
225
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
|
|
221
226
|
# @return [String] a JSON string with all String values redacted.
|
|
222
227
|
# @raise [JSON::ParserError] if +json_string+ is not valid JSON.
|
|
223
228
|
#
|
|
@@ -425,17 +430,20 @@ module DataRedactor
|
|
|
425
430
|
# Translate the user-facing +placeholder:+ value into the +(mode_int, str)+
|
|
426
431
|
# pair the C layer expects.
|
|
427
432
|
#
|
|
428
|
-
# @param placeholder [String, :tagged, :hash]
|
|
433
|
+
# @param placeholder [String, :tagged, :hash, :length, :tagged_length]
|
|
429
434
|
# @return [Array(Integer, String)]
|
|
430
435
|
# @raise [ArgumentError] if +placeholder+ is none of the accepted values.
|
|
431
436
|
def resolve_placeholder(placeholder)
|
|
432
437
|
case placeholder
|
|
433
|
-
when :tagged
|
|
434
|
-
when :hash
|
|
435
|
-
when
|
|
438
|
+
when :tagged then [PH_MODE_TAGGED, ""]
|
|
439
|
+
when :hash then [PH_MODE_HASH, ""]
|
|
440
|
+
when :length then [PH_MODE_LENGTH, ""]
|
|
441
|
+
when :tagged_length then [PH_MODE_TAGGED_LENGTH, ""]
|
|
442
|
+
when String then [PH_MODE_PLAIN, placeholder]
|
|
436
443
|
else
|
|
437
444
|
raise ArgumentError,
|
|
438
|
-
"placeholder must be a String, :tagged,
|
|
445
|
+
"placeholder must be a String, :tagged, :hash, :length, or :tagged_length " \
|
|
446
|
+
"— got #{placeholder.inspect}"
|
|
439
447
|
end
|
|
440
448
|
end
|
|
441
449
|
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_redactor
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.17.0
|
|
5
5
|
platform: aarch64-linux
|
|
6
6
|
authors:
|
|
7
7
|
- Daniele Frisanco
|
|
@@ -120,7 +120,9 @@ files:
|
|
|
120
120
|
- lib/data_redactor/integrations/openai.rb
|
|
121
121
|
- lib/data_redactor/integrations/rack.rb
|
|
122
122
|
- lib/data_redactor/integrations/rails.rb
|
|
123
|
+
- lib/data_redactor/integrations/ruby_llm.rb
|
|
123
124
|
- lib/data_redactor/name_pattern.rb
|
|
125
|
+
- lib/data_redactor/refinements.rb
|
|
124
126
|
- lib/data_redactor/version.rb
|
|
125
127
|
homepage: https://github.com/danielefrisanco/data_redactor
|
|
126
128
|
licenses:
|