data_redactor 0.14.0 → 0.16.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b29290519836ca25d5188a5ef4da2585bd7f11faa0c072927863c637fb618eeb
-  data.tar.gz: 465091099d2fcf4b990d4e4259c3c4ad549588839d918d831c9747236f84e864
+  metadata.gz: 62a1726acf7948f272e0e04bc44dac5068063731953cc7500803f2e1f8c052ed
+  data.tar.gz: 64d9f0fb75ef197facaf6538a4c46c5dad11a37e196e8b37de83ba2e629aeb24
 SHA512:
-  metadata.gz: fbc51cb331674163af43d4e952bce6ec936db4e3235ca356082a83211ae552d84409bebcdffcb364c09ed8099504ac8418d2fffed3d273d4392d762d99098d59
-  data.tar.gz: e57d9545b5acec4ca25c1c5a30b1987d3d9769f027725e6613f396e7b2bedbe352620278a6cb8c613d5e1a1c1ecabb0e446bf2a4b6ae0339bedf8a4563a33b01
+  metadata.gz: ae2923599925002c945f3a488e8eafac91037075a88815975a3c920cddc195732f4186fca51dd6b98485a60e84d9ef2d55fb1fa439e9d88a3203c8f4cd6b9f75
+  data.tar.gz: c8f6691b41a419dd5adfbe89b0399d4624f3ae06b7c5a10fdffcbaee774b2009567ba687ab65b46aa0554dcbb31408212c3ebe7bef66dfb892f19191776fa8fe

data/CHANGELOG.md

@@ -7,7 +7,68 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.14.0] - 2026-06-17
+## [0.16.0] - 2026-06-21
+
+### Added
+- **Opt-in `#redact` refinements.** `require "data_redactor/refinements"` then
+  `using DataRedactor::Refinements` adds `#redact` to `String` (→
+  `DataRedactor.redact`) and to `Hash`/`Array` (→ `DataRedactor.redact_deep`), e.g.
+  `"email a@b.com".redact` and `chat.ask(user_input.redact)`. Refinements are
+  lexically scoped, so they never pollute the core classes globally — apps that
+  don't opt in are unaffected and there is no collision risk. Forwards
+  `only:`/`except:`/`placeholder:`; never mutates the receiver.
+  `DataRedactor.redact` remains the primary API.
+- **Length-aware placeholder modes.** `placeholder: :length` replaces each match
+  with `[REDACTED:N]` and `placeholder: :tagged_length` with `[REDACTED:TAGNAME:N]`,
+  where `N` is the **byte length** of the redacted value. Readers can gauge what
+  was there without seeing it. Both compose with `only:`/`except:` and are
+  forwarded by `redact_deep`, `redact_json`, and the integrations. Additive —
+  two new values for the existing `placeholder:` keyword; no behaviour changes.
+- **CI: ASan/UBSan memory-safety gate.** New job builds the matcher engine
+  standalone under `-fsanitize=address,undefined` and drives it over an
+  adversarial corpus + seeded fuzz loop (catches the `OP_EOL`-class OOB read).
+- **CI: throughput-trend history + PR comment.** New job records the C/pure-Ruby
+  ratio over time (history in `actions/cache`, no gh-pages), posts a sticky PR
+  comment comparing each run to the previous point, and fails on a >10% drop.
+
+## [0.15.0] - 2026-06-17
+
+### Changed
+- **Overlap resolution is now longest-match-wins** (was earlier-index-wins). When
+  two patterns match overlapping spans, the engine keeps the **longer** span;
+  equal-length ties go to the lower pattern index (preserving prior behaviour for
+  same-length matches). The previous "earliest pattern by index wins any region it
+  can match" semantic was an accidental by-product of sequential per-pattern
+  rewriting, and it could leave a secret **partly unredacted** — e.g.
+  `AKIA…EXAMPLE` followed by 20 more alphanumeric bytes used to redact only the
+  20-char access-key prefix and leak the trailing 20 bytes; it now redacts the full
+  40-char secret. The public API (`redact`, `scan`) is unchanged; `scan` may report
+  one longer match where it previously reported several shorter overlapping ones.
+  Aligns with Onigmo/PCRE/RE2/Hyperscan semantics. Resolver only — no measurable
+  throughput change (still ~2.4× over pure-Ruby on the 1 MB log).
+
+### Added
+- **CI throughput regression gate** (`throughput-gate` job). Runs
+  `benchmark/ci_throughput_gate.rb`, which gates on the ratio of the C engine to
+  a pure-Ruby gsub loop over the same patterns (the ratio cancels CI-runner
+  speed variance, unlike absolute MB/s). Loose floor (1.5×; known result
+  ~2.25×), informational throughput output, plus a correctness guard so an
+  engine that redacts less cannot pass as "faster". Repo/CI only — not packaged.
+
+## [0.14.1] - 2026-06-17
+
+### Changed
+- **Bounded the greedy tails of seven built-in token patterns** (`jwt`,
+  `grafana_api_token`, `ssh_public_key`, `bearer_token`, `anthropic_api_key`,
+  `openai_project_api_key`, `sendgrid_api_key`). Open-ended quantifiers (`+` and
+  `{n,}`) are capped at the POSIX `RE_DUP_MAX` of 255 (`{n,255}`), matching the
+  existing `hashicorp_vault_batch_token` precedent. A token is unusable once its
+  front is redacted, so a bounded prefix is sufficient to neutralize it. This
+  restores a finite `max_len` for these patterns (re-enabling the engine's
+  literal back-up skip) and removes a theoretical O(N²) worst case where a
+  crafted prefix plus a megabyte of matching characters forces a long greedy
+  scan. Tokens longer than 255 characters are still neutralized — only a
+  cryptographically-dead tail may remain.
 
 ### Added
 - **Key-name-anchored secret redaction** (`:credentials`). A new pattern tier
@@ -275,7 +336,10 @@ features as 0.7.1 plus the pipeline fix.
 - `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
 - RSpec suite with one example per pattern.
 
-[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...HEAD
+[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.16.0...HEAD
+[0.16.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...v0.16.0
+[0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
+[0.14.1]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...v0.14.1
 [0.14.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.11.0...v0.13.0
 [0.11.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.10.1...v0.11.0

data/README.md

@@ -6,16 +6,22 @@
 
 A Ruby gem with a C extension for high-performance regex-based redaction of sensitive data from strings.
 
+> 📄 The engineering behind the v19 matching engine is written up as an experience
+> report, *"The Fastest Engine Is Not the Shippable Engine: Replacing a Regex Engine
+> for Data Redaction Under Production Constraints,"* currently under review at
+> *Software: Practice and Experience* (Manuscript ID 7985366). Source and the
+> reproducibility bundle are in [`paper/`](paper/).
+
 ## What it does
 
 DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs,
 credit cards, national IDs, emails, phone numbers, IPs, and more — and replaces
 each match with a placeholder. The scanning runs in a C extension backed by a
 zero-dependency Thompson NFA → lazy-DFA multi-pattern engine (v19) that scans
-all 88 built-in patterns in a single pass — 2–2.5× faster than pure-Ruby `gsub`
+every built-in pattern in a single pass — 2–2.5× faster than pure-Ruby `gsub`
 on large payloads, with no external library dependencies.
 
-It ships **88 built-in patterns** across 15+ countries, grouped into tags
+It ships **89 built-in patterns** across 15+ countries, grouped into tags
 (`:credentials`, `:financial`, `:contact`, ...) so you can redact only what you
 care about. Beyond plain strings it can walk nested Hashes, Arrays, and JSON,
 audit a payload without mutating it (`scan`), and plug into Logger, Rails, and
@@ -103,9 +109,18 @@ DataRedactor.redact(text, placeholder: :hash)
 # "user@example.com"  → "[CONTACT_3d7a]"
 # "user@example.com"  → "[CONTACT_3d7a]"  (same every time)
 # "other@example.com" → "[CONTACT_91fc]"  (different value, different hash)
+
+# Length — embeds the byte length of the redacted value, so readers can
+# gauge what was there without seeing it.
+DataRedactor.redact(text, placeholder: :length)
+# "user@example.com"  → "[REDACTED:16]"
+
+# Tagged length — tag name plus byte length.
+DataRedactor.redact(text, placeholder: :tagged_length)
+# "user@example.com"  → "[REDACTED:CONTACT:16]"
 ```
 
-All three modes compose with `only:` and `except:`:
+All modes compose with `only:` and `except:`:
 
 ```ruby
 DataRedactor.redact(text, only: :contact, placeholder: :tagged)
@@ -165,6 +180,24 @@ safe_json = DataRedactor.redact_json('{"email":"alice@example.com","count":3}')
 DataRedactor.redact_json("not json")  # => JSON::ParserError
 ```
 
+### `#redact` refinements (opt-in)
+
+Prefer `"text".redact` over `DataRedactor.redact("text")`? Opt into the refinement. It adds `#redact` to `String` (via `redact`) and to `Hash`/`Array` (via `redact_deep`) **only in the files that `using` it** — refinements are lexically scoped, so the core classes are never monkey-patched globally and there is no collision risk for apps that don't opt in. `DataRedactor.redact` remains the primary API.
+
+```ruby
+require "data_redactor/refinements"
+using DataRedactor::Refinements
+
+"email alice@example.com".redact            # => "email [REDACTED]"
+{ token: "AKIAIOSFODNN7EXAMPLE" }.redact    # => { token: "[REDACTED]" }
+["a@b.com", 3].redact                       # => ["[REDACTED]", 3]
+
+# Handy right before sending text to an LLM:
+chat.ask(user_input.redact)
+```
+
+`#redact` forwards `only:`/`except:`/`placeholder:` and never mutates the receiver. Without `using DataRedactor::Refinements` in the current file, `#redact` is not defined.
+
 ### Custom patterns
 
 Teams often have internal IDs that the gem can't ship. Register them at boot — or at runtime from any thread (registration is thread-safe, see [Thread safety](#thread-safety)):
@@ -309,7 +342,25 @@ safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
 
 `content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
 
-## Detected patterns (88 total)
+### RubyLLM
+
+[RubyLLM](https://rubyllm.com) is a unified Ruby client for every major LLM provider — and a perfect match for `data_redactor`: anything you send to a model is exactly the kind of free text that leaks secrets and PII. Because RubyLLM takes plain strings, you can scrub them with `DataRedactor.redact` before they leave the process — no extra integration required:
+
+```ruby
+require "ruby_llm"
+require "data_redactor"
+
+chat = RubyLLM.chat(model: "claude-opus-4-8")
+chat.with_instructions(DataRedactor.redact("You are a support agent for ACME Corp."))
+
+user_input = "My card is 4111 1111 1111 1111 and my email is alice@example.com"
+chat.ask(DataRedactor.redact(user_input))
+# the model receives: "My card is [REDACTED] and my email is [REDACTED]"
+```
+
+Wrap each prompt (and any `with_instructions` system prompt) in `DataRedactor.redact` before passing it to `ask`. This is a per-call step you opt into — RubyLLM does not yet expose a request hook for automatic, transparent redaction of every outbound call ([crmne/ruby_llm#765](https://github.com/crmne/ruby_llm/issues/765) tracks the connection-middleware hook that would enable it).
+
+## Detected patterns (89 total)
 
 The table below is a representative sample. Use `DataRedactor.pattern_names` for the canonical, machine-readable list — it stays in sync with the C extension automatically.
 
@@ -435,7 +486,7 @@ redactor/
 │   ├── README.md                 # How to run, what each script measures
 │   ├── support/corpus.rb         # Shared payload builders + pure-Ruby baseline redactor
 │   ├── throughput.rb             # MB/s on representative payloads
-│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same 88 patterns)
+│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same patterns)
 │   ├── scaling.rb                # Runtime vs input size 1KB → 50MB
 │   └── per_pattern.rb            # Per-pattern scan cost
 └── docs/                         # Design and execution docs for future work
@@ -523,7 +574,7 @@ different angles. They are **not** packaged with the gem.
 ```bash
 bundle install                                   # pulls benchmark-ips, benchmark-memory (dev deps)
 bundle exec rake compile
-bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same 88 patterns
+bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same patterns
 bundle exec ruby benchmark/throughput.rb         # MB/s on a log line, JSON, 1MB and 10MB log files
 bundle exec ruby benchmark/scaling.rb            # runtime vs input size (1KB → 50MB), confirms linear scaling
 bundle exec ruby benchmark/per_pattern.rb        # per-pattern scan cost over a 1MB payload
@@ -535,11 +586,8 @@ C engine uses, via `DataRedactor::BUILTIN_PATTERN_SOURCES`).
 
 ### Performance (0.10.0 — v19 multi-pattern engine)
 
-As of 0.10.0 the C extension runs a **Thompson NFA → lazy-DFA multi-pattern
-engine** (v19) that scans the input once across all 88 built-in patterns,
-with two selective-merge passes (pure-digit group + IBAN union) that further
-reduce work for the most common pattern classes. Custom patterns (`add_pattern`)
-still use the glibc path (required for correct UTF-8 diacritic matching).
+Measured on the v19 engine ([How it works](#how-it-works)) vs a pure-Ruby `gsub`
+loop over the same patterns:
 
 | Payload               | v19 engine (0.10.0) | Pure-Ruby `gsub` | Ratio           |
 |-----------------------|---------------------|------------------|-----------------|
@@ -576,14 +624,14 @@ machine-dependent, but the flat curve is not.
 
 ## How it works
 
-1. At load time, `Init_data_redactor` compiles all 85 regex patterns once using `regcomp` (POSIX ERE) and stores them as static `regex_t` structs. Patterns marked as boundary-wrapped are expanded with `wrap_boundary()` before compilation.
-2. `DataRedactor.redact(text)` receives a Ruby `String`, converts it to a C `char*` via `StringValueCStr`, and runs each compiled pattern in sequence on a working buffer.
-3. For each pattern, `replace_all_matches` iterates using `regexec`, copies non-matching segments to a fresh output buffer, and inserts `[REDACTED]` in place of each match. For boundary-wrapped patterns, `regexec` is called with `nmatch=4` and sub-match groups `[1]`/`[3]` identify the boundary characters so they are preserved verbatim.
-4. The output buffer is grown with `realloc` as needed. After all patterns are applied the result is returned as a Ruby `String` via `rb_str_new_cstr`. All intermediate `malloc`/`strdup` allocations are explicitly `free`d.
+1. At load time, `mm_init()` compiles every built-in pattern from a Thompson NFA into bytecode, lazily building each pattern's DFA on first use (interned and cached). Boundary-wrapped patterns are expanded with the word-boundary group before compilation.
+2. `DataRedactor.redact(text)` / `scan(text)` hand the input to the v19 engine, which scans it **once** and emits `(pattern_id, start, length)` events for every enabled pattern. Two selective-merge passes (a pure-digit group and an IBAN union) collapse the most common pattern classes into shared scans. The single pass over the original buffer is what makes the engine O(N).
+3. The raw events are resolved by `mm_resolve` under the **longest-match-wins** policy: overlapping spans are reduced to a non-overlapping set keeping the longest match at each position, with the lower pattern index breaking equal-length ties.
+4. `redact` rewrites the surviving spans to placeholders in one buffer build (preserving the boundary characters of boundary-wrapped matches); `scan` returns the event list with byte offsets into the original string. Custom patterns (`add_pattern`) run on the glibc `regexec` path afterward — required for correct UTF-8 diacritic matching.
 
 ## Memory management
 
-All C-side buffers are heap-allocated with `malloc`/`strdup` and freed before the function returns. The only Ruby-managed allocation is the final return value from `rb_str_new_cstr`. No Ruby objects are created mid-processing, so GC cannot collect anything out from under the C code.
+All C-side working buffers are heap-allocated and freed before the call returns; the only Ruby-managed allocation is the final result `String`. No Ruby objects are created mid-scan, so GC cannot collect anything out from under the C code. Per-thread engine scratch (NFA state, lazy-DFA cache) is freed automatically when the thread exits — see [Thread safety](#thread-safety).
 
 ## Thread safety
 
@@ -601,7 +649,6 @@ Released under the [MIT License](LICENSE).
 
 ## Known limitations
 
-- **Pattern ordering matters** — patterns run sequentially. An early broad pattern (e.g. the 9-digit passport) may consume digits that a later pattern (e.g. credit card) depends on. Boundary wrapping mitigates this for pure-digit patterns.
 - **AWS Secret Key (pattern 1)** — 40 consecutive base64 characters is a broad match. It can produce false positives in base64-encoded content such as embedded images or binary blobs.
 - **Duplicate digit patterns** — several national ID formats share the same digit-length (11 digits: PESEL, Norwegian Fødselsnummer, Belgian National Number). They are kept as separate slots for clarity but the practical effect is that any 11-digit boundary-delimited number will be redacted.
-- **Single-pass overlap semantics** — built-in patterns are resolved by an index-order greedy claim: the lower-index pattern wins any region it matches. When two secrets abut with no separator, a rewrite-created word boundary can cause the second to be missed. This is rare in real text (secrets are almost always separator-delimited) and will be fixed by the upcoming longest-match-wins resolver in 1.0.
+- **Overlap resolution is longest-match-wins** — when two patterns match overlapping spans the engine keeps the longer span; equal-length ties go to the lower pattern index. This favours redacting *more* when uncertain (a 40-char secret is redacted whole rather than leaking the bytes past a shorter prefix match). When two secrets abut with **no separator** between them, a boundary-wrapped pattern can fail to match because the original buffer has no word boundary where one token meets the next, leaving the abutting token unredacted. This is rare in real text (secrets are almost always separator-delimited).

data/ext/data_redactor/data_redactor.c

@@ -68,9 +68,11 @@ void Init_data_redactor(void) {
     rb_define_const(mDataRedactor, "BUILTIN_PATTERN_BOUNDARY", rb_ary_freeze(builtin_boundary));
 
     /* Placeholder mode constants. */
-    rb_define_const(mDataRedactor, "PH_MODE_PLAIN",  INT2NUM(PLACEHOLDER_MODE_PLAIN));
-    rb_define_const(mDataRedactor, "PH_MODE_TAGGED", INT2NUM(PLACEHOLDER_MODE_TAGGED));
-    rb_define_const(mDataRedactor, "PH_MODE_HASH",   INT2NUM(PLACEHOLDER_MODE_HASH));
+    rb_define_const(mDataRedactor, "PH_MODE_PLAIN",         INT2NUM(PLACEHOLDER_MODE_PLAIN));
+    rb_define_const(mDataRedactor, "PH_MODE_TAGGED",        INT2NUM(PLACEHOLDER_MODE_TAGGED));
+    rb_define_const(mDataRedactor, "PH_MODE_HASH",          INT2NUM(PLACEHOLDER_MODE_HASH));
+    rb_define_const(mDataRedactor, "PH_MODE_LENGTH",        INT2NUM(PLACEHOLDER_MODE_LENGTH));
+    rb_define_const(mDataRedactor, "PH_MODE_TAGGED_LENGTH", INT2NUM(PLACEHOLDER_MODE_TAGGED_LENGTH));
 
     /* Tag bitmask values used by the Ruby wrapper to build only/except masks. */
     rb_define_const(mDataRedactor, "TAG_CREDENTIALS", INT2NUM(TAG_CREDENTIALS));

data/ext/data_redactor/matcher.c

@@ -16,10 +16,10 @@
  *
  *   2. Output contract. mm_scan() takes an enable_bits gate and emits ORIGINAL-
  *      frame (pattern_id, start, span) events for ALL enabled patterns in one
- *      pass; it does NOT model the gem's cross-pattern sequential rewrite. The
- *      caller applies mm_resolve() (index-order greedy claim) to reproduce
- *      today's "earlier-index pattern wins" semantics byte-for-byte. See
- *      TODO.md §1d Gap 5 and the AKIA specs in spec/data_redactor_spec.rb.
+ *      pass. The caller applies mm_resolve() (longest-match-wins greedy claim:
+ *      longest span at each position wins, equal lengths broken by lower
+ *      pattern_id) to pick the final non-overlapping set. See TODO.md §1d Gap 5
+ *      and the overlap-resolution specs in spec/data_redactor_spec.rb.
  *
  * The infix-literal classification and the BM_INFIX hint table below are ported
  * from prototypes/multi_pattern_matcher/gen_patterns.rb (which derived them from the
@@ -1260,14 +1260,14 @@ size_t mm_scan(const char *input, size_t len,
     return count;
 }
 
-/* Order events for the index-order greedy claim: ascending pattern_id, then
- * ascending start (so a lower-index pattern always gets first claim on a region;
- * within a pattern, earlier matches are seen first). */
+/* Order events for the longest-match-wins greedy claim: ascending start, then
+ * descending length (so the longest span at a given start is seen first), then
+ * ascending pattern_id (lower index wins a tie of equal length). */
 static int ev_cmp_resolve(const void *a, const void *b) {
     const mm_match_t *x = a, *y = b;
-    if (x->pattern_id != y->pattern_id) return x->pattern_id - y->pattern_id;
     if (x->start != y->start) return x->start < y->start ? -1 : 1;
-    return 0;
+    if (x->length != y->length) return x->length > y->length ? -1 : 1;
+    return x->pattern_id - y->pattern_id;
 }
 
 /* Order kept events for emission: ascending start. */
@@ -1281,12 +1281,12 @@ size_t mm_resolve(mm_match_t *ev, size_t n) {
     if (n == 0) return 0;
     qsort(ev, n, sizeof(mm_match_t), ev_cmp_resolve);
 
-    /* Greedy claim in (pattern_id, start) order. An event is kept iff its span
-     * [start, start+length) does not overlap any already-kept span. Kept spans
-     * are accumulated in `kept`; we check membership against them. n is small
-     * for typical inputs, but to stay linear-ish we keep `kept` sorted by start
-     * and binary-search the neighbourhood. For simplicity and because match
-     * counts are modest, a linear overlap check against the kept set is used. */
+    /* Greedy claim in (start, -length, pattern_id) order: the longest span at
+     * each position is offered first and claims its region; any later (shorter,
+     * or equal-length higher-id) event overlapping an already-kept span is
+     * dropped. An event is kept iff its span [start, start+length) does not
+     * overlap any already-kept span. Match counts are modest, so a linear
+     * overlap check against the kept set is used. */
     mm_match_t *kept = mm_xmalloc(n * sizeof(mm_match_t));
     size_t nk = 0;
     for (size_t i = 0; i < n; i++) {

data/ext/data_redactor/matcher.h

@@ -53,19 +53,20 @@ void mm_clear_custom(void);
  * array disables out-of-range patterns. Events carry ORIGINAL-frame offsets.
  *
  * Events are NOT pre-resolved for cross-pattern overlap — the caller applies
- * the index-order greedy claim (mm_resolve) to reproduce the gem's sequential
- * per-pattern rewrite semantics.
+ * the longest-match-wins greedy claim (mm_resolve) to pick the final
+ * non-overlapping set.
  */
 size_t mm_scan(const char *input, size_t len,
                const int *enable_bits, size_t n_bits,
                mm_match_t *out, size_t max);
 
 /*
- * Resolve raw scan events into the non-overlapping set the gem's sequential
- * per-pattern rewrite would produce: in (pattern_id, start) order, keep an
- * event iff its CORE span does not overlap an already-kept span. Sorts `ev`
- * in place and returns the kept count (compacted to the front of `ev`), in
- * ascending start order. n_total is the pattern-id upper bound for ordering.
+ * Resolve raw scan events into the final non-overlapping set under the
+ * longest-match-wins policy: process events in (start asc, length desc,
+ * pattern_id asc) order and keep an event iff its CORE span does not overlap an
+ * already-kept span. The longest match at each position wins; equal-length ties
+ * go to the lower pattern_id. Sorts `ev` in place and returns the kept count
+ * (compacted to the front of `ev`), in ascending start order.
  */
 size_t mm_resolve(mm_match_t *ev, size_t n);
 

data/ext/data_redactor/patterns.c

@@ -425,18 +425,21 @@ const char *pattern_strings[NUM_PATTERNS] = {
     /* ---- Tier 2: Long prefixed tokens ---- */
     /*  6: GitHub PAT fine-grained (github_pat_ + 82 chars) */
     "github_pat_[0-9a-zA-Z_]{82}",
-    /*  7: JWT (three base64url segments) */
-    "eyJ[A-Za-z0-9_-]{10,}\\.eyJ[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]+",
+    /*  7: JWT (three base64url segments). Tails bounded at RE_DUP_MAX (255):
+     * a JWT is unusable once its front is gone, so a bounded prefix is enough to
+     * neutralize it. Bounding restores a finite max_len (re-enables the engine's
+     * literal back-up skip) and removes the O(N^2) greedy-tail worst case. */
+    "eyJ[A-Za-z0-9_-]{10,255}\\.eyJ[A-Za-z0-9_-]{10,255}\\.[A-Za-z0-9_-]{1,255}",
     /*  8: Grafana API Token (base64 of {\"k\":\") */
-    "eyJrIjoi[A-Za-z0-9_=-]{42,}",
+    "eyJrIjoi[A-Za-z0-9_=-]{42,255}",
     /*  9: SSH Public Key */
-    "ssh-(rsa|ed25519|ecdsa) [a-zA-Z0-9/+=]{20,}",
+    "ssh-(rsa|ed25519|ecdsa) [a-zA-Z0-9/+=]{20,255}",
     /* 10: Bearer Token */
-    "[Bb]earer [a-zA-Z0-9_.=/+:-]{12,}",
+    "[Bb]earer [a-zA-Z0-9_.=/+:-]{12,255}",
     /* 11: Anthropic API Key (sk-ant-apiNN-... ~ 95+ chars) */
-    "sk-ant-api[0-9]{2}-[A-Za-z0-9_-]{90,}",
+    "sk-ant-api[0-9]{2}-[A-Za-z0-9_-]{90,255}",
     /* 12: OpenAI Project API Key (sk-proj-...) */
-    "sk-proj-[A-Za-z0-9_-]{20,}",
+    "sk-proj-[A-Za-z0-9_-]{20,255}",
     /* 13: Google API Key (AIza + 35 chars) */
     "AIza[0-9A-Za-z_-]{35}",
     /* 14: AWS Access Key ID (all prefixes + 16 chars) */
@@ -444,7 +447,7 @@ const char *pattern_strings[NUM_PATTERNS] = {
     /* 15: AWS Secret Access Key (40 base64 chars) */
     "[A-Za-z0-9/+=]{40}",
     /* 16: SendGrid API Key */
-    "SG\\.[a-zA-Z0-9_-]{5,}\\.[a-zA-Z0-9_-]{5,}",
+    "SG\\.[a-zA-Z0-9_-]{5,255}\\.[a-zA-Z0-9_-]{5,255}",
     /* 17: Amazon MWS Auth Token */
     "amzn\\.mws\\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}",
     /* 18: LaunchDarkly API Key (api-UUID or sdk-UUID) */

data/ext/data_redactor/placeholder.c

@@ -20,6 +20,10 @@ size_t write_placeholder(char *buf, const placeholder_t *ph,
             unsigned int h = djb2(match, match_len) & 0xFFFF;
             return (size_t)sprintf(buf, "[%s_%04x]", ph->str, h);
         }
+        case PLACEHOLDER_MODE_LENGTH:
+            return (size_t)sprintf(buf, "[REDACTED:%zu]", match_len);
+        case PLACEHOLDER_MODE_TAGGED_LENGTH:
+            return (size_t)sprintf(buf, "[REDACTED:%s:%zu]", ph->str, match_len);
         default: /* PLACEHOLDER_MODE_PLAIN */
             {
                 size_t len = strlen(ph->str);
@@ -29,11 +33,17 @@ size_t write_placeholder(char *buf, const placeholder_t *ph,
     }
 }
 
+/* Widest decimal a size_t byte-length can print to (UINT64_MAX is 20 digits). */
+#define MAX_LEN_DIGITS 20
+
 size_t max_placeholder_len(const placeholder_t *ph) {
     size_t tag_len = strlen(ph->str);
     switch (ph->mode) {
         case PLACEHOLDER_MODE_TAGGED: return 2 + 9 + tag_len + 1; /* "[REDACTED:" + tag + "]" */
         case PLACEHOLDER_MODE_HASH:   return 1 + tag_len + 1 + 4 + 1; /* "[" + tag + "_" + 4hex + "]" */
+        case PLACEHOLDER_MODE_LENGTH: return 10 + MAX_LEN_DIGITS + 1; /* "[REDACTED:" + digits + "]" */
+        case PLACEHOLDER_MODE_TAGGED_LENGTH:
+            return 10 + tag_len + 1 + MAX_LEN_DIGITS + 1; /* "[REDACTED:" + tag + ":" + digits + "]" */
         default:                      return tag_len;
     }
 }

data/ext/data_redactor/placeholder.h

@@ -3,9 +3,11 @@
 
 #include <stddef.h>
 
-#define PLACEHOLDER_MODE_PLAIN  0  /* use ph.str verbatim                  */
-#define PLACEHOLDER_MODE_TAGGED 1  /* "[REDACTED:TAGNAME]"                 */
-#define PLACEHOLDER_MODE_HASH   2  /* "[TAGNAME_xxxx]" (4-hex djb2 suffix) */
+#define PLACEHOLDER_MODE_PLAIN         0  /* use ph.str verbatim                  */
+#define PLACEHOLDER_MODE_TAGGED        1  /* "[REDACTED:TAGNAME]"                 */
+#define PLACEHOLDER_MODE_HASH          2  /* "[TAGNAME_xxxx]" (4-hex djb2 suffix) */
+#define PLACEHOLDER_MODE_LENGTH        3  /* "[REDACTED:16]" (byte-length)        */
+#define PLACEHOLDER_MODE_TAGGED_LENGTH 4  /* "[REDACTED:TAGNAME:16]"              */
 
 typedef struct {
     int         mode;

data/lib/data_redactor/refinements.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require "data_redactor"
+
+module DataRedactor
+  # Opt-in refinements that add a `#redact` method to `String`, `Hash`, and
+  # `Array` as sugar over {DataRedactor.redact} / {DataRedactor.redact_deep}.
+  #
+  # Refinements are lexically scoped: `#redact` exists only in files that
+  # `using DataRedactor::Refinements`, so loading this file never pollutes the
+  # core classes globally. Apps that don't opt in are unaffected, and there is
+  # no collision risk with other libraries' `String#redact`.
+  #
+  # `DataRedactor.redact` remains the primary API; this is convenience only.
+  #
+  # @example
+  #   require "data_redactor/refinements"
+  #   using DataRedactor::Refinements
+  #
+  #   "email alice@example.com".redact            #=> "email [REDACTED]"
+  #   { token: "AKIAIOSFODNN7EXAMPLE" }.redact    #=> { token: "[REDACTED]" }
+  #   chat.ask(user_input.redact)                 # scrub before sending to an LLM
+  module Refinements
+    refine String do
+      # Redact this String via {DataRedactor.redact}. Returns a new String; the
+      # receiver is not mutated.
+      #
+      # @param only forwarded to {DataRedactor.redact}
+      # @param except forwarded to {DataRedactor.redact}
+      # @param placeholder forwarded to {DataRedactor.redact}
+      # @return [String] the redacted copy.
+      # @example
+      #   "ssn 123-45-6789".redact #=> "ssn [REDACTED]"
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+
+    refine Hash do
+      # Deep-redact this Hash's String values via {DataRedactor.redact_deep}.
+      # Returns a deep copy; the receiver is not mutated and keys are untouched.
+      #
+      # @param only forwarded to {DataRedactor.redact_deep}
+      # @param except forwarded to {DataRedactor.redact_deep}
+      # @param placeholder forwarded to {DataRedactor.redact_deep}
+      # @return [Hash] a deep copy with String leaves redacted.
+      # @example
+      #   { email: "a@b.com" }.redact #=> { email: "[REDACTED]" }
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+
+    refine Array do
+      # Deep-redact this Array's String elements via {DataRedactor.redact_deep}.
+      # Returns a deep copy; the receiver is not mutated.
+      #
+      # @param only forwarded to {DataRedactor.redact_deep}
+      # @param except forwarded to {DataRedactor.redact_deep}
+      # @param placeholder forwarded to {DataRedactor.redact_deep}
+      # @return [Array] a deep copy with String leaves redacted.
+      # @example
+      #   ["a@b.com", 3].redact #=> ["[REDACTED]", 3]
+      def redact(only: nil, except: nil, placeholder: DataRedactor::PLACEHOLDER_DEFAULT)
+        DataRedactor.redact_deep(self, only: only, except: except, placeholder: placeholder)
+      end
+    end
+  end
+end

data/lib/data_redactor/version.rb

@@ -1,4 +1,4 @@
 module DataRedactor
   # Current gem version. Follows {https://semver.org Semantic Versioning 2.0.0}.
-  VERSION = "0.14.0"
+  VERSION = "0.16.0"
 end

data/lib/data_redactor.rb

@@ -23,8 +23,10 @@ require_relative "data_redactor/name_pattern"
 #
 # @example Custom placeholder
 #   DataRedactor.redact(text, placeholder: "***")
-#   DataRedactor.redact(text, placeholder: :tagged) # => "[REDACTED:CONTACT]"
-#   DataRedactor.redact(text, placeholder: :hash)   # => "[CONTACT_a3f9]"
+#   DataRedactor.redact(text, placeholder: :tagged)        # => "[REDACTED:CONTACT]"
+#   DataRedactor.redact(text, placeholder: :hash)          # => "[CONTACT_a3f9]"
+#   DataRedactor.redact(text, placeholder: :length)        # => "[REDACTED:16]"
+#   DataRedactor.redact(text, placeholder: :tagged_length) # => "[REDACTED:CONTACT:16]"
 #
 # @example Audit / dry-run
 #   DataRedactor.scan(text)
@@ -125,12 +127,15 @@ module DataRedactor
   #   and/or pattern name(s).
   # @param except [Symbol, String, Array, nil] exclude the given tag(s)
   #   and/or pattern name(s). May be combined with +only:+.
-  # @param placeholder [String, :tagged, :hash] replacement strategy.
-  #   A String is used verbatim. +:tagged+ produces +[REDACTED:TAGNAME]+.
-  #   +:hash+ produces a deterministic +[TAGNAME_xxxx]+ token (4-hex djb2)
-  #   so the same input value always maps to the same token.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length]
+  #   replacement strategy. A String is used verbatim. +:tagged+ produces
+  #   +[REDACTED:TAGNAME]+. +:hash+ produces a deterministic +[TAGNAME_xxxx]+
+  #   token (4-hex djb2) so the same input value always maps to the same token.
+  #   +:length+ produces +[REDACTED:N]+ and +:tagged_length+ produces
+  #   +[REDACTED:TAGNAME:N]+, where +N+ is the byte length of the redacted value.
   # @return [String] a new string with every match replaced.
-  # @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash.
+  # @raise [ArgumentError] if +placeholder:+ is not a String/:tagged/:hash/
+  #   :length/:tagged_length.
   # @raise [UnknownTagError] if any Symbol in +only:+/+except:+ is not in {TAGS}.
   # @raise [UnknownPatternError] if any String in +only:+/+except:+ is not in {pattern_names}.
   #
@@ -194,7 +199,7 @@ module DataRedactor
   #   Any type is accepted; non-String scalars are returned as-is.
   # @param only [Symbol, String, Array, nil] forwarded to {redact}.
   # @param except [Symbol, String, Array, nil] forwarded to {redact}.
-  # @param placeholder [String, :tagged, :hash] forwarded to {redact}.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
   # @return [Hash, Array, String, Object] a new structure of the same shape
   #   with all String leaves redacted.
   # @raise [ArgumentError] if the structure contains a circular reference.
@@ -217,7 +222,7 @@ module DataRedactor
   # @param json_string [String] valid JSON input.
   # @param only [Symbol, String, Array, nil] forwarded to {redact}.
   # @param except [Symbol, String, Array, nil] forwarded to {redact}.
-  # @param placeholder [String, :tagged, :hash] forwarded to {redact}.
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length] forwarded to {redact}.
   # @return [String] a JSON string with all String values redacted.
   # @raise [JSON::ParserError] if +json_string+ is not valid JSON.
   #
@@ -425,17 +430,20 @@ module DataRedactor
   # Translate the user-facing +placeholder:+ value into the +(mode_int, str)+
   # pair the C layer expects.
   #
-  # @param placeholder [String, :tagged, :hash]
+  # @param placeholder [String, :tagged, :hash, :length, :tagged_length]
   # @return [Array(Integer, String)]
   # @raise [ArgumentError] if +placeholder+ is none of the accepted values.
   def resolve_placeholder(placeholder)
     case placeholder
-    when :tagged then [PH_MODE_TAGGED, ""]
-    when :hash   then [PH_MODE_HASH,   ""]
-    when String  then [PH_MODE_PLAIN,  placeholder]
+    when :tagged        then [PH_MODE_TAGGED,        ""]
+    when :hash          then [PH_MODE_HASH,          ""]
+    when :length        then [PH_MODE_LENGTH,        ""]
+    when :tagged_length then [PH_MODE_TAGGED_LENGTH, ""]
+    when String         then [PH_MODE_PLAIN,         placeholder]
     else
       raise ArgumentError,
-        "placeholder must be a String, :tagged, or :hash — got #{placeholder.inspect}"
+        "placeholder must be a String, :tagged, :hash, :length, or :tagged_length " \
+        "— got #{placeholder.inspect}"
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.14.0
+  version: 0.16.0
 platform: ruby
 authors:
 - Daniele Frisanco
@@ -145,6 +145,7 @@ files:
 - lib/data_redactor/integrations/rack.rb
 - lib/data_redactor/integrations/rails.rb
 - lib/data_redactor/name_pattern.rb
+- lib/data_redactor/refinements.rb
 - lib/data_redactor/version.rb
 homepage: https://github.com/danielefrisanco/data_redactor
 licenses: