data_redactor 0.14.0-x86_64-linux → 0.15.0-x86_64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 86ed390986b76cb2bcb48a80c322be42b8bd91dcd5b9afb8299a6ab4c0dd9419
-  data.tar.gz: ca7260eff77b03d15392b0c763e56ee7b5bf146eb7f0b7597f78f41d2a6dfe08
+  metadata.gz: 9e46ff430308e5ed5a1908907a393e88ec383c0f1cfa7a068fd39ada4a196d59
+  data.tar.gz: '094f0ebbfdbd0299377adc33db8725e2dec5f974df4281403bbb829e09ee8519'
 SHA512:
-  metadata.gz: a683277146fbb1e19caefffe829a9de090d48b181f0d15cc98a625509f47da1a7cf4f7f69df165f4b077a321f2ff1ee125ef4881e5746949b62f6db87269e022
-  data.tar.gz: ecb6d0acfcf1a9b9c2c79f1d0e016ae4ad13d0c938e31c9d97f0b3bdc8f323ac7c7c1d1a2828067b3a5b9f33895e0e9bc3bb6a69bd9c7f324e2055cdeb082ee8
+  metadata.gz: b1b4862afe360943b379874dbc87853d25c2b686d84fe94db2e1b0946b843ec365b722223fc1c719767558657661a1bef0d30e7296f9b5da85711be19e86deda
+  data.tar.gz: 1e77bf2887c050776b7765d2d946d4f2b0650dfc29f5360859f08ddc24232b9bc34482ec3cf9cacf53726379b0f3fcb8c3700866e1870b747074e0eb4178667c

data/CHANGELOG.md

@@ -7,7 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.14.0] - 2026-06-17
+## [0.15.0] - 2026-06-17
+
+### Changed
+- **Overlap resolution is now longest-match-wins** (was earlier-index-wins). When
+  two patterns match overlapping spans, the engine keeps the **longer** span;
+  equal-length ties go to the lower pattern index (preserving prior behaviour for
+  same-length matches). The previous "earliest pattern by index wins any region it
+  can match" semantic was an accidental by-product of sequential per-pattern
+  rewriting, and it could leave a secret **partly unredacted** — e.g.
+  `AKIA…EXAMPLE` followed by 20 more alphanumeric bytes used to redact only the
+  20-char access-key prefix and leak the trailing 20 bytes; it now redacts the full
+  40-char secret. The public API (`redact`, `scan`) is unchanged; `scan` may report
+  one longer match where it previously reported several shorter overlapping ones.
+  Aligns with Onigmo/PCRE/RE2/Hyperscan semantics. Resolver only — no measurable
+  throughput change (still ~2.4× over pure-Ruby on the 1 MB log).
+
+### Added
+- **CI throughput regression gate** (`throughput-gate` job). Runs
+  `benchmark/ci_throughput_gate.rb`, which gates on the ratio of the C engine to
+  a pure-Ruby gsub loop over the same patterns (the ratio cancels CI-runner
+  speed variance, unlike absolute MB/s). Loose floor (1.5×; known result
+  ~2.25×), informational throughput output, plus a correctness guard so an
+  engine that redacts less cannot pass as "faster". Repo/CI only — not packaged.
+
+## [0.14.1] - 2026-06-17
+
+### Changed
+- **Bounded the greedy tails of seven built-in token patterns** (`jwt`,
+  `grafana_api_token`, `ssh_public_key`, `bearer_token`, `anthropic_api_key`,
+  `openai_project_api_key`, `sendgrid_api_key`). Open-ended quantifiers (`+` and
+  `{n,}`) are capped at the POSIX `RE_DUP_MAX` of 255 (`{n,255}`), matching the
+  existing `hashicorp_vault_batch_token` precedent. A token is unusable once its
+  front is redacted, so a bounded prefix is sufficient to neutralize it. This
+  restores a finite `max_len` for these patterns (re-enabling the engine's
+  literal back-up skip) and removes a theoretical O(N²) worst case where a
+  crafted prefix plus a megabyte of matching characters forces a long greedy
+  scan. Tokens longer than 255 characters are still neutralized — only a
+  cryptographically-dead tail may remain.
 
 ### Added
 - **Key-name-anchored secret redaction** (`:credentials`). A new pattern tier
@@ -275,7 +312,9 @@ features as 0.7.1 plus the pipeline fix.
 - `DataRedactor.redact(text)` module function returning the input with every match replaced by `[REDACTED]`.
 - RSpec suite with one example per pattern.
 
-[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...HEAD
+[Unreleased]: https://github.com/danielefrisanco/data_redactor/compare/v0.15.0...HEAD
+[0.15.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.1...v0.15.0
+[0.14.1]: https://github.com/danielefrisanco/data_redactor/compare/v0.14.0...v0.14.1
 [0.14.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.11.0...v0.13.0
 [0.11.0]: https://github.com/danielefrisanco/data_redactor/compare/v0.10.1...v0.11.0

data/README.md

@@ -12,10 +12,10 @@ DataRedactor scans text for sensitive data — API keys and cloud secrets, IBANs
 credit cards, national IDs, emails, phone numbers, IPs, and more — and replaces
 each match with a placeholder. The scanning runs in a C extension backed by a
 zero-dependency Thompson NFA → lazy-DFA multi-pattern engine (v19) that scans
-all 88 built-in patterns in a single pass — 2–2.5× faster than pure-Ruby `gsub`
+every built-in pattern in a single pass — 2–2.5× faster than pure-Ruby `gsub`
 on large payloads, with no external library dependencies.
 
-It ships **88 built-in patterns** across 15+ countries, grouped into tags
+It ships **89 built-in patterns** across 15+ countries, grouped into tags
 (`:credentials`, `:financial`, `:contact`, ...) so you can redact only what you
 care about. Beyond plain strings it can walk nested Hashes, Arrays, and JSON,
 audit a payload without mutating it (`scan`), and plug into Logger, Rails, and
@@ -309,7 +309,7 @@ safe_response = DataRedactor::Integrations::OpenAI.redact_response(response)
 
 `content` may be a plain String or an array of content blocks/parts (`{ type: "text", text: "..." }`) — only the `text` of `text` blocks is redacted; image and other block types pass through untouched. For Claude, a top-level `system:` String is also redacted; for OpenAI, a `{ role: "system" }` message in the array is redacted like any other. Pass a bare `messages` array or the whole request Hash (with a `messages` key) — either works.
 
-## Detected patterns (88 total)
+## Detected patterns (89 total)
 
 The table below is a representative sample. Use `DataRedactor.pattern_names` for the canonical, machine-readable list — it stays in sync with the C extension automatically.
 
@@ -435,7 +435,7 @@ redactor/
 │   ├── README.md                 # How to run, what each script measures
 │   ├── support/corpus.rb         # Shared payload builders + pure-Ruby baseline redactor
 │   ├── throughput.rb             # MB/s on representative payloads
-│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same 88 patterns)
+│   ├── vs_pure_ruby.rb           # C extension vs pure-Ruby gsub (same patterns)
 │   ├── scaling.rb                # Runtime vs input size 1KB → 50MB
 │   └── per_pattern.rb            # Per-pattern scan cost
 └── docs/                         # Design and execution docs for future work
@@ -523,7 +523,7 @@ different angles. They are **not** packaged with the gem.
 ```bash
 bundle install                                   # pulls benchmark-ips, benchmark-memory (dev deps)
 bundle exec rake compile
-bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same 88 patterns
+bundle exec ruby benchmark/vs_pure_ruby.rb       # head-to-head vs pure-Ruby gsub, same patterns
 bundle exec ruby benchmark/throughput.rb         # MB/s on a log line, JSON, 1MB and 10MB log files
 bundle exec ruby benchmark/scaling.rb            # runtime vs input size (1KB → 50MB), confirms linear scaling
 bundle exec ruby benchmark/per_pattern.rb        # per-pattern scan cost over a 1MB payload
@@ -535,11 +535,8 @@ C engine uses, via `DataRedactor::BUILTIN_PATTERN_SOURCES`).
 
 ### Performance (0.10.0 — v19 multi-pattern engine)
 
-As of 0.10.0 the C extension runs a **Thompson NFA → lazy-DFA multi-pattern
-engine** (v19) that scans the input once across all 88 built-in patterns,
-with two selective-merge passes (pure-digit group + IBAN union) that further
-reduce work for the most common pattern classes. Custom patterns (`add_pattern`)
-still use the glibc path (required for correct UTF-8 diacritic matching).
+Measured on the v19 engine ([How it works](#how-it-works)) vs a pure-Ruby `gsub`
+loop over the same patterns:
 
 | Payload               | v19 engine (0.10.0) | Pure-Ruby `gsub` | Ratio           |
 |-----------------------|---------------------|------------------|-----------------|
@@ -576,14 +573,14 @@ machine-dependent, but the flat curve is not.
 
 ## How it works
 
-1. At load time, `Init_data_redactor` compiles all 85 regex patterns once using `regcomp` (POSIX ERE) and stores them as static `regex_t` structs. Patterns marked as boundary-wrapped are expanded with `wrap_boundary()` before compilation.
-2. `DataRedactor.redact(text)` receives a Ruby `String`, converts it to a C `char*` via `StringValueCStr`, and runs each compiled pattern in sequence on a working buffer.
-3. For each pattern, `replace_all_matches` iterates using `regexec`, copies non-matching segments to a fresh output buffer, and inserts `[REDACTED]` in place of each match. For boundary-wrapped patterns, `regexec` is called with `nmatch=4` and sub-match groups `[1]`/`[3]` identify the boundary characters so they are preserved verbatim.
-4. The output buffer is grown with `realloc` as needed. After all patterns are applied the result is returned as a Ruby `String` via `rb_str_new_cstr`. All intermediate `malloc`/`strdup` allocations are explicitly `free`d.
+1. At load time, `mm_init()` compiles every built-in pattern from a Thompson NFA into bytecode, lazily building each pattern's DFA on first use (interned and cached). Boundary-wrapped patterns are expanded with the word-boundary group before compilation.
+2. `DataRedactor.redact(text)` / `scan(text)` hand the input to the v19 engine, which scans it **once** and emits `(pattern_id, start, length)` events for every enabled pattern. Two selective-merge passes (a pure-digit group and an IBAN union) collapse the most common pattern classes into shared scans. The single pass over the original buffer is what makes the engine O(N).
+3. The raw events are resolved by `mm_resolve` under the **longest-match-wins** policy: overlapping spans are reduced to a non-overlapping set keeping the longest match at each position, with the lower pattern index breaking equal-length ties.
+4. `redact` rewrites the surviving spans to placeholders in one buffer build (preserving the boundary characters of boundary-wrapped matches); `scan` returns the event list with byte offsets into the original string. Custom patterns (`add_pattern`) run on the glibc `regexec` path afterward — required for correct UTF-8 diacritic matching.
 
 ## Memory management
 
-All C-side buffers are heap-allocated with `malloc`/`strdup` and freed before the function returns. The only Ruby-managed allocation is the final return value from `rb_str_new_cstr`. No Ruby objects are created mid-processing, so GC cannot collect anything out from under the C code.
+All C-side working buffers are heap-allocated and freed before the call returns; the only Ruby-managed allocation is the final result `String`. No Ruby objects are created mid-scan, so GC cannot collect anything out from under the C code. Per-thread engine scratch (NFA state, lazy-DFA cache) is freed automatically when the thread exits — see [Thread safety](#thread-safety).
 
 ## Thread safety
 
@@ -601,7 +598,6 @@ Released under the [MIT License](LICENSE).
 
 ## Known limitations
 
-- **Pattern ordering matters** — patterns run sequentially. An early broad pattern (e.g. the 9-digit passport) may consume digits that a later pattern (e.g. credit card) depends on. Boundary wrapping mitigates this for pure-digit patterns.
 - **AWS Secret Key (pattern 1)** — 40 consecutive base64 characters is a broad match. It can produce false positives in base64-encoded content such as embedded images or binary blobs.
 - **Duplicate digit patterns** — several national ID formats share the same digit-length (11 digits: PESEL, Norwegian Fødselsnummer, Belgian National Number). They are kept as separate slots for clarity but the practical effect is that any 11-digit boundary-delimited number will be redacted.
-- **Single-pass overlap semantics** — built-in patterns are resolved by an index-order greedy claim: the lower-index pattern wins any region it matches. When two secrets abut with no separator, a rewrite-created word boundary can cause the second to be missed. This is rare in real text (secrets are almost always separator-delimited) and will be fixed by the upcoming longest-match-wins resolver in 1.0.
+- **Overlap resolution is longest-match-wins** — when two patterns match overlapping spans the engine keeps the longer span; equal-length ties go to the lower pattern index. This favours redacting *more* when uncertain (a 40-char secret is redacted whole rather than leaking the bytes past a shorter prefix match). When two secrets abut with **no separator** between them, a boundary-wrapped pattern can fail to match because the original buffer has no word boundary where one token meets the next, leaving the abutting token unredacted. This is rare in real text (secrets are almost always separator-delimited).

data/lib/data_redactor/version.rb

@@ -1,4 +1,4 @@
 module DataRedactor
   # Current gem version. Follows {https://semver.org Semantic Versioning 2.0.0}.
-  VERSION = "0.14.0"
+  VERSION = "0.15.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: data_redactor
 version: !ruby/object:Gem::Version
-  version: 0.14.0
+  version: 0.15.0
 platform: x86_64-linux
 authors:
 - Daniele Frisanco